Tactile media content processing method and device, equipment and storage medium

By generating and converting the tactile media exchange format, the problem of difficulty in expressing and transmitting multiple tactile media content in the prior art is solved, and efficient tactile media content transmission and rich user experience are achieved.

CN119065488APending Publication Date: 2024-12-03TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310644146.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The prior art is difficult to express and transmit a variety of tactile media content in a unified manner, resulting in the user's tactile experience when consuming media.

Method used

A method for processing tactile media content is proposed, and by generating a tactile media exchange format, it includes at least one of a tactile experience object, a person information object, a device information object, a tactile mode object, a tactile channel object, a tactile event object, and a component component object. The data object is converted into a haptic media binary bitstream, encapsulated as a haptic media file to transfer to the client.

Benefits of technology

It realizes efficient, unified expression and streamlined tactile media content, improves the user's tactile media consumption experience, and supports the rendering and presentation of a variety of tactile signals.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065488A_ABST
    Figure CN119065488A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a tactile media content processing method and device, equipment and a storage medium. The method comprises the following steps: acquiring at least one haptic media signal for presenting an expected haptic media effect; generating a tactile media exchange format according to the at least one tactile media signal, wherein the tactile media exchange format comprises at least one data object of a tactile experience object, a figure information object, a device information object, a tactile mode object, a tactile channel object, a tactile event object and a component component object; performing conversion processing on the tactile media exchange format to obtain a tactile media binary bit stream; and packaging the tactile media binary bit stream into a tactile media file, and transmitting the tactile media file to a client. The invention provides a universal tactile media content representation mode, supports the conversion from the tactile media exchange format to the tactile media binary bit stream, and also realizes the streaming transmission of the tactile media content.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of multimedia data processing, and in particular to a method, device, equipment and computer-readable storage medium for processing tactile media content. Background Art

[0002] The presentation of immersive media content is often accompanied by a variety of wearable devices or interactive devices. Therefore, in addition to the traditional visual and auditory presentations, immersive media also has a tactile presentation method. With the popularization of wearable and interactive devices, the tactile presentation that users can perceive when consuming media content will no longer be limited to basic vibrotactile sensations, but will include a more real-world tactile presentation experience that includes all-round physical sensations such as vibration, pressure, speed, acceleration, temperature, humidity, and smell. Therefore, how to uniformly express tactile media content is a pain point in tactile media consumption. Summary of the invention

[0003] The embodiments of the present application respectively provide a method for processing tactile media content, an apparatus for processing tactile media content, a device for processing tactile media content, a computer-readable storage medium, and a computer program product.

[0004] On the one hand, an embodiment of the present application provides a method for processing tactile media content, the method comprising: obtaining at least one tactile media signal; generating a tactile media exchange format according to the at least one tactile media signal, the tactile media exchange format comprising at least one data object selected from the group consisting of a tactile experience object, a character information object, a device information object, a tactile mode object, a tactile channel object, a tactile event object, and a component component object; converting the tactile media exchange format to obtain a tactile media binary bit stream; encapsulating the tactile media binary bit stream into a tactile media file, and transmitting the tactile media file to a client.

[0005] On the other hand, an embodiment of the present application also provides a device for processing tactile media content, which includes: a signal acquisition module, configured to acquire at least one tactile media signal; an exchange format generation module, configured to generate a tactile media exchange format according to the at least one tactile media signal, wherein the tactile media exchange format includes at least one data object selected from the group consisting of a tactile experience object, a character information object, a device information object, a tactile mode object, a tactile channel object, a tactile event object, and a component component object; an exchange format conversion module, configured to convert the tactile media exchange format to obtain a tactile media binary bit stream; and an encapsulation and transmission module, configured to encapsulate the tactile media binary bit stream into a tactile media file and transmit the tactile media file to a client.

[0006] On the other hand, an embodiment of the present application also provides a tactile media content processing device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the tactile media content processing device implements the steps in the tactile media content processing method as described above.

[0007] On the other hand, an embodiment of the present application further provides a computer-readable storage medium having computer-readable instructions stored thereon. When the computer-readable instructions are executed by a processor of a computer, the computer executes the steps in the method for processing tactile media content as described above.

[0008] On the other hand, an embodiment of the present application further provides a computer program product, including a computer program, which implements the steps in the method for processing tactile media content as described above when executed by a processor.

[0009] The embodiments of the present application propose a universal method for representing tactile media content. Specifically, different types of tactile media signals are represented accordingly by including at least one data object in a tactile media exchange format document, while supporting conversion from the tactile media exchange format to a tactile media binary bit stream, thereby achieving a unified expression of tactile media content. Furthermore, the tactile media binary bit stream obtained by conversion is encapsulated into a tactile media file, and the tactile media file is transmitted to a client, thereby achieving streaming transmission of tactile media content.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 The data structure relationship of each data object in the tactile media exchange format proposed in the embodiment of the present application is shown;

[0012] Figure 2 A flowchart of a method for processing tactile media content proposed in an embodiment of the present application is shown;

[0013] Figure 3 A block diagram of a device for processing tactile media content proposed in an embodiment of the present application is shown;

[0014] Figure 4 A schematic diagram of the structure of a computer system of a device for processing tactile media content suitable for implementing an embodiment of the present application is shown. DETAILED DESCRIPTION

[0015] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0016] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0017] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0018] The term "plurality" used in this application refers to two or more than two. "And / or" describes the relationship between the associated objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0019] The terms "first", "second", "third" and "fourth" in the specification and claims of this application and the drawings are used to distinguish different objects of use rather than to describe a specific order. The terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0020] First of all, it should be noted that immersive media can be called immersive or immersive media. In a broad sense, the use of audio and video technology to create an immersive feeling is called immersive media. For example, a phone call allows the parties to talk as if they were in person, and another example is watching a movie on TV. Although the viewer is sitting at home, he seems to be on the scene of the movie. The presentation of immersive media content is often accompanied by a variety of wearable devices or interactive devices. Therefore, in addition to the traditional visual and auditory presentation, immersive media also has a tactile presentation method. It can be understood that touch is the sensory experience obtained by the human body through contact, such as vibration, pressure, temperature, speed, electrotactile, etc.

[0021] Haptic presentation uses a combination of hardware and software to allow characters to receive information through their body parts to gain an embedded physical sense, conveying key information about the system being used. For example, a mobile phone vibrates to alert its owner that a message has been received. This vibration is a type of haptic presentation. In practical applications, haptic presentation can enhance auditory and visual presentation and improve the character experience.

[0022] At present, tactile information presentation has been widely used, and is generally divided into three categories: vibrotactile, kinematic tactile, and electrotactile. Vibrotactile directly presents tactile sensations in the form of vibrations, such as simulating vibrations of a specific frequency and intensity through the vibration of the motor of the terminal device, such as simulating specific shooting effects through vibrations in shooting games. Kinematic tactile presents tactile sensations by simulating the motion state of an object, such as the weight or pressure of an object. For example, in a driving video game, when moving at a higher speed or operating a heavier vehicle, the steering wheel may resist turning. This feedback directly affects the muscles of the game player, and the game player must apply more force to get the desired response from the steering wheel. Electrotactile presents tactile sensations by simulating feedback information of a specific texture through electrical stimulation, that is, electrotactile presentation uses electrical pulses to provide tactile stimulation to the nerve endings of the character's skin, such as creating a highly realistic experience for a character wearing a suit or gloves equipped with electrotactile technology.

[0023] It can be seen that in the current tactile presentation applications in the industry, from the vibration tactile of commonly used devices in daily life to the diversified tactile presentation in subdivided fields, tactile presentation itself has become a presentation method that people are accustomed to. With the popularization and development of wearable devices and interactive devices, the tactile presentation that people can perceive when consuming media content will no longer be limited to basic vibration tactile, but will include vibration, pressure, speed, acceleration, temperature, humidity, smell and other full-dimensional body sensations that are closer to the real world.

[0024] In order to express tactile media content efficiently and uniformly to enhance the character experience of tactile media consumption, the embodiments of the present application provide a tactile media content processing method, a tactile media content processing device, a tactile media content processing equipment, a computer scale storage medium, and a computer program product.

[0025] Before describing the embodiments of the present application in detail, a general tactile media content representation method proposed in the present application is explained first.

[0026] The universal representation method of tactile media content proposed in this application mainly includes two parts. The first part is the generation of a tactile media exchange format based on the tactile media content, and the second part is the conversion from the tactile media exchange format to a tactile media binary bit stream. These two parts will be described in detail below.

[0027] First, the tactile media exchange format mentioned in the present application is used to represent a tactile experience, corresponding to at least one tactile media signal, and the tactile media signal can be understood as a signal used to represent a tactile experience of a specific modality and rendered on a specific device.

[0028] In order to facilitate the processing of tactile media signals, the tactile media exchange format defined in this application is the JSON format, but it does not mean that the tactile media exchange format can only be limited to the JSON format. It can be understood that JSON, also known as "object notation", is a lightweight data exchange format that uses a text format that is completely independent of the programming language to store and represent data. The concise and clear hierarchical structure makes JSON an ideal data exchange language that is easy for people to read and write, and is also easy for machines to parse and generate, and effectively improves network transmission efficiency.

[0029] The various data structures included in the tactile media exchange format proposed in this application are shown in Table 1 below:

[0030]

[0031] Table 1

[0032] It should be noted that this application refers to AVS_haptics in Table 1 above as a tactile experience object, AVS_haptics_avatar as a character information object, AVS_haptics_device as a device information object, AVS_haptics_pattern as a tactile pattern object, AVS_haptics_channel as a tactile channel object, AVS_haptics_event as a tactile event object, and AVS_haptics_event_component as a component component object. These data objects constitute the basic data format of the tactile media exchange format. For the description of each data object, please refer to the records in Table 1 above, which will not be repeated here.

[0033] It should also be noted that the tactile media exchange format generated in an actual application scenario may only include at least one data object shown in Table 1 above. For example, a tactile experience may not contain character information or device information, and the generated tactile media exchange format will not contain the corresponding character object or device information object.

[0034] See also Figure 1 , Figure 1 The data structure connection of each data object in the tactile media exchange format proposed in the present application is shown. It can be seen that there is a hierarchical attribution association between the tactile experience object, the tactile pattern object, the tactile channel object and the tactile event object, and the hierarchy between them is in descending order. The tactile experience object is the highest-level element contained in the tactile media exchange format, which indicates the tactile experience and can also be understood as the information entry of the tactile experience. A tactile experience corresponds to tactile experience metadata and at least one tactile pattern, and may contain relevant data of at least one specific device for presenting the tactile experience. A tactile pattern corresponds to tactile pattern metadata and at least one tactile channel, and may contain relevant data of knowledge events. It can be understood that a knowledge event refers to a tactile event that occurs repeatedly in a tactile experience. Defining a recurring tactile event as a knowledge event can make the generated tactile media exchange format more concise. A tactile channel corresponds to tactile channel metadata and at least one tactile event. A tactile event corresponds to tactile event metadata and at least one tactile event component.

[0035] The following describes the attributes contained in each of the data objects contained in the tactile media exchange format.

[0036] Table 2 below shows the attribute definitions contained in the tactile experience object AVS_haptics:

[0037]

[0038]

[0039] Table 2

[0040] In this application, version in Table 2 above is referred to as the version attribute, creation_date is referred to as the creation time attribute, description is referred to as the description information attribute, event_layer_flag is referred to as the attribute indicating whether the tactile event is divided into layers, avatar is referred to as the attribute indicating the character information related to the current tactile experience, device is referred to as the attribute indicating the device information related to the current tactile experience, and pattern is referred to as the attribute indicating the tactile pattern related to the current tactile experience.

[0041] It should be noted that the "necessary" items mentioned in Table 2 above indicate that the corresponding attributes may not exist in the tactile experience object in actual application scenarios. For example, as mentioned above, the tactile experience in actual application scenarios may not contain character information or device information, so the avatar or device attributes will not exist in the tactile experience object.

[0042] It should also be noted that event_layer_flag needs to be included in one of the data objects among the tactile experience object, tactile attribute object, and tactile channel object, and the hierarchical division used to represent the tactile event is limited to the corresponding data object. For example, when the tactile attribute object contains event_layer_flag, the values ​​of tactile events under different tactile attributes cannot be compared with each other.

[0043] Table 3 below shows the attribute definitions contained in the character information object AVS_haptics_avatar:

[0044]

[0045] Table 3

[0046] In this application, id in Table 3 above is referred to as the attribute indicating the identifier of the character, avatar_model_type is referred to as the attribute indicating the type of the character model, and avatar_model_uri is referred to as the attribute indicating the URI address of the character model file. It should be understood that the character mentioned in this application may refer to the recipient of the tactile experience, usually the user of a wearable device or an interactive device. A character model refers to an abstract model used to represent a body part of a character, and the type of the character model is the specification followed by the definition of the character model.

[0047] It should also be noted that avatar_model_uri is an optional attribute in the character information object, which can be understood as this application does not limit the character model file to be stored on an Internet device such as a resource storage server, for example, it may also be stored locally on the current device. When the character model file is stored on an Internet device, it is necessary to obtain and use the character model file according to the corresponding URI (Uniform Resource Identifier).

[0048] Table 4 below shows the attribute definitions contained in the device information object AVS_haptics_device:

[0049]

[0050]

[0051] Table 4

[0052] In this application, the id in Table 4 above is referred to as the device identification attribute, device_name is referred to as the device name attribute, refer_avatar is referred to as the attribute indicating the character corresponding to the current device, refer_avatar_parts is referred to as the attribute indicating the specific information of the character corresponding to the current device, and body_parts is referred to as the attribute indicating the body part of the character corresponding to the current device.

[0053] An exemplary definition of a character's body parts is shown in Table 5 below:

[0054]

[0055] Table 5

[0056] Table 6 below shows the attribute definitions contained in the haptic pattern object AVS_haptics_pattern:

[0057]

[0058]

[0059] Table 6

[0060] In this application, the id in Table 6 above is referred to as the mode identifier attribute, pattern_type is referred to as the mode type attribute, refer_devices is referred to as the attribute indicating the device corresponding to the current mode, refer_avatars is referred to as the attribute indicating the character corresponding to the current mode, event_semantics_flag is referred to as the attribute indicating whether the tactile event corresponding to the current mode has semantics, event_semantics_type is referred to as the attribute indicating the specification followed by the semantics of the tactile event, event_layer_flag is referred to as the attribute indicating whether the tactile event is divided into layers, library_events is referred to as the attribute indicating the knowledge tactile event contained in the current mode, and channels is referred to as the attribute indicating the tactile channels contained in the current mode.

[0061] It should be noted that event_semantics_type is a conditional required attribute. It can be understood that when the value of the event_semantics_flag attribute is true, it means that the tactile event corresponding to the current mode has semantics, so the event_semantics_type attribute must exist.

[0062] In addition, as an exemplary implementation, event_semantics_type takes a value of 0 to indicate that the semantics of the tactile event follows the specifications defined in this standard, takes a value of 1 to indicate that the semantics of the tactile event follows the specifications of an external standard, and takes a value of 2 to indicate that the semantics of the tactile event is user-defined. The standard mentioned here refers to the format standard of the tactile media exchange format mentioned in this application.

[0063] Table 7 below shows the attribute definitions contained in the haptic channel object AVS_haptics_channel:

[0064]

[0065]

[0066] Table 7

[0067] In this application, the id in Table 7 above is referred to as the channel identifier attribute, description is referred to as the channel description attribute, refer_devices is referred to as the attribute indicating the device corresponding to the current channel, refer_avatars is referred to as the attribute indicating the character corresponding to the current channel, gain is referred to as the channel gain attribute, channel_alternative_flag is referred to as the attribute indicating whether the current channel is in an interchangeable relationship with other tactile channels, alternative_group_id is referred to as the attribute indicating the identifier of the alternative channel group to which the current channel belongs, channel_group_flag is referred to as the attribute indicating whether the current channel belongs to a certain channel group, channel_group_id is referred to as the attribute indicating the identifier of the channel group to which the current channel belongs, mixing_weight is referred to as the attribute indicating the weight of the current channel in the corresponding channel group, event_layer_flag is referred to as the attribute indicating whether the tactile event is divided into layers, priority is referred to as the attribute indicating the priority of the current channel, and events is referred to as the attribute indicating the tactile events contained in the current channel.

[0068] It should be noted that when the replaceable channel group to which the current channel belongs exists, for the optional refer_devices, refer_avatars and priority, only the channel with the smallest identifier value in the channel group is indicated, and it is not repeated in the metadata of other channels in the channel group.

[0069] It should also be noted that the default values ​​of channel_alternative_flag and channel_group_flag are 0, which means that the current channel and other channels are not interchangeable by default, and the current channel does not belong to a channel group. In actual application scenarios, the attribute values ​​need to be modified according to actual needs. In addition, alternative_group_id only exists when channel_alternative_flag is true, and channel_group_id and mixing_weight only exist when channel_group_flag is true. priority is an optional attribute in the tactile channel object, which can be understood as not prioritizing different tactile channels in actual application scenarios.

[0070] Table 8 below shows the attribute definitions contained in the haptic event object AVS_haptics_event:

[0071]

[0072]

[0073] Table 8

[0074] In this application, the id in Table 8 above is referred to as the event identifier attribute, effect_semantic is referred to as the event semantic attribute, event_layer is referred to as the event layer attribute, event_type is referred to as the event type attribute, refer_library_event is referred to as the attribute of the knowledge tactile event identifier corresponding to the event type being a reference event, relative_time is referred to as the attribute indicating the time offset of the current event, duration is referred to as the attribute indicating the duration of the current event, amplitude is referred to as the attribute indicating the maximum amplitude value of the signal of the current event, base_frequency is referred to as the attribute indicating the signal base frequency of the current event, and event_components is referred to as the attribute indicating the components that make up the current event.

[0075] It should be noted that effect_semantic is a required attribute in the tactile event object only when the event_semantics_flag attribute of the tactile mode corresponding to the tactile event is true. event_layer is also a required attribute in the tactile event object only when the event_layer_flag field in the tactile experience, tactile mode or tactile channel corresponding to the tactile event is true. When the tactile event is presented, the low-level tactile events are presented first. If the level field does not exist in the tactile event, the level of the tactile event is 0 by default.

[0076] Table 9 below shows the attribute definition contained in the component object AVS_haptics_event_component:

[0077]

[0078] Table 9

[0079] In this application, refer_event_id in Table 9 above is referred to as the attribute indicating the identifier of the tactile event to which the current component belongs, relative_time is referred to as the attribute indicating the time offset of the current component relative to the tactile event, relative_amplitude is referred to as the attribute indicating the amplitude value ratio of the current component relative to the tactile event, and relative_frequency is referred to as the attribute indicating the frequency offset of the current component relative to the tactile event.

[0080] From the above Tables 1-9 and Figure 1It can be seen that the tactile media exchange format proposed in this application can be summarized as including the following information, so as to express the tactile media content efficiently and uniformly:

[0081] 1) Basic metadata of tactile media, presentation device information, and presentation location information;

[0082] 2) Organizing the tactile media content into different levels according to the signal type, signal rendering location and other principles, and indicating the metadata information of each level;

[0083] 3) Define the basic level of the tactile media signal according to the division methods such as time, space, and frequency, and indicate the metadata information of the basic level.

[0084] The present application also proposes a method of converting the tactile media exchange format shown above into a tactile media binary bit stream, so as to facilitate subsequent file packaging and transmission. The tactile media exchange format and the tactile media binary bit stream may correspond to each other in data structure.

[0085] The tactile media binary bit stream consists of a binary header and a binary payload, wherein the binary header contains metadata related to the tactile experience, tactile mode, and tactile channel, and the binary payload contains metadata related to the tactile event.

[0086] The tactile media exchange format is converted into a tactile media binary bit stream, that is, each attribute in each data object included in the tactile media exchange format is converted into a binary bit format according to a predefined attribute conversion rule.

[0087] The syntax for converting the tactile experience object into a binary format is shown in Table 10 below:

[0088]

[0089]

[0090] Table 10

[0091] Among them, versionStringSize, dateStringSize, and descriptionStringSize respectively indicate the number of bytes of the corresponding string, avatarCount, deviceCount, and patternCount respectively indicate the number of corresponding data structures, and the remaining fields and data structures correspond to the corresponding fields and data structures defined in the exchange format.

[0092] The syntax for converting the character information object into a binary format is shown in Table 11 below:

[0093]

[0094] Table 11

[0095] Among them, avatarModelUriStringSize indicates the number of bytes of the corresponding string, and the remaining fields and data structures correspond to the corresponding fields and data structures defined in the exchange format.

[0096] The syntax for converting the device information object to binary format is shown in Table 12 below:

[0097]

[0098]

[0099] Table 12

[0100] Among them, referAvatarFlag, referAvatarPartsFlag, and bodyPartsFlag respectively indicate whether to refer to character information, whether to refer to character body information, and whether to correspond to body part information; deviceNameStringSize and avatarPartsStringSize respectively indicate the length of the device name string and the length of the character body information string; the remaining fields and data structures correspond to the corresponding fields and data structures defined in the exchange format.

[0101] The syntax for converting the tactile mode object into a binary bit format is shown in Table 13 below:

[0102]

[0103]

[0104] Table 13

[0105] Among them, referDeviceCount, referAvatarCount, libraryEventCount, and channelCount respectively indicate the number of reference devices, the number of reference characters, the number of knowledge events, and the number of channels, libraryEventId indicates the identifier of the knowledge event, and the remaining fields and data structures correspond to the corresponding fields and data structures defined in the exchange format.

[0106] patternType indicates the pattern type, and its values ​​are shown in Table 14 below:

[0107] Value meaning 0 other 1 Vibrotactile 2 pressure 3 Acceleration 4 speed 5 temperature 6 Electrotactile 7-255 reserve

[0108] Table 14

[0109] The syntax for converting the tactile channel object into a binary bit format is shown in Table 15 below:

[0110]

[0111]

[0112] Table 15

[0113] Among them, descriptionStringSize indicates the number of bytes of the corresponding string, referDeviceCount, referAvatarCount, and eventCount respectively indicate the number of reference devices, the number of reference characters, and the number of events, alternativeFlag, groupFlag, weightFlag, and priorityFlag respectively indicate whether the current channel belongs to an alternative channel group, whether it belongs to a channel group, whether it contains weight information, and whether it contains priority information, eventId indicates the identifier of the event contained in the current channel, and the remaining fields and data structures correspond to the corresponding fields and data structures defined in the exchange format.

[0114] The syntax for converting the haptic event object to a binary format is shown in Table 16 below:

[0115]

[0116]

[0117]

[0118] Table 16

[0119] Among them, eventStartCode indicates the start code of the event; eventType takes a value of 0 to indicate an instantaneous event, a value of 1 to indicate a duration, and a value of 2 to indicate a reference time. For an instantaneous event, its duration is a fixed value, which usually depends on the device; referDeviceCount, referAvatarCount, and eventComponentCount indicate the number of reference devices, the number of reference characters, and the number of event components, respectively; libEventFlag takes a value of 1 to indicate that the current event is a knowledge event, and takes a value of 0 to indicate that the current event is not a knowledge event. Knowledge events do not have a time offset value; the remaining fields and data structures correspond to the corresponding fields and data structures defined in the exchange format; eventSemanticsStringSize indicates the string length of the tactile event semantics; eventSemantics indicates the semantics of the tactile event.

[0120] The syntax for converting the binary bit format of the component object is shown in Table 17 below:

[0121]

[0122]

[0123] Table 17

[0124] It can be seen from Tables 10-17 above that the attribute conversion method of the present application for converting the tactile media exchange format can be summarized as follows:

[0125] 1) Convert the attributes of string type, integer type, enum type or number type in each data object into binary bit stream segments of a number of integer bytes, wherein the byte length corresponding to each binary bit stream segment may be a preset value;

[0126] 2) Convert the Boolean attribute into a binary bit stream segment of several bits;

[0127] 3) Count the number of array elements contained in the array type array attribute, and convert the binary bit stream segment of the corresponding attribute based on the number of array elements.

[0128] Of course, as an optional implementation, the byte length of the binary bit stream segment obtained after converting the attributes of the above-mentioned data types can also be used as a corresponding adjustment, and it is not limited to the byte length shown in Table 10-17.

[0129] It should also be noted that when converting the attributes in the tactile event object, when the eventSemanticsType value in the tactile mode object is 0, that is, the semantics of the tactile event follows the current standard, the tactile event semantics can be defined in one of the following ways:

[0130] 1) Defined as a two-layer semantic structure, with different levels of semantics separated by spaces or " / ", for example, "texture / rock" means that the tactile event simulates the texture of stone;

[0131] 2) Defined as a three-layer semantic structure, with different levels of semantics separated by spaces or " / ", such as "game / weapon / shotgun", which means that the tactile event simulates the sound of a weapon in the game;

[0132] 3) It is defined as a multi-layer semantic structure, with different levels of semantics separated by spaces.

[0133] Therefore, optionally, when performing binary bit format conversion on the tactile event object, the conversion method definition of the above-mentioned hierarchical tactile semantics may also be as shown in the following Table 18:

[0134]

[0135]

[0136] Table 18

[0137] Optionally, for the semantics of the tactile event of the string type for readability, the conversion method can also be defined as a specific binary field, as shown in Table 18 below:

[0138]

[0139] Table 19

[0140] Take the 2-layer syntax as an example:

[0141] eventSemanticsLevelNum = 2;

[0142] semanticsLevelBitsNum[0]=4, eventSemantics[0]=0001, corresponding to texture;

[0143] semanticsLevelBitsNum[1]=4, eventSemantics[0]=0010, corresponding to rock.

[0144] Based on the general tactile media content representation method described above, an embodiment of the present application proposes a method for processing tactile media content, which is applied to tactile feedback related products, such as the server side, player side and intermediate node of an immersive system, etc., and is not limited here.

[0145] like Figure 2 As shown, the method comprises the following steps:

[0146] S210, acquiring at least one tactile media signal;

[0147] S220, generating a tactile media exchange format according to at least one tactile media signal, the tactile media exchange format comprising at least one data object among a tactile experience object, a character information object, a device information object, a tactile mode object, a tactile channel object, a tactile event object, and a component object;

[0148] S230, converting the tactile media exchange format to obtain a tactile media binary bit stream;

[0149] S240: encapsulate the tactile media binary bit stream into a tactile media file, and transmit the tactile media file to the client.

[0150] It should be noted that the tactile media information obtained in S210 may be produced or collected according to the expected tactile media effect, for example. The method of generating the tactile media exchange format in S220 and the method of converting the tactile media binary bit stream in S230 have been embodied in the previously recorded content, and will not be repeated here. S240 encapsulates the tactile media binary bit stream into a tactile media file, for example, it may be encapsulated based on ISOBMFF (ISO Based Media File Format, a media file format based on ISO standards), or it may be encapsulated in other media file formats, and this is not limited here. The encapsulated tactile media file may be streamed to the client, for example, the tactile media file may be sent to the client in its entirety, or at least one file fragment may be generated based on the tactile media file, and the at least one file fragment may be transmitted to the client. After receiving the file or the file fragment, the client may obtain the tactile media file containing the tactile signal by parsing the file or the file fragment, and obtain the tactile media signal after decapsulation and decoding and rendering. In this way, efficient and unified expression of tactile media content may be achieved.

[0151] It should also be noted that in streaming transmission, the server needs to send a signaling message describing the media resource to the client, and the client then requests a specific media resource according to the signaling message. Therefore, in this embodiment, the tactile media file is transmitted to the client after receiving the corresponding request signaling sent by the client.

[0152] The following is an example of each data object in the tactile media exchange format generated in S220. It should be noted that the definition of the attributes included in each of the following data objects can be found in the above records, which will not be repeated here.

[0153] The tactile experience object includes at least one of a version attribute, a creation time attribute, a description information attribute, an attribute indicating a tactile mode related to the current tactile experience, an attribute indicating character information related to the current tactile experience, an attribute indicating device information related to the current tactile experience, and an attribute indicating whether the tactile event is divided into levels.

[0154] The character information object includes at least one of an attribute indicating an identifier of a character, an attribute indicating a type of a character model, and an attribute indicating a URI address of a character model file.

[0155] The device information object includes at least one of a device identification attribute, a device name attribute, an attribute indicating a person corresponding to the current device, an attribute indicating specific information of the person corresponding to the current device, and an attribute indicating a body part of the person corresponding to the current device.

[0156] The tactile mode object includes at least one of a mode identifier attribute, a mode type attribute, an attribute indicating a device corresponding to the current mode, an attribute indicating a character corresponding to the current mode, an attribute indicating whether a tactile event corresponding to the current mode has semantics, an attribute indicating the specification followed by the semantics of the tactile event, an attribute indicating whether the tactile event is divided into levels, an attribute indicating knowledge tactile events included in the current mode, and an attribute indicating a tactile channel included in the current mode.

[0157] The tactile channel object includes at least one of a channel identifier attribute, a channel description attribute, an attribute indicating a device corresponding to the current channel, an attribute indicating a character corresponding to the current channel, a channel gain attribute, an attribute indicating whether the current channel is in a replaceable relationship with other tactile channels, an attribute indicating an identifier of a replaceable channel group to which the current channel belongs, an attribute indicating whether the current channel belongs to a certain channel group, an attribute indicating an identifier of the channel group to which the current channel belongs, an attribute indicating a weight of the current channel in the corresponding channel group, an attribute indicating whether tactile events are hierarchical, an attribute indicating a priority of the current channel, and an attribute indicating tactile events included in the current channel;

[0158] Among them, when there is a replaceable channel group to which the current channel belongs, among the attributes indicating the device corresponding to the current channel, the attributes indicating the person corresponding to the current channel, and the attributes indicating the priority of the current channel, only the tactile channel with the smallest identifier in the replaceable channel group is indicated.

[0159] The tactile event object includes at least one of an event identifier attribute, an event semantic attribute, an event level attribute, an event type attribute, an attribute indicating the knowledge tactile event identifier corresponding to the event type being a reference event, an attribute indicating the time offset of the current event, an attribute indicating the duration of the current event, an attribute indicating the maximum amplitude value of the signal of the current event, an attribute indicating the signal reference frequency of the current event, and an attribute indicating the components that constitute the current event.

[0160] The component component object includes at least one of an attribute indicating an identifier of the tactile event to which the current component belongs, an attribute indicating a time offset of the current component relative to the tactile event, an attribute indicating a ratio of an amplitude value of the current component relative to the tactile event, and an attribute indicating a frequency offset of the current component relative to the tactile event.

[0161] It should also be noted that the tactile media binary bit stream mentioned in S230 is composed of a binary header and a binary payload. In an exemplary embodiment, S230 specifically includes the following steps:

[0162] S231, performing binary bit format conversion on each attribute in each data object included in the tactile media exchange format according to a predefined attribute conversion method, to obtain a binary stream segment corresponding to each data object;

[0163] S232, forming a binary header based on the binary bit stream segments corresponding to the tactile experience object, the tactile mode object and the tactile channel object, and forming a binary payload based on the binary bit stream segments corresponding to the tactile event object and the component object, so as to obtain a tactile media binary bit stream.

[0164] The predefined attribute conversion method mentioned in S231 includes at least one of the following:

[0165] 1) Convert the string type, integer type, enumeration type or numeric type attribute into a binary bit stream segment of a certain integer number of bytes;

[0166] 2) Convert the Boolean attribute into a binary bit stream segment of several bits;

[0167] 3) Counting the number of array elements contained in the array-type attribute, and converting the binary bit stream segment of the corresponding attribute based on the number of array elements;

[0168] 4) When converting the tactile event object into a binary bit format, adding a start code corresponding to the corresponding tactile event as a new attribute of the tactile event object, and converting to obtain a corresponding binary bit stream segment;

[0169] 5) When converting the event semantic attributes contained in the tactile event object into binary bit format, if the specification followed by the semantics of the corresponding tactile event is the current standard, the semantic conversion of the corresponding tactile event is performed according to the semantic structure of the preset level, and the corresponding binary bit stream segment is obtained based on the converted semantics. The semantic structure of the preset level indicates that the semantics of different levels are divided by specified symbols, or converted into specified binary fields.

[0170] It should be noted that the contents of any of the above records have been described in detail in the aforementioned records, so they will not be elaborated on here.

[0171] Figure 3 FIG. 1 is a block diagram of a tactile media content processing device shown in an exemplary embodiment of the present application. Figure 3 As shown, the exemplary tactile media content processing device 300 includes:

[0172] A signal acquisition module 310, configured to acquire at least one tactile media signal;

[0173] The exchange format generating module 320 is configured to generate a tactile media exchange format according to at least one tactile media signal, wherein the tactile media exchange format includes at least one data object of a tactile experience object, a character information object, a device information object, a tactile mode object, a tactile channel object, a tactile event object, and a component component object;

[0174] The exchange format conversion module 330 is configured to convert the tactile media exchange format to obtain a tactile media binary bit stream;

[0175] The packaging and transmission module 340 is configured to package the haptic media binary bit stream into a haptic media file and transmit the haptic media file to the client.

[0176] In another exemplary embodiment, the tactile media binary bit stream is composed of a binary header and a binary payload, and the exchange format conversion module 330 includes:

[0177] a conversion unit configured to perform binary bit format conversion on each data object included in the tactile media exchange format based on a predefined attribute conversion method;

[0178] The bit stream acquisition unit is configured to use the binary bit data corresponding to the converted tactile experience object, tactile mode object and tactile channel object as a binary header, and use the binary bit data corresponding to the converted tactile event object and component object as a binary payload to obtain the tactile media binary bit stream.

[0179] It should be noted that the tactile media content processing device provided in the above embodiment and the tactile media content processing method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In actual application, the tactile media content processing device provided in the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0180] An embodiment of the present application also provides a tactile media content processing device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the tactile media content processing device implements the tactile media content processing method provided in the above-mentioned embodiments.

[0181] Figure 4The schematic diagram of the structure of the computer system of the tactile media content processing device suitable for implementing the embodiment of the present application is shown. It should be noted that: Figure 4 The computer system 400 of the tactile media content processing device shown is only an example and should not bring any limitation to the functions and application scope of the embodiments of the present application.

[0182] like Figure 4 As shown, the computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 402 or the program loaded from the storage part 408 to the random access memory (RAM) 403, such as executing the method described in the above embodiment. In the RAM 403, various programs and data required for system operation are also stored. The CPU 401, the ROM 402 and the RAM 403 are connected to each other through the bus 404. The input / output (I / O) interface 405 is also connected to the bus 404.

[0183] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed so that a computer program read therefrom is installed into the storage section 408 as needed.

[0184] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 409, and / or installed from a removable medium 411. When the computer program is executed by a central processing unit (CPU) 401, various functions defined in the system of the present application are executed.

[0185] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer program contained in the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0186] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0187] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0188] Another aspect of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for processing tactile media content as described above is implemented. The computer-readable storage medium may be included in the tactile media content processing device described in the above embodiment, or may exist independently without being assembled into the tactile media content processing device.

[0189] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method for processing tactile media content provided in each of the above embodiments.

[0190] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

[0191] It should also be noted that in the specific implementation of the present application, data related to tactile media content, character information, device information, etc. is involved. When the above embodiments of the present application are applied to specific products or technologies, the user's separate permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

Claims

1. A method for processing tactile media content, characterized in that: The method comprises: acquiring at least one tactile media signal; generating a tactile media exchange format according to the at least one tactile media signal, wherein the tactile media exchange format includes at least one data object selected from the group consisting of a tactile experience object, a character information object, a device information object, a tactile mode object, a tactile channel object, a tactile event object, and a component component object; Converting the tactile media exchange format to obtain a tactile media binary bit stream; The haptic media binary bit stream is encapsulated into a haptic media file, and the haptic media file is transmitted to a client.

2. The method according to claim 1, characterized in that The tactile experience object includes at least one of a version attribute, a creation time attribute, a description information attribute, an attribute indicating a tactile mode related to a current tactile experience, an attribute indicating character information related to a current tactile experience, an attribute indicating device information related to a current tactile experience, and an attribute indicating whether a tactile event is divided into levels.

3. The method according to claim 1, characterized in that The character information object includes at least one of an attribute indicating an identifier of a character, an attribute indicating a type of a character model, and an attribute indicating a URI address of a character model file.

4. The method according to claim 1, characterized in that: The device information object includes at least one of a device identification attribute, a device name attribute, an attribute indicating a person corresponding to the current device, an attribute indicating specific information of the person corresponding to the current device, and an attribute indicating a body part of the person corresponding to the current device.

5. The method according to claim 1, characterized in that: The tactile mode object includes at least one of a mode identifier attribute, a mode type attribute, an attribute indicating a device corresponding to the current mode, an attribute indicating a character corresponding to the current mode, an attribute indicating whether a tactile event corresponding to the current mode has semantics, an attribute indicating the specification followed by the semantics of the tactile event, an attribute indicating whether the tactile event is divided into levels, an attribute indicating knowledge tactile events included in the current mode, and an attribute indicating a tactile channel included in the current mode.

6. The method according to claim 1, characterized in that The tactile channel object includes at least one of a channel identifier attribute, a channel description attribute, an attribute indicating a device corresponding to the current channel, an attribute indicating a character corresponding to the current channel, a channel gain attribute, an attribute indicating whether the current channel is in a replaceable relationship with other tactile channels, an attribute indicating an identifier of a replaceable channel group to which the current channel belongs, an attribute indicating whether the current channel belongs to a certain channel group, an attribute indicating an identifier of the channel group to which the current channel belongs, an attribute indicating a weight of the current channel in the corresponding channel group, an attribute indicating whether tactile events are divided into levels, an attribute indicating a priority of the current channel, and an attribute indicating tactile events included in the current channel; Among them, when there is a replaceable channel group to which the current channel belongs, among the attributes indicating the device corresponding to the current channel, the attributes indicating the person corresponding to the current channel, and the attributes indicating the priority of the current channel, only the tactile channel with the smallest identifier in the replaceable channel group is indicated.

7. The method according to claim 1, characterized in that The tactile event object includes at least one of an event identifier attribute, an event semantic attribute, an event level attribute, an event type attribute, an attribute indicating the knowledge tactile event identifier corresponding to the event type being a reference event, an attribute indicating the time offset of the current event, an attribute indicating the duration of the current event, an attribute indicating the maximum amplitude value of the signal of the current event, an attribute indicating the signal reference frequency of the current event, and an attribute indicating the components that constitute the current event.

8. The method according to claim 1, characterized in that The component component object includes at least one of an attribute indicating an identifier of the tactile event to which the current component belongs, an attribute indicating a time offset of the current component relative to the tactile event, an attribute indicating a ratio of an amplitude value of the current component relative to the tactile event, and an attribute indicating a frequency offset of the current component relative to the tactile event.

9. The method according to any one of claims 1 to 8, characterized in that: The tactile media binary bit stream is composed of a binary header and a binary payload; the tactile media exchange format is converted to obtain the tactile media binary bit stream, including: According to a predefined attribute conversion method, performing binary bit format conversion on each attribute in each data object included in the tactile media exchange format to obtain a binary stream segment corresponding to each data object; A binary header is formed based on the binary bit stream segments corresponding to the tactile experience object, the tactile mode object and the tactile channel object, and a binary payload is formed based on the binary bit stream segments corresponding to the tactile event object and the component object, so as to obtain the tactile media binary bit stream.

10. The method according to claim 9, characterized in that The predefined attribute conversion method includes at least one of the following: Convert string, integer, enumeration or numeric attributes into binary bit stream segments of a certain integer number of bytes; Convert Boolean attributes into binary bit stream segments of several bits; The number of array elements contained in the array-type attribute is counted, and the binary bit stream segment of the corresponding attribute is converted based on the number of array elements.

11. The method according to claim 9, characterized in that The predefined field conversion method includes at least one of the following: When converting the tactile event object into a binary bit format, adding a start code corresponding to the corresponding tactile event as a new attribute of the tactile event object, and converting to obtain a corresponding binary bit stream segment; When the event semantic attributes contained in the tactile event object are converted into binary bit format, if the specification followed by the semantics of the corresponding tactile event is the current standard, the semantic conversion of the corresponding tactile event is performed according to the semantic structure of the preset level, and the corresponding binary bit stream segment is obtained based on the converted semantics. The semantic structure of the preset level indicates that the semantics of different levels are divided by specified symbols, or converted into specified binary fields.

12. The method according to any one of claims 1 to 8, characterized in that: The step of encapsulating the tactile media binary bit stream into a tactile media file and transmitting the tactile media file to a client comprises: sending the tactile media file to the client; Alternatively, at least one file segment is generated based on the tactile media file, and the at least one file segment is transmitted to the client.

13. A tactile media content processing device, characterized in that: The device comprises: a signal acquisition module, configured to acquire at least one tactile media signal; an exchange format generating module, configured to generate a tactile media exchange format according to the at least one tactile media signal, wherein the tactile media exchange format includes at least one data object selected from the group consisting of a tactile experience object, a character information object, a device information object, a tactile mode object, a tactile channel object, a tactile event object, and a component component object; An exchange format conversion module, configured to convert the tactile media exchange format to obtain a tactile media binary bit stream; The packaging and transmission module is configured to package the tactile media binary bit stream into a tactile media file and transmit the tactile media file to the client.

14. A device for processing tactile media content, characterized in that: include: one or more processors; A memory for storing one or more programs, which, when executed by the one or more processors, enables the tactile media content processing device to implement the tactile media content processing method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the method for processing tactile media content according to any one of claims 1 to 12.