Method and apparatus for processing tactile media content, and device, storage medium, program product and storage method
By defining the tactile media exchange format and converting it into binary bitstream, the problem of inconsistent tactile media content expression in the prior art is solved, efficient tactile media content transmission is achieved, and the tactile presentation quality of immersive media experience is improved.
Patent Information
- Application Number
- PCT/CN2024/095458
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2024-05-27
- Publication Date
- 2024-12-05
AI Technical Summary
The prior art is difficult to express and transmit multiple tactile media content in a unified manner, resulting in discontinuity and inconsistency of tactile presentation by users in immersive media experience.
A method and device for processing tactile media content is proposed, by defining the tactile media exchange format (including tactile experience objects, character information objects, device information objects, tactile mode objects, tactile channel objects, tactile event objects, component component objects) and It is converted into a binary bit stream and encapsulated into a tactile media file to achieve unified expression and streaming transmission.
It realizes efficient, unified expression and streamlined tactile media content, and improves the quality of tactile presentation of users in immersive media experience.
Smart Images

Figure CN2024095458_05122024_PF_FP_ABST
Abstract
Description
Method and device for processing tactile media content, equipment, storage medium, program product, and storage method Technical Field
[0001] The present application relates to the technical field of multimedia data processing, and in particular to a method, apparatus, device, computer-readable storage medium, computer program product, and storage method for processing tactile media content.
[0002] Background of the Invention
[0003] Immersive media content is often presented through various wearable or interactive devices. Therefore, in addition to traditional visual and auditory presentations, immersive media also includes tactile presentation. With the increasing popularity of wearable and interactive devices, the tactile experience users experience when consuming media content will no longer be limited to basic vibrotactile sensations. Instead, it will include a full range of haptic sensations, including vibration, pressure, speed, acceleration, temperature, humidity, and smell, offering a more realistic, tactile experience. Therefore, consistently presenting tactile media content is a major challenge for tactile media consumption.
[0004] Summary of the Invention
[0005] The embodiments of the present application provide a method for processing tactile media content, an apparatus for processing tactile media content, a device for processing tactile media content, a computer-readable storage medium, and a computer program product.
[0006] On the one hand, an embodiment of the present application provides a method for processing tactile media content, the method comprising: obtaining at least one tactile media signal; generating a tactile media exchange format based on the at least one tactile media signal, the tactile media exchange format comprising at least one data object selected from the group consisting of a tactile experience object, a character information object, a device information object, a tactile mode object, a tactile channel object, a tactile event object, and a component component object; converting the tactile media exchange format to obtain a tactile media binary bit stream; encapsulating the tactile media binary bit stream into a tactile media file, and transmitting the tactile media file to a client.
[0007] On the other hand, an embodiment of the present application also provides a device for processing tactile media content, which includes: a signal acquisition module, configured to acquire at least one tactile media signal; an exchange format generation module, configured to generate a tactile media exchange format based on the at least one tactile media signal, the tactile media exchange format including at least one data object selected from the group consisting of a tactile experience object, a character information object, a device information object, a tactile mode object, a tactile channel object, a tactile event object, and a component component object; an exchange format conversion module, configured to convert the tactile media exchange format to obtain a tactile media binary bit stream; and an encapsulation and transmission module, configured to encapsulate the tactile media binary bit stream into a tactile media file and transmit the tactile media file to a client.
[0008] On the other hand, an embodiment of the present application also provides a method for processing tactile media content, including: obtaining a tactile media file; or obtaining at least one file fragment generated based on the tactile media file, and parsing the at least one file fragment to obtain the tactile media file; decapsulating the tactile media file to obtain a tactile media binary bit stream; performing format inverse conversion on the tactile media binary bit stream to obtain a tactile media exchange format, wherein the tactile media exchange format includes at least one data object selected from the group consisting of a tactile experience object, a character information object, a device information object, a tactile mode object, a tactile channel object, a tactile event object, and a component component object; parsing the tactile media exchange format to obtain at least one tactile media signal; and rendering and presenting the at least one tactile media signal.
[0009] On the other hand, an embodiment of the present application also provides a device for processing tactile media content, including: a first module for obtaining a tactile media file; or obtaining at least one file fragment generated based on the tactile media file, and parsing the at least one file fragment to obtain the tactile media file; a second module for decapsulating the tactile media file to obtain a tactile media binary bit stream; a third module for performing format inverse conversion on the tactile media binary bit stream to obtain a tactile media exchange format, wherein the tactile media exchange format includes at least one data object selected from the group consisting of a tactile experience object, a character information object, a device information object, a tactile mode object, a tactile channel object, a tactile event object, and a component component object; a fourth module for parsing the tactile media exchange format to obtain at least one tactile media signal; and a fifth module for rendering and presenting the at least one tactile media signal.
[0010] On the other hand, an embodiment of the present application also provides a tactile media content processing device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the tactile media content processing device implements the steps in the tactile media content processing method as described above.
[0011] On the other hand, an embodiment of the present application further provides a computer-readable storage medium having computer-readable instructions stored thereon. When the computer-readable instructions are executed by a processor of a computer, the computer executes the steps in the method for processing tactile media content as described above.
[0012] On the other hand, an embodiment of the present application further provides a computer program product, including a computer program, which implements the steps of the above-mentioned method for processing tactile media content when executed by a processor.
[0013] On the other hand, an embodiment of the present application further provides a method for storing a tactile media file, characterized in that the tactile media file is stored on a non-volatile computer-readable storage medium, and the tactile media file is generated according to the method described above, or decoded based on the method described below.
[0014] The embodiments of the present application propose a universal method for representing tactile media content. Specifically, different types of tactile media signals are represented accordingly by including at least one data object in a tactile media exchange format document, while supporting conversion from the tactile media exchange format to a tactile media binary bit stream, thereby achieving a unified expression of tactile media content. Furthermore, the tactile media binary bit stream obtained by conversion is encapsulated into a tactile media file and the tactile media file is transmitted to a client, thereby achieving streaming transmission of tactile media content.
[0015] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.
[0016] BRIEF DESCRIPTION OF THE DRAWINGS
[0017] FIG1 shows the data structure relationship between various data objects in the tactile media exchange format proposed in an embodiment of the present application;
[0018] FIG2 shows a flow chart of a method for processing tactile media content proposed in an embodiment of the present application;
[0019] FIG3 shows a block diagram of a device for processing tactile media content according to an embodiment of the present application;
[0020] FIG4 shows a schematic structural diagram of a computer system of a device for processing tactile media content suitable for implementing an embodiment of the present application.
[0021] Modes for Carrying Out the Invention
[0022] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
[0023] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0024] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.
[0025] In this application, "plurality" refers to two or more. "And / or" describes the relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.
[0026] The terms "first," "second," "third," and "fourth," etc. in the specification, claims, and drawings of this application are used to distinguish different objects of use, rather than to describe a specific order. The terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.
[0027] First, it's important to note that immersive media can be referred to as "immersive" or "immersive media." Broadly speaking, immersive media is any media that uses audio and video technologies to create a sense of being present. For example, a phone call allows all parties to speak as if they were in person, or a movie on television allows viewers to feel as if they were right there on the set, even though they're sitting at home. Immersive media content is often presented through various wearable or interactive devices. Therefore, in addition to traditional visual and auditory presentations, immersive media also includes tactile presentation. As you can see, touch is the sensory experience the human body experiences through contact, such as vibration, pressure, temperature, speed, and electrotactile sensations.
[0028] Haptics, through a combination of hardware and software, allow characters to receive information through their body parts, providing an embedded physical sensation and conveying key information about the system they are using. For example, a phone vibrates to alert its owner of an incoming message; this vibration is a type of haptics. In practical applications, haptics can enhance auditory and visual perception, improving the character experience.
[0029] Currently, tactile information presentation has been widely used and is generally categorized into three types: vibrotactile, kinematic, and electrotactile. Vibrotactile is a type of tactile presentation that directly delivers tactile sensations through vibration. For example, the vibration of a terminal device's motor simulates vibrations of a specific frequency and intensity, such as in a shooting game where vibrations simulate the specific effects of shooting. Kinematic tactile is a type of tactile presentation that simulates the motion of an object, such as weight or pressure. For example, in a driving video game, when moving at high speeds or operating a heavy vehicle, the steering wheel may resist turning. This feedback directly affects the player's muscles, forcing them to apply more force to obtain the desired response from the steering wheel. Electrotactile is a type of tactile presentation that simulates feedback information of a specific texture through electrical stimulation. In other words, electrotactile presentation uses electrical pulses to deliver tactile stimulation to the nerve endings of a character's skin. This can create a highly realistic experience for a character wearing a suit or glove equipped with electrotactile technology.
[0030] It can be seen that in the current tactile presentation applications in the industry, from the vibration tactile sense of common daily devices to the diversified tactile presentation in niche fields, tactile presentation itself has become a presentation method that people are accustomed to. With the popularization and development of wearable and interactive devices, the tactile presentation that people can perceive when consuming media content will no longer be limited to basic vibration tactile. Instead, it will include vibration, pressure, speed, acceleration, temperature, humidity, smell, and other comprehensive physical sensations, a more real-world tactile presentation experience.
[0031] In order to efficiently and uniformly express tactile media content and enhance the user experience of tactile media consumption, embodiments of the present application provide a tactile media content processing method, a tactile media content processing apparatus, a tactile media content processing device, a computer scale storage medium, and a computer program product.
[0032] Before describing the embodiments of the present application in detail, a general tactile media content representation method proposed in the present application is first explained.
[0033] The universal representation method of tactile media content proposed in this application mainly includes two parts. The first part is the generation of a tactile media exchange format based on the tactile media content, and the second part is the conversion from the tactile media exchange format to a tactile media binary bit stream. These two parts will be described in detail below.
[0034] First, the tactile media exchange format mentioned in this application is used to represent a tactile experience, corresponding to at least one tactile media signal. The tactile media signal can be understood as a signal used to represent a tactile experience of a specific modality and rendered on a specific device.
[0035] To facilitate the processing of tactile media signals, this application defines the tactile media exchange format as JSON. However, this does not limit the tactile media exchange format to JSON. JSON, also known as "object notation," is a lightweight data exchange format that uses a text format completely independent of programming languages to store and represent data. Its concise and clear hierarchical structure makes JSON an ideal data exchange language, easy for humans to read and write, as well as easy for machines to parse and generate, effectively improving network transmission efficiency.
[0036] The various data structures included in the tactile media exchange format proposed in this application are shown in Table 1 below:
[0037] Table 1
[0038] It should be noted that this application refers to AVS_haptics in Table 1 above as a tactile experience object, AVS_haptics_avatar as a character information object, AVS_haptics_device as a device information object, AVS_haptics_pattern as a tactile pattern object, AVS_haptics_channel as a tactile channel object, AVS_haptics_event as a tactile event object, and AVS_haptics_event_component as a component object. These data objects constitute the basic data format of the tactile media exchange format. For a description of each data object, please refer to the records in Table 1 above, which will not be repeated here.
[0039] It should also be noted that the tactile media exchange format generated in an actual application scenario may only include at least one data object shown in Table 1 above. For example, a tactile experience may not contain character information or device information, and the generated tactile media exchange format will not contain the corresponding character object or device information object.
[0040] Please also refer to Figure 1, which shows the data structure connection between the various data objects in the tactile media exchange format proposed in this application. It can be seen that the tactile experience object, tactile pattern object, tactile channel object, and tactile event object have a hierarchical affiliation, and their hierarchies are in descending order. The tactile experience object is the highest-level element contained in the tactile media exchange format. It indicates the tactile experience and can also be understood as the information entry for the tactile experience. A tactile experience corresponds to tactile experience metadata and at least one tactile pattern, and may also contain data related to at least one specific device used to present the tactile experience. A tactile pattern corresponds to tactile pattern metadata and at least one tactile channel, and may contain data related to knowledge events. It can be understood that knowledge events are tactile events that occur repeatedly in a tactile experience. Defining recurring tactile events as knowledge events can make the generated tactile media exchange format more concise. A tactile channel corresponds to tactile channel metadata and at least one tactile event. A tactile event corresponds to tactile event metadata and at least one tactile event component.
[0041] The following describes the attributes contained in each data object contained in the tactile media exchange format.
[0042] Table 2 below shows the attribute definitions contained in the haptic experience object AVS_haptics:
[0043] Table 2
[0044] In this application, version in Table 2 above is referred to as the version attribute, creation_date is referred to as the creation time attribute, description is referred to as the description information attribute, event_layer_flag is referred to as the attribute indicating whether the tactile event is divided into layers, avatar is referred to as the attribute indicating the character information related to the current tactile experience, device is referred to as the attribute indicating the device information related to the current tactile experience, and pattern is referred to as the attribute indicating the tactile pattern related to the current tactile experience.
[0045] It should be noted that the "Required" items in Table 2 above indicate that the corresponding attributes may not exist in the haptic experience object in actual application scenarios. For example, as mentioned above, in actual application scenarios, the haptic experience may not include avatar or device information, so the avatar or device attributes will not exist in the haptic experience object.
[0046] It's also important to note that the event_layer_flag must be included in one of the following data objects: the haptic experience object, the haptic attribute object, or the haptic channel object. The hierarchical division of haptic events is limited to the corresponding data object. For example, when the event_layer_flag is included in the haptic attribute object, the values of haptic events under different haptic attributes cannot be compared.
[0047] Table 3 below shows the attribute definitions contained in the character information object AVS_haptics_avatar:
[0048] Table 3
[0049] This application refers to the id attribute in Table 3 above as the attribute indicating the character's identifier, avatar_model_type as the attribute indicating the type of the character model, and avatar_model_uri as the attribute indicating the URI address of the character model file. It should be understood that the character referred to in this application can refer to the recipient of the tactile experience, typically the user of a wearable device or interactive device. A character model refers to an abstract model used to represent a character's body parts, and the character model type defines the specifications followed by the character model.
[0050] It should also be noted that avatar_model_uri is an optional attribute in the character information object. This means that this application does not limit the character model file to being stored on an internet device such as a resource storage server. For example, it can also be stored locally on the current device. When the character model file is stored on an internet device, it is necessary to obtain and use the corresponding URI (Uniform Resource Identifier) of the character model file.
[0051] Table 4 below shows the attribute definitions contained in the device information object AVS_haptics_device:
[0052] Table 4
[0053] In this application, the id in Table 4 above is referred to as the device identification attribute, device_name is referred to as the device name attribute, refer_avatar is referred to as the attribute indicating the character corresponding to the current device, refer_avatar_parts is referred to as the attribute indicating the specific information of the character corresponding to the current device, and body_parts is referred to as the attribute indicating the body parts of the character corresponding to the current device.
[0054] An exemplary definition of a character's body parts is shown in Table 5 below:
[0055] Table 5
[0056] Table 6 below shows the attribute definitions contained in the haptic pattern object AVS_haptics_pattern:
[0057] Table 6
[0058] In this application, the id in Table 6 above is referred to as the mode identifier attribute, pattern_type is referred to as the mode type attribute, refer_devices is referred to as the attribute indicating the device corresponding to the current mode, refer_avatars is referred to as the attribute indicating the character corresponding to the current mode, event_semantics_flag is referred to as the attribute indicating whether the tactile event corresponding to the current mode has semantics, event_semantics_type is referred to as the attribute indicating the specification followed by the tactile event semantics, event_layer_flag is referred to as the attribute indicating whether the tactile event is divided into layers, library_events is referred to as the attribute indicating the knowledge tactile event contained in the current mode, and channels is referred to as the attribute indicating the tactile channels contained in the current mode.
[0059] It should be noted that event_semantics_type is a conditional mandatory attribute. It can be understood that when the value of the event_semantics_flag attribute is true, it means that the tactile event corresponding to the current mode has semantics, so the event_semantics_type attribute must exist.
[0060] In another exemplary embodiment, a value of 0 for event_semantics_type indicates that the semantics of the haptic event follow the specifications defined by this standard, a value of 1 indicates that the semantics of the haptic event follow the specifications of an external standard, and a value of 2 indicates that the semantics of the haptic event are user-defined. The "this standard" mentioned herein refers to the format standard of the Haptic Media Exchange Format mentioned in this application.
[0061] Table 7 below shows the attribute definitions contained in the haptic channel object AVS_haptics_channel:
[0062] Table 7
[0063] In this application, the id in Table 7 above is referred to as the channel identifier attribute, description is referred to as the channel description attribute, refer_devices is referred to as the attribute indicating the device corresponding to the current channel, refer_avatars is referred to as the attribute indicating the character corresponding to the current channel, gain is referred to as the channel gain attribute, channel_alternative_flag is referred to as the attribute indicating whether the current channel is in an interchangeable relationship with other tactile channels, alternative_group_id is referred to as the attribute indicating the identifier of the alternative channel group to which the current channel belongs, channel_group_flag is referred to as the attribute indicating whether the current channel belongs to a certain channel group, channel_group_id is referred to as the attribute indicating the identifier of the channel group to which the current channel belongs, mixing_weight is referred to as the attribute indicating the weight of the current channel in the corresponding channel group, event_layer_flag is referred to as the attribute indicating whether the tactile events are divided into layers, priority is referred to as the attribute indicating the priority of the current channel, and events is referred to as the attribute indicating the tactile events contained in the current channel.
[0064] It should be noted that when the replaceable channel group to which the current channel belongs exists, for the optional refer_devices, refer_avatars and priority, only the channel with the smallest identifier value in the channel group is indicated, and is not repeated in the metadata of other channels in the channel group.
[0065] It should also be noted that the default values of channel_alternative_flag and channel_group_flag are 0, which means that the current channel and other channels are not in a replaceable relationship by default, and the current channel does not belong to a certain channel group. In actual application scenarios, the attribute values need to be modified according to actual needs. In addition, alternative_group_id only exists when the value of channel_alternative_flag is true, and channel_group_id and mixing_weight only exist when the value of channel_group_flag is true. Priority is an unnecessary attribute in the tactile channel object. It can be understood that in actual application scenarios, it is not necessary to divide priorities for different tactile channels. The following Table 8 shows the attribute definitions contained in the tactile event object AVS_haptics_event:
[0066] Table 8
[0067] In this application, the id in Table 8 above is referred to as the event identifier attribute, effect_semantic is referred to as the event semantic attribute, event_layer is referred to as the event layer attribute, event_type is referred to as the event type attribute, refer_library_event is referred to as the attribute of the knowledge tactile event identifier corresponding to the event type being a reference event, relative_time is referred to as the attribute indicating the time offset of the current event, duration is referred to as the attribute indicating the duration of the current event, amplitude is referred to as the attribute indicating the maximum amplitude value of the signal of the current event, base_frequency is referred to as the attribute indicating the signal base frequency of the current event, and event_components is referred to as the attribute indicating the components that make up the current event.
[0068] It should be noted that effect_semantic is a required attribute in the tactile event object only when the event_semantics_flag attribute of the tactile mode corresponding to the tactile event is set to true. event_layer is also a required attribute in the tactile event object only when the event_layer_flag field in the tactile experience, tactile mode, or tactile channel corresponding to the tactile event is set to true. When presenting a tactile event, lower-level tactile events are presented first. If a tactile event does not have a level field, the default level of the tactile event is 0.
[0069] Table 9 below shows the attribute definitions contained in the component object AVS_haptics_event_component:
[0070] Table 9
[0071] In this application, refer_event_id in Table 9 above is referred to as the attribute indicating the identifier of the tactile event to which the current component belongs, relative_time is referred to as the attribute indicating the time offset of the current component relative to the tactile event, relative_amplitude is referred to as the attribute indicating the amplitude value ratio of the current component relative to the tactile event, and relative_frequency is referred to as the attribute indicating the frequency offset of the current component relative to the tactile event.
[0072] As can be seen from Tables 1-9 and FIG1 , the tactile media exchange format proposed in this application can be summarized as including the following information, thereby efficiently and uniformly expressing tactile media content:
[0073] 1) Basic metadata of tactile media, presentation device information, and presentation location information;
[0074] 2) Organizing tactile media content into different levels according to signal type, signal rendering location, etc., and indicating metadata information for each level;
[0075] 3) Define the basic level of the tactile media signal according to the division methods such as time, space, and frequency, and indicate the metadata information of the basic level.
[0076] This application also proposes a method for converting the tactile media exchange format shown above into a tactile media binary bit stream to facilitate subsequent file packaging and transmission. The tactile media exchange format and the tactile media binary bit stream may correspond to each other in data structure.
[0077] The tactile media binary bit stream consists of a binary header and a binary payload, where the binary header contains metadata related to the tactile experience, tactile mode, and tactile channel, and the binary payload contains metadata related to the tactile event.
[0078] The tactile media exchange format is converted into a tactile media binary bit stream, that is, each attribute in each data object included in the tactile media exchange format is converted into a binary bit format according to a predefined attribute conversion rule.
[0079] The syntax for converting the tactile experience object into a binary format is shown in Table 10 below:
[0080] Table 10
[0081] Among them, versionStringSize, dateStringSize, and descriptionStringSize respectively indicate the number of bytes of the corresponding string, avatarCount, deviceCount, and patternCount respectively indicate the number of corresponding data structures, and the remaining fields and data structures correspond to the corresponding fields and data structures defined in the exchange format.
[0082] The syntax for converting the character information object into a binary format is shown in Table 11 below:
[0083] Table 11
[0084] Among them, avatarModelUriStringSize indicates the number of bytes of the corresponding string, and the remaining fields and data structures correspond to the corresponding fields and data structures defined in the exchange format.
[0085] The syntax for converting the device information object to binary format is shown in Table 12 below:
[0086] Table 12
[0087] Among them, referAvatarFlag, referAvatarPartsFlag, and bodyPartsFlag respectively indicate whether to refer to character information, whether to refer to character body information, and whether it corresponds to body part information; deviceNameStringSize and avatarPartsStringSize respectively indicate the length of the device name string and the length of the character body information string; the remaining fields and data structures correspond to the corresponding fields and data structures defined in the exchange format.
[0088] The syntax for converting the tactile mode object into a binary format is shown in Table 13 below:
[0089] Table 13
[0090] Among them, referDeviceCount, referAvatarCount, libraryEventCount, and channelCount respectively indicate the number of reference devices, the number of reference characters, the number of knowledge events, and the number of channels, libraryEventId indicates the identifier of the knowledge event, and the remaining fields and data structures correspond to the corresponding fields and data structures defined in the exchange format.
[0091] patternType indicates the pattern type. Its values are shown in Table 14 below:
[0092] Table 14
[0093] The syntax for converting the tactile channel object into a binary format is shown in Table 15 below:
[0094] Table 15
[0095] Among them, descriptionStringSize indicates the number of bytes of the corresponding string, referDeviceCount, referAvatarCount, and eventCount respectively indicate the number of reference devices, the number of reference characters, and the number of events, alternativeFlag, groupFlag, weightFlag, and priorityFlag respectively indicate whether the current channel belongs to an alternative channel group, whether it belongs to a channel group, whether it contains weight information, and whether it contains priority information, eventId indicates the identifier of the event contained in the current channel, and the remaining fields and data structures correspond to the corresponding fields and data structures defined in the exchange format.
[0096] The syntax for converting the haptic event object into a binary format is shown in Table 16 below:
[0097] Table 16
[0098] Among them, eventStartCode represents the start code of the event; eventType takes a value of 0 to indicate an instantaneous event, a value of 1 to indicate a duration, and a value of 2 to indicate a reference time. For instantaneous events, the duration is a fixed value and usually depends on the device; referDeviceCount, referAvatarCount, and eventComponentCount indicate the number of reference devices, the number of reference characters, and the number of event components, respectively; libEventFlag takes a value of 1 to indicate that the current event is a knowledge event, and a value of 0 to indicate that the current event is not a knowledge event. Knowledge events do not have a time offset value; the remaining fields and data structures correspond to the corresponding fields and data structures defined in the exchange format; eventSemanticsStringSize indicates the string length of the tactile event semantics; eventSemantics indicates the semantics of the tactile event.
[0099] The syntax for converting the binary bit format of the component object is shown in Table 17 below:
[0100] Table 17
[0101] As can be seen from Tables 10-17 above, the attribute conversion method for converting the tactile media exchange format in this application can be summarized as follows:
[0102] 1) Converting the attributes of string type, integer type, enum type, or number type in each data object into binary bit stream segments of a number of integer bytes, where the byte length corresponding to each binary bit stream segment may be a preset value;
[0103] 2) Convert the Boolean attribute into a binary bit stream segment of several bits;
[0104] 3) Count the number of array elements contained in the array type array attribute, and convert the binary bit stream segment of the corresponding attribute based on the number of array elements.
[0105] Of course, as an optional implementation, the byte length of the binary bit stream segment obtained after converting the attributes of the above-mentioned various data types can also be adjusted accordingly, and it is not limited to the byte length shown in Table 10-17.
[0106] It should also be noted that when converting the various properties in the tactile event object, when the event SemanticsType value in the tactile mode object is 0, that is, the semantics of the tactile event follows the current standard, the tactile event semantics can be defined in one of the following ways:
[0107] 1) Defined as a two-layer semantic structure, with different levels of semantics separated by spaces or " / ". For example, "texture / rock" means that the tactile event simulates the texture of stone;
[0108] 2) Defined as a three-layer semantic structure, with different levels of semantics separated by spaces or " / ", such as "game / weapon / shotgun", indicating that the tactile event simulates the sound of a gunshot in the game;
[0109] 3) Defined as a multi-layer semantic structure, with different levels of semantics separated by spaces.
[0110] Therefore, optionally, when performing binary bit format conversion on the haptic event object, the conversion method of the above-mentioned hierarchical haptic semantics may be defined as shown in the following Table 18:
[0111] Table 18
[0112] Optionally, for the semantics of tactile events of readable string type, the conversion method can also be defined as a specific binary field, as shown in Table 18 below:
[0113] Table 19
[0114] Take the 2-layer grammar as an example:
[0115] eventSemanticsLevelNum = 2;
[0116] semanticsLevelBitsNum[0]=4, eventSemantics[0]=0001, corresponding to texture;
[0117] semanticsLevelBitsNum[1]=4, eventSemantics[0]=0010, corresponding to rock.
[0118] Based on the general tactile media content representation method described above, an embodiment of the present application proposes a method for processing tactile media content. The method is applied to tactile feedback-related products, such as the server side, player side, and intermediate nodes of an immersive system, and is not limited here.
[0119] As shown in FIG2 , the method includes the following steps:
[0120] S210, obtaining at least one tactile media signal;
[0121] S220, generating a tactile media exchange format according to at least one tactile media signal, where the tactile media exchange format includes at least one data object selected from the group consisting of a tactile experience object, a character information object, a device information object, a tactile mode object, a tactile channel object, a tactile event object, and a component object;
[0122] S230, converting the tactile media exchange format to obtain a tactile media binary bit stream;
[0123] S240 : Encapsulate the haptic media binary bit stream into a haptic media file, and transmit the haptic media file to the client.
[0124] It should be noted that the tactile media information obtained in S210 can be produced or collected based on the expected tactile media effect. The method for generating the tactile media exchange format in S220 and the method for converting the tactile media binary bit stream in S230 have been described in the previous content and will not be repeated here. S240 encapsulates the tactile media binary bit stream into a tactile media file. For example, this can be achieved by encapsulating based on ISOBMFF (ISO Based Media File Format, a media file format based on ISO standards), or using other media file formats, which is not limited here. The encapsulated tactile media file can be streamed to the client, for example, by sending the tactile media file to the client in its entirety, or by generating at least one file fragment based on the tactile media file and transmitting the at least one file fragment to the client. After receiving the file or file fragment, the client can parse the file or file fragment to obtain a tactile media file containing the tactile signal. After decapsulation and decoding, the tactile media signal is obtained and rendered. In this way, efficient and unified expression of tactile media content can be achieved.
[0125] It should also be noted that in streaming, the server needs to send a signaling message describing the media resource to the client, and the client then requests the specific media resource based on the signaling message. Therefore, in this embodiment, the haptic media file can be transmitted to the client after receiving the corresponding request signaling sent by the client.
[0126] The following is an example of each data object in the tactile media exchange format generated in S220. It should be noted that the definition of the attributes included in each of the following data objects can be found in the above description and will not be repeated here.
[0127] The tactile experience object includes at least one of a version attribute, a creation time attribute, a description information attribute, an attribute indicating a tactile mode related to the current tactile experience, an attribute indicating character information related to the current tactile experience, an attribute indicating device information related to the current tactile experience, and an attribute indicating whether the tactile event is divided into levels.
[0128] The character information object includes at least one of an attribute indicating an identifier of a character, an attribute indicating a type of a character model, and an attribute indicating a URI address of a character model file.
[0129] The device information object includes at least one of a device identification attribute, a device name attribute, an attribute indicating a person corresponding to the current device, an attribute indicating specific information of the person corresponding to the current device, and an attribute indicating a body part of the person corresponding to the current device.
[0130] The tactile mode object includes at least one of a mode identifier attribute, a mode type attribute, an attribute indicating the device corresponding to the current mode, an attribute indicating the person corresponding to the current mode, an attribute indicating whether the tactile event corresponding to the current mode has semantics, an attribute indicating the specifications followed by the tactile event semantics, an attribute indicating whether the tactile event is divided into levels, an attribute indicating the knowledge tactile events contained in the current mode, and an attribute indicating the tactile channels contained in the current mode.
[0131] The haptic channel object includes at least one of a channel identifier attribute, a channel description attribute, an attribute indicating a device corresponding to the current channel, an attribute indicating a person corresponding to the current channel, a channel gain attribute, an attribute indicating whether the current channel is in a replaceable relationship with other haptic channels, an attribute indicating an identifier of an replaceable channel group to which the current channel belongs, an attribute indicating whether the current channel belongs to a certain channel group, an attribute indicating an identifier of the channel group to which the current channel belongs, an attribute indicating a weight of the current channel in the corresponding channel group, an attribute indicating whether haptic events are hierarchical, an attribute indicating a priority of the current channel, and an attribute indicating haptic events included in the current channel.
[0132] Among them, when there is a replaceable channel group to which the current channel belongs, among the attributes indicating the device corresponding to the current channel, the attributes indicating the person corresponding to the current channel, and the attributes indicating the priority of the current channel, only the tactile channel with the smallest identifier in the replaceable channel group is indicated.
[0133] The tactile event object includes at least one of an event identifier attribute, an event semantic attribute, an event level attribute, an event type attribute, an attribute indicating the knowledge tactile event identifier corresponding to the event type being a reference event, an attribute indicating the time offset of the current event, an attribute indicating the duration of the current event, an attribute indicating the maximum amplitude value of the signal of the current event, an attribute indicating the signal reference frequency of the current event, and an attribute indicating the components that constitute the current event.
[0134] The component component object includes at least one of an attribute indicating an identifier of the tactile event to which the current component belongs, an attribute indicating a time offset of the current component relative to the tactile event, an attribute indicating a ratio of an amplitude value of the current component relative to the tactile event, and an attribute indicating a frequency offset of the current component relative to the tactile event.
[0135] It should also be noted that the tactile media binary bit stream mentioned in S230 is composed of a binary header and a binary payload. In an exemplary embodiment, S230 specifically includes the following steps:
[0136] S231, performing binary bit format conversion on each attribute in each data object included in the tactile media exchange format according to a predefined attribute conversion method to obtain a binary stream segment corresponding to each data object;
[0137] S232: Form a binary header based on the binary bit stream segments corresponding to the tactile experience object, the tactile pattern object, and the tactile channel object, and form a binary payload based on the binary bit stream segments corresponding to the tactile event object and the component object, to obtain a tactile media binary bit stream.
[0138] The predefined attribute conversion methods mentioned in S231 include at least one of the following:
[0139] 1) Convert the string type, integer type, enumeration type or numeric type attribute into a binary bit stream segment of a certain integer number of bytes;
[0140] 2) Convert the Boolean attribute into a binary bit stream segment of several bits;
[0141] 3) Counting the number of array elements contained in the array-type attribute, and converting the binary bit stream segment of the corresponding attribute based on the number of array elements;
[0142] 4) When converting the tactile event object into a binary bit format, adding the start code corresponding to the corresponding tactile event as a new attribute of the tactile event object, and converting it to obtain a corresponding binary bit stream segment;
[0143] 5) When converting the event semantic attributes contained in the tactile event object into binary bit format, if the specification followed by the semantics of the corresponding tactile event is the current standard, the semantic conversion of the corresponding tactile event is performed according to the semantic structure of the preset level, and the corresponding binary bit stream segment is obtained based on the converted semantics. The semantic structure of the preset level indicates that the semantics of different levels are divided by specified symbols, or converted into specified binary fields.
[0144] It should be noted that the contents of any of the above records have been described in detail in the aforementioned records, so they will not be explained in detail here.
[0145] FIG3 is a block diagram of a tactile media content processing apparatus according to an exemplary embodiment of the present application. As shown in FIG3 , the exemplary tactile media content processing apparatus 300 includes:
[0146] a signal acquisition module 310 configured to acquire at least one tactile media signal;
[0147] an exchange format generating module 320 configured to generate a tactile media exchange format according to at least one tactile media signal, the tactile media exchange format including at least one data object selected from the group consisting of a tactile experience object, a character information object, a device information object, a tactile mode object, a tactile channel object, a tactile event object, and a component object;
[0148] The exchange format conversion module 330 is configured to convert the tactile media exchange format to obtain a tactile media binary bit stream;
[0149] The packaging and transmission module 340 is configured to package the haptic media binary bit stream into a haptic media file and transmit the haptic media file to the client.
[0150] In another exemplary embodiment, the tactile media binary bit stream consists of a binary header and a binary payload, and the exchange format conversion module 330 includes:
[0151] a conversion unit configured to perform binary bit format conversion on each data object included in the tactile media exchange format based on a predefined attribute conversion method;
[0152] The bit stream acquisition unit is configured to use the binary bit data corresponding to the converted tactile experience object, tactile pattern object and tactile channel object as a binary header, and use the binary bit data corresponding to the converted tactile event object and component object as a binary payload to obtain the tactile media binary bit stream.
[0153] It should be noted that the tactile media content processing apparatus provided in the above-mentioned embodiments and the tactile media content processing method provided in the above-mentioned embodiments are based on the same concept. The specific manner in which the various modules and units perform their operations has been described in detail in the method embodiments and will not be repeated here. In actual applications, the tactile media content processing apparatus provided in the above-mentioned embodiments can, as needed, allocate the aforementioned functions to different functional modules, i.e., divide the internal structure of the apparatus into different functional modules to perform all or part of the functions described above. This is not a limitation herein.
[0154] An embodiment of the present application also provides a tactile media content processing device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the tactile media content processing device implements the tactile media content processing method provided in the above-mentioned embodiments.
[0155] Figure 4 shows a schematic diagram of the structure of a computer system suitable for implementing a tactile media content processing device according to an embodiment of the present application. It should be noted that the computer system 400 of the tactile media content processing device shown in Figure 4 is merely an example and should not limit the functionality and scope of use of the embodiments of the present application.
[0156] As shown in Figure 4, computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 402 or programs loaded from storage unit 408 into random access memory (RAM) 403, such as executing the methods described in the above embodiments. Various programs and data required for system operation are also stored in RAM 403. CPU 401, ROM 402, and RAM 403 are connected to each other via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.
[0157] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, and the like; an output section 407 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 408 including a hard disk and the like; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. Removable media 411, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 410 as needed, so that computer programs read therefrom can be installed into the storage section 408 as needed.
[0158] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 409, and / or installed from a removable medium 411. When the computer program is executed by the central processing unit (CPU) 401, the various functions defined in the system of the present application are executed.
[0159] It should be noted that the computer-readable medium shown in the embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. The computer program contained in the computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.
[0160] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0161] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.
[0162] Another aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method for processing tactile media content. The computer-readable storage medium may be included in the tactile media content processing device described in the above embodiments, or may exist independently and not be incorporated into the tactile media content processing device.
[0163] Another aspect of the present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method for processing tactile media content provided in each of the above embodiments.
[0164] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. Ordinary technicians in this field can easily make corresponding changes or modifications based on the main ideas and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.
[0165] It should also be noted that in the specific implementation of this application, data related to tactile media content, character information, device information, etc. is involved. When the above embodiments of this application are applied to specific products or technologies, the user's separate permission or consent is required, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.
Claims
1. A method for processing tactile media content, characterized in that: The method comprises: acquiring at least one tactile media signal; generating a tactile media exchange format according to the at least one tactile media signal, wherein the tactile media exchange format includes at least one data object of a tactile experience object, a character information object, a device information object, a tactile mode object, a tactile channel object, a tactile event object, and a component component object; Converting the tactile media exchange format to obtain a tactile media binary bit stream; The haptic media binary bit stream is encapsulated into a haptic media file, and the haptic media file is transmitted to a client.
2. The method according to claim 1, characterized in that The tactile experience object includes at least one of a version attribute, a creation time attribute, a description information attribute, an attribute indicating a tactile mode related to a current tactile experience, an attribute indicating character information related to a current tactile experience, an attribute indicating device information related to a current tactile experience, and an attribute indicating whether a tactile event is divided into levels.
3. The method according to claim 1, characterized in that The character information object includes at least one of an attribute indicating an identifier of a character, an attribute indicating a type of a character model, and an attribute indicating a URI address of a character model file.
4. The method according to claim 1, characterized in that The device information object includes at least one of a device identification attribute, a device name attribute, an attribute indicating a person corresponding to the current device, an attribute indicating specific information of the person corresponding to the current device, and an attribute indicating a body part of the person corresponding to the current device.
5. The method according to claim 1, characterized in that The tactile mode object includes at least one of a mode identifier attribute, a mode type attribute, an attribute indicating a device corresponding to the current mode, an attribute indicating a character corresponding to the current mode, an attribute indicating whether a tactile event corresponding to the current mode has semantics, an attribute indicating the specification followed by the semantics of the tactile event, an attribute indicating whether the tactile event is divided into levels, an attribute indicating knowledge tactile events included in the current mode, and an attribute indicating a tactile channel included in the current mode.
6. The method according to claim 1, characterized in that The tactile channel object includes at least one of a channel identifier attribute, a channel description attribute, an attribute indicating a device corresponding to the current channel, an attribute indicating a character corresponding to the current channel, a channel gain attribute, an attribute indicating whether the current channel is in a replaceable relationship with other tactile channels, an attribute indicating an identifier of a replaceable channel group to which the current channel belongs, an attribute indicating whether the current channel belongs to a certain channel group, an attribute indicating an identifier of the channel group to which the current channel belongs, an attribute indicating a weight of the current channel in the corresponding channel group, an attribute indicating whether tactile events are divided into levels, an attribute indicating a priority of the current channel, and an attribute indicating tactile events included in the current channel; Among them, when there is a replaceable channel group to which the current channel belongs, among the attributes indicating the device corresponding to the current channel, the attributes indicating the person corresponding to the current channel, and the attributes indicating the priority of the current channel, only the tactile channel with the smallest identifier in the replaceable channel group is indicated.
7. The method according to claim 1, characterized in that The tactile event object includes at least one of an event identifier attribute, an event semantic attribute, an event level attribute, an event type attribute, an attribute indicating the knowledge tactile event identifier corresponding to the event type being a reference event, an attribute indicating the time offset of the current event, an attribute indicating the duration of the current event, an attribute indicating the maximum amplitude value of the signal of the current event, an attribute indicating the signal reference frequency of the current event, and an attribute indicating the components that constitute the current event.
8. The method according to claim 1, characterized in that The component component object includes at least one of an attribute indicating an identifier of the tactile event to which the current component belongs, an attribute indicating a time offset of the current component relative to the tactile event, an attribute indicating a ratio of an amplitude value of the current component relative to the tactile event, and an attribute indicating a frequency offset of the current component relative to the tactile event.
9. The method according to any one of claims 1 to 8, characterized in that The tactile media binary bit stream is composed of a binary header and a binary payload; the tactile media exchange format is converted to obtain the tactile media binary bit stream, including: According to a predefined attribute conversion method, performing binary bit format conversion on each attribute in each data object included in the tactile media exchange format to obtain a binary stream segment corresponding to each data object; A binary header is formed based on the binary bit stream segments corresponding to the tactile experience object, the tactile mode object and the tactile channel object, and a binary payload is formed based on the binary bit stream segments corresponding to the tactile event object and the component object, so as to obtain the tactile media binary bit stream.
10. The method according to claim 9, characterized in that The predefined attribute conversion method includes at least one of the following: Convert string, integer, enumeration or numeric attributes into binary bit stream segments of a certain integer number of bytes; Convert Boolean attributes into binary bit stream segments of several bits; The number of array elements contained in the array-type attribute is counted, and the binary bit stream segment of the corresponding attribute is converted based on the number of array elements.
11. The method according to claim 9, characterized in that The predefined field conversion method includes at least one of the following: When converting the tactile event object into a binary bit format, adding a start code corresponding to the corresponding tactile event as a new attribute of the tactile event object, and converting to obtain a corresponding binary bit stream segment; When the event semantic attributes contained in the tactile event object are converted into binary bit format, if the specification followed by the semantics of the corresponding tactile event is the current standard, the semantic conversion of the corresponding tactile event is performed according to the semantic structure of the preset level, and the corresponding binary bit stream segment is obtained based on the converted semantics. The semantic structure of the preset level indicates that the semantics of different levels are divided by specified symbols, or converted into specified binary fields.
12. The method according to any one of claims 1 to 8, characterized in that The step of encapsulating the tactile media binary bit stream into a tactile media file and transmitting the tactile media file to a client comprises: sending the tactile media file to the client; Alternatively, at least one file segment is generated based on the tactile media file, and the at least one file segment is transmitted to the client.
13. A tactile media content processing device, characterized in that: The device comprises: a signal acquisition module, configured to acquire at least one tactile media signal; an exchange format generating module, configured to generate a tactile media exchange format according to the at least one tactile media signal, wherein the tactile media exchange format includes at least one data object selected from the group consisting of a tactile experience object, a character information object, a device information object, a tactile mode object, a tactile channel object, a tactile event object, and a component component object; An exchange format conversion module, configured to convert the tactile media exchange format to obtain a tactile media binary bit stream; a packaging and transmission module configured to package the tactile media binary bit stream into a tactile media file, And transmit the tactile media file to the client.
14. A method for processing tactile media content, characterized in that: The method comprises: Acquire a tactile media file; or acquire at least one file segment generated based on the tactile media file, and parse the at least one file segment to obtain the tactile media file; Decapsulating the tactile media file to obtain a tactile media binary bit stream; Performing format inverse conversion on the tactile media binary bit stream to obtain a tactile media exchange format, wherein the tactile media exchange format includes at least one data object selected from the group consisting of a tactile experience object, a character information object, a device information object, a tactile mode object, a tactile channel object, a tactile event object, and a component component object; Parsing the tactile media exchange format to obtain at least one tactile media signal; The at least one haptic media signal is rendered and presented.
15. A tactile media content processing device, characterized in that: The method comprises: A first module is used to obtain a tactile media file; or obtain at least one file segment generated based on the tactile media file, and parse the at least one file segment to obtain the tactile media file; A second module is used to decapsulate the tactile media file to obtain a tactile media binary bit stream; A third module is used to perform format inverse conversion on the tactile media binary bit stream to obtain a tactile media exchange format, wherein the tactile media exchange format includes at least one data object selected from the group consisting of a tactile experience object, a character information object, a device information object, a tactile mode object, a tactile channel object, a tactile event object, and a component component object; A fourth module is used to parse the tactile media exchange format to obtain at least one tactile media signal; The fifth module is used to render and present the at least one tactile media signal.
16. A device for processing tactile media content, characterized in that: include: one or more processors; A memory for storing one or more programs, which, when executed by the one or more processors, enables the tactile media content processing device to implement the tactile media content processing method as described in any one of claims 1 to 12 and 14.
17. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the method for processing tactile media content according to any one of claims 1 to 12 and 14.
18. A program product, characterized in that include: A computer program, wherein when the computer program is executed by a processor, the method for processing tactile media content according to any one of claims 1 to 12 and 14 is implemented.
19. A method for storing a tactile media file, characterized in that: A tactile media file is stored on a non-volatile computer-readable storage medium, wherein the tactile media file is generated according to the method of any one of claims 1 to 12 or is decoded based on the method of claim 14.
Citation Information
Patent Citations
Method and apparatus for encoding and decoding haptic information in multi-media files
CN104049889A
Screen recording method and device, equipment and storage medium
CN113973226A
Automated haptics generation and distribution
US20230044961A1
Haptic scene representation format
WO2022053439A1
