Tactile information processing method and apparatus, electronic device, and storage medium

Abstract

The present application provides a tactile information processing method and apparatus, an electronic device, and a storage medium, which are applied to the technical field of multimedia information processing. The method comprises: decapsulating a media file comprising target tactile information to obtain a binary bitstream corresponding to the target tactile information, the bitstream comprising one or more data units related to the target tactile information, and the data units comprising one or more data packets; and decoding the binary bitstream corresponding to the target tactile information to obtain an exchange format of the target tactile information, thereby enabling the target tactile information to be rendered and presented on the basis of the exchange format of the target tactile information. The solution provided in embodiments of the present application facilitates reducing the space occupied by tactile media content, and reduces bandwidth usage during transmission of the tactile media content.

Description

Method, device, electronic device and storage medium for processing tactile information

[0001] This application claims priority to Chinese patent application No. 2023116953158, filed with the Patent Office of China on December 11, 2023, entitled “Method, device, electronic device and storage medium for processing tactile information”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments of the present application relate to the field of multimedia information processing technology, and in particular to a method, device, electronic device, and storage medium for processing tactile information. Background Art

[0003] The presentation of immersive media content is often accompanied by the use of various wearable or interactive devices. In addition to visual and auditory presentation, immersive media also offers tactile presentation. Tactile presentation can be achieved through a tactile presentation mechanism that combines hardware and software. Tactile presentation provides an embedded physical sensation. Users receive information through their bodies, thereby conveying key information about the system they are using. For example, vibration is a type of tactile presentation; a mobile phone vibrates to alert users to incoming calls or messages.

[0004] In the related art, tactile media content occupies a large storage space and consumes a large amount of bandwidth during transmission. The related art urgently needs a solution that can save space and reduce bandwidth usage during the transmission of tactile media content.

[0005] Summary of the Invention

[0006] The present application provides a tactile information processing method, device, electronic device, and storage medium, which can save space occupied by tactile media content and reduce bandwidth usage during tactile media content transmission.

[0007] In a first aspect, an embodiment of the present application provides a method for processing tactile information, which is executed by a processor, and the method includes: decapsulating a media file containing target tactile information to obtain a binary bit stream corresponding to the target tactile information, the bit stream including: one or more data units about the target tactile information, the data units including one or more data packets; and decoding the binary bit stream corresponding to the target tactile information to obtain an exchange format of the target tactile information, wherein the exchange format of the target tactile information is used to render and present the target tactile information.

[0008] In a second aspect, an embodiment of the present application provides a device for processing tactile information, the device comprising: a decapsulation module and a decoding module; the decapsulation module is used to decapsulate a media file containing target tactile information to obtain a binary bit stream corresponding to the target tactile information, the bit stream comprising: one or more data units regarding the target tactile information, the data units comprising one or more data packets; and the decoding module is used to decode the binary bit stream corresponding to the target tactile information to obtain an exchange format of the target tactile information, wherein the exchange format of the target tactile information is used to render and present the target tactile information.

[0009] In a third aspect, an embodiment of the present application provides a method for processing tactile information, which is executed by a processor. The method includes: encoding the exchange format of the target tactile information to obtain a binary bit stream corresponding to the target tactile information, the bit stream including: one or more data units about the target tactile information, the data unit including one or more data packets; and encapsulating the bit stream to obtain a media file containing the tactile information.

[0010] In a fourth aspect, an embodiment of the present application provides a device for processing tactile information, the device comprising: an encoding module and an encapsulation module; the encoding module is used to encode the exchange format of the target tactile information to obtain a binary bit stream corresponding to the target tactile information, the bit stream comprising: one or more data units regarding the target tactile information, the data units comprising one or more data packets; and the encapsulation module is used to encapsulate the bit stream to obtain a media file containing the tactile information.

[0011] In a fifth aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory. The memory is configured to store a computer program, and the processor is configured to call and execute the computer program stored in the memory to execute the tactile information processing method provided in the first aspect or the second aspect.

[0012] In a sixth aspect, embodiments of the present application provide a chip for implementing the method of any aspect of the first aspect or its respective implementations. Specifically, the chip includes a processor for calling and executing a computer program from a memory, causing a device equipped with the chip to execute the method for processing tactile information as provided in the first or second aspect.

[0013] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program enables a computer to execute the method for processing tactile information provided in the first aspect or the second aspect.

[0014] In an eighth aspect, an embodiment of the present application provides a computer program product, comprising computer program instructions, wherein the computer program instructions enable a computer to execute the method for processing tactile information provided in the first aspect or the second aspect.

[0015] In a ninth aspect, an embodiment of the present application provides a computer program, which, when executed on a computer, enables the computer to execute the method for processing tactile information provided in the first aspect or the second aspect.

[0016] In summary, in the solution provided by the embodiment of the present application, the exchange format of the target tactile information is encoded to obtain a binary bit stream corresponding to the target tactile information. The above-mentioned binary bit stream is further encapsulated to obtain a media file containing the tactile information. The embodiment of the present application provides a solution for encoding the exchange format of tactile information into a binary bit stream. Specifically, the data is organized and divided in the form of data units and data packets in the above-mentioned binary bit stream, and a data unit contains one or more data packets. The solution provided by the embodiment of the present application is conducive to saving the space occupied by tactile media content, and can reduce bandwidth occupancy during the transmission of tactile media content. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] FIG1 is a schematic diagram of the system architecture for representing and transmitting tactile information in the video coding standard AVS;

[0019] FIG2 is a schematic diagram of an application scenario of processing tactile information provided by an embodiment of the present application;

[0020] FIG3 is a schematic diagram of information interaction of a method for processing tactile information according to an embodiment of the present application;

[0021] FIG4 is a schematic diagram of a data structure representing tactile information in the video coding standard AVS;

[0022] FIG5 is a flow chart of a method for processing tactile information according to an embodiment of the present application;

[0023] FIG6 is a schematic diagram of the relationship between data units and data packets provided in an embodiment of the present application;

[0024] FIG7 is a flow chart of a method for processing tactile information according to an embodiment of the present application;

[0025] FIG8 is a schematic diagram of the structure of a device for processing tactile information provided in an embodiment of the present application;

[0026] FIG9 is a schematic structural diagram of a device for processing tactile information provided in an embodiment of the present application;

[0027] FIG10 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0029] It should be noted that the terms "first," "second," and the like in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In the embodiments of the present application, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A, but that B can also be determined based on A and / or other information. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or devices. In the description of this application, unless otherwise specified, "plurality" refers to two or more than two.

[0030] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0031] Tactile information presentation can be divided into the following categories: vibrotactile, kinematic, and electrotactile. Specific descriptions are as follows:

[0032] 1. Vibrotactile

[0033] Direct tactile presentation in the form of vibration can simulate vibrations of specific frequencies and intensities through the vibration of the terminal device's motor. For example, in a shooting game, vibrations can be used to simulate the specific effects of shooting.

[0034] 2. Kinematic touch

[0035] Kinematic haptics simulate the motion of an object, such as weight or pressure. For example, in a driving video game, the steering wheel might resist turning when moving at higher speeds or operating a heavier vehicle. This type of feedback directly affects the user's muscles. In the driving game example, the user must apply more force to get the desired response from the steering wheel.

[0036] 3. Electrotactile

[0037] Electrotactile presentation uses electrical stimulation to simulate specific texture feedback. Specifically, electrical pulses are used to deliver tactile stimulation to the nerve endings in the user's skin. Electrotactile presentation can create a highly realistic experience for users wearing suits or gloves equipped with electrotactile technology. Almost any sensation can be simulated using electrical pulses, including temperature changes, pressure changes, and the sensation of moisture.

[0038] With the popularization of wearable devices and interactive devices, the tactile presentation that users can perceive when consuming media content will no longer be limited to basic vibration touch, but can also include vibration, pressure, speed, acceleration, temperature, humidity, smell and other full-dimensional tactile presentation experiences that are closer to the real world.

[0039] Figure 1 is a schematic diagram of the tactile information representation and transmission system architecture 100 of the Chinese National Video Coding Standard (AVS). Referring to Figure 1 , the tactile information exchange format X can be generated based on the acquired signal A or transcoded from the existing tactile signal format B. The tactile information exchange format X can be directly rendered and presented on the client, or it can be encoded to generate a tactile information elementary stream (also referred to as an elementary bitstream, or bitstream) E. The tactile information elementary stream is encapsulated according to the corresponding media container file format (e.g., ISO Based Media File Format (ISOBMFF)), resulting in a sequence Fs of initialization segments and media segments for streaming, or a media file F for file playback. The sequence Fs for streaming is combined with media presentation description information and transmitted to the player using a transmission mechanism. The file is decapsulated, and the received file F' or segment Fs' is extracted, followed by metadata parsing. The tactile information elementary stream E' is then decoded to generate the tactile information exchange format X', which can ultimately be rendered and presented on the client.

[0040] The tactile media exchange format is used to describe the signals contained in the tactile media. The tactile media exchange format defined by the AVS tactile information representation and transmission system architecture 100 shown in FIG1 is in the form of a human-readable JavaScript Object Notation (JSON) file. However, the AVS tactile information representation and transmission system architecture 100 shown in FIG1 does not specify how the tactile information exchange format X is encoded to produce a bitstream (as shown in the dashed box in FIG1 ). Therefore, encapsulating and transmitting the tactile media exchange format in the form of a JSON file presents issues that require optimization in terms of memory and bandwidth usage.

[0041] Therefore, an embodiment of the present application proposes a method for processing tactile media content, encoding the exchange format of tactile information to obtain a binary bit stream corresponding to the target tactile information. The above-mentioned binary bit stream is further encapsulated to obtain a media file containing the tactile information. An embodiment of the present application provides a solution for encoding the exchange format of tactile information into a binary bit stream. Specifically, the data is organized and divided in the form of data units and data packets in the above-mentioned binary bit stream, and a data unit contains one or more data packets. The solution provided by the embodiment of the present application is conducive to saving the space occupied by tactile media content.

[0042] Figure 2 is a schematic diagram of an application scenario 200 for processing tactile information according to an embodiment of the present application. Referring to Figure 2 , in this application scenario 200, a terminal 220 can implement tactile presentation, such as vibration. Data can be exchanged between a server 210 and the terminal 220 via a communication network 20. In some embodiments, the terminal 220 can be understood as a decapsulation device, and the server 210 can be understood as a packaging device.

[0043] The terminal 220 can provide computing resources for running the algorithm model during the implementation process. For example, the terminal 220 can specifically be a tactile feedback related product, and a player end or other intermediate node of the immersive system, such as a smart phone, a desktop computer, a mobile computing device, a notebook (e.g., laptop) computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a car computer, a navigation system, a digital phone, a video phone, a television, a sensor device and a server, etc., which are not limited in the embodiment of the present application; the server 210 can specifically be a server end of the immersive system, such as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The communication network 20 can include wired or wireless communication methods for direct or indirect connection, which are not limited in the embodiment of the present application.

[0044] It should be noted that FIG2 is only an example of an application scenario provided by an embodiment of the present application. The application scenarios of the embodiment of the present application include but are not limited to those shown in FIG2 .

[0045] Based on the scenario shown in FIG2 , a tactile information processing method P300 as shown in FIG3 may be executed. Referring to FIG3 , the tactile information processing method P300 includes: S30 to S312 .

[0046] In S30 , the server 210 produces or collects a haptic media signal according to the expected haptic media effect, and generates a haptic media signal exchange format.

[0047] In S32 , the server 210 compresses and encodes the tactile media signal into a tactile media bit stream according to the solution provided in the embodiment of the present application (such as the solution provided in the embodiment of FIG. 5 ).

[0048] In S34 , the server 210 encapsulates the haptic media bitstream into a haptic media file, where the haptic media file includes one or more haptic media tracks.

[0049] In an exemplary embodiment, after determining the haptic media file, the server 210 may further slice the haptic media file into a plurality of haptic media file segments.

[0050] In S36 , the server 210 sends the haptic media file or the haptic media file segment to the terminal 220 , so that the client installed in the terminal can obtain the haptic media file or the haptic media file segment.

[0051] In S38 , the client of the terminal 220 decapsulates the haptic media file to obtain a haptic media bitstream.

[0052] In S310 , the client of the terminal 220 decodes the tactile media bitstream according to the solution provided in the embodiment of the present application (such as the solution provided in the embodiment of FIG. 7 ) to obtain the exchange format of the tactile media signal.

[0053] In S312 , the client of the terminal 220 renders the exchange format of the haptic media signal and performs haptic presentation.

[0054] Before describing the technical solutions of the embodiments of the present application in detail, we first introduce the data structures defined in the prior art for representing tactile media. FIG4 is a schematic diagram of an AVS tactile information representation data structure 400. Referring to FIG4 , the data structures defined in the prior art for representing tactile media are as follows:

[0055] The highest-level data structure for tactile information representation is the tactile experience, which is used to describe all tactile experience information in a file or bitstream. The tactile experience includes metadata related to the tactile experience, possible device information, and one or more tactile modes.

[0056] A tactile pattern corresponds to a certain type of tactile signal (such as vibration, pressure, temperature, etc.). The tactile pattern includes metadata information related to the tactile pattern, possibly predefined knowledge tactile events, and one or more tactile channels.

[0057] A haptic channel contains all or part of the haptic signal of the corresponding haptic mode. A haptic channel includes metadata information related to the haptic channel and one or more haptic events. Generally speaking, haptic signals corresponding to different rendering devices in the same haptic mode can be organized into different haptic channels.

[0058] A tactile event is a basic tactile signal unit, which includes metadata information related to the tactile event and possible tactile event components, wherein the tactile event components are the time domain or frequency domain components of the tactile event.

[0059] 1. The data structure of tactile experience is defined in Table 1;

[0060] Table 1

[0061] 2. The data structure definition of the attribute device information of the tactile experience is shown in Table 2;

[0062] Table 2

[0063] 3. The data structure of the tactile mode is defined in Table 3;

[0064] Table 3

[0065] 4. Knowledge tactile events;

[0066] A knowledge tactile event is a special tactile event whose data structure follows the data structure definition of a tactile event. A knowledge tactile event is usually a predefined tactile event that can be repeatedly referenced to avoid repeated parsing of the same tactile event.

[0067] 5. The tactile channel data structure is defined in Table 4;

[0068] Table 4

[0069] 6. The data structure of tactile events is defined in Table 5;

[0070] Table 5

[0071] 7. Attributes of tactile events The data structure of the tactile event component is defined as shown in Table 6;

[0072] Table 6

[0073] The following describes the technical solutions of the embodiments of the present application in detail through some embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0074] Figure 5 is a flow chart illustrating a method P500 for processing tactile information according to an embodiment of the present application. Method P500 is executed by an electronic device capable of decoding a tactile media bitstream, such as terminal 220 in Figure 2 . The following description uses a decapsulation device as an example. Referring to Figure 5 , method P500 includes steps S510 and S520.

[0075] In S510 , a decapsulation device decapsulates a media file containing target tactile information to obtain a binary bit stream corresponding to the target tactile information. The bit stream includes one or more data units regarding the target tactile information, and the data units include one or more data packets.

[0076] The embodiment of the present application adds several descriptive fields based on the AVS tactile information representation data structure 400 shown in FIG. 4 , including field extensions of the tactile media bitstream, to support the method steps of the embodiment of the present application.

[0077] The binary bit stream obtained by encoding and compressing the tactile media interchange format is a self-contained transmission stream format composed of a series of data units (AHAS Units), and each data unit (AHAS Unit) corresponds to the presentation time of the tactile media.

[0078] Exemplarily, FIG6 is a schematic diagram of the relationship between data units and data packets provided in an embodiment of the present application. Referring to FIG6 , the binary bit stream includes: the i-1th data unit (AHAS Unit (i-1)), the i-th data unit (AHAS Unit (i)) and the i+1th data unit (AHAS Unit (i+1)), and so on. Each data unit includes at least one data packet, such as the i-th data unit (AHAS Unit (i)) in FIG6 includes a data unit header (AHAS Unit Header), data packet 0 (AHAS Packet 0), data packet 1 (AHAS Packet 1), data packet 2 (AHAS Packet 2)...data packet n (AHAS Packet n). Among them, each data packet contains a tactile signal or metadata or auxiliary information of a tactile signal. It will be described in detail in the following embodiments.

[0079] As can be seen from Figure 6, in the embodiment of the present application, data is organized and divided in the bit stream in the form of data units and data packets. Furthermore, data units and data packets are classified into different types, and the type of data unit and data packet determines the type of data contained therein. Table 7 shows the types of data units provided in the embodiment of the present application and the types of data packets contained therein.

[0080] Table 7

[0081] As shown in Table 7, data units are classified into at least one of the following types:

[0082] 1. Initialization type data unit:

[0083] An initialization-type data unit contains one Type 1 packet, one or more Type 2 packets, one or more Type 3 packets, zero, one or more Type 4 packets, and zero or one Type 5 packet. Note that if a Type 4 packet exists in the bitstream, the initialization-type data unit must contain a Type 4 packet.

[0084] 2. Data units of time domain type or space domain type:

[0085] The data unit of the time domain type or the space domain type contains one or more data packets of the seventh type and zero or one data packet of the fifth type.

[0086] 3. Silent type data unit:

[0087] The data unit of the silence type contains a sixth type data packet.

[0088] After introducing the relationship between data units and data packets, as well as the types of data units and data packets in the embodiments of this application, the syntax structures of bit streams, data units, and data packets are respectively introduced through Tables 8 to 10. In the embodiments of this application, the syntax and semantics of the bit stream are shown in Table 8.

[0089] Table 8

[0090] Among them, endOfStream() is a function for determining whether the data unit in the bit stream ends. If the data unit in the bit stream ends, the function returns 1; if the data unit in the bit stream does not end, the function returns 0.

[0091] In the embodiment of the present application, the syntax and semantics of the data unit (AHAS Unit) are shown in Table 9. The number of bits shown in Table 9 is exemplary data and may also be other values, which are not limited in the embodiment of the present application.

[0092] Table 9

[0093] Referring to Table 9, AHASUnitType indicates the type of the data unit. In the embodiment of the present application, different types of data units correspond to different binary values. Referring to Table 9, it can be seen that the percentage of the type of the data unit is 3. For example, the type of the data unit is a silent type, which can be represented as "100". Exemplarily, the meaning of the binary value of this field can be shown in Table 9.1.

[0094] Table 9.1

[0095] Referring to Table 9, AHASUnitDependency represents a decoding dependency indicator.

[0096] If the value of this field is the first target value, it means that the current data unit does not depend on other data units of the same type during decoding; if the value of this field is not the above-mentioned first target value, it means that the current data unit depends on other data units of the same type during decoding.

[0097] In some embodiments, the first target value may be "0" instead of "1." Because initialization-type data units and silence-type data units do not rely on other data units of the same type during decoding, the value of this field in initialization-type data units and silence-type data units is 0.

[0098] If the data unit of the initialization type is followed by a data unit of the time domain or space domain type and the AHASUnitDependency field value of the data unit of the time domain or space domain type is the first target value (e.g., 0), then the data unit of the initialization type can be considered as a decoding initial unit, that is, a unit that can directly start decoding the bit stream, that is, a random access point for decoding the bit stream. In an exemplary embodiment, the current bit stream can be decoded directly from the data unit of the initialization type. The embodiment of the present application does not limit the above-mentioned first target value.

[0099] Referring to Table 9, AHASUnitLayerFlag represents a layer indicator.

[0100] The above-mentioned level indicator is represented by a binary value. When the value of this field is the second target value, it indicates that the level of the data unit is not indicated; when the value of this field is not the above-mentioned second target value, it indicates that the level of the data unit is indicated. In an exemplary embodiment, the above-mentioned second target value can be "0", and the value other than the above-mentioned second target value can be "1". Then, when the level indicator AHASUnitLayerFlag has a value of 0, it indicates that the level of the data unit is not indicated; when the value of this field is 1, it indicates that the level of the data unit is indicated. Of course, the embodiment of the present application does not limit the above-mentioned second target value.

[0101] Referring to Table 9, AHASLayer represents the level identifier of the data unit.

[0102] In the embodiment of the present application, the above-mentioned level identifier is also represented by a binary value. At the same time, the priority of the data unit can be determined by the value of the level identifier.

[0103] In some exemplary embodiments, a smaller value of this field indicates a higher priority for the data unit; conversely, a larger value of this field indicates a lower priority for the data unit. It is understood that in other embodiments, a larger value of the AHASLayer field indicates a higher priority for the data unit; conversely, a smaller value of this field indicates a lower priority for the data unit. This embodiment of the present application is not limited to this.

[0104] Referring to Table 9, AHASUnitTimestamp represents the first offset timestamp. Different timestamps correspond to different binary values. In the embodiment shown in Table 9, the first offset timestamp may occupy 32 bits.

[0105] The first offset timestamp represents the offset time of the data unit relative to the start time of the tactile experience, in clocks. The number of clocks contained in one second is specified by the timescale field in the experience metadata. For example, the ratio of the first offset timestamp AHASUnitTimestamp to timescale is the number of seconds that the tactile experience lasted.

[0106] For different types of data units, the value of this field can follow the following constraints:

[0107] -For initialized data units, this field has no actual meaning;

[0108] -For a silence type data unit, this field shall be greater than the timestamp of any data packet in the previous temporal or spatial type data unit of the silence type data unit.

[0109] -For a data unit of a time domain type or a spatial domain type, the timestamp of any data packet contained in the i-th data unit shall be less than the timestamp of any data packet contained in the i+1-th data unit. In some embodiments, this field shall be the minimum value of the timestamps of the M data packets contained therein. In other embodiments, this field shall be the maximum value of the timestamps of the M data packets contained therein.

[0110] The first offset timestamp can be used to implement random access to the tactile signal during transmission. The specific implementation will be described in detail in the subsequent embodiment of the second offset timestamp.

[0111] Referring to Table 9, AHASUnitLength represents the number of bytes of all data packets in the data unit. Different numbers of bytes correspond to different binary values. In the embodiment shown in Table 9, the number of bytes of all data packets in the above data unit can occupy 32 bits.

[0112] Referring to Table 9, endOfUnit() is a function for determining whether the data packet in the data unit ends. If the data packet in the data unit ends, the function returns 1; if the data unit does not end, the function returns 0. Exemplarily, the data unit ends after parsing AHASUnitLength bytes of data packet data contained in the data packet.

[0113] Next, the syntax and semantics of the data unit (AHAS Packet) are introduced through Table 10. The number of bits shown in Table 10 is exemplary data and may also be other values, which are not limited in the embodiments of the present application.

[0114] Table 10

[0115] Referring to Table 10, AHASPacketType indicates the type of data packet. In the embodiments of the present application, different data packet types correspond to different binary values. Referring to Table 10, it can be seen that the percentage of the data packet type is 4. For example, the data packet type of type 6 containing silence information can be represented as "0110". For example, the binary value meaning of this field can be shown in Table 10.1.

[0116] Table 10.1

[0117] Referring to Table 10, AHASPacketLength is used to indicate the number of bytes in the payload of the data packet, that is, the number of bytes in AHASPacketPayload. In the embodiment of the present application, the number of bytes in the payload of the data packet can be represented by a binary value. Referring to Table 10, it can be seen that the percentage of bytes in the payload of the data packet is 20.

[0118] Referring to Table 10, AHASPacketTimestamp represents the second offset timestamp. Different timestamps correspond to different binary values. In the embodiment shown in Table 10, the second offset timestamp may occupy 24 bits.

[0119] The second offset timestamp is used to represent the offset relative to the first timestamp of the data unit containing the data packet, in units of clocks. The number of clocks contained in one second is specified by the timescale field in the experience metadata. Specifically, in some embodiments, the second offset timestamp AHASPacketTimestamp represents the minimum value among the timestamps of all tactile events contained in the data packet. In other embodiments, the second offset timestamp AHASPacketTimestamp represents the maximum value among the timestamps of all tactile events contained in the data packet.

[0120] In the embodiment of the present application, the timestamp of the current tactile signal data packet may be determined according to the second offset timestamp of the data packet and the first offset timestamp of the data unit where the data packet is located.

[0121] When the first offset timestamp AHASUnitTimestamp represents the minimum value of the timestamps of all the data packets it contains, AHASUnitTimestamp+AHASPacketTimestamp can be determined as the timestamp of the current tactile signal data packet. When the first offset timestamp AHASUnitTimestamp represents the maximum value of the timestamps of all the data packets it contains, AHASUnitTimestamp-AHASPacketTimestamp can be determined as the timestamp of the current tactile signal data packet.

[0122] It can be seen that the solution provided by the embodiment of the present application can determine the timestamp of the current tactile signal data packet through the second offset timestamp of the current data packet and the first offset timestamp of the data unit where the data packet is located, thereby supporting random access to the tactile signal during transmission.

[0123] Refer to Table 10. byteAlignment() is a function that determines byte alignment. If the data contained in the current packet is an integer number of bytes, it returns 0. If the data contained in the current packet is less than an integer number of bytes, it pads the last byte and returns 0.

[0124] Referring to Table 10, the syntax of AHASPacketPayload can be introduced in conjunction with Table 10.2.

[0125] Table 10.2

[0126] Depending on the packet type, the syntax in Table 10.2 will be used in conjunction with the payload syntax of the corresponding packet type, as shown in Tables 10.2-A through 10.2-G. The bit numbers shown in Tables 10.2-A through 10.2-G are exemplary and may be other values. This embodiment of the present application does not limit this.

[0127] In the case where the data packet type is the first type including haptic experience metadata, the syntax of the payload of the data packet shown in Table 10.2-A will be used during the execution of the syntax in Table 10.2.

[0128] Table 10.2-A

[0129] When the data packet type is the first type containing tactile experience metadata, the payload of the data packet includes one or more of the following fields shown in Table 10.2-A. Each field in Table 10.2-A can be represented by a binary value. Specifically, the specific indication information of each field is as follows:

[0130] Version is used to indicate the standard version information that the tactile experience complies with; the binary value used to represent Version can occupy 8 bits.

[0131] dateLength is a string indicating the length of the creation date of the haptic experience. The binary value representing dateLength can occupy 8 bits.

[0132] Date is used to indicate the creation date of the tactile experience. The binary value used to represent DateDate can occupy dateLength×8 bits.

[0133] descriptionLength is used to indicate the length of the string describing the tactile experience; the binary value used to represent DateDate can occupy 8 bits.

[0134] Description is used to indicate the tactile experience description; the binary value used to represent Description can occupy descriptionLength×8 bits.

[0135] Timescale is used to indicate the time scale of the tactile experience. The value of this field represents the number of clock units contained in one second. The binary value used to represent Timescale can occupy 16 bits.

[0136] patternCount is used to indicate the number of patterns included in the tactile experience; the binary value used to represent patternCount can occupy 8 bits.

[0137] deviceCount is used to indicate the number of devices corresponding to the tactile experience; the binary value used to represent deviceCount can occupy 8 bits.

[0138] For the syntax of readDevice() in Table 10.2-A, please refer to Table 10.2-A1.

[0139] Table 10.2-A1

[0140] The payload of the above data packet also includes: one or more of the following fields as shown in Table 10.2-A1 regarding the jth device. Among them, each field in Table 10.2-A1 can be represented by a binary value. Specifically, the specific indication information of each field is as follows:

[0141] deviceId is used to indicate the identifier of the j-th device; the binary value used to represent deviceId can occupy 8 bits.

[0142] nameLength is used to indicate the length of the character string of the j-th device name; the binary value used to represent nameLength can occupy 8 bits.

[0143] deviceName is used to indicate the jth human-readable device name. The binary value used to represent deviceName can occupy nameLength × 8 bits.

[0144] bodyPartFlag is an indicator of whether the device corresponds to a body part; the binary value used to represent bodyPartFlag may occupy 1 bit; for example, when the value is 1, it indicates the body part information corresponding to the device; when the value is 0, it does not indicate the body part information corresponding to the device.

[0145] bodyParts is used to indicate the body part corresponding to the device; the binary value used to represent bodyParts can occupy 8 bits.

[0146] In the case where the data packet type is the second type including tactile mode metadata, the syntax of the payload of the data packet shown in Table 10.2-B will be used during the syntax execution process of Table 10.2.

[0147] Table 10.2-B

[0148] In the case where the data packet type is the second type including tactile mode metadata, the payload of the data packet includes one or more of the following fields shown in Table 10.2-B. Each field in Table 10.2-B can be represented by a binary value. Specifically, the specific indication information of each field is as follows:

[0149] Id is used to indicate the identifier of the tactile mode; the binary value used to represent Id may occupy 8 bits.

[0150] patternType is used to indicate the type of tactile pattern. The binary value used to represent patternType can occupy 8 bits. For example, if the data pattern type is acceleration, it can be represented as "0000 0010". For example, the meaning of the binary value of this field can be shown in Table 10.2-B1.

[0151] Table 10.2-B1

[0152] semanticisFlag is an indicator for indicating whether the tactile event corresponding to the tactile mode has semantics; the binary value used to indicate semanticisFlag may occupy 1 bit, for example, when the value is 1, it indicates that the tactile event corresponding to the tactile mode has semantics, and when the value is 0, it indicates that the tactile event corresponding to the tactile mode does not have semantics.

[0153] sementicsType is an indicator used to indicate the specifications followed by the semantics of tactile events. The binary value used to indicate sementicsType can occupy 8 bits. For example, a value of 0 indicates that the semantics of the tactile event follow the specifications defined in this standard; a value of 1 indicates that the semantics of the tactile event follow the specifications of an external standard; and a value of 2 indicates that the semantics of the tactile event are user-defined.

[0154] referenceDeviceCount is used to indicate the number of devices corresponding to the tactile mode; the binary value used to indicate referenceDeviceCount may occupy 8 bits.

[0155] referenceDeviceId is used to indicate the identifier of the device corresponding to the tactile mode; the binary value used to represent referenceDeviceId may occupy 8 bits.

[0156] channelCount is used to indicate the number of tactile channels included in the tactile pattern; the binary value used to represent channelCount may occupy 16 bits.

[0157] In the case where the data packet type is the third type containing tactile channel metadata, the syntax of the payload of the data packet shown in Table 10.2-C will be used during the execution of the syntax in Table 10.2.

[0158] Table 10.2-C

[0159] When the data packet type is the third type including tactile channel metadata, the payload of the data packet includes one or more of the following fields shown in Table 10.2-C. Each field in Table 10.2-C can be represented by a binary value. Specifically, the specific indication information of each field is as follows:

[0160] Id is used to indicate the identifier of the tactile channel; the binary value used to represent Id may occupy 16 bits.

[0161] perceptionId is used to indicate the identifier of the tactile mode corresponding to the tactile channel; the binary value used to represent perceptionId can occupy 8 bits.

[0162] descriptionLength is used to indicate the length of the character string of the tactile channel description; the binary value used to represent descriptionLength can occupy 8 bits.

[0163] Description is used to indicate the tactile channel description; the binary value used to represent Description may occupy 8×descriptionLength bits.

[0164] Gain indicates the gain of the tactile channel. The binary value representing Gain occupies 32 bits, with units ranging from 2 to 31. This gain is applied to all events in the current channel, restoring the normalized signal value to the original value.

[0165] referenceDeviceCount is used to indicate the number of devices corresponding to the tactile channel; the binary value used to represent referenceDeviceCount may occupy 8 bits.

[0166] referenceDeviceId is used to indicate the identifier of the device corresponding to the tactile channel; the binary value used to represent referenceDeviceId can occupy 8 bits.

[0167] eventsCount is used to indicate the number of tactile events contained in the tactile channel; the binary value used to represent eventsCount can occupy 16 bits.

[0168] In the case where the data packet type is the seventh type containing a haptic signal, the syntax of the payload of the data packet shown in Table 10.2-D will be used during the execution of the syntax in Table 10.2.

[0169] Table 10.2-D

[0170] In the case where the data packet type is the seventh type including a tactile signal, the payload of the data packet includes one or more of the following fields shown in Table 10.2-D. Each field in Table 10.2-D may be represented by a binary value. Specifically, the specific indication information of each field is as follows:

[0171] packetDependency is an indicator of whether the current data packet depends on other data packets of the same type when decoding. The binary value used to represent packetDependency can occupy 1 bit. For example, when the value of this field is 0, it indicates that the current data packet does not depend on other data packets of the same type when decoding. When the value of this field is 1, it indicates that the current data packet depends on other data packets of the same type when decoding.

[0172] perceptionId is used to indicate the identifier of the tactile mode corresponding to the corresponding tactile event in the data packet; the binary value used to represent perceptionId can occupy 8 bits.

[0173] channelId is used to indicate the identifier of the tactile channel corresponding to the corresponding tactile event in the data packet; the binary value used to represent channelId may occupy 8 bits.

[0174] eventsCount is used to indicate the number of tactile events corresponding to the data packet; the binary value used to represent eventsCount may occupy 16 bits.

[0175] For the syntax of readEvent() in Table 10.2-D, please refer to Table 10.2-D1.

[0176] Table 10.2-D1

[0177] The payload of the above data packet also includes: one or more of the following fields as shown in Table 10.2-D1 regarding the mth tactile event. Among them, each field in Table 10.2-D1 can be represented by a binary value. Specifically, the specific indication information of each field is as follows:

[0178] eventId is used to indicate the identifier of a tactile event; a binary value used to represent eventId may occupy 16 bits.

[0179] eventType is used to indicate the type of tactile event; the binary value used to represent eventType can occupy 4 bits. For example, a value of 0 indicates that the event is an instantaneous event, a value of 1 indicates that the event is a continuous event, and a value of 2 indicates that the event is a reference event, etc.

[0180] eventSementicsFlag is an indicator value indicating whether the semantics of a tactile event is indicated; the binary value used to indicate eventSementicsFlag may occupy 1 bit, for example, when the value is 0, the semantics of a tactile event is not indicated, and when the value is 1, the semantics of a tactile event is indicated.

[0181] semanticKeywords represents the semantic information of the tactile event; the binary value used to represent semanticKeywords can occupy 12 bits. The semantics of the tactile event are event auxiliary information keywords that describe the content creator's intention.

[0182] referEventId is used to indicate the knowledge tactile event identifier corresponding to the reference event; the binary value used to represent referEventId can occupy 16 bits.

[0183] relativePosition is used to indicate the time or space offset of a tactile event; the binary value used to represent relativePosition may occupy 32 bits.

[0184] Duration is used to indicate the duration of a tactile event. This attribute is present when the event type is a continuous event. The binary value used to represent Duration can occupy 32 bits.

[0185] Amplitude is used to indicate the maximum amplitude value of the tactile event signal; the binary value used to represent Amplitude can occupy 8 bits.

[0186] baseFrequency is used to indicate the base frequency of the signal of the tactile event; the binary value used to represent baseFrequency may occupy 16 bits.

[0187] numComponents is used to indicate the number of tactile event components. The tactile event components may be time domain components or frequency domain components of the tactile event. The binary value representing numComponents may occupy 16 bits.

[0188] For the syntax of readEventComponent() in Table 10.2-D1, please refer to Table 10.2-D2.

[0189] Table 10.2-D2

[0190] The payload of the above data packet also includes: one or more of the following fields as shown in Table 10.2-D2 regarding the kth tactile event component. Among them, each field in Table 10.2-D2 can be represented by a binary value. Specifically, the specific indication information of each field is as follows:

[0191] refer_event_id is used to indicate the identifier of the tactile event to which the tactile event component belongs; the binary value used to represent refer_event_id may occupy 16 bits.

[0192] position_flag is an indicator used to indicate whether the tactile event component contains a time-space domain offset; the binary value used to indicate position_flag can occupy 1 bit. For example, when the position_flag value is 1, it indicates that the tactile event component contains a time-space domain offset, and when the position_flag value is 0, it indicates that the tactile event component does not contain a time-space domain offset.

[0193] Amplitude_flag is an indicator used to indicate whether the tactile event component contains an amplitude value offset; the binary value used to indicate amplitude_flag can occupy 1 bit. For example, when the value of amplitude_flag is 1, it indicates that the tactile event component contains an amplitude value offset, and when the value is 0, it indicates that the tactile event component does not contain an amplitude value offset.

[0194] frequency_flag is an indicator for indicating whether the tactile event component includes a frequency offset; the binary value used to indicate frequency_flag may occupy 1 bit, for example, when the value of frequency_flag is 1, it indicates that the tactile event component includes a frequency offset, and when the value is 0, it indicates that the tactile event component does not include a frequency offset.

[0195] relative_position is used to represent the time or space offset of the tactile event component relative to the tactile event; the binary value used to represent relative_position may occupy 16 bits.

[0196] relative_amplitude is used to represent the amplitude ratio of the tactile event component to the tactile event, with a value range of 0 to 215 in units of 2-15; the binary value used to represent relative_amplitude can occupy 16 bits.

[0197] relative_frequency is used to represent the frequency offset of the tactile event component relative to the tactile event; the binary value used to represent relative_frequency may occupy 16 bits.

[0198] In the case where the data packet type is the fourth type containing a knowledge tactile event, the syntax of the payload of the data packet shown in Table 10.2-E will be used during execution of the syntax in Table 10.2.

[0199] Table 10.2-E

[0200] When the data packet type is the fourth type including knowledge tactile events, the payload of the data packet includes one or more of the following fields shown in Table 10.2-E. Each field in Table 10.2-E can be represented by a binary value. Specifically, the specific indication information of each field is as follows:

[0201] patternId is used to represent the identifier of the tactile pattern corresponding to the knowledge tactile event; the binary value used to represent patternId can occupy 8 bits.

[0202] eventsCount is used to represent the number of knowledge tactile events corresponding to the data packet; the binary value used to represent eventsCount can occupy 16 bits.

[0203] For the syntax of readLibraryEvent() in Table 10.2-E, please refer to Table 10.2-E1.

[0204] Table 102-E1

[0205] The payload of the above data packet also includes: one or more of the following fields as shown in Table 10.2-E1 regarding the hth knowledge tactile event. Among them, each field in Table 10.2-E1 can be represented by a binary value. Specifically, the specific indication information of each field is as follows:

[0206] eventId is used to represent the identifier of the knowledge tactile event; the binary value used to represent eventId can occupy 16 bits.

[0207] eventType is used to indicate the type of the knowledge tactile event. The binary value used to indicate eventType can occupy 4 bits. For example, a value of 0 indicates that the event is an instantaneous event, and a value of 1 indicates that the event is a continuous event.

[0208] eventSementicsFlag is an indicator for indicating whether the semantics of the knowledge tactile event is indicated; the binary value used to indicate eventSementicsFlag may occupy 1 bit, for example, when the value is 0, the semantics of the knowledge tactile event is not indicated, and when the value is 1, the semantics of the knowledge tactile event is indicated.

[0209] SemanticKeywords is used to represent the semantic information of an event; the binary value used to represent semanticKeywords can occupy 12 bits. The semantics of a knowledge tactile event is the event auxiliary information keyword that describes the content creator's intention.

[0210] relativePosition is used to represent the time or space offset of the knowledge tactile event; the binary value used to represent relativePosition can occupy 32 bits.

[0211] Duration is used to indicate the duration of a knowledge tactile event. This attribute exists when the event type is a continuous event. The binary value used to indicate Duration can occupy 32 bits.

[0212] Amplitude is used to represent the maximum amplitude value of the signal of the knowledge tactile event; the binary value used to represent Amplitude can occupy 8 bits.

[0213] baseFrequency is used to represent the signal base frequency of the knowledge tactile event; the binary value used to represent baseFrequency can occupy 16 bits.

[0214] numComponents is used to represent the number of knowledge tactile event components. The knowledge tactile event components may be time domain components or frequency domain components of the knowledge tactile event. The binary value used to represent numComponents may occupy 16 bits.

[0215] For the syntax of readEventComponent() in Table 10.2-E1, please refer to Table 10.2-D2. We will not go into details here.

[0216] In the case where the data packet type is the sixth type containing silent data, the syntax of the payload of the data packet shown in Table 10.2-F will be used during the execution of the syntax in Table 10.2.

[0217] Table 10.2-F

[0218] When the data packet type is type 6 including silent data, the payload of the data packet includes the fields shown in Table 10.2-F. The fields in Table 10.2-F can be represented by binary values. Specifically, the specific indication information of the field is as follows:

[0219] silentDuration is used to indicate the length of the silent period, in units of time ticks. The binary value used to indicate silentDuration may occupy 16 bits.

[0220] When the data packet type is the fifth type including a cyclic check code, the syntax execution process of Table 10.2 will turn to the syntax of the payload of the data packet shown in Table 10.2-G.

[0221] Table 10.2-G

[0222] When the data packet type is the fifth type including a cyclic check code, the payload of the data packet includes one or more of the following fields shown in Table 10.2-G. Each field in Table 10.2-G can be represented by a binary value. Specifically, the specific indication information of each field is as follows:

[0223] CRCType is used to indicate the type of cyclic check code. The binary value used to indicate CRCTypey can occupy 8 bits. For example, a value of 0 indicates that the cyclic check code is a 16-bit check code, and the data packet protected by the cyclic check code is the data packet following the cyclic check code data packet. A value of 1 indicates that the cyclic check code is a 32-bit check code, and the data packet protected by the cyclic check code is the data packet following the cyclic check code data packet. A value of 2 indicates that the cyclic check code is a 16-bit check code, and the data packet range protected by the cyclic check code is protectedPacketsCount data packets following the cyclic check code data packet. A value of 3 indicates that the cyclic check code is a 32-bit check code, and the data packet range protected by the cyclic check code is protectedPacketsCount data packets following the cyclic check code data packet.

[0224] CRC16Value is used to represent a 16-bit checksum. The binary value used to represent CRC16Value can occupy 16 bits.

[0225] CRC32Value is used to represent a 32-bit checksum. The binary value used to represent CRC32Value can occupy 32 bits.

[0226] protectedPacketsCount is used to represent the number of packets protected by the cyclic check code; the binary value used to represent protectedPacketsCount can occupy 8 bits.

[0227] Continuing to refer to FIG. 5 , in S520 , the decapsulation device decodes the binary bit stream corresponding to the target tactile information to obtain the exchange format of the target tactile information.

[0228] The data exchange format refers to a standardized format used when exchanging data between different systems. In an embodiment of the present application, the exchange format of the above-mentioned target tactile information is a standardized format used for exchanging data between a server side (i.e., the encapsulation device side) and a terminal (i.e., the decapsulation device side) (such as an interactive device) that presents the tactile information. Specifically, the terminal (such as an interactive device) can obtain the exchange format of the above-mentioned target tactile information after decoding the bit stream. Furthermore, the above-mentioned target tactile information can be rendered and presented based on the exchange format of the above-mentioned target tactile information. For example, if the above-mentioned target tactile information is information about vibration, the terminal can present the corresponding vibration effect at the terminal according to the exchange format of the above-mentioned target information. Among them, the exchange format of tactile information is not limited to JSON, but can also be other types of data exchange formats, such as data exchange formats such as Extensible Markup Language (XML) and Comma-Separated Values ​​(CSV).

[0229] In an embodiment of the present application, the decapsulation device is based on the AVS tactile information representation data structure 400 shown in Figure 4, and the field extension of the tactile media bit stream provided in the embodiment of the present application (such as the embodiments provided in Tables 7 to 10.2 above), and can decode the binary bit stream corresponding to the above-mentioned target tactile information to obtain the exchange format of the above-mentioned target tactile information.

[0230] Furthermore, the decapsulation device (eg, a terminal client) may render the exchange format of the target tactile information and perform tactile presentation.

[0231] In the solution provided by the embodiment shown in Figure 5, the decapsulation device decodes the binary bit stream corresponding to the target tactile information to obtain the exchange format of the target tactile information. The above-mentioned binary bit stream is obtained based on the solution provided by the embodiment of the present application for encoding the exchange format of tactile information into a binary bit stream. Specifically, the data is organized and divided in the form of data units and data packets in the above-mentioned binary bit stream, and a data unit contains one or more data packets. The solution provided by the embodiment of the present application is conducive to saving the space occupied by tactile media content and can reduce bandwidth occupancy during the transmission of tactile media content. It also supports random access to tactile signals during transmission.

[0232] The solution for decoding the corresponding bit stream of target tactile information in the embodiment corresponding to Figure 5 is described in detail. The following embodiment of encoding the exchange format of target tactile information provided by the present application to obtain the corresponding bit stream of tactile information is described in detail with reference to Figure 7.

[0233] Figure 7 is a flow chart illustrating a method P700 for processing tactile information according to an embodiment of the present application. Method P700 is executed by an electronic device capable of encoding the target tactile information in an exchange format to obtain a tactile media bitstream, such as server 210 in Figure 2 or by a processor. Referring to Figure 7 , method P700 includes steps S710 and S720.

[0234] In S710 , the exchange format of the target tactile information is encoded to obtain a binary bit stream corresponding to the target tactile information, where the bit stream includes one or more data units regarding the target tactile information, and the data units include one or more data packets.

[0235] The embodiment of the present application adds several descriptive fields based on the AVS tactile information representation data structure 400 shown in FIG. 4 , including field extensions of the tactile media bitstream, to support the method steps of the embodiment of the present application.

[0236] The data exchange format refers to a standardized format used when exchanging data between different systems. In an embodiment of the present application, the exchange format of the target tactile information is a standardized format used for exchanging data between the server side (i.e., the encapsulation device side) and the terminal (i.e., the decapsulation device side) (such as the interactive device) that presents the tactile information. Specifically, in order to facilitate transmission, the server side encodes the exchange format of the target tactile information into a binary bit stream, and transmits the binary bit stream to the terminal (such as the interactive device). Thus, after obtaining the exchange format of the target tactile information, the terminal (such as the interactive device) can render and present based on the target tactile information. For example, if the target tactile information is information about vibration, the terminal can present the corresponding vibration effect at the terminal according to the exchange format of the target information. Among them, the exchange format of tactile information is not limited to JSON, but can also be other types of data exchange formats, such as Extensible Markup Language (XML), Comma-Separated Values ​​(CSV) and other data exchange formats.

[0237] The binary bit stream obtained by encoding and compressing the tactile media interchange format is a self-contained transmission stream format composed of a series of data units (AHAS Units), and each data unit (AHAS Unit) corresponds to the presentation time of the tactile media.

[0238] Exemplarily, FIG6 is a schematic diagram of the relationship between data units and data packets provided in an embodiment of the present application. Referring to FIG6 , the binary bit stream includes: the i-1th data unit (AHAS Unit (i-1)), the i-th data unit (AHAS Unit (i)) and the i+1th data unit (AHAS Unit (i+1)), and so on. Each data unit includes at least one data packet, such as the i-th data unit (AHAS Unit (i)) in FIG6 includes a data unit header (AHAS Unit Header), data packet 0 (AHAS Packet 0), data packet 1 (AHAS Packet 1), data packet 2 (AHAS Packet 2)...data packet n (AHAS Packet n). Among them, each data packet contains a tactile signal or metadata or auxiliary information of a tactile signal. It will be described in detail in the following embodiments.

[0239] As can be seen from Figure 6, in the embodiment of the present application, data is organized and divided in the bit stream in the form of data units and data packets. Furthermore, data units and data packets are classified into different types, and the type of data unit and data packet determines the type of data contained therein. Table 7 shows the types of data units provided in the embodiment of the present application and the types of data packets contained therein.

[0240] As shown in Table 7, data units are classified into at least one of the following types:

[0241] 1. Initialization type data unit:

[0242] The data unit of the initialization type contains one data packet of the first type, one or more data packets of the second type, one or more data packets of the third type, zero, one or more data packets of the fourth type, and zero or one data packet of the fifth type.

[0243] 2. Data units of time domain type or space domain type:

[0244] The data unit of the time domain type or the space domain type contains one or more data packets of the seventh type and zero or one data packet of the fifth type.

[0245] 3. Silent type data unit:

[0246] The data unit of the silence type contains a sixth type data packet.

[0247] After introducing the relationship between data units and data packets, as well as the types of data units and data packets in the embodiment of the present application, the syntax structures of bit streams, data units, and data packets will be introduced respectively through Tables 8 to 10. In the embodiment of the present application, the syntax and semantics of the bit stream are shown in Table 8.

[0248] In the embodiment of the present application, the syntax and semantics of the data unit (AHAS Unit) are shown in Table 11. The number of bits shown in Table 11 is exemplary data and may also be other values, which are not limited in the embodiment of the present application.

[0249] Table 11

[0250] Referring to Table 11, AHASUnitType indicates the type of the data unit. In the embodiment of the present application, different types of data units correspond to different binary values. Referring to Table 11, it can be seen that the percentage of the type of the data unit is 3. For example, the type of the data unit is a silent type, which can be represented as "100". Exemplarily, the meaning of the binary value of this field can be shown in Table 11.1.

[0251] Table 11.1

[0252] Referring to Table 11, AHASUnitDependency represents an encoding dependency indicator.

[0253] If the value of this field is the first target value, it means that the current data unit does not depend on other data units of the same type during encoding; if the value of this field is not the above-mentioned first target value, it means that the current data unit depends on other data units of the same type during encoding.

[0254] In some embodiments, the first target value may be "0" instead of "1." Because initialization-type data units and silence-type data units do not rely on other data units of the same type during encoding, the value of this field in initialization-type data units and silence-type data units is 0.

[0255] If the data unit of the initialization type is followed by a data unit of the time domain or space domain type and the AHASUnitDependency field value of the data unit of the time domain or space domain type is the first target value (e.g., 0), then the data unit of the initialization type can be considered as an initial unit for encoding, that is, a unit that can directly start encoding the bit stream, that is, a random access point for encoding the bit stream. In an exemplary embodiment, the current bit stream can be encoded directly from the data unit of the initialization type. The embodiment of the present application does not limit the above-mentioned first target value.

[0256] Referring to Table 11, AHASUnitLayerFlag represents a layer indicator.

[0257] The above-mentioned level indicator is represented by a binary value. When the value of this field is the second target value, it indicates that the level of the data unit is not indicated; when the value of this field is not the above-mentioned second target value, it indicates that the level of the data unit is indicated. In an exemplary embodiment, the above-mentioned second target value can be "0", and the value other than the above-mentioned second target value can be "1". Then, when the level indicator AHASUnitLayerFlag has a value of 0, it indicates that the level of the data unit is not indicated; when the value of this field is 1, it indicates that the level of the data unit is indicated. Of course, the embodiment of the present application does not limit the above-mentioned second target value.

[0258] Referring to Table 11, AHASLayer represents the level identifier of the data unit.

[0259] In the embodiment of the present application, the above-mentioned level identifier is also represented by a binary value. At the same time, the priority of the data unit can be determined by the value of the level identifier.

[0260] In some exemplary embodiments, a smaller value of this field indicates a higher priority for the data unit; conversely, a larger value of this field indicates a lower priority for the data unit. It is understood that in other embodiments, a larger value of the AHASLayer field indicates a higher priority for the data unit; conversely, a smaller value of this field indicates a lower priority for the data unit. This embodiment of the present application is not limited to this.

[0261] Referring to Table 11, AHASUnitTimestamp represents the first offset timestamp. Different timestamps correspond to different binary values. In the embodiment shown in Table 11, the first offset timestamp may occupy 32 bits.

[0262] The first offset timestamp represents the offset time of the data unit relative to the start time of the tactile experience, in clocks. The number of clocks contained in one second is specified by the timescale field in the experience metadata. For example, the ratio of the first offset timestamp AHASUnitTimestamp to timescale is the number of seconds that the tactile experience lasted.

[0263] For different types of data units, the value of this field can follow the following constraints:

[0264] -For initialized data units, this field has no actual meaning;

[0265] -For a silence type data unit, this field should be greater than the timestamp of any data packet in the previous time domain / spatial domain type data unit of the silence type data unit.

[0266] -For a data unit of a time domain type or a spatial domain type, the timestamp of any data packet contained in the i-th data unit shall be less than the timestamp of any data packet contained in the i+1-th data unit. In some embodiments, this field shall be the minimum value of the timestamps of the M data packets contained therein. In other embodiments, this field shall be the maximum value of the timestamps of the M data packets contained therein.

[0267] The first offset timestamp can be used to implement random access to the tactile signal during transmission. The specific implementation will be described in detail in the subsequent embodiment of the second offset timestamp.

[0268] Referring to Table 11, AHASUnitLength represents the number of bytes of all data packets in the data unit. Different numbers of bytes correspond to different binary values. In the embodiment shown in Table 9, the number of bytes of all data packets in the above data unit can occupy 32 bits.

[0269] Referring to Table 11, endOfUnit() is a function for determining whether the data packet in the data unit ends. If the data packet in the data unit ends, the function returns 1; if the data unit does not end, the function returns 0. Exemplarily, the data unit ends after parsing AHASUnitLength bytes of data packet data contained in the data packet.

[0270] The syntax and semantics of the data unit (AHAS Packet) are introduced through Table 12. The number of bits shown in Table 12 is exemplary data and may also be other values, which are not limited in the embodiments of the present application.

[0271] Table 12

[0272] Referring to Table 12, AHASPacketType indicates the type of data packet. In the embodiment of the present application, different data packet types correspond to different binary values. Referring to Table 12, it can be seen that the percentage of the data packet type is 4. For example, the data packet type is the sixth type containing silent information, which can be represented as "0110". Exemplarily, the binary value meaning of this field can be as shown in Table 10.1 above and will not be repeated here.

[0273] Referring to Table 12, AHASPacketLength is used to indicate the number of bytes in the payload of the data packet, that is, the number of bytes in AHASPacketPayload. In the embodiment of the present application, the number of bytes in the payload of the data packet can be represented by a binary value. Referring to Table 12, it can be seen that the percentage of bytes in the payload of the data packet is 20.

[0274] Referring to Table 12, AHASPacketTimestamp represents the second offset timestamp. Different timestamps correspond to different binary values. In the embodiment shown in Table 12, the second offset timestamp may occupy 24 bits.

[0275] The second offset timestamp is used to represent the offset relative to the first timestamp of the data unit containing the data packet, in units of clocks. The number of clocks contained in one second is specified by the timescale field in the experience metadata. Specifically, in some embodiments, the second offset timestamp AHASPacketTimestamp represents the minimum value among the timestamps of all tactile events contained in the data packet. In other embodiments, the second offset timestamp AHASPacketTimestamp represents the maximum value among the timestamps of all tactile events contained in the data packet.

[0276] In the embodiment of the present application, the timestamp of the current tactile signal data packet may be determined according to the second offset timestamp of the data packet and the first offset timestamp of the data unit where the data packet is located.

[0277] When the first offset timestamp AHASUnitTimestamp represents the minimum value of the timestamps of all the data packets it contains, AHASUnitTimestamp+AHASPacketTimestamp can be determined as the timestamp of the current tactile signal data packet. When the first offset timestamp AHASUnitTimestamp represents the maximum value of the timestamps of all the data packets it contains, AHASUnitTimestamp-AHASPacketTimestamp can be determined as the timestamp of the current tactile signal data packet.

[0278] It can be seen that the solution provided by the embodiment of the present application can determine the timestamp of the current tactile signal data packet through the second offset timestamp of the current data packet and the first offset timestamp of the data unit where the data packet is located, thereby supporting random access to the tactile signal during transmission.

[0279] Refer to Table 12. byteAlignment() is a function that determines byte alignment. If the data contained in the current packet is an integer number of bytes, it returns 0. If the data contained in the current packet is less than an integer number of bytes, it pads the last byte and returns 0.

[0280] Referring to Table 12, the syntax of AHASPacketPayload can be introduced in conjunction with Table 12.2 provided above.

[0281] Table 12.2

[0282] Depending on the packet type, the syntax in Table 12.2 will be used in conjunction with the payload syntax of the corresponding packet type, as shown in Tables 12.2-A through 12.2-G. The bit numbers shown in Tables 12.2-A through 12.2-G are exemplary and may have other values. This is not a limitation of the present invention.

[0283] In the case where the data packet type is the first type including haptic experience metadata, the syntax of the payload of the data packet shown in Table 12.2-A will be used during the execution of the syntax in Table 12.2.

[0284] Table 12.2-A

[0285] When the data packet type is the first type containing tactile experience metadata, the payload of the data packet includes one or more of the following fields shown in Table 12.2-A. Each field in Table 12.2-A can be represented by a binary value. Specifically, the specific indication information of each field is as follows:

[0286] Version is used to indicate the standard version information that the tactile experience complies with; the binary value used to represent Version can occupy 8 bits.

[0287] dateLength is a string indicating the length of the creation date of the haptic experience. The binary value representing dateLength can occupy 8 bits.

[0288] Date is used to indicate the creation date of the tactile experience. The binary value used to represent DateDate can occupy dateLength×8 bits.

[0289] descriptionLength is used to indicate the length of the string describing the tactile experience; the binary value used to represent DateDate can occupy 8 bits.

[0290] Description is used to indicate the tactile experience description; the binary value used to represent Description can occupy descriptionLength×8 bits.

[0291] Timescale is used to indicate the time scale of the tactile experience. The value of this field represents the number of clock units contained in one second. The binary value used to represent Timescale can occupy 16 bits.

[0292] patternCount is used to indicate the number of patterns included in the tactile experience; the binary value used to represent patternCount can occupy 8 bits.

[0293] deviceCount is used to indicate the number of devices corresponding to the tactile experience; the binary value used to represent deviceCount can occupy 8 bits.

[0294] For the syntax of readDevice() in Table 12.2-A, please refer to Table 12.2-A1.

[0295] Table 12.2-A1

[0296] The payload of the above data packet also includes: one or more of the following fields as shown in Table 12.2-A1 regarding the jth device. Among them, each field in Table 12.2-A1 can be represented by a binary value. Specifically, the specific indication information of each field is as follows:

[0297] deviceId is used to indicate the identifier of the j-th device; the binary value used to represent deviceId can occupy 8 bits.

[0298] nameLength is used to indicate the length of the character string of the j-th device name; the binary value used to represent nameLength can occupy 8 bits.

[0299] deviceName is used to indicate the jth human-readable device name. The binary value used to represent deviceName can occupy nameLength × 8 bits.

[0300] bodyPartFlag is an indicator of whether the device corresponds to a body part; the binary value used to represent bodyPartFlag may occupy 1 bit; for example, when the value is 1, it indicates the body part information corresponding to the device; when the value is 0, it does not indicate the body part information corresponding to the device.

[0301] bodyParts is used to indicate the body part corresponding to the device; the binary value used to represent bodyParts can occupy 8 bits.

[0302] In the case where the data packet type is the second type including tactile mode metadata, the syntax of the payload of the data packet shown in Table 12.2-B will be used during the execution of the syntax in Table 12.2.

[0303] Table 12.2-B

[0304] In the case where the data packet type is the second type including tactile mode metadata, the payload of the data packet includes one or more of the following fields shown in Table 12.2-B. Each field in Table 12.2-B can be represented by a binary value. Specifically, the specific indication information of each field is as follows:

[0305] Id is used to indicate the identifier of the tactile mode; the binary value used to represent Id may occupy 8 bits.

[0306] patternType is used to indicate the type of tactile pattern. The binary value used to represent patternType can occupy 8 bits. For example, if the data pattern type is acceleration, it can be represented as "0000 0010". For example, the meaning of the binary value of this field can be shown in Table 12.2-B1.

[0307] Table 12.2-B1

[0308] semanticisFlag is an indicator for indicating whether the tactile event corresponding to the tactile mode has semantics; the binary value used to indicate semanticisFlag may occupy 1 bit, for example, when the value is 1, it indicates that the tactile event corresponding to the tactile mode has semantics, and when the value is 0, it indicates that the tactile event corresponding to the tactile mode does not have semantics.

[0309] sementicsType is an indicator used to indicate the specifications followed by the semantics of tactile events. The binary value used to indicate sementicsType can occupy 8 bits. For example, a value of 0 indicates that the semantics of the tactile event follow the specifications defined in this standard; a value of 1 indicates that the semantics of the tactile event follow the specifications of an external standard; and a value of 2 indicates that the semantics of the tactile event are user-defined.

[0310] referenceDeviceCount is used to indicate the number of devices corresponding to the tactile mode; the binary value used to indicate referenceDeviceCount may occupy 8 bits.

[0311] referenceDeviceId is used to indicate the identifier of the device corresponding to the tactile mode; the binary value used to represent referenceDeviceId may occupy 8 bits.

[0312] channelCount is used to indicate the number of tactile channels included in the tactile pattern; the binary value used to represent channelCount may occupy 16 bits.

[0313] In the case where the data packet type is the third type containing tactile channel metadata, the syntax of the payload of the data packet shown in Table 12.2-C will be used during the execution of the syntax in Table 12.2.

[0314] Table 12.2-C

[0315] When the data packet type is the third type including tactile channel metadata, the payload of the data packet includes one or more of the following fields shown in Table 12.2-C. Each field in Table 12.2-C can be represented by a binary value. Specifically, the specific indication information of each field is as follows:

[0316] Id is used to indicate the identifier of the tactile channel; the binary value used to represent Id may occupy 16 bits.

[0317] perceptionId is used to indicate the identifier of the tactile mode corresponding to the tactile channel; the binary value used to represent perceptionId can occupy 8 bits.

[0318] descriptionLength is used to indicate the length of the character string of the tactile channel description; the binary value used to represent descriptionLength can occupy 8 bits.

[0319] Description is used to indicate the tactile channel description; the binary value used to represent Description may occupy 8×descriptionLength bits.

[0320] Gain indicates the gain of the tactile channel. The binary value representing Gain occupies 32 bits, with units ranging from 2 to 31. This gain is applied to all events in the current channel, restoring the normalized signal value to the original value.

[0321] referenceDeviceCount is used to indicate the number of devices corresponding to the tactile channel; the binary value used to represent referenceDeviceCount may occupy 8 bits.

[0322] referenceDeviceId is used to indicate the identifier of the device corresponding to the tactile channel; the binary value used to represent referenceDeviceId can occupy 8 bits.

[0323] eventsCount is used to indicate the number of tactile events contained in the tactile channel; the binary value used to represent eventsCount can occupy 16 bits.

[0324] In the case where the data packet type is the seventh type containing a haptic signal, the syntax of the payload of the data packet shown in Table 12.2-D will be used during the execution of the syntax in Table 12.2.

[0325] Table 12.2-D

[0326] In the case where the data packet type is the seventh type including a tactile signal, the payload of the data packet includes one or more of the following fields shown in Table 12.2-D. Each field in Table 12.2-D may be represented by a binary value. Specifically, the specific indication information of each field is as follows:

[0327] packetDependency is an indicator of whether the current data packet depends on other data packets of the same type during encoding. The binary value used to represent packetDependency can occupy 1 bit. For example, when the value of this field is 0, it means that the current data packet does not depend on other data packets of the same type during encoding. When the value of this field is 1, it means that the current data packet depends on other data packets of the same type during encoding.

[0328] perceptionId is used to indicate the identifier of the tactile mode corresponding to the corresponding tactile event in the data packet; the binary value used to represent perceptionId can occupy 8 bits.

[0329] channelId is used to indicate the identifier of the tactile channel corresponding to the corresponding tactile event in the data packet; the binary value used to represent channelId may occupy 8 bits.

[0330] eventsCount is used to indicate the number of tactile events corresponding to the data packet; the binary value used to represent eventsCount may occupy 16 bits.

[0331] For the syntax of readEvent() in Table 12.2-D, please refer to Table 12.2-D1.

[0332] Table 12.2-D1

[0333] The payload of the above data packet also includes: one or more of the following fields as shown in Table 12.2-D1 regarding the mth tactile event. Among them, each field in Table 12.2-D1 can be represented by a binary value. Specifically, the specific indication information of each field is as follows:

[0334] eventId is used to indicate the identifier of a tactile event; a binary value used to represent eventId may occupy 16 bits.

[0335] eventType is used to indicate the type of tactile event; the binary value used to represent eventType can occupy 4 bits. For example, a value of 0 indicates that the event is an instantaneous event, a value of 1 indicates that the event is a continuous event, and a value of 2 indicates that the event is a reference event, etc.

[0336] eventSementicsFlag is an indicator value indicating whether the semantics of a tactile event is indicated; the binary value used to indicate eventSementicsFlag may occupy 1 bit, for example, when the value is 0, the semantics of a tactile event is not indicated, and when the value is 1, the semantics of a tactile event is indicated.

[0337] semanticKeywords represents the semantic information of the tactile event; the binary value used to represent semanticKeywords can occupy 12 bits. The semantics of the tactile event are event auxiliary information keywords that describe the content creator's intention.

[0338] referEventId is used to indicate the knowledge tactile event identifier corresponding to the reference event; the binary value used to represent referEventId can occupy 16 bits.

[0339] relativePosition is used to indicate the time or space offset of a tactile event; the binary value used to represent relativePosition may occupy 32 bits.

[0340] Duration is used to indicate the duration of a tactile event. This attribute is present when the event type is a continuous event. The binary value used to represent Duration can occupy 32 bits.

[0341] Amplitude is used to indicate the maximum amplitude value of the tactile event signal; the binary value used to represent Amplitude can occupy 8 bits.

[0342] baseFrequency is used to indicate the base frequency of the signal of the tactile event; the binary value used to represent baseFrequency may occupy 16 bits.

[0343] numComponents is used to indicate the number of tactile event components. The tactile event components may be time domain components or frequency domain components of the tactile event. The binary value representing numComponents may occupy 16 bits.

[0344] For the syntax of readEventComponent() in Table 12.2-D1, please refer to Table 12.2-D2.

[0345] Table 12.2-D2

[0346] The payload of the above data packet also includes: one or more of the following fields as shown in Table 12.2-D2 regarding the kth tactile event component. Among them, each field in Table 12.2-D2 can be represented by a binary value. Specifically, the specific indication information of each field is as follows:

[0347] refer_event_id is used to indicate the identifier of the tactile event to which the tactile event component belongs; the binary value used to represent refer_event_id may occupy 16 bits.

[0348] position_flag is an indicator used to indicate whether the tactile event component contains a time-space domain offset; the binary value used to indicate position_flag can occupy 1 bit. For example, when the position_flag value is 1, it indicates that the tactile event component contains a time-space domain offset, and when the position_flag value is 0, it indicates that the tactile event component does not contain a time-space domain offset.

[0349] Amplitude_flag is an indicator used to indicate whether the tactile event component contains an amplitude value offset; the binary value used to indicate amplitude_flag can occupy 1 bit. For example, when the value of amplitude_flag is 1, it indicates that the tactile event component contains an amplitude value offset, and when the value is 0, it indicates that the tactile event component does not contain an amplitude value offset.

[0350] frequency_flag is an indicator for indicating whether the tactile event component includes a frequency offset; the binary value used to indicate frequency_flag may occupy 1 bit, for example, when the value of frequency_flag is 1, it indicates that the tactile event component includes a frequency offset, and when the value is 0, it indicates that the tactile event component does not include a frequency offset.

[0351] relative_position is used to represent the time or space offset of the tactile event component relative to the tactile event; the binary value used to represent relative_position may occupy 16 bits.

[0352] relative_amplitude is used to represent the amplitude ratio of the tactile event component to the tactile event, with a value range of 0 to 215 in units of 2-15; the binary value used to represent relative_amplitude can occupy 16 bits.

[0353] relative_frequency is used to represent the frequency offset of the tactile event component relative to the tactile event; the binary value used to represent relative_frequency may occupy 16 bits.

[0354] In the case where the data packet type is the fourth type containing a knowledge tactile event, the syntax of the payload of the data packet shown in Table 12.2-E will be used during execution of the syntax in Table 12.2.

[0355] Table 12.2-E

[0356] When the data packet type is the fourth type including knowledge tactile events, the payload of the data packet includes one or more of the following fields shown in Table 12.2-E. Each field in Table 12.2-E can be represented by a binary value. Specifically, the specific indication information of each field is as follows:

[0357] patternId is used to represent the identifier of the tactile pattern corresponding to the knowledge tactile event; the binary value used to represent patternId can occupy 8 bits.

[0358] eventsCount is used to represent the number of knowledge tactile events corresponding to the data packet; the binary value used to represent eventsCount can occupy 16 bits.

[0359] For the syntax of readLibraryEvent() in Table 12.2-E, please refer to Table 12.2-E1.

[0360] Table 12.2-E1

[0361] The payload of the above data packet also includes: one or more of the following fields as shown in Table 12.2-E1 regarding the hth knowledge tactile event. Among them, each field in Table 12.2-E1 can be represented by a binary value. Specifically, the specific indication information of each field is as follows:

[0362] eventId is used to represent the identifier of the knowledge tactile event; the binary value used to represent eventId can occupy 16 bits.

[0363] eventType is used to indicate the type of the knowledge tactile event. The binary value used to indicate eventType can occupy 4 bits. For example, a value of 0 indicates that the event is an instantaneous event, and a value of 1 indicates that the event is a continuous event.

[0364] eventSementicsFlag is an indicator for indicating whether the semantics of the knowledge tactile event is indicated; the binary value used to indicate eventSementicsFlag may occupy 1 bit, for example, when the value is 0, the semantics of the knowledge tactile event is not indicated, and when the value is 1, the semantics of the knowledge tactile event is indicated.

[0365] SemanticKeywords is used to represent the semantic information of an event; the binary value used to represent semanticKeywords can occupy 12 bits. The semantics of a knowledge tactile event is the event auxiliary information keyword that describes the content creator's intention.

[0366] relativePosition is used to represent the time or space offset of the knowledge tactile event; the binary value used to represent relativePosition can occupy 32 bits.

[0367] Duration is used to indicate the duration of a knowledge tactile event. This attribute exists when the event type is a continuous event. The binary value used to indicate Duration can occupy 32 bits.

[0368] Amplitude is used to represent the maximum amplitude value of the signal of the knowledge tactile event; the binary value used to represent Amplitude can occupy 8 bits.

[0369] baseFrequency is used to represent the signal base frequency of the knowledge tactile event; the binary value used to represent baseFrequency can occupy 16 bits.

[0370] numComponents is used to represent the number of knowledge tactile event components. The knowledge tactile event components may be time domain components or frequency domain components of the knowledge tactile event. The binary value used to represent numComponents may occupy 16 bits.

[0371] For the syntax of readEventComponent() in Table 12.2-E1, please refer to Table 12.2-D2. We will not go into details here.

[0372] In the case where the data packet type is the sixth type containing silent data, the syntax of the payload of the data packet shown in Table 12.2-F will be used during the execution of the syntax in Table 12.2.

[0373] Table 12.2-F

[0374] When the data packet type is the sixth type including silent data, the payload of the data packet includes the fields shown in Table 12.2-F. The fields in Table 12.2-F can be represented by binary values. Specifically, the specific indication information of the field is as follows:

[0375] silentDuration is used to indicate the length of the silent period, in units of time ticks. The binary value used to indicate silentDuration may occupy 16 bits.

[0376] When the data packet type is the fifth type including a cyclic check code, the syntax execution process of Table 12.2 will turn to the syntax of the payload of the data packet shown in Table 12.2-G.

[0377] Table 12.2-G

[0378] When the data packet type is the fifth type including a cyclic check code, the payload of the data packet includes one or more of the following fields shown in Table 12.2-G. Each field in Table 12.2-G can be represented by a binary value. Specifically, the specific indication information of each field is as follows:

[0379] CRCType is used to indicate the type of cyclic check code. The binary value used to indicate CRCTypey can occupy 8 bits. For example, a value of 0 indicates that the cyclic check code is a 16-bit check code, and the data packet protected by the cyclic check code is the data packet following the cyclic check code data packet. A value of 1 indicates that the cyclic check code is a 32-bit check code, and the data packet protected by the cyclic check code is the data packet following the cyclic check code data packet. A value of 2 indicates that the cyclic check code is a 16-bit check code, and the data packet range protected by the cyclic check code is protectedPacketsCount data packets following the cyclic check code data packet. A value of 3 indicates that the cyclic check code is a 32-bit check code, and the data packet range protected by the cyclic check code is protectedPacketsCount data packets following the cyclic check code data packet.

[0380] CRC16Value is used to represent a 16-bit checksum. The binary value used to represent CRC16Value can occupy 16 bits.

[0381] CRC32Value is used to represent a 32-bit checksum. The binary value used to represent CRC32Value can occupy 32 bits.

[0382] protectedPacketsCount is used to represent the number of packets protected by the cyclic check code; the binary value used to represent protectedPacketsCount can occupy 8 bits.

[0383] Based on the AVS tactile information representation data structure 400 shown in Figure 4 and the field extension of the tactile media bit stream provided in the embodiments of the present application (such as the embodiments provided in Tables 7, 8, 11-12.2 above), it is possible to encode the exchange format of the target tactile information, and then obtain a binary bit stream corresponding to the above target tactile information.

[0384] Continuing to refer to FIG. 7 , in S720 , the bit stream is encapsulated to obtain a media file containing the tactile information.

[0385] The server side (ie, the packaging device) may package the bit stream, for example, by packaging the bit stream based on ISOBMFF to obtain a media file containing the tactile information.

[0386] In the solution provided by the embodiment shown in Figure 7, the exchange format of the target tactile information is encoded to obtain a binary bit stream corresponding to the target tactile information. The above-mentioned binary bit stream is further encapsulated to obtain a media file containing tactile information. The embodiment of the present application provides a solution for encoding the exchange format of tactile information into a binary bit stream. Specifically, the data is organized and divided in the form of data units and data packets in the above-mentioned binary bit stream, and a data unit contains one or more data packets. The solution provided by the embodiment of the present application is conducive to saving the space occupied by tactile media content, and can reduce bandwidth occupancy during the transmission of tactile media content. It also supports random access to tactile signals during transmission.

[0387] The above describes in detail an embodiment of a method for processing tactile information of the present application in conjunction with FIG. 1 to FIG. 7 . The following describes in detail an embodiment of a device of the present application in conjunction with FIG. 8 and FIG. 9 .

[0388] FIG8 is a schematic diagram of the structure of a tactile information processing device 800 provided in an embodiment of the present application. The tactile information processing device 800 is configured in an electronic device capable of decoding a tactile media bitstream, such as a decapsulation device. The decoding and encapsulation device may be the terminal 220 in FIG2 . As shown in FIG8 , the tactile information processing device 800 includes: a decapsulation module 810 and a decoding module 820;

[0389] The above-mentioned decapsulation module 810 is used to decapsulate the media file containing the target tactile information to obtain a binary bit stream corresponding to the target tactile information, and the bit stream includes: one or more data units about the target tactile information, and the data unit includes one or more data packets; and the above-mentioned decoding module 820 is used to decode the binary bit stream corresponding to the target tactile information to obtain the exchange format of the target tactile information, wherein the exchange format of the target tactile information is used to render and present the target tactile information.

[0390] In some embodiments, based on the above scheme, the grammatical structure of the data unit includes: the type of the data unit, different types correspond to different binary values; wherein, the type of the data unit includes: initialization type, silence type, time domain type and space domain type.

[0391] In some embodiments, based on the above scheme, when the type of the data unit is a time domain type or a spatial domain type, the grammatical structure of the data unit also includes: a first offset timestamp, different timestamps correspond to different binary values; wherein, the first offset timestamp represents the minimum or maximum value among the timestamps of all data packets contained in the data unit, and the timestamp of the i-th data unit is less than the timestamp of the i+1-th data unit, and the value of i is a positive integer.

[0392] In some embodiments, based on the above scheme, when the type of the data unit is a silence type, the grammatical structure of the data unit also includes: a first offset timestamp, different timestamps correspond to different binary values; wherein, the first offset timestamp is greater than the maximum value of the timestamps of all data packets contained in the target data unit, and the target data unit is a time domain type data unit or a spatial domain type data unit before the silence type data unit.

[0393] In some embodiments, based on the above scheme, the grammatical structure of the data unit also includes: a decoding dependency indicator, different decoding dependency indicators correspond to different binary values; wherein, when the decoding dependency indicator of the current data unit is the first target value, the current data unit does not depend on other data units of the same type during decoding; when the decoding dependency indicator of the current data unit is not the above-mentioned first target value, the current data unit depends on other data units of the same type during decoding.

[0394] In some embodiments, based on the above scheme, if the decoding dependency indicator contained in the next time domain type or spatial domain type data unit of the initialization type data unit is a first target value, then the initialization type data unit is a decoding initialization unit.

[0395] In some embodiments, based on the above scheme, the grammatical structure of the data unit also includes: a level indicator and a level identifier, different level indicators correspond to different binary values, and different level identifiers correspond to different binary values; wherein, if the level indicator is the second target value, the priority of the data unit is determined by the value of the level identifier.

[0396] In some embodiments, based on the above solution, the grammatical structure of the data packet includes: the type of the data packet, where different types correspond to different binary values;

[0397] The types of the data packets include:

[0398] The first type contains haptic experience metadata,

[0399] The second type contains tactile mode metadata,

[0400] A third type containing haptic channel metadata,

[0401] The fourth type includes knowledge tactile events,

[0402] The fifth type includes cyclic check codes,

[0403] The sixth type contains silent information and

[0404] The seventh type includes tactile signals.

[0405] In some embodiments, based on the above scheme, when the type of the data packet is the seventh type, the grammatical structure of the data packet also includes: a second offset timestamp, different timestamps correspond to different binary values; wherein, the second offset timestamp represents the offset relative to the first timestamp of the data unit where the data packet is located, and the second offset timestamp represents the minimum or maximum value among the timestamps of all tactile events contained in the data packet.

[0406] In some embodiments, based on the above solution, the grammatical structure of the data packet further includes: the number of bytes of the payload, where different numbers of bytes correspond to different binary values; wherein,

[0407] If the type of the data packet is the first type, the payload of the data packet includes one or more of the following information represented by binary values: the standard version followed by the tactile experience, the string length of the creation date of the tactile experience, the creation date of the tactile experience, the string length of the tactile experience description, the tactile experience description, the time scale of the tactile experience, the number of modes included in the tactile experience, and the number of devices corresponding to the tactile experience; and, if the type of the data packet is the first type, the payload of the data packet also includes one or more of the following information about the j-th device: a device identifier, a string length of the device name, a human-readable device name, and an indicator of whether the device corresponds to a body part, where the value of j is a positive integer not greater than the number of devices corresponding to the tactile experience;

[0408] If the type of the data packet is the second type, the payload of the data packet includes one or more of the following information represented by binary values: an identifier of a tactile mode, a type of the tactile mode, an indicator of whether a tactile event corresponding to the tactile mode has semantics, an indicator of a specification followed by the semantics of the tactile event, the number of devices corresponding to the tactile mode, an identifier of a device corresponding to the tactile mode, and the number of tactile channels included in the tactile mode;

[0409] If the type of the data packet is the third type, the payload of the data packet includes one or more of the following information represented by binary values: a tactile channel identifier, a tactile mode identifier corresponding to the tactile channel, a character string length of the tactile channel description, a tactile channel description, a gain of the tactile channel, the number of devices corresponding to the tactile channel, a device identifier of each device corresponding to the tactile channel, and the number of tactile events contained in the tactile channel;

[0410] If the type of the data packet is the seventh type, the load of the data packet includes one or more of the following information represented by binary values: an indicator of whether the current data packet depends on other data packets of the same type when decoding, an identifier of the tactile mode corresponding to the tactile event corresponding to the current data packet, an identifier of the tactile channel corresponding to the tactile event in the current data packet, and the number of tactile events in the data packet; and, if the type of the data packet is the seventh type, the load of the data packet also includes one or more of the following information about the mth tactile event: a tactile event identifier, a type of tactile event, an indicator of whether to indicate the semantics of the tactile event, semantic information of the tactile event, a knowledge tactile event identifier corresponding to a reference event, a time or space offset of the tactile event, a duration of the tactile event, the maximum signal amplitude value of the event, the signal reference frequency of the tactile event, and the number of tactile event components, where m is a positive integer not greater than the number of tactile events in the data packet; and, if the type of the data packet is the seventh type, the payload of the data packet further includes one or more of the following information about the kth tactile event component: an identifier of the tactile event to which the tactile event component belongs, an indicator of whether the tactile event component contains a spatiotemporal offset, an indicator of whether the tactile event component contains an amplitude offset, an indicator of whether the tactile event component contains a frequency offset, a time or space offset of the tactile event component relative to the tactile event, and an amplitude ratio of the tactile event component relative to the tactile event, where k is a positive integer not greater than the number of tactile event components in the tactile event;

[0411] If the type of the data packet is the fourth type, the load of the data packet includes one or more of the following information represented by binary values: an identifier of the tactile pattern corresponding to the knowledge tactile event, and the number of knowledge tactile events corresponding to the data packet; and if the type of the data packet is the fourth type, the load of the data packet also includes one or more of the following information about the hth knowledge tactile event: an identifier of the knowledge tactile event, a type of the knowledge tactile event, an indicator of whether the semantics of the knowledge tactile event is indicated, semantic information of the knowledge tactile event, a time or space offset of the knowledge tactile event, a duration of the knowledge tactile event, a maximum signal amplitude value of the knowledge tactile event, a signal reference frequency of the knowledge tactile event, and tactile The number of event components, h is a positive integer not greater than the number of the knowledge tactile events; and if the type of the data packet is the fourth type, the payload of the data packet also includes one or more of the following information about the kth tactile event component: an identifier of the knowledge tactile event to which the tactile event component belongs, an indicator of whether the tactile event component contains a time-space domain offset, an indicator of whether the tactile event component contains an amplitude value offset, an indicator of whether the tactile event component contains a frequency offset, a time or space offset of the tactile event component relative to the knowledge tactile event, and an amplitude value ratio of the tactile event component relative to the knowledge tactile event, and k is a positive integer not greater than the number of tactile event components in the knowledge tactile event;

[0412] If the type of the data packet is the sixth type, the payload of the data packet includes the following information represented by a binary value: the length of the silence period;

[0413] If the type of the data packet is the fifth type, the payload of the data packet includes one or more of the following information represented by binary values: the type of cyclic check code, different types of cyclic check codes, and the number of data packets protected by the cyclic check code.

[0414] It should be understood that the device embodiment and the embodiment of the method for processing tactile information can correspond to each other, and similar descriptions can refer to the method embodiment. To avoid repetition, no further details are given here. Specifically, the tactile information processing device shown in FIG8 can execute the embodiment of the method for processing tactile information shown in FIG5, and the aforementioned and other operations and / or functions of each module in the tactile information processing device are respectively for implementing the embodiment of the method for processing tactile information shown in FIG5. For the sake of brevity, no further details are given here.

[0415] FIG9 is a schematic diagram of a tactile information processing device 900 according to an embodiment of the present application. The tactile information processing device 900 is configured in an electronic device capable of encoding and exchanging tactile media bitstreams, such as the server 210 in FIG2 . As shown in FIG9 , the tactile information processing device 900 includes: an encoding module 910 and an encapsulation module 920;

[0416] Among them, the above-mentioned encoding module 910 is used to: encode the exchange format of the target tactile information to obtain a binary bit stream corresponding to the target tactile information, and the bit stream includes: one or more data units about the target tactile information, and the data unit includes one or more data packets; and the above-mentioned encapsulation module 920 is used to: encapsulate the bit stream to obtain a media file containing the tactile information.

[0417] In an exemplary embodiment, based on the above scheme, the grammatical structure of the data unit includes: the type of the data unit, different types correspond to different binary values; wherein the type of the data unit includes: initialization type, silence type, time domain type and space domain type.

[0418] In an exemplary embodiment, based on the above scheme,

[0419] In the case where the type of the data unit is a time domain type or a spatial domain type, the syntax structure of the data unit further includes: a first offset timestamp, where different timestamps correspond to different binary values; wherein the first offset timestamp represents the minimum or maximum value among the timestamps of all data packets contained in the data unit, and the timestamp of the i-th data unit is less than the timestamp of the i+1-th data unit, and the value of i is a positive integer;

[0420] In the case where the type of the data unit is a silence type, the syntax structure of the data unit further includes: a first offset timestamp, where different timestamps correspond to different binary values; wherein the first offset timestamp is greater than the maximum value of the timestamps of all data packets included in the target data unit, and the target data unit is a data unit of a time domain type or a data unit of a space domain type preceding the data unit of the silence type;

[0421] The grammatical structure of the data unit further includes: a coding dependency indicator, different coding dependency indicators correspond to different binary values; when the coding dependency indicator of the current data unit is a first target value, the current data unit does not depend on other data units of the same type during encoding; when the coding dependency indicator of the current data unit is not the above-mentioned first target value, the current data unit depends on other data units of the same type during encoding; if the coding dependency indicator included in the next time domain type or spatial domain type data unit of the initialization type data unit is the first target value, then the initialization type data unit is a coding initial unit;

[0422] The grammatical structure of the data unit also includes: a level indicator and a level identifier, different level indicators correspond to different binary values, and different level identifiers correspond to different binary values; wherein, if the level indicator is the second target value, the priority of the data unit is determined by the value of the level identifier.

[0423] In an exemplary embodiment, based on the above solution, the grammatical structure of the data packet includes: the type of the data packet, where different types correspond to different binary values;

[0424] The types of the data packets include:

[0425] The first type contains haptic experience metadata,

[0426] The second type contains tactile mode metadata,

[0427] A third type containing haptic channel metadata,

[0428] The fourth type includes knowledge tactile events,

[0429] The fifth type includes cyclic check codes,

[0430] The sixth type contains silent information and

[0431] The seventh type includes tactile signals.

[0432] In an exemplary embodiment, based on the above scheme, when the type of the data packet is the seventh type, the grammatical structure of the data packet also includes: a second offset timestamp, different timestamps correspond to different binary values; wherein, the second offset timestamp represents the offset relative to the first timestamp of the data unit where the data packet is located, and the second offset timestamp represents the minimum or maximum value among the timestamps of all tactile events contained in the data packet.

[0433] In an exemplary embodiment, based on the above solution, the grammatical structure of the data packet further includes: the number of bytes of the payload, where different numbers of bytes correspond to different binary values; wherein,

[0434] If the type of the data packet is the first type, the payload of the data packet includes one or more of the following information represented by binary values: the standard version followed by the tactile experience, the string length of the creation date of the tactile experience, the creation date of the tactile experience, the string length of the tactile experience description, the tactile experience description, the time scale of the tactile experience, the number of modes included in the tactile experience, and the number of devices corresponding to the tactile experience; and, if the type of the data packet is the first type, the payload of the data packet also includes one or more of the following information about the j-th device: a device identifier, a string length of the device name, a human-readable device name, and an indicator of whether the device corresponds to a body part, where the value of j is a positive integer not greater than the number of devices corresponding to the tactile experience;

[0435] If the type of the data packet is the second type, the payload of the data packet includes one or more of the following information represented by binary values: an identifier of a tactile mode, a type of the tactile mode, an indicator of whether a tactile event corresponding to the tactile mode has semantics, an indicator of a specification followed by the semantics of the tactile event, the number of devices corresponding to the tactile mode, an identifier of a device corresponding to the tactile mode, and the number of tactile channels included in the tactile mode;

[0436] If the type of the data packet is the third type, the payload of the data packet includes one or more of the following information represented by binary values: a tactile channel identifier, a tactile mode identifier corresponding to the tactile channel, a character string length of the tactile channel description, a tactile channel description, a gain of the tactile channel, the number of devices corresponding to the tactile channel, a device identifier of each device corresponding to the tactile channel, and the number of tactile events contained in the tactile channel;

[0437] If the type of the data packet is the seventh type, the load of the data packet includes one or more of the following information represented by binary values: an indicator of whether the current data packet depends on other data packets of the same type when encoding, an identifier of the tactile mode corresponding to the tactile event in the current data packet, an identifier of the tactile channel corresponding to the tactile event in the current data packet, and the number of tactile events in the data packet; and, if the type of the data packet is the seventh type, the load of the data packet also includes one or more of the following information about the mth tactile event: a tactile event identifier, a type of tactile event, an indicator of whether to indicate the semantics of the tactile event, semantic information of the tactile event, a knowledge tactile event identifier corresponding to a reference event, a time or space offset of the tactile event, a duration of the tactile event, the maximum signal amplitude value of the event, the signal reference frequency of the tactile event, and the number of tactile event components, where m is a positive integer not greater than the number of tactile events in the data packet; and, if the type of the data packet is the seventh type, the payload of the data packet further includes one or more of the following information about the kth tactile event component: an identifier of the tactile event to which the tactile event component belongs, an indicator of whether the tactile event component contains a spatiotemporal offset, an indicator of whether the tactile event component contains an amplitude offset, an indicator of whether the tactile event component contains a frequency offset, a time or space offset of the tactile event component relative to the tactile event, and an amplitude ratio of the tactile event component relative to the tactile event, where k is a positive integer not greater than the number of tactile event components in the tactile event;

[0438] If the type of the data packet is the fourth type, the load of the data packet includes one or more of the following information represented by binary values: an identifier of the tactile pattern corresponding to the knowledge tactile event, and the number of knowledge tactile events corresponding to the data packet; and if the type of the data packet is the fourth type, the load of the data packet also includes one or more of the following information about the hth knowledge tactile event: an identifier of the knowledge tactile event, a type of the knowledge tactile event, an indicator of whether the semantics of the knowledge tactile event is indicated, semantic information of the knowledge tactile event, a time or space offset of the knowledge tactile event, a duration of the knowledge tactile event, a maximum signal amplitude value of the knowledge tactile event, a signal reference frequency of the knowledge tactile event, and tactile The number of event components, h is a positive integer not greater than the number of the knowledge tactile events; and if the type of the data packet is the fourth type, the payload of the data packet also includes one or more of the following information about the kth tactile event component: an identifier of the knowledge tactile event to which the tactile event component belongs, an indicator of whether the tactile event component contains a time-space domain offset, an indicator of whether the tactile event component contains an amplitude value offset, an indicator of whether the tactile event component contains a frequency offset, a time or space offset of the tactile event component relative to the knowledge tactile event, and an amplitude value ratio of the tactile event component relative to the knowledge tactile event, and k is a positive integer not greater than the number of tactile event components in the knowledge tactile event;

[0439] If the type of the data packet is the sixth type, the payload of the data packet includes the following information represented by a binary value: the length of the silence period;

[0440] If the type of the data packet is the fifth type, the payload of the data packet includes one or more of the following information represented by binary values: the type of cyclic check code, different types of cyclic check codes, and the number of data packets protected by the cyclic check code.

[0441] It should be understood that the device embodiment and the embodiment of the method for processing tactile information may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, no further details will be given here. Specifically, the tactile information processing device shown in FIG9 can execute the embodiment of the method for processing tactile information shown in FIG7, and the aforementioned and other operations and / or functions of each module in the tactile information processing device are respectively for implementing the embodiment of the method for processing tactile information shown in FIG7. For the sake of brevity, no further details will be given here.

[0442] The apparatus of the embodiment of the present application is described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, or can be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in conjunction with its hardware.

[0443] Figure 10 is a structural schematic diagram of an electronic device 1000 provided in an embodiment of the present application. The electronic device 1000 in Figure 10 can be used to execute the above-mentioned tactile information processing method provided in the embodiment of Figure 5, or the electronic device 1000 can be used to execute the above-mentioned tactile information processing method provided in the embodiment of Figure 7.

[0444] As shown in FIG10 , the electronic device 1000 may include:

[0445] The memory 1010 and the processor 1020 are configured to store a computer program 1030 and transmit the program code 1030 to the processor 1020. In other words, the processor 1020 can call and run the computer program 1030 from the memory 1010 to implement the tactile information processing method in the embodiment of the present application.

[0446] For example, the processor 1020 may be configured to execute the steps of the aforementioned method for processing tactile information according to the instructions in the computer program 1030 .

[0447] In some embodiments of the present application, the processor 1020 may include but is not limited to:

[0448] General-purpose processor, Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0449] In some embodiments of the present application, the memory 1010 includes but is not limited to:

[0450] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0451] In some embodiments of the present application, the computer program 1030 may be divided into one or more modules, which are stored in the memory 1010 and executed by the processor 1020 to implement the tactile information processing method of the present application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 1030 in the electronic device.

[0452] As shown in FIG10 , the electronic device 1000 may further include:

[0453] The transceiver 1040 may be connected to the processor 1020 or the memory 1010 .

[0454] The processor 1020 may control the transceiver 1040 to communicate with other devices. Specifically, the processor 1020 may send information or data to other devices or receive information or data sent by other devices. The transceiver 1040 may include a transmitter and a receiver. The transceiver 1040 may further include one or more antennas.

[0455] It should be understood that the various components in the electronic device 1000 are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0456] According to one aspect of the present application, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a computer, the computer is enabled to perform the method of the above-mentioned method embodiment. Alternatively, the present application also provides a computer program product containing instructions. When the computer is executed by the instructions, the computer is enabled to perform the method of the above-mentioned method embodiment.

[0457] According to another aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method of the above-described method embodiment.

[0458] In other words, when implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0459] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0460] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0461] Modules described as separate components may or may not be physically separate, and components displayed as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected based on actual needs to achieve the purpose of the present embodiment. For example, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module.

[0462] The above content is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for processing tactile information, executed by a processor, characterized in that: The method comprises: Decapsulating a media file containing target tactile information to obtain a binary bit stream corresponding to the target tactile information, wherein the bit stream includes: one or more data units regarding the target tactile information, wherein the data unit includes one or more data packets; The binary bit stream corresponding to the target tactile information is decoded to obtain an exchange format of the target tactile information, wherein the exchange format of the target tactile information is used to render and present the target tactile information.

2. The method according to claim 1, characterized in that The grammatical structure of the data unit includes: the type of the data unit, different types correspond to different binary values; The type of the data unit includes at least one of the following: an initialization type, a silence type, a time domain type, and a space domain type.

3. The method according to claim 2, characterized in that In the case where the type of the data unit is a time domain type or a space domain type, the syntax structure of the data unit further includes: a first offset timestamp, different timestamps corresponding to different binary values; The first offset timestamp represents the minimum or maximum value among the timestamps of all data packets contained in the data unit, and the timestamp of the i-th data unit is less than the timestamp of the i+1-th data unit, and the value of i is a positive integer.

4. The method according to claim 2 or 3, characterized in that: In the case where the type of the data unit is a silence type, the syntax structure of the data unit further includes: a first offset timestamp, different timestamps corresponding to different binary values; The first offset timestamp is greater than the maximum value of the timestamps of all data packets contained in the target data unit, and the target data unit is a time domain type data unit or a space domain type data unit preceding the silence type data unit.

5. The method according to any one of claims 2 to 4, characterized in that The syntax structure of the data unit further includes: a decoding dependency indicator, different decoding dependency indicators correspond to different binary values; Among them, when the value of the decoding dependency indicator of the current data unit is the first target value, it means that the current data unit does not depend on other data units of the same type during decoding; when the value of the decoding dependency indicator of the current data unit is not the first target value, it means that the current data unit depends on other data units of the same type during decoding.

6. The method according to any one of claims 2 to 5, characterized in that If the value of the decoding dependency indicator contained in the next time domain type or space domain type data unit of the initialization type data unit is the first target value, then the initialization type data unit is a decoding initialization unit.

7. The method according to any one of claims 2 to 6, characterized in that The grammatical structure of the data unit further includes: a level indicator and a level identifier, different level indicators correspond to different binary values, and different level identifiers correspond to different binary values; If the level indicator is the second target value, the priority of the data unit is determined by the value of the level indicator.

8. The method according to any one of claims 1 to 7, characterized in that The grammatical structure of the data packet includes: the type of the data packet, different types correspond to different binary values; The type of the data packet includes at least one of the following: The first type contains the haptic experience metadata, A second type containing haptic mode metadata, A third type containing haptic channel metadata, The fourth type includes knowledge tactile events, The fifth type includes cyclic checksums, A sixth type containing silent information, and A seventh type includes tactile signals.

9. The method according to any one of claims 1 to 8, characterized in that In the case where the type of the data packet is the seventh type, the syntax structure of the data packet further includes: a second offset timestamp, different timestamps corresponding to different binary values; The second offset timestamp represents an offset relative to the first timestamp of the data unit where the data packet is located, and the second offset timestamp represents the minimum value or the maximum value of the timestamps of all tactile events contained in the data packet. Maximum value.

10. The method according to any one of claims 1 to 9, characterized in that The grammatical structure of the data packet also includes: the number of bytes of the payload, and different numbers of bytes correspond to different binary values; wherein, If the type of the data packet is the first type, the payload of the data packet includes one or more of the following information represented by binary values: the standard version followed by the tactile experience, the string length of the creation date of the tactile experience, the creation date of the tactile experience, the string length of the tactile experience description, the tactile experience description, the time scale of the tactile experience, the number of modes included in the tactile experience, and the number of devices corresponding to the tactile experience; and, if the type of the data packet is the first type, the payload of the data packet also includes one or more of the following information about the j-th device: device identification, the string length of the device name, a human-readable device name, an indicator of whether the device corresponds to a body part, and the value of j is a positive integer not greater than the number of devices corresponding to the tactile experience.

11. The method according to any one of claims 1 to 10, characterized in that The grammatical structure of the data packet also includes: the number of bytes of the payload, and different numbers of bytes correspond to different binary values; wherein, If the type of the data packet is the second type, the payload of the data packet includes one or more of the following information represented by binary values: an identifier of the tactile mode, the type of the tactile mode, an indicator of whether the tactile event corresponding to the tactile mode has semantics, an indicator of the specification followed by the semantics of the tactile event, the number of devices corresponding to the tactile mode, the device identifier corresponding to the tactile mode, and the number of tactile channels included in the tactile mode.

12. The method according to any one of claims 1 to 11, characterized in that The grammatical structure of the data packet also includes: the number of bytes of the payload, and different numbers of bytes correspond to different binary values; wherein, If the type of the data packet is the third type, the payload of the data packet includes one or more of the following information represented by binary values: a tactile channel identifier, a tactile mode identifier corresponding to the tactile channel, a string length of a tactile channel description, a tactile channel description, a gain of a tactile channel, the number of devices corresponding to the tactile channel, a device identifier of each device corresponding to the tactile channel, and the number of tactile events contained in the tactile channel.

13. The method according to any one of claims 1 to 12, characterized in that The grammatical structure of the data packet also includes: the number of bytes of the payload, and different numbers of bytes correspond to different binary values; wherein, If the type of the data packet is the seventh type, the load of the data packet includes one or more of the following information represented by binary values: an indicator of whether the current data packet depends on other data packets of the same type when decoding, an identifier of the tactile mode corresponding to the tactile event corresponding to the current data packet, an identifier of the tactile channel corresponding to the tactile event in the current data packet, and the number of tactile events in the data packet; and, if the type of the data packet is the seventh type, the load of the data packet also includes one or more of the following information about the mth tactile event: a tactile event identifier, a type of tactile event, an indicator of whether to indicate the semantics of the tactile event, semantic information of the tactile event, a knowledge tactile event identifier corresponding to a reference event, a time or space offset of the tactile event, a duration of the tactile event, and a value of the tactile event. The maximum amplitude value of the signal of the tactile event component, the signal reference frequency of the tactile event, and the number of tactile event components, m is a positive integer not greater than the number of tactile events in the data packet; and if the type of the data packet is the seventh type, the payload of the data packet also includes one or more of the following information about the kth tactile event component: an identifier of the tactile event to which the tactile event component belongs, an indicator of whether the tactile event component contains a time-space domain offset, an indicator of whether the tactile event component contains an amplitude value offset, an indicator of whether the tactile event component contains a frequency offset, the time or space offset of the tactile event component relative to the tactile event, and the amplitude value ratio of the tactile event component to the tactile event, k is a positive integer not greater than the number of tactile event components in the tactile event.

14. The method according to any one of claims 1 to 13, characterized in that The grammatical structure of the data packet also includes: the number of bytes of the payload, and different numbers of bytes correspond to different binary values; wherein, If the type of the data packet is the fourth type, the load of the data packet includes one or more of the following information represented by binary values: an identifier of the tactile pattern corresponding to the knowledge tactile event, and the number of knowledge tactile events corresponding to the data packet; and if the type of the data packet is the fourth type, the load of the data packet also includes one or more of the following information about the hth knowledge tactile event: an identifier of the knowledge tactile event, a type of the knowledge tactile event, an indicator of whether the semantics of the knowledge tactile event is indicated, semantic information of the knowledge tactile event, a time or space offset of the knowledge tactile event, a duration of the knowledge tactile event, and a value of the knowledge tactile event. The maximum amplitude value of the signal of the kth tactile event, the signal reference frequency of the knowledge tactile event, and the number of tactile event components, where h is a positive integer not greater than the number of the knowledge tactile events; and, if the type of the data packet is the fourth type, the load of the data packet also includes one or more of the following information about the kth tactile event component: an identifier of the knowledge tactile event to which the tactile event component belongs, an indicator of whether the tactile event component contains a time-space domain offset, an indicator of whether the tactile event component contains an amplitude value offset, an indicator of whether the tactile event component contains a frequency offset, and a time or space information of the tactile event component relative to the knowledge tactile event. The offset between the haptic event components and the amplitude value ratio of the haptic event components to the knowledge haptic event, and the value of k is a positive integer not greater than the number of haptic event components in the knowledge haptic event.

15. The method according to any one of claims 1 to 14, characterized in that The grammatical structure of the data packet also includes: the number of bytes of the payload, and different numbers of bytes correspond to different binary values; wherein, If the type of the data packet is the sixth type, the payload of the data packet includes a length of the silence time represented by a binary value.

16. The method according to any one of claims 1 to 15, characterized in that The grammatical structure of the data packet also includes: the number of bytes of the payload, and different numbers of bytes correspond to different binary values; wherein, If the type of the data packet is the fifth type, the payload of the data packet includes one or more of the following information represented by binary values: the type of cyclic check code, different types of cyclic check codes, and the number of data packets protected by the cyclic check code.

17. A method for processing tactile information, executed by a processor, characterized in that: The method comprises: Encoding the exchange format of the target tactile information to obtain a binary bit stream corresponding to the target tactile information, wherein the bit stream includes: one or more data units regarding the target tactile information, wherein the data unit includes one or more data packets; The bit stream is encapsulated to obtain a media file containing the tactile information.

18. The method according to claim 17, characterized in that The grammatical structure of the data unit includes: the type of the data unit, different types correspond to different binary values; The type of the data unit includes at least one of the following: an initialization type, a silence type, a time domain type, and a space domain type.

19. The method according to claim 18, characterized in that In the case where the type of the data unit is a time domain type or a spatial domain type, the grammatical structure of the data unit further includes: a first offset timestamp, where different timestamps correspond to different binary values; wherein the first offset timestamp represents the minimum or maximum value of the timestamps of all data packets contained in the data unit, and the timestamp of the i-th data unit is less than the timestamp of the i+1-th data unit, and the value of i is a positive integer; In the case where the type of the data unit is a silence type, the syntax structure of the data unit further includes: a first offset timestamp, and different timestamps correspond to different binary values; wherein the first offset timestamp is greater than the maximum value of the timestamps of all data packets included in the target data unit, and the target data unit is a data unit of a time domain type or a data unit of a space domain type before the data unit of the silence type; The grammatical structure of the data unit also includes: a coding dependency indicator, different coding dependency indicators correspond to different binary values; when the value of the coding dependency indicator of the current data unit is the first target value, it indicates that the current data unit does not depend on other data units of the same type during encoding; when the value of the coding dependency indicator of the current data unit is not the first target value, it indicates that the current data unit depends on other data units of the same type during encoding; if the value of the coding dependency indicator contained in the next time domain type or spatial domain type data unit of the initialization type data unit is the first target value, it indicates that the initialization type data unit is an initial coding unit; The grammatical structure of the data unit also includes: a level indicator and a level identifier, different level indicators correspond to different binary values, and different level identifiers correspond to different binary values; wherein, if the level indicator is the second target value, the priority of the data unit is determined by the value of the level identifier.

20. The method according to any one of claims 17 to 19, characterized in that The grammatical structure of the data packet includes: the type of the data packet, different types correspond to different binary values; The type of the data packet includes at least one of the following: The first type contains the haptic experience metadata, A second type containing haptic mode metadata, A third type containing haptic channel metadata, The fourth type includes knowledge tactile events, The fifth type includes cyclic checksums, The sixth type contains silent information and A seventh type includes tactile signals.

21. The method according to any one of claims 17 to 20, characterized in that In the case where the type of the data packet is the seventh type, the syntax structure of the data packet further includes: a second offset timestamp, different timestamps corresponding to different binary values; The second offset timestamp represents an offset relative to the first timestamp of the data unit where the data packet is located, and the second offset timestamp represents a minimum value or a maximum value among the timestamps of all tactile events contained in the data packet.

22. The method according to any one of claims 17 to 21, characterized in that The grammatical structure of the data packet also includes: the number of bytes of the payload, and different numbers of bytes correspond to different binary values; wherein, If the type of the data packet is the first type, the payload of the data packet includes one or more of the following information represented by binary values: the standard version followed by the tactile experience, the string length of the creation date of the tactile experience, the creation date of the tactile experience, the string length of the tactile experience description, the tactile experience description, the time scale of the tactile experience, the number of modes included in the tactile experience, and the number of devices corresponding to the tactile experience; and, if the type of the data packet is the first type, the payload of the data packet also includes one or more of the following information about the j-th device: device identification, the string length of the device name, a human-readable device name, an indicator of whether the device corresponds to a body part, and the value of j is a positive integer not greater than the number of devices corresponding to the tactile experience.

23. The method according to any one of claims 17 to 22, characterized in that The grammatical structure of the data packet also includes: the number of bytes of the payload, and different numbers of bytes correspond to different binary values; wherein, If the type of the data packet is the second type, the payload of the data packet includes one or more of the following information represented by binary values: an identifier of the tactile mode, the type of the tactile mode, an indicator of whether the tactile event corresponding to the tactile mode has semantics, an indicator of the specification followed by the semantics of the tactile event, the number of devices corresponding to the tactile mode, the device identifier corresponding to the tactile mode, and the number of tactile channels included in the tactile mode.

24. The method according to any one of claims 17 to 23, characterized in that The grammatical structure of the data packet also includes: the number of bytes of the payload, and different numbers of bytes correspond to different binary values; wherein, If the type of the data packet is the third type, the payload of the data packet includes one or more of the following information represented by binary values: a tactile channel identifier, a tactile mode identifier corresponding to the tactile channel, a string length of a tactile channel description, a tactile channel description, a gain of a tactile channel, the number of devices corresponding to the tactile channel, a device identifier of each device corresponding to the tactile channel, and the number of tactile events contained in the tactile channel.

25. The method according to any one of claims 17 to 24, characterized in that The grammatical structure of the data packet also includes: the number of bytes of the payload, and different numbers of bytes correspond to different binary values; wherein, If the type of the data packet is the seventh type, the payload of the data packet includes one or more of the following information represented by binary values: an indicator of whether the current data packet depends on other data packets of the same type when encoding, an identifier of the tactile mode corresponding to the tactile event corresponding to the current data packet, an identifier of the tactile channel corresponding to the tactile event in the current data packet, and the number of tactile events in the data packet; and, if the type of the data packet is the seventh type, the payload of the data packet also includes one or more of the following information about the mth tactile event: a tactile event identifier, a type of tactile event, an indicator of whether the semantics of the tactile event is indicated, semantic information of the tactile event, a knowledge tactile event identifier corresponding to a reference event, a time or space offset of the tactile event, a duration of the tactile event, and a value of the tactile event. The maximum amplitude value of the signal of the tactile event component, the signal reference frequency of the tactile event, and the number of tactile event components, m is a positive integer not greater than the number of tactile events in the data packet; and if the type of the data packet is the seventh type, the payload of the data packet also includes one or more of the following information about the kth tactile event component: an identifier of the tactile event to which the tactile event component belongs, an indicator of whether the tactile event component contains a time-space domain offset, an indicator of whether the tactile event component contains an amplitude value offset, an indicator of whether the tactile event component contains a frequency offset, the time or space offset of the tactile event component relative to the tactile event, and the amplitude value ratio of the tactile event component to the tactile event, k is a positive integer not greater than the number of tactile event components in the tactile event.

26. The method according to any one of claims 17 to 25, characterized in that The grammatical structure of the data packet also includes: the number of bytes of the payload, and different numbers of bytes correspond to different binary values; wherein, If the type of the data packet is the fourth type, the load of the data packet includes one or more of the following information represented by binary values: an identifier of the tactile pattern corresponding to the knowledge tactile event, and the number of knowledge tactile events corresponding to the data packet; and, if the type of the data packet is the fourth type, the load of the data packet also includes one or more of the following information about the hth knowledge tactile event: an identifier of the knowledge tactile event, a type of the knowledge tactile event, an indicator of whether the semantics of the knowledge tactile event is indicated, semantic information of the knowledge tactile event, a time or space offset of the knowledge tactile event, a duration of the knowledge tactile event, a maximum signal amplitude value of the knowledge tactile event, a signal reference frequency of the knowledge tactile event, and the number of tactile event components, where h is a positive integer not greater than the number of the knowledge tactile events; and, if the type of the data packet is the fourth type, the load of the data packet also includes information about the kth tactile event. One or more of the following information of the component: an identifier of the knowledge tactile event to which the tactile event component belongs, an indicator of whether the tactile event component contains a time-space domain offset, an indicator of whether the tactile event component contains an amplitude value offset, an indicator of whether the tactile event component contains a frequency offset, the time or space offset of the tactile event component relative to the knowledge tactile event, and the amplitude value ratio of the tactile event component relative to the knowledge tactile event, where k is a positive integer not greater than the number of tactile event components in the knowledge tactile event.

27. The method according to any one of claims 17 to 26, characterized in that The grammatical structure of the data packet also includes: the number of bytes of the payload, and different numbers of bytes correspond to different binary values; wherein, If the type of the data packet is the sixth type, the payload of the data packet includes the following information represented by a binary value: the length of the silence time.

28. The method according to any one of claims 17 to 27, characterized in that The grammatical structure of the data packet also includes: the number of bytes of the payload, and different numbers of bytes correspond to different binary values; wherein, If the type of the data packet is the fifth type, the payload of the data packet includes one or more of the following information represented by binary values: the type of cyclic check code, different types of cyclic check codes, and the number of data packets protected by the cyclic check code.

29. An electronic device comprising a processor and a memory; The memory is used to store computer programs; The processor is used to execute the computer program to implement the method for processing tactile information as described in any one of claims 1 to 16 or 17 to 28.

30. A computer-readable storage medium, characterized in that: For storing computer programs; The computer program enables a computer to execute the method for processing tactile information as described in any one of claims 1 to 16 or 17 to 28.

31. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the method for processing tactile information as described in any one of claims 1 to 16 or 17 to 28.

Images