Tactile information processing method and device, electronic equipment and storage medium

Abstract

The invention provides a tactile information processing method and device, electronic equipment and a storage medium, and is applied to the technical field of multimedia information processing. The method comprises the steps that a media file containing target tactile information is decapsulated, a binary bit stream corresponding to the target tactile information is obtained, the bit stream comprises one or more data units related to the target tactile information, and each data unit comprises one or more data packets; and decoding the binary bit stream corresponding to the target tactile information to obtain an exchange format of the target tactile information, so that the target tactile information can be rendered and presented based on the exchange format of the target tactile information. According to the scheme provided by the embodiment of the invention, the occupied space of the tactile media content can be saved, and the bandwidth occupation can be reduced in the transmission process of the tactile media content.

Description

Technical Field

[0001] Embodiments of the present application relate to the technical field of multimedia information processing, and in particular, to a method for processing tactile information, a device for processing tactile information, an electronic device, and a computer-readable storage medium. Background Art

[0002] The presentation of immersive media content often accompanies the use of various wearable devices or interactive devices. In terms of the presentation method of immersive media, in addition to visual and auditory presentations, it also has a tactile presentation method. Tactile presentation can be achieved through a tactile presentation mechanism that combines hardware and software. The tactile presentation method provides an embedded body feeling. The user receives information through the body, thereby transmitting key information of the system being used to the user. For example, vibration is a type of tactile presentation, and the user is reminded of an incoming call or message by the vibration of the mobile phone.

[0003] In related technologies, tactile media content occupies a large amount of storage space and consumes a large amount of bandwidth during transmission. There is an urgent need in related technologies for a solution that can save occupied space and reduce bandwidth occupancy during the transmission of tactile media content. Summary of the Invention

[0004] The present application provides a method for processing tactile information, a device for processing tactile information, an electronic device, and a computer-readable storage medium, which can save the occupied space of tactile media content and reduce bandwidth occupancy during the transmission of tactile media content.

[0005] In a first aspect, the present application provides a method for processing tactile information, the method including: decompressing a media file including target tactile information to obtain a binary bitstream corresponding to the target tactile information, the bitstream including: one or more data units regarding the target tactile information, and the data unit including one or more data packets; and decoding the binary bitstream corresponding to the target tactile information to obtain an exchange format of the target tactile information, where the exchange format of the target tactile information is used for rendering and presenting the target tactile information.

[0006] In a second aspect, the present application provides a tactile information processing device, which includes a decompression module and a decoding module. The decompression module is configured to decompress a media file containing target tactile information to obtain a binary bitstream corresponding to the target tactile information. The bitstream includes one or more data units regarding the target tactile information, and each data unit includes one or more data packets. Further, the decoding module is configured to decode the binary bitstream corresponding to the target tactile information to obtain an exchange format of the target tactile information, where the exchange format of the target tactile information is used to render and present the target tactile information.

[0007] In a third aspect, the present application provides a method for processing tactile information, which includes: encoding an exchange format of target tactile information to obtain a binary bitstream corresponding to the target tactile information. The bitstream includes one or more data units regarding the target tactile information, and each data unit includes one or more data packets. Further, encapsulating the bitstream to obtain a media file containing the tactile information.

[0008] In a fourth aspect, the present application provides a tactile information processing device, which includes an encoding module and an encapsulation module. The encoding module is configured to encode an exchange format of target tactile information to obtain a binary bitstream corresponding to the target tactile information. The bitstream includes one or more data units regarding the target tactile information, and each data unit includes one or more data packets. Further, the encapsulation module is configured to encapsulate the bitstream to obtain a media file containing the tactile information.

[0009] In a fifth aspect, an electronic device is provided, which includes a processor and a memory. The memory is used to store a computer program, and the processor is configured to call and run the computer program stored in the memory to execute the method for processing tactile information provided in the first aspect or the second aspect above.

[0010] In a sixth aspect, a chip is provided, which is used to implement the method in any aspect or its various implementation manners in the first aspect above. Specifically, the chip includes a processor configured to call and run a computer program from a memory, so that a device installed with the chip executes the method for processing tactile information provided in the first aspect or the second aspect above.

[0011] In a seventh aspect, a computer-readable storage medium is provided, which is used to store a computer program, and the computer program causes a computer to execute the method for processing tactile information provided in the first aspect or the second aspect above.

[0012] In an eighth aspect, a computer program product is provided, including computer program instructions that cause a computer to execute the method for processing tactile information provided in the above first aspect or the above second aspect.

[0013] In a ninth aspect, a computer program is provided, which, when running on a computer, causes the computer to execute the method for processing tactile information provided in the above first aspect or the above second aspect.

[0014] In summary, in the solution provided by the embodiments of the present application, the exchange format of the target tactile information is encoded to obtain a binary bitstream corresponding to the target tactile information. Further, the above binary bitstream is encapsulated to obtain a media file containing the tactile information. The embodiments of the present application provide a solution for encoding the exchange format of tactile information into a binary bitstream. Specifically, data is organized and divided in the above binary bitstream in the form of data units and data packets, and one data unit contains one or more data packets. The solution provided by the embodiments of the present application is beneficial to saving the occupied space of tactile media content and reducing bandwidth occupancy during the transmission of tactile media content. Description of the Drawings

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0016] Figure 1 It is a schematic diagram of the system architecture for the representation and transmission of tactile information in the video coding standard AVS;

[0017] Figure 2 It is a schematic diagram of the application scenario for the processing of tactile information provided by the embodiments of the present application;

[0018] Figure 3 It is an information interaction schematic diagram of the method for processing tactile information provided by the embodiments of the present application;

[0019] Figure 4 It is a schematic diagram of the data structure for the representation of tactile information in the video coding standard AVS;

[0020] Figure 5 It is a schematic flowchart of the method for processing tactile information provided by the embodiments of the present application;

[0021] Figure 6 It is a schematic diagram of the relationship between data units and data packets provided by the embodiments of the present application;

[0022] Figure 7Schematic flowchart of the method for processing tactile information provided by an embodiment of the present application;

[0023] Figure 8 Schematic structural diagram of the device for processing tactile information provided by an embodiment of the present application;

[0024] Figure 9 Schematic structural diagram of the device for processing tactile information provided by an embodiment of the present application;

[0025] Figure 10 Schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0026] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In the embodiments of the present application, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, but B can also be determined according to A and / or other information. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or server including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices. In the description of the present application, unless otherwise specified, "a plurality of" means two or more than two.

[0028] In the embodiments of the present application, the term "module" or "unit" refers to a computer program with a predetermined function or a part of a computer program, which works together with other relevant parts to achieve a predetermined goal, and can be fully or partially implemented by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit including the function of the module or unit

[0029] Haptic information presentation can be classified into the following categories: vibrotactile, kinesthetic, and electrohaptic. The specific descriptions are as follows:

[0030] 1. Vibrotactile

[0031] Direct haptic presentation in the form of vibration can simulate vibrations of specific frequencies and intensities through the motor vibration of the terminal device. For example, in a shooting game, specific effects of shooting are simulated through vibration.

[0032] 2. Kinesthetic

[0033] Kinesthetic haptics simulates the motion states of an object such as weight or pressure. For example, in a driving video game, when moving at a high speed or operating a heavy vehicle, the steering wheel may resist turning. This type of feedback directly affects the user's muscles. In the example of the driving game, the user has to apply more force to obtain the desired response from the steering wheel.

[0034] 3. Electrohaptic

[0035] Electrohaptic presentation simulates feedback information of specific textures through electrical stimulation. Specifically, electrical pulses can be used to provide haptic stimuli to the nerve endings of the user's skin. Electrohaptic presentation can create a highly realistic experience for users wearing a suit or gloves equipped with electrohaptic technology. Almost any sensation can be simulated with electrical pulses, such as including: temperature changes, pressure changes, a sense of dampness, etc.

[0036] With the popularization of wearable devices and interactive devices, the haptic presentation that users can perceive when consuming media content will no longer be limited to basic vibrotactile, but can also include a more real-world haptic presentation experience with all-round body sensations such as vibration, pressure, speed, acceleration, temperature, humidity, smell, etc.

[0037] Figure 1 Schematic diagram of the haptic information representation and transmission system architecture 100 for the Chinese national video coding standard (Audio Video Coding Standard, AVS). Refer to Figure 1, the tactile information exchange format X can be generated based on the acquisition signal A or transcoded from the existing tactile signal format B. The tactile information exchange format X can be directly rendered on the client side, or encoded to generate the basic tactile information stream (also known as the basic bitstream, bitstream) E. The basic tactile information stream is encapsulated according to the corresponding media container file format (e.g., the ISO Based Media File Format (ISOBMFF)), to obtain a sequence Fs of initialization segments and media segments for streaming transmission or a media file F for file playback. The sequence Fs for streaming transmission is combined with the media presentation description information, and Fs is transmitted to the player using the transmission mechanism. The file F' or segment Fs' received by the file de-encapsulation process is de-encapsulated, the basic tactile information stream E' is extracted and the metadata is parsed, and then decoded to generate the tactile information exchange format X', which can finally be rendered on the client side.

[0038] The exchange format of tactile media is used to describe the signals contained in the tactile media. For example Figure 1 The exchange format of the tactile media defined by the provided AVS tactile information representation and transmission system architecture 100 is in the form of a human-readable JavaScript Object Notation (JSON) file. However, as Figure 1 The provided AVS tactile information representation and transmission system architecture 100 does not specify in detail how the tactile information exchange format X is encoded to obtain the bitstream (such as Figure 1 the dashed box in). Therefore, if the tactile media exchange format in the form of a JSON file is encapsulated and transmitted, there are problems that the memory and bandwidth occupancy need to be optimized.

[0039] Therefore, the embodiment of the present application proposes a method for processing tactile media content, encoding the exchange format of tactile information to obtain a binary bitstream corresponding to the target tactile information. Further, the above binary bitstream is encapsulated to obtain a media file containing the tactile information. The embodiment of the present application provides a solution for encoding the exchange format of tactile information into a binary bitstream. Specifically, the data is organized and divided in the above binary bitstream in the form of data units and data packets, and one data unit contains one or more data packets. The solution provided by the embodiment of the present application is beneficial to saving the occupied space of tactile media content.

[0040] Figure 2 It is a schematic diagram of the application scenario 200 for the processing of tactile information provided by the embodiment of the present application. Refer to Figure 2 , in the application scenario 200 for the processing of this tactile information, the terminal 220 can achieve tactile rendering, such as vibration. Data transmission and interaction can be carried out between the server 210 and the terminal 220 through the communication network 20.

[0041] During the implementation process, the above-mentioned terminal 220 can provide computing resources for running the algorithm model. For example, the above-mentioned terminal 220 can specifically be a product related to tactile feedback, or the player side or other intermediate nodes of an immersive system. For example, it can include a smart phone, a desktop computer, a mobile computing device, a notebook (e.g., laptop) computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, an in-vehicle computer, a navigator, a digital phone, a video phone, a television set, a sensing device, and a server, etc., which are not limited in the embodiments of the present application; the above-mentioned server 210 can specifically be the server side of an immersive system, such as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms. The above-mentioned communication network 20 can include direct or indirect connections in wired or wireless communication manners, which are not limited in the embodiments of the present application.

[0042] It should be noted that Figure 2 This is only an example of the application scenario provided by the embodiments of the present application. The application scenarios of the embodiments of the present application include but are not limited to Figure 2 as shown.

[0043] Based on the scenario as Figure 2 shown, a processing method P300 of tactile information as Figure 3 shown can be executed. Referring to Figure 3 , the processing method P300 of tactile information includes: S30 to S312.

[0044] In S30, the server 210 produces or acquires a tactile media signal according to the expected tactile media effect, and generates a tactile media signal exchange format.

[0045] In S32, the server 210 compresses and encodes the tactile media signal into a tactile media bitstream according to the solution provided by the embodiments of the present application (such as Figure 5 the solution provided by the embodiments).

[0046] In S34, the server 210 encapsulates the tactile media bitstream into a tactile media file, and the tactile media file contains one or more tactile media tracks.

[0047] In an exemplary embodiment, after determining the tactile media file, the server 210 can also slice the tactile media file into multiple tactile media file segments.

[0048] In S36, the server 210 sends the haptic media file or the above-mentioned haptic media file segment to the terminal 220 for the client installed on the terminal to obtain the above-mentioned haptic media file or haptic media file segment.

[0049] In S38, the client of the terminal 220 unpacks the haptic media file to obtain a haptic media bitstream.

[0050] In S310, the client of the terminal 220 will, according to the solution provided in the embodiments of the present application (such as Figure 7 the solution provided in the embodiment), decode the haptic media bitstream to obtain the exchange format of the haptic media signal.

[0051] In S312, the client of the terminal 220 renders the exchange format of the haptic media signal and performs haptic rendering.

[0052] Before elaborating on the technical solution of the embodiments of the present application in detail, first introduce the data structure defined for representing haptic media in the prior art. Among them, Figure 4 is a schematic diagram of the AVS haptic information representation data structure 400. Refer to Figure 4 , the data structure defined for representing haptic media in the prior art is as follows:

[0053] The highest-level data structure for haptic information representation is haptic experience, which is used to describe all haptic experience information in a file or bitstream. Haptic experience includes metadata information related to haptic experience, possible device information, and one or more haptic patterns.

[0054] A haptic pattern corresponds to a haptic signal of a certain type (such as vibration, pressure, temperature, etc.). The haptic pattern includes metadata information related to the haptic pattern, possibly predefined knowledge haptic events, and one or more haptic channels.

[0055] A haptic channel contains all or part of the haptic signals of the corresponding haptic pattern. The haptic channel includes metadata information related to the haptic channel and one or more haptic events. Generally speaking, haptic signals corresponding to different rendering devices under the same haptic pattern can be organized into different haptic channels.

[0056] A haptic event is a basic haptic signal unit. The haptic event includes metadata information related to the haptic event and possible haptic event components. Among them, the haptic event component is the time-domain or frequency-domain component of the haptic event.

[0057] 1. The data structure definition of haptic experience is shown in Table 1;

[0058] Table 1

[0059] Attribute Description version Indicates the standard version information followed by the haptic experience. creation_date Indicates the creation time of the haptic experience. description Indicates the description information of the haptic experience, which is user-defined. timescale The time scale of the haptic experience, indicating the number of clock units contained in one second. devices A list of device information elements, indicating one or more device information related to the haptic experience. patterns A list of haptic pattern elements, indicating one or more haptic patterns contained in the haptic experience.

[0060] 2. The data structure definition of the attribute device information for the tactile experience is shown in Table 2;

[0061] Table 2

[0062] Attribute Description id Indicates the identifier of the device device_name Indicates the (human-readable) name of the device body_parts Indicates the body part corresponding to the device

[0063] 3. The data structure definition of the tactile mode is shown in Table 3;

[0064] Table 3

[0065]

[0066]

[0067] 4. Knowledge tactile events;

[0068] Knowledge tactile events are special tactile events, and their data structure follows the data structure definition of tactile events. Knowledge tactile events are usually predefined and reusable tactile events to avoid repeated parsing of the same tactile event.

[0069] 5. The data structure definition of the tactile channel is shown in Table 4;

[0070] Table 4

[0071]

[0072] 6. The data structure definition of the tactile event is shown in Table 5;

[0073] Table 5

[0074]

[0075]

[0076] 7. The data structure definition of the attribute tactile event component of the tactile event is shown in Table 6;

[0077] Table 6

[0078] Attribute Description refer_event_id Indicates the identifier of the haptic event to which the haptic event component belongs. relative_time Indicates the time offset of the haptic event component relative to the haptic event. relative_amplitude Indicates the amplitude value ratio of the haptic event component relative to the haptic event, with a value range of 0 to 1. relative_frequency Indicates the frequency offset of the haptic event component relative to the haptic event.

[0079] The technical solutions of the embodiments of the present application will be described in detail below through some embodiments. These embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0080] Figure 5 It is a schematic flowchart of the processing method P500 for tactile information provided by the embodiments of the present application. Among them, the execution subject of the method P500 is an electronic device that can decode the tactile media bitstream, such as Figure 2 the terminal 220 inFigure 5 , method P500 includes: S510 and S520.

[0081] In S510, the media file containing the target tactile information is unpacked to obtain the binary bitstream corresponding to the above-mentioned target tactile information. The above bitstream includes: one or more data units regarding the above-mentioned target tactile information, and the above data unit includes one or more data packets.

[0082] In the embodiments of the present application, on the basis of the AVS tactile information representation data structure 400 shown in Figure 4 , several descriptive fields are added, including the field extension of the tactile media bitstream, to support the method steps of the embodiments of the present application.

[0083] Among them, the binary bitstream encoded and compressed by the tactile media exchange format is a self - contained transport stream format composed of a series of data units (AHAS Unit), and each data unit (AHAS Unit) corresponds to the presentation time of the tactile media.

[0084] Exemplarily, Figure 6 is a schematic diagram of the relationship between the data unit and the data packet provided by the embodiments of the present application. Refer to Figure 6 , the binary bitstream includes: the (i - 1)th data unit (AHAS Unit(i - 1)), the ith data unit (AHAS Unit(i)), the (i + 1)th data unit (AHAS Unit(i + 1)), and so on. Each data unit includes at least one data packet. For example, Figure 6 the ith data unit (AHAS Unit(i)) in includes an AHAS Unit Header, AHAS Packet 0, AHAS Packet 1, AHAS Packet 2... AHAS Packet n. Among them, each data packet contains tactile signals or metadata or auxiliary information of tactile signals. It will be specifically introduced in the following embodiments.

[0085] Through Figure 6 It can be seen that in the embodiments of the present application, the data is organized and divided in the form of data units and data packets in the bitstream. At the same time, the data units and data packets are divided into different types, and the types of data units and data packets determine the types of data they contain. Among them, Table 7 shows the types of data units provided by the embodiments of the present application and the types of data packets they contain.

[0086] Table 7

[0087]

[0088]

[0089] As can be seen from Table 7, the data units are divided into the following types:

[0090] 1. Data units of the initialization type:

[0091] The data units of the initialization type include a data packet of the first type, one or more data packets of the second type, one or more data packets of the third type, zero, one or more data packets of the fourth type, and zero or one data packet of the fifth type. It should be noted that if there is a data packet of the fourth type in the bitstream, the data units of the initialization type must include a data packet of the fourth type.

[0092] 2. Data units of the time domain type or the spatial domain type:

[0093] The data units of the time domain type or the spatial domain type include one or more data packets of the seventh type and zero or one data packet of the fifth type.

[0094] 3. Data units of the silent type:

[0095] The data units of the silent type include a data packet of the sixth type.

[0096] After introducing the relationship between the data units and the data packets in the embodiments of the present application, as well as the types of the data units and the types of the data packets, the following will introduce the syntax structures of the bitstream, the data units, and the data packets respectively through Tables 8-10. In the embodiments of the present application, the syntax and semantics of the bitstream are shown in Table 8.

[0097] Table 8

[0098]

[0099] Among them, endOfStream() is a function for determining whether the data units in the bitstream have ended. If the data units in the bitstream have ended, the function returns 1; if the data units in the bitstream have not ended, the function returns 0.

[0100] In the embodiments of the present application, the syntax and semantics of the data units (AHAS Unit) are shown in Table 9. Among them, the number of bits shown in Table 9 is exemplary data and can also be other values. The embodiments of the present application do not make any limitations in this regard.

[0101] Table 9

[0102]

[0103] Referring to Table 9, AHASUnitType represents the type of data unit. In the embodiments of the present application, different types of data units correspond to different binary values. As can be seen from Table 9, the number of bits occupied by the type of data unit is 3. For example, if the type of data unit is the silent type, it can be represented as "100". Exemplarily, the meaning of the binary values of this field can be as shown in Table 9.1.

[0104] Table 9.1

[0105]

[0106] Referring to Table 9, AHASUnitDependency represents the decoding dependency indicator.

[0107] When the value of this field is the first target value, it indicates that the current data unit does not depend on other data units of the same type during decoding; when the value of this field is not the above first target value, it indicates that the current data unit depends on other data units of the same type during decoding.

[0108] In some embodiments, the above first target value may be "0", and the value that is not the above first target value may specifically be "1". Since the initialization type data unit and the silent type data unit do not depend on other data units of the same type during decoding, the value of this field in the initialization type data unit and the silent type data unit is 0.

[0109] If the initialization type data unit is followed by a time domain / spatial domain type data unit and the value of the AHASUnitDependency field of the time domain / spatial domain type data unit is the first target value (e.g., 0), then it can be considered that the initialization type data unit can be used as the decoding initial unit, that is, the unit that can directly start decoding the bitstream, which is also the random access point for decoding the bitstream. In an exemplary embodiment, the current bitstream can be directly decoded starting from the initialization type data unit. The embodiments of the present application do not limit the above first target value.

[0110] Referring to Table 9, AHASUnitLayerFlag represents the layer indicator.

[0111] The above layer indicator is represented by binary values. When the value of this field is the second target value, it indicates that the layer of the data unit is not indicated; when the value of this field is not the above second target value, it indicates that the layer of the data unit is indicated. In an exemplary embodiment, the above second target value may be "0", and the value that is not the above second target value may be "1". Then, when the layer indicator AHASUnitLayerFlag has a value of 0, it indicates that the layer of the data unit is not indicated; when the value of this field is 1, it indicates that the layer of the data unit is indicated. Of course, the embodiments of the present application do not limit the above second target value.

[0112] Referring to Table 9, AHASLayer represents the hierarchical identifier of the data unit.

[0113] In the embodiments of the present application, the above hierarchical identifier is also represented by binary values. Meanwhile, the priority of the data unit can be determined by the value of the hierarchical identifier.

[0114] In some exemplary embodiments, the smaller the value of this field, the higher the priority of the data unit; conversely, the larger the value of this field, the lower the priority of the data unit. It can be understood that in other embodiments, it can also be that the larger the value of the AHASLayer field, the higher the priority of the data unit; conversely, the smaller the value of this field, the lower the priority of the data unit. The embodiments of the present application do not make limitations in this regard.

[0115] Referring to Table 9, AHASUnitTimestamp represents the first offset timestamp. Among them, different timestamps correspond to different binary values. In the embodiment shown in Table 9, the above first offset timestamp can occupy 32 bits.

[0116] The above first offset timestamp represents the offset time of the data unit relative to the start time of the tactile experience, in units of the clock. The number of clocks included in one second is specified by the timescale field in the experience metadata. Exemplarily, the ratio of the above first offset timestamp AHASUnitTimestamp to timescale is the number of seconds passed in the tactile experience.

[0117] For different types of data units, the value of this field can follow the following constraints:

[0118] - For initialization-type data units, this field has no practical significance;

[0119] - For silent-type data units, this field should be greater than the timestamp of any data packet in the previous time domain / spatial domain type data unit of the silent-type data unit.

[0120] - For time domain type or spatial domain type data units, the timestamp of any data packet included in the i-th data unit should be less than the timestamp of any data packet included in the (i + 1)-th data unit. In some embodiments, this field should be the minimum value of the timestamps of the M data packets it contains. In other embodiments, this field should be the maximum value of the timestamps of the M data packets it contains.

[0121] The above first offset timestamp can be used to achieve random access during the transmission of tactile signals, and the specific implementation will be introduced in detail in the subsequent embodiments regarding the second offset timestamp.

[0122] Referring to Table 9, AHASUnitLength represents the number of bytes of all data packets within a data unit. Among them, different numbers of bytes correspond to different binary values. In the embodiment shown in Table 9, the number of bytes of all data packets within the above-mentioned data unit can occupy 32 bits.

[0123] Referring to Table 9, endOfUnit() is a function for determining whether the data packet in the data unit has ended. If the data packet in the data unit has ended, this function returns 1; if the data unit has not ended, this function returns 0. Exemplarily, when the data packet data of AHASUnitLength bytes included in the data packet is parsed, the data unit ends.

[0124] Next, the syntax and semantics of the data unit (AHAS Packet) are introduced through Table 10. Among them, the number of bits shown in Table 10 is exemplary data and can also be other values, which are not limited in the embodiments of the present application.

[0125] Table 10

[0126]

[0127] Referring to Table 10, AHASPacketType represents the type of the data packet. In the embodiments of the present application, different types of data packets correspond to different binary values. Referring to Table 10, it can be seen that the number of bits occupied by the type of the data packet is 4. For example, if the type of the data packet is the sixth type containing silent information, it can be represented as "0110". Exemplarily, the binary value meaning of this field can be as shown in Table 10.1.

[0128] Table 10.1

[0129] AHASPacketType (type of data packet) Value Remarks PACTYPE_METADATAEXPERIENCE 0 The first type containing haptic experience metadata PACTYPE_METADATAPATTERN 1 The second type containing haptic pattern metadata PACTYPE_METADATACHANNEL 2 The third type containing haptic channel metadata PACTYPE_LIBRARYEVENTS 3 The fourth type containing knowledge haptic events PACTYPE_CRC 4 The fifth type containing cyclic redundancy check codes PACTYPE_SILENTINFO 5 The sixth type containing silent information PACTYPE_DATA 6 The seventh type containing haptic signals Reserved 7～15

[0130] Referring to Table 10, AHASPacketLength is used to indicate the number of bytes of the payload of the data packet, that is, the number of bytes of AHASPacketPayload. In the embodiments of the present application, the number of bytes of the payload of the data packet can be represented by a binary value. Referring to Table 10, it can be seen that the number of bits occupied by the number of bytes of the payload of the data packet is 20.

[0131] Referring to Table 10, AHASPacketTimestamp represents the second offset timestamp. Among them, different timestamps correspond to different binary values. In the embodiment shown in Table 10, the above-mentioned first offset timestamp can occupy 24 bits.

[0132] The above-mentioned second offset timestamp is used to represent the offset relative to the first timestamp of the data unit where the data packet is located, in units of clocks. The number of clocks contained in one second is specified by the timescale field in the experience metadata. Specifically, in some embodiments, the above-mentioned second offset timestamp AHASPacketTimestamp represents the minimum value among the timestamps of all haptic events included in the data packet. In other embodiments, the above-mentioned second offset timestamp AHASPacketTimestamp represents the maximum value among the timestamps of all haptic events included in the data packet.

[0133] In the embodiments of the present application, the timestamp of the current haptic signal data packet can be determined according to the second offset timestamp of the data packet and the first offset timestamp of the data unit where the data packet is located.

[0134] In the case where the first offset timestamp AHASUnitTimestamp represents the minimum value of the timestamps of all data packets it contains, AHASUnitTimestamp + AHASPacketTimestamp can be determined as the timestamp of the current haptic signal data packet. In the case where the first offset timestamp AHASUnitTimestamp represents the maximum value of the timestamps of all data packets it contains, AHASUnitTimestamp - AHASPacketTimestamp can be determined as the timestamp of the current haptic signal data packet.

[0135] It can be seen that the solution provided by the embodiments of the present application can determine the timestamp of the current haptic signal data packet through the second offset timestamp of the current data packet and the first offset timestamp of the data unit where the data packet is located, thereby enabling random access to haptic signals during transmission.

[0136] Referring to Table 10, byteAlignment() is a function for judging byte alignment. If the data contained in the current data packet is an integer number of bytes, 0 is returned; if the data contained in the current data packet is less than an integer number of bytes, 0 is returned after padding the last byte.

[0137] Referring to Table 10, the syntax of AHASPacketPayload can be introduced in combination with Table 10.2.

[0138] Table 10.2

[0139]

[0140] According to the different types of data packets, during the execution of the syntax in Table 10.2, it will turn to the syntax of the payload of the corresponding type of data packet shown in one of Table 10.2-A to Table 10.2-G. Among them, the number of bits shown in Table 10.2-A to Table 10.2-G is exemplary data and can also be other values, which are not limited in the embodiments of the present application.

[0141] When the type of the data packet is the first type including haptic experience metadata, during the execution of the syntax in Table 10.2, it will turn to the syntax of the payload of this data packet shown in Table 10.2-A.

[0142] Table 10.2-A

[0143]

[0144]

[0145] When the type of the data packet is the first type including haptic experience metadata, the payload of this data packet includes one or more of the following fields shown in Table 10.2-A. Each field in Table 10.2-A can be represented by binary values. Specifically, the specific indication information of each field is as follows:

[0146] Version is used to indicate the standard version information followed by the haptic experience; the binary value used to represent Version can occupy 8 bits.

[0147] dateLength is used to indicate the string length of the creation date of the haptic experience; the binary value used to represent dateLength can occupy 8 bits.

[0148] Date is used to indicate the creation date of the haptic experience; the binary value used to represent Date can occupy dateLength×8 bits.

[0149] descriptionLength is used to indicate the string length of the description of the haptic experience; the binary value used to represent Date can occupy 8 bits.

[0150] Description is used to indicate the description of the haptic experience; the binary value used to represent Description can occupy descriptionLength×8 bits.

[0151] Timescale is used to indicate the time scale of the haptic experience, and the value of this field represents the number of clock units included in one second; the binary value used to represent Timescale can occupy 16 bits.

[0152] patternCount is used to indicate the number of patterns included in the tactile experience; the binary value used to represent patternCount can occupy 8 bits.

[0153] deviceCount is used to indicate the number of devices corresponding to the tactile experience; the binary value used to represent deviceCount can occupy 8 bits.

[0154] Regarding readDevice() in Table 10.2-A, its syntax can be referred to Table 10.2-A1.

[0155] Table 10.2-A1

[0156]

[0157] The payload of the above data packet further includes: one or more of the following fields shown in Table 10.2-A1 regarding the j-th device. Among them, each field in Table 10.2-A1 can be reflected by binary values. Specifically, the specific indication information of each field is as follows:

[0158] deviceId is used to indicate the identifier of the j-th device; the binary value used to represent deviceId can occupy 8 bits.

[0159] nameLength is used to indicate the string length of the name of the j-th device; the binary value used to represent nameLength can occupy 8 bits.

[0160] deviceName is used to indicate the human-readable name of the j-th device; the binary value used to represent deviceName can occupy nameLength × 8 bits.

[0161] bodyPartFlag is an indicator for whether the device corresponds to a body part; the binary value used to represent bodyPartFlag can occupy 1 bit; for example, when the value is 1, it indicates the body part information corresponding to the device; when the value is 0, it does not indicate the body part information corresponding to the device.

[0162] bodyParts is used to indicate the body part corresponding to the device; the binary value used to represent bodyParts can occupy 8 bits.

[0163] In the case where the data packet type is the second type including tactile pattern metadata, during the execution of the syntax in Table 10.2, the syntax of the payload of this data packet shown in Table 10.2-B will be turned to.

[0164] Table 10.2-B

[0165]

[0166] When the type of the data packet is the second type including haptic mode metadata, the payload of the data packet includes one or more of the following fields shown in Table 10.2-B. Each field in Table 10.2-B can be represented by binary values. Specifically, the specific indication information for each field is as follows:

[0167] Id is used to indicate the identifier of the haptic mode; the binary value used to represent Id can occupy 8 bits.

[0168] patternType is used to indicate the type of the haptic mode; the binary value used to represent patternType can occupy 8 bits. For example, if the type of the data pattern is acceleration, it can be represented as "0000 0010". Exemplarily, the meaning of the binary value of this field can be as shown in Table 10.2-B1.

[0169] Table 10.2-B1

[0170] patternType value Meaning 0 Vibrotactile 1 Pressure 2 Acceleration 3 Velocity 4 Temperature 5 Electrotactile 6～255 Reserved

[0171] semanticisFlag is an indicator used to represent whether there is semantics in the haptic event corresponding to the haptic mode; the binary value used to represent semanticisFlag can occupy 1 bit. For example, when the value is 1, it means that there is semantics in the haptic event corresponding to the haptic mode, and when the value is 0, it means that there is no semantics in the haptic event corresponding to the haptic mode.

[0172] sementicsType is an indicator used to represent the specification followed by the semantics of the haptic event; the binary value used to represent sementicsType can occupy 8 bits. For example, when the value is 0, it means that the semantics of the haptic event follows the specification defined in this standard; when the value is 1, it means that the semantics of the haptic event follows the specification of an external standard; when the value is 2, it means that the semantics of the haptic event is user-defined.

[0173] referenceDeviceCount is used to represent the number of devices corresponding to the haptic mode; the binary value used to represent referenceDeviceCount can occupy 8 bits.

[0174] referenceDeviceId is used to indicate the identifier of the device corresponding to the haptic mode; the binary value used to represent referenceDeviceId can occupy 8 bits.

[0175] channelCount is used to indicate the number of haptic channels included in the haptic mode; the binary value used to represent channelCount can occupy 16 bits.

[0176] When the data packet type is the third type that includes haptic channel metadata, during the execution of the syntax in Table 10.2, the syntax of the payload of this data packet shown in Table 10.2-C will be followed.

[0177] Table 10.2-C

[0178]

[0179]

[0180] When the type of the data packet is the third type that includes haptic channel metadata, the payload of this data packet includes one or more of the following fields shown in Table 10.2-C. Each field in Table 10.2-C can be represented by a binary value. Specifically, the specific indication information for each field is as follows:

[0181] Id is used to indicate the identifier of the haptic channel; the binary value used to represent Id can occupy 16 bits.

[0182] perceptionId is used to indicate the identifier of the haptic mode corresponding to the haptic channel; the binary value used to represent perceptionId can occupy 8 bits.

[0183] descriptionLength is used to indicate the string length of the haptic channel description; the binary value used to represent descriptionLength can occupy 8 bits.

[0184] Description is used to indicate the haptic channel description; the binary value used to represent Description can occupy 8 × descriptionLength bits.

[0185] Gain is used to indicate the gain of the haptic channel; the binary value used to represent Gain can occupy 32 bits. Specifically, it is in units of 2 - 31. This gain Gain is applied to all events included in the current channel and is used to restore the normalized signal value to the original signal value.

[0186] referenceDeviceCount is used to indicate the number of devices corresponding to the haptic channel; the binary value used to represent referenceDeviceCount can occupy 8 bits.

[0187] The referenceDeviceId is used to indicate the identifier of the device corresponding to the tactile channel; the binary value used to represent the referenceDeviceId can occupy 8 bits.

[0188] The eventsCount is used to indicate the number of tactile events included in the tactile channel; the binary value used to represent the eventsCount can occupy 16 bits.

[0189] In the case where the data packet type is the seventh type containing tactile signals, during the execution of the syntax in Table 10.2, the syntax of the payload of this data packet shown in Table 10.2-D will be followed.

[0190] Table 10.2-D

[0191]

[0192] In the case where the data packet type is the seventh type containing tactile signals, the payload of this data packet includes one or more of the following fields shown in Table 10.2-D. Each field in Table 10.2-D can be represented by a binary value. Specifically, the specific indication information for each field is as follows:

[0193] The packetDependency is an indicator used to indicate whether the current data packet depends on other data packets of the same type during decoding; the binary value used to represent the packetDependency can occupy 1 bit; for example, when the value of this field is 0, it means that the current data packet does not depend on other data packets of the same type during decoding, and when the value of this field is 1, it means that the current data packet depends on other data packets of the same type during decoding.

[0194] The perceptionId is used to indicate the identifier of the tactile pattern corresponding to the tactile event in the data packet; the binary value used to represent the perceptionId can occupy 8 bits.

[0195] The channelId is used to indicate the identifier of the tactile channel corresponding to the tactile event in the data packet; the binary value used to represent the channelId can occupy 8 bits.

[0196] The eventsCount is used to indicate the number of tactile events corresponding to the data packet; the binary value used to represent the eventsCount can occupy 16 bits.

[0197] Regarding readEvent() in Table 10.2-D, its syntax can refer to Table 10.2-D1.

[0198] Table 10.2-D1

[0199]

[0200] The payload of the above data packet further includes one or more of the following fields shown in Table 10.2-D1 regarding the m-th haptic event. Among them, each field in Table 10.2-D1 can be represented by binary values. Specifically, the specific indication information of each field is as follows:

[0201] eventId is used to indicate the identifier of the haptic event; the binary value used to represent eventId can occupy 16 bits.

[0202] eventType is used to indicate the type of the haptic event; the binary value used to represent eventType can occupy 4 bits. For example, when the value of this field is 0, it indicates that the event is an instantaneous event; when the value of this field is 1, it indicates that the event is a continuous event; when the value of this field is 2, it indicates that the event is a reference event, etc.

[0203] eventSementicsFlag is an indicator value used to indicate whether to indicate the semantics of the haptic event; the binary value used to represent eventSementicsFlag can occupy 1 bit. For example, when the value is 0, it does not indicate the semantics of the haptic event; when the value is 1, it indicates the semantics of the haptic event.

[0204] semanticKeywords represents the semantic information of the haptic event; the binary value used to represent semanticKeywords can occupy 12 bits. The semantics of the haptic event are the keyword of the event auxiliary information that describes the intention of the content producer.

[0205] referEventId is used to indicate the identifier of the knowledge haptic event corresponding to the reference event; the binary value used to represent referEventId can occupy 16 bits.

[0206] relativePosition is used to indicate the time or space offset of the haptic event; the binary value used to represent relativePosition can occupy 32 bits.

[0207] Duration is used to indicate the duration of the haptic event, and this attribute exists when the event type is a continuous event; the binary value used to represent Duration can occupy 32 bits.

[0208] Amplitude is used to indicate the maximum amplitude value of the signal of the haptic event; the binary value used to represent Amplitude can occupy 8 bits.

[0209] baseFrequency is used to indicate the signal reference frequency of a haptic event; the binary value used to represent baseFrequency can occupy 16 bits.

[0210] numComponents is used to indicate the number of components of a haptic event, and the components of a haptic event can be time-domain components or frequency-domain components of the haptic event; the binary value used to represent numComponents can occupy 16 bits.

[0211] Regarding readEventComponent() in Table 10.2-D1, its syntax can be referred to in Table 10.2-D2.

[0212] Table 10.2-D2

[0213]

[0214] The payload of the above data packet further includes one or more of the following fields shown in Table 10.2-D2 for the k-th haptic event component. Among them, each field in Table 10.2-D2 can be reflected by a binary value. Specifically, the specific indication information of each field is as follows:

[0215] refer_event_id is used to indicate the identifier of the haptic event to which the haptic event component belongs; the binary value used to represent refer_event_id can occupy 16 bits.

[0216] position_flag is an indicator used to represent whether the spatio-temporal domain offset is included in the haptic event component; the binary value used to represent position_flag can occupy 1 bit. For example, when position_flag takes the value of 1, it indicates that the spatio-temporal domain offset is included in the haptic event component, and when it takes the value of 0, it indicates that the spatio-temporal domain offset is not included in the haptic event component.

[0217] amplitude_flag is an indicator used to represent whether the amplitude value offset is included in the haptic event component; the binary value used to represent amplitude_flag can occupy 1 bit. For example, when amplitude_flag takes the value of 1, it indicates that the amplitude value offset is included in the haptic event component, and when it takes the value of 0, it indicates that the amplitude value offset is not included in the haptic event component.

[0218] frequency_flag is an indicator used to represent whether a haptic event component contains a frequency offset; the binary value used to represent frequency_flag can occupy 1 bit. For example, when the value of frequency_flag is 1, it indicates that the haptic event component contains a frequency offset, and when the value is 0, it indicates that the haptic event component does not contain a frequency offset.

[0219] relative_position is used to represent the temporal or spatial offset of a haptic event component relative to a haptic event; the binary value used to represent relative_position can occupy 16 bits.

[0220] relative_amplitude is used to represent the amplitude value ratio of a haptic event component relative to a haptic event, with a value range of 0 to 215 in units of 2-15; the binary value used to represent relative_amplitude can occupy 16 bits.

[0221] relative_frequency is used to represent the frequency offset of a haptic event component relative to a haptic event; the binary value used to represent relative_frequency can occupy 16 bits.

[0222] In the case where the data packet type is the fourth type that includes a knowledge haptic event, during the execution of the syntax in Table 10.2, the syntax of the payload of this data packet shown in Table 10.2-E will be followed.

[0223] Table 10.2-E

[0224]

[0225] In the case where the data packet type is the fourth type that includes a knowledge haptic event, the payload of this data packet includes one or more of the following fields shown in Table 10.2-E. Each field in Table 10.2-E can be represented by a binary value. Specifically, the specific indication information for each field is as follows:

[0226] patternId is used to represent the identifier of the haptic pattern corresponding to a knowledge haptic event; the binary value used to represent patternId can occupy 8 bits.

[0227] eventsCount is used to represent the number of knowledge haptic events corresponding to a data packet; the binary value used to represent eventsCount can occupy 16 bits.

[0228] Regarding readLibraryEvent() in Table 10.2-E, its syntax can be referred to in Table 10.2-E1.

[0229] Table 10.2-E1

[0230]

[0231] The payload of the above data packet also includes one or more of the following fields shown in Table 10.2-E1 for the h-th knowledge haptic event. Among them, each field in Table 10.2-E1 can be represented by binary values. Specifically, the specific indication information for each field is as follows:

[0232] eventId is used to represent the identifier of the knowledge haptic event; the binary value used to represent eventId can occupy 16 bits.

[0233] eventType is used to represent the type of the knowledge haptic event; the binary value used to represent eventType can occupy 4 bits. For example, when the value of this field is 0, it indicates that the event is an instantaneous event, and when the value of this field is 1, it indicates that the event is a continuous event.

[0234] eventSementicsFlag is an indicator used to represent whether to indicate the semantics of the knowledge haptic event; the binary value used to represent eventSementicsFlag can occupy 1 bit. For example, when the value is 0, it does not indicate the semantics of the knowledge haptic event, and when the value is 1, it indicates the semantics of the knowledge haptic event.

[0235] semanticKeywords is used to represent the semantic information of the event; the binary value used to represent semanticKeywords can occupy 12 bits. The semantics of the knowledge haptic event are the keyword of the event auxiliary information that describes the intention of the content producer.

[0236] relativePosition is used to represent the time or space offset of the knowledge haptic event; the binary value used to represent relativePosition can occupy 32 bits.

[0237] Duration is used to represent the duration of the knowledge haptic event, and this attribute exists when the event type is a continuous event; the binary value used to represent Duration can occupy 32 bits.

[0238] Amplitude is used to represent the maximum amplitude value of the signal of the knowledge haptic event; the binary value used to represent Amplitude can occupy 8 bits.

[0239] baseFrequency is used to represent the signal reference frequency of a knowledge tactile event; the binary value used to represent baseFrequency can occupy 16 bits.

[0240] numComponents is used to represent the number of components of a knowledge tactile event. The components of a knowledge tactile event can be time-domain components or frequency-domain components of the knowledge tactile event; the binary value used to represent numComponents can occupy 16 bits.

[0241] Regarding readEventComponent() in Table 10.2-E1, its syntax can be referred to in Table 10.2-D2. This will not be elaborated here.

[0242] In the case where the data packet type is the sixth type containing silent data, during the execution of the syntax in Table 10.2, the syntax of the payload of this data packet shown in Table 10.2-F will be turned to.

[0243] Table 10.2-F

[0244]

[0245] In the case where the data packet type is the sixth type containing silent data, the payload of this data packet includes the fields shown in Table 10.2-F. The fields in Table 10.2-F can be reflected by binary values. Specifically, the specific indication information of this field is as follows:

[0246] silentDuration is used to represent the length of the silent time, in units of time scale; the binary value used to represent silentDuration can occupy 16 bits.

[0247] In the case where the data packet type is the fifth type containing a cyclic redundancy check code, during the execution of the syntax in Table 10.2, the syntax of the payload of this data packet shown in Table 10.2-G will be turned to.

[0248] Table 10.2-G

[0249]

[0250] In the case where the data packet type is the fifth type containing a cyclic redundancy check code, the payload of this data packet includes one or more of the following fields shown in Table 10.2-G. Each field in Table 10.2-G can be reflected by binary values. Specifically, the specific indication information of each field is as follows:

[0251] CRCType is used to represent the type of cyclic redundancy check code; the binary value used to represent CRCTypey can occupy 8 bits; for example, when the value is 0, it means the cyclic redundancy check code is a 16-bit check code, and the data packet protected by this cyclic redundancy check code is the data packet after the cyclic redundancy check code data packet; when the value is 1, it means the cyclic redundancy check code is a 32-bit check code, and the data packet protected by this cyclic redundancy check code is the data packet after the cyclic redundancy check code data packet; when the value is 2, it means the cyclic redundancy check code is a 16-bit check code, and the range of data packets protected by this cyclic redundancy check code is the protectedPacketsCount data packets after the cyclic redundancy check code data packet. When the value is 3, it means the cyclic redundancy check code is a 32-bit check code, and the range of data packets protected by this cyclic redundancy check code is the protectedPacketsCount data packets after the cyclic redundancy check code data packet.

[0252] CRC16Value is used to represent a 16-bit check code; the binary value used to represent CRC16Value can occupy 16 bits.

[0253] CRC32Value is used to represent a 32-bit check code; the binary value used to represent CRC32Value can occupy 32 bits.

[0254] protectedPacketsCount is used for the number of data packets protected by the cyclic redundancy check code; the binary value used to represent protectedPacketsCount can occupy 8 bits.

[0255] Continue to refer to Figure 5 , in S520, decode the binary bitstream corresponding to the above target tactile information to obtain the interchange format of the above target tactile information.

[0256] A data exchange format refers to a standardized format used for data exchange between different systems. In the embodiments of this application, the exchange format of the above-mentioned target tactile information is a standardized format used for data exchange between the server side and the terminal (such as an interaction device) that presents tactile information. Specifically, the terminal (such as an interaction device) can obtain the exchange format of the above-mentioned target tactile information after decoding the bitstream. Further, based on the exchange format of the above-mentioned target tactile information, the above-mentioned target tactile information can be rendered and presented. For example, if the above-mentioned target tactile information is information about vibration, the terminal can present the corresponding vibration effect on the terminal according to the exchange format of the above-mentioned target information. Among them, the exchange format of tactile information is not limited to JSON, and can also be other types of data exchange formats, such as Extensible Markup Language (XML), Comma-Separated Values (CSV), and other data exchange formats.

[0257] Based on the AVS tactile information representation data structure 400 shown in Figure 4 and the field extension of the tactile media bitstream provided by the embodiments of this application (such as the embodiments provided in Tables 7 - 10.2 above), it is possible to decode the binary bitstream corresponding to the above-mentioned target tactile information to obtain the exchange format of the above-mentioned target tactile information.

[0258] Further, the terminal client can render the exchange format of the above-mentioned target tactile information and perform tactile presentation.

[0259] Figure 5 In the solution provided by the shown embodiment, the binary bitstream corresponding to the target tactile information is decoded to obtain the exchange format of the target tactile information. Among them, the above-mentioned binary bitstream is obtained based on the scheme provided by the embodiments of this application for encoding the exchange format of tactile information into a binary bitstream. Specifically, the data is organized and divided in the form of data units and data packets in the above-mentioned binary bitstream. One data unit contains one or more data packets. The scheme provided by the embodiments of this application is beneficial to saving the occupied space of tactile media content and can reduce the bandwidth occupancy during the transmission of tactile media content. At the same time, it also supports random access of tactile signals during transmission.

[0260] Figure 5 The corresponding embodiment introduces in detail the scheme for decoding the bitstream corresponding to the target tactile information. The following Figure 7 introduces in detail the embodiment provided by this application for encoding the exchange format of the target tactile information to obtain the corresponding bitstream of the tactile information.

[0261] Figure 7This is a schematic flowchart of the processing method P700 for tactile information provided by an embodiment of the present application. Among them, the execution subject of method P700 is an electronic device that can encode the exchange format of target tactile information to obtain a tactile media bitstream, such as Figure 2 the server 210 in Figure 7 . Method P700 includes: S710 and S720.

[0262] In S710, encode the exchange format of the target tactile information to obtain a binary bitstream corresponding to the above target tactile information. The above bitstream includes: one or more data units regarding the above target tactile information, and the above data unit includes one or more data packets.

[0263] An embodiment of the present application adds several descriptive fields, including field extensions of the tactile media bitstream, on the basis of the AVS tactile information representation data structure 400 shown in Figure 4 to support the method steps of the embodiment of the present application.

[0264] The data exchange format refers to a standardized format used when exchanging data between different systems. In an embodiment of the present application, the exchange format of the above target tactile information is a standardized format used for exchanging data between the server side and a terminal (such as an interaction device) that presents tactile information. Specifically, for the convenience of transmission, the server side encodes the exchange format of the target tactile information into a binary bitstream and transmits the binary bitstream to the terminal (such as an interaction device). Thus, after obtaining the exchange format of the above target tactile information, the terminal (such as an interaction device) can perform rendering and presentation based on the above target tactile information. For example, if the above target tactile information is information about vibration, the terminal can present a corresponding vibration effect according to the exchange format of the above target information at the terminal. Among them, the exchange format of tactile information is not limited to JSON, and can also be other types of data exchange formats, such as Extensible Markup Language (XML), Comma-Separated Values (CSV), and other data exchange formats.

[0265] Among them, the binary bitstream encoded and compressed by the tactile media exchange format is a self-contained transport stream format composed of a series of data units (AHAS Unit), and each data unit (AHAS Unit) corresponds to the presentation time of the tactile media.

[0266] Exemplarily, Figure 6 This is a schematic diagram of the relationship between data units and data packets provided by an embodiment of the present application. Refer to Figure 6, the binary bitstream includes: the (i-1)th data unit (AHAS Unit(i-1)), the ith data unit (AHAS Unit(i)), the (i+1)th data unit (AHAS Unit(i+1)), and so on. Each data unit includes at least one data packet, such as Figure 6 the ith data unit (AHAS Unit(i)) in Figure 6 includes a data unit header (AHAS Unit Header), data packet 0 (AHAS Packet 0), data packet 1 (AHAS Packet 1), data packet 2 (AHAS Packet 2), …… data packet n (AHASPacket n). Among them, each data packet contains a tactile signal or metadata or auxiliary information of the tactile signal. This will be specifically introduced in the following embodiments.

[0267] Through Figure 6 It can be seen that in the embodiments of the present application, the data is organized and divided in the form of data units and data packets in the bitstream. At the same time, the data units and data packets are divided into different types, and the types of data units and data packets determine the types of data they contain. Among them, Table 7 shows the types of data units provided in the embodiments of the present application and the types of data packets they contain.

[0268] It can be seen from Table 7 that the data units are divided into the following types:

[0269] 1. Data units of initialization type:

[0270] The data units of initialization type contain a data packet of the first type, one or more data packets of the second type, one or more data packets of the third type, zero, one or more data packets of the fourth type, and zero or one data packet of the fifth type. Data units of time domain / space domain type:

[0271] 2. Data units of time domain type or space domain type:

[0272] The data units of time domain type or space domain type contain one or more data packets of the seventh type and zero or one data packet of the fifth type.

[0273] 3. Data units of silent type:

[0274] The data units of silent type contain a data packet of the sixth type.

[0275] After introducing the relationship between data units and data packets in the embodiments of the present application, as well as the types of data units and data packets, the syntax structures of the bitstream, data units, and data packets will be introduced respectively through Tables 8-10 below. In the embodiments of the present application, the syntax and semantics of the bitstream are shown in Table 8

[0276] In the embodiments of the present application, the syntax and semantics of the data unit (AHAS Unit) are shown in Table 11. The number of bits shown in Table 11 is exemplary data and can also be other values. The embodiments of the present application do not limit this.

[0277] Table 11

[0278]

[0279]

[0280] Referring to Table 11, AHASUnitType represents the type of the data unit. In the embodiments of the present application, different types of data units correspond to different binary values. Referring to Table 9, it can be seen that the number of bits occupied by the type of the data unit is 3. For example, if the type of the data unit is the silent type, it can be represented as "100". Exemplarily, the meaning of the binary values of this field can be shown in Table 11.1.

[0281] Table 9.1

[0282]

[0283] Referring to Table 11, AHASUnitDependency represents the coding dependency indicator.

[0284] When the value of this field is the first target value, it means that the current data unit does not depend on other data units of the same type during coding; when the value of this field is not the above first target value, it means that the current data unit depends on other data units of the same type during coding.

[0285] In some embodiments, the above first target value can be "0", and the value that is not the above first target value can be "1". Since the initialization type data unit and the silent type data unit do not depend on other data units of the same type during coding, the value of this field in the initialization type data unit and the silent type data unit is 0.

[0286] If the initialization type data unit is followed by a time domain / spatial domain type data unit and the value of the AHASUnitDependency field of this time domain / spatial domain type data unit is the first target value (e.g., 0), then it can be considered that this initialization type data unit can be used as the coding initial unit, that is, the unit that can directly start coding the bitstream, which is also the random access point of the coded bitstream. In an exemplary embodiment, the current bitstream can be directly coded starting from this initialization type data unit. The embodiments of the present application do not limit the above first target value.

[0287] Referring to Table 11, AHASUnitLayerFlag represents the layer indicator.

[0288] The above hierarchical indicator is represented by binary values. When the value of this field is the second target value, it indicates that the hierarchy of the data unit is not indicated; when the value of this field is not the above second target value, it indicates the hierarchy of the data unit. In an exemplary embodiment, the above second target value may be "0", and the value that is not the above second target value may be "1". Then, when the value of the hierarchical indicator AHASUnitLayerFlag is 0, it indicates that the hierarchy of the data unit is not indicated; when the value of this field is 1, it indicates the hierarchy of the data unit. Of course, the embodiments of the present application do not limit the above second target value.

[0289] Referring to Table 11, AHASLayer represents the hierarchical identifier of the data unit.

[0290] In the embodiments of the present application, the above hierarchical identifier is also represented by binary values. At the same time, the priority of the data unit can be determined by the value of the hierarchical identifier.

[0291] In some exemplary embodiments, the smaller the value of this field, the higher the priority of the data unit; conversely, the larger the value of this field, the lower the priority of the data unit. It can be understood that in other embodiments, it may also be that the larger the value of the AHASLayer field, the higher the priority of the data unit; conversely, the smaller the value of this field, the lower the priority of the data unit. The embodiments of the present application do not limit this.

[0292] Referring to Table 11, AHASUnitTimestamp represents the first offset timestamp. Among them, different timestamps correspond to different binary values. In the embodiment shown in Table 11, the above first offset timestamp may occupy 32 bits.

[0293] The above first offset timestamp represents the offset time of the data unit relative to the start time of the tactile experience, in units of clocks. The number of clocks included in one second is specified by the timescale field in the experience metadata. Exemplarily, the ratio of the above first offset timestamp AHASUnitTimestamp to timescale is the number of seconds elapsed in the tactile experience.

[0294] For data units of different types, the value of this field may follow the following constraints:

[0295] - For initialization-type data units, this field has no practical significance;

[0296] - For silent-type data units, this field should be greater than the timestamp of any data packet in the previous time domain / spatial domain type data unit of the silent-type data unit.

[0297] - For data units of the time domain type or the space domain type, the timestamp of any data packet included in the i-th data unit shall be less than the timestamp of any data packet included in the (i + 1)-th data unit. In some embodiments, this field shall be the minimum value of the timestamps of the M data packets it contains. In other embodiments, this field shall be the maximum value of the timestamps of the M data packets it contains.

[0298] The above first offset timestamp can be used to achieve random access during the transmission of tactile signals. The specific implementation will be described in detail in the subsequent embodiments regarding the second offset timestamp.

[0299] Referring to Table 11, AHASUnitLength represents the number of bytes of all data packets within the data unit. Among them, different numbers of bytes correspond to different binary values. In the embodiment shown in Table 9, the number of bytes of all data packets within the above data unit can occupy 32 bits.

[0300] Referring to Table 11, endOfUnit() is a function for determining whether the data packets in the data unit have ended. If the data packets in the data unit have ended, this function returns 1; if the data unit has not ended, this function returns 0. Exemplarily, when the data packet data of AHASUnitLength bytes included in the data packet is parsed, the data unit ends.

[0301] The syntax and semantics of the data unit (AHAS Packet) are introduced through Table 12. Among them, the number of bits shown in Table 12 is exemplary data and can also be other values. The embodiments of the present application do not limit this.

[0302] Table 12

[0303]

[0304] Referring to Table 12, AHASPacketType represents the type of the data packet. In the embodiments of the present application, different types of data packets correspond to different binary values. Referring to Table 12, it can be seen that the number of bits occupied by the type of the data packet is 4. For example, if the type of the data packet is the sixth type including silent information, it can be represented as "0110". Exemplarily, the meaning of the binary value of this field can be as shown in Table 10.1 above and will not be elaborated here.

[0305] Referring to Table 12, AHASPacketLength is used to indicate the number of bytes of the payload of the data packet, that is, the number of bytes of AHASPacketPayload. In the embodiments of the present application, the number of bytes of the payload of the data packet can be represented by a binary value. Referring to Table 12, it can be seen that the number of bits occupied by the number of bytes of the payload of the data packet is 20.

[0306] Referring to Table 12, AHASPacketTimestamp represents the second offset timestamp. Among them, different timestamps correspond to different binary values. In the embodiment shown in Table 12, the above first offset timestamp may occupy 24 bits.

[0307] The above second offset timestamp is used to represent the offset relative to the first timestamp of the data unit where the data packet is located, in units of the clock. The number of clocks included in one second is specified by the timescale field in the experience metadata. Specifically, in some embodiments, the above second offset timestamp AHASPacketTimestamp represents the minimum value among the timestamps of all haptic events included in the data packet. In other embodiments, the above second offset timestamp AHASPacketTimestamp represents the maximum value among the timestamps of all haptic events included in the data packet.

[0308] In the embodiments of the present application, the timestamp of the current haptic signal data packet can be determined according to the second offset timestamp of the data packet and the first offset timestamp of the data unit where the data packet is located.

[0309] In the case where the first offset timestamp AHASUnitTimestamp represents the minimum value of the timestamps of all data packets it contains, AHASUnitTimestamp + AHASPacketTimestamp can be determined as the timestamp of the current haptic signal data packet. In the case where the first offset timestamp AHASUnitTimestamp represents the maximum value of the timestamps of all data packets it contains, AHASUnitTimestamp - AHASPacketTimestamp can be determined as the timestamp of the current haptic signal data packet.

[0310] It can be seen that the solution provided by the embodiments of the present application can determine the timestamp of the current haptic signal data packet through the second offset timestamp of the current data packet and the first offset timestamp of the data unit where the data packet is located, so as to support random access during the transmission of haptic signals.

[0311] Referring to Table 12, byteAlignment() is a function for judging byte alignment. If the data included in the current data packet is an integer number of bytes, it returns 0; if the data included in the current data packet is less than an integer number of bytes, after padding the last byte, it returns 0.

[0312] Referring to Table 12, the syntax of AHASPacketPayload can be introduced in combination with Table 12.2 provided above.

[0313] Table 12.2

[0314]

[0315] According to different data packet types, during the execution of the syntax in Table 12.2, it will turn to the syntax of the payload of the corresponding type of data packet shown in one of Table 12.2-A to Table 12.2-G. Among them, the number of bits shown in Table 12.2-A to Table 12.2-G is exemplary data and can also be other values, which are not limited in the embodiments of the present application.

[0316] When the data packet type is the first type including haptic experience metadata, during the execution of the syntax in Table 12.2, it will turn to the syntax of the payload of this data packet shown in Table 12.2-A.

[0317] Table 12.2-A

[0318]

[0319]

[0320] When the type of the data packet is the first type including haptic experience metadata, the payload of this data packet includes one or more of the following fields shown in Table 12.2-A. Each field in Table 12.2-A can be represented by binary values. Specifically, the specific indication information of each field is as follows:

[0321] Version is used to indicate the standard version information followed by the haptic experience; the binary value used to represent Version can occupy 8 bits.

[0322] dateLength is used to indicate the string length of the creation date of the haptic experience; the binary value used to represent dateLength can occupy 8 bits.

[0323] Date is used to indicate the creation date of the haptic experience; the binary value used to represent Date can occupy dateLength × 8 bits.

[0324] descriptionLength is used to indicate the string length of the description of the haptic experience; the binary value used to represent Date can occupy 8 bits.

[0325] Description is used to indicate the description of the haptic experience; the binary value used to represent Description can occupy descriptionLength × 8 bits.

[0326] Timescale is used to indicate the time scale of the tactile experience. The value of this field represents the number of clock units contained within one second. The binary value used to represent Timescale can occupy 16 bits.

[0327] patternCount is used to indicate the number of patterns included in the tactile experience. The binary value used to represent patternCount can occupy 8 bits.

[0328] deviceCount is used to indicate the number of devices corresponding to the tactile experience. The binary value used to represent deviceCount can occupy 8 bits.

[0329] Regarding readDevice() in Table 12.2-A, its syntax can be referred to in Table 12.2-A1.

[0330] Table 12.2-A1

[0331]

[0332] The payload of the above data packet also includes one or more of the following fields for the j-th device as shown in Table 12.2-A1. Among them, each field in Table 12.2-A1 can be reflected by a binary value. Specifically, the specific indication information for each field is as follows:

[0333] deviceId is used to indicate the identifier of the j-th device. The binary value used to represent deviceId can occupy 8 bits.

[0334] nameLength is used to indicate the string length of the name of the j-th device. The binary value used to represent nameLength can occupy 8 bits.

[0335] deviceName is used to indicate the human-readable name of the j-th device. The binary value used to represent deviceName can occupy nameLength × 8 bits.

[0336] bodyPartFlag is an indicator for whether the device corresponds to a body part. The binary value used to represent bodyPartFlag can occupy 1 bit. For example, when the value is 1, it indicates the body part information corresponding to the device; when the value is 0, it does not indicate the body part information corresponding to the device.

[0337] bodyParts is used to indicate the body part corresponding to the device. The binary value used to represent bodyParts can occupy 8 bits.

[0338] In the case where the data packet type is the second type including haptic mode metadata, during the execution of the syntax in Table 12.2, the syntax of the payload of this data packet shown in Table 12.2-B will be turned to.

[0339] Table 12.2-B

[0340]

[0341]

[0342] In the case where the type of the data packet is the second type including haptic mode metadata, the payload of this data packet includes one or more of the following fields shown in Table 12.2-B. Each field in Table 12.2-B can be represented by binary values. Specifically, the specific indication information of each field is as follows:

[0343] Id is used to indicate the identifier of the haptic mode; the binary value used to represent Id can occupy 8 bits.

[0344] patternType is used to indicate the type of the haptic mode; the binary value used to represent patternType can occupy 8 bits. For example, the type of the data pattern is acceleration, which can be represented as "0000 0010". Exemplarily, the meaning of the binary value of this field can be as shown in Table 12.2-B1.

[0345] Table 12.2-B1

[0346] patternType value Meaning 0 Vibrotactile 1 Pressure 2 Acceleration 3 Velocity 4 Temperature 5 Electrotactile 6～255 Reserved

[0347] semanticisFlag is an indicator used to represent whether there is semantics in the haptic event corresponding to the haptic mode; the binary value used to represent semanticisFlag can occupy 1 bit. For example, when the value is 1, it means that there is semantics in the haptic event corresponding to the haptic mode, and when the value is 0, it means that there is no semantics in the haptic event corresponding to the haptic mode.

[0348] sementicsType is an indicator used to represent the specification followed by the semantics of the haptic event; the binary value used to represent sementicsType can occupy 8 bits. For example, when the value is 0, it means that the semantics of the haptic event follows the specification defined in this standard; when the value is 1, it means that the semantics of the haptic event follows the specification of an external standard; when the value is 2, it means that the semantics of the haptic event is user-defined.

[0349] referenceDeviceCount is used to represent the number of devices corresponding to the haptic mode; the binary value used to represent referenceDeviceCount can occupy 8 bits.

[0350] The referenceDeviceId is used to indicate the identifier of the device corresponding to the haptic mode; the binary value representing the referenceDeviceId can occupy 8 bits.

[0351] The channelCount is used to indicate the number of haptic channels included in the haptic mode; the binary value representing the channelCount can occupy 16 bits.

[0352] In the case where the data packet type is the third type including haptic channel metadata, during the execution of the syntax in Table 12.2, the syntax of the payload of this data packet shown in Table 12.2-C will be followed.

[0353] Table 12.2-C

[0354]

[0355]

[0356] In the case where the type of the data packet is the third type including haptic channel metadata, the payload of this data packet includes one or more of the following fields shown in Table 12.2-C. Each field in Table 12.2-C can be represented by a binary value. Specifically, the specific indication information for each field is as follows:

[0357] The Id is used to indicate the identifier of the haptic channel; the binary value representing the Id can occupy 16 bits.

[0358] The perceptionId is used to indicate the identifier of the haptic mode corresponding to the haptic channel; the binary value representing the perceptionId can occupy 8 bits.

[0359] The descriptionLength is used to indicate the string length of the haptic channel description; the binary value representing the descriptionLength can occupy 8 bits.

[0360] The Description is used to indicate the haptic channel description; the binary value representing the Description can occupy 8 × descriptionLength bits.

[0361] The Gain is used to indicate the gain of the haptic channel; the binary value representing the Gain can occupy 32 bits. Specifically, it is in units of 2-31. This gain Gain is applied to all events included in the current channel and is used to restore the normalized signal value to the original signal value.

[0362] The referenceDeviceCount is used to indicate the number of devices corresponding to the haptic channel; the binary value used to represent the referenceDeviceCount can occupy 8 bits.

[0363] The referenceDeviceId is used to indicate the identifier of the device corresponding to the haptic channel; the binary value used to represent the referenceDeviceId can occupy 8 bits.

[0364] The eventsCount is used to indicate the number of haptic events included in the haptic channel; the binary value used to represent the eventsCount can occupy 16 bits.

[0365] When the packet type is the seventh type containing haptic signals, during the execution of the syntax in Table 12.2, the syntax of the payload of this packet shown in Table 12.2-D will be followed.

[0366] Table 12.2-D

[0367]

[0368] When the type of the packet is the seventh type containing haptic signals, the payload of this packet includes one or more of the following fields shown in Table 12.2-D. Each field in Table 12.2-D can be represented by a binary value. Specifically, the specific indication information of each field is as follows:

[0369] The packetDependency is an indicator used to indicate whether the current packet depends on other packets of the same type during encoding; the binary value used to represent the packetDependency can occupy 1 bit; for example, when the value of this field is 0, it means that the current packet does not depend on other packets of the same type during encoding, and when the value of this field is 1, it means that the current packet depends on other packets of the same type during encoding.

[0370] The perceptionId is used to indicate the identifier of the haptic mode corresponding to the haptic event in the packet; the binary value used to represent the perceptionId can occupy 8 bits.

[0371] The channelId is used to indicate the identifier of the haptic channel corresponding to the haptic event in the packet; the binary value used to represent the channelId can occupy 8 bits.

[0372] The eventsCount is used to indicate the number of haptic events corresponding to the packet; the binary value used to represent the eventsCount can occupy 16 bits.

[0373] Regarding readEvent() in Table 12.2-D, its syntax can be referred to in Table 12.2-D1.

[0374] Table 12.2-D1

[0375]

[0376] The payload of the above data packet also includes one or more of the following fields shown in Table 12.2-D1 regarding the m-th tactile event. Among them, each field in Table 12.2-D1 can be represented by binary values. Specifically, the specific indication information of each field is as follows:

[0377] eventId is used to indicate the identifier of the tactile event; the binary value used to represent eventId can occupy 16 bits.

[0378] eventType is used to indicate the type of the tactile event; the binary value used to represent eventType can occupy 4 bits. For example, when the value of this field is 0, it indicates that the event is an instantaneous event; when the value of this field is 1, it indicates that the event is a continuous event; when the value of this field is 2, it indicates that the event is a reference event, etc.

[0379] eventSementicsFlag is an indicator value used to represent whether to indicate the semantics of the tactile event; the binary value used to represent eventSementicsFlag can occupy 1 bit. For example, when the value is 0, it does not indicate the semantics of the tactile event; when the value is 1, it indicates the semantics of the tactile event.

[0380] semanticKeywords represents the semantic information of the tactile event; the binary value used to represent semanticKeywords can occupy 12 bits. The semantics of the tactile event are the keyword of the event auxiliary information describing the intention of the content producer.

[0381] referEventId is used to indicate the identifier of the knowledge tactile event corresponding to the reference event; the binary value used to represent referEventId can occupy 16 bits.

[0382] relativePosition is used to indicate the time or space offset of the tactile event; the binary value used to represent relativePosition can occupy 32 bits.

[0383] Duration is used to indicate the duration of the tactile event, which exists when the event type is a continuous event; the binary value used to represent Duration can occupy 32 bits.

[0384] Amplitude is used to indicate the maximum amplitude value of the signal for a haptic event; the binary value used to represent Amplitude can occupy 8 bits.

[0385] baseFrequency is used to indicate the reference frequency of the signal for a haptic event; the binary value used to represent baseFrequency can occupy 16 bits.

[0386] numComponents is used to indicate the number of components of a haptic event. The components of a haptic event can be time-domain components or frequency-domain components of the haptic event; the binary value used to represent numComponents can occupy 16 bits.

[0387] Regarding readEventComponent() in Table 12.2-D1, its syntax can be referred to in Table 12.2-D2.

[0388] Table 12.2-D2

[0389]

[0390]

[0391] The payload of the above data packet further includes one or more of the following fields shown in Table 12.2-D2 for the kth haptic event component. Among them, each field in Table 12.2-D2 can be reflected by a binary value. Specifically, the specific indication information of each field is as follows:

[0392] refer_event_id is used to indicate the identifier of the haptic event to which the haptic event component belongs; the binary value used to represent refer_event_id can occupy 16 bits.

[0393] position_flag is an indicator used to represent whether the spatio-temporal domain offset is included in the haptic event component; the binary value used to represent position_flag can occupy 1 bit. For example, when position_flag has a value of 1, it indicates that the spatio-temporal domain offset is included in the haptic event component, and when the value is 0, it indicates that the spatio-temporal domain offset is not included in the haptic event component.

[0394] amplitude_flag is an indicator used to represent whether the amplitude value offset is included in the haptic event component; the binary value used to represent amplitude_flag can occupy 1 bit. For example, when amplitude_flag has a value of 1, it indicates that the amplitude value offset is included in the haptic event component, and when the value is 0, it indicates that the amplitude value offset is not included in the haptic event component.

[0395] The frequency_flag is an indicator used to represent whether a frequency offset is included in the haptic event component; the binary value used to represent the frequency_flag can occupy 1 bit. For example, when the value of the frequency_flag is 1, it indicates that a frequency offset is included in the haptic event component, and when the value is 0, it indicates that no frequency offset is included in the haptic event component.

[0396] The relative_position is used to represent the temporal or spatial offset of the haptic event component relative to the haptic event; the binary value used to represent the relative_position can occupy 16 bits.

[0397] The relative_amplitude is used to represent the amplitude value ratio of the haptic event component relative to the haptic event, with a value range of 0 to 215 in units of 2-15; the binary value used to represent the relative_amplitude can occupy 16 bits.

[0398] The relative_frequency is used to represent the frequency offset of the haptic event component relative to the haptic event; the binary value used to represent the relative_frequency can occupy 16 bits.

[0399] In the case where the data packet type is the fourth type including the knowledge haptic event, during the execution of the syntax in Table 12.2, the syntax of the payload of this data packet shown in Table 12.2-E will be followed.

[0400] Table 12.2-E

[0401]

[0402] In the case where the data packet type is the fourth type including the knowledge haptic event, the payload of this data packet includes one or more of the following fields shown in Table 12.2-E. Each field in Table 12.2-E can be represented by a binary value. Specifically, the specific indication information of each field is as follows:

[0403] The patternId is an identifier used to represent the haptic pattern corresponding to the knowledge haptic event; the binary value used to represent the patternId can occupy 8 bits.

[0404] The eventsCount is used to represent the number of knowledge haptic events corresponding to the data packet; the binary value used to represent the eventsCount can occupy 16 bits.

[0405] Regarding readLibraryEvent() in Table 12.2-E, its syntax can be referred to in Table 12.2-E1.

[0406] Table 12.2-E1

[0407]

[0408]

[0409] The payload of the above data packet also includes one or more of the following fields shown in Table 12.2-E1 regarding the hth knowledge haptic event. Among them, each field in Table 12.2-E1 can be represented by binary values. Specifically, the specific indication information for each field is as follows:

[0410] eventId is used to represent the identifier of the knowledge haptic event; the binary value used to represent eventId can occupy 16 bits.

[0411] eventType is used to represent the type of the knowledge haptic event; the binary value used to represent eventType can occupy 4 bits. For example, when the value of this field is 0, it indicates that the event is an instantaneous event, and when the value of this field is 1, it indicates that the event is a continuous event.

[0412] eventSementicsFlag is an indicator used to represent whether to indicate the semantics of the knowledge haptic event; the binary value used to represent eventSementicsFlag can occupy 1 bit. For example, when the value is 0, it does not indicate the semantics of the knowledge haptic event, and when the value is 1, it indicates the semantics of the knowledge haptic event.

[0413] semanticKeywords is used to represent the semantic information of the event; the binary value used to represent semanticKeywords can occupy 12 bits. The semantics of the knowledge haptic event are the keyword of the event auxiliary information that describes the intention of the content producer.

[0414] relativePosition is used to represent the time or space offset of the knowledge haptic event; the binary value used to represent relativePosition can occupy 32 bits.

[0415] Duration is used to represent the duration of the knowledge haptic event, and this attribute exists when the event type is a continuous event; the binary value used to represent Duration can occupy 32 bits.

[0416] Amplitude represents the maximum amplitude value of the signal for a knowledge tactile event; the binary value used to represent Amplitude can occupy 8 bits.

[0417] baseFrequency represents the reference frequency of the signal for a knowledge tactile event; the binary value used to represent baseFrequency can occupy 16 bits.

[0418] numComponents represents the number of components of a knowledge tactile event, and the components of a knowledge tactile event can be time-domain components or frequency-domain components of the knowledge tactile event; the binary value used to represent numComponents can occupy 16 bits.

[0419] Regarding readEventComponent() in Table 12.2-E1, its syntax can be referred to in Table 12.2-D2. It will not be elaborated here.

[0420] In the case where the data packet type is the sixth type containing silent data, during the execution of the syntax in Table 12.2, it will turn to the syntax of the payload of this data packet shown in Table 12.2-F.

[0421] Table 12.2-F

[0422]

[0423] In the case where the data packet type is the sixth type containing silent data, the payload of this data packet includes the fields shown in Table 12.2-F. The fields in Table 12.2-F can be reflected by binary values. Specifically, the specific indication information of this field is as follows:

[0424] silentDuration represents the length of the silent time, in time scale units; the binary value used to represent silentDuration can occupy 16 bits.

[0425] In the case where the data packet type is the fifth type containing a cyclic redundancy check code, during the execution of the syntax in Table 12.2, it will turn to the syntax of the payload of this data packet shown in Table 12.2-G.

[0426] Table 12.2-G

[0427]

[0428] In the case where the data packet type is the fifth type containing a cyclic redundancy check code, the payload of this data packet includes one or more of the following fields shown in Table 12.2-G. Each field in Table 12.2-G can be reflected by binary values. Specifically, the specific indication information of each field is as follows:

[0429] CRCType is used to represent the type of cyclic redundancy check code; the binary value used to represent CRCTypey can occupy 8 bits; for example, when the value is 0, it means the cyclic redundancy check code is a 16-bit check code, and the data packet protected by this cyclic redundancy check code is the data packet after the cyclic redundancy check code data packet; when the value is 1, it means the cyclic redundancy check code is a 32-bit check code, and the data packet protected by this cyclic redundancy check code is the data packet after the cyclic redundancy check code data packet; when the value is 2, it means the cyclic redundancy check code is a 16-bit check code, and the range of data packets protected by this cyclic redundancy check code is the protectedPacketsCount data packets after the cyclic redundancy check code data packet. When the value is 3, it means the cyclic redundancy check code is a 32-bit check code, and the range of data packets protected by this cyclic redundancy check code is the protectedPacketsCount data packets after the cyclic redundancy check code data packet.

[0430] CRC16Value is used to represent a 16-bit check code; the binary value used to represent CRC16Value can occupy 16 bits.

[0431] CRC32Value is used to represent a 32-bit check code; the binary value used to represent CRC32Value can occupy 32 bits.

[0432] protectedPacketsCount is used for the number of data packets protected by the cyclic redundancy check code; the binary value used to represent protectedPacketsCount can occupy 8 bits.

[0433] Based on the AVS tactile information representation data structure 400 shown as Figure 4 and the field extension of the tactile media bitstream provided by the embodiments of the present application (such as the embodiments provided in Table 7, Table 8, Table 11 - Table 12.2 above), it is possible to encode the exchange format of the target tactile information, and then obtain the binary bitstream corresponding to the above target tactile information.

[0434] Continue to refer to Figure 7 , in S720, the above bitstream is encapsulated to obtain a media file containing the above tactile information.

[0435] The server side can encapsulate the above bitstream. For example, the above bitstream is encapsulated based on ISOBMFF to obtain a media file containing the above tactile information.

[0436] Figure 7In the solution provided by the illustrated embodiment, the exchange format of the target tactile information is encoded to obtain a binary bitstream corresponding to the target tactile information. Further, the above binary bitstream is encapsulated to obtain a media file containing the tactile information. The embodiments of the present application provide a solution for encoding the exchange format of tactile information into a binary bitstream. Specifically, data is organized and divided in the above binary bitstream in the form of data units and data packets, and one data unit contains one or more data packets. The solution provided by the embodiments of the present application helps to save the occupied space of tactile media content and can reduce the bandwidth occupancy during the transmission of tactile media content. At the same time, it also supports random access of tactile signals during transmission.

[0437] As described above in conjunction with Figures 1 to 7 , the embodiments of the method for processing tactile information of the present application have been described in detail. Below, in conjunction with Figure 8 and Figure 9 , the device embodiments of the present application will be described in detail.

[0438] Figure 8 FIG. is a schematic structural diagram of a tactile information processing device 800 provided by an embodiment of the present application. The tactile information processing device 800 is configured in an electronic device that can decode a tactile media bitstream, such as Figure 2 the terminal 220 in Figure 8 . As shown in

[0439] , the tactile information processing device 800 includes: a de-encapsulation module 810 and a decoding module 820;

[0440] In some embodiments, based on the above solution, the syntax structure of the data unit includes: the type of the data unit, and different types correspond to different binary values; wherein, the types of the data unit include: initialization type, silent type, time domain type, and spatial domain type.

[0441] In some embodiments, based on the above solution, when the type of the data unit is a time domain type or a spatial domain type, the syntax structure of the data unit further includes: a first offset timestamp, where different timestamps correspond to different binary values; wherein, the first offset timestamp represents the minimum or maximum value among the timestamps of all data packets included in the data unit, and the timestamp of the i-th data unit is less than the timestamp of the (i + 1)-th data unit, where i is a positive integer.

[0442] In some embodiments, based on the above solution, when the type of the data unit is a silent type, the syntax structure of the data unit further includes: a first offset timestamp, where different timestamps correspond to different binary values; wherein, the first offset timestamp is greater than the maximum value among the timestamps of all data packets included in the target data unit, and the target data unit is the previous time domain type data unit or spatial domain type data unit of the silent type data unit.

[0443] In some embodiments, based on the above solution, the syntax structure of the data unit further includes: a decoding dependency indicator, where different decoding dependency indicators correspond to different binary values; wherein, when the decoding dependency indicator of the current data unit is a first target value, the current data unit does not depend on other data units of the same type during decoding; when the decoding dependency indicator of the current data unit is not the above first target value, the current data unit depends on other data units of the same type during decoding.

[0444] In some embodiments, based on the above solution, if the decoding dependency indicator included in the next time domain type or spatial domain type data unit of the initialization type data unit is a first target value, then the initialization type data unit is a decoding initial unit.

[0445] In some embodiments, based on the above solution, the syntax structure of the data unit further includes: a layer indicator and a layer identifier, where different layer indicators correspond to different binary values, and different layer identifiers correspond to different binary values; wherein, if the layer indicator is a second target value, the priority of the data unit is determined by the value of the layer identifier.

[0446] In some embodiments, based on the above solution, the syntax structure of the data packet includes: the type of the data packet, where different types correspond to different binary values;

[0447] Among them, the types of the data packet include:

[0448] A first type including haptic experience metadata,

[0449] A second type including haptic pattern metadata,

[0450] A third type including haptic channel metadata,

[0451] A fourth type including knowledge haptic events,

[0452] A fifth type including cyclic redundancy check codes,

[0453] A sixth type including silence information and

[0454] A seventh type including haptic signals.

[0455] In some embodiments, based on the above solution, when the type of the data packet is the seventh type, the syntax structure of the data packet further includes: a second offset timestamp, where different timestamps correspond to different binary values; wherein, the second offset timestamp represents the offset relative to the first timestamp of the data unit where the data packet is located, and the second offset timestamp represents the minimum or maximum value among the timestamps of all haptic events included in the data packet.

[0456] In some embodiments, based on the above solution, the syntax structure of the data packet further includes: the number of bytes of the payload, where different numbers of bytes correspond to different binary values; wherein,

[0457] If the type of the data packet is the first type, the payload of the data packet includes one or more of the following information represented by binary values: the standard version followed by the haptic experience, the string length of the creation date of the haptic experience, the creation date of the haptic experience, the string length of the haptic experience description, the haptic experience description, the time scale of the haptic experience, the number of patterns included in the haptic experience, and the number of devices corresponding to the haptic experience; and, if the type of the data packet is the first type, the payload of the data packet further includes one or more of the following information about the j-th device: the device identifier, the string length of the device name, the human-readable device name, and an indicator of whether the device corresponds to a body part, where j is a positive integer not greater than the number of devices corresponding to the haptic experience;

[0458] If the type of the data packet is the second type, the payload of the data packet includes one or more of the following information represented by binary values: the identifier of the haptic pattern, the type of the haptic pattern, an indicator of whether there is semantics for the haptic events corresponding to the haptic pattern, an indicator of the specification followed by the haptic event semantics, the number of devices corresponding to the haptic pattern, the device identifier corresponding to the haptic pattern, and the number of haptic channels included in the haptic pattern;

[0459] If the type of the data packet is the third type, the payload of the data packet includes one or more of the following information represented by binary values: tactile channel identifier, tactile mode identifier corresponding to the tactile channel, string length of the tactile channel description, tactile channel description, gain of the tactile channel, number of devices corresponding to the tactile channel, device identifier of each device corresponding to the tactile channel, and number of tactile events included in the tactile channel;

[0460] If the type of the data packet is the seventh type, the payload of the data packet includes one or more of the following information represented by binary values: indicator of whether the current data packet depends on other data packets of the same type during decoding, identifier of the tactile mode corresponding to the tactile event in the current data packet, identifier of the tactile channel corresponding to the tactile event in the current data packet, number of tactile events in the data packet; and, if the type of the data packet is the seventh type, the payload of the data packet further includes one or more of the following information about the m-th tactile event: tactile event identifier, type of the tactile event, indicator of whether to indicate the semantics of the tactile event, semantic information of the tactile event, knowledge tactile event identifier corresponding to the reference event, time or space offset of the tactile event, duration of the tactile event, maximum amplitude value of the signal of the tactile event, reference frequency of the signal of the tactile event, and number of tactile event components, where m is a positive integer not greater than the number of tactile events in the data packet; and, if the type of the data packet is the seventh type, the payload of the data packet further includes one or more of the following information about the k-th tactile event component: identifier of the tactile event to which the tactile event component belongs, indicator of whether the tactile event component includes a spatio-temporal domain offset, indicator of whether the tactile event component includes an amplitude value offset, indicator of whether the tactile event component includes a frequency offset, time or space offset of the tactile event component relative to the tactile event, and amplitude value ratio of the tactile event component relative to the tactile event, where k is a positive integer not greater than the number of tactile event components in the tactile event;

[0461] If the type of the data packet is the fourth type, the payload of the data packet includes one or more of the following information represented by binary values: the identifier of the tactile pattern corresponding to the knowledge tactile event, the number of knowledge tactile events corresponding to the data packet; and, if the type of the data packet is the fourth type, the payload of the data packet further includes one or more of the following information about the h-th knowledge tactile event: the identifier of the knowledge tactile event, the type of the knowledge tactile event, an indicator indicating whether to indicate the semantics of the knowledge tactile event, the semantic information of the knowledge tactile event, the time or space offset of the knowledge tactile event, the duration of the knowledge tactile event, the maximum amplitude value of the signal of the knowledge tactile event, the reference frequency of the signal of the knowledge tactile event, and the number of tactile event components, where h is a positive integer not greater than the number of knowledge tactile events; and, if the type of the data packet is the fourth type, the payload of the data packet further includes one or more of the following information about the k-th tactile event component: the identifier of the knowledge tactile event to which the tactile event component belongs, an indicator indicating whether the tactile event component includes a spatio-temporal domain offset, an indicator indicating whether the tactile event component includes an amplitude value offset, an indicator indicating whether the tactile event component includes a frequency offset, the time or space offset of the tactile event component relative to the knowledge tactile event, and the amplitude value ratio of the tactile event component relative to the knowledge tactile event, where k is a positive integer not greater than the number of tactile event components in the knowledge tactile event;

[0462] If the type of the data packet is the sixth type, the payload of the data packet includes the following information represented by binary values: the length of the silent time;

[0463] If the type of the data packet is the fifth type, the payload of the data packet includes one or more of the following information represented by binary values: the type of cyclic redundancy check code, different types of cyclic redundancy check codes, and the number of data packets protected by the cyclic redundancy check code.

[0464] It should be understood that the device embodiments and the embodiments of the method for processing tactile information can correspond to each other, and similar descriptions can refer to the method embodiments. To avoid repetition, they are not elaborated here. Specifically, Figure 8 the shown tactile information processing device can execute the embodiments of the method for processing tactile information as Figure 5 shown, and the foregoing and other operations and / or functions of each module in the tactile information processing device are respectively for implementing the embodiments of the method for processing tactile information as Figure 5 shown. For the sake of brevity, they are not elaborated here.

[0465] Figure 9It is a schematic structural diagram of a tactile information processing device 900 provided by an embodiment of the present application. The tactile information processing device 900 is configured in an electronic device that can obtain a tactile media bitstream in an encoded exchange format, such as Figure 2 the server 210 in Figure 9 As shown, the tactile information processing device 900 includes: an encoding module 910 and a packaging module 920;

[0466] Among them, the above-mentioned encoding module 910 is used for: encoding the exchange format of the target tactile information to obtain a binary bitstream corresponding to the target tactile information, the bitstream includes: one or more data units regarding the target tactile information, and the data unit includes one or more data packets; and, the above-mentioned packaging module 920 is used for: packaging the bitstream to obtain a media file containing the tactile information.

[0467] In an exemplary embodiment, based on the above solution, the syntax structure of the data unit includes: the type of the data unit, and different types correspond to different binary values; among them, the type of the data unit includes: initialization type, silent type, time domain type, and spatial domain type.

[0468] In an exemplary embodiment, based on the above solution,

[0469] When the type of the data unit is the time domain type or the spatial domain type, the syntax structure of the data unit further includes: a first offset timestamp, and different timestamps correspond to different binary values; among them, the first offset timestamp represents the minimum or maximum value of the timestamps of all data packets included in the data unit, and the timestamp of the i-th data unit is less than the timestamp of the (i + 1)-th data unit, where i is a positive integer;

[0470] When the type of the data unit is the silent type, the syntax structure of the data unit further includes: a first offset timestamp, and different timestamps correspond to different binary values; among them, the first offset timestamp is greater than the maximum value of the timestamps of all data packets included in the target data unit, and the target data unit is the previous time domain type data unit or spatial domain type data unit of the silent type data unit;

[0471] The syntax structure of the data unit further includes: an encoding dependency indicator, where different encoding dependency indicators correspond to different binary values; when the encoding dependency indicator of the current data unit is a first target value, the current data unit does not depend on other data units of the same type during encoding; when the encoding dependency indicator of the current data unit is not the above first target value, the current data unit depends on other data units of the same type during encoding; if the encoding dependency indicator included in the next time domain type or spatial domain type data unit of the initialization type data unit is the first target value, then the initialization type data unit is an encoding initial unit;

[0472] The syntax structure of the data unit further includes: a layer indicator and a layer identifier, where different layer indicators correspond to different binary values, and different layer identifiers correspond to different binary values; among them, if the layer indicator is a second target value, the priority of the data unit is determined by the value of the layer identifier.

[0473] In an exemplary embodiment, based on the above solution, the syntax structure of the data packet includes: the type of the data packet, where different types correspond to different binary values;

[0474] Among them, the types of the data packet include:

[0475] A first type including haptic experience metadata,

[0476] A second type including haptic mode metadata,

[0477] A third type including haptic channel metadata,

[0478] A fourth type including knowledge haptic events,

[0479] A fifth type including a cyclic redundancy check code,

[0480] A sixth type including silence information, and

[0481] A seventh type including haptic signals.

[0482] In an exemplary embodiment, based on the above solution, when the type of the data packet is the seventh type, the syntax structure of the data packet further includes: a second offset timestamp, where different timestamps correspond to different binary values; among them, the second offset timestamp represents the offset relative to the first timestamp of the data unit where the data packet is located, and the second offset timestamp represents the minimum or maximum value among the timestamps of all haptic events included in the data packet.

[0483] In an exemplary embodiment, based on the above solution, the syntax structure of the data packet further includes: the number of bytes of the payload, where different numbers of bytes correspond to different binary values; wherein,

[0484] If the type of the data packet is the first type, the payload of the data packet includes one or more of the following information represented by binary values: the standard version followed by the tactile experience, the string length of the creation date of the tactile experience, the creation date of the tactile experience, the string length of the description of the tactile experience, the description of the tactile experience, the time scale of the tactile experience, the number of patterns included in the tactile experience, and the number of devices corresponding to the tactile experience; and, if the type of the data packet is the first type, the payload of the data packet further includes one or more of the following information about the jth device: the device identifier, the string length of the device name, the human-readable device name, an indicator of whether the device corresponds to a body part, where j is a positive integer not greater than the number of devices corresponding to the tactile experience;

[0485] If the type of the data packet is the second type, the payload of the data packet includes one or more of the following information represented by binary values: the identifier of the tactile pattern, the type of the tactile pattern, an indicator of whether there is semantics for the tactile event corresponding to the tactile pattern, an indicator of the specification followed by the tactile event semantics, the number of devices corresponding to the tactile pattern, the device identifier corresponding to the tactile pattern, and the number of tactile channels included in the tactile pattern;

[0486] If the type of the data packet is the third type, the payload of the data packet includes one or more of the following information represented by binary values: the tactile channel identifier, the tactile pattern identifier corresponding to the tactile channel, the string length of the description of the tactile channel, the description of the tactile channel, the gain of the tactile channel, the number of devices corresponding to the tactile channel, the device identifier of each device corresponding to the tactile channel, and the number of tactile events included in the tactile channel;

[0487] If the type of the data packet is the seventh type, the payload of the data packet includes one or more of the following information represented by binary values: an indicator of whether the current data packet depends on other data packets of the same type during encoding, an identifier of the haptic pattern corresponding to the haptic event in the current data packet, an identifier of the haptic channel corresponding to the haptic event in the current data packet, and the number of haptic events in the data packet; and, if the type of the data packet is the seventh type, the payload of the data packet further includes one or more of the following information about the m-th haptic event: a haptic event identifier, a type of the haptic event, an indicator of whether to indicate the semantics of the haptic event, semantic information of the haptic event, a knowledge haptic event identifier corresponding to the reference event, a temporal or spatial offset of the haptic event, a duration of the haptic event, a maximum amplitude value of the signal of the haptic event, a reference frequency of the signal of the haptic event, and the number of haptic event components, where m is a positive integer not greater than the number of haptic events in the data packet; and, if the type of the data packet is the seventh type, the payload of the data packet further includes one or more of the following information about the k-th haptic event component: an identifier of the haptic event to which the haptic event component belongs, an indicator of whether the haptic event component includes a spatio-temporal domain offset, an indicator of whether the haptic event component includes an amplitude value offset, an indicator of whether the haptic event component includes a frequency offset, a temporal or spatial offset of the haptic event component relative to the haptic event, and an amplitude value ratio of the haptic event component relative to the haptic event, where k is a positive integer not greater than the number of haptic event components in the haptic event;

[0488] If the type of the data packet is the fourth type, the payload of the data packet includes one or more of the following information represented by binary values: the identifier of the tactile pattern corresponding to the knowledge tactile event, the number of knowledge tactile events corresponding to the data packet; and, if the type of the data packet is the fourth type, the payload of the data packet further includes one or more of the following information about the h-th knowledge tactile event: the identifier of the knowledge tactile event, the type of the knowledge tactile event, the indicator indicating whether to indicate the semantics of the knowledge tactile event, the semantic information of the knowledge tactile event, the time or space offset of the knowledge tactile event, the duration of the knowledge tactile event, the maximum amplitude value of the signal of the knowledge tactile event, the reference frequency of the signal of the knowledge tactile event, and the number of tactile event components, where h is a positive integer not greater than the number of knowledge tactile events; and, if the type of the data packet is the fourth type, the payload of the data packet further includes one or more of the following information about the k-th tactile event component: the identifier of the knowledge tactile event to which the tactile event component belongs, the indicator indicating whether the tactile event component includes a spatio-temporal domain offset, the indicator indicating whether the tactile event component includes an amplitude value offset, the indicator indicating whether the tactile event component includes a frequency offset, the time or space offset of the tactile event component relative to the knowledge tactile event, and the amplitude value ratio of the tactile event component relative to the knowledge tactile event, where k is a positive integer not greater than the number of tactile event components in the knowledge tactile event;

[0489] If the type of the data packet is the sixth type, the payload of the data packet includes the following information represented by binary values: the length of the silent time;

[0490] If the type of the data packet is the fifth type, the payload of the data packet includes one or more of the following information represented by binary values: the type of cyclic redundancy check code, different types of cyclic redundancy check codes, the number of data packets protected by the cyclic redundancy check code.

[0491] It should be understood that the device embodiments and the embodiments of the method for processing tactile information can correspond to each other, and similar descriptions can refer to the method embodiments. To avoid repetition, it will not be elaborated here. Specifically, Figure 9 the shown tactile information processing device can execute the embodiments of the method for processing tactile information as Figure 7 shown, and the foregoing and other operations and / or functions of each module in the tactile information processing device are respectively for implementing the embodiments of the method for processing tactile information as Figure 7 shown. For the sake of brevity, it will not be elaborated here.

[0492] In the foregoing, the apparatus according to the embodiments of the present application has been described from the perspective of functional modules. It should be understood that the functional modules can be implemented in the form of hardware, or can be implemented by instructions in software form, or can be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiments in the present application can be completed by the integrated logic circuit in the hardware in the processor and / or instructions in software form. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware decoding processor, or can be executed and completed by a combination of the hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps in the above method embodiments.

[0493] Figure 10 FIG. 4 is a schematic structural diagram of an electronic device 1000 provided by an embodiment of the present application. Figure 10 The electronic device 1000 can be used to execute the above-mentioned method for processing tactile information as provided in Figure 5 the embodiment, or the electronic device 1000 can be used to execute the above-mentioned method for processing tactile information as provided in Figure 7 the embodiment.

[0494] As Figure 10 shown, the electronic device 1000 may include:

[0495] A memory 1010 and a processor 1020. The memory 1010 is used to store a computer program 1030 and transmit the program code 1030 to the processor 1020. In other words, the processor 1020 can call and run the computer program 1030 from the memory 1010 to implement the method for processing tactile information in the embodiments of the present application.

[0496] For example, the processor 1020 can be used to execute the steps in the above-mentioned method for processing tactile information according to the instructions in the computer program 1030.

[0497] In some embodiments of the present application, the processor 1020 may include but is not limited to:

[0498] General-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like.

[0499] In some embodiments of the present application, the memory 1010 includes, but is not limited to:

[0500] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synch link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0501] In some embodiments of the present application, the computer program 1030 can be divided into one or more modules, and the one or more modules are stored in the memory 1010 and executed by the processor 1020 to complete the method for processing tactile information of the present application. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 1030 in the electronic device.

[0502] As Figure 10 shown, the electronic device 1000 may further include:

[0503] A transceiver 1040, which can be connected to the processor 1020 or the memory 1010.

[0504] Among them, the processor 1020 can control the transceiver 1040 to communicate with other devices. Specifically, it can send information or data to other devices, or receive information or data sent by other devices. The transceiver 1040 can include a transmitter and a receiver. The transceiver 1040 can further include an antenna, and the number of antennas can be one or more.

[0505] It should be understood that each component in the electronic device 1000 is connected through a bus system. Among them, the bus system includes not only a data bus, but also a power bus, a control bus, and a status signal bus.

[0506] According to one aspect of the present application, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a computer, the computer can execute the method of the above method embodiment. Or, the embodiment of the present application further provides a computer program product including instructions. When the instructions are executed by a computer, the computer executes the method of the above method embodiment.

[0507] According to another aspect of the present application, a computer program product or a computer program is provided. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method of the above method embodiment.

[0508] In other words, when implemented using software, it can be implemented in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of this application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a digital video disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0509] Those of ordinary skill in the art will realize that the modules and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0510] In several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there can be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of devices or modules can be in electrical, mechanical, or other forms.

[0511] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. For example, in each embodiment of the present application, the functional modules can be integrated into one processing module, or each module can exist physically alone, or two or more modules can be integrated into one module.

[0512] The above content is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for processing tactile information, characterized in that, the method includes: Unpacking a media file containing target tactile information to obtain a binary bitstream corresponding to the target tactile information, the bitstream including: one or more data units regarding the target tactile information, and the data unit including one or more data packets; Decoding the binary bitstream corresponding to the target tactile information to obtain an exchange format of the target tactile information, wherein the exchange format of the target tactile information is used for rendering and presenting the target tactile information.

2. The method according to claim 1, characterized in that, the syntax structure of the data unit includes: the type of the data unit, and different types correspond to different binary values; wherein, the types of the data unit include: initialization type, silence type, time domain type, and spatial domain type.

3. The method according to claim 2, characterized in that, when the type of the data unit is a time domain type or a spatial domain type, the syntax structure of the data unit further includes: a first offset timestamp, and different timestamps correspond to different binary values; wherein, the first offset timestamp represents the minimum or maximum value among the timestamps of all data packets included in the data unit, and the timestamp of the i-th data unit is less than the timestamp of the (i + 1)-th data unit, and i is a positive integer.

4. The method according to claim 2, characterized in that, when the type of the data unit is a silence type, the syntax structure of the data unit further includes: a first offset timestamp, and different timestamps correspond to different binary values; wherein, the first offset timestamp is greater than the maximum value among the timestamps of all data packets included in a target data unit, and the target data unit is a time domain type data unit or a spatial domain type data unit before the silence type data unit.

5. The method according to claim 2, characterized in that, the syntax structure of the data unit further includes: a decoding dependency indicator, and different decoding dependency indicators correspond to different binary values; wherein, when the decoding dependency indicator of the current data unit is a first target value, the current data unit does not depend on other data units of the same type during decoding; when the decoding dependency indicator of the current data unit is not the first target value, the current data unit depends on other data units of the same type during decoding.

6. The method according to claim 5, characterized in that, if the decoding dependency indicator included in the next time domain type or spatial domain type data unit of the initialization type data unit is a first target value, then the initialization type data unit is a decoding initial unit.

7. The method according to claim 2, characterized in that, the syntax structure of the data unit further includes: a hierarchy indicator and a hierarchy identifier, and different hierarchy indicators correspond to different binary values, and different hierarchy identifiers correspond to different binary values; Wherein, if the hierarchical indicator is the second target value, the priority of the data unit is determined by the value of the hierarchical identifier.

8. The method according to claim 1, wherein, the syntax structure of the data packet includes: the type of the data packet, and different types correspond to different binary values; wherein, the types of the data packet include: a first type including haptic experience metadata, a second type including haptic pattern metadata, a third type including haptic channel metadata, a fourth type including knowledge haptic events, a fifth type including cyclic redundancy check codes, a sixth type including silence information, and a seventh type including haptic signals.

9. The method according to claim 8, wherein, when the type of the data packet is the seventh type, the syntax structure of the data packet further includes: a second offset timestamp, and different timestamps correspond to different binary values; wherein, the second offset timestamp represents the offset relative to the first timestamp of the data unit where the data packet is located, and the second offset timestamp represents the minimum or maximum value among the timestamps of all haptic events included in the data packet.

10. The method according to claim 8, wherein, the syntax structure of the data packet further includes: the number of bytes of the payload, and different numbers of bytes correspond to different binary values; wherein, if the type of the data packet is the first type, the payload of the data packet includes one or more of the following information represented by binary values: the standard version followed by the haptic experience, the string length of the creation date of the haptic experience, the creation date of the haptic experience, the string length of the haptic experience description, the haptic experience description, the time scale of the haptic experience, the number of patterns included in the haptic experience, and the number of devices corresponding to the haptic experience; and, if the type of the data packet is the first type, the payload of the data packet further includes one or more of the following information about the jth device: the device identifier, the string length of the device name, the human-readable device name, the indicator of whether the device corresponds to a body part, where j is a positive integer not greater than the number of devices corresponding to the haptic experience; if the type of the data packet is the second type, the payload of the data packet includes one or more of the following information represented by binary values: the identifier of the haptic pattern, the type of the haptic pattern, the indicator of whether there is semantics for the haptic event corresponding to the haptic pattern, the indicator of the specification followed by the haptic event semantics, the number of devices corresponding to the haptic pattern, the device identifier corresponding to the haptic pattern, and the number of haptic channels included in the haptic pattern; if the type of the data packet is the third type, the payload of the data packet includes one or more of the following information represented by binary values: the haptic channel identifier, the haptic pattern identifier corresponding to the haptic channel, the string length of the haptic channel description, the haptic channel description, the gain of the haptic channel, the number of devices corresponding to the haptic channel, the device identifier of each device corresponding to the haptic channel, and the number of haptic events included in the haptic channel; If the type of the data packet is the seventh type, the payload of the data packet includes one or more of the following information represented by binary values: an indicator of whether the current data packet depends on other data packets of the same type when decoded, an identifier of the haptic pattern corresponding to the haptic event in the current data packet, an identifier of the haptic channel corresponding to the haptic event in the current data packet, the number of haptic events in the data packet; and, if the type of the data packet is the seventh type, the payload of the data packet further includes one or more of the following information about the m-th haptic event: a haptic event identifier, the type of the haptic event, an indicator of whether to indicate the semantics of the haptic event, the semantic information of the haptic event, the knowledge haptic event identifier corresponding to the reference event, the temporal or spatial offset of the haptic event, the duration of the haptic event, the maximum amplitude value of the signal of the haptic event, the reference frequency of the signal of the haptic event, and the number of haptic event components, where m is a positive integer not greater than the number of haptic events in the data packet; and, if the type of the data packet is the seventh type, the payload of the data packet further includes one or more of the following information about the k-th haptic event component: the identifier of the haptic event to which the haptic event component belongs, an indicator of whether the haptic event component includes a spatio-temporal domain offset, an indicator of whether the haptic event component includes an amplitude offset, an indicator of whether the haptic event component includes a frequency offset, the temporal or spatial offset of the haptic event component relative to the haptic event, and the amplitude ratio of the haptic event component relative to the haptic event, where k is a positive integer not greater than the number of haptic event components in the haptic event; If the type of the data packet is the fourth type, the payload of the data packet includes one or more of the following information represented by binary values: the identifier of the tactile pattern corresponding to the knowledge tactile event, the number of knowledge tactile events corresponding to the data packet; and, if the type of the data packet is the fourth type, the payload of the data packet further includes one or more of the following information about the h-th knowledge tactile event: the identifier of the knowledge tactile event, the type of the knowledge tactile event, an indicator indicating whether to indicate the semantics of the knowledge tactile event, the semantic information of the knowledge tactile event, the time or space offset of the knowledge tactile event, the duration of the knowledge tactile event, the maximum amplitude value of the signal of the knowledge tactile event, the reference frequency of the signal of the knowledge tactile event, and the number of tactile event components, where h is a positive integer not greater than the number of knowledge tactile events; and, if the type of the data packet is the fourth type, the payload of the data packet further includes one or more of the following information about the k-th tactile event component: the identifier of the knowledge tactile event to which the tactile event component belongs, an indicator indicating whether the tactile event component includes a spatio-temporal domain offset, an indicator indicating whether the tactile event component includes an amplitude value offset, an indicator indicating whether the tactile event component includes a frequency offset, the time or space offset of the tactile event component relative to the knowledge tactile event, and the amplitude value ratio of the tactile event component relative to the knowledge tactile event, where k is a positive integer not greater than the number of tactile event components in the knowledge tactile event; If the type of the data packet is the sixth type, the payload of the data packet includes the following information represented by binary values: the length of the silent time; If the type of the data packet is the fifth type, the payload of the data packet includes one or more of the following information represented by binary values: the type of the cyclic redundancy check code, different types of cyclic redundancy check codes, the number of data packets protected by the cyclic redundancy check code.

11. A method for processing tactile information, characterized in that, the method includes: encoding the exchange format of the target tactile information to obtain a binary bitstream corresponding to the target tactile information, the bitstream including: one or more data units about the target tactile information, and the data unit includes one or more data packets; encapsulating the bitstream to obtain a media file containing the tactile information.

12. The method according to claim 11, characterized in that, the syntax structure of the data unit includes: the type of the data unit, and different types correspond to different binary values; wherein, the type of the data unit includes: initialization type, silent type, time domain type, and space domain type.

13. The method according to claim 12, characterized in that, When the type of the data unit is a time domain type or a spatial domain type, the syntax structure of the data unit further includes: a first offset timestamp, where different timestamps correspond to different binary values; wherein, the first offset timestamp represents the minimum or maximum value among the timestamps of all data packets included in the data unit, and the timestamp of the i-th data unit is less than the timestamp of the (i + 1)-th data unit, and i is a positive integer; When the type of the data unit is a silent type, the syntax structure of the data unit further includes: a first offset timestamp, where different timestamps correspond to different binary values; wherein, the first offset timestamp is greater than the maximum value among the timestamps of all data packets included in the target data unit, and the target data unit is the previous time domain type data unit or spatial domain type data unit of the silent type data unit; The syntax structure of the data unit further includes: an encoding dependency indicator, where different encoding dependency indicators correspond to different binary values; when the encoding dependency indicator of the current data unit is the first target value, the current data unit does not depend on other data units of the same type during encoding; when the encoding dependency indicator of the current data unit is not the first target value, the current data unit depends on other data units of the same type during encoding; if the encoding dependency indicator included in the next time domain type or spatial domain type data unit of the initialization type data unit is the first target value, then the initialization type data unit is the encoding initial unit; The syntax structure of the data unit further includes: a layer indicator and a layer identifier, where different layer indicators correspond to different binary values, and different layer identifiers correspond to different binary values; wherein, if the layer indicator is the second target value, the priority of the data unit is determined by the value of the layer identifier.

14. The method according to claim 11, wherein, the syntax structure of the data packet includes: the type of the data packet, where different types correspond to different binary values; wherein, the type of the data packet includes: a first type including haptic experience metadata, a second type including haptic mode metadata, a third type including haptic channel metadata, a fourth type including knowledge haptic events, a fifth type including a cyclic redundancy check code, a sixth type including silent information, and a seventh type including haptic signals.

15. The method according to claim 14, wherein, when the type of the data packet is the seventh type, the syntax structure of the data packet further includes: a second offset timestamp, where different timestamps correspond to different binary values; wherein, the second offset timestamp represents the offset relative to the first timestamp of the data unit where the data packet is located, and the second offset timestamp represents the minimum or maximum value among the timestamps of all haptic events included in the data packet.

16. The method according to claim 14, wherein, The syntax structure of the data packet further includes: the number of bytes of the payload, and different numbers of bytes correspond to different binary values; wherein, if the type of the data packet is the first type, the payload of the data packet includes one or more of the following information represented by binary values: the standard version followed by the tactile experience, the string length of the creation date of the tactile experience, the creation date of the tactile experience, the string length of the tactile experience description, the tactile experience description, the time scale of the tactile experience, the number of patterns included in the tactile experience, and the number of devices corresponding to the tactile experience; and, if the type of the data packet is the first type, the payload of the data packet further includes one or more of the following information about the jth device: the device identifier, the string length of the device name, the human-readable device name, and an indicator of whether the device corresponds to a body part, where j is a positive integer not greater than the number of devices corresponding to the tactile experience; if the type of the data packet is the second type, the payload of the data packet includes one or more of the following information represented by binary values: the identifier of the tactile pattern, the type of the tactile pattern, an indicator of whether there is semantics for the tactile event corresponding to the tactile pattern, an indicator of the specification followed by the tactile event semantics, the number of devices corresponding to the tactile pattern, the device identifier corresponding to the tactile pattern, and the number of tactile channels included in the tactile pattern; if the type of the data packet is the third type, the payload of the data packet includes one or more of the following information represented by binary values: the tactile channel identifier, the tactile pattern identifier corresponding to the tactile channel, the string length of the tactile channel description, the tactile channel description, the gain of the tactile channel, the number of devices corresponding to the tactile channel, the device identifier of each device corresponding to the tactile channel, and the number of tactile events included in the tactile channel; If the type of the data packet is the seventh type, the payload of the data packet includes one or more of the following information represented by binary values: an indicator of whether the current data packet depends on other data packets of the same type during encoding, an identifier of the haptic pattern corresponding to the haptic event in the current data packet, an identifier of the haptic channel corresponding to the haptic event in the current data packet, and the number of haptic events in the data packet; and, if the type of the data packet is the seventh type, the payload of the data packet further includes one or more of the following information about the m-th haptic event: a haptic event identifier, a type of the haptic event, an indicator of whether to indicate the semantics of the haptic event, semantic information of the haptic event, a knowledge haptic event identifier corresponding to the reference event, a time or space offset of the haptic event, a duration of the haptic event, a maximum amplitude value of the signal of the haptic event, a reference frequency of the signal of the haptic event, and the number of haptic event components, where m is a positive integer not greater than the number of haptic events in the data packet; and, if the type of the data packet is the seventh type, the payload of the data packet further includes one or more of the following information about the k-th haptic event component: an identifier of the haptic event to which the haptic event component belongs, an indicator of whether the haptic event component includes a spatio-temporal domain offset, an indicator of whether the haptic event component includes an amplitude value offset, an indicator of whether the haptic event component includes a frequency offset, a time or space offset of the haptic event component relative to the haptic event, and an amplitude value ratio of the haptic event component relative to the haptic event, where k is a positive integer not greater than the number of haptic event components in the haptic event; If the type of the data packet is the fourth type, the payload of the data packet includes one or more of the following information represented by binary values: the identifier of the haptic pattern corresponding to the knowledge haptic event, the number of knowledge haptic events corresponding to the data packet; and, if the type of the data packet is the fourth type, the payload of the data packet further includes one or more of the following information about the h-th knowledge haptic event: the identifier of the knowledge haptic event, the type of the knowledge haptic event, an indicator indicating whether to indicate the semantics of the knowledge haptic event, the semantic information of the knowledge haptic event, the time or space offset of the knowledge haptic event, the duration of the knowledge haptic event, the maximum amplitude value of the signal of the knowledge haptic event, the reference frequency of the signal of the knowledge haptic event, and the number of haptic event components, where h is a positive integer not greater than the number of the knowledge haptic events; and, if the type of the data packet is the fourth type, the payload of the data packet further includes one or more of the following information about the k-th haptic event component: the identifier of the knowledge haptic event to which the haptic event component belongs, an indicator indicating whether the haptic event component includes a spatio-temporal domain offset, an indicator indicating whether the haptic event component includes an amplitude value offset, an indicator indicating whether the haptic event component includes a frequency offset, the time or space offset of the haptic event component relative to the knowledge haptic event, and the amplitude value ratio of the haptic event component relative to the knowledge haptic event, where k is a positive integer not greater than the number of haptic event components in the knowledge haptic event; If the type of the data packet is the sixth type, the payload of the data packet includes the following information represented by binary values: the length of the silent time; If the type of the data packet is the fifth type, the payload of the data packet includes one or more of the following information represented by binary values: the type of the cyclic redundancy check code, different types of cyclic redundancy check codes, the number of data packets protected by the cyclic redundancy check code.

17. A processing device for haptic information, characterized in that, the device includes: a de-encapsulation module, configured to de-encapsulate a media file containing target haptic information to obtain a binary bitstream corresponding to the target haptic information, where the bitstream includes: one or more data units about the target haptic information, and the data unit includes one or more data packets; a decoding module, configured to decode the binary bitstream corresponding to the target haptic information to obtain an exchange format of the target haptic information, where the exchange format of the target haptic information is used to render and present the target haptic information.

18. A processing device for haptic information, characterized in that, the device includes: an encoding module, configured to encode an exchange format of target haptic information to obtain a binary bitstream corresponding to the target haptic information, where the bitstream includes: one or more data units about the target haptic information, and the data unit includes one or more data packets; an encapsulation module, configured to encapsulate the bitstream to obtain a media file containing the haptic information.

19. An electronic device, including a processor and a memory; The memory is used for storing a computer program; The processor is used for executing the computer program to implement the method for processing tactile information according to any one of claims 1 to 16 above.

20. A computer-readable storage medium, characterized in that, it is used for storing a computer program; the computer program causes a computer to execute the method for processing tactile information according to any one of claims 1 to 16 above.

Images