Data processing method and apparatus for haptic media, and computer device and storage medium

Abstract

Provided in the present application are a data processing method and apparatus for haptic media, and a computer device and a storage medium. The data processing method for haptic media comprises: acquiring representation information of a transient haptic signal in haptic media, wherein the representation information of the transient haptic signal is used for indicating the duration of the transient haptic signal; and presenting the transient haptic signal in the haptic media according to the representation information of the transient haptic signal. Using the present application can accurately present the transient haptic signal in the haptic media, improving the consumption experience of the haptic media.

Description

Data processing method and device for tactile media, computer equipment, and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on November 29, 2023, with application number 202311622744.2 and application name “Data processing method, device, computer equipment, and storage medium for tactile media”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, in particular to the field of immersive media technology, and specifically to a tactile media data processing method, a tactile media data processing device, a computer device, and a computer-readable storage medium. Background Art

[0003] With the continuous development of immersive media, in addition to the traditional visual and auditory presentations, immersive media also includes new presentation methods such as touch, such as vibration touch, electric touch, and pressure touch, etc. Immersive media presented in the form of touch can be called tactile media, and tactile media can include one or more tactile signals. At present, the instantaneous tactile signals in tactile media are processed by the media presentation end. This makes the presentation of instantaneous tactile signals inaccurate, resulting in differences in the presentation of the same instantaneous tactile signal on different media presentation ends, further leading to a low consumer experience of tactile media. Therefore, how to accurately present the instantaneous tactile signals in tactile media and improve the consumer experience of tactile media has become a current research hotspot.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a tactile media data processing method, apparatus, computer equipment, and storage medium, which can accurately present instantaneous tactile signals in tactile media and enhance the tactile media consumption experience.

[0006] In one aspect, an embodiment of the present application provides a method for processing tactile media data, the method comprising:

[0007] Acquiring representation information of the instantaneous tactile signal in the tactile medium, where the representation information of the instantaneous tactile signal is used to indicate a duration of the instantaneous tactile signal;

[0008] The instantaneous tactile signal is presented in the tactile medium according to the representation information of the instantaneous tactile signal.

[0009] Accordingly, an embodiment of the present application provides a tactile media data processing device, the tactile media data processing device comprising:

[0010] an acquiring unit, configured to acquire representation information of the instantaneous tactile signal in the tactile medium, wherein the representation information of the instantaneous tactile signal is used to indicate a duration of the instantaneous tactile signal;

[0011] The processing unit is configured to present the instantaneous tactile signal in the tactile medium according to the representation information of the instantaneous tactile signal.

[0012] On the other hand, an embodiment of the present application provides a method for processing tactile media data, the method comprising:

[0013] Obtaining the duration of instantaneous tactile signals in tactile media;

[0014] generating representation information of the instantaneous tactile signal according to the duration of the instantaneous tactile signal;

[0015] The representation information of the instantaneous tactile signal is encapsulated into a media format file of the tactile media.

[0016] Accordingly, an embodiment of the present application provides a tactile media data processing device, the tactile media data processing device comprising:

[0017] an acquiring unit, configured to acquire the duration of the instantaneous tactile signal in the tactile medium;

[0018] a processing unit, configured to generate representation information of the transient tactile signal according to a duration of the transient tactile signal;

[0019] The processing unit is further configured to encapsulate the representation information of the instantaneous tactile signal into a media format file of the tactile media.

[0020] Accordingly, an embodiment of the present application provides a computer device, comprising:

[0021] a processor suitable for implementing a computer program;

[0022] A computer-readable storage medium stores a computer program, wherein the computer program is suitable for being loaded by a processor and executing the above-mentioned tactile media data processing method.

[0023] Accordingly, an embodiment of the present application provides a computer-readable storage medium storing a computer program. When the computer program is read and executed by a processor of a computer device, the computer device executes the above-mentioned tactile media data processing method.

[0024] Accordingly, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the aforementioned tactile media data processing method.

[0025] In an embodiment of the present application, representation information of an instantaneous tactile signal in a tactile medium is defined, and the representation information of the instantaneous tactile signal can be used to indicate the duration of the instantaneous tactile signal; that is, through the representation information of the tactile signal, the presentation form of the instantaneous tactile signal in the tactile medium can be accurately defined, thereby accurately presenting the instantaneous tactile signal in the tactile medium based on the duration indicated by the representation information of the tactile signal, thereby improving the consumption experience of the tactile media. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] FIG1a is a schematic diagram of a 6DoF camera provided in an embodiment of the present application;

[0027] FIG1b is a schematic diagram of a 3DoF provided in an embodiment of the present application;

[0028] FIG1c is a schematic diagram of a 3DoF+ provided in an embodiment of the present application;

[0029] FIG2 is a schematic diagram of a data structure of information representing tactile media provided by an embodiment of the present application;

[0030] FIG3 is a schematic diagram of the architecture of a tactile media processing system provided in an embodiment of the present application;

[0031] FIG4 is a schematic diagram of a tactile media processing flow in a tactile media processing system provided in an embodiment of the present application;

[0032] FIG5 is a flow chart of a method for processing tactile media data according to an embodiment of the present application;

[0033] FIG6 is a schematic diagram of the structure of a bit stream file of a tactile media provided in an embodiment of the present application;

[0034] FIG7 is a flow chart of another method for processing tactile media data provided in an embodiment of the present application;

[0035] FIG8 is a schematic structural diagram of a tactile media data processing device provided in an embodiment of the present application;

[0036] FIG9 is a schematic structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to more clearly understand the technical solutions provided by the embodiments of the present application, some technical terms involved in the present application are first introduced here.

[0038] 1. Immersive Media

[0039] Immersive media refers to a media format file that can provide immersive media content, so that consumers immersed in the media content can obtain visual, auditory, tactile and other sensory experiences in the real world. Immersive media may include but is not limited to at least one of the following (the term "at least one" in the embodiments of this application means one or more, as explained herein): audio media, video media, tactile media, etc. Consumers may include but are not limited to at least one of the following: listeners of audio media, viewers of video media, users of tactile media, etc. Immersive media can be divided into 6DoF (Degree of Freedom) immersive media, 3DoF immersive media, and 3DoF+ immersive media according to the degree of freedom of consumers when consuming media content. Among them, as shown in Figure 1a, 6DoF means that consumers of immersive media can freely translate along the X-axis, Y-axis, and Z-axis. For example, consumers of immersive media can move freely in three-dimensional 360-degree VR (Virtual Reality) content. Similar to 6DoF are 3DoF and 3DoF+ production technologies. As shown in Figure 1b, 3DoF means that the consumer of immersive media is fixed at the center point of a three-dimensional space, and the consumer's head rotates along the X-axis, Y-axis, and Z-axis to view the images provided by the media content. As shown in Figure 1c, 3DoF+ means that when the virtual scene provided by the immersive media has a certain depth information, the consumer's head can move within a limited space based on 3DoF to view the images provided by the media content.

[0040] 2. Haptics

[0041] Immersive media content is often presented through various smart devices, such as wearables or interactive devices. Therefore, in addition to traditional visual and auditory presentations, immersive media also incorporates a new sense of touch. Haptics, through a tactile presentation mechanism that combines hardware and software, allows consumers to receive information through their bodies, providing an embedded physical sensation that conveys key information about the system they are using. For example, a device may vibrate to alert the user of a received message. This vibration is a form of tactile presentation. Haptics can also enhance auditory and visual presentations, improving the consumer experience.

[0042] Haptics can include, but are not limited to, one or more of the following: vibrotactile, kinematic, and electrotactile. Vibrotactile refers to the simulation of vibrations of a specific frequency and intensity through the vibrations of a device's motor. For example, in a shooting game, vibrations can simulate the specific effects of using a shooting tool. Kinematic haptics refers to the simulation of the weight or pressure of an object by a kinematic haptic system. Kinematic haptics can include, but are not limited to, velocity and acceleration. For example, in a driving game, when moving at high speeds or operating a heavy vehicle, the steering wheel may resist turning. This type of feedback directly impacts the consumer. In the driving game example, the consumer must apply more force to get the desired response from the steering wheel. Electrotactile uses electrical pulses to deliver tactile stimulation to the consumer's nerve endings. Electrotactile can create a highly realistic experience for consumers wearing suits or gloves equipped with electrotactile technology. Almost any sensation can be simulated using electrical pulses: temperature changes, pressure changes, even the sensation of moisture. With the popularization of wearable devices and interactive devices, the tactile sensations that consumers perceive when consuming immersive media content can include vibration, pressure, speed, acceleration, temperature, humidity, smell, and other all-round physical sensations, which is closer to the real-world tactile experience.

[0043] 3. Tactile media:

[0044] Haptic media refers to immersive media of the tactile type, a media format file that can provide consumers with a real-world tactile sensory experience. Haptic media can contain one or more haptic signals (Haptics Signals), which represent a specific modal tactile experience and can be rendered and presented on a specific device. In terms of presentation type, haptic signals can include, but are not limited to, vibrotactile signals, pressure signals, velocity signals, temperature signals, and so on. The tactile type of haptic media varies depending on the haptic signal. For example, if the haptic signal is a vibrotactile signal, the tactile type of the haptic media is vibrotactile media; or, if the haptic signal is an electrotactile signal, the tactile type of the haptic media is electrotactile media. In terms of presentation duration, haptic signals can include, but are not limited to, instantaneous haptic signals and continuous haptic signals. Continuous haptic signals have a longer duration than instantaneous haptic signals. Specifically, for vibrotactile signals, the duration can be measured by the number of vibrations of the vibration motor at a reference frequency. In terms of presentation form, tactile signals may include but are not limited to: reference tactile signals and ordinary tactile signals. Reference tactile signals refer to tactile signals that refer to knowledge tactile signals. Knowledge tactile signals refer to predefined tactile signals that can be repeatedly referenced to avoid repeated analysis of the same tactile signals. Ordinary tactile signals refer to tactile signals that do not refer to knowledge tactile signals.

[0045] The representation information of tactile media can be used to indicate attribute information of the tactile media. The attribute information of the tactile media is related to the presentation form of the tactile media. The data structure of the representation information of tactile media is introduced below with reference to FIG2 . FIG2 shows the data structure of the representation information of tactile media in the AVS (Audio Video Coding Standard) format. The data structure of the representation information of tactile media is expanded hierarchically.

[0046] The highest-level data structure in the representation information of tactile media is the tactile experience, which can be used to describe all the tactile experience information in a file or bitstream. The tactile experience includes metadata information related to the tactile experience, possible device information (for example, device 1-device N in Figure 2), and one or more tactile patterns (for example, tactile pattern 1-tactile pattern N in Figure 2). A tactile pattern corresponds to a tactile signal of a certain tactile type (for example, vibration, pressure, temperature, etc.). The tactile pattern includes metadata information related to the tactile pattern, possible predefined knowledge tactile events (for example, knowledge tactile event 1-knowledge tactile event N in Figure 2), and one or more tactile channels (for example, tactile channel 1-tactile channel N in Figure 2). A tactile channel contains all or part of the tactile signal of the corresponding tactile pattern. The tactile channel includes metadata information related to the tactile channel and one or more tactile events (for example, tactile event 1-tactile event N in Figure 2); generally speaking, tactile signals corresponding to different rendering devices in the same tactile pattern can be organized into different tactile channels. A tactile event is a basic tactile signal unit. In other words, a tactile event is equivalent to a tactile signal. A tactile event includes metadata related to the tactile event and possible tactile event components (e.g., tactile event component 1 through tactile event component N in FIG2 ). A tactile event component refers to the time domain or frequency domain component of a tactile event.

[0047] The following describes the data structures of tactile experience, tactile mode, knowledge tactile event, tactile channel, tactile event, and tactile event component:

[0048] 1. The data structure definition of tactile experience is shown in Table 1 below:

[0049] Table 1 Data structure of tactile experience

[0050] Table 1 above shows the attribute information of the data experience, including the attribute information of the following attributes: version (version field), creation_date (creation time field), description (description field), timescale (time scale field), devices (device information element list field) and patterns (tactile pattern element list field).

[0051] The data structure of the device information is defined as shown in Table 2 below:

[0052] Table 2 Data structure of device information

[0053] Table 2 above shows the data structure of the device information, including the attribute information of the following attributes: id (device identification field), device_name (device name field) and body_parts (device corresponding body part field).

[0054] 2. The data structure definition of the tactile mode is shown in Table 3 below:

[0055] Table 3 Data structure of tactile mode

[0056] As shown in Table 3 above, the data structure of the tactile mode includes attribute information of the following attributes: id (tactile mode identification field), pattern_type (tactile mode type field), refer_devices (tactile mode corresponding device identification field), event_semantics_flag (tactile mode semantics indication field), event_semantics_type (tactile event semantics follow specification field), library_events (knowledge tactile event element list field) and channels (tactile channel element list field).

[0057] 3. Data structure of knowledge tactile events:

[0058] A knowledge tactile event is a special tactile event whose data structure follows the data structure definition of a tactile event. A knowledge tactile event is usually a predefined tactile event that can be repeatedly referenced to avoid repeated parsing of the same tactile event.

[0059] 4. The data structure of the tactile channel is defined as shown in Table 4 below:

[0060] Table 4 Data structure of tactile channel

[0061] As shown in Table 4 above, the data structure of the tactile channel includes attribute information of the following attributes: id (tactile channel identifier field), description (tactile channel description field), refer_devices (device field corresponding to the tactile mode), gain (tactile channel gain field), and events (tactile event element list field).

[0062] 5. The data structure of the tactile event is defined as shown in Table 5 below:

[0063] Table 5 Data structure of tactile events

[0064] As shown in Table 5 above, the data structure of the tactile event includes attribute information of the following attributes: id (tactile event identifier field), effect_semantic (tactile event semantic field), event_type (tactile event type field), refer_library_event (reference knowledge event field), relative_time (time offset field), duration (duration field), amplitude (amplitude field), base_frequency (base frequency field) and event_components (event component field).

[0065] 6. The data structure definition of the tactile event component is shown in Table 6 below:

[0066] Table 6 Data structure of haptic event component

[0067] As shown in Table 6 above, the data structure of the tactile event component includes attribute information of the following attributes: refer_event_id (the event identifier field to which the component belongs), relative_time (the time offset field of the component relative to the event), relative_amplitude (the amplitude offset field of the component relative to the event), and relative_frequency (the frequency offset field of the component relative to the event).

[0068] 4. Track:

[0069] A track refers to a collection of media data in the process of encapsulating a media format file. A track consists of multiple samples with a time sequence. A media format file may contain one or more tracks. Schematically, for example, a video media format file may include but is not limited to: a video media track, an audio media track, and a subtitle media track. In particular, metadata information can also be included in a media format file as a media type in the form of a metadata track. The so-called metadata information is a general term for information related to the presentation of tactile media. The metadata information may include description information of the media content of the tactile media, dependency information on which the tactile media depends, and signaling information related to the presentation of the media content of the tactile media, etc. For example, metadata information related to the tactile experience may include description information of the media content corresponding to the tactile experience, and signaling information related to the presentation of the media content corresponding to the tactile experience, etc. Similarly, metadata information related to the tactile pattern may include description information of the media content corresponding to the tactile pattern, and signaling information related to the presentation of the media content corresponding to the tactile pattern, etc.

[0070] 5. ISOBMFF (ISO Based Media File Format, a media format file format based on ISO standards):

[0071] ISOBMFF is an encapsulation standard for media format files, and a typical ISOBMFF file is an MP4 file.

[0072] Based on the introduction of the above technical terms, an embodiment of the present application provides a data processing method for tactile media, which adds representation information of instantaneous tactile signals in tactile media. The representation information of instantaneous tactile signals can be used to indicate the duration of instantaneous tactile signals. Through the representation information of tactile signals, the presentation form of instantaneous tactile signals in tactile media can be accurately defined, thereby accurately presenting instantaneous tactile signals in tactile media and improving the consumption experience of tactile media.

[0073] Furthermore, in the data processing method for tactile media provided in the embodiment of the present application, the representation information of the instantaneous tactile signal can be added to the exchange format file of the tactile media. The exchange format file of the tactile media refers to the representation information of the tactile media and can be used to define the attribute information of the tactile media. By adding the representation information of the instantaneous tactile signal to the exchange format file of the tactile media, the instantaneous tactile signal in the tactile media can be accurately presented at the presentation level of the tactile media, thereby improving the consumption experience of the tactile media. Optionally, the representation information of the instantaneous tactile signal can also be added to the bit stream file of the tactile media. The bit stream file of the tactile media is a binary bit stream obtained by compressing (i.e., encoding) the exchange format file of the tactile media. By adding the representation information of the instantaneous tactile signal to the bit stream file of the tactile media, the tactile media can be processed accordingly at the bit stream level of the tactile media. Optionally, the representation information of the instantaneous tactile signal can also be added to the media format file of the tactile media. The media format file of the tactile media is obtained by encapsulating the bit stream file of the tactile media. By adding the representation information of the instantaneous tactile signal to the media format file of the tactile media, the tactile media can be processed accordingly at the encapsulation level of the tactile media.

[0074] The following describes a tactile media processing system provided in an embodiment of the present application. The tactile media processing system can be used to implement a tactile media data processing method provided in an embodiment of the present application.

[0075] As shown in Figure 3, the tactile media processing system may include a service device 301 and a consumer device 302. The service device 301 can serve as an encoding end of the tactile media to encode and encapsulate the tactile media to form a media format file of the tactile media. The consumer device 302 can serve as a decoding end of the tactile media (or can be called the media presentation end mentioned above) to decapsulate, decode and consume the media format file of the tactile media, thereby presenting the tactile media. The embodiment of the present application does not limit the connection method between the service device 301 and the consumer device 302. A direct communication connection can be established between the service device 301 and the consumer device 302 by means of wired communication, or an indirect communication connection can be established between the service device 301 and the consumer device 302 by means of wireless communication.

[0076] The service device 301 can be a terminal device or a server; the consumer device 302 can also be a terminal device or a server. The terminal device can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, in-vehicle terminal, smart TV, smart wearable device, smart interactive device, etc., but is not limited to these. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0077] The transmission process of tactile media between the service device 301 and the consumer device 302 can be carried out based on various transmission protocols (or transmission signaling). The transmission protocols here may include but are not limited to: DASH (Dynamic Adaptive Streaming over HTTP, dynamic adaptive streaming media transmission) protocol, HLS (HTTP Live Streaming, dynamic bit rate adaptive transmission) protocol, SMTP (Smart Media Transport Protocol, smart media transmission protocol), TCP (Transmission Control Protocol, transmission control protocol), etc.

[0078] When the service device 301 executes the data processing method for tactile media provided in the embodiment of the present application, the data processing process mainly involves the following: (1) the acquisition process of tactile media; (2) the encoding and file packaging process of tactile media. When the consumer device 302 executes the data processing method for tactile media provided in the embodiment of the present application, the data processing process mainly involves the following: (3) the file decapsulation and decoding process of tactile media; (4) the presentation process of tactile media. The data processing processes of the service device 301 and the consumer device 302 are respectively introduced below:

[0079] (1) The acquisition process of tactile media.

[0080] The service device 301 can acquire tactile media and generate an exchange format file for the tactile media. The exchange format file for the tactile media contains information representing instantaneous tactile signals. Specifically, the service device 301 can acquire tactile media, which can include one or more tactile signals. Different tactile signals may require different methods for acquiring the corresponding tactile media. For example, for vibrotactile signals, the corresponding vibrotactile media can be acquired by capturing vibrotactile signals with specific frequencies and intensities through a capture device (e.g., a sensor) associated with the service device 301. Association here means that the service device 301 can communicate with the capture device, and the capture device can send the captured tactile signals to the service device 301. For another example, for electrotactile signals, the corresponding electrotactile media can be acquired by capturing electrical pulses through a capture device associated with the service device 301 to form electrotactile signals. The capture device can be determined based on the type of tactile signal being captured, and may include, but is not limited to, a camera device, a sensor device, or a scanning device. Camera devices may include ordinary cameras, stereo cameras, and light field cameras, among others. Sensor devices may include laser devices, radar devices, and others. The scanning device may include a three-dimensional laser scanning device, etc.

[0081] (2) The process of encoding and file packaging of tactile media.

[0082] The service device 301 may encode the exchange format file of the tactile media to obtain a bit stream file of the tactile media. The service device 301 may encapsulate the bit stream file of the tactile media to obtain an encapsulated file of the tactile media.

[0083] Optionally, the service device 301 may add the representation information of the transient haptic signal to the bit stream file of the haptic media; or, the service device 301 may add the representation information of the transient haptic signal to the media format file of the haptic media.

[0084] (3) The process of decapsulating and decoding files of tactile media.

[0085] The consumer device 302 can obtain the media format file and corresponding media presentation description information of the tactile media through the service device 301. The media presentation description information includes relevant information about the media format file of the tactile media. The file decapsulation process of the consumer device 302 is the inverse of the file encapsulation process of the service device 301. The consumer device 302 decapsulates the media format file according to the file format requirements of the tactile media to obtain the bitstream file of the tactile media. The decoding process of the consumer device 302 is the inverse of the encoding process of the service device 301. The consumer device 302 decodes the bitstream file of the tactile media to restore the exchange format file of the tactile media. The exchange format file of the tactile media may include representation information of the instantaneous tactile signal.

[0086] Optionally, if representation information of the instantaneous tactile signal is added during the encapsulation process of the tactile media, the representation information of the instantaneous tactile signal can be decapsulated and obtained during the decapsulation process of the tactile media; or, if identification information of the instantaneous tactile signal is added during the encoding process of the tactile media, the representation information of the instantaneous tactile signal can be decoded and obtained during the decoding process of the tactile media.

[0087] (4) The presentation process of tactile media.

[0088] The consumer device 302 can render the decoded haptic media exchange format file to present the haptic media. Specifically, the consumer device 302 can render the information representing the instantaneous haptic signal in the haptic media exchange format file to present the instantaneous haptic signal in the haptic media.

[0089] Furthermore, the data processing process of the aforementioned service device 301 and consumer device 302 can be summarized as the flowchart shown in Figure 4. As shown in Figure 4, the service device 301 can: collect tactile media B, which contains instantaneous tactile signal A; generate an exchange format file for tactile media B, which contains representation information of instantaneous tactile signal A; encode the exchange format file for tactile media B to obtain a bitstream file E for the tactile media; and encapsulate the bitstream file E for the tactile media to obtain a media format file for the tactile media. Furthermore, the service device 301 can also add the representation information of the instantaneous tactile signal to the bitstream file for the tactile media. The service device 301 can also add the representation information of the instantaneous tactile signal to the media format file for the tactile media.

[0090] Specifically, the service device 301 synthesizes the bitstream file E of the haptic media into a media format file F for file playback, or an initialization segment and a media format file segment (Fs) for streaming, based on a specific media container file format. That is, the media format file of the haptic media may include the media format file F or the media format file segment Fs. The media container file format may be the ISO base media format file format specified in International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC) 14496-12.

[0091] For the consumer device 302: it can receive the media format file of the tactile media sent by the service device 301, and the media format file may include: a media format file F′ for file playback, or an initialization segment and a segment Fs′ of the media format file for streaming; it can decapsulate the media format file E of the tactile media to obtain the bitstream file E′ of the tactile media; it can decode the bitstream file E′ of the tactile media to obtain the exchange format file of the tactile media D′, and the exchange format file of the tactile media D′ includes the representation information of the instantaneous tactile signal A′; it can render the tactile media D′ according to the exchange format file of the tactile media D′ to present the tactile media D′, and in particular, it can render the instantaneous tactile signal A′ according to the representation information of the instantaneous tactile signal A′ to present the instantaneous tactile signal A′. In addition, the consumer device 302 may also obtain the representation information of the instantaneous haptic signal from the bitstream file of the haptic media. The consumer device 302 may also obtain the representation information of the instantaneous haptic signal from the media format file of the haptic media.

[0092] It should be noted that the data processing flow of tactile media under the above-mentioned tactile media processing system can be applied to tactile feedback-related products, as well as service nodes (i.e., encoding terminals), consumer nodes (i.e., decoding terminals), and intermediate nodes (i.e., relay terminals) of immersive systems. It is understandable that the data processing method of tactile media involved in this application can also be implemented based on cloud technology; for example, a cloud server implemented based on cloud technology can be used as an encoding terminal or decoding terminal, wherein cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network within a wide area network or a local area network to realize data calculation, storage, processing, and sharing.

[0093] This embodiment of the present application provides a method for processing tactile media data. This method primarily describes the representation information of instantaneous tactile signals added by a decoder. This method can be executed by a computer device, such as the consumer device 302 in the tactile media processing system shown in FIG. 3 . As shown in FIG. 5 , this method can include steps S501 and S502:

[0094] S501 : Acquire representation information of an instantaneous tactile signal in a tactile medium. The representation information of the instantaneous tactile signal is used to indicate a duration of the instantaneous tactile signal.

[0095] Methods for obtaining the representation information of the instantaneous tactile signal in the tactile media may include at least one of the following: first, obtaining the representation information of the instantaneous tactile signal from an exchange format file of the tactile media; second, obtaining the representation information of the instantaneous tactile signal from a bit stream file of the tactile media; and third, obtaining the representation information of the instantaneous tactile signal from a media format file of the tactile media. The following describes these three methods for obtaining the representation information of the instantaneous tactile signal:

[0096] 1. Obtaining the representation information of the instantaneous tactile signal from the tactile media exchange format file:

[0097] In the method for obtaining the representation information of the instantaneous tactile signal, an exchange format file of tactile media can be obtained. The exchange format file of tactile media can be used to define the attribute information of the tactile media. The method for obtaining the exchange format file of tactile media can include: decapsulating the media format file of the tactile media to obtain a bitstream file of the tactile media, and decoding the bitstream file of the tactile media to obtain the exchange format file of the tactile media. In this method for obtaining the exchange format file of tactile media, the consumer device 302 can be understood as a decoding end of the tactile media. Alternatively, an exchange format file of tactile media can be generated. In this method for obtaining the exchange format file of tactile media, the consumer device 302 can be understood as an encoding end and a decoding end of the tactile media, and the consumer device 302 can independently produce and present the tactile media.

[0098] Furthermore, in the tactile media exchange format file, the method for obtaining the representation information of the instantaneous tactile signal may include any of the following: obtaining the representation information of the instantaneous tactile signal from the data structure of the tactile pattern, obtaining the representation information of the instantaneous tactile signal from the data structure of the knowledge tactile event, and obtaining the representation information of the instantaneous tactile signal from the data structure of the tactile event. The following introduces the three methods for obtaining the representation information of the instantaneous tactile signal in the tactile media exchange format file:

[0099] (1) Obtaining the representation information of the instantaneous tactile signal from the data structure of the tactile pattern:

[0100] The haptic media exchange format file may include a haptic pattern data structure, which may be used to define attribute information of the haptic pattern. A haptic pattern may include one or more haptic channels and related metadata. A haptic pattern corresponds to a haptic signal of a haptic type. After adding information representing the instantaneous haptic signal, the haptic pattern data structure (AVS_haptics_pattern) is shown in Table 7 below:

[0101] Table 7 Data structure of tactile mode

[0102] Comparing Table 7 with the data structure of the tactile mode before the addition of the transient tactile signal representation information in Table 3 above, it can be seen that: after the addition, the data structure of the tactile mode can include a transient duration field (transient_duration); the representation information of the transient tactile signal can include the transient duration field, which can be used to indicate the duration of the transient tactile signal in the current tactile mode. Moreover, when the timescale is used as the unit, transient_duration / timescale can be used to indicate the duration of the transient tactile signal in seconds.

[0103] Optionally, for vibrotactile haptics, transient_duration can also be defined according to the base frequency of the vibration motor. That is, transient_duration is used to indicate the number of times the vibration motor vibrates at the base frequency. In this case, the semantics of transient_duration are as follows: it indicates the measurement of the transient vibration signal in the vibrotactile mode. The value of this field represents the number of times the vibration motor vibrates at the base frequency. In other words, the data structure of the tactile pattern can also include a tactile pattern type field (pattern_type). The tactile pattern type field can be used to indicate the type of tactile pattern of the tactile media. When the tactile pattern type field indicates that the type of tactile pattern of the tactile media is vibrotactile, the transient duration field can be used to indicate the duration of the transient tactile signal in the current vibrotactile mode, which is the number of times the vibration motor vibrates at the base frequency.

[0104] (2) Obtaining the representation information of the instantaneous tactile signal from the data structure of the knowledge tactile event:

[0105] The exchange format file of tactile media may include the data structure of knowledge tactile events, and the data structure of knowledge tactile events may be used to define the attribute information of knowledge tactile events; knowledge tactile events are special tactile events, and their data structure may follow the data structure definition of tactile events. Knowledge tactile events are usually predefined tactile events that can be repeatedly referenced to avoid repeated parsing of the same tactile events. In other words, for the representation information of instantaneous tactile signals, the currently defined knowledge tactile events may be used to define special knowledge tactile events to represent the duration of the instantaneous tactile signal in the corresponding tactile mode. After adding the representation information of the instantaneous tactile signal, the data structure of the knowledge tactile event is shown in Table 8 below:

[0106] Table 8 Data structure of knowledge tactile events

[0107] Comparing Table 8 with the data structure of the tactile event before the representation information of the instantaneous tactile signal is newly added in Table 5 above, it can be seen that: after the addition, the data structure of the knowledge tactile event may include a tactile event type field (event_type), and the tactile event type field may be used to indicate the event type of the knowledge tactile event of the tactile media, and the event type may include an instantaneous event or a continuous event; when the tactile event type field indicates that the event type of the knowledge tactile event of the tactile media is an instantaneous event, and the data structure of the knowledge tactile event also includes a duration field (duration), the representation information of the instantaneous tactile signal may include a duration field, and the duration field may be used to indicate the duration of all instantaneous tactile events in the current tactile mode; wherein, in a tactile mode, there is a knowledge tactile event with a duration field and an event type of an instantaneous event.

[0108] In this case, for a continuous event, when the haptic event type field indicates that the event type of the knowledge haptic event of the haptic medium is a continuous event, the duration field may be used to indicate the duration of the corresponding knowledge haptic event.

[0109] Continuing to compare the data structure of the tactile event with Table 8 before adding the representation information of the instantaneous tactile signal in Table 5 above, it can be seen that: the data structure of the knowledge tactile event is determined based on the data structure of the tactile event, the data structure of the tactile event may include a reference knowledge event field (refer_library_event), the data structure of the knowledge tactile event may not include the reference knowledge event field, the reference knowledge event field can be used to indicate the knowledge tactile event identifier corresponding to the tactile event when the event type of the tactile event is a reference event; the data structure of the tactile event may include a relative time field (relative_time), the data structure of the knowledge tactile event may not include the relative time field, the relative time field can be used to indicate the time offset of the tactile event.

[0110] The data structure of a tactile event may include an amplitude field (amplitude), a base frequency field (base_frequency), and an event component field (event_components); when the tactile event type field indicates that the event type of the knowledge tactile event of the tactile media is an instantaneous event, and the data structure of the knowledge tactile event includes a duration field, the data structure of the knowledge tactile event may not include the amplitude field, the base frequency field, and the event component field; the amplitude field may be used to indicate the maximum amplitude value of the signal of the tactile event; the base frequency field may be used to indicate the base frequency of the signal of the tactile event; the event component field may be used to indicate a list of tactile event component elements, and the list of tactile event component elements is used to indicate the components of the tactile event, and the components of the tactile event include the time domain component or the frequency domain component of the tactile event.

[0111] (3) Obtaining the representation information of the instantaneous tactile signal from the data structure of the tactile event:

[0112] The haptic media exchange format file can include the data structure of haptic events, which can be used to define the attribute information of haptic events. A haptic event can contain a complete haptic signal and related metadata. The representation information of the instantaneous haptic signal can be further directly merged with the currently defined continuous haptic signal. That is, for the representation information of haptic media, there are only two types of haptic events: ordinary events and reference events. Ordinary events are haptic signals that do not reference knowledge haptic events. Ordinary events must have continuous events, and reference events are haptic signals that need to reference knowledge haptic events. After adding the representation information of the instantaneous haptic signal, the data structure of the haptic event (AVS_haptics_event) is shown in Table 9 below:

[0113] Table 9 Data structure of tactile events

[0114] Comparing Table 9 with the data structure of the tactile event before the representation information of the instantaneous tactile signal is newly added in Table 5 above, it can be seen that: after the addition, the data structure of the tactile event includes a tactile event type field (event_type), and the tactile event type field can be used to indicate the event type of the tactile event of the tactile media, and the event type includes a reference event or a common event; when the tactile event type field indicates that the event type of the tactile event of the tactile media is a reference event, the data structure of the tactile event also includes a reference knowledge event field (refer_library_event), and the representation information of the instantaneous tactile signal may include the reference knowledge event field; the reference knowledge event field can be used to indicate the knowledge tactile event identifier corresponding to the tactile event whose event type is a reference event; wherein, the duration of the tactile event whose event type is a reference event is the same as the duration of the knowledge tactile event indicated by the corresponding knowledge tactile event identifier.

[0115] For a common event, when the haptic event type field indicates that the event type of the haptic event of the haptic media is a common event, the data structure of the haptic event may further include a relative time field (relative_time), which may be used to indicate a time offset of the haptic event.

[0116] 2. Obtaining the representation information of the instantaneous tactile signal from the bit stream file of the tactile media:

[0117] In this method for obtaining information representing a transient tactile signal, a bitstream file of tactile media can be obtained. At the encoding end, the bitstream file of tactile media can be obtained by encoding a tactile media exchange format file. At the decoding end, the bitstream file of tactile media can be obtained by decapsulating the tactile media media format file. As shown in FIG6 , after the tactile media exchange format file is compressed into a binary bitstream (i.e., the tactile media bitstream file), the data therein exists in the binary bitstream in the form of packets. Specifically, the tactile media bitstream file can include multiple bitstream units (e.g., ..., bitstream unit i-1, bitstream unit i, bitstream unit i+1, ... in FIG6 ), each of which can include a bitstream unit header and multiple bitstream data packets (e.g., bitstream unit i in FIG6 includes a bitstream unit header and bitstream data packets 0 to bitstream data packets n). Each bitstream data packet can be considered a data unit in the binary bitstream. In particular, there are some data units used to contain metadata in the exchange format file, such as a tactile pattern metadata package and a knowledge tactile event metadata package; wherein the tactile pattern metadata package can be obtained by encoding the tactile pattern metadata, and the knowledge tactile event metadata package can be obtained by encoding the knowledge tactile event metadata.

[0118] Furthermore, in the bitstream file of the tactile media, the method for obtaining the representation information of the instantaneous tactile signal may include any of the following: obtaining the representation information of the instantaneous tactile signal from the tactile pattern metadata package, or obtaining the representation information of the instantaneous tactile signal from the knowledge tactile event metadata package. The following describes the two methods for obtaining the representation information of the instantaneous tactile signal from the bitstream file of the tactile media:

[0119] (1) Obtain the representation information of the instantaneous tactile signal from the tactile pattern metadata package:

[0120] The syntax of the tactile pattern metadata packet (readMetadataPattern()) is shown in Table 10 below:

[0121] Table 10

[0122] The syntax elements in the tactile mode metadata package shown in Table 10 are explained as follows:

[0123] id (tactile mode identification field): an identifier indicating the tactile mode. The value of this field must be greater than or equal to 0.

[0124] patternType (tactile pattern type field): indicates the type of the tactile pattern. The meanings of the values ​​of this field are shown in Table 11 below:

[0125] Table 11

[0126] eventLibraryCount (knowledge tactile event number field): indicates the number of knowledge tactile events.

[0127] sementicsFlag (tactile mode semantic flag field): indicates whether the tactile event corresponding to the tactile mode has semantics.

[0128] event_semantics_type (Haptic Event Semantics Follow Specification Field): Indicates the specification followed by the haptic event semantics. When this field is 0, the haptic event semantics follow the specification defined by this standard; when it is 1, the haptic event semantics follow the specification of an external standard; when it is 2, the haptic event semantics are user-defined.

[0129] transient_duration (short duration field): indicates the duration of the transient tactile signal in the current tactile mode. In units of timescale, transient_duration / timescale is the number of seconds that the transient signal lasts. (Corresponding to the optional solution of obtaining the duration of the transient tactile signal from the exchange format file of the tactile media, the meaning of this field can also be: indicating the measurement of the transient vibration signal in the vibration tactile mode, the value of this field represents the number of times the vibration motor vibrates at the reference frequency).

[0130] referenceDeviceCount (field of the number of devices corresponding to the tactile mode): indicates the number of devices corresponding to the tactile mode.

[0131] referenceDeviceId (tactile mode corresponding device identification field): indicates the identifier of the device corresponding to the tactile mode.

[0132] Based on the syntax of the tactile mode metadata package shown in Table 10 above, it can be seen that: the tactile mode metadata package of the tactile media may include a transient duration field (transient_duration); the representation information of the transient tactile signal may include a transient duration field, and the transient duration field may be used to indicate the duration of the transient tactile signal in the current tactile mode.

[0133] Optionally, for vibration tactile, the tactile pattern metadata package may further include a tactile pattern type field (pattern_type), which may be used to indicate the type of the tactile pattern of the tactile medium; when the tactile pattern type field indicates that the type of the tactile pattern of the tactile medium is vibration tactile, the short duration field is used to indicate that the duration of the instantaneous tactile signal in the current vibration tactile mode may be the number of times the vibration motor vibrates at the reference frequency.

[0134] (2) Obtaining the representation information of the instantaneous tactile signal from the knowledge tactile event metadata package:

[0135] The syntax of the knowledge tactile event metadata package (readLibraryEvents()) is shown in Table 12 below:

[0136] Table 12

[0137] The syntax elements in the tactile mode metadata package shown in Table 12 are explained as follows:

[0138] id (haptic event identifier field): indicates an identifier of a haptic event.

[0139] eventType (tactile event type field): Indicates the type of tactile event. A value of 0 indicates an instantaneous event; a value of 1 indicates a continuous event; a value of 2 indicates a reference event. For knowledge tactile signals, this field cannot have a value of 2.

[0140] hasSemantic (tactile event semantic indication field): When the value is 0, it indicates that the event has no semantics; when the value is 1, it indicates that the event has special semantics.

[0141] semanticKeywords (tactile event semantic field): represents the semantics of the event.

[0142] durationFlag (duration flag field): When the value is 0, it does not indicate the duration of the event; when the value is 1, it indicates the duration of the event.

[0143] durationTime (Duration field): Indicates the duration of the event. For knowledge haptic events, if the event type is instantaneous and this field is present, this field indicates the duration of all instantaneous events in the current haptic mode. For a haptic mode, there can only be one instantaneous knowledge haptic event with a duration field.

[0144] If the event type is a continuous event, this field must be present and indicates the continuous event of the corresponding knowledge tactile event.

[0145] This field uses the timescale as the unit, and transient_duration / timescale is the duration of the transient signal in seconds.

[0146] baseFrequency (base frequency field): indicates the signal base frequency of the haptic event.

[0147] baseAmplitude (base amplitude field): indicates the maximum amplitude value of the signal of the tactile event.

[0148] eventComponentsCount (tactile event component count field): indicates the number of event components included in the tactile event.

[0149] Mask (tactile event component control field): used to control the corresponding field indication of the tactile event component.

[0150] relativeTime (relative time field): indicates the time offset value of the touch event component relative to the touch event.

[0151] relativeAmplitude (relative amplitude field): indicates the amplitude offset value of the haptic event component relative to the haptic event.

[0152] relativeFrequency (relative frequency field): indicates the frequency offset value of the haptic event component relative to the haptic event.

[0153] Based on the syntax of the knowledge tactile event metadata package shown in Table 12 above, it can be seen that: the knowledge tactile event metadata package may include an event type field (eventType) and a duration flag field (durationFlag); the event type field may be used to indicate the event type of the knowledge tactile event, and the event type may include an instantaneous event or a continuous event; the duration flag field may be used to indicate whether the duration of the knowledge tactile event is to be indicated; if the duration flag field indicates that the duration of the knowledge tactile event needs to be indicated, the knowledge tactile event metadata package may also include a duration field (durationTime), and the representation information of the instantaneous tactile signal may include the duration field; if the event type field indicates that the event type of the knowledge tactile event of the tactile media is an instantaneous event, the duration field may be used to indicate the duration of all instantaneous tactile signals in the current tactile mode; wherein, in a tactile mode, there is a knowledge tactile event with a duration field and the event type is the instantaneous event.

[0154] In this case, for a continuous event, when the event type field indicates that the event type of the knowledge haptic event of the tactile media is a continuous event, the duration field may be used to indicate the duration of the corresponding knowledge haptic event.

[0155] 3. Obtaining representation information of instantaneous tactile signals from a media format file of the tactile media:

[0156] In this method for obtaining representation information of a transient tactile signal, a tactile media format file can be obtained. At the encoding end, the tactile media media format file can be obtained by encoding a bitstream file of the tactile media. At the decoding end, the tactile media media format file can be sent by the encoding end. The representation information of the transient tactile signal is specifically obtained from a tactile pattern description data box in the tactile media media format file. The tactile pattern description data box can be used to indicate metadata information about the tactile pattern of the tactile media. The syntax of the tactile pattern description data box (HapticPatternDescriptionBox()) is first introduced. The syntax of the tactile pattern description data box is as follows:

[0157] The syntax elements of the haptic mode description data box are explained as follows:

[0158] pattern_id (tactile pattern identification field): an identifier indicating the tactile pattern. The value of this field must be greater than or equal to 0.

[0159] pattern_type (tactile pattern type field): indicates the type of the tactile pattern. The meaning of the value of this field is shown in Table 11 above.

[0160] library_event_count (knowledge tactile event number field): indicates the number of knowledge tactile events.

[0161] event_semantics_flag (tactile mode semantics flag field): indicates whether the tactile event corresponding to the tactile mode has semantics.

[0162] event_semantics_type (Haptic Event Semantics Follow Specification Field): Indicates the specification followed by the haptic event semantics. When this field is 0, the haptic event semantics follow the specification defined by this standard; when it is 1, the haptic event semantics follow the specification of an external standard; when it is 2, the haptic event semantics are user-defined.

[0163] transient_duration (short duration field): indicates the duration of the transient tactile signal in the current tactile mode. In units of timescale, transient_duration / timescale is the number of seconds that the transient signal lasts. (Corresponding to the optional solution of obtaining the duration of the transient tactile signal from the exchange format file of the tactile media, the meaning of this field can also be: indicating the measurement of the transient vibration signal in the vibration tactile mode, the value of this field represents the number of times the vibration motor vibrates at the reference frequency).

[0164] Based on the syntax of the above-mentioned tactile pattern description data box, it can be seen that the tactile pattern description data box may include a transient duration field (transient_duration); the representation information of the transient tactile signal may include a transient duration field, which may be used to indicate the duration of the transient tactile signal in the current tactile mode.

[0165] Optionally, for vibration tactile, the tactile pattern description data box may further include a tactile pattern type field (pattern_type), which may be used to indicate the type of the tactile pattern of the tactile medium; when the tactile pattern type field indicates that the type of the tactile pattern of the tactile medium is vibration tactile, the short duration field is used to indicate that the duration of the instantaneous tactile signal in the current vibration tactile mode may be the number of times the vibration motor vibrates at the reference frequency.

[0166] S502: Present the instantaneous tactile signal in the tactile medium according to the representation information of the instantaneous tactile signal.

[0167] After the representation information of the instantaneous haptic signal is acquired, the representation information of the instantaneous haptic signal may be rendered to obtain the instantaneous haptic signal.

[0168] In an embodiment of the present application, representation information of instantaneous tactile signals in tactile media is defined, and the representation information of instantaneous tactile signals can be used to indicate the duration of the instantaneous tactile signals; that is, through the representation information of the tactile signals, the presentation form of the instantaneous tactile signals in the tactile media can be accurately defined, thereby accurately presenting the instantaneous tactile signals in the tactile media and improving the consumption experience of the tactile media.

[0169] This embodiment of the present application provides a method for processing tactile media data. This method primarily describes the representation information of instantaneous tactile signals added by the encoder. This method can be executed by a computer device, such as the service device 301 in the tactile media processing system shown in FIG. 3 . As shown in FIG. 7 , this method can include steps S701 through S703:

[0170] S701: Obtain the duration of an instantaneous tactile signal in a tactile medium.

[0171] S702: Generate representation information of the instantaneous tactile signal according to the duration of the instantaneous tactile signal.

[0172] S703: Encapsulate the representation information of the instantaneous tactile signal into a media format file of the tactile media.

[0173] The representation information of the instantaneous tactile signal can be defined in the exchange format file of the tactile media. That is, the exchange format file of the tactile media can include the representation information of the instantaneous tactile signal. In this case, encapsulating the representation information of the instantaneous tactile signal into the media format file of the tactile media can include: encoding the exchange format file of the tactile media to obtain a bit stream file of the tactile media; and encapsulating the bit stream file of the tactile media to obtain a media format file of the tactile media.

[0174] Furthermore, the tactile media exchange format file may include at least a tactile pattern data structure, a knowledge tactile event data structure, and a tactile event data structure. The representation information of the instantaneous tactile signal is defined in the tactile media exchange format file and may include at least one of the following: the representation information of the instantaneous tactile signal may be defined in the tactile pattern data structure, the representation information of the instantaneous tactile signal may be defined in the knowledge tactile event data structure, and the representation information of the instantaneous tactile signal may be defined in the tactile event data structure. These three situations are introduced below:

[0175] (1) The representation information of the instantaneous tactile signal is defined in the data structure of the tactile pattern:

[0176] The data structure of the tactile pattern can be used to define the attribute information of the tactile pattern of the tactile media, and the data structure of the tactile pattern includes a short duration field; in this case, the representation information of the instantaneous tactile signal may include the short duration field, and generating the representation information of the instantaneous tactile signal based on the duration of the instantaneous tactile signal may include: setting the value of the short duration field to the duration of the instantaneous tactile signal.

[0177] Optionally, for vibration tactile sensation, the data structure of the tactile pattern may further include a tactile pattern type field, which may be used to indicate the type of the tactile pattern of the tactile medium; when the tactile pattern type field indicates that the type of the tactile pattern of the tactile medium is vibration tactile sensation, the duration of the instantaneous tactile signal may refer to the number of times the vibration motor vibrates at a reference frequency.

[0178] (2) The representation information of the instantaneous tactile signal is defined in the data structure of the knowledge tactile event:

[0179] The data structure of the knowledge tactile event can be used to define the attribute information of the knowledge tactile event of the tactile media. The data structure of the knowledge tactile event includes a tactile event type field. The tactile event type field can be used to indicate the event type of the knowledge tactile event of the tactile media. The event type may include an instantaneous event or a continuous event. When the tactile event type field indicates that the event type of the knowledge tactile event of the tactile media is an instantaneous event, and the data structure of the knowledge tactile event also includes a duration field, the representation information of the instantaneous tactile signal may include a duration field. In this case, generating the representation information of the instantaneous tactile signal according to the duration of the instantaneous tactile signal may include: setting the value of the duration field to the duration of the instantaneous tactile signal. In a tactile pattern, there is a knowledge tactile event with a duration field and an event type of an instantaneous event.

[0180] In this case, for a continuous event, when the haptic event type field indicates that the event type of the knowledge haptic event of the haptic medium is a continuous event, the duration field may be used to indicate the duration of the corresponding knowledge haptic event.

[0181] In particular, the data structure of the knowledge tactile event can be determined based on the data structure of the tactile event. The data structure of the tactile event may include a reference knowledge event field, and the data structure of the knowledge tactile event may not include the reference knowledge event field. The reference knowledge event field can be used to indicate the knowledge tactile event identifier corresponding to the tactile event when the event type of the tactile event is a reference event; the data structure of the tactile event may include a relative time field, and the data structure of the knowledge tactile event may not include the relative time field. The relative time field can be used to indicate the time offset of the tactile event.

[0182] In addition, the data structure of the knowledge tactile event can be determined based on the data structure of the tactile event. The data structure of the tactile event can include an amplitude field, a reference frequency field, and an event component field. When the tactile event type field indicates that the event type of the knowledge tactile event of the tactile media is an instantaneous event, and the data structure of the knowledge tactile event includes a duration field, the data structure of the knowledge tactile event may not include the amplitude field, the reference frequency field, and the event component field. Among them, the amplitude field can be used to indicate the maximum amplitude value of the tactile event signal; the reference frequency field can be used to indicate the reference frequency of the tactile event signal; and the event component field can be used to indicate a list of tactile event component elements. The list of tactile event component elements can be used to indicate the components of the tactile event. The components of the tactile event can include time domain components or frequency domain components of the tactile event.

[0183] (3) The representation information of the instantaneous tactile signal is defined in the data structure of the tactile event:

[0184] The data structure of a tactile event can be used to define the attribute information of a tactile event of a tactile media. The data structure of the tactile event can include a tactile event type field. The tactile event type field can be used to indicate the event type of the tactile event of the tactile media. The event type can include a reference event or a common event. When the event type field indicates that the event type of the tactile event of the media is a reference event, the data structure of the reference event also includes a reference knowledge event field. The representation information of the tactile signal can include the reference knowledge event field. In this case, generating the representation information of the tactile signal based on the duration of the tactile signal can include: setting the value of the reference knowledge event field to the knowledge tactile event identifier corresponding to the tactile event whose event type is a reference event; wherein the duration of the tactile event whose event type is a reference event is the same as the duration of the knowledge tactile event indicated by the corresponding knowledge tactile event identifier.

[0185] Optionally, for a common event, when the haptic event type field indicates that the event type of the haptic event of the haptic media is a common event, the data structure of the haptic event may further include a relative time field, which may be used to indicate a time offset of the haptic event.

[0186] The above steps S701 to S703 mainly introduce the process of encapsulating the representation information of the instantaneous tactile signal into the media format file of the tactile media when the representation information of the instantaneous tactile signal is contained in the exchange format file of the tactile media. Optionally, the representation information of the tactile signal can also be added to the bit stream file of the tactile media, and the bit stream file of the tactile media is encapsulated to obtain the media format file of the tactile media. The bit stream file of the tactile media may include at least one of the following: a tactile pattern metadata package and a knowledge tactile event metadata package, that is, the representation information of the tactile signal can be added to the tactile pattern metadata package, or the representation information of the tactile signal can be added to the knowledge tactile event metadata package. These two situations are introduced below respectively:

[0187] ① Add the representation information of the tactile signal to the tactile mode metadata package:

[0188] The haptic pattern metadata packet may be obtained by encoding haptic pattern metadata, and the representation information of the transient haptic signal may be added to the haptic pattern metadata packet. The haptic pattern metadata packet of the haptic media may include a short duration field, and the representation information of the transient haptic signal may include the short duration field. In this case, generating the representation information of the transient haptic signal based on the duration of the transient haptic signal may include setting the value of the short duration field to the duration of the transient haptic signal.

[0189] Optionally, for vibration tactile, the tactile pattern metadata package may further include a tactile pattern type field, which may be used to indicate the type of the tactile pattern of the tactile media; when the tactile pattern type field indicates that the type of the tactile pattern of the tactile media is vibration tactile, the duration of the instantaneous tactile signal may refer to the number of times the vibration motor vibrates at a reference frequency.

[0190] ②Add the representation information of the tactile signal to the knowledge tactile event metadata package:

[0191] The knowledge tactile event metadata package may be obtained by encoding the knowledge tactile event metadata, and the representation information of the instantaneous tactile signal may be added to the knowledge tactile event metadata package; the knowledge tactile event metadata package may include a tactile event type field and a duration flag field; the tactile event type field may be used to indicate the event type of the knowledge tactile event, and the event type may include an instantaneous event or a continuous event; the duration flag field may be used to indicate whether the duration of the knowledge tactile event is indicated. In this case, based on the duration of the instantaneous tactile signal, the representation information of the instantaneous tactile signal is generated, which may include: the value of the duration representation field may be set to a first preset value, indicating that the duration of the knowledge tactile event needs to be indicated; the duration field may be added to the knowledge tactile event metadata package, and the representation information of the instantaneous tactile signal may include a continuous event field; and the value of the duration field may be set to the duration of the instantaneous tactile signal. Among them, when the tactile event type field indicates that the event type of the knowledge tactile event of the tactile media is an instantaneous event, the duration field can be used to indicate the duration of all instantaneous tactile signals in the current tactile mode; there is a knowledge tactile event with a duration field and an event type of an instantaneous event in a tactile mode.

[0192] In this case, for a continuous event, when the haptic event type field indicates that the event type of the knowledge haptic event of the haptic medium is a continuous event, the duration field may be used to indicate the duration of the corresponding knowledge haptic event.

[0193] The above ①-② describe the case of adding the representation information of the instantaneous haptic signal to the bit stream file of the haptic media. Optionally, the representation information of the instantaneous haptic signal is also added to the media format file of the haptic media.

[0194] Specifically, a media format file for haptic media may include a haptic pattern description data box, which may be used to indicate metadata information about the haptic pattern of the haptic media. Representation information for a transient haptic signal is added to the haptic pattern description data box. The haptic pattern description data box may include a short duration field, and the representation information for a transient haptic signal may include the short duration field. In this case, generating representation information for the transient haptic signal based on the duration of the transient haptic signal may include setting the value of the short duration field to the duration of the transient haptic signal.

[0195] Optionally, for vibration tactile sensation, the tactile pattern description data box may further include a tactile pattern type field, which may be used to indicate the type of the tactile pattern of the tactile medium; when the tactile pattern type field indicates that the type of the tactile pattern of the tactile medium is vibration tactile sensation, the duration of the instantaneous tactile signal may refer to the number of times the vibration motor vibrates at a reference frequency.

[0196] In an embodiment of the present application, representation information of instantaneous tactile signals in tactile media is defined, and the representation information of instantaneous tactile signals can be used to indicate the duration of the instantaneous tactile signals; that is, through the representation information of the tactile signals, the presentation form of the instantaneous tactile signals in the tactile media can be accurately defined, thereby accurately presenting the instantaneous tactile signals in the tactile media and improving the consumption experience of the tactile media.

[0197] Please refer to Figure 8, which is a schematic diagram of the structure of a tactile media data processing device provided in an embodiment of the present application. This tactile media data processing device can be installed in a computer device provided in an embodiment of the present application. The computer device can be the consumer device 302 in the tactile media processing system shown in Figure 3 above. The tactile media data processing device shown in Figure 8 can be a computer program running on the computer device. This tactile media data processing device can be used to perform some or all of the steps in the method embodiment shown in Figure 5. Referring to Figure 8, this tactile media data processing device can include the following units:

[0198] An acquiring unit 801 is configured to acquire representation information of an instantaneous tactile signal in a tactile medium, where the representation information of the instantaneous tactile signal is used to indicate a duration of the instantaneous tactile signal;

[0199] The processing unit 802 is configured to present the instantaneous tactile signal in the tactile medium according to the representation information of the instantaneous tactile signal.

[0200] In one implementation, the acquiring unit 801 is configured to acquire the representation information of the instantaneous tactile signal in the tactile media by performing the following steps: acquiring an exchange format file of the tactile media; and acquiring the representation information of the instantaneous tactile signal from the exchange format file of the tactile media.

[0201] In one implementation, the exchange format file of the tactile media includes a data structure of the tactile pattern of the tactile media, the data structure of the tactile pattern is used to define attribute information of the tactile pattern of the tactile media, and the data structure of the tactile pattern includes a short duration field; the representation information of the transient tactile signal includes a short duration field, and the short duration field is used to indicate the duration of the transient tactile signal in the current tactile mode.

[0202] In one implementation, the data structure of the tactile pattern also includes a tactile pattern type field, which is used to indicate the type of the tactile pattern of the tactile medium; when the tactile pattern type field indicates that the type of the tactile pattern of the tactile medium is vibration tactile, the short duration field is used to indicate that the duration of the instantaneous tactile signal in the current vibration tactile mode is the number of times the vibration motor vibrates at the reference frequency.

[0203] In one implementation, an exchange format file of tactile media includes a data structure of knowledge tactile events of the tactile media, the data structure of the knowledge tactile events is used to define attribute information of the knowledge tactile events of the tactile media, the data structure of the knowledge tactile events includes a tactile event type field, the tactile event type field is used to indicate the event type of the knowledge tactile event of the tactile media, and the event type includes an instantaneous event or a continuous event; when the tactile event type field indicates that the event type of the knowledge tactile event of the tactile media is an instantaneous event, and the data structure of the knowledge tactile event also includes a duration field, the representation information of the instantaneous tactile signal includes a duration field, and the duration field is used to indicate the duration of all instantaneous tactile events in the current tactile mode; wherein, in a tactile mode, there is a knowledge tactile event with a duration field and an event type of an instantaneous event.

[0204] In one implementation, when the haptic event type field indicates that the event type of the knowledge haptic event of the haptic media is a continuous event, the duration field is used to indicate the duration of the corresponding knowledge haptic event.

[0205] In one implementation, the data structure of the knowledge tactile event is determined based on the data structure of the tactile event. The data structure of the tactile event includes a reference knowledge event field. The data structure of the knowledge tactile event does not include the reference knowledge event field. The reference knowledge event field is used to indicate the knowledge tactile event identifier corresponding to the tactile event when the event type of the tactile event is a reference event; the data structure of the tactile event includes a relative time field. The data structure of the knowledge tactile event does not include the relative time field. The relative time field is used to indicate the time offset of the tactile event.

[0206] In one implementation, the data structure of the knowledge tactile event is determined based on the data structure of the tactile event, and the data structure of the tactile event includes an amplitude field, a reference frequency field, and an event component field; when the tactile event type field indicates that the event type of the knowledge tactile event of the tactile media is an instantaneous event, and the data structure of the knowledge tactile event includes a duration field, the data structure of the knowledge tactile event does not include the amplitude field, the reference frequency field, and the event component field; the amplitude field is used to indicate the maximum amplitude value of the signal of the tactile event; the reference frequency field is used to indicate the signal reference frequency of the tactile event; the event component field is used to indicate a list of tactile event component elements, and the tactile event component element list is used to indicate the components of the tactile event, and the components of the tactile event include the time domain component or the frequency domain component of the tactile event.

[0207] In one implementation, the exchange format file of tactile media includes a data structure of tactile events of the tactile media, the data structure of the tactile events is used to define attribute information of the tactile events of the tactile media, the data structure of the tactile events includes a tactile event type field, the tactile event type field is used to indicate the event type of the tactile event of the tactile media, and the event type includes a reference event or an ordinary event; when the tactile event type field indicates that the event type of the tactile event of the tactile media is a reference event, the data structure of the tactile event also includes a reference knowledge event field, and the representation information of the instantaneous tactile signal includes the reference knowledge event field; the reference knowledge event field is used to indicate the knowledge tactile event identifier corresponding to the tactile event whose event type is a reference event; wherein, the duration of the tactile event whose event type is a reference event is the same as the duration of the knowledge tactile event indicated by the corresponding knowledge tactile event identifier.

[0208] In one implementation, when the haptic event type field indicates that the event type of the haptic event of the haptic media is a common event, the data structure of the haptic event further includes a relative time field, and the relative time field is used to indicate a time offset of the haptic event.

[0209] In one implementation, the acquiring unit 801 is configured to acquire the representation information of the instantaneous tactile signal in the tactile media by performing the following steps: acquiring a bitstream file of the tactile media; and acquiring the representation information of the instantaneous tactile signal from the bitstream file of the tactile media.

[0210] In one implementation, the bitstream file of the tactile media includes a tactile pattern metadata package, which is obtained by encoding the tactile pattern metadata; the tactile pattern metadata package of the tactile media includes a short duration field; the representation information of the transient tactile signal includes a short duration field, which is used to indicate the duration of the transient tactile signal in the current tactile mode.

[0211] In one implementation, the tactile pattern metadata package also includes a tactile pattern type field, which is used to indicate the type of the tactile pattern of the tactile medium; when the tactile pattern type field indicates that the type of the tactile pattern of the tactile medium is vibration tactile, the short duration field is used to indicate that the duration of the instantaneous tactile signal in the current vibration tactile mode is the number of times the vibration motor vibrates at the reference frequency.

[0212] In one implementation, the bitstream file of the tactile media includes a knowledge tactile event metadata package, which is obtained by encoding the knowledge tactile event metadata; the knowledge tactile event metadata package includes an event type field and a duration flag field; the event type field is used to indicate the event type of the knowledge tactile event, and the event type includes an instantaneous event or a continuous event; the duration flag field is used to indicate whether the duration of the knowledge tactile event is indicated; if the duration flag field indicates that the duration of the knowledge tactile event needs to be indicated, the knowledge tactile event metadata package also includes a duration field, and the representation information of the instantaneous tactile signal includes the duration field; if the event type field indicates that the event type of the knowledge tactile event of the tactile media is an instantaneous event, then the duration field is used to indicate the duration of all instantaneous tactile signals in the current tactile mode; wherein, in a tactile mode, there is a knowledge tactile event with a duration field and an event type of an instantaneous event.

[0213] In one implementation, when the event type field indicates that the event type of the knowledge haptic event of the haptic media is a continuous event, the duration field is used to indicate the duration of the corresponding knowledge haptic event.

[0214] In one implementation, the acquiring unit 801 is configured to acquire the representation information of the instantaneous tactile signal in the tactile media by performing the following steps: acquiring a media format file of the tactile media; and acquiring the representation information of the instantaneous tactile signal from the media format file of the tactile media.

[0215] In one implementation, the media format file of the tactile media includes a tactile pattern description data box, which is used to indicate metadata information of the tactile pattern of the tactile media; the tactile pattern description data box includes a short duration field; and the representation information of the transient tactile signal includes a short duration field, which is used to indicate the duration of the transient tactile signal in the current tactile mode.

[0216] In one implementation, the tactile pattern description data box also includes a tactile pattern type field, which is used to indicate the type of the tactile pattern of the tactile medium; when the tactile pattern type field indicates that the type of the tactile pattern of the tactile medium is vibration tactile, the short duration field is used to indicate that the duration of the instantaneous tactile signal in the current vibration tactile mode is the number of times the vibration motor vibrates at the reference frequency.

[0217] In one implementation, the processing unit 802 is configured to perform the following steps when presenting the instantaneous tactile signal in the tactile medium according to the representation information of the instantaneous tactile signal: rendering the representation information of the instantaneous tactile signal to obtain the instantaneous tactile signal.

[0218] According to another embodiment of the present application, the various units in the tactile media data processing device shown in Figure 8 can be individually or entirely combined into one or several other units to form a structure, or one (or some) of the units can be further divided into multiple functionally smaller units to form a structure, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In actual applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the tactile media data processing device can also include other units. In actual applications, these functions can also be assisted by other units and can be implemented by the collaboration of multiple units.

[0219] According to another embodiment of the present application, a tactile media data processing device as shown in FIG. 8 and the tactile media data processing method of the present application embodiment can be constructed by running a computer program capable of executing some or all of the steps involved in the method shown in FIG. 5 on a general-purpose computing device, such as a computer, including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on a computer-readable storage medium, for example, and loaded into the computing device via the computer-readable storage medium and executed therein.

[0220] In an embodiment of the present application, representation information of instantaneous tactile signals in tactile media is defined, and the representation information of instantaneous tactile signals can be used to indicate the duration of the instantaneous tactile signals; that is, through the representation information of the tactile signals, the presentation form of the instantaneous tactile signals in the tactile media can be accurately defined, thereby accurately presenting the instantaneous tactile signals in the tactile media and improving the consumption experience of the tactile media.

[0221] Please refer to Figure 8, which is a schematic diagram of the structure of a tactile media data processing device provided in an embodiment of the present application. This tactile media data processing device can be installed in a computer device provided in an embodiment of the present application. The computer device can be the service device 301 in the tactile media processing system shown in Figure 3 above. The tactile media data processing device shown in Figure 8 can be a computer program running on the computer device. This tactile media data processing device can be used to execute some or all of the steps in the method embodiment shown in Figure 7. Referring to Figure 8, this tactile media data processing device can include the following units:

[0222] An acquisition unit 801 is configured to acquire the duration of an instantaneous tactile signal in a tactile medium;

[0223] a processing unit 802, configured to generate representation information of the transient tactile signal according to the duration of the transient tactile signal;

[0224] The processing unit 802 is further configured to encapsulate the representation information of the instantaneous tactile signal into a media format file of the tactile media.

[0225] In one implementation, the tactile media exchange format file includes representation information of the instantaneous tactile signal; the processing unit 802 is configured to encapsulate the representation information of the instantaneous tactile signal into the media format file of the tactile media, specifically performing the following steps: encoding the tactile media exchange format file to obtain a bitstream file of the tactile media; and encapsulating the bitstream file of the tactile media to obtain a media format file of the tactile media.

[0226] In one implementation, the processing unit 802 is further configured to perform the following step: adding representation information of the instantaneous haptic signal to a bit stream file of the haptic media.

[0227] In one implementation, the processing unit 802 is further configured to perform the following step: adding the representation information of the instantaneous haptic signal to the media format file of the haptic media.

[0228] According to another embodiment of the present application, the various units in the tactile media data processing device shown in Figure 8 can be individually or entirely combined into one or several other units to form a structure, or one (or some) of the units can be further divided into multiple functionally smaller units to form a structure, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In actual applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the tactile media data processing device can also include other units. In actual applications, these functions can also be assisted by other units and can be implemented by the collaboration of multiple units.

[0229] According to another embodiment of the present application, a tactile media data processing device as shown in FIG. 8 and the tactile media data processing method of the present application embodiment can be constructed by running a computer program capable of executing some or all of the steps involved in the method shown in FIG. 7 on a general-purpose computing device, such as a computer, including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on a computer-readable storage medium, for example, and loaded into the computing device via the computer-readable storage medium and executed therein.

[0230] In an embodiment of the present application, representation information of instantaneous tactile signals in tactile media is defined, and the representation information of instantaneous tactile signals can be used to indicate the duration of the instantaneous tactile signals; that is, through the representation information of the tactile signals, the presentation form of the instantaneous tactile signals in the tactile media can be accurately defined, thereby accurately presenting the instantaneous tactile signals in the tactile media and improving the consumption experience of the tactile media.

[0231] Please refer to Figure 9, which is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device shown in Figure 9 includes at least a processor 901, an input interface 902, an output interface 903, and a computer-readable storage medium 904. The processor 901, input interface 902, output interface 903, and computer-readable storage medium 904 may be connected via a bus or other means.

[0232] The computer-readable storage medium 904 can be stored in a memory of a computer device. The computer-readable storage medium 904 is used to store a computer program, which includes computer instructions. The processor 901 is used to execute the computer program stored in the computer-readable storage medium 904. The processor 901 (also known as the CPU (Central Processing Unit)) is the computing core and control core of the computer device and is suitable for implementing the computer program, specifically loading and executing the computer program to implement the corresponding method flow or corresponding function.

[0233] The embodiment of the present application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the operating system of the computer device. In addition, a computer program suitable for being loaded and executed by the processor is also stored in the storage space. It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory (Non-Volatile Memory), such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.

[0234] In some embodiments, the computer device may be the consumer device 302 in the tactile media processing system shown in FIG. 3 , and the processor 901 may load and execute a computer program stored in the computer-readable storage medium 904 to implement the corresponding steps in the data processing method for tactile media shown in FIG. 5 .

[0235] In other embodiments, the computer device may be the service device 301 in the tactile media processing system shown in FIG. 3 , and the processor 901 may load and execute a computer program stored in the computer-readable storage medium 904 to implement the corresponding steps in the data processing method for tactile media shown in FIG. 7 .

[0236] In an embodiment of the present application, representation information of instantaneous tactile signals in tactile media is defined, and the representation information of instantaneous tactile signals can be used to indicate the duration of the instantaneous tactile signals; that is, through the representation information of the tactile signals, the presentation form of the instantaneous tactile signals in the tactile media can be accurately defined, thereby accurately presenting the instantaneous tactile signals in the tactile media and improving the consumption experience of the tactile media.

[0237] The present application also provides a computer program product, including a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the aforementioned tactile media data processing method.

[0238] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

Claims

1. A method for processing tactile media data, characterized in that: The method is performed by a consumer device, and the method includes: Acquiring representation information of an instantaneous tactile signal in a tactile medium, wherein the representation information of the instantaneous tactile signal is used to indicate a duration of the instantaneous tactile signal; The transient haptic signal in the haptic medium is presented according to the representation information of the transient haptic signal.

2. The method according to claim 1, characterized in that The step of obtaining the representation information of the instantaneous tactile signal in the tactile medium includes: Obtaining an exchange format file of the tactile media; The representation information of the instantaneous tactile signal is obtained from the exchange format file of the tactile medium.

3. The method according to any one of claims 1 to 2, characterized in that: The tactile media exchange format file includes a data structure of a tactile pattern of the tactile media, the data structure of the tactile pattern is used to define attribute information of the tactile pattern of the tactile media, and the data structure of the tactile pattern includes a short duration field; The representation information of the transient haptic signal includes the short duration field, and the short duration field is used to indicate the duration of the transient haptic signal in the current haptic mode.

4. The method according to any one of claims 1 to 3, characterized in that: The data structure of the tactile pattern further includes a tactile pattern type field, and the tactile pattern type field is used to indicate the type of the tactile pattern of the tactile media; When the tactile mode type field indicates that the type of the tactile mode of the tactile medium is vibration tactile, the short duration field is used to indicate that the duration of the instantaneous tactile signal in the current vibration tactile mode is the number of times the vibration motor vibrates at the reference frequency.

5. The method according to any one of claims 1 to 4, characterized in that: The exchange format file of the tactile media includes a data structure of a knowledge tactile event of the tactile media, the data structure of the knowledge tactile event is used to define attribute information of the knowledge tactile event of the tactile media, the data structure of the knowledge tactile event includes a tactile event type field, the tactile event type field is used to indicate an event type of the knowledge tactile event of the tactile media, and the event type includes an instantaneous event or a continuous event; When the tactile event type field indicates that the event type of the knowledge tactile event of the tactile media is the instantaneous event, and the data structure of the knowledge tactile event further includes a duration field, the representation information of the instantaneous tactile signal includes the duration field, and the duration field is used to indicate the duration of all instantaneous tactile events in the current tactile mode; There is a knowledge tactile event in a tactile pattern, which has the duration field and the event type is the instantaneous event.

6. The method according to any one of claims 1 to 5, characterized in that: When the haptic event type field indicates that the event type of the knowledge haptic event of the haptic media is the continuous event, the duration field is used to indicate the duration of the corresponding knowledge haptic event.

7. The method according to any one of claims 1 to 6, characterized in that: The data structure of the knowledge tactile event is determined according to the data structure of the tactile event, the data structure of the tactile event includes a reference knowledge event field, the data structure of the knowledge tactile event does not include the reference knowledge event field, the reference knowledge event field is used to indicate the knowledge tactile event identifier corresponding to the tactile event when the event type of the tactile event is a reference event; The data structure of the tactile event includes a relative time field, while the data structure of the known tactile event does not include the relative time field, wherein the relative time field is used to indicate a time offset of the tactile event.

8. The method according to any one of claims 1 to 7, characterized in that: The data structure of the knowledge tactile event is determined according to the data structure of the tactile event, and the data structure of the tactile event includes an amplitude field, a reference frequency field, and an event component field; when the tactile event type field indicates that the event type of the knowledge tactile event of the tactile media is the instantaneous event, and the data structure of the knowledge tactile event includes a duration field, the data structure of the knowledge tactile event does not include the amplitude field, the reference frequency field, and the event component field; The amplitude field is used to indicate the maximum amplitude value of the signal of the tactile event; the reference frequency field is used to indicate the signal reference frequency of the tactile event; the event component field is used to indicate a tactile event component element list, and the tactile event component element list is used to indicate the components of the tactile event, and the components of the tactile event include the time domain component or the frequency domain component of the tactile event.

9. The method according to any one of claims 1 to 8, characterized in that The tactile media exchange format file includes a data structure of a tactile event of the tactile media, the data structure of the tactile event is used to define attribute information of the tactile event of the tactile media, the data structure of the tactile event includes a tactile event type field, the tactile event type field is used to indicate an event type of the tactile event of the tactile media, and the event type includes a reference event or a common event; When the haptic event type field indicates that the event type of the haptic event of the haptic medium is the reference event, the haptic event The data structure further includes a reference knowledge event field, and the representation information of the instantaneous tactile signal includes the reference knowledge event field; the reference knowledge event field is used to indicate a knowledge tactile event identifier corresponding to the tactile event whose event type is the reference event; The duration of the tactile event whose event type is the reference event is the same as the duration of the knowledge tactile event indicated by the corresponding knowledge tactile event identifier.

10. The method according to any one of claims 1 to 9, characterized in that: When the tactile event type field indicates that the event type of the tactile event of the tactile media is the common event, the data structure of the tactile event further includes a relative time field, and the relative time field is used to indicate a time offset of the tactile event.

11. The method according to any one of claims 1 to 10, characterized in that: The step of obtaining the representation information of the instantaneous tactile signal in the tactile medium includes: Acquire a bitstream file of the tactile media; The representation information of the instantaneous haptic signal is obtained from the bit stream file of the haptic medium.

12. The method according to any one of claims 1 to 11, characterized in that: The bitstream file of the tactile media includes a tactile mode metadata packet, wherein the tactile mode metadata packet is obtained by encoding the tactile mode metadata; the tactile mode metadata packet of the tactile media includes a short duration field; The representation information of the transient haptic signal includes the short duration field, and the short duration field is used to indicate the duration of the transient haptic signal in the current haptic mode.

13. The method according to any one of claims 1 to 12, characterized in that: The tactile mode metadata package further includes a tactile mode type field, and the tactile mode type field is used to indicate the type of the tactile mode of the tactile media; When the tactile mode type field indicates that the type of the tactile mode of the tactile medium is vibration tactile, the short duration field is used to indicate that the duration of the instantaneous tactile signal in the current vibration tactile mode is the number of times the vibration motor vibrates at the reference frequency.

14. The method according to any one of claims 1 to 13, characterized in that: The bit stream file of the tactile media includes a knowledge tactile event metadata package, and the knowledge tactile event metadata package is obtained by encoding the knowledge tactile event metadata; The knowledge tactile event metadata package includes an event type field and a duration flag field; the event type field is used to indicate the event type of the knowledge tactile event, and the event type includes an instantaneous event or a continuous event; The duration flag field is used to indicate whether to indicate the duration of the knowledge tactile event; If the duration flag field indicates that the duration of the knowledge tactile event needs to be indicated, the knowledge tactile event metadata package further includes a duration field, and the representation information of the instantaneous tactile signal includes the duration field; if the event type field indicates that the event type of the knowledge tactile event of the tactile media is the instantaneous event, the duration field is used to indicate the duration of all instantaneous tactile signals in the current tactile mode; There is a knowledge tactile event in a tactile pattern, which has the duration field and the event type is the instantaneous event.

15. The method according to any one of claims 1 to 14, characterized in that: When the event type field indicates that the event type of the knowledge tactile event of the tactile media is the continuous event, the duration field is used to indicate the duration of the corresponding knowledge tactile event.

16. The method according to any one of claims 1 to 15, characterized in that: The step of obtaining the representation information of the instantaneous tactile signal in the tactile medium includes: Acquire a media format file of the tactile media; The representation information of the instantaneous tactile signal is obtained from a media format file of the tactile media.

17. The method according to any one of claims 1 to 16, characterized in that: The media format file of the tactile media includes a tactile pattern description data box, wherein the tactile pattern description data box is used to indicate metadata information of the tactile pattern of the tactile media; the tactile pattern description data box includes a transient duration field; The representation information of the transient haptic signal includes the short duration field, and the short duration field is used to indicate the duration of the transient haptic signal in the current haptic mode.

18. The method according to any one of claims 1 to 17, characterized in that: The tactile pattern description data box further includes a tactile pattern type field, and the tactile pattern type field is used to indicate the type of the tactile pattern of the tactile media; When the tactile mode type field indicates that the type of the tactile mode of the tactile medium is vibration tactile, the short duration field is used to indicate that the duration of the instantaneous tactile signal in the current vibration tactile mode is the number of times the vibration motor vibrates at the reference frequency.

19. The method according to any one of claims 1 to 18, characterized in that: The presenting the instantaneous tactile signal in the tactile medium according to the representation information of the instantaneous tactile signal comprises: The representation information of the instantaneous tactile signal is rendered to obtain the instantaneous tactile signal.

20. A method for processing tactile media data, characterized in that: The method is performed by a service device, and includes: Obtaining the duration of an instantaneous tactile signal in a tactile medium; generating representation information of the instantaneous tactile signal according to the duration of the instantaneous tactile signal; The representation information of the instantaneous haptic signal is encapsulated into a media format file of the haptic media.

21. The method of claim 20, wherein: The tactile media exchange format file includes the representation information of the instantaneous tactile signal; the step of encapsulating the representation information of the instantaneous tactile signal into the media format file of the tactile media includes: Encoding the exchange format file of the tactile media to obtain a bit stream file of the tactile media; The bit stream file of the tactile media is packaged to obtain a media format file of the tactile media.

22. The method according to any one of claims 20 to 21, characterized in that: The method further comprises: The representation information of the transient haptic signal is added to a bit stream file of the haptic medium.

23. The method according to any one of claims 20 to 22, characterized in that: The method further comprises: The representation information of the transient haptic signal is added to a media format file of the haptic media.

24. A tactile media data processing device, characterized in that: include: an acquisition unit, configured to acquire representation information of an instantaneous tactile signal in a tactile medium, wherein the representation information of the instantaneous tactile signal is used to indicate a duration of the instantaneous tactile signal; A processing unit is used to present the instantaneous tactile signal in the tactile medium according to the representation information of the instantaneous tactile signal.

25. A tactile media data processing device, characterized in that: include: an acquisition unit, used for acquiring the duration of the instantaneous tactile signal in the tactile medium; a processing unit, configured to generate representation information of the instantaneous tactile signal according to the duration of the instantaneous tactile signal; The processing unit is further used to encapsulate the representation information of the instantaneous tactile signal into a media format file of the tactile media.

26. A computer device, characterized in that: The computer device comprises: a processor suitable for implementing a computer program; A computer-readable storage medium storing a computer program, wherein the computer program is suitable for being loaded by the processor and executing the tactile media data processing method according to any one of claims 1 to 19, or the tactile media data processing method according to any one of claims 20 to 23.

27. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the tactile media data processing method according to any one of claims 1 to 19, or the tactile media data processing method according to any one of claims 20 to 23.

28. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method for processing tactile media data as claimed in any one of claims 1 to 19 or the method for processing tactile media data as claimed in any one of claims 20 to 23 is implemented.

Images