How to generate a sparse ISOBMFF haptic track

Abstract

A haptic signal processing method, apparatus, and system are provided. The processing process can include receiving a haptic track including multiple types of Moving Picture Experts Group (MPEG) Immersive Haptic Stream (MIHS) units. The processing process can include obtaining a first type of MIHS unit including haptic information from the haptic track and a second type of MIHS unit from the haptic track, the second type of MIHS unit being an empty unit including only duration information. The processing can include determining quiet periods of the haptic track based on the duration information in the second type of MIHS unit.

Description

Technical Field

[0001]

[0001] Cross - reference to Related Applications This application claims priority to U.S. Provisional Application No. 63 / 416,790, filed on October 17, 2022, and U.S. Application No. 18 / 487,743, filed on October 16, 2023, the disclosures of which are hereby incorporated by reference in their entirety.

[0002]

[0002] Technical Field This disclosure relates to a group of advanced video coding technologies. More specifically, this disclosure relates to encoding and decoding haptic experiences for multimedia presentations.

Background Art

[0003]

[0003] Haptic experience is becoming part of multimedia presentations. In applications where multimedia presentations include haptic aspects, haptic signals can be delivered to a device or a wearable device, and users can feel haptic (or skin sensations) during the use of the application in conjunction with visual and / or auditory media experiences.

[0004]

[0004] Recognizing the growing popularity of haptic experiences in multimedia presentations, the Motion Picture Experts Group (MPEG) has started working on the transmission of compressed haptic signals in ISO - based media file format (ISOBMFF), in addition to compression standards for haptics (both MPEG - DASH and MPEG - I).

[0005]

[0005] In many applications, haptic effects are sparse while visual and auditory tracks are continuous. For example, during short durations of a media presentation, haptic media effects need to be rendered along with audiovisual samples, but at other times, the haptic track is "quiet". Current ISOBMFF transmission of haptic signals does not address the quiet periods of the haptic track. Therefore, a solution is needed to address this problem. [Overview of the project]

[0006]

[0006] According to the embodiment, it is possible to provide a method for decoding sparse haptic data.

[0007] The method involves receiving a haptic track containing multiple types of Video Expert Group (MPEG) immersive haptics stream (MIHS) units; Steps include acquiring a first-type MIHS unit containing tactile information from a haptic track; A step of obtaining a second type MIHS unit from a haptic track, wherein the second type MIHS unit is an empty unit containing only duration information; and The procedure may include a step of determining the quiescent period of the haptic track based on the duration information in the second type of MIHS unit.

[0008]

[0007] According to one embodiment, it is possible to provide a device for decoding sparse haptic data. The device may include at least one memory configured to store program code and at least one processor configured to read program code and operate as directed by the program code.

[0009] The program code is a first receive code configured to cause at least one processor to perform the step of receiving a haptic track containing multiple types of Video Expert Group (MPEG) Immersive Haptics Stream (MIHS) units; A first acquisition code configured to cause at least one processor to perform the step of acquiring a first type of MIHS unit containing haptic information from a haptic track; A second acquisition code configured to cause at least one processor to perform the step of acquiring a second type MIHS unit from a haptic track, wherein the second type MIHS unit is an empty unit containing only duration information; and The system may include a first decision code configured to cause at least one processor to perform the step of determining the quiescent period of a haptic track based on duration information in a second type of MIHS unit.

[0010]

[0008] According to the embodiment, it is possible to provide a non-temporary computer-readable medium for storing computer instructions. The instructions may include one or more instructions, and when one or more instructions are executed by one or more processors of the device for decoding sparse haptic data, the instructions are sent to one or more processors: Steps include receiving a haptic track containing multiple types of Video Expert Group (MPEG) Immersive Haptics Stream (MIHS) units; Steps include acquiring a first-type MIHS unit containing tactile information from a haptic track; A step of obtaining a second type MIHS unit from a haptic track, wherein the second type MIHS unit is an empty unit containing only duration information; and Based on the duration information in the second type of MIHS unit, the system performs a step to determine the quiescent period of the haptic track. [Brief explanation of the drawing]

[0011]

[0009] Further features, properties, and various advantages of the disclosed subject matter will become clearer from the following detailed description and accompanying drawings. [Figure 1]

[0010] Figure 1 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment of the present disclosure. [Figure 2]

[0011] Figure 2 is a schematic diagram of a simplified block diagram of a streaming system according to an embodiment of the present disclosure. [Figure 3]

[0012] Figure 3 is a schematic diagram of a simplified block diagram of a haptic encoder according to an embodiment of the present disclosure. [Figure 4]

[0013] Figure 4 is a schematic diagram of a simplified block diagram of a haptic decoder and haptic renderer according to an embodiment of the present disclosure. [Figure 5]

[0014] Figure 5 is an illustrative diagram of an MIHS unit or haptic sample showing a haptic track having sparse characteristics according to an embodiment of the present disclosure. [Figure 6]

[0015] Figure 6 is an exemplary flowchart illustrating the process for decoding haptic data according to an embodiment of the present disclosure. [Figure 7]

[0016] Figure 7 shows a computer system suitable for implementing an embodiment. [Modes for carrying out the invention]

[0012]

[0017] According to one aspect of the present disclosure, a method, system, and non-temporary storage medium for parallel processing of dynamic mesh compression are provided. Embodiments of the present disclosure can also be applied to static meshes.

[0013]

[0018] An embodiment of the present disclosure for implementing the encoding and decoding structure of the present disclosure will be described with reference to Figure 1-2.

[0014]

[0019] Figure 1 shows a simplified block diagram of a communication system 100 according to one embodiment of the present disclosure. The system 100 may include at least two terminals 110, 120 interconnected via a network 150. In the case of one-way data transmission, the first terminal 110 may encode video data, which may include mesh data at its local location, for transmission to the other terminal 120 via the network 150. The second terminal 120 may receive the encoded video data from the other terminal via the network 150, decode the encoded data, and display the restored video data. One-way data transmission may be common in media service applications and the like.

[0020] Figure 1 shows a second pair of terminals 130,140 provided to support the bidirectional transmission of coded video, for example, during a video conference. For bidirectional data transmission, each terminal 130,140 is capable of coding video data captured at its local location for transmission to other terminals over the network 150. Each terminal 130,140 is also capable of receiving coded video data transmitted by other terminals, decoding the coded data, and displaying the restored video data on a local display device.

[0015]

[0021] In FIG. 1, terminals 110 - 140 may be, for example, a server, a personal computer, a smartphone, and / or any other type of terminal. For example, the terminals (110 - 140) may be a laptop computer, a tablet computer, a media player, and / or a dedicated video conferencing device. Network 150 represents any number of networks that carry coded video data between terminals 110 - 140 and includes, for example, wired and / or wireless communication networks. Communication network 150 can exchange data over circuit - switched channels and / or packet - switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this description, the architecture and topology of network 150 may not be important for the operation of the present disclosure, unless otherwise described below.

[0016]

[0022] FIG. 2 shows the arrangement of video encoders and decoders in a streaming environment as an example of an application for the disclosed subject matter. The disclosed subject matter can be used with other video - enabled applications, including, for example, video conferencing, digital TV, storage of compressed video on digital media (including CDs, DVDs, memory sticks, etc.).

[0017]

[0023] As shown in FIG. 2, streaming system 200 can include a capture subsystem 213 that includes a video source 201 and an encoder 203. Streaming system 200 may further include at least one streaming server 205 and / or at least one streaming client 206.

[0018]

[0024] The video source 201 can, for example, create a stream 202 that includes a 3D mesh and metadata associated with the 3D mesh. The video source 201 may include, for example, a 3D sensor (e.g., a depth sensor) or 3D imaging technology (e.g., a digital camera), and a computing device configured to generate a 3D mesh using data received from the 3D imaging technology or 3D sensor. The sample stream 202, which may have a larger data volume compared to the encoded video bitstream, can be processed by an encoder 203 coupled to the video source 201. The encoder 203 includes hardware, software, or a combination thereof, and can operate or implement aspects of the disclosed subject matter as described in detail below. The encoder 203 can also generate an encoded video bitstream 204. The encoded video bitstream 204 may have a smaller data volume compared to the uncompressed stream 202 and can be stored in the streaming server 205 for future use. One or more streaming clients 206 can access the streaming server 205 to retrieve a video bitstream 209 which may be a copy of the encoded video bitstream 204.

[0019]

[0025] The streaming client 206 may include a video decoder 210 and a display 212. The video decoder 210 can, for example, decode a video bitstream 209, which is an input copy of an encoded video bitstream 204, and generate an output video sample stream 211 that can be rendered on the display 212 or another rendering device (not shown). In some streaming systems, the video bitstreams 204,209 may be encoded according to a specific video coding / compression standard.

[0020]

[0026] An embodiment of the present disclosure that implements the haptic encoder 300 and the haptic decoder 350 will be described with reference to Figure 3-4.

[0021]

[0027] As shown in Figure 3, the haptic encoder 300 is capable of receiving both descriptive and waveform haptic data. Therefore, the haptic encoder 300 can handle three types of input files: .ohm metadata files (Object Haptic Metadata, ohm, which is a text file format related to haptic metadata), Descriptive haptic files (.ivs, .ahap, and .hjif) or Waveform PCM file (.wav) It is possible to process this. An example of descriptive data is: .ahap (Apple Haptic and Audio Pattern) is a JSON-like file format from Apple that specifies haptic patterns (representing the expected haptic output with a group of modulated continuous signals and a group of modulated parameterized transients). From Immersion, .ivs (representing the expected haptic output by a set of base effects parameterized by a set of parameters), or The proposed MPEG format may include the .hjif (Haptics JSON Interchange Format) file. An example of a waveform pulse-code modulation (PCM) signal can include a .ohm input file containing metadata information.

[0022]

[0028] According to the embodiment, the haptic encoder 300 can process two types of input files in different ways. For descriptive content, the haptic encoder 300 can semantically analyze the input and (if necessary) transcode the data into a proposed coded representation.

[0023]

[0029] According to the embodiment, the .ohm metadata input file may contain a description of the haptic system and setup. In particular, it may contain the names of each relevant haptic file (either descriptive or PCM) along with a description of the signal. A mapping between each channel of the signal and a target area of ​​the user's body is also provided. For the .ohm metadata input file, the haptic encoder performs metadata extraction by searching for the relevant haptic file from the URI, encoding it based on its type, and extracting metadata from the .ohm file and mapping it to metadata information in the data model.

[0024]

[0030] According to the embodiment, descriptive haptic files (e.g., .ivs, .ahap, and .hjif) may be encoded by a simple process. The haptic encoder 300 first specifically identifies the input format. If the input format is an .hjif file, transcoding is not necessary, and the file can be further edited, compressed into a binary format, and finally packetized into an MIHS stream. If an ahap or .ivs input file is used, transcoding is required. The haptic encoder 301 first semantically parses the input file information and transcodes it so that it is formatted into a selected data model. After transcoding, the data can be written (exported) as an .hjif file, an .hmpg binary file, or an MIHS stream.

[0025]

[0031] According to the embodiment, the haptic encoder 300 can perform signal analysis to interpret the signal structure of a .wav file and convert it into a proposed encoded representation. In the case of waveform PCM content, the signal analysis process may be divided into two sub-processes by the haptic encoder 300. After performing frequency band decomposition on the signal, in the first sub-process, the low frequencies can be encoded using a keyframe extraction process. The low-frequency bands are then reconstructed, and the error between this signal and the original low-frequency signal can be calculated. This remaining signal is added to the original high-frequency bands before encoding using wavelet transform, and encoding using wavelet transform is the second sub-process. According to the embodiment, when multiple low-frequency bands are used, the residual errors from all low-frequency bands are added to the high-frequency bands before encoding. In embodiments where multiple high-frequency bands are used, the residual errors from the low-frequency bands are added to the first high-frequency band before encoding.

[0026]

[0032] According to the embodiment, keyframe extraction includes extracting the low-frequency band from the frequency band decomposition and analyzing its contents in the time domain. According to the embodiment, wavelet processing may include extracting the high-frequency band from the frequency band decomposition and low-frequency residuals and dividing it into blocks of equal size. These equal-sized signal blocks are then analyzed using a psychohaptic model. Lossy compression (non-lossless compression) can be applied by wavelet transforming the blocks and quantizing them, with the assistance of the psychohaptic model. Finally, each block is stored as an individual effect in a single band, which is then formatted. Binary compression can be applied using appropriate coding techniques, such as the SPIHT (Set partitioning in hierarchical trees) algorithm and arithmetic coding (AC), to apply lossless compression.

[0027]

[0033] As shown in Figure 3, the haptic encoder 300 can be configured to encode descriptive and quantized haptic data in three formats, namely, Exchange format (.hjif), Binary compression format (.hmpg), and Streaming formats (e.g., MPEG immersive haptic stream (MIHS)) It is possible to output the following. The hjif format is a human-readable format based on JSON, which can be easily parsed and manually edited, making it an ideal exchange format, especially when designing / creating content. For distribution purposes, .hjif data can be compressed into a more memory-efficient binary .hmpg bitstream. This compression can be non-lossless, and various parameters affect the encoding depth of the amplitude and frequency that make up the bitstream. For streaming purposes, the data can be compressed and packetized into an MPEG-I haptic stream (MIHS). The three formats mentioned above have complementary purposes, and non-lossless one-to-one conversions can occur between them.

[0028]

[0034] As shown in Figure 4, the haptic decoder 350 can accept either the .hmpg compressed binary file format or the MIHS bitstream as input. The haptic decoder 350 can output the .hjif exchange format, which can be used directly for rendering. Both input formats can be decompressed to extract both metadata and the data itself from the file and map it to a selected data structure. The data can then be exported in .hjif format to the haptic renderer 380.

[0029]

[0035] As shown in Figure 4, the renderer 380 includes a synthesizer. The synthesizer can render haptic data from an hjif input file into a PCM output file. Rendering and / or synthesizing is informative. According to the embodiment, the synthesizer analyzes the input file and performs a high-level synthesis distribution among vectors, wavelets, etc. The synthesis process then descends to the bandwidth components of the codec from which the synthesis process is invoked. Then, all bandwidths of a given channel are mixed by a simple addition operator to reproduce the desired haptic signal.

[0030]

[0036] According to the embodiment, the haptic experience defines the root of the hierarchical data model. It provides information about the file date and format version, describes the haptic experience, lists the various avatars (i.e., bodily representations) used when experiencing it, and defines all haptic perceptions.

[0031]

[0037] According to the embodiments, tactile signals may be encoded on multiple channels. In some embodiments, tactile channels may define signals represented at specific body positions using dedicated actuators / devices. Metadata stored at the channel level may include information such as the gain associated with the channel, the weights to be mixed, the desired body position for haptic feedback, and optionally a reference device and / or direction. Additional information such as a desired sampling frequency or sample count may also be provided. Finally, the haptic data of the channels is contained in a set of tactile bands defined by their frequency ranges. A tactile band describes the tactile signal of the channel within a given frequency range. Bands are defined by a sequential list of tactile effect types, each containing a set of keyframes. For any type of tactile band, the tactile effect can be defined using at least position (temporal or spatial) and type. Depending on the band type and effect type, additional properties may be specified, including phase, base signal, composition, and a number of sequential haptic keyframes describing the effect.

[0032]

[0038] According to the embodiments, a haptic data hierarchy is defined in this disclosure.

[0033] ●Tactile channels ○Tactile band ●Tactile effects

[0039] According to the embodiment, a self-contained stream format for transferring MPEG-I haptic data may use a packetized approach and may include the following two levels of packetization: An MPEG-I Haptic Stream (MIHS) unit that covers a duration and contains zero or more MIHS packets; MIHS packets containing metadata or haptic effect data. In embodiments, an MIHS unit may be referred to as a network abstraction layer unit associated with haptic data. In embodiments, an MIHS unit may be referred to as an MIHS sample associated with haptic data.

[0034]

[0040] According to the embodiment, each MIHS unit covers a non-overlapping duration of the haptic presentation time; that is, an MIHS unit starts from the end of the previous MIHS unit and covers a duration defined by its duration field. An MIHS unit follows the next MIHS unit unless it is the last MIHS unit of the haptic experience. All MIHS packets of an MIHS unit have the start time and duration of the MIHS unit they contain. MIHS units may be transmitted on a haptic track.

[0035]

[0041] The binary delivery format is occasionally encapsulated in the ISOBMFF file format for delivery. The binary haptic delivery format requires that the ISOBMFF samples be occasionally divided in the form of MIHS units. Each ISOBMFF sample and / or MIHS unit is capable of covering information for a certain duration, and the samples do not overlap in time. The current encapsulation proposal assumes that all ISOBMFF samples contain the binary haptic delivery. However, as mentioned above, haptic effects are sparse while visual and audio tracks are consecutive; for example, during short durations of a media presentation, haptic media effects need to be rendered along with audiovisual samples, but at other times, the haptic track is "quiet". Therefore, there is a need for additional methods to more efficiently signal and / or process the "quiet" portions of the haptic track.

[0036]

[0042] According to aspects of this disclosure, one or more empty haptic samples (also known as MIHS units) are added to the ISOBMFF haptic technology, resulting in more efficient signaling and decoding of durations during which there are no haptic signals to be represented on the ISOBMFF haptic track.

[0037]

[0043] Each ISOBMFF haptic track consists of one or more haptic samples (e.g., MIHS units). Each sample defines a haptic signal for a given duration. According to embodiments of the present disclosure, a new haptic sample is added that is different from the existing haptic samples, and the haptic track is empty for its duration, i.e., it indicates that there is no haptic effect on the renderer (also known as an empty MIHS unit).

[0038]

[0044] Such "quietness" instructions for haptic tracks improve efficiency by preventing unnecessary searches of haptic packets during the "quietness" duration. Embodiments of the present disclosure also support random access to haptic tracks, as this facilitates navigation to the next meaningful packet.

[0039]

[0045] Figure 5 is an illustrative figure showing how haptic samples with effects (e.g., MIHS units) and empty haptic samples (e.g., empty MIHS units) are used to represent quiescent periods within a haptic track. Quiescent periods are long periods without haptic effects.

[0040]

[0046] As shown in Figure 5, the haptic track according to this disclosure may have two types of haptic samples or MIHS units. A first type of MIHS unit may include a haptic sample that carries haptic delivery format information. A second type of MIHS unit may include an empty haptic sample that only has duration and does not contain tactile information.

[0041]

[0047] One of the advantages of the proposed disclosure is that knowledge of quiescent periods in a haptic track may allow for the fragmentation and re-fragmentation of the haptic track. Furthermore, knowing the quiescent periods in a haptic track would make the retrieval and delivery of haptic signals more efficient, for example, by not requiring the delivery of the haptic track during quiescent periods.

[0042]

[0048] For example, the start of a sensation may be defined as a common anchor point for all effects in the stream. For instance, the first effect may have position 0, and the positions of all other effects can be defined relative to the first position. Then, in the case of ISOBMFF carrying for a haptic channel, the position of the effect should be relative to the start time of the sample carrying the effect. Then, when a haptic channel is carried in ISOBMFF, the position of its effect needs to be adjusted. Similarly, after analyzing the ISOBMFF and before sending it to the haptic decoder, the position of the effect needs to be readjusted by adding the start time of the sample.

[0043]

[0049] In another or the same example, two types of ISOBMFF tracks may be provided: one where the start time is tracked as an anchor relative to the effect's position, and another where the sample start time is the anchor.

[0044]

[0050] In another or the same example, a sample structure in a haptics elementary stream may be defined. In this case, a haptic channel may consist of one or more samples / frames, and the timing of each effect in each sample / frame is relative to that sample.

[0045]

[0051] Embodiments of the present disclosure provide a method or process for encoding, decoding, or transporting compressed haptic signals in an ISOBMFF track, in which case the track may consist of two types of haptic samples, one having haptic binary information and the other having empty haptic information. The empty haptic information may only include the duration of the sample, and thus the representation of the quiescent period of a haptic track using such a representation enables more compact tracks, as well as enabling track manipulation at the file format level because the quiescent period of the track is marked. In embodiments, a parser and a file format packaging unit can use this information for file format manipulation without having to parse the haptic elementary stream.

[0046]

[0052] Figure 6 shows a process 600 for decoding sparse haptic data according to an embodiment.

[0047]

[0053] Operation 605 is capable of receiving haptic tracks containing more than one type of Video Expert Group (MPEG) Immersive Haptics Stream (MIHS) unit. In Operation 610, a first-type MIHS unit containing haptic information is acquired from the haptic track, and the haptic information within the first-type MIHS unit is in binary format.

[0048]

[0054] In Operation 615, a second type MIHS unit is acquired from a haptic track, and the second type MIHS unit is an empty unit containing only duration information. In embodiments, the first and second type MIHS units are signaled with a high-level syntax stream.

[0049]

[0055] In some embodiments, the duration information in the second type MIHS unit indicates the length of time without haptic effects, and the second type MIHS unit does not contain haptic information. Therefore, the second type MIHS unit represents the quiescent period of the haptic track. In embodiments, the quiescent period of the haptic track is used for fragmentation of the haptic track.

[0050]

[0056] In one embodiment, based on the determination of the quiescent period, it is required that no MIHS units be delivered during the quiescent period.

[0051]

[0057] In Operation 620, it is possible to determine the quiescent period of the haptic track based on the duration information in the second type of MIHS unit.

[0052]

[0058] Those skilled in the art will understand that the techniques described herein may be implemented on both the encoder and decoder sides. The techniques described above can be implemented as computer software using computer-readable instructions and can be physically stored on one or more computer-readable media. For example, Figure 7 shows a computer system (700) suitable for implementing a particular embodiment disclosed.

[0053]

[0059] Computer software can be coded using any suitable machine code or computer language that may be subject to assembly, compilation, linking, or similar mechanisms, to create code that includes instructions that can be executed directly by a computer's central processing unit (CPU), graphics processing unit (GPU), etc., or instructions that are executed via interpretation, microcode execution, etc.

[0054]

[0060] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, and Internet of Things devices.

[0055]

[0061] The components shown in Figure 7 with respect to the computer system 700 are illustrative and are not intended to imply any limitation on the scope of functionality or use of the computer software that realizes embodiments of this disclosure. Furthermore, the configuration of the components should not be construed as having any dependency or requirement on any one or combination of components shown in any non-limiting embodiment of the computer system 700.

[0056]

[0062] The computer system 700 may include certain human interface input devices. Such human interface input devices may respond to input from one or more human users, for example, through tactile input (e.g., keystrokes, swipes, data glove movements), auditory input (e.g., voice, applause), visual input (e.g., gestures), or olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., conversations, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), and video (e.g., 2D video, 3D video including stereoscopic pictures).

[0057]

[0063] The input human interface device may include one or more of the following (though only one of each is depicted): keyboard 701, mouse 702, trackpad 703, touchscreen 710, data glove, joystick 705, microphone 706, scanner 707, and camera 708.

[0058]

[0064] The computer system 700 may also include certain human interface output devices. Such human interface output devices can stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback via a touchscreen 710, data glove, or joystick 705, although there may also be tactile feedback devices that do not serve as input devices). For example, such devices may include auditory output devices (e.g., speakers 709, headphones (not shown)), visual output devices (e.g., screens 710 including CRT screens, LCD screens, plasma screens, OLED screens, each having or not having touch screen input functionality, each having or not having haptic feedback functionality, some of which may be capable of outputting two-dimensional visual output, three-dimensional or more output by means such as stereoscopic output; virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0059]

[0065] The computer system 700 may also include human-accessible storage devices and associated media, such as optical media including CD / DVD ROM / RW 720 using media 721 such as CD / DVD, thumb drives 722, removable hard drives or solid-state drives 723, legacy magnetic media such as tapes and floppy disks (not shown), and specialized ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0060]

[0066] Those skilled in the art will also understand that the term “computer-readable medium” as used in relation to the subject matter disclosed herein does not include a transmission medium, carrier wave, or other transient signal.

[0061]

[0067] The computer system 700 may also include interfaces to one or more communication networks. These networks may be, for example, wireless, wired, or optical. Furthermore, these networks may be local, wide-area, metropolitan, automotive, real-time, or latency-tolerant. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), wired or wireless wide-area digital networks for television (including cable TV, satellite TV, and terrestrial TV), and automotive industries including CANBus. Certain networks generally require external network interface adapters attached to specific general-purpose data ports or peripheral buses 749 (e.g., USB ports on the computer system 700); others are generally integrated into the core of the computer system 700 by being attached to system buses as described below (e.g., Ethernet interfaces are integrated within PC computer systems, and cellular network interfaces are integrated within smartphone computer systems). Using any of these networks, the computer system 700 can communicate with other entities. Such communication can be one-way receive-only (e.g., broadcast television), one-way transmit-only (e.g., CANbus to a specific CANbus device), or two-way, for example, to other computer systems using local or wide-area digital networks. Such communication may include communication to the cloud computing environment 755. Specific protocols and protocol stacks can be used for each of these networks and network interfaces, as described above.

[0062]

[0068] The aforementioned human interface device, human-accessible storage device, and network interface 754 can be mounted on the core 740 of the computer system 700.

[0063]

[0069] The core 740 may include one or more central processing units (CPUs) 741, graphics processing units (GPUs) 742, special programmable processing units in the form of field-programmable gate areas (FPGAs) 743, hardware accelerators 744 for specific tasks, etc. These devices, along with read-only memory (ROM) 745, random-access memory 746, and internal mass storage (e.g., internal non-user-accessible hard drives, SSDs, etc.) 747, can be connected via a system bus 748. In some computer systems, the system bus 748 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices can be connected directly to the core's system bus 748 or via a peripheral bus 749. Peripheral bus architectures include PCI, USB, etc. A graphics adapter 750 may be included in the core 740.

[0064]

[0070] The CPU 741, GPU 742, FPGA 743, and accelerator 744 can be combined to execute specific instructions that constitute the aforementioned computer code. The computer code can be stored in ROM 745 or RAM 746. Temporary data can be stored in RAM 746, while persistent data can be stored, for example, in internal mass storage 747. High-speed storage and retrieval of any memory device may be possible by utilizing cache memory, which can be closely associated with one or more CPUs 741, GPUs 742, mass storage 747, ROM 745, RAM 746, etc.

[0065]

[0071] Computer-readable media can have computer code thereon for performing various computer execution operations. The media and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be of a type that is well known and available to those skilled in the art in the field of computer software.

[0066]

[0072] As an example, and not limited to, a computer system architecture 700, specifically a computer system having a core 740, can provide functionality as a result of the operation of a processor (including CPUs, GPUs, FPGAs, accelerators, etc.) that runs software embodied in one or more tangible computer-readable media. Such computer-readable media can be media related to user-accessible mass storage as described above, as well as specific storage of the core 740 of a non-transient nature, such as the mass storage 747 or ROM 745 inside the core. Software that implements various embodiments of the present disclosure can be stored in such devices and executed by the core 740. The computer-readable media can include one or more memory devices or chips, depending on the specific needs. The software can cause the core 740 and in particular the processors (including CPUs, GPUs, FPGAs, etc.) therein to execute specific processes or specific parts of specific processes described herein, including defining data structures to be stored in RAM 746 and modifying such data structures according to processes defined by the software. As an addition or alternative, a computer system may provide functionality as a result of logic wired or otherwise incorporated within a circuit (e.g., accelerator 744), which may perform, in lieu of or in conjunction with, a particular process or a particular part of a particular process described herein. References to software may include logic, and vice versa, as appropriate. References to computer-readable media may include circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both, as appropriate. This disclosure encompasses any appropriate combination of hardware and software.

[0067]

[0073] While this disclosure describes several non-limiting embodiments, there are many modifications, substitutions, and alternative equivalents that fall within the scope of this disclosure. Therefore, it will be understood that many systems and methods, not expressly illustrated or described herein, that embody the principles of this disclosure and thus fall within its spirit and scope, can be devised by those skilled in the art.

[0068]

[0074] (Note 1) A method for decoding sparse haptic data, which is performed by at least one processor, and the method is: Steps include receiving a haptic track containing multiple types of Video Expert Group (MPEG) Immersive Haptics Stream (MIHS) units; A step of acquiring a first type MIHS unit, which is a non-empty unit containing tactile information, from the haptic track; A step of acquiring a second type MIHS unit from the haptic track, wherein the second type MIHS unit is an empty unit containing only duration information without tactile information; and A step of determining the quiescent period of the haptic track based on the duration information in the second type of MIHS unit; A method that includes this.

[0069] (Note 2) The method described in Appendix 1, wherein the duration information in the second type of MIHS unit indicates the length of time during which no haptic effect is present.

[0070] (Note 3) A method according to Appendix 1, further comprising the step of requesting that, based on the determination of the quiescent period, the MIHS unit not be delivered during the continuation of the quiescent period.

[0071] (Note 4) The method described in Appendix 1, wherein the second type of MIHS unit does not include the tactile information.

[0072] (Note 5) The method described in Appendix 1, wherein the second type of MIHS unit represents the quiescent period of the haptic track.

[0073] (Note 6) The method described in Appendix 1, wherein the first type MIHS unit and the second type MIHS unit are signaled with high-level syntax.

[0074] (Note 7) The method described in Appendix 1, wherein the tactile information contained in the first type MIHS unit is in binary format.

[0075] (Note 8) The method described in Appendix 1, wherein the quiescence period of the haptic track is used for fragmentation of the haptic track.

[0076] (Note 9) A device for decoding sparse haptic data: At least one memory configured to store program code; A processor configured to read the program code and operate as instructed by the program code; The program code includes: A first receive code configured to cause at least one processor to perform the step of receiving a haptic track comprising multiple types of Video Expert Group (MPEG) Immersive Haptics Stream (MIHS) units; A first acquisition code configured to cause at least one processor to perform the step of acquiring a first type of MIHS unit, which is a non-empty unit containing tactile information, from the haptic track; A second acquisition code configured to cause at least one processor to perform the step of acquiring a second type of MIHS unit from the haptic track, wherein the second type of MIHS unit is an empty unit containing only duration information without tactile information; and A first decision code configured to cause at least one processor to perform the step of determining the quiescent period of the haptic track based on the duration information in the second type of MIHS unit; A device including a device.

[0077] (Note 10) The apparatus described in Appendix 9, wherein the duration information in the second type MIHS unit indicates the length of time during which no tactile effect is present.

[0078] (Note 11) The apparatus as described in Appendix 9, wherein the program code further includes request code configured to cause the at least one processor to perform the step of requesting that, based on the determination of the quiescent period, the MIHS unit not be delivered during the continuation of the quiescent period.

[0079] (Note 12) In the apparatus described in Appendix 9, the second type of MIHS unit is an apparatus that does not include the tactile information.

[0080] (Note 13) The apparatus described in Appendix 9, wherein the second type MIHS unit represents the quiescent period of the haptic track.

[0081] (Note 14) The apparatus described in Appendix 9, wherein the first type MIHS unit and the second type MIHS unit are signaled in high-level syntax.

[0082] (Note 15) The apparatus described in Appendix 9, wherein the tactile information contained in the first type MIHS unit is in binary format.

[0083] (Note 16) A non-temporary, computer-readable storage medium for storing instructions, wherein the one or more instructions are executed by one or more processors of a device for decoding sparse haptic data, and then to the one or more processors: Steps include receiving a haptic track containing multiple types of Video Expert Group (MPEG) Immersive Haptics Stream (MIHS) units; A step of acquiring a first type MIHS unit, which is a non-empty unit containing tactile information, from the haptic track; A step of acquiring a second type MIHS unit from the haptic track, wherein the second type MIHS unit is an empty unit containing only duration information without tactile information; and A step of determining the quiescent period of the haptic track based on the duration information in the second type of MIHS unit; A non-temporary, computer-readable storage medium that enables execution of a function.

[0084] (Note 17) A non-temporary computer-readable storage medium as described in Appendix 16, wherein the duration information in the second type of MIHS unit indicates the length of time during which no tactile effect is present.

[0085] (Note 18) A non-temporary computer-readable storage medium as described in Appendix 16, wherein the instruction causes one or more processors to perform the step of requesting that no MIHS units be delivered during the continuation of the quiet period, based on the determination of the quiet period.

[0086] (Note 19) In the non-temporary computer-readable storage medium described in Appendix 16, the second type of MIHS unit is a storage medium that does not contain tactile information.

[0087] (Note 20) A non-temporary computer-readable storage medium as described in Appendix 16, wherein the second type of MIHS unit represents the quiet period of the haptic track.

Claims

1. A method for decoding sparse haptic data, which is performed by at least one processor, the method being: Steps include receiving a haptic track containing multiple types of Video Expert Group (MPEG) Immersive Haptics Stream (MIHS) units; A step of acquiring a first type MIHS unit, which is a non-empty unit containing tactile information, from the haptic track; A step of acquiring a second type MIHS unit from the haptic track, wherein the second type MIHS unit is an empty unit containing only duration information without haptic information, and the duration information indicates the length of time during which no haptic effect is present; and A step of determining the quiescent period of the haptic track based on the duration information in the second type of MIHS unit; A method that includes this.

2. A method according to claim 1, further comprising the step of requesting, based on the determination of the quiescent period, that no MIHS units be delivered during the continuation of the quiescent period.

3. The method according to claim 1, wherein the second type of MIHS unit does not include the tactile information.

4. The method according to claim 1, wherein the second type of MIHS unit represents the quiescent period of the haptic track.

5. The method according to claim 1, wherein the first type MIHS unit and the second type MIHS unit are signaled with high-level syntax.

6. The method according to claim 1, wherein the tactile information contained in the first type MIHS unit is in binary format.

7. A method according to claim 1, wherein the quiescence period of the haptic track is used for fragmentation of the haptic track.

8. A device for decoding sparse haptic data: At least one memory configured to store program code; A processor configured to read the program code and operate as instructed by the program code; An apparatus comprising, wherein the program code is configured to perform the method described in any one of claims 1 to 7.

9. A computer program that causes a computer processor to perform the method described in any one of claims 1 to 7.

10. A method for encoding sparse haptic data, which is performed by at least one processor, the method being: The steps include generating a haptic track containing multiple types of Video Expert Group (MPEG) Immersive Haptics Stream (MIHS) units and transmitting the haptic track to a decoder; The haptic track includes a first type of MIHS unit which is a non-empty unit containing tactile information, and a second type of MIHS unit which is an empty unit containing only duration information without tactile information, the duration information indicating the length of time during which no tactile effect is present. A method for determining the quiescent period of the haptic track based on duration information in the second type of MIHS unit.

Images