How to generate sparse ISOBMFF haptic tracks

Abstract

A haptic signal processing method, apparatus, and system are provided. The processing process can include receiving a haptic track including multiple types of Moving Picture Experts Group (MPEG) Immersive Haptic Stream (MIHS) units. The processing process can include obtaining a first type of MIHS unit including haptic information from the haptic track and a second type of MIHS unit from the haptic track, the second type of MIHS unit being an empty unit including only duration information. The processing can include determining quiet periods of the haptic track based on the duration information in the second type of MIHS unit.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 416,790, filed October 17, 2022, and U.S. Application No. 18 / 487,743, filed October 16, 2023, the disclosures of which are incorporated herein by reference in their entireties.

[0002]

[0002] Technical field This disclosure relates to a family of advanced video coding techniques, and more particularly to encoding and decoding haptic experiences for multimedia presentations. [Background technology]

[0003]

[0003] Haptic experiences are becoming part of multimedia presentations. In applications where the multimedia presentation includes haptic aspects, haptic signals can be delivered to a device or wearable device, allowing users to experience haptic (or skin) sensations while using the application in coordination with their visual and / or auditory media experience.

[0004]

[0004] Recognizing the growing popularity of haptic experiences in multimedia presentations, the motion picture experts group (MPEG) began working on transmission of compressed haptic signals in the ISO based media file format (ISOBMFF) in addition to compression standards for haptics (both MPEG-DASH and MPEG-I).

[0005] In many applications, haptic effects are sparse while the visual and auditory tracks are continuous; for example, haptic media effects need to be rendered along with audiovisual samples for short durations of a media presentation, while at other times the haptic track is "quiet." Current ISOBMFF transmission of haptic signals does not address quiet periods in the haptic track. Therefore, a solution is needed to address this issue. Summary of the Invention

[0006] According to an embodiment, it is possible to provide a method for decoding sparse haptic data.

[0007] The method includes receiving a haptic track including multiple types of Moving Picture Experts Group (MPEG) immersive haptics stream (MIHS) units; obtaining a first type of MIHS unit containing tactile information from the haptic track; obtaining a second type of MIHS unit from the haptic track, the second type of MIHS unit being an empty unit containing only duration information; and The method may include determining a quiet period of the haptic track based on the duration information in the second type MIHS unit.

[0008] According to an embodiment, an apparatus for decoding sparse haptic data can be provided, which can include at least one memory configured to store program code and at least one processor configured to read the program code and operate as directed by the program code.

[0009] The program code includes first receiving code configured to cause at least one processor to perform the steps of receiving a haptic track including multiple types of Moving Picture Experts Group (MPEG) Immersive Haptic Stream (MIHS) units; first acquisition code configured to cause at least one processor to perform the steps of acquiring a first type of MIHS unit including haptic information from the haptic track; second acquisition code configured to cause the at least one processor to execute the steps of: acquiring a second type of MIHS unit from the haptic track, the second type of MIHS unit being an empty unit that includes only duration information; and The method may include first decision code configured to cause at least one processor to perform a step of determining quiet periods of the haptic track based on the duration information in the second type MIHS unit.

[0010] According to an embodiment, a non-transitory computer-readable medium may be provided that stores computer instructions. The instructions may include one or more instructions that, when executed by one or more processors of a device for decoding sparse haptic data, cause the one or more processors to: receiving a haptic track including multiple types of Moving Picture Experts Group (MPEG) Immersive Haptic Stream (MIHS) units; obtaining a first type of MIHS unit containing tactile information from the haptic track; obtaining a second type of MIHS unit from the haptic track, the second type of MIHS unit being an empty unit containing only duration information; and Determining a quiet period of the haptic track based on the duration information in the second type MIHS unit. [Brief explanation of the drawings]

[0011]

[0009] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. [Figure 1] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a schematic diagram of a simplified block diagram of a streaming system according to an embodiment of the present disclosure. [Figure 3]

[0012] FIG. 3 is a schematic diagram of a simplified block diagram of a haptic encoder according to an embodiment of the present disclosure. [Figure 4]

[0013] FIG. 4 is a schematic diagram of a simplified block diagram of a haptic decoder and a haptic renderer according to an embodiment of the disclosure. [Figure 5]

[0014] FIG. 5 is an exemplary illustration of an MIHS unit or haptic sample showing a haptic track with sparse features according to an embodiment of the present disclosure. [Figure 6]

[0015] FIG. 6 is an exemplary flow chart illustrating a process for decoding haptic data according to an embodiment of the present disclosure. [Figure 7]

[0016] FIG. 7 is a diagram of a computer system suitable for practicing embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0012]

[0017] According to one aspect of the present disclosure, a method, system, and non-transitory storage medium for parallel processing of dynamic mesh compression are provided. Embodiments of the present disclosure may also be applied to static meshes.

[0013]

[0018] 1-2, an embodiment of the present disclosure for implementing the encoding and decoding structure of the present disclosure will be described.

[0014]

[0019] 1 illustrates a simplified block diagram of a communication system 100 according to one embodiment of the present disclosure. The system 100 may include at least two terminals 110, 120 interconnected via a network 150. In a unidirectional data transmission scenario, a first terminal 110 may code video data, which may include mesh data, at a local location for transmission to another terminal 120 via the network 150. A second terminal 120 may receive the coded video data of the other terminal from the network 150, decode the coded data, and display the recovered video data. Unidirectional data transmission may be common in media service applications, etc.

[0020] 1 shows a second pair of terminals 130, 140 configured to support bidirectional transmission of coded video, such as might occur during a video conference. For bidirectional transmission of data, each terminal 130, 140 is capable of coding video data captured locally for transmission to the other terminal over network 150. Each terminal 130, 140 is also capable of receiving coded video data transmitted by the other terminal, decoding the coded data, and displaying the recovered video data on a local display device.

[0015]

[0021] In FIG. 1 , terminals 110-140 may be, for example, servers, personal computers, smartphones, and / or any other type of terminal. For example, terminals 110-140 may be laptop computers, tablet computers, media players, and / or dedicated videoconferencing devices. Network 150 represents any number of networks that carry coded video data between terminals 110-140, including, for example, wired and / or wireless communication networks. Communication network 150 may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of network 150 may not be important to the operation of the present disclosure, unless otherwise described below.

[0016]

[0022] Figure 2 shows the arrangement of a video encoder and decoder in a streaming environment as an example application for the disclosed subject matter. The disclosed subject matter can be used with other video-enabled applications including, for example, video conferencing, digital TV, storage of compressed video on digital media (including CDs, DVDs, memory sticks, etc.), etc.

[0017]

[0023] 2, the streaming system 200 may include a capture subsystem 213 that includes a video source 201 and an encoder 203. The streaming system 200 may further include at least one streaming server 205 and / or at least one streaming client 206.

[0018]

[0024] The video source 201 can, for example, create a stream 202 that includes a 3D mesh and metadata associated with the 3D mesh. The video source 201 can include, for example, a 3D sensor (e.g., a depth sensor) or 3D imaging technology (e.g., a digital camera) and a computing device configured to generate the 3D mesh using data received from the 3D imaging technology or 3D sensor. The sample stream 202, which may have a large amount of data compared to an encoded video bitstream, can be processed by an encoder 203 coupled to the video source 201. The encoder 203 can include hardware, software, or a combination thereof and can enable or implement aspects of the disclosed subject matter, as described in detail below. The encoder 203 can also generate an encoded video bitstream 204. The encoded video bitstream 204 can have a smaller amount of data compared to the uncompressed stream 202 and can be stored on a streaming server 205 for future use. One or more streaming clients 206 can access the streaming server 205 to retrieve a video bitstream 209 , which may be a copy of the encoded video bitstream 204 .

[0019]

[0025] Streaming client 206 may include a video decoder 210 and a display 212. Video decoder 210 may decode video bitstream 209, which may be, for example, an input copy of encoded video bitstream 204, and generate an output video sample stream 211 that may be rendered on display 212 or another rendering device (not shown). In some streaming systems, video bitstreams 204, 209 may be encoded according to a particular video coding / compression standard.

[0020]

[0026] 3-4, embodiments of the present disclosure that implement haptic encoder 300 and haptic decoder 350 are described.

[0021]

[0027] 3, haptic encoder 300 is capable of receiving both descriptive and waveform haptic data. Thus, haptic encoder 300 accepts three types of input files: .ohm metadata files (Object Haptic Metadata (ohm), a text file format for haptic metadata), Descriptive haptic files (.ivs, .ahap, and .hjif) or Waveform PCM file (.wav) Examples of descriptive data are: .ahap (Apple Haptic and Audio Pattern) from Apple, a JSON-like file format for specifying haptic patterns (representing the desired haptic output in terms of a set of modulated continuous signals and a set of modulated parameterized transients); .ivs from Immersion (representing the expected haptic output with a set of base effects parameterized by a set of parameters), or Potentially includes the proposed MPEG format .hjif (Haptics JSON Interchange Format). An example of a waveform pulse-code modulation (PCM) signal may include an .ohm input file containing metadata information.

[0022]

[0028] According to an embodiment, haptic encoder 300 can process two types of input files differently: For descriptive content, haptic encoder 300 can semantically analyze the input and (if necessary) transcode the data into a proposed coded representation.

[0023]

[0029] According to an embodiment, the .ohm metadata input file may include a description of the haptic system and setup. In particular, it may include the name of each associated haptic file (either descriptive or PCM) along with a description of the signal. A mapping between each channel of the signal and a target location on the user's body is also provided. For the .ohm metadata input file, the haptic encoder performs metadata extraction by retrieving the associated haptic file from its URI, encoding it based on its type, and extracting metadata from the .ohm file and mapping it to metadata information in the data model.

[0024]

[0030] According to an embodiment, descriptive haptic files (e.g., .ivs, .ahap, and .hjif) may be encoded using a simple process. Haptic encoder 300 first specifically identifies the input format. If the input format is an .hjif file, no transcoding is necessary; the file can be further edited, compressed into a binary format, and finally packetized into an MIHS stream. If an ahap or .ivs input file is used, transcoding is required. Haptic encoder 301 first semantically analyzes the input file information and transcodes it so that it is formatted into the selected data model. After transcoding, the data can be exported as an .hjif file, an .hmpg binary file, or an MIHS stream.

[0025]

[0031] According to an embodiment, haptic encoder 300 can perform signal analysis to interpret the signal structure of a .wav file and convert it into the proposed coding representation. For waveform PCM content, the signal analysis process can be divided into two sub-processes by haptic encoder 300. After performing frequency band decomposition on the signal, in the first sub-process, the low frequencies can be coded using a keyframe extraction process. The low frequency band is then reconstructed, and the error between this signal and the original low frequency signal can be calculated. This residual signal is added to the original high frequency band before encoding using a wavelet transform, which is the second sub-process. According to an embodiment, if multiple low frequency bands are used, the residual error from all low frequency bands is added to the high frequency band before encoding. In an embodiment where multiple high frequency bands are used, the residual error from the low frequency band is added to the first high frequency band before encoding.

[0026]

[0032] According to an embodiment, keyframe extraction involves extracting a low-frequency band from the frequency band decomposition and analyzing its content in the time domain. According to an embodiment, wavelet processing may involve extracting a high-frequency band from the frequency band decomposition and the low-frequency residual and dividing it into equal-sized blocks. These equal-sized signal blocks are then analyzed with a psychohaptic model. Lossy compression (non-lossless compression) can be applied by wavelet transforming the blocks and quantizing them with the aid of the psychohaptic model. Finally, each block is saved as an individual effect in a single band, where it is formatted. Binary compression can be applied using appropriate coding techniques, such as the Set Partitioning in Hierarchical Trees (SPIHT) algorithm and Arithmetic Coding (AC), to apply lossless compression.

[0027]

[0033] As shown in FIG. 3, haptic encoder 300 can be configured to encode descriptive, quantized haptic data in three different formats: Interchange format (.hjif), Binary compressed format (.hmpg), and Streaming formats (e.g., MPEG immersive haptic stream (MIHS)) The hjif format is a human-readable format based on JSON and can be easily parsed and manually edited, making it an ideal exchange format, especially for content design and creation. For distribution purposes, .hjif data can be compressed into a more memory-efficient binary .hmpg bitstream. This compression can be lossless, and various parameters affect the amplitude and frequency coding depth that makes up the bitstream. For streaming purposes, data can be compressed and packetized into the MPEG-I Haptic Stream (MIHS). The three formats above have complementary purposes, and non-lossless one-to-one conversions can be performed between them.

[0028]

[0034] As shown in Figure 4, haptic decoder 350 can take as input either the .hmpg compressed binary file format or an MIHS bitstream. Haptic decoder 350 can output the .hjif interchange format, which can be used directly for rendering. Both input formats can undergo binary decompression to extract both the metadata and the data itself from the file and map it to a selected data structure. The data can then be exported to haptic renderer 380 in the .hjif format.

[0029]

[0035] As shown in FIG. 4, renderer 380 includes a synthesizer. The synthesizer can render haptic data from an hjif input file into a PCM output file. Rendering and / or synthesizing can be informative. According to an embodiment, the synthesizer analyzes the input file and performs high-level synthesis distribution between vectors, wavelets, etc. The synthesis process then descends to the band components of the codec in which the synthesis process is invoked. All bands of a given channel are then mixed with a simple additive operator to reproduce the desired haptic signal.

[0030]

[0036] According to an embodiment, a haptic experience defines the root of a hierarchical data model that provides information about the file date and format version, describes the haptic experience, lists the various avatars (i.e., body representations) used in experiencing it, and defines all haptic perceptions.

[0031]

[0037] According to an embodiment, haptic signals may be encoded on multiple channels. In some embodiments, a haptic channel may define a signal to be rendered at a specific body position using a dedicated actuator / device. Metadata stored at the channel level may include information such as the channel's associated gain, blending weight, desired body position of haptic feedback, and optionally a reference device and / or orientation. Additional information such as a desired sampling frequency or sample count may also be provided. Finally, a channel's haptic data is contained in a set of haptic bands defined by their frequency ranges. A haptic band describes the channel's haptic signal in a given frequency range. Bands are defined by a type and a sequential list of haptic effects, each containing a set of keyframes. For any type of haptic band, a haptic effect can be defined by at least its position (temporal or spatial) and type. Depending on the band type and effect type, additional properties may be specified, including phase, base signal, composition, and a number of consecutive haptic keyframes describing the effect.

[0032]

[0038] According to an embodiment, a haptic data hierarchy is defined in this disclosure.

[0033] ●Tactile Channel ○Tactile band ●Haptic effects

[0039] According to an embodiment, a self-contained stream format for transporting MPEG-I haptic data may use a packetized approach and may include two levels of packetization: An MPEG-I Haptic Stream (MIHS) unit, covering a duration and containing zero or more MIHS packets; MIHS packets that contain metadata or haptic effect data. In embodiments, the MIHS unit may be referred to as a network abstraction layer unit associated with the haptic data. In embodiments, the MIHS unit may be referred to as an MIHS sample associated with the haptic data.

[0034]

[0040] According to an embodiment, each MIHS unit covers a non-overlapping duration of the haptic presentation time, i.e., an MIHS unit starts from the end of the previous MIHS unit and covers the duration defined by its duration field. An MIHS unit continues with the next MIHS unit unless it is the last MIHS unit of the haptic experience. Every MIHS packet of an MIHS unit has the start time and duration of the MIHS unit it contains. MIHS units may be transmitted in a haptic track.

[0035]

[0041] Binary delivery formats are sometimes encapsulated in the ISOBMFF file format for delivery. The binary haptic delivery format requires that ISOBMFF samples are sometimes split into MIHS units. Each ISOBMFF sample and / or MIHS unit can cover information for a certain duration, and samples do not overlap in time. The current encapsulation proposal assumes that every ISOBMFF sample contains a binary haptic delivery. However, as mentioned above, haptic effects are sparse while the visual and audio tracks are continuous. For example, during short durations of a media presentation, haptic media effects need to be rendered along with the audiovisual samples, while at other times the haptic track is "quiet." Therefore, there is a need for additional methods to more efficiently signal and / or process the "quiet" portions of a haptic track.

[0036]

[0042] According to aspects of the present disclosure, one or more empty haptic samples (also known as MIHS units) are added to the ISOBMFF haptic technology, so that durations in which no haptic signal is to be represented in the ISOBMFF haptic track can be more efficiently signaled and decoded.

[0037]

[0043] Each ISOBMFF haptic track consists of one or more haptic samples (e.g., MIHS units). Each sample defines a haptic signal for a certain duration. According to embodiments of the present disclosure, new haptic samples (also known as empty MIHS units) that are different from the existing haptic samples are added to indicate that the haptic track is empty for that duration, i.e., has no haptic effect to the renderer.

[0038]

[0044] Such indication of haptic track "quietness" improves efficiency by preventing unnecessary searches for haptic packets within the "quiet" duration. Embodiments of the present disclosure also support random access of haptic tracks, making navigation to the next meaningful packet easier.

[0039]

[0045] 5 is an example diagram 500 illustrating the use of haptic samples with effects (e.g., MIHS units) and empty haptic samples (e.g., empty MIHS units) to indicate quiet periods in a haptic track. Quiet periods are long periods without haptic effects.

[0040]

[0046] As shown in Figure 5, a haptic track according to the present disclosure can have two types of haptic samples or MIHS units. The first type of MIHS unit can include a haptic sample that carries haptic delivery format information. The second type of MIHS unit can include an empty haptic sample that has only a duration and no haptic information.

[0041]

[0047] One advantage of the proposed disclosure is that knowledge of quiet periods in a haptic track may enable fragmentation and re-fragmentation of the haptic track. Furthermore, knowing the quiet periods in a haptic track may make retrieval and delivery of haptic signals more efficient, e.g., not requiring delivery of the haptic track during quiet periods.

[0042]

[0048] In one example, the start of a haptic experience may be defined as a common anchor point for all effects in a stream. For example, the first effect may have position 0, and the positions of all other effects may be defined relative to the first position. Then, for ISOBMFF transport of the haptic channel, the position of an effect should be relative to the start time of the sample that carries the effect. Then, when the haptic channel is transported with ISOBMFF, the position of the effect needs to be adjusted. Similarly, after parsing the ISOBMFF and before sending it to the haptic decoder, the position of the effect needs to be readjusted by adding the start time of the sample.

[0043]

[0049] In another or the same example, two types of ISOBMFF tracks may be provided: the first is a track that tracks the start time as an anchor for the position of the effect, and the second is a track where the sample start time is the anchor.

[0044]

[0050] In another or the same example, a sample structure in a haptics elementary stream may be defined, where a haptic channel may consist of one or more samples / frames, and the timing of each effect in each sample / frame is relative to that sample.

[0045]

[0051] According to an embodiment of the present disclosure, a method or process is provided for encoding, decoding, or conveying compressed haptic signals in an ISOBMFF track, where the track can consist of two types of haptic samples: one with haptic binary information and one with empty haptic information. The empty haptic information may only include the duration of the sample. Therefore, representing the quiet periods of a haptic track using such a representation not only allows for more compact tracks, but also enables manipulation of the track at the file format level because the quiet periods of the track are marked. In an embodiment, a parser and file format packaging unit can use this information for file format manipulation without having to parse the haptic elementary stream.

[0046]

[0052] FIG. 6 illustrates a process 600 for decoding sparse haptic data according to an embodiment.

[0047]

[0053] In operation 605, a haptic track including more than one type of Moving Picture Experts Group (MPEG) Immersive Haptic Stream (MIHS) unit can be received. In operation 610, a first type of MIHS unit containing haptic information is obtained from the haptic track, the haptic information in the first type of MIHS unit being in a binary format.

[0048]

[0054] In operation 615, a second type MIHS unit is obtained from the haptic track, the second type MIHS unit being an empty unit containing only duration information. In an embodiment, the first type MIHS unit and the second type MIHS unit are signaled in a high-level syntax stream.

[0049]

[0055] In some embodiments, the duration information in the second type MIHS unit indicates a length of time without a haptic effect, and the second type MIHS unit does not contain haptic information. Thus, the second type MIHS unit represents a quiet period in the haptic track. In some embodiments, the quiet period in the haptic track is used for segmenting the haptic track.

[0050]

[0056] In an embodiment, based on the quiet period determination, no delivery of MIHS units during the quiet period is required during the quiet period.

[0051]

[0057] In operation 620, a quiet period of the haptic track can be determined based on the duration information in the second type MIHS unit.

[0052]

[0058] Those skilled in the art will appreciate that the techniques described herein may be implemented on both the encoder and decoder sides. The techniques described above may be implemented as computer software using computer-readable instructions and may be physically stored on one or more computer-readable media. For example, Figure 7 illustrates a computer system (700) suitable for implementing certain disclosed embodiments.

[0053]

[0059] Computer software may be coded using any suitable machine code or computer language that may be subject to assembly, compilation, linking, or similar mechanisms to create code that contains instructions that may be executed directly by a computer central processing unit (CPU), graphics processing unit (GPU), etc., or that may go through interpretation, microcode execution, etc.

[0054]

[0060] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0055]

[0061] 7 for computer system 700 are illustrative and are not intended to suggest any limitation on the scope of functionality or application of the computer software implementing embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement related to any one or combination of components illustrated in the non-limiting embodiment of computer system 700.

[0056]

[0062] The computer system 700 may include certain human interface input devices that may respond to input by one or more human users, for example, via tactile input (e.g., keystrokes, swipes, data glove movements), auditory input (e.g., voice, clapping), visual input (e.g., gestures), or olfactory input (not shown). Human interface devices may also be used to capture certain media that do not necessarily involve direct conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still-image cameras), and video (e.g., two-dimensional video, three-dimensional video, including stereoscopic pictures).

[0057]

[0063] The input human interface devices may include one or more of the following (only one of each is depicted): a keyboard 701, a mouse 702, a trackpad 703, a touch screen 710, a data glove, a joystick 705, a microphone 706, a scanner 707, and a camera 708.

[0058]

[0064] The computer system 700 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen 710, data gloves, or joystick 705, although haptic feedback devices that do not serve as input devices may also exist). For example, such devices may include auditory output devices (e.g., speakers 709, headphones (not shown)), visual output devices (e.g., screens 710 including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output, three- or more-dimensional output by means such as stereoscopic output; virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0059]

[0065] The computer system 700 may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 720 using media such as CD / DVD 721, thumb drives 722, removable hard drives or solid state drives 723, legacy magnetic media (not shown) such as tape and floppy disks (not shown), and specialized ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0060]

[0066] Those skilled in the art will also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not encompass transmission media, carrier waves, or other transitional signals.

[0061]

[0067] The computer system 700 may also include interfaces to one or more communications networks. Networks may be, for example, wireless, wired, or optical. Networks may further be local, wide-area, metropolitan, automotive, real-time, delay-tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), TV wired or wireless wide-area digital networks (including cable TV, satellite TV, and terrestrial TV), automotive networks including CANBus, etc. Particular networks generally require an external network interface adapter attached to a particular general-purpose data port or peripheral bus 749 (e.g., a USB port on the computer system 700); others are commonly integrated into the core of the computer system 700 by attaching to a system bus, as described below (e.g., an Ethernet interface is integrated in a PC computer system, and a cellular network interface is integrated in a smartphone computer system). Using any of these networks, computer system 700 can communicate with other entities. Such communications can be one-way receive-only (e.g., broadcast TV), one-way transmit-only (e.g., CANbus to certain CANbus devices), or two-way, such as to other computer systems using local or wide-area digital networks. Such communications can include communications to cloud computing environment 755. Specific protocols and protocol stacks can be used with each of these networks and network interfaces, as described above.

[0062]

[0068] The aforementioned human interface devices, human accessible storage devices, and network interface 754 may be attached to core 740 of computer system 700 .

[0063]

[0069] The core 740 may include one or more central processing units (CPUs) 741, graphics processing units (GPUs) 742, specialized programmable processing units in the form of field programmable gate arrays (FPGAs) 743, task-specific hardware accelerators 744, etc. These devices, along with read-only memory (ROM) 745, random access memory 746, and internal mass storage (e.g., internal non-user-accessible hard drives, SSDs, etc.) 747, may be connected via a system bus 748. In some computer systems, the system bus 748 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus 748 or via a peripheral bus 749. Peripheral bus architectures include PCI, USB, etc. A graphics adapter 750 may be included in the core 740.

[0064]

[0070] The CPU 741, GPU 742, FPGA 743, and accelerator 744 may combine to execute specific instructions that may constitute the aforementioned computer code. The computer code may be stored in ROM 745 or RAM 746. Temporary data may be stored in RAM 746, while persistent data may be stored in, for example, internal mass storage 747. Fast storage and retrieval from any memory device may be enabled through the use of cache memory, which may be closely associated with one or more of the CPU 741, GPU 742, mass storage 747, ROM 745, RAM 746, etc.

[0065]

[0071] The computer-readable medium can have computer code thereon for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those having skill in the computer software arts.

[0066]

[0072] By way of example, and not limitation, computer system architecture 700, and specifically a computer system having core 740, may provide functionality as a result of operations by a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be media associated with user-accessible mass storage as described above, as well as specific storage of the core 740 that is non-transitory in nature, such as internal core mass storage 747 or ROM 745. Software implementing various embodiments of the present disclosure may be stored on such devices and executed by the core 740. The computer-readable media may include one or more memory devices or chips, depending on particular needs. The software may cause the core 740, and particularly the processors therein (including a CPU, GPU, FPGA, etc.), to perform specific processes or portions of specific processes described herein, including defining data structures stored in RAM 746 and modifying such data structures according to processes defined by the software. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embedded in circuitry (e.g., accelerator 744), which may execute in place of or in conjunction with software to perform a particular process or portion of a particular process described herein. References to software include logic, and vice versa, where appropriate. References to computer-readable media may include circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry embodying logic for execution, or both, as appropriate. The present disclosure encompasses any appropriate combination of hardware and software.

[0067]

[0073] While this disclosure describes a number of non-limiting embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. It will thus be understood that those skilled in the art will be able to devise numerous systems and methods that, while not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope.

[0068]

[0074] (Appendix 1) 1. A method for decoding sparse haptic data, executed by at least one processor, the method comprising: receiving a haptic track including multiple types of Moving Picture Experts Group (MPEG) Immersive Haptic Stream (MIHS) units; obtaining a first type of MIHS unit from the haptic track, the MIHS unit being a non-empty unit containing haptic information; obtaining a second type of MIHS unit from the haptic track, the second type of MIHS unit being an empty unit containing only duration information without haptic information; and determining a quiet period of the haptic track based on duration information in the second type MIHS unit; A method comprising:

[0069] (Appendix 2) 2. The method of claim 1, wherein the duration information in the second type of MIHS unit indicates the length of a period during which no haptic effect is present.

[0070] (Appendix 3) 2. The method of claim 1, further comprising the step of requesting, based on the determination of the quiet period, that no MIHS units be delivered for the duration of the quiet period.

[0071] (Appendix 4) 2. The method of claim 1, wherein the second type of MIHS unit does not include the tactile information.

[0072] (Appendix 5) 2. The method of claim 1, wherein the second type of MIHS unit represents a quiet period of the haptic track.

[0073] (Appendix 6) 2. The method of claim 1, wherein the first type MIHS unit and the second type MIHS unit are signaled in high level syntax.

[0074] (Appendix 7) 2. The method of claim 1, wherein the haptic information included in the first type MIHS unit is in binary format.

[0075] (Appendix 8) 2. The method of claim 1, wherein quiet periods of the haptic track are used to fragment the haptic track.

[0076] (Appendix 9) 1. An apparatus for decoding sparse haptic data, comprising: at least one memory configured to store program code; at least one processor configured to read said program code and to operate as directed by said program code; the program code comprising: first receiving code configured to cause the at least one processor to receive a haptic track including multiple types of Moving Picture Experts Group (MPEG) Immersive Haptic Stream (MIHS) units; first acquisition code configured to cause the at least one processor to execute a step of acquiring a first type of MIHS unit from the haptic track, the MIHS unit being a non-empty unit containing haptic information; second acquisition code configured to cause the at least one processor to execute a step of acquiring a second type of MIHS unit from the haptic track, the second type of MIHS unit being an empty unit that includes only duration information without any haptic information; and first decision code configured to cause the at least one processor to execute a step of determining a quiet period of the haptic track based on duration information in the second type MIHS unit; 1. An apparatus comprising:

[0077] (Appendix 10) 10. The apparatus of claim 9, wherein the duration information in the second type MIHS unit indicates a length of time during which no haptic effect is present.

[0078] (Appendix 11) 10. The apparatus of claim 9, wherein the program code further includes request code configured to cause the at least one processor to execute, based on the determination of the quiet period, requesting that no MIHS units be delivered for the duration of the quiet period.

[0079] (Appendix 12) 10. The device of claim 9, wherein the second type MIHS unit does not include the tactile information.

[0080] (Appendix 13) 10. The apparatus of claim 9, wherein the second type of MIHS unit represents a quiet period of the haptic track.

[0081] (Appendix 14) 10. The apparatus of claim 9, wherein the first type MIHS unit and the second type MIHS unit are signaled in high level syntax.

[0082] (Appendix 15) 10. The apparatus of claim 9, wherein the haptic information included in the first type MIHS unit is in binary format.

[0083] (Appendix 16) A non-transitory computer-readable storage medium storing instructions, the one or more instructions, when executed by one or more processors of a device for decoding sparse haptic data, causing the one or more processors to: receiving a haptic track including multiple types of Moving Picture Experts Group (MPEG) Immersive Haptic Stream (MIHS) units; obtaining a first type of MIHS unit from the haptic track, the MIHS unit being a non-empty unit containing haptic information; obtaining a second type of MIHS unit from the haptic track, the second type of MIHS unit being an empty unit containing only duration information without haptic information; and determining a quiet period of the haptic track based on duration information in the second type MIHS unit; A non-transitory computer-readable storage medium that causes the

[0084] (Appendix 17) 17. The non-transitory computer-readable storage medium of claim 16, wherein the duration information in the second type of MIHS unit indicates a length of time during which no haptic effect is present.

[0085] (Appendix 18) 17. The non-transitory computer-readable storage medium of claim 16, wherein the instructions cause the one or more processors to perform, based on the determination of the quiet period, requesting not to deliver MIHS units for the duration of the quiet period.

[0086] (Appendix 19) 17. The non-transitory computer-readable storage medium of claim 16, wherein the second type MIHS unit does not include the tactile information.

[0087] (Appendix 20) 17. The non-transitory computer-readable storage medium of claim 16, wherein the second type of MIHS unit represents a quiet period of the haptic track.

Claims

1. 1. A method for decoding sparse haptic data, executed by at least one processor, the method comprising: receiving a haptic track including multiple types of Moving Picture Experts Group (MPEG) Immersive Haptic Stream (MIHS) units; obtaining a first type of MIHS unit from the haptic track, the MIHS unit being a non-empty unit containing haptic information; obtaining a second type of MIHS unit from the haptic track, the second type of MIHS unit being an empty unit containing only duration information without haptic information; and determining a quiet period of the haptic track based on duration information in the second type MIHS unit; A method comprising:

2. 10. The method of claim 1, wherein the duration information in the second type of MIHS unit indicates a length of time during which no haptic effect is present.

3. 10. The method of claim 1, further comprising the step of requesting, based on the determination of the quiet period, that no MIHS units be delivered for the duration of the quiet period.

4. The method of claim 1 , wherein the second type of MIHS unit does not include the tactile information.

5. The method of claim 1 , wherein the second type of MIHS unit represents a quiet period of the haptic track.

6. 2. The method of claim 1, wherein the first type MIHS unit and the second type MIHS unit are signaled in a high level syntax.

7. 2. The method of claim 1, wherein the haptic information contained in the first type of MIHS unit is in binary format.

8. The method of claim 1 , wherein quiet periods in the haptic track are used to fragment the haptic track.

9. 1. An apparatus for decoding sparse haptic data, comprising: at least one memory configured to store program code; at least one processor configured to read the program code and to act as directed by the program code; 9. An apparatus comprising: a program code configured to perform a method according to any one of claims 1 to 8.

10. A computer program product that causes a computer processor to carry out a method according to any one of claims 1 to 8.

11. 1. A method for encoding sparse haptic data, executed by at least one processor, the method comprising: generating a haptic track including multiple types of Moving Picture Experts Group (MPEG) Immersive Haptic Stream (MIHS) units and receiving the haptic track to a decoder; the haptic track includes a first type of MIHS unit that is a non-empty unit containing haptic information, and a second type of MIHS unit, the second type of MIHS unit being an empty unit containing only duration information without haptic information; The method further comprises determining a quiet period of the haptic track based on duration information in the second type of MIHS unit.

Images