Method and apparatus for defining frame and time-specific reference NAL structure in a haptic signal

Abstract

A method, apparatus, and system for haptic signal processing are provided. The process can include receiving a bitstream having one or more haptic channels, and can obtain one or more moving picture experts group (MPEG) immersive haptics stream (MIHS) units for the one or more haptic channels from the bitstream, the MIHS units including timing information for one or more haptic effects. The process can also include obtaining timing information for the one or more haptic effects from the one or more MIHS units, the timing information for the one or more haptic effects including offsets associated with the one or more haptic effects. The one or more haptic effects can be rendered based on the obtained timing information.

Description

[Technical Field]

[0001] Reference to Related Applications This application claims priority to U.S. Provisional Application No. 63 / 416,795, filed October 17, 2022, and U.S. Application No. 18 / 487,590, filed October 16, 2023, the entire disclosures of which are incorporated herein by reference.

[0002] This disclosure is directed to a collection of advanced video coding techniques. In particular, this disclosure is directed to encoding and decoding haptic experiences in multimedia presentations. [Background technology]

[0003] Haptic experiences are part of a multimedia presentation. In applications where the multimedia presentation includes a haptic aspect, haptic signals are passed to a device or wearable that allows a user to feel haptic sensations in coordination with their visual and / or audio media experiences while using the application.

[0004] Recognizing the growing popularity of haptic experiences in multimedia presentations, work has been initiated by the motion picture experts group (MPEG) on haptic compression standards (both MPEG-DASH and MPEG-I standards) and the transport of compressed haptic signaling in the ISO based media file format (ISOBMFF). Summary of the Invention [Problem to be solved by the invention]

[0005] One of the problems to be solved when incorporating haptic aspects into a multimedia presentation is the lack of a concept of a haptic frame or a haptic sample. Furthermore, there is no concept of a haptic time reference NAL unit (network abstraction layer unit). A solution is needed to address this problem. [Means for solving the problem]

[0006] According to an embodiment, there may be provided a method for encoding or decoding haptic data, the method being executable by at least one processor and including the steps of receiving a bitstream comprising one or more haptic channels, obtaining one or more moving picture experts group (MPEG) immersive haptics stream (MIHS) units for the one or more haptic channels from the bitstream, the MIHS units comprising timing information for one or more haptic effects, obtaining the timing information for the one or more haptic effects from the one or more MIHS units, the timing information for the one or more haptic effects comprising offsets associated with the one or more haptic effects, and rendering the bitstream based on the obtained timing information.

[0007] According to an embodiment, an apparatus for haptic processing can be provided. The apparatus can include at least one memory configured to store program code and at least one processor configured to load the program code and operate as directed by the program code. The program code can include first receiving code configured to cause the at least one processor to receive a bitstream comprising one or more haptic channels; first acquiring code configured to cause the at least one processor to acquire one or more moving picture experts group (MPEG) immersive haptics stream (MIHS) units of the one or more haptic channels from the bitstream, the MIHS units comprising timing information for one or more haptic effects; second acquiring code configured to cause the at least one processor to acquire timing information for the one or more haptic effects from the one or more MIHS units, the timing information for the one or more haptic effects comprising offsets associated with the one or more haptic effects; and rendering code configured to cause the at least one processor to render the bitstream based on the acquired timing information.

[0008] According to an embodiment, a non-transitory computer-readable medium can be provided that stores computer instructions that, when executed by one or more processors of a device for haptic processing, can include one or more instructions that cause the one or more processors to: receive a bitstream comprising one or more haptic channels; retrieve one or more moving picture experts group (MPEG) immersive haptics stream (MIHS) units for the one or more haptic channels from the bitstream, the MIHS units comprising timing information for one or more haptic effects; retrieve the timing information for the one or more haptic effects from the one or more MIHS units, the timing information for the one or more haptic effects comprising offsets associated with the one or more haptic effects; and render the bitstream based on the retrieved timing information.

[0009] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a schematic diagram of a simplified block diagram of a streaming system according to an embodiment of the present disclosure. [Figure 3A] FIG. 1 is a schematic diagram of a simplified block diagram of a haptic encoder according to an embodiment of the present disclosure. [Figure 3B] FIG. 1 is a schematic diagram of a simplified block diagram of a haptics decoder and a haptics renderer according to an embodiment of the present disclosure. [Figure 4] 1 is an exemplary diagram of a haptic frame according to an embodiment of the present disclosure. [Figure 5] FIG. 1 is an exemplary diagram of an MIHS unit according to an embodiment of the present disclosure. [Figure 6] FIG. 10 is an exemplary flow diagram illustrating a process for processing haptic media according to an embodiment of the present disclosure. [Figure 7] FIG. 1 illustrates a computer system suitable for implementing embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0011] According to one aspect of the present disclosure, a method, system, and non-transitory storage medium are provided for parallel processing of dynamic mesh compression. Embodiments of the present disclosure may also be applied to static meshes.

[0012] 1-2, one embodiment of the present disclosure for implementing the encoding and decoding structure of the present disclosure is described.

[0013] 1 illustrates a simplified block diagram of a communication system 100 according to one embodiment of the present disclosure. The system 100 may include at least two terminals 110, 120 interconnected via a network 150. In a one-way data transmission scenario, a first terminal 110 may encode video data, which may include mesh data, at a local location for transmission to the other terminal 120 via the network 150. The second terminal 120 may receive the other terminal's encoded video data from the network 150, decode the encoded data, and display the recovered video data. One-way data transmission may be common in media delivery applications, for example.

[0014] 1 shows a second pair of terminals 130, 140 arranged to support bidirectional transmission of encoded video, such as may occur during a video conference. For bidirectional transmission of data, each terminal 130, 140 may encode video data captured at a local location for transmission to the other terminal over network 150. Each terminal 130, 140 may also receive encoded video data transmitted by the other terminal, decode the encoded data, and display the recovered video data on a local display device.

[0015] In FIG. 1 , terminals 110-140 may be, for example, servers, personal computers, smartphones, and / or any other type of terminal. For example, terminals (110-140) may be laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 150 represents any number of networks that convey coded video data between terminals 110-140, including, for example, wired and / or wireless communication networks. Communication network 150 may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of network 150 may not be important to the operation of the present disclosure unless otherwise described herein.

[0016] Figure 2 illustrates the deployment of a video encoder and decoder in a streaming environment as an example of an application of the disclosed subject matter, which may be used in other video-enabled applications, including, for example, video conferencing, digital TV, and storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0017] 2, the streaming system 200 may include a capture subsystem 213 that includes a video source 201 and an encoder 203. The streaming system 200 may further include at least one streaming server 205 and / or at least one streaming client 206.

[0018] The video source 201 may, for example, create a stream 202 that includes a 3D mesh and metadata associated with the 3D mesh. The video source 201 may, for example, include a 3D sensor (e.g., a depth sensor) or 3D imaging technology (e.g., one or more digital cameras) and a computing device configured to generate a 3D mesh using data received from the 3D sensor or 3D imaging technology. The sample stream 202 may have a high amount of data compared to an encoded video bitstream and may be processed by an encoder 203 coupled to the video source 201. The encoder 203 may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoder 203 may also further generate an encoded video bitstream 204. The encoded video bitstream 204 may have a low amount of data compared to the uncompressed stream 202 and may be stored on a streaming server 205 for later use. One or more streaming clients 206 may access the streaming server 205 to obtain a video bitstream 209 , which may be a copy of the encoded video bitstream 204 .

[0019] Streaming client 206 may include a video decoder 210 and a display 212. Video decoder 210 may, for example, decode video bitstream 209, which is a copy of input encoded video bitstream 204, and create an output video sample stream 211 that can be rendered on display 212 or another rendering device (not shown). In some streaming systems, video bitstreams 204, 209 may be encoded according to a particular video encoding / compression standard.

[0020] 3A and 3B, one embodiment of the present disclosure for implementing haptic encoder 300 and haptic decoder 350 is described.

[0021] 3A, haptics encoder 300 can receive both descriptive haptic data and waveform haptic data. Thus, haptics encoder 300 can process three types of input files: .ohm metadata files (object haptics metadata: a text file format for haptic metadata), descriptive haptic files (.ivs, .ahap, and .hjif), or waveform PCM files (.wav). Examples of descriptive data include Apple's .ahap (Apple Haptic and Audio Pattern: a JSON-like file format for specifying haptic patterns) (which represents the expected haptic output resulting from a set of modulated continuous signals and a set of parameters derived from the modulated transient signals), Immersion's .ivs (which uses a set of parameters to represent the expected haptic output resulting from parameters derived from a set of basis effects), and the proposed MPEG format .hjif (Haptics JSON Interchange Format). An example of a waveform pulse-code modulation (PCM) signal would include an .ohm input file containing metadata information.

[0022] According to one embodiment, haptics encoder 300 can process the two types of input files differently: For described content, haptics encoder 300 can perform a semantic analysis on the input and transcode the data into a proposed encoded representation (if necessary).

[0023] According to one embodiment, an .ohm metadata input file can include a description of the haptic system and setup. In particular, it can include a description of the signal as well as the name of each associated haptic file (either a description file or a PCM file). It can also provide a correspondence between each channel of the signal and a target part of the user's body. For an .ohm metadata input file, the haptic encoder extracts the metadata by retrieving the associated haptic file from its URI, encoding it by extracting metadata from the .ohm file based on its type, and mapping the encoding to the metadata information in the data model.

[0024] According to one embodiment, descriptive haptic files (e.g., .ivs, .ahap, and .hjif) can be encoded through a simple process. First, haptics encoder 300 specifically identifies the input format. If the input format is an .hjif file, no transcoding is required; the file can be further edited, compressed into a binary format, and finally packetized into an MIHS stream. Transcoding is required when using an .ahap or .ivs input file. First, haptics encoder 300 performs a semantic analysis on the input file information and transcodes it so that it is formatted into the selected data model. After transcoding, the data can be exported as an .hjif file, an .hmpg binary file, or an MIHS stream.

[0025] According to one embodiment, the haptics encoder 300 can perform signal analysis to interpret the signal structure of a .wav file and convert it to the proposed coding representation. For waveform PCM content, the signal analysis process can be divided into two subprocesses by the haptics encoder 300. After performing frequency band decomposition on the signal, the first subprocess can encode low frequencies using a keyframe extraction process. One or more low-frequency bands can then be reconstructed, and the error between this signal and the original low-frequency signal can be calculated. This residual signal can be added to one or more original high-frequency bands before encoding using a wavelet transform. Encoding using a wavelet transform is the second subprocess. According to one embodiment, if several low-frequency bands are used, the residual errors of all low-frequency bands are added to the high-frequency band before encoding. In embodiments where several high-frequency bands are used, the residual errors of one or more low-frequency bands are added to the first high-frequency band before encoding.

[0026] According to one embodiment, keyframe extraction involves obtaining a low-frequency band from the frequency band decomposition and analyzing its content in the time domain. According to one embodiment, wavelet processing involves obtaining a high-frequency band from the frequency band decomposition and the low-frequency residual and dividing it into equal-sized blocks. These equal-sized signal blocks are then analyzed with a psychohaptic model. Lossy compression can be applied by wavelet transforming and quantizing the blocks with the aid of the psychohaptic model. Finally, each block is saved as a separate effect of one band. This is done during formatting. For binary compression, lossless compression can be applied using an appropriate coding technique, such as the Set Partitioning in Hierarchical Trees (SPIHT) algorithm or arithmetic coding (AC).

[0027] As shown in FIG. 3A , haptic encoder 300 can be configured to encode descriptive haptic data and quantized haptic data. Haptic encoder 300 can output three types of formats: an interchange format (.hjif), a binary compressed format (.hmpg), and a streaming format (e.g., MPEG immersive haptic stream (MIHS)). The .hjif format is a human-readable format based on JSON. The ease of parsing and manually editing the .hjif format makes it an ideal interchange format, especially during content design and creation. For distribution, the .hjif data can be compressed into a more memory-efficient binary .hmpg bitstream. This compression can be lossy, using various parameters that affect the coding depth of the amplitude and frequency components that make up the bitstream. For streaming, data can be compressed and packetized to obtain an MPEG-I haptic stream (MIHS). These three formats serve complementary purposes, and lossy, one-to-one conversions can be performed between them.

[0028] As shown in FIG. 3B, haptics decoder 350 can accept either the .hmpg compressed binary file format or an MIHS bitstream as input. Haptics decoder 350 can output the .hjif interchange format, which can be used directly for rendering. The two input formats can undergo binary decompression, extracting both the metadata and the data itself from the file and mapping it to a selected data structure. The data can then be exported to haptics renderer 380 in .hjif format.

[0029] As shown in FIG. 3B, renderer 380 includes a synthesizer. The synthesizer can render haptic data obtained from an .hjif input file to obtain a PCM output file. Rendering and / or synthesis is highly informative. According to one embodiment, the synthesizer analyzes the input file and performs high-level synthesis distribution, such as between vectors or wavelets. The synthesis process then flows down to the codec band components, where the synthesis process is invoked. All bands for a particular channel are then mixed using a simple summation operator to recreate the desired haptic signal.

[0030] According to an embodiment, the root of the hierarchical data model is defined by the haptic experience, which provides information about the file date and format version, describes the haptic experience, lists the various avatars (i.e., body representations) used throughout the experience, and defines all haptic perceptions.

[0031] According to one embodiment, a self-contained stream format for transporting MPEG-I haptic data can use a packetization approach, which can include two levels of packetization: an MPEG-I haptic stream (MIHS) unit that covers a time span and contains zero or more MIHS packets, and an MIHS packet that contains metadata or haptic effect data.

[0032] In an embodiment, each MIHS unit covers a non-overlapping haptic delivery time span, i.e., each MIHS unit starts at the end of the previous MIHS unit and covers the time span defined by its time span field. An MIHS unit is followed by the next MIHS unit if it is not the last MIHS unit of the haptic experience. All MIHS packets of an MIHS unit have the start time and time span of the MIHS unit that contains them.

[0033] In some embodiments, an MIHS unit may be referred to as a network abstraction layer (NAL) unit associated with haptic data. In some embodiments, an MIHS unit may be referred to as an MIHS sample associated with haptic data.

[0034] According to one embodiment, an MIHS unit may be a synchronous unit or an asynchronous unit. A synchronous unit resets previous effects, providing a haptic experience independent of the previous MIHS unit. An asynchronous unit is a continuation of a previous MIHS unit, and cannot be decoded and rendered independently without decoding the previous MIHS unit.

[0035] According to embodiments, haptic signals can be encoded in multiple channels. In some embodiments, haptic channels can be used to define signals to be rendered at specific body positions using dedicated actuators / devices. Metadata stored at the channel level can include information such as channel-associated gain, mix weight, desired body position of haptic feedback, and, optionally, information such as a reference device and / or orientation. Additional information such as a desired sampling frequency and number of samples can also be provided. Finally, haptic data for a channel is stored in a set of haptic bands defined by frequency ranges. A haptic band describes the haptic signal for a channel within a specific frequency range. A band is defined by a type of haptic effect and an ordered list, with each band containing a set of keyframes. In embodiments, for any type of haptic band, a haptic effect can be defined using at least a position and a type. Depending on the type of band and the type of effect, additional properties must be specified, including phase, base signal, composition, and the number of consecutive haptic keyframes that describe the effect. For any type of haptic band, a haptic effect can be defined using at least a position and a type. The position may indicate the temporal or spatial position of the effect. In some embodiments, a value of 0 is the relative start position of the experience according to the dependent variable of the configured sensory modality. The unit of temporal haptic feedback may be milliseconds by default, and the unit of spatial haptic feedback may be millimeters by default. Because the binary delivery format does not have any concept of fixed time intervals, i.e., frames or samples, this embodiment discloses the "start position of the experience."

[0036] Depending on the type of band and the type of effect, additional properties can be specified, including phase, base signal, composition, and the number of consecutive haptic keyframes that describe the effect.

[0037] According to one embodiment of the present disclosure, a new element of a frame can be defined in the haptic data hierarchy. Thus, according to one embodiment, each haptic channel consists of one or more frames. A time duration is defined for each frame, and no two frames overlap.

[0038] According to one embodiment, this disclosure defines a new element in the haptic data hierarchy called a frame. ● Haptic Channel Haptic frame ■ Haptic band ● Haptic effects

[0039] Each haptic channel consists of one or more frames, as shown in Figure 4. A time span is defined for each frame, and no two frames overlap. As shown in Figure 4, the temporal position of an effect is defined as an offset relative to the start timing of the frame that carries the effect.

[0040] One immediate benefit of using haptic frames in a haptic elementary stream is that it makes it easier and more efficient to map the stream to ISOBMFF or any other file format that uses temporal samples. In one example, each frame of haptics maps to one ISOBMFF sample.

[0041] In an embodiment of the present disclosure, a haptic synchronization frame is defined to be a haptic frame that resets the output of all previous effects. That is, if the player starts before or at that frame, the output will be the same. The value of such a frame is that it provides a random access point for the player. Any playback at this point will result in consistent playback regardless of the start time of playback. The synchronization frame can be converted to an ISOBMFF synchronization sample. For example, the synchronization frame becomes an ISOBMFF synchronization sample. Therefore, the random accessibility of the synchronization frame can be used at the system layer.

[0042] Another aspect of the present disclosure, namely, a network abstraction layer (also referred to as an MPEG immersive haptic stream (MIHS) unit), which is an alternative to or combination with the haptic frames disclosed herein, can provide timing information for the haptic channel shown in Figure 5. In embodiments, NAL units are referred to as MIHS units, and interchangeably, MIHS units are also referred to as NAL units.

[0043] In one embodiment, an MIHS unit can contain necessary timing information (start time and / or duration). In contrast to a channel frame, one MIHS unit can contain one or more channels. It should be noted that the haptics delivery syntax naturally includes MIHS unit syntax. Similar to video NAL unit types, MIHS units can be defined using synchronous / random access or asynchronous types to specify random access. As explained above, a synchronous MIHS unit resets all previous haptic effects and initiates one or more new effects.

[0044] According to one embodiment, a haptic stream can include both frames and timed-referenced MIHS units. In this embodiment, a timed-referenced MIHS unit can include one or more frames. The timing information for the MIHS units may be selectable.

[0045] As shown in FIG. 6, process 600 illustrates an exemplary process for decoding haptic data.

[0046] At operation 605, a media stream including one or more haptic channels may be received. In some embodiments, the media stream may include one or more audio / video channels. In some embodiments, as an example, the media stream may include an .ahap, .hjif, or .ivs input stream.

[0047] At act 610, one or more moving picture experts group (MPEG) immersive haptics stream (MIHS) units can be obtained for one or more haptic channels from the media stream. In embodiments, the MIHS units can include timing information for one or more haptic effects. In some embodiments, each of the one or more haptic channels has at least one associated MIHS unit. The MIHS unit can also include one or more haptic frames, each associated with a non-overlapping time span in the MIHS unit.

[0048] Timing information for one or more haptic effects obtained from one or more MIHS units can be obtained at act 615. In an embodiment, the timing information for the one or more haptic effects comprises an offset associated with the one or more haptic effects.

[0049] In an embodiment, an MIHS unit includes an MIHS start time that is used as an anchor for the start of one or more haptic effects, and an offset associated with the one or more haptic effects indicates an effect start time that corresponds to the MIHS start time.

[0050] In an embodiment, the MIHS unit further comprises timing information for one or more haptic bands, whereby an operation is further provided of obtaining timing information for one or more haptic bands from the one or more MIHS units, wherein the timing information for one or more haptic bands includes an offset associated with the one or more haptic bands.

[0051] In an embodiment, the MIHS unit may include an MIHS start time that is used as an anchor for the start of one or more haptic bands, and an offset associated with the one or more haptic bands indicates a band start time that corresponds to the MIHS start time.

[0052] At operation 620, the media stream is rendered based on the obtained timing information.

[0053] In some embodiments, synchronized MIHS units can be obtained from a media stream. In embodiments, synchronized MIHS units are a special type of MIHS unit configured to provide a reset point in a bitstream. In embodiments, synchronized MIHS units are mapped to synchronization samples in a video bitstream that correspond to one or more haptic channels.

[0054] Those skilled in the art will appreciate that the techniques described in this disclosure can be implemented on both the encoder and decoder sides. The techniques described above can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 7 illustrates a computer system 700 suitable for implementing certain embodiments of the present disclosure.

[0055] Computer software can be encoded using any suitable machine code or computer language that can be subjected to mechanisms such as assembly, compilation, linking, etc. to create code containing instructions that can be executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc. directly or via interpretation, microcode execution, etc.

[0056] The instructions may be executed on various types of computers or computer components, including, for example, personal computers, tablet computers, servers, smartphones, gaming consoles, Internet of Things devices, and the like.

[0057] 7 for computer system 700 are examples and are not intended to suggest any limitation on the scope of use or functionality of the computer software implementing embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the non-limiting embodiment of computer system 700.

[0058] The computer system 700 may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users via, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., speaking, clapping), visual input (e.g., gestures), or olfactory input (not shown). The human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0059] The input human interface devices may include one or more of a keyboard 701, a mouse 702, a trackpad 703, a touchscreen 710, a data glove, a joystick 705, a microphone 706, a scanner 707, and a camera 708 (only one of each is shown).

[0060] The computer system 700 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen 710, data gloves, or joystick 705, although haptic feedback devices that do not function as input devices may also be present). For example, such devices may include audio output devices (such as speakers 709, headphones (not shown)), visual output devices (such as screens 710, including CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touchscreen input capabilities, each with or without haptic feedback capabilities, some of which may output two-dimensional visual output or output in more than three dimensions, such as through means of stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0061] The computer system 700 may also include human-accessible storage devices and media associated with the storage devices, such as optical media including CD / DVD ROM / RW 720 with media 721 such as CD / DVD, thumb drive 722, removable hard drive or solid state drive 723, legacy magnetic media such as tape or floppy disk (not shown), dedicated ROM / ASIC / PLD based devices such as security dongles (not shown), etc.

[0062] Those skilled in the art will also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0063] The computer system 700 may also include interfaces to one or more communication networks. Networks may be, for example, wireless, wired, or optical. Networks may further be local, wide-area, metropolitan, vehicular, and industrial, real-time, delay-tolerant, and the like. Examples of networks include local area networks such as Ethernet; cellular networks including WLAN, GSM, 3G, 4G, 5G, LTE, and the like; television wired or wireless wide-area digital networks including cable, satellite, and terrestrial television; vehicular and industrial networks including CANBus; and the like. Certain networks generally require an external network interface adapter connected to a particular general-purpose data port or peripheral bus 749 (e.g., a USB port on the computer system 700); others are generally integrated into the core of the computer system 700 by connection to a system bus, as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system 700 can communicate with other entities. Such communications may be, for example, one-way, receive-only (e.g., broadcast television), one-way transmit-only (e.g., CANbus to a particular CANbus device), or two-way, to other computer systems using local or wide area digital networks. Such communications may include communications to cloud computing environment 755. Specific protocols and protocol stacks may be used in each of these networks and network interfaces, as described above.

[0064] The aforementioned human interface devices, human-accessible storage devices, and network interface 754 may be connected to core 740 of computer system 700 .

[0065] The core 740 may include one or more central processing units (CPUs) 741, graphics processing units (GPUs) 742, specialized programmable processing units in the form of field programmable gate arrays (FPGAs) 743, hardware accelerators 744 for specific tasks, etc. These devices may be connected via a system bus 748, along with read-only memory (ROM) 745, random access memory 746, and internal mass storage such as an internal non-user-accessible hard drive, SSD, etc. 747. In some computer systems, the system bus 748 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus 748 or via a peripheral bus 749. Architectures for peripheral buses include PCI, USB, etc. A graphics adapter 750 may also be included in the core 740.

[0066] The CPU 741, GPU 742, FPGA 743, and accelerator 744 may combine to execute specific instructions that may constitute the aforementioned computer code. That computer code may be stored in ROM 745 or RAM 746. Transient data may also be stored in RAM 746, while persistent data may be stored, for example, in internal mass storage 747. Rapid storage and retrieval from any of the memory devices may be enabled through the use of cache memory, which may be closely associated with one or more of the CPU 741, GPU 742, mass storage 747, ROM 745, RAM 746, etc.

[0067] The computer-readable medium may bear computer code for performing various computer-implemented operations, and the medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0068] By way of example and not limitation, a computer system having the architecture of computer system 700, and in particular core 740, can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage as described above, as well as media associated with specific storage of core 740 that is non-transitory in nature, such as core internal mass storage 747 or ROM 745. Software implementing various embodiments of the present disclosure can be stored on such devices and executed by core 740. Computer-readable media can include one or more memory devices or chips, depending on particular needs. The software can cause core 740, and in particular the processors (including a CPU, GPU, FPGA, etc.) in the core, to perform particular processes or particular portions of particular processes described herein, including defining data structures stored in RAM 746 and modifying such data structures according to processes defined by the software. Additionally or alternatively, a computer system may provide functionality as a result of hardwired or otherwise embodied logic in circuitry (e.g., accelerator 744) that can operate in place of or together with software to perform particular processes or portions of particular processes described herein. Where appropriate, references to software may encompass logic, and vice versa. Where appropriate, references to computer-readable media may encompass circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0069] While this disclosure has described several non-limiting embodiments, there are modifications, permutations, and various substitute equivalents that fall within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope. [Explanation of symbols]

[0070] 100 Communication Systems 110 terminals 120 terminals 130 terminals 150 Communication Network 200 Streaming System 201 Video Source 202 Sample Stream 203 Encoder 204 Video Bitstream 205 Streaming Server 206 Streaming Client 209 Video Bitstream 210 Video Decoder 211 Video Sample Stream 212 Display 213 Capture Subsystem 300 Haptic Encoder 350 Haptic Decoder 380 Haptic Renderer 600 processes 605 operation 610 operation 615 operation 620 operation 700 Computer Systems 701 Keyboard 702 Mouse 703 Trackpad 705 Joystick 706 Mike 707 Scanner 708 Camera 709 Speaker 710 Touchscreen 720 CD / DVD ROM / RW 721 Medium 722 thumb drive 723 Solid State Drive 740 cores 741 CPU 742 GPU 743 FPGA 744 Hardware Accelerator 745 ROM 746 RAM 747 Core internal mass storage unit 748 System Bus 749 Peripheral Bus 750 graphics adapter 754 Network Interface 755 Cloud Computing Environment

Claims

1. 1. A method for decoding information in haptic data, the method being executed by at least one processor, the method comprising: receiving a bitstream comprising one or more haptic channels; obtaining one or more moving picture experts group (MPEG) immersive haptics stream (MIHS) units for the one or more haptic channels from the bitstream, the MIHS unit providing timing information for one or more haptic effects; obtaining the timing information of the one or more haptic effects from the one or more MIHS units, the timing information of the one or more haptic effects comprising an offset associated with the one or more haptic effects; decoding the bitstream based on the obtained timing information; A method comprising:

2. The method comprises: obtaining synchronous MIHS units from the bitstream, the synchronous MIHS unit is a special type of MIHS unit configured to provide a reset point in the bitstream; the synchronous MIHS units are mapped to synchronous samples of a video bitstream corresponding to the one or more haptic channels. The method of claim 1 further comprising:

3. the MIHS unit includes an MIHS start time that is used as an anchor for the start of the one or more haptic effects; the offset associated with the one or more haptic effects indicates an effect start time that corresponds to the MIHS start time. The method of claim 1.

4. The MIHS unit further comprises timing information for one or more haptic bands, and the method further comprises: obtaining the timing information for the one or more haptic bands from the one or more MIHS units, the timing information for the one or more haptic bands comprising an offset associated with the one or more haptic bands; The method of claim 1 further comprising:

5. the MIHS unit comprises an MIHS start time used as an anchor for the start of the one or more haptic bands; the offset associated with the one or more haptic bands indicates a band start time corresponding to the MIHS start time; The method of claim 4.

6. The method of claim 1 , wherein each of the one or more haptic channels has at least one associated MIHS unit.

7. The method of claim 1 , wherein the MIHS unit further comprises one or more haptic frames, each haptic frame associated with a non-overlapping time span in the MIHS unit.

8. 1. An apparatus for decoding haptic data, the apparatus comprising: at least one memory configured to store program code; at least one processor configured to read said program code and to operate as directed by said program code, said program code comprising: first receiving code configured to cause the at least one processor to receive a bitstream comprising one or more haptic channels; first obtaining code configured to cause the at least one processor to obtain one or more moving picture experts group (MPEG) immersive haptics stream (MIHS) units of the one or more haptics channels from the bitstream, the first obtaining code comprising: a first acquisition code, the MIHS unit comprising timing information for one or more haptic effects; second acquisition code configured to cause the at least one processor to acquire the timing information of the one or more haptic effects from the one or more MIHS units, the timing information of the one or more haptic effects comprising an offset associated with the one or more haptic effects; and rendering code configured to cause the at least one processor to render the bitstream based on the obtained timing information; At least one processor, including An apparatus comprising:

9. The program code a third obtaining code configured to cause the at least one processor to obtain a synchronous MIHS unit from the bitstream, the third obtaining code comprising: the synchronous MIHS unit is a special type of MIHS unit configured to provide a reset point in the bitstream; a third acquisition code, wherein the synchronous MIHS units are mapped to synchronous samples of a video bitstream corresponding to the one or more haptic channels; The apparatus of claim 8, further comprising:

10. the MIHS unit includes an MIHS start time that is used as an anchor for the start of the one or more haptic effects; the offset associated with the one or more haptic effects indicates an effect start time that corresponds to the MIHS start time.

9. The apparatus of claim 8.

11. the MIHS unit further comprises timing information for one or more haptic bands, and the program code further comprises: fourth acquisition code configured to cause the at least one processor to acquire the timing information of the one or more haptic bands from the one or more MIHS units, the timing information of the one or more haptic bands comprising an offset associated with the one or more haptic bands; The apparatus of claim 8, further comprising:

12. the MIHS unit comprises an MIHS start time used as an anchor for the start of the one or more haptic bands; the offset associated with the one or more haptic bands indicates a band start time corresponding to the MIHS start time; 12. The apparatus of claim 11.

13. The device of claim 8 , wherein each of the one or more haptic channels has at least one associated MIHS unit.

14. The apparatus of claim 8 , wherein the MIHS unit further comprises one or more haptic frames, each haptic frame associated with a non-overlapping time span in the MIHS unit.

15. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a device that decodes haptic data, receiving a bitstream comprising one or more haptic channels; obtaining one or more moving picture experts group (MPEG) immersive haptics stream (MIHS) units for the one or more haptic channels from the bitstream; the MIHS unit comprises timing information for one or more haptic effects; obtaining the timing information of the one or more haptic effects from the one or more MIHS units, the timing information of the one or more haptic effects comprising an offset associated with the one or more haptic effects; rendering the bitstream based on the obtained timing information; a non-transitory computer-readable medium comprising one or more instructions that cause the one or more processors to:

16. and further causing the one or more processors to obtain synchronous MIHS units from the bitstream by the instructions; the synchronous MIHS unit is a special type of MIHS unit configured to provide a reset point in the bitstream; the synchronous MIHS units are mapped to synchronous samples of a video bitstream corresponding to the one or more haptic channels; 16. The non-transitory computer-readable medium of claim 15.

17. the MIHS unit includes an MIHS start time that is used as an anchor for the start of the one or more haptic effects; the offset associated with the one or more haptic effects indicates an effect start time that corresponds to the MIHS start time.

16. The non-transitory computer-readable medium of claim 15.

18. 16. The non-transitory computer-readable medium of claim 15, wherein the MIHS unit further comprises timing information for one or more haptic bands, and the instructions further cause the one or more processors to obtain the timing information for the one or more haptic bands from the one or more MIHS units, and the timing information for the one or more haptic bands comprises an offset associated with the one or more haptic bands.

19. the MIHS unit comprises an MIHS start time used as an anchor for the start of the one or more haptic bands; the offset associated with the one or more haptic bands indicates a band start time corresponding to the MIHS start time; 20. The non-transitory computer-readable medium of claim 18.

20. 16. The non-transitory computer-readable medium of claim 15, wherein the MIHS unit further comprises one or more haptic frames, each haptic frame associated with a non-overlapping time span in the MIHS unit.

Images