Method and apparatus for defining frame and time-specified reference NAL structure in haptic signals

Abstract

A method, apparatus, and system for haptic signal processing are provided. The process can include receiving a bitstream having one or more haptic channels, and can obtain one or more moving picture experts group (MPEG) immersive haptics stream (MIHS) units for the one or more haptic channels from the bitstream, the MIHS units including timing information for one or more haptic effects. The process can also include obtaining timing information for the one or more haptic effects from the one or more MIHS units, the timing information for the one or more haptic effects including offsets associated with the one or more haptic effects. The one or more haptic effects can be rendered based on the obtained timing information.

Description

Technical Field

[0001] Reference to Related Application The present application claims priority from U.S. Provisional Application No. 63 / 416,795 filed on October 17, 2022 and U.S. Application No. 18 / 487,590 filed on October 16, 2023. The entire disclosures of the aforementioned U.S. provisional application and U.S. application are incorporated herein by reference.

[0002] The present disclosure relates to a set of advanced video coding techniques. In particular, the present disclosure relates to encoding and decoding of haptic sensations for multimedia presentations.

Background Art

[0003] Haptic sensations form a part of multimedia presentations. In applications where multimedia presentations include a haptic sensation aspect, haptic signals are transmitted to a device or wearable device, allowing a user to perceive haptic sensations in conjunction with visual sensations and / or audio media sensations during use of the application.

[0004] It has been recognized that haptic sensations in multimedia presentations are growing in popularity, and activities have been initiated by the Moving Picture Experts Group (MPEG) regarding haptic compression standards (both MPEG-DASH and MPEG-I standards) and carriage of compressed haptic signaling in the ISO based media file format (ISOBMFF).

Summary of Invention

Problem to be Solved by the Invention

[0005] One of the problems that needs to be solved when incorporating haptic experiences into multimedia presentations is the lack of concepts such as haptic frames and haptic samples. Furthermore, there is no concept of haptic time-referenced NAL units (network abstraction layer units). A solution is needed to address this problem. [Means for solving the problem]

[0006] In one embodiment, a method for encoding or decoding haptic data can be provided. The method can be performed by at least one processor and may include the steps of: receiving a bitstream having one or more haptic channels; obtaining one or more moving picture experts group (MPEG) immersive haptics stream (MIHS) units of one or more haptic channels from the bitstream, wherein each MIHS unit comprises timing information for one or more haptic effects; obtaining timing information for one or more haptic effects from the one or more MIHS units, wherein the timing information for one or more haptic effects comprises offsets associated with one or more haptic effects; and rendering the bitstream based on the obtained timing information.

[0007] In one embodiment, an apparatus for haptics processing can be provided. The apparatus may include at least one memory configured to store program code, and at least one processor configured to read the program code and operate as instructed by the program code. The program code may include a first receive code configured to cause at least one processor to receive a bitstream having one or more haptics channels, a first acquire code configured to cause at least one processor to acquire one or more moving picture experts group (MPEG) immersive haptics stream (MIHS) units of one or more haptics channels from the bitstream, wherein the MIHS unit comprises timing information for one or more haptics effects, a second acquire code configured to cause at least one processor to acquire timing information for one or more haptics effects from one or more MIHS units, wherein the timing information for one or more haptics effects comprises offsets associated with one or more haptics effects, and a rendering code configured to cause at least one processor to render the bitstream based on the acquired timing information.

[0008] In one embodiment, a non-temporary computer-readable medium for storing computer instructions can be provided. The instructions may include one or more instructions that cause one or more processors to perform the following actions when executed by one or more processors of a device for haptics processing: receive a bitstream having one or more haptics channels; obtain one or more moving picture experts group (MPEG) immersive haptics stream (MIHS) units of one or more haptics channels from the bitstream, wherein each MIHS unit comprises timing information for one or more haptics effects; obtain timing information for one or more haptics effects from the one or more MIHS units, wherein the timing information for one or more haptics effects comprises offsets associated with one or more haptics effects; and render the bitstream based on the obtained timing information.

[0009] The following detailed description and attached drawings will further clarify the characteristics, properties, and various effects of the disclosed protected subject matter. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic diagram of a simplified block diagram of a communication system according to an embodiment of the present disclosure. [Figure 2] This is a schematic diagram of a simplified block diagram of a streaming system according to an embodiment of the present disclosure. [Figure 3A] This is a schematic diagram of a simplified block diagram of a haptic encoder according to an embodiment of the present disclosure. [Figure 3B] This is a schematic diagram of a simplified block diagram of a haptics decoder and haptics renderer according to an embodiment of the present disclosure. [Figure 4] This is an illustrative diagram of a haptics frame according to an embodiment of the present disclosure. [Figure 5] This is an illustrative diagram of an MIHS unit according to an embodiment of the present disclosure. [Figure 6] This is an exemplary flowchart illustrating a process for processing haptic media according to an embodiment of the present disclosure. [Figure 7] This is a diagram of a computer system suitable for implementing an embodiment. [Modes for carrying out the invention]

[0011] According to one aspect of the present disclosure, a method, system, and non-temporary storage medium for parallel processing of dynamic mesh compression are provided. Embodiments of the present disclosure may also be applied to static meshes.

[0012] Referring to Figures 1 and 2, one embodiment of the present disclosure for carrying out the encoding and decoding structure of the present disclosure is described.

[0013] Figure 1 shows a simplified block diagram of a communication system 100 according to one embodiment of the present disclosure. The system 100 may include at least two terminals 110, 120 interconnected via a network 150. In the case of one-way data transmission, the first terminal 110 can encode video data, which may include mesh data, at its local location for transmission to the other terminal 120 via the network 150. The second terminal 120 can receive the encoded video data from the other terminal via the network 150, decode the encoded data, and display the restored video data. One-way data transmission may be common in applications such as media provision.

[0014] Figure 1 shows a second pair of terminals 130, 140 provided to support the bidirectional transmission of encoded video, which may occur, for example, during a video conference. In the case of bidirectional data transmission, each terminal 130, 140 can encode video data captured at a local location for transmission to the other terminal via the network 150. Each terminal 130, 140 can also receive encoded video data transmitted by the other terminal, decode the encoded data, and display the restored video data on a local display device.

[0015] In Figure 1, terminals 110-140 may be, for example, servers, personal computers, and smartphones, and / or any other type of terminal. For example, terminals (110-140) may be laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 150 represents any number of networks that transmit encoded video data between terminals 110-140, including, for example, wired and / or wireless communication networks. Communication network 150 can exchange data over circuit-switched and / or packet-switched channels. Typical networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this description, the architecture and topology of network 150 may not be important to the operation of this disclosure unless described below herein.

[0016] Figure 2 shows an example of a disclosed protected use case, illustrating the arrangement of a video encoder and decoder in a streaming environment. The disclosed protected material may be used in other video-enabled applications, such as video conferencing, digital TV, and storing compressed video on digital media including CDs, DVDs, and memory sticks.

[0017] As shown in Figure 2, the streaming system 200 may include a capture subsystem 213 that includes a video source 201 and an encoder 203. The streaming system 200 may further include at least one streaming server 205 and / or at least one streaming client 206.

[0018] The video source 201 can, for example, generate a stream 202 including a 3D mesh and metadata associated with the 3D mesh. The video source 201 may, for example, include a 3D sensor (e.g., a depth sensor) or a 3D imaging technique (e.g., one or more digital cameras), and a computing device configured to generate the 3D mesh using data received from the 3D sensor or 3D imaging technique. The sample stream 202 may have a larger data volume compared to an encoded video bitstream, and can be processed by an encoder 203 coupled to the video source 201. The encoder 203 may include hardware, software, or a combination thereof, to enable or implement the disclosed claimed aspects, as will be described in more detail below. The encoder 203 can further generate an encoded video bitstream 204. The encoded video bitstream 204 may have a smaller data volume compared to the uncompressed stream 202, and can be stored on a streaming server 205 for later use. One or more streaming clients 206 can access the streaming server 205 to obtain a video bitstream 209, which can be a copy of the encoded video bitstream 204.

[0019] The streaming client 206 can include a video decoder 210 and a display 212. The video decoder 210 can, for example, decode the video bitstream 209, which is a copy of the input encoded video bitstream 204, and generate an output video sample stream 211 that can be rendered on the display 212 or another rendering device (not shown). In some streaming systems, the video bitstreams 204 and 209 can be encoded in accordance with a specific video encoding / compression standard.

[0020] Referring to FIGS. 3A and 3B, an embodiment of the present disclosure for implementing a haptics encoder 300 and a haptics decoder 350 will be described.

[0021] As shown in FIG. 3A, the haptics encoder 300 can receive both descriptive haptics data and waveform haptics data. Therefore, it can be said that the haptics encoder 300 can process three types of input files: .ohm metadata files (Object Haptics Metadata: a text file format for haptics metadata), descriptive haptics files (.ivs, .ahap and .hjif), and waveform PCM files (.wav). Examples of descriptive data include Apple's .ahap (Apple Haptic and Audio Pattern: a file format similar to JSON that specifies haptic patterns, which represents an expected haptics output obtained from a set of modulated continuous signals and a set of parameters obtained from modulated transient signals), Immersion's .ivs (which represents an expected haptics output obtained from parameters derived from a set of basis effects using a set of parameters), and the proposed MPEG format .hjif (Haptics JSON Interchange Format). Examples of waveform pulse-code modulation (PCM) signals include .ohm input files containing metadata information.

[0022] According to one embodiment, the haptics encoder 300 can process two types of input files differently. In the case of descriptive content, the haptics encoder 300 can perform semantic analysis on the input and transcode the data into the proposed coded representation (if necessary).

[0023] In one embodiment, the .ohm metadata input file may include a description of the haptics system and setup. In particular, it may include the names of each associated haptics file (either a description file or a PCM file) along with a description of the signal. It may also provide a correspondence between each channel of the signal and a target part of the user's body. In the case of the .ohm metadata input file, the haptics encoder extracts metadata by obtaining the associated haptics file from the URI, encodes the obtained metadata by extracting metadata from the .ohm file based on its type, and maps the encoded metadata to the metadata information of the data model.

[0024] In one embodiment, descriptive haptics files (e.g., .ivs, .ahap, and .hjif) can be encoded through a simple process. First, the haptics encoder 300 specifically identifies the input format. If the input format is an .hjif file, transcoding is not necessary; the file can be further edited, compressed into binary format, and finally packetized into an MIHS stream. Transcoding is required when using an .ahap or .ivs input file. First, the haptics encoder 300 performs semantic analysis on the input file information and transcodes it so that it is formatted according to the selected data model. After transcoding, the data can be exported as an .hjif file, an .hmpg binary file, or an MIHS stream.

[0025] In one embodiment, the haptic encoder 300 can perform signal analysis to interpret the signal structure of a .wav file and convert it into a proposed encoded representation. In the case of waveform PCM content, the signal analysis process can be divided into two subprocesses by the haptic encoder 300. After performing frequency band decomposition on the signal, in the first subprocess, the low frequencies can be encoded using a keyframe extraction process. Then, one or more low-frequency bands can be reconstructed, and the error between this signal and the original low-frequency signal can be calculated. This residual signal can be added to one or more of the original high-frequency bands before encoding using a wavelet transform. Encoding using a wavelet transform is the second subprocess. In one embodiment, if several low-frequency bands are used, the residual errors of all low-frequency bands are added to the high-frequency bands before encoding. In embodiments where several high-frequency bands are used, the residual errors of one or more low-frequency bands are added to the first high-frequency band before encoding.

[0026] In one embodiment, keyframe extraction involves obtaining low-frequency bands from the frequency band decomposition and analyzing their contents in the time domain. In another embodiment, wavelet processing may involve obtaining high-frequency bands from the frequency band decomposition and low-frequency residuals, and dividing these into blocks of equal size. These signal blocks of equal size are then analyzed using a psychohaptic model. Lossy compression can be applied by quantizing the blocks using wavelet transforms, with the assistance of the psychohaptic model. Finally, each block is saved as a separate effect of one band. This is done during formatting. In binary compression, lossless compression can be applied using appropriate coding techniques, such as the Set partitioning in hierarchical trees (SPIHT) algorithm or arithmetic coding (AC).

[0027] As shown in Figure 3A, the haptics encoder 300 can be configured to encode descriptive haptics data and quantized haptics data, and the haptics encoder 300 can output three types of formats: an interchange format (.hjif), a binary compressed format (.hmpg), and a streaming format (e.g., MPEG immersive haptic stream (MIHS)). The .hjif format is a human-readable format based on JSON, and because the .hjif format can be easily parsed and manually edited, it is an ideal interchange format, especially when designing / creating content. For distribution, the .hjif data can be compressed into a more memory-efficient binary .hmpg bitstream. This compression may be lossy compression using various parameters that affect the encoding depth of the amplitude and frequency that make up the bitstream. For streaming, the data can be compressed and packetized to obtain an MPEG-I haptic stream (MIHS). The three formats described above serve complementary purposes, and lossy one-to-one conversions can be performed between them.

[0028] As shown in Figure 3B, the haptics decoder 350 can accept either a .hmpg compressed binary file format or a MIHS bitstream as input. The haptics decoder 350 can output a .hjif interchange format that can be used directly for rendering. Both input formats can undergo binary decompression, extracting both metadata and the data itself from the file and mapping it to a selected data structure. The data can then be exported to the haptics renderer 380 in .hjif format.

[0029] As shown in Figure 3B, the renderer 380 includes a synthesizer. The synthesizer can render haptic data obtained from a .hjif input file to obtain a PCM output file. The information transfer capability for rendering and / or synthesis is high. In one embodiment, the synthesizer analyzes the input file and performs high-level synthesis distribution between vectors, wavelets, etc. The synthesis process then descends to the band components of the codec, and the synthesis process is invoked. Subsequently, all bands of a particular channel are mixed by a simple adder to recreate the desired haptic signal.

[0030] In one embodiment, the root of the hierarchical data model is defined by a haptic experience. This provides information about the file's date and format version, describes the haptic experience, enumerates the various avatars (i.e., bodily representations) used throughout the experience, and defines all haptic perceptions.

[0031] In one embodiment, a self-contained stream format for transporting MPEG-I haptic data may employ a packetization approach, which may include two levels of packetization: an MPEG-I haptic stream (MIHS) unit covering a time span and containing zero or more MIHS packets, and an MIHS packet containing metadata or haptic effect data.

[0032] In this embodiment, each MIHS unit covers a non-overlapping haptics delivery time width, i.e., each MIHS unit starts from the end of the previous MIHS unit and covers a time width defined by its time width field. Following an MIHS unit, if it is not the last MIHS unit of the haptics experience, the next MIHS unit follows. All MIHS packets in an MIHS unit have the start time and time width of the MIHS unit containing these MIHS packets.

[0033] In some embodiments, an MIHS unit may be referred to as a NAL unit (network abstraction layer unit) related to haptic data. In some embodiments, an MIHS unit may be referred to as an MIHS sample related to haptic data.

[0034] In one embodiment, the MIHS unit may be either a synchronous or asynchronous unit. In the case of a synchronous unit, it resets the previous effect, providing a haptic experience independent of the previous MIHS unit. An asynchronous unit is a continuation of the previous MIHS unit, and it cannot be decoded or rendered independently without decoding the previous MIHS unit.

[0035] In some embodiments, haptic signals can be encoded in multiple channels. In some embodiments, haptic channels can be used to define signals that are rendered at specific body positions using dedicated actuators / devices. Metadata stored at the channel level may include information such as gain associated with the channel, mix weights, and desired body positions for haptic feedback, and may include information such as reference devices and / or directions as appropriate. Additional information such as desired sampling frequency and number of samples may also be provided. Finally, the haptic data of the channels is stored in a set of haptic bands defined by frequency ranges. The haptic signals of channels in a particular frequency range are described by haptic bands. The bands are defined by the type of haptic effect and a sequential list, with each band containing a set of keyframes. In some embodiments, for any type of haptic band, the haptic effect can be defined using at least position and type. Depending on the band type and the type of effect, additional properties must be specified, including phase, base signal, composition, and the number of consecutive haptic keyframes describing the effect. For any type of haptic band, the haptic effect can be defined using at least position and type. The position may indicate either the temporal or spatial position of the effect. In some embodiments, a value of 0 represents the relative starting position of the experience according to the dependent variable of the set perceptual modality. The unit of temporal haptic feedback may be milliseconds by default, and the unit of spatial haptic feedback may be millimeters by default. Since the binary delivery format does not have any defined time intervals, i.e., the concept of frames or samples, in this embodiment the "starting position of the experience" is disclosed.

[0036] Depending on the band type and effect type, additional properties can be specified, including phase, base signal, composition, and the number of consecutive haptic keyframes describing the effect.

[0037] In one embodiment of this disclosure, a new element of frames can be defined in the haptics data hierarchy. Thus, in one embodiment, each haptics channel consists of one or more frames. A time width is defined for each frame, and no two frames overlap.

[0038] In one embodiment, this disclosure defines a new element called a frame in the haptics data hierarchy. ● Haptics Channel 〇 Haptics Frame ■ Haptics Band ● Haptic effect

[0039] As shown in Figure 4, each haptic channel consists of one or more frames. Each frame has a defined time width, and no two frames overlap. As shown in Figure 4, the temporal position of the effect is defined as an offset from the start timing of the frame that carries the effect.

[0040] One direct advantage of using haptic frames in haptic elementary streams is that mapping the stream to ISOBMFF or any other file format that uses temporal samples becomes easier and more efficient. For example, each haptic frame is mapped to one ISOBMFF sample.

[0041] In embodiments of this disclosure, a haptic synchronization frame is defined as a haptic frame that resets the output of all previous effects. That is, if the player starts before or at this frame, the output will be the same. The value of such a frame is that it provides the player with a random access point. Any playback performed at this point will result in consistent playback, regardless of the start time of playback. The synchronization frame can be converted to an ISOBMFF synchronization sample. For example, the synchronization frame becomes an ISOBMFF synchronization sample. Therefore, the random accessibility of the synchronization frame can be used at the system layer.

[0042] Another aspect of this disclosure, which replaces or combines with the haptic frames disclosed herein, namely a network abstraction layer (also referred to as an MPEG immersive haptic stream (MIHS) unit), can provide timing information for the haptic channels shown in Figure 5. In this embodiment, the NAL unit is referred to as the MIHS unit, and conversely, the MIHS unit is also referred to as the NAL unit.

[0043] In one embodiment, an MIHS unit may have the necessary timing information (start time and / or duration). In contrast to channel frames, one or more channels can be contained in a single MIHS unit. It should be noted that the haptics distribution syntax naturally includes the MIHS unit syntax. Similar to the video NAL unit type, MIHS units can be defined using synchronous / random access type or asynchronous type to specify random access. As described above, a synchronous MIHS unit resets all previous haptic effects and initiates one or more new effects.

[0044] In one embodiment, the haptics stream can include both frames and timed-referenced MIHS units. In this embodiment, the timed-referenced MIHS unit can include one or more frames. The timing information of the MIHS unit may be selected as appropriate.

[0045] As shown in Figure 6, process 600 represents an exemplary process for decoding haptic data.

[0046] In operation 605, a media stream containing one or more haptic channels can be received. In some embodiments, the media stream may include one or more audio / video channels. In some embodiments, for example, the media stream may include a .ahap, .hjif, or .ivs input stream.

[0047] In operation 610, one or more moving picture experts group (MPEG) immersive haptics stream (MIHS) units can be obtained from the media stream for one or more haptics channels. In embodiments, an MIHS unit may include timing information for one or more haptic effects. In some embodiments, each of the one or more haptics channels has at least one associated MIHS unit. An MIHS unit may also include one or more haptic frames, each haptic frame relating to a non-overlapping time span in the MIHS unit.

[0048] Operation 615 allows for the acquisition of timing information for one or more haptic effects obtained from one or more MIHS units. In the embodiment, the timing information for one or more haptic effects includes offsets associated with one or more haptic effects.

[0049] In the embodiment, the MIHS unit includes an MIHS start time used as an anchor for the starting point of one or more haptic effects, and the offset associated with one or more haptic effects indicates the effect start time corresponding to the MIHS start time.

[0050] In the embodiment, the MIHS unit further comprises timing information for one or more haptic bands, thereby providing an operation to acquire timing information for one or more haptic bands from one or more MIHS units, the timing information for one or more haptic bands includes offsets associated with one or more haptic bands.

[0051] In the embodiment, the MIHS unit may include an MIHS start time used as an anchor for the starting point of one or more haptic bands, and the offset associated with one or more haptic bands indicates the band start time corresponding to the MIHS start time.

[0052] Operation 620 renders the media stream based on the acquired timing information.

[0053] In some embodiments, a synchronous MIHS unit can be obtained from a media stream. In embodiments, the synchronous MIHS unit is a special type of MIHS unit configured to provide a reset point in the bitstream. In embodiments, the synchronous MIHS unit is mapped to a synchronous sample of the video bitstream corresponding to one or more haptic channels.

[0054] Those skilled in the art will understand that the techniques described herein can be implemented on both the encoder and decoder sides. The techniques described above are implemented as computer software using computer-readable instructions and can be physically stored on one or more computer-readable media. For example, Figure 7 shows a computer system 700 suitable for implementing a particular embodiment of the herein.

[0055] Computer software can be encoded using any suitable machine code or computer language to which mechanisms such as assembly, compilation, and linking can be applied to create code that contains instructions that can be executed directly or via interpretation, microcode execution, etc., by a computer central processing unit (CPU), graphics processing unit (GPU), etc.

[0056] Instructions can be executed on various types of computers or computer components, including, for example, personal computers, tablet computers, servers, smartphones, game consoles, and Internet of Things devices.

[0057] The components shown in Figure 7 for the computer system 700 are illustrative and are not intended to imply any limitation on the scope or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should not be construed as dependent on or requiring any one or any combination of components shown in non-limiting embodiments of the computer system 700.

[0058] The computer system 700 may include certain human interface input devices. Such human interface input devices may respond to input from one or more human users via, for example, tactile input (keystrokes, swipes, data glove movements, etc.), voice input (vocalization, clapping, etc.), visual input (gestures, etc.), or olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as sound (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images acquired from still image cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0059] The input human interface device may include one or more of the following (only one of each is shown): a keyboard 701, a mouse 702, a trackpad 703, a touchscreen 710, a data glove, a joystick 705, a microphone 706, a scanner 707, and a camera 708.

[0060] The computer system 700 may also include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more human users, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen 710, data glove, or joystick 705, although there may also be haptic feedback devices that do not function as input devices). For example, such devices may include audio output devices (e.g., speaker 709, headphones (not shown)), visual output devices (e.g., screen 710 including CRT screens, LCD screens, plasma screens, OLED screens, etc., each with or without touchscreen input functionality, each with or without haptic feedback functionality, some of which may be able to output two-dimensional visual output or output beyond three dimensions through means such as stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0061] The computer system 700 may also include human-accessible storage devices and media associated with storage devices such as CD / DVD ROM / RW 720 having media such as CD / DVD 721, thumb drives 722, removable hard drives or solid-state drives 723, legacy magnetic media such as tapes and floppy disks (not shown), and optical media such as dedicated ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0062] Those skilled in the art will also understand that the term “computer-readable medium” as used in relation to the subject matter protected by this disclosure does not include transmission media, carrier waves, or other transient signals.

[0063] The computer system 700 may also include interfaces to one or more communication networks. These networks may be, for example, wireless, wired, or optical. Networks can further be local, wide-area, metropolitan, automotive, and industrial, real-time, latency-tolerant, etc. Examples of networks include local area networks such as Ethernet, cellular networks including wireless LAN, GSM, 3G, 4G, 5G, LTE, etc., wired or wireless wide-area digital networks for television including cable television, satellite television, and terrestrial television, and automotive and industrial networks including CANBus. Certain networks generally require an external network interface adapter connected to a specific general-purpose data port or peripheral bus 749 (e.g., a USB port on the computer system 700), while others are generally integrated into the core of the computer system 700 by connection to the system bus, as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system 700 can communicate with other entities. Such communications may be one-way, receive only (e.g., broadcast television), transmit only (e.g., CANbus to a specific CANbus device), or bidirectional to other computer systems using local or wide-area digital networks. Such communications may include communications to cloud computing environments 755. Specific protocols and protocol stacks may be used in each of those networks and network interfaces, as described above.

[0064] The aforementioned human interface device, human-accessible storage device, and network interface 754 can be connected to the core 740 of the computer system 700.

[0065] The core 740 may include one or more central processing units (CPUs) 741, graphics processing units (GPUs) 742, specialized programmable processing units in the form of field-programmable gate areas (FPGAs) 743, hardware accelerators 744 for specific tasks, etc. These devices may be connected via a system bus 748 along with internal mass storage units such as read-only memory (ROM) 745, random access memory 746, and internal non-user-accessible hard drives, SSDs, etc. 747. In some computer systems, the system bus 748 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core's system bus 748 or via a peripheral bus 749. Architectures for peripheral buses include PCI, USB, etc. A graphics adapter 750 may be included in the core 740.

[0066] The CPU 741, GPU 742, FPGA 743, and accelerator 744 can execute specific instructions that, in combination, constitute the aforementioned computer code. This computer code can be stored in ROM 745 or RAM 746. Transient data can also be stored in RAM 746, while persistent data can be stored, for example, in the internal mass storage unit 747. High-speed storage and ingestion into any of the memory devices can be enabled by using cache memory that can be closely associated with one or more CPUs 741, GPUs 742, mass storage units 747, ROMs 745, RAM 746, etc.

[0067] A computer-readable medium may have computer code for performing various computer operations. The medium and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be of a type that is well known and available to those skilled in the computer software technology.

[0068] For example, a computer system having the architecture of computer system 700, particularly the core 740, may provide functionality as a result of a processor (including CPUs, GPUs, FPGAs, accelerators, etc.) that runs software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage as described above, as well as media associated with specific storage of the core 740 that are non-temporary in nature, such as the core internal mass storage unit 747 or ROM 745. Software implementing various embodiments of this disclosure may be stored in such devices and executed by the core 740. The computer-readable media may include one or more memory devices or chips as required by specific needs. The software may cause the core 740, specifically the processor (including CPUs, GPUs, FPGAs, etc.) within the core, to execute specific processes or specific parts of specific processes as described herein, including defining data structures stored in RAM 746 and modifying such data structures according to processes defined by the software. In addition to or instead of the foregoing, a computer system may also provide functions resulting from logic (e.g., accelerator 744) wired or otherwise embodied in a circuit to perform a particular process or a particular part of a particular process described herein, either in place of or in conjunction with the software. Where necessary, references to software may encompass logic and vice versa. References to computer-readable media may, where necessary, encompass circuits that store software for execution (such as integrated circuits (ICs)), circuits that embody logic for execution, or both. This disclosure encompasses any suitable combination of hardware and software.

[0069] While this disclosure has described several non-limiting embodiments, there are many variations, rearrangements, and alternative equivalents that fall within the scope of this disclosure. Those skilled in the art will therefore understand that numerous systems and methods not expressly shown or described herein can be devised to embody the principles of this disclosure and thus fall within its spirit and scope. [Explanation of symbols]

[0070] 100 Communication Systems 110 terminals 120 devices 130 devices 150 Communication Networks 200 Streaming Systems 201 Video Sources 202 Sample Streams 203 Encoder 204 video bitstream 205 Streaming Servers 206 Streaming Clients 209 video bitstreams 210 Video Decoder 211 Video Sample Stream 212 displays 213 Capture Subsystem 300 haptic encoders 350 Haptics Decoders 380 Haptics Renderer 600 processes 605 operation 610 operation 615 operation 620 operation 700 Computer Systems 701 Keyboard 702 Mouse 703 Trackpad 705 Joystick 706 Mike 707 Scanner 708 Camera 709 Speaker 710 Touchscreen 720 CD / DVD ROM / RW 721 Medium 722 Thumb Drive 723 Solid State Drive 740 cores 741 CPU 742 GPU 743 FPGA 744 Hardware Accelerators 745 ROM 746 RAM 747 Core internal large-capacity storage unit 748 System Bus 749 Local Buses 750 Graphics Adapter 754 Network Interfaces 755 Cloud Computing Environment

Claims

1. A method for decoding information from haptic data, wherein the method is performed by at least one processor, A step of receiving a bitstream having one or more haptic channels, A step of obtaining one or more moving picture experts group (MPEG) immersive haptics stream (MIHS) units of one or more haptics channels from the bitstream, The aforementioned one or more MIHS units have timing information for one or more haptic effects, and a step, A step of obtaining the timing information of one or more haptic effects from one or more MIHS units, wherein the timing information of one or more haptic effects includes an offset associated with the one or more haptic effects. A step of obtaining a synchronized MIHS unit from the bitstream, The aforementioned synchronous MIHS unit is a type of MIHS unit configured to provide a reset point in the bitstream that resets the output of all previous haptic effects, The synchronized MIHS unit is mapped to a synchronized sample of the video bitstream corresponding to one or more haptic channels, in steps and The steps include decoding the bitstream based on the acquired timing information and A method that includes [a certain feature].

2. The MIHS unit includes an MIHS start time used as an anchor used at the starting point of one or more haptic effects. The offset associated with the one or more haptic effects indicates the effect start time corresponding to the MIHS start time. The method according to claim 1.

3. The MIHS unit further comprises timing information for one or more haptic bands, and the method is A step of obtaining timing information of one or more haptic bands from one or more MIHS units, wherein the timing information of one or more haptic bands includes offsets associated with the one or more haptic bands. The method according to claim 1, further comprising:

4. The MIHS unit includes an MIHS start time used as an anchor used at the starting point of one or more haptic bands. The offset associated with one or more haptic bands indicates the band start time corresponding to the MIHS start time. The method according to claim 3.

5. The method according to claim 1, wherein each of the one or more haptic channels has at least one associated MIHS unit.

6. The method according to claim 1, wherein the MIHS unit further comprises one or more haptic frames, each haptic frame relating to a non-overlapping time width in the MIHS unit.

7. An apparatus configured to perform the method described in any one of claims 1 to 6.

8. A computer program that causes at least one processor to perform the method described in any one of claims 1 to 6.

Images