Method, apparatus, and computer program for decoding time-referenced motion picture experts group (MPEG) immersive haptics stream (MIHS) units

Abstract

A method, an apparatus, and a system for haptic signal processing are provided. The process may include receiving a media stream including at least one haptic track and at least one media track. The process may include obtaining at least one MIHS unit from the media stream and obtaining, from the media stream, a start time associated with each of the at least one MIHS unit. The process may include rendering the media stream based on the start times. Each of the at least one MIHS unit does not overlap in time, and each MIHS unit in the at least one MIHS unit includes a start time for the MIHS unit, and the MIHS unit is associated with one or more haptic channels.

Description

[Technical Field]

[0001] This disclosure is directed to a set of advanced video coding techniques, and more particularly, to encoding and decoding haptic experiences for multimedia presentations. [Background technology]

[0002] Haptic experiences have become part of multimedia presentations. In applications where the multimedia presentation includes aspects of a haptic experience, haptic signals are delivered to a device or wearable, allowing a user to feel the haptic experience in coordination with the visual and / or audio media experience while using the application.

[0003] Recognizing the growing popularity of haptic experiences in multimedia presentations, the motion picture experts group (MPEG) began work on developing compression standards for haptics (both MPEG-DASH and MPEG-I) along with the transmission of compressed haptic signaling in the ISO based media file format (ISOBMFF).

[0004] One of the problems to be solved when incorporating aspects of a haptic experience within a multimedia presentation is the lack of a haptic access unit concept that would allow for mapping a haptic delivery format to a file format with an ISOBMFF sample structure. This makes the mapping difficult and often requires tedious and computationally inefficient modification of haptic sample offsets. Therefore, a solution that addresses this issue is needed. Summary of the Invention

[0005] According to an embodiment, a method for processing haptic data may include steps of receiving a media stream including at least one haptic track and at least one media track; obtaining at least one MIHS unit from the media stream, wherein each of the at least one MIHS unit does not overlap in time, and wherein an MIHS unit within the at least one MIHS unit includes a start time for the MIHS unit, and the MIHS unit is associated with one or more haptic channels; obtaining each start time associated with the at least one MIHS unit from the media stream; and rendering data in the at least one haptic track based on each start time.

[0006] According to an embodiment, an apparatus for processing haptic data may be provided. The apparatus may include at least one memory configured to store program code and at least one processor configured to read the program code and operate as directed by the program code. The program code may include: first receiving code configured to cause the at least one processor to receive a media stream including at least one haptic track and at least one media track; first acquisition code configured to cause the at least one processor to acquire at least one MIHS unit from the media stream, wherein each of the at least one MIHS unit does not overlap in time, each MIHS unit in the at least one MIHS unit includes a start time for the MIHS unit, and the MIHS unit is associated with one or more haptic channels; second acquisition code configured to cause the at least one processor to acquire from the media stream each start time associated with the at least one MIHS unit; and rendering code configured to cause the at least one processor to render the media stream based on the respective start times.

[0007] According to an embodiment, a non-transitory computer-readable medium storing computer instructions may be provided. The instructions, when executed by one or more processors of a device for processing haptic data, may cause the one or more processors to perform the following steps: receiving a media stream including at least one haptic track and at least one media track; obtaining at least one MIHS unit from the media stream, wherein each of the at least one MIHS unit does not overlap in time, wherein the MIHS unit in the at least one MIHS unit includes a start time for the MIHS unit, and the MIHS unit is associated with one or more haptic channels; obtaining from the media stream each start time associated with the at least one MIHS unit; and rendering the media stream based on the each start time.

[0008] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a schematic diagram of a simplified block diagram of a communication system in accordance with an embodiment of the present disclosure. [Figure 2] FIG. 1 is a schematic diagram of a simplified block diagram of a streaming system in accordance with an embodiment of the present disclosure. [Figure 3] FIG. 2 is a schematic diagram of a simplified block diagram of a haptic encoder according to an embodiment of the present disclosure. [Figure 4] FIG. 2 is a schematic diagram of a simplified block diagram of a haptic decoder and a haptic renderer according to an embodiment of the present disclosure. [Figure 5] 10 is an exemplary diagram of a timing model for a haptic track according to an embodiment of the present disclosure. [Figure 6]FIG. 10 is an example flow diagram illustrating a process for decoding haptic data according to an embodiment of the present disclosure. [Figure 7] FIG. 1 illustrates a computer system suitable for implementing embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0010] According to aspects of the present disclosure, a method, system, and non-transitory storage medium are provided for parallel processing of dynamic mesh compression.Embodiments of the present disclosure are also applicable to static meshes.

[0011] 1 and 2, an embodiment of the present disclosure for implementing the encoding and decoding structure of the present disclosure will be described.

[0012] 1 illustrates a simplified block diagram of a communication system 100 in accordance with an embodiment of the present disclosure. The system 100 may include at least two terminals 110, 120 interconnected via a network 150. In the case of a unidirectional transmission of data, the first terminal 110 may code video data, which may include mesh data, at a local location for transmission to the other terminal 120 via the network 150. The second terminal 120 may receive the other terminal's coded video data from the network 150, decode the coded data, and display the recovered video data. Unidirectional data transmission may be common in media serving applications, etc.

[0013] 1 depicts a second pair of terminals 130, 140 provided to support bidirectional transmission of coded video, such as may occur during a video conference. For bidirectional transmission of data, each terminal device 130, 140 may code video data captured at a local location for transmission to the other terminal over network 150. Each terminal 130, 140 may also receive coded video data transmitted by the other terminal, decode the coded video data, and display the recovered video data on a local display device.

[0014] In FIG. 1 , terminals 110-140 may be represented as, for example, servers, personal computers, and smartphones, and / or any other type of terminal. For example, terminals 110-140 may be laptop computers, tablet computers, media players, and / or dedicated video conferencing devices. Network 150 represents any number of networks that carry coded video data between terminals 110-140, including, for example, wireline and / or wireless communication networks. Communication network 150 may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this discussion, the architecture and topology of network 150 may be irrelevant to the operation of the present disclosure unless otherwise described hereinafter.

[0015] 2 depicts the placement of a video encoder and decoder in a streaming environment as an example of an application of the disclosed subject matter. The disclosed subject matter can be used in other video-enabled applications, including, for example, video conferencing, digital TV, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0016] 2, the streaming system 200 may include a capture subsystem 213 that includes a video source 201 and an encoder 203. The streaming system 200 may further include at least one streaming server 205 and / or at least one streaming client 206.

[0017] The video source 201 may generate a stream 202 including, for example, a 3D mesh and metadata associated with the 3D mesh. The video source 201 may include, for example, a 3D sensor (e.g., a depth sensor) or 3D imaging technology (e.g., a digital camera) and a computing device configured to generate a 3D mesh using data received from the 3D sensor or 3D imaging technology. The sample stream 202, which may have a high data volume compared to an encoded video bitstream, may be processed by an encoder 203 coupled to the video source 201. The encoder 203 may include hardware, software, or a combination thereof for enabling or implementing aspects of the disclosed subject matter, as described in more detail below. The encoder 203 may also generate an encoded video bitstream 204. The encoded video bitstream 204 may have a lower data volume compared to the uncompressed stream 202 and may be stored on a streaming server 205 for future use. One or more streaming clients 206 may access the streaming server 205 to retrieve a video bitstream 209, which may be a copy of the encoded video bitstream 204.

[0018] The streaming client 206 may include a video decoder 210 and a display 212. The video decoder 210 may, for example, decode a video bitstream 209, which may be an incoming copy of the encoded video bitstream 204, and generate an outgoing video sample stream 211 that may be rendered on a display 212 or other rendering device (not shown). In some streaming systems, the video bitstreams 204, 209 may be encoded according to a particular video coding / compression standard.

[0019] 3 and 4, embodiments of the present disclosure for implementing haptic encoder 300 and haptic decoder 350 will be described.

[0020] As shown in FIG. 3 , haptic encoder 300 can receive both descriptive data and waveform haptic data. Thus, haptic encoder 300 can process three types of input files: .ohm metadata files (Object Haptic Metadata—a text file format for haptic metadata), descriptive haptic files (.ivs, .ahap, and .hjif), or waveform PCM files (.wav). Examples of descriptive data include Apple's .ahap (Apple Haptic and Audio Pattern—a JSON-like file format for specifying haptic patterns) (which represents the expected haptic output through a set of modulated continuous signals and a set of modulated parameterized temporal signals), Immersion's .ivs (which represents the expected haptic output through a set of basic effects parameterized by a set of parameters), or the proposed MPEG format .hjif (Haptics JSON Interchange Format). An example of a waveform pulse code modulation (PCM) signal is an .ohm input file containing metadata information.

[0021] Depending on the embodiment, haptic encoder 300 may process two types of input files differently: For described content, haptic encoder 300 may semantically analyze the input to transcode the data (if necessary) into a proposed coded representation.

[0022] According to an embodiment, the .ohm metadata input file can include a description of the haptic system and setup. In particular, it may include the name of each associated haptic file (either description or PCM) along with a description of the signal. It also provides a mapping between each channel of the signal and a target body part on the user's body. For the .ohm metadata input file, haptic encoder 300 performs metadata extraction by retrieving the associated haptic file from the URI, encoding it based on its type, and extracting metadata from the .ohm file and mapping it to the metadata information in the data model.

[0023] According to an embodiment, descriptive haptic files (e.g., .ivs, .ahap, and .hjif) can be encoded through a simple process. Haptic encoder 300 first specifically identifies the input format. If the input format is an .hjif file, no transcoding is required; the file can be further edited, compressed into a binary format, and finally packetized into an MIHS stream. If an .ahap or .ivs input file is used, transcoding is required. Haptic encoder 300 first semantically analyzes the input file information and transcodes it so that it is formatted into the selected data model. After transcoding, the data can be exported as an .hjif file, an .hmpg binary file, or an MIHS stream.

[0024] According to an embodiment, haptic encoder 300 can perform signal analysis to interpret the signal structure of a .wav file and convert it into the proposed coded representation. For waveform PCM content, the signal analysis process can be divided into two subprocesses by haptic encoder 300. After performing frequency band decomposition on the signal, in the first subprocess, low frequencies can be encoded using a keyframe extraction process. The low-frequency band can then be reconstructed, and the error between this signal and the original low-frequency signal can be calculated. This residual signal can then be added to the original high-frequency band before encoding using a wavelet transform. Wavelet transform encoding is the second subprocess. According to an embodiment, when several low-frequency bands are used, the residual error from all low-frequency bands is added to the high-frequency band before encoding. In an embodiment where several high-frequency bands are used, the residual error from the low frequencies is added to the first high-frequency band before encoding.

[0025] According to an embodiment, keyframe extraction involves obtaining a low-frequency band from the frequency band decomposition and analyzing its content in the time domain. According to an embodiment, wavelet processing may involve obtaining a high-frequency band from the frequency band decomposition and the low-frequency residual and dividing it into equal-sized blocks. These equal-sized signal blocks are then analyzed with a psychohaptic model. Lossy compression may be applied by wavelet transforming the blocks and quantizing them with the help of the psychohaptic model. Finally, each block is saved as an individual effect within a single band. This is done during formatting. Binary compression may be applied using lossless compression using appropriate coding techniques, such as the Set Partitioning in Hierarchical Trees (SPIHT) algorithm and Arithmetic Coding (AC).

[0026] As shown in FIG. 3 , haptic encoder 300 may be configured to encode description data and quantized haptic data and may output three formats: an interchange format (.hjif), a binary compressed format (.hmpg), and a streaming format (e.g., MPEG Immersive Haptic Stream (MIHS)). The .hjif format is a human-readable format based on JSON that is easily parsed and manually edited, making it an ideal interchange format, especially when designing and creating content. For distribution, the .hjif data can be compressed into a more storage-efficient binary .hmpg bitstream. This compression can be lossy, using various parameters that affect the coding depth of the amplitude and frequency that make up the bitstream. For streaming, the data can be compressed and packetized into an MPEG-I haptic stream (MIHS). The three formats above have complementary purposes, and lossy one-to-one conversions can be performed between them.

[0027] As shown in FIG. 4, haptic decoder 350 can take either the .hmpg compressed binary file format or an MIHS bitstream as input. Haptic decoder 350 can output the .hjif interchange format, which can be used directly for rendering. The two input formats undergo binary decompression to extract both the metadata and the data itself from the file and map it to a selected data structure. The data can then be exported to haptic renderer 380 in the .hjif format.

[0028] As shown in FIG. 4, renderer 380 includes a synthesizer. The synthesizer can render haptic data from an .hjif input file to a PCM output file. The rendering and / or synthesizing is informative. According to an embodiment, the synthesizer parses the input file and performs advanced synthesis distribution between vectors, wavelets, etc. The synthesis process then proceeds to the band components of the codec in which the synthesis process is invoked. All bands of a given channel are then mixed by a simple additive operator to recreate the desired haptic signal.

[0029] According to an embodiment, the haptic experience defines the root of a hierarchical data model: it provides information about the file date and format version, it describes the haptic experience, it lists the various avatars (i.e., body representations) used throughout the experience, and it defines all haptic perceptions.

[0030] According to an embodiment, a haptic signal can be encoded in multiple channels. In some embodiments, a haptic channel can define a signal to be rendered at a specific body position by a dedicated actuator / device. Metadata stored at the channel level can include information such as the gain, blend weight, desired body position of haptic feedback, and optionally, a reference device and / or direction associated with that channel. Additional information such as a desired sampling frequency or number of samples can also be provided. Ultimately, the haptic data for a channel is contained in a set of haptic bands defined by their frequency ranges. A haptic band describes the haptic signal of a channel within a given frequency range. A band is defined by a type and a sequential list of haptic effects, each containing a set of keyframes. For any type of haptic band, a haptic effect can be defined by at least its position (temporal or spatial) and type. Depending on the type of band and the type of effect, further characteristics describing the effect can be specified, including the phase, base signal, configuration, and number of consecutive haptic keyframes.

[0031] According to an embodiment, a haptic data hierarchy is defined in this disclosure. ●Haptic Channel ○Haptic band Haptic Effects

[0032] According to an embodiment, a self-contained stream format for carrying MPEG-I haptic data may use a packetization approach and may include two levels of packetization: an MPEG-I haptic stream (MIHS) unit, which covers a period of time and includes zero or more MIHS packets, and an MIHS packet, which includes metadata or haptic effect data. In an embodiment, an MIHS unit may be referred to as a network abstraction layer unit associated with the haptic data. In an embodiment, an MIHS unit may be referred to as an MIHS sample associated with the haptic data.

[0033] According to an embodiment, each MIHS unit can cover a non-overlapping duration of haptic rendering time. That is, it can start at the end of the previous MIHS unit and cover the time defined by its duration field. An MIHS unit can be followed by the next MIHS unit until it is the last MIHS unit of the haptic experience. Every MIHS packet of an MIHS unit can have the start time and duration of the contained MIHS unit.

[0034] Depending on the embodiment, an MIHS unit may be a synchronous (sync) unit or an asynchronous (non-sync) unit. A synchronous unit resets the previous effect, providing a haptic experience independent of the previous MIHS unit. An asynchronous unit is a continuation of the previous MIHS unit and cannot be decoded and rendered independently without decoding the previous MIHS unit.

[0035] Embodiments of the present disclosure concern temporal reference elements (also referred to as haptic access units, access units, or MIHS units) that are orthogonal to the haptic data hierarchy. That is, each temporal reference MIHS unit contains one or more channels, each channel contains one or more bands, and each band contains one or more haptic effects. In this disclosure, the term temporal reference MIHS unit is used synonymously with the term haptic access unit.

[0036] As shown in FIG. 5, the channel information, band information, and haptic effects are packed into MIHS unit MIHS1.

[0037] According to an embodiment, MIHS units do not overlap in time. In an embodiment, each MIHS unit may have a start time and / or a duration. The start time and duration may be defined in units of a time scale. The time scale may be defined by the number of ticks in that time scale per second. An MIHS unit with a duration can only start at the end of a previous access unit. If an MIHS unit has a start time, it overwrites the duration of the previous MIHS unit if the start time of the current MIHS unit appears before the end of the previous MIHS unit's duration.

[0038] According to the embodiment, the position of an MIHS effect is defined as an offset from the start time of the MIHS unit that carries that effect. In other or the same embodiment, the position of a haptic effect in an MIHS unit may be defined as an offset from the start time of the MIHS unit that carries that effect. In other or the same embodiment, an MIHS unit may not contain an effect. That is, an empty access unit that does not show a haptic effect begins during this access unit.

[0039] In accordance with aspects of the present disclosure, an MIHS unit may consist of multiple MIHS packets, each with a type that defines what information the packet carries. Exemplary MIHS packet types are sensory information, body part information, device information, channel information, band information, effect information for effects carrying the MIHS unit, or empty or synchronization information for MIHS packets in a synchronous MIHS unit. A synchronous MIHS unit can reset all previous effects and is a random access point.

[0040] An advantage of using MIHS units in haptic elementary streams is that mapping the haptic stream to ISOBMFF or any other time-sample-based file format is efficient and simple. In some embodiments, each MIHS unit can be mapped to an ISOBMFF sample in the corresponding audio / video stream.

[0041] Therefore, according to an embodiment, a method for defining a time reference access unit for a haptic signal may be provided. In the method, information may be packaged into one or more MIHS units. Each MIHS unit may cover a period of time and have a start time and / or duration in units of a time scale. All haptic aspects, such as perception, device, channel, band, and haptic effect, may be packaged in the MIHS unit. Metadata or binary aspects of the information may be included in the MIHS unit or MIHS packet, and the anchor point for the start time of any haptic effect within the MIHS unit is the start time of the MIHS unit, and the start time of any effect does not exceed the duration of the MIHS that carries this effect. The type of MIHS unit may be identified by a field in the MIHS unit header. Some MIHS units are defined as synchronous access units, which reset all previous effects, thereby providing random access points / synchronization points within the stream and are mapped to ISOBMFF synchronization samples.

[0042] In one example, the start of an experience may be defined as a common anchor point for all effects in a stream. For example, the first effect can have position 0, and the positions of all other effects can be defined relative to the position of the first effect. Then, in the case of ISOBMFF delivery of a haptic channel, the effect position should be relative to the start time of the sample carrying that effect. Then, when a haptic channel is delivered with ISOBMFF, the position of that effect needs to be adjusted. Similarly, after parsing the ISOBMFF, the position of the effect needs to be readjusted by adding the start time of the sample before sending it to the haptic decoder.

[0043] In another or the same example, two types of ISOBMFF tracks may be provided: first, tracks where the track start time is tracked as an anchor to the position of the effect, and second, tracks where the sample start time is tracked as an anchor.

[0044] In another or the same example, a sample structure within a haptic elementary stream may be defined, where a haptic channel may consist of one or more samples / frames, and the timing of each effect within each sample / frame is relative to that sample.

[0045] As an example, according to one embodiment, a method for decoding MIHS units may include receiving a media stream including at least one haptic track and at least one media track; obtaining at least one MIHS unit from the media stream, wherein none of the at least one MIHS unit overlaps in time with other MIHS units, the MIHS unit includes a start time for the MIHS unit, and the MIHS unit is associated with one or more haptic channels; obtaining from the media stream each start time associated with the at least one MIHS unit; and decoding the media stream based on each start time.

[0046] FIG. 6 describes a process 600 for decoding a temporal reference MIHS unit.

[0047] At act 605, a media stream may be received, the media stream including at least one haptic track and at least one media track.

[0048] At operation 610, at least one MIHS unit is obtained from the metadata stream. In an embodiment, each of the at least one MIHS unit is non-overlapping in time. In an embodiment, an MIHS unit in the at least one MIHS unit may include a start time of the MIHS unit. In an embodiment, the MIHS unit may be associated with one or more haptic channels.

[0049] According to an embodiment, the MIHS unit may further include one or more haptic effects, wherein a haptic effect among the one or more haptic effects includes an offset indicating a start time of the haptic effect relative to a start time of the MIHS unit. In an embodiment, the offset is less than or equal to the duration of the MIHS unit.

[0050] In embodiments, when an MIHS unit does not include a haptic effect, the duration indicates the length of time that no haptic effect is present in the media stream. In some embodiments, a synchronous MIHS unit from at least one haptic track is mapped to an ISO Base Media File Format (ISOBMFF) synchronous sample from at least one media track.

[0051] According to an embodiment, an MIHS unit may include one or more MIHS packets, each of which includes a type parameter that defines what information the MIHS packet carries. The type parameter may be one of a sensory information type, a body part information type, a device information type, a channel information type, a band information type, and an effect loading type.

[0052] A respective start time associated with at least one MIHS unit may be obtained at operation 615. In an embodiment, the MIHS unit may further include a duration, and the start time of the MIHS unit and the duration of the MIHS may be defined in units of a time scale, and the time scale may be defined by the number of tracks per second in the time scale.

[0053] At operation 620, the media streams may be rendered and / or displayed based on their respective start times.

[0054] In an embodiment, the MIHS unit may be signaled in a higher level syntax. The MIHS unit may further include one or more haptic bands that further include one or more haptic effects.

[0055] Those skilled in the art will appreciate that the techniques described herein may be implemented on both the encoder and decoder sides. The techniques described above may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 7 illustrates a computer system 700 suitable for implementing certain embodiments of the present disclosure.

[0056] The computer software can be coded in any suitable machine code or computer language that can be subjected to mechanisms such as assembly, compilation, linking, etc. to generate code containing instructions that can be executed by a central processing unit (CPU), graphics processing unit (GPU), etc. directly or through interpretation, microcode execution, etc.

[0057] The instructions may be executable by various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming consoles, Internet of Things devices, and the like.

[0058] 7 for computer system 700 are examples and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components described in the non-limiting embodiment of computer system 700.

[0059] The computer system 700 may also include certain human interface input devices. Such human interface input devices may respond to input by one or more users through, for example, tactile input (e.g., keyboard, swipe, dataglove movement), audio input (e.g., voice, claps), visual input (e.g., gestures), or olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still camera), and video (e.g., two-dimensional video, three-dimensional video, including stereoscopic video).

[0060] The input human interface devices may include one or more of a keyboard 701, a mouse 702, a trackpad 703, a touchscreen 710, a data glove, a joystick 705, a microphone 706, a scanner 707, and a camera 708 (only one of each is shown).

[0061] The computer system 700 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the user's senses through, for example, tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen 710, data gloves, or joystick 705; however, haptic feedback devices that do not function as input devices may also be present). For example, such devices may include audio output devices (e.g., speakers 709, headphones (not shown)), visual output devices (e.g., screen 710, including CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touchscreen input capabilities, each with or without haptic feedback capabilities, some of which may output two-dimensional visual output or output in more than three dimensions, such as by means of stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0062] The computer system 700 may also include human-accessible storage devices and their associated media, such as CD / DVD ROM / RW 720, including CD / DVD or similar media 721, thumb drives 722, removable hard disk or solid state drives 723, legacy magnetic media, such as tape and floppy disks (not shown), dedicated ROM / ASIC / PLD-based devices, such as security dongles (not shown), and the like.

[0063] Those skilled in the art will also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transitory signals.

[0064] The computer system 700 may also include interfaces to one or more communication networks. Networks may be, for example, wireless, wireline, or optical. Networks may further be local, wide-area, metropolitan, vehicular, and industrial, real-time, delay-tolerant, and the like. Examples of networks include local area networks such as Ethernet; wireless LANs; cellular networks including GSM, 3G, 4G, 5G, LTE, and the like; TV wireline or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV; and vehicle and factory networks including CAN bus. Certain networks generally require an external network interface adapter attached to a particular general-purpose digital port or peripheral bus 749 (e.g., a USB port on the computer system 700). Others are generally built into the core of the computer system 700 by attachment to a system bus as described below (e.g., an Ethernet network interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system 700 can communicate with other entities. Such communications may be one-way receive-only (e.g., broadcast TV) or one-way transmit-only (e.g., a CAN bus to a particular CAN bus device), or may be two-way to other computer systems, for example, using local or wide area digital networks. Such communications may include communications to a cloud computing environment 755. Specific protocols or protocol stacks may be used with each of the networks and network interfaces as described above.

[0065] The above-mentioned human interface devices, human-accessible storage devices, and network interface 754 may be attached to the core 740 of the computer system 700 .

[0066] The core 740 may include one or more central processing units (CPUs) 741, graphics processing units (GPUs) 742, dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) 743, hardware accelerators for specific tasks 744, etc. These devices may be connected through a system bus 748, along with read-only memory (ROM) 745, random access memory (RAM) 746, internal mass storage devices such as internal non-user-accessible hard drives, SSDs, etc. 747. In some computer systems, the system bus 748 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached to the core's system bus 748 directly or through a peripheral bus 749. Architectures for peripheral buses include PCI, USB, etc. A graphics adapter 750 may also be included in the core 740.

[0067] The CPU 741, GPU 742, FPGA 743, and accelerator 744 are capable of executing specific instructions that, in combination, may constitute the above-described computer code. The computer code may be stored in ROM 745 or RAM 746. Temporary data may also be stored in RAM 746, while persistent data may be stored, for example, in an internal mass storage device 747. Rapid storage and retrieval from any of the memory devices may be enabled through the use of cache memory. The cache memory may be closely associated with one or more of the CPU 741, GPU 742, mass storage device 747, ROM 745, RAM 746, etc.

[0068] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those skilled in the computer software arts.

[0069] By way of example, and not limitation, a computer system having architecture 700, and specifically core 740, can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with the user-accessible mass storage devices previously introduced, in addition to specific storage devices of core 740 that are non-transitory in nature, such as core internal mass storage 747 or ROM 745. Software implementing various embodiments of the present disclosure can be stored on such devices and executable by core 740. Computer-readable media can include one or more memory devices or chips, depending on particular needs. The software can cause core 740, and specifically the processor therein (including a CPU, GPU, FPGA, etc.), to perform particular processes or portions of particular processes described herein, including defining data structures stored in RAM 746 and modifying such data structures in accordance with software-defined processes. Additionally, or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator 744) that can operate in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. References to software can encompass logic, where appropriate, and vice versa. References to computer-readable media can encompass circuitry (e.g., integrated circuits (ICs)) storing software for execution, circuitry embodying logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0070] While this disclosure has described several exemplary embodiments, alternatives, permutations, and various substitute equivalents exist and are included within the scope of this disclosure. Thus, it will be apparent to those skilled in the art that numerous systems and methods, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope.

[0071] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Provisional Patent Application No. 63 / 417,184, filed October 18, 2022, and U.S. Patent Application No. 18 / 487,870, filed October 16, 2023. The disclosures of these U.S. applications are incorporated herein by reference in their entirety.

Claims

1. 1. A method for decoding a time-referenced Motion Picture Experts Group (MPEG) Immersive Haptics Stream (MIHS) unit, the method being executed by at least one processor and comprising: receiving a media stream including at least one haptic track and at least one media track; obtaining at least one MIHS unit from the media stream, none of the at least one MIHS unit overlapping in time with other MIHS units, each MIHS unit including a start time for that MIHS unit, and each MIHS unit associated with one or more haptic channels; obtaining from the media stream each start time associated with the at least one MIHS unit; decoding the media streams based on the respective start times; A method having the following.

2. The MIHS unit further includes a duration. The method of claim 1.

3. the start time of the MIHS unit and the duration of the MIHS unit are defined in units of a time scale; the time scale is defined by the number of tracks per second within the time scale; The method of claim 2.

4. the MIHS unit further includes one or more haptic effects; a haptic effect among the one or more haptic effects includes an offset indicating a start time of the haptic effect relative to the start time of the MIHS unit; The method of claim 2.

5. the offset is less than or equal to the duration of the MIHS unit; The method of claim 4.

6. the MIHS unit does not include a haptic effect, and if the MIHS unit does not include a haptic effect, the duration indicates an amount of time during which no haptic effect is present in the media stream. The method of claim 2.

7. The MIHS unit includes one or more MIHS packets, each MIHS packet including a type parameter that defines what information the MIHS packet carries. The method of claim 1.

8. The type parameter is Perceptual information type, Body part information type, Device information type, Channel information type, Bandwidth information type, and Effect-equipped type One of the The method of claim 7.

9. The MIHS unit is signaled in a higher level syntax. The method of claim 1.

10. the MIHS unit further includes one or more haptic bands that further include one or more haptic effects; The method of claim 1.

11. synchronous MIHS units from the at least one haptic track are mapped to ISO Base Media File Format (ISOBMFF) synchronous samples from the at least one media track. The method of claim 1.

12. 1. An apparatus for decoding a time-referenced Motion Picture Experts Group (MPEG) immersive haptics stream (MIHS) unit, comprising: at least one memory configured to store program code; at least one processor configured to read the program code and to act as directed by the program code; The program code, when executed by the at least one processor, causes the at least one processor to perform the method of any one of claims 1 to 11. Device.

13. A computer program comprising instructions, The instructions, when executed by one or more processors of a device for decoding time-referenced Motion Picture Experts Group (MPEG) Immersive Haptics Stream (MIHS) units, cause the one or more processors to perform the method of any one of claims 1 to 11. Computer program.

Images