Method and apparatus for hierarchically encoding semantic information associated with a haptic effect

Abstract

A method is disclosed in which semantic information associated with at least one haptic effect is first obtained (S700). The semantic information is based on a multi-layer hierarchical structure of the semantic information. The decomposed semantic information is then encoded (S704).

Description

[Technical Field]

[0001] At least one embodiment generally relates to a method and apparatus for encoding semantic information associated with a haptic effect. Corresponding rendering methods and apparatus are also disclosed. [Background technology]

[0002] This application claims priority to European Application No. 23305342.0, filed March 14, 2023, and European Application No. 23305500.3, filed April 6, 2023, both of which are incorporated herein by reference in their entireties.

[0003] Immersive systems rely on feedback and interaction to provide users with a fully immersive experience. Interaction may involve traditional control methods that meet the user's needs. Current visual and auditory feedback provides a satisfactory level of realistic immersion. Additional feedback can be provided through haptic effects, allowing human users to perceive the virtual environment with their senses, thereby achieving a superior immersive experience and increased realism. However, haptic technology remains an area of ​​potential advancement to improve the overall user experience in immersive systems.

[0004] Traditionally, an immersive system may include a 3D scene representing a virtual environment and virtual objects placed within the 3D scene. To enhance user interaction with elements of the virtual environment, haptic feedback may be used through stimulation of haptic actuators. Such interaction is based on the concept of a "haptic object," which corresponds to a physical phenomenon conveyed to the user. In an immersive scene, a haptic object provides a haptic effect on a haptic rendering device by defining appropriate haptic actuator stimulation to mimic the physical phenomenon. Different types of haptic actuators allow for the reproduction of different types of haptic feedback.

[0005] An example of a haptic object is an explosion, which can be simulated using vibrations and heat, allowing for different haptic effects to be combined for the user to enhance realism. Immersive scenes typically contain multiple haptic objects, for example, using a first haptic object associated with a global effect and a second haptic object associated with a local effect.

[0006] The principles described herein apply to any immersive environment that utilizes haptics, such as augmented reality, virtual reality, mixed reality, or haptics-enhanced video (or omnidirectional / 360° video) rendering, and more generally to any haptics-based user experience. A scene in an example of such an immersive environment is considered an immersive scene.

[0007] Haptics refers to the sense of touch and includes two dimensions: tactile and kinematic. The first dimension relates to tactile sensations such as friction, roughness, hardness, and temperature, sensed through mechanoreceptors in the skin (Merkel cells, Ruffini endings, Meissner's corpuscles, and Pacinian corpuscles). The second dimension relates to the sense of force / torque, position, and motion / velocity provided by mechanoreceptors in muscles, tendons, and joints. Haptics also contributes to the perception of self-motion and contributes to the proprioceptive system (i.e., perception of one's own body). Thus, the perception of acceleration, velocity, or any body model can be assimilated as a haptic effect. The frequency range is approximately 0–1 kHz, depending on the modality. Most existing haptic signal generators generate vibrations. Exemplary haptic actuators are linear resonant actuators (LRAs), eccentric rotating masses (ERMs), and voice-coil linear motors. These actuators can be incorporated into haptic suits as well as haptic-rendering devices such as smartphones and game controllers.

[0008] Several formats have been defined for encoding haptic signals, including high-level descriptions using XML-like formats (e.g., MPEG-V), parametric representations using JSON-like formats such as Apple Haptic Audio Pattern (AHAP) and Immersion Corporation's HAPT format, and waveform encoding (IEEE 1918.1.1 standard for haptic and kinesthetic signals is currently in development). The HAPT format was recently incorporated into the MPEG ISOBMFF file format specification (ISO / IEC 14496 Part 12). Additionally, the GL Transmission Format (glTF®) is a royalty-free specification for efficient transmission and loading of 3D scenes and models by applications. This format defines an extensible, common public format for 3D content tools and services, streamlining authoring workflows and enabling industry-wide content interoperability.

[0009] Additionally, a new haptic file format has been defined within the MPEG standardization group, which involves a coded representation of haptics. The encoded haptic description file can be output as a human-readable JSON interchange format (e.g., a .gmpg file) or as a compressed binary delivery format (e.g., .mpg) that is particularly suitable for transmission to haptic rendering devices. [Brief explanation of the drawings]

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

[0011] In one embodiment, a method for hierarchically encoding semantic information associated with a haptic effect is disclosed. Various multi-layer hierarchical structures are defined that are suitable for decomposing semantic information associated with various haptic effects into different levels of semantic information.

[0012] FIG. 1 is a block diagram illustrating an example of an immersive system 10 in which various aspects and embodiments are implemented. In the immersive system 10, a user, Alice, uses a haptic rendering device 100 to interact with a server 180, which hosts an immersive scene 190, via a communication network 170. This immersive scene 190 may include various data and / or files representing various elements required for its rendering (e.g., a scene description 191, audio data, video data, a 3D model, and a haptic description file 192). While FIG. 1 illustrates the different elements of the immersive scene 190 as separate elements, the principles described herein also apply when these elements are directly integrated into the scene description and not as separate elements. A mixture of the two options is also possible, with some elements integrated into the scene description and other elements residing in separate files. The immersive scene 190 may be generated under the control of an immersive experience editor 110, which allows for the placement of different elements and the design of an immersive experience. Appropriate description files and various data files representing the immersive experience are generated by an immersive scene generator 111 (also known as an encoder) and encoded in a format suitable for transmission to a haptic rendering device. The immersive experience editor 110 runs on a computer that generates the immersive scene, which is typically hosted on a server. For simplicity, the immersive experience editor 110 is shown as being directly connected to the immersive scene 190 via dotted line 171. In practice, the immersive scene 190 is hosted on a server 180, and the computer running the immersive experience editor 110 is connected to the server 180 via a communications network 170.

[0013] Haptic rendering device 100 includes processor 101. Processor 101 can be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors in conjunction with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor can perform data processing such as decoding and rendering haptic signals, input / output processing, and / or other functions to operate the device within an immersive system.

[0014] The processor 101 may be connected to an input unit 102 configured to communicate user actions. For this purpose, multiple types of inputs and modalities can be used. A physical keypad or a touch-sensitive surface are typical examples of inputs suitable for this application, although voice control can also be used. Furthermore, the input unit 102 may include a digital camera capable of capturing still or video images in two dimensions, or more complex sensors capable of determining depth information in addition to images or videos, thus capturing a full 3D representation. The processor 101 may be connected to a display unit 103 configured to output visual data to be displayed on a screen. Multiple types of displays can be used for this purpose, such as liquid crystal display (LCD) or organic light-emitting diode (OLED) units. The processor 101 may also be connected to an audio unit 104 configured to generate audio data that is converted into acoustic waves via an adaptive transducer, e.g., a speaker. The processor 101 may be connected to a communication interface 105 configured to exchange data with external devices. Communication preferably uses a wireless communication standard (e.g., cellular (e.g., LTE) communication, Wi-Fi communication, etc.) to provide mobility for the haptic rendering device. Processor 101 may access information and store data from memory 106, which may include multiple types of memory, including random access memory (RAM), read-only memory (ROM), a hard disk, a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, or any other type of storage device. By way of example, processor 101 may access information and store data from memory not physically located on haptic rendering device 100, such as memory on a server, a home computer, or other external device.

[0015] Processor 101 is connected to haptic unit 107 configured to provide haptic feedback to the user, the haptic feedback being described in haptic description file 192 associated with scene description 191 of immersive scene 190. Haptic description file 192 describes the type of feedback to be provided according to a syntax described below. Such description files are typically transmitted from server 180 to haptic rendering device 100. Haptic unit 107 can include a single haptic actuator or multiple haptic actuators located at multiple locations on the haptic rendering device. Different haptic units may have different numbers of actuators and / or the actuators may be located at different locations on the haptic rendering device.

[0016] Processor 101 can be configured to generate haptic signals. In other words, processor 101 can be configured to apply low-level signals to haptic actuators to generate haptic effects. Such low-level signals can be represented in various formats, such as by metadata or parameters in a description file or using digital encoding of an extracted analog signal (e.g., PCM or LPCM).

[0017] Processor 101 may be configured to receive power from power source 108 and distribute and / or control the power to other components within device 100. The power source may be any suitable device for providing power to the device. By way of example, the power source may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0018] While FIG. 1 depicts the processor 101 and the other elements 102-108 as separate components, it will be understood that these elements may be integrated within an electronic package or chip. It will be understood that the haptic device 100 may include any subcombination of the elements described herein while remaining consistent with the present embodiment. The processor 101 may further be connected to other peripheral devices or units not shown in FIG. 1 , which may include one or more software and / or hardware modules that provide additional features, capabilities, and / or wired or wireless connectivity. For example, peripheral devices may include sensors such as a universal serial bus (USB) port, a vibration device, a television receiver / transmitter, a hands-free headset, a Bluetooth module, a frequency modulation (FM) radio unit, a digital music player, a media player, a video game player module, an internet browser, etc. For example, the processor 101 may be connected to a localization unit configured to locate the haptic rendering device within its environment. The localization unit may integrate a GPS chipset that provides longitude and latitude location information about the current location of the haptic rendering device, and may also integrate other motion sensors such as an accelerometer and / or electronic compass that provide location services.

[0019] Typical examples of haptic rendering device 100 include a haptic suit, a smartphone, a game controller, a haptic glove, a haptic chair, a haptic prop, a motion platform, etc. However, any device or device configuration that provides similar functionality can be used as haptic rendering device 100 while complying with the present principles.

[0020] In one example, haptic rendering device 100 does not include display unit 103, but does include a haptic unit. In such an embodiment, the device does not visually render a scene, but only renders haptic effects. However, the device can prepare data for display and enable another device, such as a screen, to perform the display. Examples of such devices include a haptic suit or a motion platform.

[0021] In one example, haptic rendering device 100 does not include haptic unit 107, but does include a display unit. In such an embodiment, the device does not render haptic effects and only visually renders a scene. However, haptic rendering device 100 may prepare data for rendering haptic effects, thereby enabling another device, such as a haptic prop, to perform the haptic rendering. Examples of such devices include a smartphone, a head-mounted display, or a laptop.

[0022] In one example, haptic rendering device 100 does not include a display unit or a haptic unit. In such an example, device 100 does not visually render a scene or render haptic effects. However, device 100 can prepare display data so that another device, such as a screen, can perform the display, or prepare data for rendering haptic effects so that another device, such as a piezo prop, can perform the haptic effect rendering. Examples of such devices include a computer, a game console, an optical media player, or a set-top box.

[0023] As an example, immersive scene 190 and associated elements may be hosted directly in memory 106 of haptic rendering device 100, allowing for local rendering and interaction. Alternatively, device 100 may also include immersive experience editor 110, allowing for complete standalone operation without requiring, for example, communications network 170 or server 180.

[0024] FIG. 2 shows an example flowchart illustrating a method for encoding an immersive description file. This encoding process 200 may be implemented, for example, as a module of the immersive scene generator 111 of the immersive editor 110 and typically runs on a computer that generates files describing the immersive scene. Alternatively, it may be implemented on a computer or specific hardware platform dedicated to encoding immersive description files. The inputs to the encoding method are a metadata file 201, at least one description file, and / or at least one low-level haptic signal file 203 (e.g., waveform PCM file format). The metadata file 201 may be based, for example, on the "OHM" haptic object file format. The signal file 203 represents an analog signal applied to a haptic actuator and is conventionally encoded into a waveform PCM file format (e.g., WAV file format). The description file 202 may be based, for example, on the AHAP, IVS, HJIF, or HAPT file format. AHAP is a JSON-like file format that specifies haptic patterns through key-value pairs similar to dictionary literals, but within a text file.

[0025] Metadata is extracted from the metadata file 201 (step S210), allowing the description file and / or signal file to be identified. The description file is parsed and transcoded in step S211. In step S212, the signal file 203 is processed using common signal analysis techniques to generate keyframes. Interpolation between the keyframes is then performed or wavelet coding (based on the English abbreviation SPIHT: Set Partitioning In Hierarchical Trees) is used to generate a binary encoded stream. Step S212 is shown in more detail in Figure 3. It includes frequency band decomposition of the signal in step S2120, and keyframe extraction (step S2122) or wavelet coefficient extraction (step S2124).

[0026] 2, in step S220, an exchange file 204 is generated according to one of the embodiments described herein. The exchange file 204 may be compressed (or more precisely, binary encoded) in step S230, and then packetized in step S232 for distribution in a format suitable for transmission (e.g., a distribution stream 205 that is more compact than the exchange file 204).

[0027] The exchange file 204 is a human-readable file, for example based on the glTF, XML, or JSON format. The delivery stream 205 is a binary-encoded stream that may be packetized, for example based on an MPEG file format adapted for streaming to a decoder or broadcast. As an example, the delivery stream 205 is an MIHS stream (MPEG-I Haptic Stream).

[0028] 4 shows an example of a process for decoding a delivery stream 205 and generating an immersive description file. The delivery stream 205 is first depacketized in step S310, which is the reverse of the packetization step S232 on the encoder side. The resulting data is further decoded in step S312. The output of step S312 is an interchange file 204, which is the reverse of the compression step S230 on the encoder side.

[0029] In rendering process 4000, exchange file 204 output by step S312 may be analyzed in synthesis step S410 to generate an appropriate haptic signal 206. In a variation shown by dotted lines, haptic signal 206 may be synthesized directly from exchange file 204 generated by encoding process 200. Decoding process 3000 and rendering process 4000 may both be implemented by, for example, haptic rendering device 100 of FIG. 1. Decoding process 3000 and rendering process 4000 may also be performed by a device separate from but in communication with the haptic rendering device, such as a computer, a set-top box, a smartphone, or a computing instance in the cloud.

[0030] FIG. 5 shows an example of the structure of an interchange file format for describing an immersive scene. Data structure 300 represents immersive scene 190, which can be broken down into a series of layers. At the top layer, metadata 301 describes high-level metadata information about the overall haptic experience defined in data structure 300 and a list of avatars (body representations), described later in the file. These avatars allow for target locations on the body for haptic stimulation. Haptic effects are described through perceptual lists 310 through 31N. These perceptual lists correspond to haptic signals associated with specific perceptual modalities, such as vibration, force, position, velocity, and temperature. A perceptual list includes metadata 320 describing the haptic content of the signal, device 321 describing the specifications of the haptic device for which the signal was designed, and a list of haptic tracks 331 through 33N. A haptic track includes metadata 340 describing the track's content, associated gain values, mixing weights, body position information, and a reference to the haptic device specification (defined at the perceptual level). The track ultimately includes a list of haptic bands 351 through 35N, each defining a subset of signals within a given frequency range. For example, haptic band 351 could correspond to a frequency range of 0–50 Hz, while haptic band 35N could correspond to a frequency range above 2 kHz. A haptic band includes band data 360, which describes the band's frequency range, the encoding modality type (vector or wavelet), the band type (transient, curve, or wave), and optionally the curve type (cubic, linear, or unknown) or window length. A haptic band is defined by a haptic effect list 371–37N. Finally, a haptic effect includes effect data 380 and a keyframe list 391–39N, where keyframes are defined by their position (time reference), frequency, and amplitude. The effect data describes the type of underlying signal—selected from sine, square, triangle, upsawtooth, and downsawtooth—as well as providing a time reference, such as a timestamp. As shown in the example of FIG. 6, a low-level haptic signal can be reconstructed by combining keyframes of haptic effects in different bands.

[0031] FIG. 6 shows an example of a signal encoded using two haptic bands. In this approach, a low-level haptic signal is encoded using two frequency bands: low frequency band 410 and high frequency band 420. Each band defines a portion of the signal in a given frequency range. In this example, the low frequency band corresponds to frequencies below 72.5 Hz, and the high frequency band corresponds to frequencies above 72.5 Hz. On the playback side, the device combines (i.e., adds) the two portions to produce final haptic signal 440.

[0032] The frequency band data is reconstructed based on keyframes and according to the haptic band type selected from transient, curve, and wave bands. Furthermore, the wave band can use two encoding modalities: vector or wavelet. Each band consists of a set of effects, each defined by a list of keyframes, represented by points in the diagram. The data contained in the effects and keyframes is interpreted differently depending on the haptic band type and encoding modality.

[0033] In the transient band, each effect stores a series of keyframes that define the position, amplitude, and frequency. A keyframe represents a transient event. The signal is reconstructed using the type of period-based signal specified in the effect metadata, with the amplitude specified in the keyframe and the period given by the frequency of the keyframe. A transient event is a signal that is generated for a very short period of time (only a few periods). The number of periods generated is determined by the decoder.

[0034] In a curve band, each effect stores a set of keyframes that define the position (time reference) and amplitude. The keyframes represent the control points of the curve, from which the curve is generated by an interpolation process. The type of interpolation function can be either cubic or linear, and is specified in the band's metadata (380 in Figure 5). The signal can be reconstructed by interpolating between its amplitudes based on the time references of the keyframes.

[0035] For vector wave bands, the effect stores a series of keyframes that define the position (time reference), amplitude and frequency. In this case, a signal is generated using the period-based signal type specified in the effect metadata, with the amplitude specified in the keyframes and the period given by the frequency of the keyframes.

[0036] The SPIHT wavelet encoding method can be used for wavelet bands. For example, in a wavelet band, the effect can preserve the contents of one wavelet block. For each coefficient of the wavelet-transformed and quantized signal, a keyframe is included, indicating the wavelet amplitude value. The coefficients are normalized to the range [-1, 1]. Additionally, the keyframe preserves the original maximum amplitude and maximum number of bits used. In this case, the signal can be reconstructed by performing an inverse wavelet transform using the coefficients.

[0037] Frequency band decomposition can use low-pass and high-pass filters to separate a signal into low- and high-frequency bands. Each band is then processed differently. Various techniques are used to encode the high-frequency portion. One solution is to divide the high-frequency signal into small, fixed-length windows and decompose the signal across the frequency spectrum using a short-time Fourier transform (STFT). Another solution is to encode the high frequencies using a wavelet transform. The data structure shown in Figure 5 allows the definition of multiple bands with different frequency ranges. These bands are used to store the coefficients of the Fourier or wavelet transform. For the low-frequency portion of the signal, data for this frequency band is stored through a list of keyframe points, defined by timestamp and amplitude. The data also contains information about the type of interpolation method used to reconstruct the signal in this band. The keyframes (i.e., control points) that define the low-frequency band are obtained by simply extracting the local extrema of the low-frequency signal.

[0038] In the example of Figure 6, the low frequency band 410 is defined as a curve band with a single effect 411. Such a representation is particularly suited to the low frequency portion of the signal. Effect 411 is defined by keyframes 4111, 4112, 4113, 4114, 4115, 4116, 4117, 4118, and 4119. The signal in the low frequency band is generated by cubic interpolation between these keyframes. The high frequency band 420 is defined by four effects 421, 422, 423, and 424. Effect 421 is defined as a vector band defined by four keyframes 4211, 4212, 4213, and 4214.

[0039] While Figure 6 shows an example with two sets of bands defining a low frequency range and a high frequency range, the principles also apply when using two or more frequency ranges. In the latter case, the low frequency band is the lowest frequency band and the high frequency band is the highest frequency band. For example, the lowest frequency band may be encoded using a curve band with a single effect, as represented by low frequency band 410 in Figure 6. Other frequency bands may be encoded using any of the other encoding schemes, such as using a wavelet-based vector waveform band, as represented by high frequency band 420 in Figure 6, but using multiple instances of encoding for each frequency band.

[0040] The structural advantage of this solution is that it allows easy packaging of signal data, making it particularly suitable for streaming purposes. Indeed, such a linear structure allows data to be easily divided into small, contiguous packages, without the need for complex data pre-acquisition operations. The signal can be easily restored by recombining the packages, ensuring smooth signal playback. It also allows for the extraction of only the low-frequency portion without considering the high-frequency band, resulting in the reconstruction of a low-quality (but potentially satisfactory) signal.

[0041] Haptic effects can be further described using semantic information to identify the type of effect. This semantic information can be provided in addition to keyframes and wavelet coefficients. This semantic information can be useful for classifying haptic effects according to specific rules and organizing a haptic effect library. For creators, using semantic information to describe haptic effects makes it easier to design haptic experiences. In the industry, haptic editor software already implements haptic effect libraries with semantic information for classifying haptic effects. However, this type of semantic information is not represented in current formats (interchange file formats and distribution stream formats). Current formats (and codecs) use keyframes or wavelet coefficients to encode effects and other information, such as the frequency and amplitude of the signal for each keyframe.

[0042] The primary limitation of current codecs is that they force artists to create haptic effects using ad-hoc haptic editor software that can manipulate the signals. Such tools require both the haptic editor software and specialized knowledge of the human mechanoreceptors for which the haptic effect is designed. While designing haptic effects typically requires the use of such software, user-friendly tools would facilitate the adoption and design of haptics for a wider range of applications.

[0043] The second limitation of current codecs concerns the issue of device adaptation. Current formats allow for the embedding of metadata specific to the target device of the haptic experience, providing the minimum requirements for performing device adaptation. It is not easy to adapt a haptic experience designed for one device to another.

[0044] The third limitation of the codec is the design of device-specific effects, such as washout effects. Some devices require certain effects to be executed along with the haptic experience to comply with the device specifications. For example, kinesthetic devices have positional limitations that prevent the playback of haptic effects when the actuator position reaches the device's limit. Playing a special effect can force the actuator position to be reset, while altering the rendering of the effect to emulate the haptic experience. The current format does not allow for easy design of device-specific effects.

[0045] Adding semantic information to haptic effects may contribute to improving the user experience.

[0046] As an example, semantic information for a haptic effect can be stored using a string (also known as a string of characters), i.e., a series of characters.

[0047] The use of such strings can make it difficult to store and transmit semantic information in a binary format (e.g., binary delivery stream 205). There are two ways to store strings in a binary format. The first method fixes the number of characters, which can cause problems if too few characters are allowed. On the other hand, allowing too many characters can affect the bitrate of the haptic experience. The second method adds an initial field that stores the number of characters, followed by a list of characters. This causes the size of the binary format to fluctuate and does not prevent excessive character usage.

[0048] An additional constraint imposed by describing the semantic information of haptic effects as strings concerns the interpretation of the stored values: while strings allow for human-readable information, they are more difficult for computers to interpret, requiring additional software development effort.

[0049] To overcome these drawbacks, tags can be used with a predefined enumeration of common haptic effects. In binary format, a certain number of bits can be allocated to this enumeration, providing a list of haptic effects. An example of such a list is provided below.

[0050] / / Weapons and Combat ·bullet ·explosion Shotgun ·pistol ·rifle ·punch ·kick ·engine Reload etc. / / music Snare Bass drum Tom Stand tom Hi-hat Crash cymbal Ride cymbal etc. / / UX ·click Double-click ·confirmation ·Misoperation Ringtones ·message etc. / / car ·engine ·Collision sound Gear change etc. / / Character movement ·footsteps etc. / / environmental sounds ·Water drop sound Rain sounds etc.

[0051] However, this solution is not optimal, especially in binary formats, which force a fixed number of bits to be used regardless of the type of haptic effect.

[0052] In contrast, a method for encoding semantic information associated with a haptic effect is disclosed below with reference to FIG. 7, which allows embedding semantic information of the haptic effect that is adapted to the needs of current encoding schemes. More precisely, this method provides scalability through a multi-layer hierarchical structure that can be adapted to various usage scenarios. Furthermore, in some examples, the impact on the bitrate of the binary format can be limited by using a flexible approach that can be adapted to different usage scenarios. As an example, the streaming bitrate is optimized using the proposed multi-layer semantic information encoding. Advantageously, this method enables data reduction and optimization based on the semantic information.

[0053] Furthermore, encoding such semantic information facilitates the haptic content creation process by providing semantic annotations of effects that can be used in place of keyframe representations. It also further facilitates the device adaptation process by allowing each haptic device to use its own unique haptic effect library, with the effect library sharing the same semantics across all devices. For example, instead of using haptic effects described in exchange file 204, a haptic device can recognize the semantic information and generate effects from its own library, ensuring that effects are optimized for that device.

[0054] This method provides a general approach that can be applied to any hierarchical semantic structure.

[0055] 7 shows a flowchart of an example encoding method. This method enables encoding the semantics of haptic effects using a multi-layer hierarchical structure. The hierarchical structure allows haptic effects to be clustered based on their type, application area, or other related rules, and allows haptic effects to be grouped and organized.

[0056] In step S700, semantic information associated with the at least one haptic effect is obtained, for example, from the extracted metadata.

[0057] In step S702, the semantic information is decomposed into different levels (also known as layers) of semantic information hierarchically organized from high-level semantic information to low-level semantic information. The terms "level" and "layer" can be used interchangeably. This decomposition is performed according to a multi-layer hierarchical structure. The lower the level, the more precise the semantic information. An example of such decomposition is shown in FIG. 9, which depicts a multi-layer hierarchical structure using four layers for the semantic representation of haptic effects. Each element (also known as a semantic tag) in the structure represents a concept that can be divided into different concepts at lower levels, or multiple haptic effects. Furthermore, this representation is extensible. The top layer contains six elements, each of which has at least one child element and thus defines a concept. As an example, the semantic information "heat" is decomposed into three levels: the first level is "environment," the second level is "fire," and the third level is "heat." As another example, the semantic information "snare" can be decomposed into four levels: the first level is "music," the second level is "percussion," the third level is "drum," and the fourth level is "snare." Semantic information is thus represented hierarchically, with each level providing more precise information. The multi-layer hierarchical structure shown in Figure 9 is an example. Different data structures can be used to represent semantic information hierarchically. Each layer consists of a list of elements. If an element has one or more children, it is a concept; otherwise, it represents a haptic effect. Concepts can also be represented by generic haptic effects. This is useful for vertical pruning (i.e., layer pruning). A hierarchical structure allows for a pruning mechanism to limit either the depth of the structure (vertical pruning) or the number of branches used (horizontal pruning). This provides more flexibility than enumeration-based representations, which require the use of a fixed number of bits.

[0058] Table 1 below summarizes what can be considered the deep structure. The first layer contains eight initial concepts (UX, character behavior, special effects, weapons and combat, environment, texture, music, and vehicles), and four layers of semantic spread are developed on top of these.

[0059] [Table 1-1]

[0060] [Table 1-2]

[0061] A second representation using three layers is shown in Table 2. This second representation is at a lower level of abstraction, but it encompasses most of the possibilities while limiting the level of detail. For example, for the concept of "music," the third layer concept of "percussion" is divided into two haptic effects: "hard material" and "elastic material." In Table 2, seven haptic effects (glockenspiel, crotal, cowbell, snare drum, tom, kick drum, and cymbal) are associated with "percussion."

[0062] [Table 2]

[0063] A third representation using two layers is shown below in Table 3. This two-layer representation does not include a level of abstraction and maximizes the number of effects stored within the structure.

[0064] [Table 3-1]

[0065] [Table 3-2]

[0066] Thus, Tables 1, 2, and 3 detail all the haptic effects that can be expressed by decomposing semantic information into multiple layers.

[0067] In one example, the multi-layer hierarchy can be applied to codec profiles and profile levels. In fact, for the current version of the codec designed to encode haptic signals, two profiles are currently defined within the MPEG standardization group: the primary profile and the simple parametric profile. The simple parametric profile was developed to facilitate the adoption of MPEG formats for existing haptic interfaces and peripherals on the market. It targets simple devices such as mobile phones and game controllers. The primary profile targets advanced platforms such as simulators and motion platforms. This enables scalable applications and ultra-high-fidelity encoding.

[0068] In both profiles, two profile levels are defined that restrict the number of channels, the number of bands, the perceptual modalities, and fix the time scale. Table 4 provides a comparison of both profiles at the two profile levels.

[0069] [Table 4]

[0070] Considering that Table 2 represents the hierarchical structure of the Main Profile Level 2, the representation store 63 represents the final effects across the eight initial concepts and 21 intermediate concepts.

[0071] This representation can be reduced using horizontal pruning (i.e., concept pruning) to the first four initial concepts at Simplified Parametric Profile Level 2, as shown in Table 5. This pruned representation includes four initial concepts, nine intermediate concepts, and 31 effect concepts. The four initial concepts retained in Tables 2 through 5 are identified as the most relevant concepts for the devices targeted by the Simplified Parametric Profile.

[0072] [Table 5]

[0073] Similarly, the hierarchy may be further pruned vertically to retain higher layer concepts, for example those used in Primary Profile Level 1. Table 6 shows an example hierarchy for Primary Profile Level 1.

[0074] [Table 6]

[0075] Similarly, the hierarchical structure can be further simplified vertically and horizontally, for example, as used in the case of Simple Parametric Profile Level 1. Table 7 shows an example of the hierarchical structure of Simple Parametric Profile Level 1. Only 18 haptic effects are represented, distributed across four initial concepts.

[0076] [Table 7]

[0077] Other pruning strategies can also be applied. One strategy is based on the type of reference device used to design the haptic experience, and therefore focuses only on branches for this specific type of device. This pruning mechanism can be used to allocate more bits to deeper layers, or to remove excess bits from the representation. For example, an experience designed for smartphones might embed only the branch UX structure shown in the table below.

[0078] [Table 8]

[0079] Returning to FIG. 7 , in step S704, the decomposed semantic information is encoded. The decomposed semantic information may be encoded as a string (e.g., in exchange file 204) or encoded using a fixed-length or variable-length binary representation. The binary representation of the decomposed semantic information may be inserted into delivery stream 205. The encoding of semantic information associated with a haptic effect does not replace the encoding of keyframes. As an example, keyframes and semantic information can be encoded to describe a haptic effect. The synthesizer may need to determine which information to use for rendering.

[0080] For simplicity, the implementation details below use the representation in Table 2.

[0081] In the first example, the decomposed semantic information is encoded using paths that allow JSON to maintain the readability of the fields as intended by the human reader. This solution also facilitates pruning strategies. Additional fields are added at the effect level. The proposed changes to the specification (ISO / IEC 23090-31: Haptics Coding, Committee Draft, section 6.2.8) are highlighted below with underlines.

[0082] { "$schema":"http: / / json-schema.org / draft-04 / schema", "title":"MPEG_haptics_effect", "type":"object", "properties":{ "id":{ "type":"integer", "description":"Track Channel id", "minimum":0 }, "effect_type":{ "type":"string", "enum":[ "Basis", "Composite", "Reference" ], "description":"Type of effect: basis, reference or composite" }, " "semantic": { " "type": "string", " "description": "Tag providing semantic information about the effect. The Semantic information follows a hierarchical construct with each layer of the semantic separated by a / " }, "position":{ "type":"integer", "description":"Temporal or spatial position of the effect", "minimum":0 }, "phase":{ "type":"number", "description":"Phase of the effect", "minimum":0.0, "maximum":6.28318 }, "keyframes":{ "type":"array", "description":"List of keyframes", "items":{ "type":"object", "$ref":"MPEG_haptics.keyframes.schema.json" } }, "base_signal":{ "type":"string", "enum":[ "Sine", "Square", "Triangle", "SawToothUp", "SawToothDown" ], "description":"Type of the base signal" }, "composition":{ "type":"array", "items":{ "effect":{ "type":"object", "$ref":"MPEG_haptics.effect.schema.json" } } } }, "required":[ "effect_type", "position", "phase", "keyframes" ] }

[0083] An example of encoding decomposed semantic information is given below, where the semantic information of the haptic effect “Rain” is hierarchically represented as “Ambient / Water / Rain” according to the hierarchical structure of Table 2.

[0084] { "effect_type":"Basis", "position":0, "phase":0.0, "base_signal":"Square", " "semantic": "Ambient / Water / Rain", "keyframes":[ { "relative_position":0, "amplitude_modulation":-0.003921568393707275, "frequency_modulation":8 }, { "relative_position":5, "amplitude_modulation":1.0 }, { "relative_position":30, "amplitude_modulation":1.0 }, { "relative_position":60, "amplitude_modulation":-0.003921568393707275, "frequency_modulation":8 } ] }

[0085] Below is an example of the encoding of decomposed semantic information for the haptic effects "Click," "Collision," and "Doors."

[0086] The semantic information of the haptic effect “Click” is expressed hierarchically as “UX (UX) / Button (Button) / Click (Click)” according to the hierarchical structure in Table 2.

[0087] { "effect_type":"Basis", "position":0, "phase":0.0, "base_signal":"Square", "semantic":"UX / Button / Click", "keyframes":[ ] }

[0088] The semantic information of the haptic effect "Collision" is hierarchically represented as "Character Movements / Interaction / Collision" according to the hierarchical structure in Table 2.

[0089] { "effect_type":"Basis", "position":0, "phase":0.0, "base_signal":"Square", "semantic":"Character Movements / Interaction / Collision", "keyframes":[ ] }

[0090] The semantic information of the haptic effect "Collision" is hierarchically represented as "Vehicles / Motorized / Doors" according to the hierarchical structure in Table 2.

[0091] { "effect_type":"Basis", "position":0, "phase":0.0, "base_signal":"Square", "semantic":"Vehicles / Motorized / Doors", "keyframes":[ ] }

[0092] In the second example, the decomposed semantic information is encoded in a human-readable format, JSON, using a graph-like structure: for each effect, the field semantic is represented by a node.

[0093] Additional fields are added at the effect level. Proposed changes to the specification are underlined below.

[0094] { "$schema":"http: / / json-schema.org / draft-04 / schema", "title":"MPEG_haptics_effect", "type":"object", "properties":{ "id":{ "type":"integer", "description":"Track Channel id", "minimum":0 }, "effect_type":{ "type":"string", "enum":[ "Basis", "Composite", "Reference" ], "description":"Type of effect: basis, reference or composite" }, " "semantic": { " "type": "object", " "$ref": "MPEG_haptics.semanticTag.schema.json", " "description": "Tag providing semantic information about the effect. The Semantic information follows hierarchical construct with each layer of the semantic defined by a number, a name and a potential child." }, "position":{ "type":"integer", "description":"Temporal or spatial position of the effect", "minimum":0 }, "phase":{ "type":"number", "description":"Phase of the effect", "minimum":0.0, "maximum":6.28318 }, "keyframes":{ "type":"array", "description":"List of keyframes", "items":{ "type":"object", "$ref":"MPEG_haptics.keyframes.schema.json" } }, "base_signal":{ "type":"string", "enum":[ "Sine", "Square", "Triangle", "SawToothUp", "SawToothDown" ], "description":"Type of the base signal" }, "composition":{ "type":"array", "items":{ "effect":{ "type":"object", "$ref":"MPEG_haptics.effect.schema.json" } } } }, "required":[ "effect_type", "position", "phase", "keyframes" ] }

[0095] The table below details the JSON schema of the proposed semantic tag object (following a graph-like structure).

[0096] { " "$schema": "http: / / json-schema.org / draft-04 / schema", " "title": "MPEG_haptics_semanticTag", " "type ": "object", " "properties": { " "layer": { " "type": "integer", " "description": "Track Channel id", " "minimum": 0 }, " "name": { ""type": "string", "enum": " "Basis", " "Composite", " "Reference" ], " "description": "Type of effect: basis, reference or composite" }, " "child": { " "type": "object", " "$ref": "MPEG_haptics.effect.semanticTag.schema.json" } }, " "required": " "layer", " "name" ] }

[0097] Finally, the following table shows an example of encoding the decomposed semantic information for the haptic effect "Rain."

[0098] { "effect_type":"Basis", "position":0, "phase":0.0, "base_signal":"Square", " "semantic": { " "layer": 1, " "name": "Ambient", " "child": { " "layer": 2, " name":"Water", " child":{ " layer":3, " name":"Rain" } } }, "keyframes":[ { "relative_position":0, "amplitude_modulation":-0.003921568393707275, "frequency_modulation":8 }, { "relative_position":5, "amplitude_modulation":1.0 }, { "relative_position":30, "amplitude_modulation":1.0 }, { "relative_position":60, "amplitude_modulation":-0.003921568393707275, "frequency_modulation":8 } ] }

[0099] Below is another example encoding decomposed semantic information for the haptic effects "Click," "Collision," and "Doors."

[0100] The semantic information of the haptic effect "Click" is hierarchically represented as follows, according to the hierarchical structure of Table 2.

[0101] { "effect_type":"Basis", "position":0, "phase":0.0, "base_signal":"Square", " semantic":{ " layer":1, " name":"UX", " child":{ " layer":2, " name":"Button", " child":{ " layer":3, " name":"Click" } }, "keyframes":[ ] } }

[0102] The semantic information of the haptic effect “Collision” is expressed hierarchically as follows, according to the hierarchical structure of Table 2.

[0103] { "effect_type":"Basis", "position":0, "phase":0.0, "base_signal":"Square", " semantic":{ " layer":1, " name":"Movement", " child":{ " layer":2, " name":"Interaction", " child":{ " layer":3, " name":"Collision" } }, "keyframes":[ ] } }

[0104] The semantic information of the haptic effect "Doors" is hierarchically represented as follows, according to the hierarchical structure of Table 2.

[0105] { "effect_type":"Basis", "position":0, "phase":0.0, "base_signal":"Square", " semantic":{ " layer":1, " name":" Vehicles ", " child":{ " layer":2, " name":" Motorized ", " child":{ " layer":3, " name":"Doors" } }, "keyframes":[ ] } }

[0106] In other examples disclosed below, each layer of the hierarchical structure is represented using a defined number of bits. A multi-layer hierarchical structure can be represented using a fixed-size binary representation or a variable-size binary representation. For example, Table 9 shows an example of a fixed-size binary representation with an arbitrary number of layers, N. The number of bits allocated to each layer also provides room for adding new effects in the future without changing the structure itself.

[0107] [Table 9]

[0108] The overall structure is Σ i=0 N Xi bits, and this construction allows for a maximum of 2^(Σ i=0 N X i ) haptic effects can be represented. Multi-layer hierarchical structures can be easily stored using a limited number of binary representations to solve the bit rate problem that arises when using character strings and provide more flexibility than simple enumerations. Therefore, the structure in Figure 9 can be represented using 6 bits. The first layer can be represented using a minimum of 3 bits because six high-level concepts are identified. The second layer can be represented using 1 bit because it contains a maximum of 2 elements for each major concept. A similar trend is observed for the third and fourth layers, so they can both be represented using 1 bit. Table 10 shows the binary representation of the hierarchical structure in Figure 9.

[0109] [Table 10]

[0110] Table 11 shows an example of a 3-bit binary representation of each concept shown in Figure 9. The same principle applies to the structure defined in Table 2.

[0111] [Table 11]

[0112] Table 12 shows an example of a binary representation on one bit for each concept shown in Figure 9 when the first layer concept is "environment." Similar principles apply to other concepts in the first layer.

[0113] [Table 12]

[0114] Thus, if the semantic information of the haptic effect "Heat" is decomposed into "Environment / Fire / Heat," it can be represented by the binary number 001 1 1 0. The first three bits "001" identify the concept "Environment," the next bit "1" identifies "Fire," and the third bit "1" identifies "Heat." The last bit is set to "0" because no semantic information is specified at the fourth level. The decomposed semantic information is added to the delivery stream 205, for example, in formatting step S220.

[0115] The structure in Table 2 can be represented in 8 bits as shown in Table 13 below. In fact, the first layer can be represented in a minimum of 3 bits since eight superordinate concepts are identified. The second layer can be represented in 2 bits since it contains up to 4 elements for each major concept. The third layer can be represented in 3 bits since it contains up to 8 elements (for vehicles / powered vehicles).

[0116] [Table 13]

[0117] In one example, the number of bits used to represent the semantics of an effect remains the same regardless of the profile or profile level used.

[0118] The semantic information of the haptic effect "Click" decomposed into "UX (UX) / Buttons / Click" can be represented by the binary number 000 00 000. Indeed, "000" encodes the concept "UX (UX)", "00" encodes "Buttons", and 000 encodes "(Click)".

[0119] The semantic information of the haptic effect "Collision," for example, can be encoded with the binary number 001 01 000. "001" encodes "Character Movements," i.e., the second concept in Table 2. "01" encodes "Interaction," and "000" encodes "Collision." "Jumping" is encoded with 001 00 001.

[0120] Table 14 shows an example of a binary representation of the semantic information associated with the "vehicle" concept. The binary representation is specified in brackets.

[0121] [Table 14]

[0122] The semantic information for the haptic effect "Doors," for example, can be encoded with the binary numbers 110 00 001. "110" encodes the "Vehicle" concept, "00" encodes powered (the first concept in layer 2), and "001" encodes "Doors" (the second haptic effect in layer 3). "Brake" is encoded as 110 00 010.

[0123] The use of such a fixed-size binary representation allows a hierarchical structure to be represented using the same number of bits. In this representation mode, the total number of bits used for the entire structure is fixed, but the number of bits for each layer can vary.

[0124] A fixed size binary representation not only allows for structural pruning, but also provides the advantage of allocating more bits to the remaining branches or layers.

[0125] This mode has the advantage of ensuring a stable bit rate for all binary-encoded haptic effects. Table 15 below defines an example of a bitstream syntax that can be used to binary-encode the semantic information of a haptic effect. It uses an optional semantic flag "hasSemantic" and a syntax element "effectSemantic" that encodes the decomposed semantic information of the haptic effect. The example in Table 15 is based on the hierarchical structure proposed in Table 2. Therefore, the syntax element "effectSemantic" is encoded using 8 bits, allocating 3 bits to the first layer, 2 bits to the second layer, and 3 bits to the third layer. A similar syntax can be used for a different hierarchical structure; simply adjust the number of allocated bits to fit the structure. For example, in the structure of Figure 9, effectSemantic could be encoded using 6 bits instead of 8 bits. Alternatively, a Boolean value (the semantic flag "hasSemantic") indicating whether the haptic effect has associated semantic information (also known as a tag) is also encoded.

[0126] [Table 15]

[0127] Another example is the use of a variable-depth binary representation, which allows the binary representation of the hierarchical structure to be reduced as needed, requiring either prior knowledge of the number of bits depending on the pruning mechanism used, or the addition of a field indicating the number of bits used.

[0128] The number of bits allocated to represent the structure may vary depending on the profile used, limiting the depth of the hierarchical structure that is encoded or limiting the ramifications of identifying the structure relevant to the haptic experience being conveyed. The advantage of using a binary representation with varying layer depth is that it provides flexibility and improved bitrate for binary-encoded haptic experiences while still conveying essential information.

[0129] Tables 16 and 17 below define an example of a bitstream syntax that can be used for a variable layer-depth binary representation of semantic information for a haptic effect. Here, before reading the semantic tag (also called the decomposed semantic information element), the tag layer is encoded. Based on the associated structure (Table 2 in this example), this syntax indicates the number of bits to read to obtain the semantic tag. Layer 0 corresponds to no semantic information. In this scheme, semantic tags can be defined at any layer of the structure, and there is no need to provide information for lower layers.

[0130] [Table 16]

[0131] [Table 17]

[0132] The syntax element "layer" indicates the layer depth of the semantic information (aka tags), i.e., the number of layers to read. If it is 0, the effect is to include no semantic information. The syntax element "semanticLayerN" indicates the semantic tag at the Nth layer in the semantic hierarchy.

[0133] For example, if layer=2, 5 bits are encoded (and read at the decoder side), with 3 bits assigned to the first layer and 2 bits assigned to the second layer.

[0134] The binary representation of variable layer depth is useful when a vertical pruning mechanism is used, which can save bitrate. In the representation of the Simple Parametric Profile Level 2 described in Table 5, the binary representation uses 6 bits as described in Table 18. This representation allows saving 2 bits per haptic effect in the haptic experience.

[0135] [Table 18]

[0136] The semantic information "UX(UX)" can be encoded with the binary number 01 00. "01" indicates a layer depth of 1, and the next two bits represent the tag UX.

[0137] The semantic information for the haptic effect "Click" can be encoded, for example, with the binary number 11 00 0 000. Since "11" indicates a layer depth of 3, 6 bits are encoded (or read): 2 bits for the first layer ("00" indicates UX (UX)), 1 bit for the second layer ("0" indicates "Button"), and 3 bits for the third layer ("000" indicates "Click").

[0138] Semantic information for the haptic effect "Collision," for example, can be encoded with the binary number 11 01 1 000. Since "11" indicates a layer depth of 3, 6 bits are encoded (or read): 2 bits for the first layer ("01" indicates "Character movements"), 1 bit for the second layer ("1" indicates "Interaction"), and 3 bits for the third layer ("000" indicates "Collision").

[0139] For the "Doors" representation, there is no semantic information in this pruned representation. In such cases, the haptic effect can only be defined using the keyframe description.

[0140] In another example, a variable-length binary representation is used. In this example, a completely flexible hierarchical representation is defined. In this example, the number of bits used for a particular layer is indicated in the binary stream. This allows for only the minimum number of bits to be used for a particular layer, allowing for adaptation to any hierarchical data structure. For example, regardless of the profile used, a decoder can decode the data without prior knowledge of the profile, and information is needed to interpret the decoded data, but not to decode the data as in the previous embodiment. Furthermore, even if the standard updates the proposed hierarchical semantic structure, the decoder will still be able to read the data.

[0141] In this example, the number of bits used for each node in the hierarchy can be different. This solution can also be used to store the first element of a layer using fewer bits than the last element. For example, in Table 2, the semantic information (aka tag) UX / Notification / Confirmation can be encoded using only one bit in layer 3, while the "ring" or "message" tag requires two bits.

[0142] In this embodiment, for each haptic effect, the layer depth (ie, the number of layers to read) and the number of bits to read are indicated in the bitstream, as detailed in the table below.

[0143] [Table 19]

[0144] [Table 20]

[0145] The syntax element "layer" indicates the layer depth of the semantic information. If the value is 0, the effect contains no semantic information.

[0146] [Table 21]

[0147] The syntax element "layerBitSize" indicates the number of bits used to store the semantic information of a layer.

[0148] The syntax element "semanticLayer" indicates the semantic tag of the layer, which corresponds to the information defined in the semantic hierarchy.

[0149] Similar to the previous embodiment, this solution also includes information on the number of bits to read at each layer. Based on the Simple Parametric Profile Level 2 representation mentioned in Table 5, an example of semantic information / tag representation is shown below.

[0150] The semantic information "UX(UX)" can be encoded with the binary numbers 01 01 0.

[0151] [Table 22]

[0152] The semantic information "Click" can be encoded with the binary number 11 01 0 01 0 01 0.

[0153] [Table 23]

[0154] The semantic information "Touching" can be encoded with the binary number 11 01 1 01 1 10 10.

[0155] [Table 24]

[0156] The semantic information "Touching" can be encoded with the binary number 11 10 11 01 0 011 1.

[0157] [Table 25]

[0158] FIG. 8 shows a flowchart of a decoding method according to an example.

[0159] In step S800, a stream (either a delivery stream 205 or an exchange file 204) containing semantic information hierarchically encoded into layers according to the method of FIG. 7 is obtained.

[0160] In step S802, the semantic information (also known as tag) is decoded. As an example, the sequence of bits 001 1 1 0 is decoded into the semantic information "Environment / Fire / Heat" if the semantic information is encoded according to the examples disclosed with respect to Tables 11 and 12.

[0161] In one variant, the string is obtained when the semantic information is encoded as a string in a human-readable file.

[0162] In step S804, this semantic information is used to render the haptic effect. The decoded semantic information can be provided in addition to a keyframe representation of the effect. Thus, on the rendering side, the generator can use either the semantic information or the keyframe information to render the haptic effect. In other words, the synthesizer decides which information to use for rendering. As an example, the synthesizer can decide to use the decoded semantic information. This representation enables the construction of a standard haptic effect library for each actuator, facilitating content creation that is independent of the haptic device that renders the experience. This representation also allows the synthesizer to signal specific haptic effects that can be interpreted depending on the type of haptic device used.

[0163] For example, when decoding "Environment / Fire / Heat," the synthesizer may access a standard haptic effects library and render the "Heat" haptic effect from the library instead of rendering the haptic effect represented by the keyframe in the stream, thus ensuring that the rendered effect is tailored to the capabilities of the rendering device.

[0164] Furthermore, this type of hierarchical structure allows for easy data reduction by removing unnecessary data. This can be achieved by removing all haptic signal effects that match a specific semantic, or conversely, by retaining only those effects related to a specific semantic. This can typically be used to optimize haptic data streaming and transmit only the necessary information. This type of request can typically be based on the constraints of available devices or simply on user preferences. For example, if a complete haptic experience containing numerous haptic effects with different semantics is stored on a server, a client can request to retrieve only data related to the UX / Notification semantic information. The proposed solution selects appropriate data by selecting only effects with the UX / Notification semantic tag and streaming only the necessary information. This type of data filtering can be performed at different semantic levels. It can be used to select or remove data based on high-level semantic information (the first layer of the structure) or to precisely select data with fine-grained semantic data (lower semantic layers).

[0165] Various numerical values ​​are used in this application. The specific numerical values ​​are for illustrative purposes only and the described embodiments are not limited to these particular values.

[0166] In one embodiment, the encoding principles of the first or second method are applied to encode an audio signal. Such an audio signal can represent any type of audio communication, such as a background soundtrack, sound effects (e.g., explosions), or audio communication between two users. The audio signal can be part of an immersive scene or can be independent of any immersive scene, using the format described in FIG. 5 . Furthermore, the audio signal undergoes a low-pass filtering stage before being used to render a haptic signal. This encoding technique is particularly useful for low frequencies, such as subwoofer audio signals. All encoding principles are similar to those described above in the context of low-level haptic signals, but apply to more general audio signals or signals (e.g., stereo, 5.1 multi-channel audio, etc.). In fact, low-level haptic signals are very similar to audio signals and share the same characteristics. Therefore, such an embodiment is applicable to any audio distribution system, and the resulting encoded data can be stored on removable media (e.g., memory cards, USB flash drives, hard disk drives, solid-state drives, optical media, etc.) or transmitted via a communication network.

[0167] If multiple frequency bands are encoded using key frames, the principles described in the first or second embodiment are used for each frequency band encoded using key frames, and the resulting residual signals can be encoded individually as different frequency bands or jointly into a single frequency band.

[0168] Although the embodiments have been described primarily using decomposition into two frequency bands, the first embodiment The principles of the second embodiment are easily applied to applications that use more than one frequency band for decomposition.

[0169] Several embodiments have been described above, and the features of these embodiments may be provided alone or in any combination across various claim categories and types.

[0170] In one example, a method includes obtaining semantic information associated with at least one haptic effect, decomposing the semantic information into different levels of semantic information based on a multi-layer hierarchical structure of the semantic information, and encoding the decomposed semantic information.

[0171] In one example, encoding the decomposed semantic information includes encoding the decomposed semantic information as a sequence of consecutive strings, the sequence including one string for each level of semantic information.

[0172] In one example, encoding the decomposed semantic information includes encoding the decomposed semantic information as a sequence of contiguous strings organized as a graph-like structure.

[0173] In one example, encoding the decomposed semantic information includes encoding each level of semantic information using a fixed number of bits.

[0174] In one example, encoding the decomposed semantic information includes encoding a first syntax element indicating a layer depth of the semantic information, and further encoding a second syntax element representing the decomposed semantic information in a number of bits depending on the depth layer.

[0175] In one example, encoding the decomposed semantic information includes, for each layer, encoding a first syntax element indicating a layer depth of the semantic information, encoding a second syntax element indicating a number of bits used to encode the decomposed semantic information for that layer, and encoding a third syntax element representing the decomposed semantic information for that layer on the indicated number of bits.

[0176] In one example, the multi-layer hierarchy of semantic information depends on the encoding profile.

[0177] An apparatus including one or more processors and at least one memory connected to the one or more processors, wherein the one or more processors are configured to acquire semantic information associated with at least one haptic effect, decompose the semantic information into different levels of semantic information based on a multi-layer hierarchical structure of the semantic information, and encode the decomposed semantic information.

[0178] A method for rendering a haptic effect includes obtaining a stream having semantic information hierarchically encoded into layers, the semantic information being associated with the haptic effect, decoding the semantic information, and rendering the haptic effect based on the decoded semantic information.

[0179] A rendering device including one or more processors and at least one memory connected to the one or more processors, wherein the one or more processors are configured to obtain a stream including semantic information hierarchically encoded into layers, the semantic information being associated with the haptic effect, decode the semantic information, and render the haptic effect based on the decoded semantic information.

[0180] Further disclosed is a computer-readable storage medium having stored thereon instructions for implementing any of the above-disclosed methods.

[0181] Although the embodiments relate to haptic effects, those skilled in the art will appreciate that the same principles are applicable to other effects, including sensory effects, such as smell, taste, temperature, emotion, intensity enhancement, etc. Accordingly, appropriate syntax may determine appropriate parameters associated with these effects.

[0182] References to "one embodiment," "an embodiment," "one example," "an example," and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of "one embodiment," "in an embodiment," "in one implementation," or "in an implementation," as well as other variations thereof, in various places throughout this specification do not necessarily all refer to the same embodiment.

[0183] Additionally, this application or its claims may refer to "determining" various pieces of information. Determining information may include one or more operations, such as, for example, estimating information, calculating information, predicting information, or retrieving information from memory.

[0184] Additionally, this application or its claims may refer to "obtaining" various pieces of information. "Obtaining," like "accessing," is intended as a broad term. Obtaining information may include, for example, receiving information, accessing information, or retrieving information (e.g., from memory or optical media storage). Furthermore, actions such as storing, processing, transmitting, moving, copying, erasing, calculating, determining, predicting, or estimating information typically involve some form of "obtaining."

[0185] The use of phrases such as " / ," "and / or," and "at least one" (e.g., "A / B," "A and / or B," "at least one of A and B") includes selecting only the first listed option (A), selecting only the last listed option (B), or selecting both the first listed option (A) and the last listed option (B). By way of further example, phrases such as "A, B, and / or C" and "at least one of A, B, and C" are intended to encompass selecting only the first listed option (A), selecting only the second listed option (B), selecting only the third listed option (C), selecting only the first and second listed options (A and B), selecting only the first and third listed options (A and C), selecting only the second and third listed options (B and C), or selecting all three options (A, B, and C). This can be expanded depending on the number of items listed, as would be readily apparent to one of ordinary skill in this and related arts.

Claims

1. 1. A method comprising: Obtaining semantic information associated with at least one haptic effect (S700); Decomposing the semantic information into different levels of semantic information based on a multi-layer hierarchical structure of the semantic information (S702); encoding the decomposed semantic information (S704); A method comprising:

2. 2. The method of claim 1 , wherein encoding the decomposed semantic information comprises encoding the decomposed semantic information as a sequence of consecutive strings, the sequence comprising one string for each level of semantic information.

3. 10. The method of claim 1, wherein encoding the decomposed semantic information comprises encoding the decomposed semantic information as a sequence of contiguous strings organized as a graph-like structure.

4. The method of claim 1 , wherein encoding the decomposed semantic information comprises encoding each level of semantic information using a fixed number of bits.

5. 2. The method of claim 1, wherein encoding the decomposed semantic information comprises: encoding a first syntax element indicating a layer depth of the semantic information; and encoding a second syntax element representing the decomposed semantic information with a number of bits depending on the depth layer.

6. 2. The method of claim 1, wherein encoding the decomposed semantic information comprises: for each layer, encoding a first syntax element indicating a layer depth of the semantic information; encoding a second syntax element for the layer indicating a number of bits used to encode the decomposed semantic information; and encoding a third syntax element representing the decomposed semantic information for the layer on the indicated number of bits.

7. The method according to claim 1 , wherein the multi-layer hierarchical structure of semantic information depends on the encoding profile.

8. 1. An apparatus comprising one or more processors and at least one memory coupled to said one or more processors, said one or more processors comprising: obtaining semantic information associated with at least one haptic effect; Decomposing the semantic information into different levels of semantic information based on a multi-layer hierarchical structure of the semantic information; encoding the decomposed semantic information; An apparatus configured as follows.

9. 1. A method for rendering a haptic effect, comprising: Obtaining a stream with semantic information hierarchically encoded into layers, the semantic information being associated with the haptic effect (S800); Decoding the semantic information (S802); Rendering the haptic effect based on the decoded semantic information (S804); and A method comprising:

10. 1. A rendering device comprising one or more processors and at least one memory coupled to said one or more processors, said one or more processors comprising: obtaining a stream with semantic information hierarchically encoded into layers, the semantic information being associated with the haptic effect; Decoding the semantic information; rendering the haptic effect based on the decoded semantic information.

1. A rendering device configured to:

11. A computer-readable storage medium having stored thereon instructions for implementing the method of any one of claims 1 to 7.

12. 10. A computer-readable storage medium having stored thereon instructions for implementing the method of claim 9.

Images