Timeline-based representation of haptic signals

The proposed haptic data structure addresses inefficiencies in haptic signal encoding by using a timeline-based approach with library references, improving file compression and flexibility for immersive environments.

JP2024544053APending Publication Date: 2024-11-27INTERDIGITALCE PATENT HLDG SAS
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2024532153
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2022-03-15
Filing Date
2022-11-08
Publication Date
2024-11-27

AI Technical Summary

Technical Problem

Existing haptic technologies lack flexibility and efficiency in encoding and rendering haptic signals, leading to increased data size and repetitive data storage, which is unsuitable for both streaming and editing purposes.

Method used

A data structure that allows haptic effects to be defined within a timeline and referenced from a library, enabling flexible encoding and reducing redundant data storage by using base, timeline, and reference effects, optimizing file size for both streaming and editing.

Benefits of technology

The proposed data structure enhances haptic file compression and flexibility, allowing efficient use of haptic effects in immersive environments by minimizing redundant data and optimizing storage space.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024544053000001_ABST
    Figure 2024544053000001_ABST
Patent Text Reader

Abstract

A data structure that stores information representing haptic effects includes a set of haptic effects and a timeline. Haptic effects can be defined in a flexible manner, either directly in the timeline or referenced by an identifier in the timeline. A library can store effect definitions with associated identifiers. A data restructuring process is proposed to convert from a format suitable for streaming without the library to a format suitable for editing with the library and vice versa.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] At least one of the embodiments relates generally to haptics, and more specifically to defining a timeline-based representation format for haptic objects in an immersive scene. [Background technology]

[0002] Through immersive systems based on feedback and interaction, a fully immersive user experience is proposed to the user. The interaction may use traditional control methods that meet the needs of the user. Current visual and auditory feedback provides a satisfying and realistic sense of immersion. Additional feedback may be provided by haptic effects that allow the human user to perceive the virtual environment with their senses and thus obtain a better experience of full immersion with improved realism. However, haptics is one area where there is still room for progress to improve the overall user experience in immersive systems.

[0003] Typically, an immersive system may include a 3D scene representing a virtual environment, in which virtual objects are located. To enhance user interaction with elements of the virtual environment, haptic feedback by simulation of haptic actuators can be used. Such interaction is based on the concept of "haptic objects" that correspond to physical phenomena communicated to the user. In the context of an immersive scene, haptic objects make it possible to provide haptic effects by defining appropriate haptic actuator stimuli to mimic physical phenomena on a haptic rendering device. Different types of haptic actuators make it possible to restore different types of haptic feedback.

[0004] An example of a haptic object is an explosion, which can be rendered via vibration and heat, thus combining different haptic effects for the user to enhance realism. An immersive scene typically includes multiple haptic objects, for example using a first haptic object associated with a global effect and a second haptic object associated with a local effect.

[0005] The principles described herein apply to any immersive environment that uses haptics, such as augmented reality, virtual reality, mixed reality, or haptic-enhanced video (or omnidirectional / 360° video) rendering, and more generally to any haptic-based user experience. Thus, such example scenes of immersive environments are considered immersive scenes.

[0006] Haptics refers to the sense of touch and includes two dimensions: tactile and kinesthetic. The first dimension is related to tactile sensations such as friction, roughness, hardness, temperature, etc., and is felt through mechanoreceptors in the skin (Merkel cells, Ruffini endings, Meissner's corpuscles, Pacinian corpuscles). The second dimension is related to the sense of force / torque, position, and motion / velocity provided by muscles, tendons, and mechanoreceptors in the joints. Haptics also contributes to the proprioceptive system (i.e., the perception of one's own body), and thus participates in the perception of self-motion. Thus, the perception of acceleration, velocity, or any other body model can be fused as a haptic effect. The frequency range varies depending on the type of modality, but is around 0-1 KHz. Most existing devices capable of rendering haptic signals generate vibrations. Examples of such haptic actuators include linear resonant actuators (LRA), eccentric rotating masses (ERM), and voice coil linear motors. These actuators may be integrated into a haptic rendering device such as a haptic suit, but also into a smartphone or a game controller.

[0007] Several formats have been defined to encode haptic signals, either related to high-level descriptions using XML-like formats (e.g., MPEG-V), parametric representations using json-like formats, such as Apple Haptic Audio Pattern (AHAP) or Immersion Corporation's HAPT format, or waveform encoding (ongoing standardization of IEEE 1918.1.1 for haptic and kinesthetic signals). The HAPT format has recently been included in the MPEG ISOBMFF file format specification (ISO / IEC 14496 part 12). Furthermore, the GL Transmission Format (glTF™) is a royalty-free specification for the efficient transmission and loading of 3D scenes and models by applications. This format defines an extensible, common public format for 3D content tools and services that streamlines authoring workflows and enables interoperable use of content across the industry.

[0008] Furthermore, a new haptic file format is being defined within the MPEG standardization group for the coded representation of haptics. A reference model for this format has not yet been released, but is referred to herein as RM0. This reference model allows the encoded haptic description file to be exported as a human-readable JSON interchange format (e.g., a .gmpg file) or as a compressed binary delivery format (e.g., .mpg) specifically adapted for transmission to haptic rendering devices. Summary of the Invention

[0009] The embodiments relate to a data structure that stores information representing an immersive experience, including a set of haptic effects and a timeline. Haptic effects can be defined in a flexible manner, either directly in the timeline or referenced by an identifier in the timeline. A library can store effect definitions with associated identifiers. A data restructuring process is proposed that converts from a format suitable for streaming without a library to a format suitable for editing with a library and vice versa.

[0010] A first aspect of at least one embodiment is directed to a method including generating haptic data including information representing a type of effect, a list of haptic effects, and a list of haptic trajectories, the haptic trajectory including a timeline with at least one temporal reference associated with the haptic effect or an identifier in the list of haptic effects, the haptic effect including information representing a value of a signal to be applied to render the haptic effect.

[0011] A second aspect of at least one embodiment is directed to a method for rendering haptic data, the method including: acquiring haptic data including information representing a type of effect, a list of haptic effects, and a list of haptic trajectories, the haptic trajectories including a timeline with at least one temporal reference associated with the haptic effect or an identifier in the list of haptic effects, the haptic effects including information representing values ​​of signals to be applied to render the haptic effect; and providing the signal values ​​to a haptic actuator.

[0012] A third aspect of at least one embodiment is directed to a haptic rendering device including a processor, the processor configured to obtain haptic data including information representing a type of effect, a list of haptic effects, and a list of haptic trajectories, the haptic trajectories including a timeline where at least one temporal reference is associated with the haptic effect or an identifier in the list of haptic effects, and the haptic effects including information representing values ​​of signals to be applied to render the haptic effect, and to provide the signal values ​​to a haptic actuator.

[0013] A fourth aspect of at least one embodiment is directed to haptic data including information representing a type of effect, a list of haptic effects, and a list of haptic trajectories, the haptic trajectories including a timeline where at least one temporal reference is associated with the haptic effect or an identifier in the list of haptic effects, and the haptic effects include information representing values ​​of signals to be applied to render the haptic effect.

[0014] A fifth aspect of at least one embodiment is directed to a method for reconstructing haptic data generated by the first aspect, where at least one temporal reference is associated with an identifier in a list of haptic effects, the method including obtaining haptic data, analyzing the haptic data to determine an identifier of a haptic effect in a timeline, replacing the determined identifier with a copy of the haptic effect associated with the identifier, emptying the list of haptic effects, and providing reconstructed haptic data.

[0015] A sixth aspect of at least one embodiment is directed to a method for reconstructing haptic data generated according to claim 1, where at least one temporal reference is associated with a haptic effect, the method including acquiring haptic data, analyzing the haptic data to determine an identical haptic effect, inserting a copy of one of the determined identical haptic effects into a list of haptic effects, associating an identifier with the inserted identical haptic effect, replacing the identical haptic effect with its associated identifier, and providing reconstructed haptic data.

[0016] According to a seventh aspect of at least one embodiment there is provided a computer program comprising program code instructions executable by a processor to perform at least the steps of a method according to the first or second aspects.

[0017] According to an eighth aspect of at least one embodiment, there is provided a computer program product stored on a non-transitory computer readable medium and comprising program code instructions executable by a processor, the computer program product performing at least the steps of the method according to the first or second aspects. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a block diagram of an example system in which various aspects and embodiments may be implemented. [Figure 2A] FIG. 2A shows an example of a prior art coding of a haptic signal using decomposition into frequency bands. [Figure 2B] FIG. 2B shows an example of a prior art coding of a haptic signal using decomposition into frequency bands. [Diagram 3] FIG. 3 illustrates an example data structure of a haptic file according to at least a first embodiment, where the base effect is defined based on a frequency decomposition in the haptic band. [Figure 4]FIG. 4 illustrates an example data structure of a haptic file according to at least a second embodiment, where the base effects are defined based on key frames. [Diagram 5] FIG. 5 is a diagram illustrating an example of a haptic file encoded without using an effects library according to the second embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a haptic file encoded using an effects library according to the second embodiment. [Figure 7] FIG. 7 illustrates an example of an architecture for a haptic file encoder, according to at least one embodiment. [Figure 8] FIG. 8 is a diagram illustrating a data reconstruction process according to one embodiment. [Figure 9] FIG. 9 illustrates an example of a process for rendering haptic data in accordance with at least one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] FIG. 1 shows a block diagram of an example of a system in which various aspects and embodiments are implemented. In the illustrated immersive system, a user Alice uses a haptic rendering device 100 to interact through a communication network 170 with a server 180 hosting an immersive scene 190. This immersive scene 190 may contain various data and / or files representing the different elements required for its rendering (scene description 191, audio data, video data, 3D models, and haptic objects 192). The immersive scene 190 may be generated under the control of an immersive experience editor 110 that allows the different elements to be placed together to design an immersive experience. An appropriate description file and various data files representing the immersive experience are generated by an immersive scene generator 111 in a format adapted for transmission to the haptic rendering device. The immersive experience editor 110 typically runs on a computer that generates the immersive scene hosted on the server. For the sake of simplicity, the immersive experience editor 110 is shown as being directly connected to the immersive scene 190 via a dotted line 171. In practice, the computer running the immersive experience editor 110 is connected to a server 180 via a communications network 170 .

[0020] Haptic rendering device 100 comprises a processor 101. Processor 101 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor may perform data processing such as haptic signal decoding, input / output processing, and / or any other functionality that enables the device to operate in an immersive system.

[0021] The processor 101 may be coupled to an input unit 102 configured to convey user interactions. Multiple types of inputs and modalities can be used for that purpose. Physical keypads and touch-sensitive surfaces are typical examples of inputs adapted for this application, but voice control can also be used. In addition, the input unit may comprise a digital camera capable of capturing still images or videos. The processor 101 may be coupled to a display unit 103 configured to output visual data to be displayed on a screen. Multiple types of displays can be used for that purpose, such as liquid crystal display (LCD) or organic light-emitting diode (OLED) display units. The processor 101 may also be coupled to an audio unit 104 configured to render sound data that is converted into audio waves through adapted transducers, such as loudspeakers. The processor 101 may be coupled to a communication interface 105 configured to exchange data with external devices. The communication preferably uses wireless communication standards that provide mobility of the haptic rendering device, such as cellular (e.g., LTE) communication, Wi-Fi communication, etc. The processor 101 may access information from and store data in memory 106, which may comprise multiple types of memory including random access memory (RAM), read-only memory (ROM), a hard disk, a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, or any other type of memory storage device. In one embodiment, the processor 101 may access information from and store data in memory that is not physically located on the device, such as a server, a home computer, or another device.

[0022] The processor 101 may be coupled to a haptic unit 107 configured to provide haptic feedback to the user, the haptic feedback being described in haptic data 192 associated with a scene description 191 of an immersive scene 190. The haptic data 192 describes the type of feedback to be provided according to a syntax further described below. Such a description file is typically communicated from a server 180 to the haptic rendering device 100. The haptic unit 107 may comprise a single haptic actuator or multiple haptic actuators arranged at multiple locations on the haptic rendering device. Different haptic units may have different numbers of actuators and / or the actuators may be arranged differently on the haptic rendering device.

[0023] The processor 101 may receive power from a power source 108 and may be configured to distribute and / or control the power to other components in the device 100. The power source may be any suitable device for providing power to the device. By way of example, the power source may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0024] While this diagram illustrates the processor 101 and other elements 102-108 as separate components, it will be understood that these elements may be integrated together in an electronic package or chip. It will be understood that the haptic rendering device 100 may include any subcombination of elements described herein while remaining consistent with an embodiment. The processor 101 may be further coupled to other peripherals or units not shown in FIG. 1, which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals may include sensors such as a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, etc. For example, the processor 101 may be coupled to a localization unit configured to locate the haptic rendering device within its environment. The localization unit may integrate not only a GPS chipset that provides longitude and latitude position relative to the current location of the haptic rendering device, but also other motion sensors such as an accelerometer and / or electronic compass that provide localization services.

[0025] Typical examples of haptic rendering device 100 include haptic suits, smartphones, game controllers, haptic gloves, haptic chairs, haptic props, motion platforms, etc., although any device or configuration of devices providing similar functionality may be used as haptic rendering device 100 while conforming to the principles of the present disclosure.

[0026] In at least one embodiment, the device does not include a display unit, but does include a haptic unit. In such an embodiment, the device does not visually render the scene, but only renders haptic effects. However, the device may prepare data for display so that another device, such as a screen, can perform the display. Examples of such devices include a haptic suit or a motion platform.

[0027] In at least one embodiment, the device does not include a haptic unit, but does include a display unit. In such an embodiment, the device does not render haptic effects, but only visually renders the scene. However, the device may prepare data for rendering haptic effects so that another device, such as a haptic prop, can perform the haptic rendering. Examples of such devices include a smartphone, a head-mounted display, or a laptop.

[0028] In at least one embodiment, the device does not include a display unit or a haptic unit. In such an embodiment, the device does not visually render a scene and does not render haptic effects. However, the device may prepare data for display so that another device, such as a screen, can perform the display, and may prepare data for rendering haptic effects so that another device, such as a haptic prop, can perform the haptic rendering. Examples of such devices include a desktop computer, an optical media player, or a set-top box.

[0029] In at least one embodiment, the immersive scene 190 and associated elements are hosted directly within the memory 106 of the haptic rendering device 100, allowing for local rendering and interaction.

[0030] 1 as separate elements, the principles described herein also apply when these elements are integrated directly into the scene description and are not separate elements. Any mix between the two options is also possible, where some of the elements are integrated into the scene description and others are separate files.

[0031] 2A and 2B show an example of a prior art coding of a haptic signal using a decomposition into frequency bands. As shown in FIG. 2A, in this technique, the signal is coded using a list of trajectories 210 and the data is decomposed into a set of frequency bands 220. Each band defines the portion of the signal in a given frequency range using a list of streams. A stream 230 contains a number of unit signal keyframes 240 and governs their timing. A haptic signal in a trajectory can be reconstructed by combining data from streams in different bands. FIG. 2B shows a haptic signal 250 and a possible decomposition into two frequency bands. Band 260 is associated with a first frequency band and contains only one stream 261. Band 270 is associated with a second frequency band and contains streams 271, 272, 273, and 274. Each stream contains a set of keyframes that represent the values ​​of the signal to be rendered. A temporal reference, such as a timestamp, is associated with the keyframes. By aggregating keyframes from the high and low frequency band streams located at appropriate timestamps, the original low level signal can be reconstructed.

[0032] One advantage of the structure of this solution is that the signal data is easy to package and is particularly convenient for streaming purposes. In fact, with such a linear structure, the data can be easily broken down into small contiguous packages and does not require complex data prefetching operations. The signal can be easily reconstructed by piecing the packages back together, ensuring smooth playback of the signal.

[0033] However, this approach requires that multiple repeated parts of a signal must be stored multiple times. While this linear representation of data may be convenient for streaming purposes, it may be counterproductive for other purposes, as the resulting data size may increase. In addition, when creating a signal, the haptic designer must manually duplicate the same haptic effect multiple times, which is cumbersome, especially in the case of signal modifications, since the modifications also need to be duplicated or redone. Such an approach is clearly unsatisfactory for exchange formats used in the editing stage of haptic files.

[0034] In addition, the frequency band decomposition for a given trajectory is always the same. It is not possible to split the trajectory into multiple parts with different band decompositions. This limits the possibilities from a design point of view. For a given trajectory (usually associated with a single haptic actuator), it can be useful to define different effects using different frequency ranges; simple effects may only require a single frequency band, while other effects can be decomposed into multiple layers.

[0035] The embodiments described below are designed with the above in mind.

[0036] An embodiment introduces the concept of a haptic effect library, allowing haptic effects to be defined once and used multiple times. The following types of haptic effects are defined: - Base Effects: Base effects typically define the signal data used to control haptic actuators. - Timeline Effects: A timeline effect is defined as a sequence of effects, each associated with a timestamp. The effects themselves may be defined directly using signal data or may rely on references to haptic library effects. - Reference Effect: A reference to an existing effect (identified by an "id" attribute), associated with a timestamp.

[0037] In a first embodiment, basic effects are defined based on the frequency decomposition in haptic bands introduced above. A haptic trajectory is then composed of a set of timed haptic effects instead of a list of haptic bands. This solution allows defining haptic effects with different frequency bands within one trajectory. This allows more flexibility to compress signals using different frequency bands, which may improve the quality or compression level of the encoded signal. This also allows more flexibility in designing the signal to encode the haptic effect using the most reasonable band decomposition.

[0038] In a second embodiment, the base effects are defined based on key frames. This solution allows for maximizing spatial optimization by storing lower level information that can be more easily referenced. In this solution, the haptic effects defined for a given haptic band do not depend on the content described in another band.

[0039] Both embodiments allow optimizing the space consumption of haptic files: By defining haptic effects in an effects library and using the proposed referencing system, unnecessary data repetition can be avoided through data refactoring.

[0040] As shown in Figures 3 and 4, the proposed data structure allows haptic effects to be used both in the effects library and directly in each trajectory. In fact, the use of the data model proposed herein for effect definition allows for great flexibility, allowing effects to be defined directly in each trajectory or simply referenced from the library to existing effects. In addition, this model allows for beneficial data optimization for various purposes (e.g., streaming without the use of a library, or editing with intensive use of a library to minimize manual operations while modifying the haptic description file).

[0041] For example, given an existing haptic file that has multiple references to haptic effects in a library, the data can be reorganized by replacing the references with the actual effects. This linearization of the data simplifies packaging the data for streaming purposes. Conversely, if a haptic file was designed for streaming purposes, the effects library is empty. The size of the file can also be significantly reduced by parsing the data to identify and extract multiple identical effects and placing them in a library.

[0042] Such optimizations may be introduced in the "Binary Compression" step 730 according to FIG. 7, or in the file load / save stages of the haptic editor.

[0043] 3 shows an example of a data structure for haptic data, according to at least one embodiment. In this embodiment, the base effects are defined based on the frequency decomposition in the haptic bands introduced above. The haptic trajectory is then constructed from a set of timed haptic effects instead of a list of haptic bands.

[0044] A haptic description file (in other words, haptic data) 300 includes a first level 301, including a file description 302, a set of avatars 303, a set of signals 310, and a geometry 305. The file description 302 includes some generic metadata, including the file version, date, and a description of the associated haptic experience. The set of avatars 303 includes a definition of the body model to which the haptic effect is to be applied. The geometry 305 determines the volume within the immersive scene over which the haptic effect is active. The set of signals 310 aggregates signals, such as signal 311. Despite its name, a signal is a relatively high-level concept that links together haptic effects of similar categories (e.g., kinesthetic, temperature-based effects, etc.).

[0045] Signal 311 is expanded in the second row of the figure and includes metadata 321, a reference 322 to an avatar selected from set of avatars 303 (e.g., a reference to avatar 304), an effects library 330, and a set of haptic trajectories 340. Metadata 321 includes information about the type of haptic effect and the corresponding signal (vibration, pressure, temperature, etc.), as well as a description of the signal. The effects library includes a set of effects, such as effects 331 and 332. An example of effect 331 is expanded in the bottom left of the figure. Set of haptic trajectories 340 aggregates several haptic trajectories, such as 341, 342, and 343. Haptic trajectory 342 is also expanded in the middle right of the figure.

[0046] According to the first embodiment, the effect 331 is defined based on a frequency decomposition in haptic bands. At a first level, the effect comprises a set of bands 380, each band corresponding to a range of frequencies, e.g. band 381. At a second level, each band comprises a set of streams 385, each stream corresponding to a portion of the signal within a given time frame and within a given frequency range. At a third level, each stream comprises a set of keyframes (defined by frequency and amplitude) 350. For example, stream 386 comprises at least keyframes 351, 352 and 353. These keyframes carry signals that are applied to appropriate haptic actuators to finally render the haptic effect, similar to streams 271, 272, 273 and 274 in FIG. 2B. With such a definition, each effect defined in the library comprises a minimal definition of a signal, so that it may be reused multiple times within a haptic file.

[0047] Haptic trajectory 342 includes a trajectory description 361 carrying semantic information, trajectory attributes 362 carrying several attributes such as gain, mixing weights, body part masks, etc., and an effect timeline 370. Effect timeline 370 includes a set of referenced effects, e.g. effects 371, 372, 373, and 374, associated with a temporal reference such as a timestamp. From a temporal perspective, each effect may be associated with a timestamp. The decomposition of effect 371 at the bottom right of the figure shows that its structure is identical to effects defined in an effects library, such as effect 331.

[0048] Such a data structure allows the designer of a haptic file to choose for each effect whether it is better to add it to an effects library and then reference it from the effects timeline, or to insert it directly into the effects timeline. Effect timeline 370 shows both options, with effect 371 (which will likely only be used once) defined directly in the effects timeline, and effects 372, 373, and 374 referencing effects from the library, effects 331, 331, and 332 respectively. With this technique, effect 331, which is used multiple times, is defined only once, thus minimizing the bandwidth and storage space required to write the haptic file.

[0049] As mentioned above, haptic effects can be of three types: basic, timeline, or reference. A basic effect is defined by a list of haptic bands that describe a signal in the form of a stream of waves (or keyframes). A basic effect is identified by an id. An example of a basic effect is effect 331 in the effects library. A timeline effect is also identified by an id, but is defined in a timeline consisting of timed effects. An example of a timeline effect is effect 371 in effect timeline 370. Finally, a reference effect is simply a reference to an existing effect defined in the effects library. In this type of effect, the id attribute indicates the effect being referenced. An example of a reference effect is effect 373 in effect timeline.

[0050] An example JSON schema for a haptic effect according to the first embodiment is shown in Table 1.

[0051] [Table 1-1]

[0052] [Table 1-2]

[0053] The concept of haptic effects is introduced into the format. At the signal level, effects are stored in the "effect_library" as a list of effects. An example JSON schema for a haptic signal is shown in Table 2.

[0054] [Table 2-1]

[0055] [Table 2-2]

[0056] Effects defined at the signal level can then be referenced and used multiple times across multiple trajectories. Effects are organized within each haptic trajectory using a timeline as shown in the JSON schema in Table 3.

[0057] [Table 3-1]

[0058] [Table 3-2]

[0059] Table 4 shows an example of JSON syntax for a haptic file according to the first embodiment. In this example, the haptic file contains a single signal with a single trajectory. The effect library contains one basic effect that is referenced multiple times within the trajectory. The trajectory also directly defines another haptic effect. In this solution, one trajectory can contain haptic effects with different frequency bands.

[0060] [Table 4-1]

[0061] [Table 4-2]

[0062] [Table 4-3]

[0063] 4 shows an example of a data structure for haptic data according to at least the second embodiment, in which base effects are defined based on key frames.

[0064] Haptic description file (or in other words, haptic data) 400 includes a first level 401, including a file description 402, a set of avatars 403, a set of signals 410, and a geometry 405. File description 402 includes some generic metadata, including the file version, date, and a description of the associated haptic experience. Set of avatars 403 includes a definition of the body model to which the haptic effect is to be applied. Geometry 405 determines the volume within the immersive scene over which the haptic effect is active. Set of signals 410 aggregates signals, such as signal 411.

[0065] Signal 411 is expanded in the second row of the figure and includes metadata 421, a reference 422 to an avatar selected from set of avatars 403 (e.g., a reference to avatar 404), an effects library 430, and a set of haptic trajectories 440. Metadata 421 includes information about the type of signal (vibration, pressure, temperature, etc.) and a description of the signal. The effects library includes a set of effects such as effects 431 and 432. An example of effect 431 is expanded in the bottom left of the figure. Set of haptic trajectories 440 aggregates several haptic trajectories such as 441, 442, and 443. Haptic trajectory 441 is also expanded in the middle right of the figure.

[0066] Effects library 430 includes a set of effects, such as 431 and 432, where an effect is defined as a set of keyframes that represent signal values ​​to be rendered. For example, effect 431 is defined as consisting of keyframes 451, 452, and 453.

[0067] The haptic trajectory 441 includes a trajectory description 461 carrying semantic information, trajectory attributes 462 carrying several attributes such as gain, mixing weights, body part masks, and a set of haptic bands 470 such as 471, 472, 473. An example of a haptic band 471 includes several attributes 475 and an effect timeline 480. The effect timeline includes four elements 481, 482, 483, and 484. Elements 481 and 484 define an effect directly in the timeline with a set of keyframes, while elements 482 and 483 reference effects from the effect library, effects 431 and 432, respectively. Thus, element 482 references keyframes 451, 452, 453 by referencing effect 431.

[0068] An example JSON schema for a haptic effect according to the second embodiment is shown in Table 5.

[0069] [Table 5-1]

[0070] [Table 5-2]

[0071] As can be seen above, haptic effects can be of three types: foundational, timeline, or reference. Foundational effects are defined by an id and contain a list of haptic "keyframes". Optionally, a topological attribute may be added to a foundational effect (not shown). An example of a foundational effect is effect 331 in FIG. 4. Timeline effects are also identified by an id, but are defined by a list of haptic keyframes associated with timestamps. An example of a timeline effect in FIG. 4 is effect 481. Finally, a reference effect is simply a reference to an existing effect in the effects library, with the id attribute indicating the referenced effect. An example of a timeline effect in FIG. 4 is effect 482, which references effect 431 and therefore contains keyframes 451, 452, 453.

[0072] At the signal level, the set of effects is stored in an "effect_library", as shown in the example JSON schema in Table 6.

[0073] [Table 6-1]

[0074] [Table 6-2]

[0075] Effects are used at the tactile band level, as shown in Table 7.

[0076] [Table 7-1]

[0077] [Table 7-2]

[0078] Table 8 shows an example of the JSON syntax of a haptic file according to the second embodiment, based on the same example as in the first embodiment (Table 4). With this solution, the base effects can be more easily defined, but all effects of a given trajectory use the same frequency band decomposition.

[0079] [Table 8-1]

[0080] [Table 8-2]

[0081] [Table 8-3]

[0082] 5 shows an example of a haptic file encoded without an effects library according to the second embodiment. The data is stored linearly and the same effects are repeated. Such a haptic file is particularly adapted for streaming.

[0083] Fig. 6 shows an example of a haptic file encoded using an effects library according to the second embodiment. This file describes the same effects, but is optimized to reduce space consumption by using an effects library. In fact, the same effects are moved to the effects library and then referenced in the timeline. Such a haptic file is especially adapted to the editing stage. In fact, if a correction needs to be made in a keyframe, it only needs to be made once.

[0084] Fig. 7 shows an example of an architecture of a haptic file encoder 700 according to at least one embodiment. The inputs are a metadata file 701 and at least one signal file 703. The metadata file 701 is for example based on the "OHM" haptic object file format. The signal file is conventionally using a PCM encoded file, for example based on the WAV file format. The description file 702 is for example based on the AHAP or HAPT file format. The exchange file 704 is a human readable file, for example based on the glTF, XML or JSON format. The distribution file 705 is a binary encoded file, for example based on the MPEG file format adapted for streaming or broadcasting to a decoder device.

[0085] Metadata is extracted 710 from the metadata file 701, allowing the description file and / or the signal file to be identified. The description file is analyzed and transcoded in step 711. In step 712 the signal file is decomposed into frequency bands and in step 713 key frames or wavelets are extracted. Then in step 720 an exchange file 704 is generated according to a data format according to one of the embodiments described herein. This file may be compressed in step 730 and distributed in a more compact format suitable for transmission than the exchange file format.

[0086] FIG. 8 illustrates a data restructuring process according to an embodiment. This process 800 allows, for example, to convert the file of FIG. 5 into the file of FIG. 6, and vice versa. In other words, the process allows the haptic structure to be adapted to use linearized (and therefore duplicated) keyframes, or to use library effects that factorize the keyframe descriptions, if possible. In a first step 801, the signal data is analyzed. Typically, in this step, the process examines the effects defined in the timeline to identify multiple identical effects. In the example of FIG. 5, the process identifies that the combination of keyframe 1 and keyframe 2 is used four times. In a second step, the process performs one of the optimizations. In the case of data linearization 802, the goal is to obtain a data structure that is most suitable for streaming. In this case, library-based effect references in the timeline are replaced with copies of the effects, and all effects are removed from the library. In the case of data refactoring 803, the goal is the opposite, in other words to compress the data structure. For that purpose, when an identical effect is detected in the timeline, such effect is added to the effects library (if it does not already exist), and in the timeline, the effect is replaced with a reference to the effect in the library.

[0087] Finally, other types of optimizations 804 can also be added to this process. For example, the process can not only merge identical effects, but also merge effects that are similar enough to further optimize the gain in space. Such optimizations require additional computations during the signal analysis to evaluate some distance metric between the effects. An associated distance threshold would have to be provided to define the similarity between the effects for merging to occur.

[0088] This data restructuring process can also be implemented in multiple ways. For example, it can be inserted after the formatting step 720 in the exemplary architecture of Figure 7. It can also be implemented as a standalone tool that restructures and re-exports data of existing haptic files (either as exchange or distribution files), or it can be implemented as an import plug-in for haptic file editing tools. It can also be included in the binary compression step 730.

[0089] Table 9 shows an example of JSON syntax for a haptic file according to the second embodiment after the data linearization process. In fact, the two examples of Table 4 and Table 8 have already been optimized to save space and make editing of the haptic signals easier. For streaming purposes, this data organization is inconvenient. The table below shows the same example after data linearization. This process replaces all references to effects with copies of the effects and removes the original references from the library. Data restructuring based on the second embodiment results in the following file, which is much longer but easier to stream:

[0090] [Table 9-1]

[0091] [Table 9-2]

[0092] [Table 9-3]

[0093] The proposed data structures are presented here in a JSON-readable format, however the same principles apply to the binary version of the format used in the distribution files.

[0094] 9 illustrates an example of a process for rendering haptic data according to at least one embodiment. The process 900 is performed, for example, by the processor 101 of the device 100 of FIG. 1. In step 910, the processor obtains the haptic data. In step 920, the processor provides a haptic signal to a haptic actuator as described by the haptic data.

[0095] Although various embodiments have been described separately, the embodiments can be combined in any combination while respecting the principles of the present disclosure.

[0096] Although the embodiments relate to haptic effects, one skilled in the art will appreciate that the same principles can be applied to other effects, such as sensory effects, and thus including smells and tastes, and thus the appropriate syntax determines the appropriate parameters associated with these effects.

[0097] References to "one embodiment" or "embodiment" or "one implementation" or "implementation," as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with an embodiment is included in at least one embodiment. Thus, appearances of "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" appearing in various places throughout this application, as well as other variations, do not necessarily all refer to the same embodiment.

[0098] Additionally, the application may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0099] Additionally, the application or claims thereof may refer to "obtaining" various information. Obtaining, like "accessing," is intended to be a broad term. Obtaining information may include, for example, one or more of receiving information, accessing information, or retrieving information (e.g., from a memory or optical media storage device). Furthermore, "obtaining" generally involves in some way, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0100] For example, in the case of "A / B," "A and / or B," and "at least one of A and B," it should be understood that the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), or the selection of only the first and third listed alternatives (A and C), or the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A and B and C). This may be expanded as many times as the number of items listed, as would be readily apparent to one of ordinary skill in this and related arts.

Claims

1. haptic data, The type of effect and A list of haptic effects, a list of haptic trajectories, the haptic trajectories including a timeline with at least one temporal reference associated with a haptic effect or an identifier in the list of haptic effects; a haptic effect includes information representing values ​​of signals to be applied to render the haptic effect; method.

2. The method of claim 1 , wherein the signals are grouped according to a set of frequency bands.

3. The method of claim 1 , wherein the timelines are grouped according to sets of frequency bands.

4. 1. A method for rendering haptic data, comprising: haptic data, The type of effect and A list of haptic effects, obtaining haptic data including information representing a list of haptic trajectories, the haptic trajectories including a timeline with at least one temporal reference associated with a haptic effect or an identifier in the list of haptic effects, the haptic effects including information representing values ​​of signals to be applied to render the haptic effect; providing a value of the signal to a haptic actuator.

5. The method of claim 4 , wherein the signals are grouped according to a set of frequency bands.

6. The method of claim 4 , wherein the timelines are grouped according to sets of frequency bands.

7. a processor, the processor comprising: haptic data, The type of effect and A list of haptic effects, acquiring haptic data including information representing a list of haptic trajectories, the haptic trajectories including a timeline with at least one temporal reference associated with a haptic effect or an identifier in the list of haptic effects, the haptic effect including information representing values ​​of signals to be applied to render the haptic effect; configured to provide a signal value to a haptic actuator; A haptic rendering device.

8. The device of claim 7 , wherein signals are grouped according to a set of frequency bands.

9. The device of claim 7 , wherein the timelines are grouped according to sets of frequency bands.

10. haptic data, The type of effect and A list of haptic effects, a list of haptic trajectories, the haptic trajectory including a timeline with at least one temporal reference associated with a haptic effect or an identifier in the list of haptic effects; and a haptic effect including information representing values ​​of signals to be applied to render the haptic effect. Tactile data.

11. The haptic data of claim 10 , wherein the signals are grouped according to a set of frequency bands.

12. The haptic data of claim 10 , wherein the timelines are grouped according to sets of frequency bands.

13. 1. A non-transitory computer readable medium storing haptic data, the haptic data comprising: The type of effect and A list of haptic effects, A non-transitory computer-readable medium comprising: a list of haptic trajectories, the haptic trajectories including a timeline with at least one temporal reference associated with a haptic effect or an identifier in the list of haptic effects; and a list of haptic trajectories, the haptic effects including information representing values ​​of signals to be applied to render the haptic effect.

14. 14. The non-transitory computer-readable medium of claim 13, wherein the signals are grouped according to a set of frequency bands.

15. The non-transitory computer-readable medium of claim 13 , wherein the timelines are grouped according to sets of frequency bands.

16. 13. A method for reconstructing haptic data generated according to claim 1, wherein at least one temporal reference is associated with an identifier in the list of haptic effects, the method comprising: Acquiring tactile data; and analyzing the haptic data to determine an identifier for a haptic effect in a timeline; replacing the determined identifier with a copy of the haptic effect associated with the identifier; Emptying the list of haptic effects, providing said reconstructed haptic data; and A method comprising:

17. 13. A method for reconstructing haptic data generated according to claim 1, wherein at least one temporal reference is associated with a haptic effect, the method comprising: Acquiring tactile data; and analyzing the haptic data to determine a same haptic effect; inserting a copy of one of the determined identical haptic effects into the list of haptic effects; Associating an identifier with the inserted identical haptic effect; replacing the identical haptic effect with the associated identifier; and providing said reconstructed haptic data; and A method comprising:

18. A computer program comprising program code instructions which, when executed by a processor, performs the method according to any one of claims 1 to 6.

19. A computer program comprising program code instructions which, when executed by a processor, performs the method according to any of claims 16 or 17.

20. A non-transitory computer readable medium comprising program code instructions which, when executed by a processor, implements the method of any one of claims 1 to 6.

21. A non-transitory computer readable medium comprising program code instructions which, when executed by a processor, implements the method of any of claims 16 or 17.

Citation Information

Patent Citations

  • Systems, devices and methods for streaming haptic effects

    JP2020036315A

  • System and method for providing customizable haptic playback

    JP2020080152A