Scene description framework for haptic interactivity

Abstract

An immersive scene description is defined by a data structure containing information representing triggers and haptic effects. When a trigger occurs while the user is interacting with the immersive scene, parameters for the haptic effect are retrieved from the data structure and provided to the haptic actuator for rendering the haptic effect. These parameters determine at least the modality and perception of the haptic effect, and can determine at least some of the body parts to which the haptic effect should be applied. The modality relates to the type of haptic effect, such as temperature, vibration, pressure, or acceleration. Perception determines the haptic signal to be applied to the corresponding haptic actuator. This signal is generated or retrieved from a file or other device.

Description

Technical Field

[0001] At least one of the embodiments generally relates to immersive scene descriptions, and more particularly to formats and syntax for obtaining interactive haptic effects.

Background Art

[0002] This application claims the priority of European Application No. 23305357.8 filed on March 16, 2023 and European Application No. 23305464.2 filed on March 31, 2023, the entire disclosures of which are incorporated herein by reference.

[0003] In recent years, the field of virtual reality has advanced rapidly, enabling users to immerse themselves in a virtual world and interact with objects therein. A fully immersive user experience is proposed to users by immersive systems based on feedback and interaction. In interaction, conventional control methods that satisfy users' needs may be used. Current visual and auditory feedback provides a satisfactory level of realistic immersion. Additional feedback can be provided by haptic effects that allow human users to perceive the virtual environment with their senses, thus enabling a better fully immersive experience with improved realism. However, haptics remains one area that has the potential to advance in order to improve the overall user experience in immersive systems.

[0004] Traditionally, immersive systems may include 3D scenes that represent a virtual environment in which virtual objects are localized within the 3D scene. To improve user interactivity with elements of the virtual environment, haptic feedback may be used through stimuli of haptic actuators. Such interaction is based on the concept of "haptic objects" that correspond to physical phenomena to be transmitted to the user. In the context of an immersive scene, haptic objects can provide haptic effects by defining appropriate haptic actuator stimuli and mimicking physical phenomena with a haptic rendering device. Different types of haptic actuators can reproduce different types of haptic feedback.

[0005] An example of a haptic object is an explosion. Explosions are rendered using vibration and heat, thereby combining different haptic effects for the user to enhance realism. Immersive scenes typically include multiple haptic objects, for example, using a first haptic object associated with global effects and a second haptic object associated with local effects.

[0006] The principles described herein apply to any immersive environment that uses touch, such as augmented reality, virtual reality, mixed reality, or haptic-enhanced video (or omnidirectional / 360° video) rendering, and more generally to any haptic-based user experience. Thus, example scenes of such immersive environments are considered immersive scenes.

[0007] Tactile sensation refers to the sense of contact and includes two dimensions: tactile and kinesthetic. The first relates to tactile sensations such as friction, roughness, hardness, and temperature, and is felt through mechanoreceptors in the skin (Merkel cells, Ruffini endings, Meissner corpuscles, Pacinian corpuscles) and thermoreceptors. The second is linked to the sense of force / torque, position, and motion / velocity provided by muscles, tendons, and mechanoreceptors in the joints. Tactile sensation is also involved in the perception of one's own motion, as it contributes to the prosopagnostic system (i.e., the perception of one's own body). Therefore, perceptions of acceleration, velocity, or any body model can be assimilated as tactile effects. The frequency range is approximately 0–1 kHz, depending on the type of modality. Most existing devices that can render tactile signals generate vibrations. Examples of such tactile actuators include linear resonant actuators (LRAs), eccentric rotating masses (ERMs), and voice coil linear motors. These actuators are sometimes incorporated not only into haptic rendering devices such as haptic suits, but also into smartphones and game controllers.

[0008] To convey immersive scene information, it is crucial to use a syntax format that accurately describes the virtual environment and the objects within it. This syntax format can be designed to be easily interpreted by both the human user who conceives the immersive scene and the computer system that interprets and renders the immersive scene.

[0009] To encode haptic signals, several formats are defined that relate to one of the following: high-level descriptions using XML-like formats (e.g., MPEG-V), parametric representations using JSON-like formats such as Apple Haptic Audio Pattern (AHAP) or Immersion Corporation's HAPT format, or waveform coding (IEEE 1918.1.1, an ongoing standard for haptic and kinesthetic signals). The HAPT format was recently included in the MPEG ISOBMFF file format specification (ISO / IEC 14496 Part 12). Furthermore, the GL Transmission Format (glTF®) is a royalty-free specification for the efficient transmission and loading of 3D scenes and models by applications. This format defines an extensible common publishing format for 3D content tools and services that simplifies authoring workflows and enables interoperable use of content across the industry.

[0010] Furthermore, a haptic file format has been defined within the MPEG standardization group and is associated with encoded representations of haptics. Using this format, encoded haptic description files can be exported as a human-readable JSON exchange format (e.g., .hjif files) or as a compressed binary distribution format (e.g., .mpg) specifically adapted for transmission to haptic rendering devices. In addition, MPEG has published a standard focused on scene description. This MPEG-I Scene Description (SD) standard is an extension of the existing glTF™ format. Modifications to this MPEG-I SD standard are defined, focusing on several aspects, including interactivity and haptics. The embodiments described herein are built upon the MPEG-I SD standard. [Brief explanation of the drawing]

[0011] [Figure 1]This is a block diagram of an example of an immersive system in which various forms and embodiments are implemented. [Figure 2] This figure shows an example of a data structure for describing an immersive scene, according to at least one embodiment. [Figure 3] This figure shows an example of a data structure for a tactile object. [Figure 4] This figure shows an example of a concrete implementation of the data structure for a tactile object. [Figure 5] This figure shows a simple example of a data structure for interactive haptic actions. [Figure 6] This figure shows an example of a data structure for interactive haptic action according to the first embodiment. [Figure 7] This figure shows an example of a data structure for interactive haptic actions according to the second embodiment. [Figure 8] This figure shows an example of a data structure for interactive haptic action according to the third embodiment. [Figure 9] This figure shows an example of a data structure for interactive haptic actions according to the fourth embodiment. [Figure 10] This figure shows an example of a scene description architecture including interactive haptic actions according to the first embodiment. [Figure 11] This figure shows an example of a scene description architecture including interactive haptic actions according to the second embodiment. [Figure 12] This figure shows an example of a scene description architecture including interactive haptic actions according to the third embodiment. [Figure 13] This figure shows an example flowchart for addressing interactive haptic actions, according to at least one embodiment. [Modes for carrying out the invention]

[0012] The embodiments described below are designed with the foregoing in mind and introduce the concept of interactive haptic actions defined within the immersive scene description as a data structure containing information representing triggers and haptic effects. When a trigger occurs while the user is interacting with the immersive scene, parameters for the haptic effect are retrieved from the data structure and provided to the haptic actuator for rendering the haptic effect. These parameters can determine at least the modality and perception of the haptic effect and at least one part of the body to which the haptic effect should be applied. The modality relates to the type of haptic effect (e.g., temperature, vibration, pressure, acceleration). The perception determines the haptic signal to be applied to the corresponding haptic actuator. This signal can be generated or retrieved from a file or another device.

[0013] A first aspect of at least one embodiment relates to a method comprising: obtaining from an immersive scene description a data structure representing an interactive haptic action, which is associated with a virtual object or immersive scene, and defines behavior based on triggers and actions, the actions being associated with haptic effects; determining parameters of a haptic effect in response to the occurrence of an event corresponding to a trigger, which parameters include at least the type of haptic effect and the perception of a haptic signal for the haptic effect; and providing haptic data for rendering a haptic effect, at least based on the type of haptic effect and the perception of a haptic signal for the haptic effect.

[0014] A second aspect of at least one embodiment relates to a device including a processor configured to acquire a data structure representing an interactive haptic action from an immersive scene description, the data structure being associated with a virtual object or immersive scene, and defining behavior based on triggers and actions, the actions being associated with haptic effects; and determining parameters of a haptic effect in response to the occurrence of an event corresponding to a trigger, the parameters including at least the type of haptic effect and the perception of a haptic signal for the haptic effect; and providing haptic data for rendering a haptic effect, at least based on the type of haptic effect and the perception of a haptic signal for the haptic effect.

[0015] A third aspect of at least one embodiment relates to a computer program that includes program code instructions executable by a processor for implementing steps of the method according to at least one aspect of the first embodiment.

[0016] A fourth aspect of at least one embodiment relates to a non-temporary computer-readable medium that stores program code instructions executable by a processor for implementing steps of the method according to at least one aspect of the first embodiment.

[0017] Figure 1 shows a block diagram of an example of an immersive system in which various aspects and embodiments are implemented. In the immersive system depicted, user Alice uses a haptic rendering device 100 to interact with a server 180 hosting an immersive scene 190 via a communication network 170. This immersive scene 190 may include various data and / or files representing different elements necessary for its rendering (scene description 191, audio data, video data, 3D models, and haptic description file 192). The immersive scene 190 may be generated under the control of an immersion experience editor 110, which allows for the arrangement of different elements and the design of an immersive experience. The appropriate description file and various data files representing the immersive experience are generated by an immersion scene generator 111 (also called an encoder), encoded in a format suitable for transmission to the haptic rendering device, and stored on the server 180. The immersion experience editor 110 may include a graphical user interface, typically hosted by a computer and configured to generate immersive scenes. For simplicity, the diagram shows the immersion experience editor 110 directly connected to the immersion scene 190 by a dotted line 171. In reality, the immersion scene 190 is hosted on server 180, and the computer running the immersion experience editor 110 is connected to server 180 via communication network 170.

[0018] The haptic rendering device 100 includes a processor 101. The processor 101 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, and the like. The processor can perform data processing such as haptic signal decoding, input / output processing, and / or any other functions that enable the device to operate in an immersive system.

[0019] Processor 101 can be coupled to an input unit 102 configured to communicate user interactions. For that purpose, multiple types of input and modalities can be used. Physical keypads and touch-sensitive surfaces are typical examples of input units adapted for this use, although voice control can also be used. Additionally, the input unit can also include a digital camera capable of capturing still images or videos in two dimensions, or a more complex sensor capable of determining depth information in addition to images or videos, and thus capturing a complete 3D representation. Processor 101 can be coupled to a display unit configured to output visual data to be displayed on a screen 103. For that purpose, multiple types of displays, such as a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display unit, can be used. Processor 101 can also be coupled to an audio unit 104 configured to render audio data to be converted into audio waves by a suitable transducer, such as a speaker. Processor 101 can be coupled to a communication interface 105 configured to exchange data with an external device. The communication preferably uses a wireless communication standard, such as cellular (e.g., LTE) communication, Wi-Fi communication, etc., to provide the mobility of the haptic rendering device. Processor 101 can access information from a memory 106 that can include multiple types of memory, including random access memory (RAM), read-only memory (ROM), hard disk, subscriber identity module (SIM) card, memory stick, secure digital (SD) memory card, and any other type of memory storage device, and can store data in the memory 106. In an embodiment, Processor 101 can access information from a memory that is not physically located on the device, such as a server, home computer, or other device, and store data in the memory.

[0020] Processor 101 is coupled to a haptic unit 107 configured to provide haptic feedback to a user, and the haptic feedback is described in a haptic description file 192 related to the scene description 191 of the immersive scene 190. The haptic description file 192 describes the types of feedback provided according to a syntax further described below. Such a description file is typically transmitted from server 180 to haptic rendering device 100. The haptic unit 107 may include a single haptic actuator or multiple haptic actuators placed at multiple locations of the haptic rendering device. Different haptic units may have different numbers of actuators, and / or the actuators may be separately arranged in the haptic rendering device.

[0021] In at least one embodiment, processor 101 is configured to render haptic signals, i.e., apply low-level signals to haptic actuators to render haptic effects, according to embodiments further described below. Such low-level signals can be represented using different forms, for example, by metadata or parameters in the description file, or by digital encoding of sampled analog signals (e.g., PCM or LPCM).

[0022] Processor 101 may receive power from power supply 108 and may be configured to distribute and / or control power to other components of device 100. Power supply 108 may be any suitable device for supplying power to the device. By way of example, power supply 108 may include one or more dry batteries (e.g., nickel cadmium (NiCd), nickel zinc (NiZn), nickel metal hydride (NiMH), lithium ion (Li-ion), etc.), solar cells, fuel cells, and the like.

[0023] In the diagram, the processor 101 and other elements 102 through 108 are depicted as separate components, but it should be understood that these elements may be integrated into an electronic package or chip. It should be understood that the haptic rendering device 100 may include any subcombination of the elements described herein, while maintaining consistency with the embodiments described below. The processor 101 may also be further coupled to other peripherals or units not shown in Figure 1, which may include one or more software modules and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, peripherals may include sensors such as a Universal Serial Bus (USB) port, vibration devices, television transceivers, hands-free headsets, Bluetooth® modules, frequency modulation (FM) radio units, digital music players, media players, video game player modules, and internet browsers. For example, the processor 101 may be coupled to a localization unit configured to localize the haptic rendering device within its environment. The localization unit may incorporate a GPS chipset that provides the longitude and latitude of the haptic rendering device's current location, but may also incorporate motion sensors such as accelerometers and / or an electronic compass that provides localization services.

[0024] Typical examples of haptic rendering devices 100 include haptic suits, smartphones, game controllers, haptic gloves, haptic chairs, haptic props, and motion platforms. However, any device or configuration of a device that provides similar functionality can be used as a haptic rendering device 100, while still adhering to the principles of this disclosure.

[0025] In at least one embodiment, the device includes a haptic unit but does not include a display unit. In such an embodiment, the device renders only haptic effects without visually rendering the scene. However, the device may provide display data so that another device, such as a screen, can perform the display. Examples of such devices include haptic suits or motion platforms.

[0026] In at least one embodiment, the device does not include a haptic unit but includes a display unit. In such embodiments, the device merely visually renders the scene without rendering haptic effects. However, the device may provide data for rendering haptic effects so that another device, such as a haptic prop, can perform haptic rendering. Examples of such devices include smartphones, head-mounted displays, or laptops.

[0027] In at least one embodiment, the device does not include a display unit or a haptic unit. In such an embodiment, the device does not visually render the scene or render haptic effects. However, the device may provide display data so that another device, such as a screen, can perform the display, and may provide data for rendering haptic effects so that another device, such as a haptic prop, can perform the haptic rendering. Examples of such devices include computers, game consoles, optical media players, or set-top boxes.

[0028] In at least one embodiment, the immersive scene 190 and associated elements are hosted directly in the memory 106 of the haptic rendering device 100, enabling local rendering and interaction. In a variant of this embodiment, the device 100 also includes an immersive experience editor 110 that enables complete standalone operation, requiring no communication network 170 and server 180 at all.

[0029] Although the different elements of Immersion Scene 190 are depicted as separate elements in Figure 1, the principles described herein also apply when these elements are directly integrated in the scene description and are not separate elements. Any mixture between the two alternative forms is also possible, where some elements are integrated in the scene description and others are in separate files.

[0030] Although the haptic rendering device 100 is described herein as a single device, it can also be implemented as a combination of separate haptic rendering devices.

[0031] For the sake of simplicity, interaction and haptic effects are described herein using fingers touching a haptic surface as the interaction medium. However, any other element that represents the user's position in an immersive environment (such as a part of the user's body, a position provided by a force feedback device, or the localization of a head-mounted display in a virtual reality environment) can also be used based on the same principles.

[0032] Figure 2 illustrates an example of a data structure for an immersive scene description to address interactive haptic actions, according to at least one embodiment. This embodiment is based on the glTF® file format. At the core of glTF® is a JSON file that describes the structure and configuration of a scene containing a 3D model. The figure shows the relationships between the elements that make up this data structure of the immersive scene description 200. In this context, scene 201 is the top-level element that brings together all the other elements. It contains an array of nodes. Each node 202 can contain child nodes that allow for the creation of a hierarchy. Nodes may refer to meshes, cameras, or skins, and local geometry transformations may be associated with the nodes. Mesh 210 corresponds to the geometry data required to render the mesh. Skin 220 is used to perform vertex skinning so that the vertices of the mesh are affected by the skeleton bones based on their pose. Camera 225 determines the projection matrix. Light source 215 determines the lighting properties associated with the node. Buffer 255 contains the geometry of the 3D model, animation, and data used for skinning. The buffer view 250 adds structural information to the buffer data, and the access mechanism 245 defines the exact type and layout of the buffer view. The material 260 determines how the object should be rendered based on its physical material properties. The texture 265 allows the appearance of the object to be defined. The image 270 defines the image data used for the texture, and the sampler 280 describes the wrapping and scaling of the texture. Media files (e.g., audio WAV files of sound or tactile signals) can be combined within the MPEG media 205.

[0033] The immersive scene description file further includes haptic objects 230 that describe the haptic effects to be rendered. Haptic objects are identified in the file format as "MPEG_Haptic," and their syntax is further described below. Haptic objects are connected at the node level to indicate that the node references haptic media and relies on independent haptic media such as audio content or images. Haptic objects can be associated with material haptics 235, identified in the file format as "MPEG_material_haptics," which allow haptic effects to be defined based on textures 265.

[0034] The immersive scene description file further includes interactivity-related objects placed at the scene level (MPEG Scene Interactivity 203) or node level (MPEG Node Interactivity 204). These interactivity-related objects allow for the definition of interactions between the user and immersive objects (nodes), thereby creating more animated immersive scenes.

[0035] Other elements are conventional elements of immersive representation and are not relevant to the embodiments; therefore, they are not described herein.

[0036] The embodiments described herein aim to enhance the quality of the user experience by adding haptic effects to the interactivity of immersive scenes, thereby enabling users to physically perceive the interaction through vibration, heat, movement, and the like.

[0037] According to the embodiment, these elements of an immersive scene description file based on the glTF® format enable the definition of an immersive scene with interactive haptic actions.

[0038] Figure 3 illustrates an example of a data structure for a tactile object. Data structure 300 represents the tactile object 230 in Figure 2 and can be broken down into a set of layers. At the top layer, metadata 301 describes high-level metadata information about the overall tactile experience defined in data structure 300, and a list of avatars 302 (i.e., bodily representations) that will later be referenced in the file. These avatars allow specifying the target location of tactile stimuli on the body. Tactile effects are described by a list of perceptions 310, 31N. These perceptions correspond to tactile signals associated with specific perceptual modalities such as vibration, force, position, velocity, and temperature. Perceptions include metadata 320 for describing the parameters of the tactile signal (such as modality), a device 321 for describing the specifications of the reference tactile device on which the signal was designed, and a list of tactile channels 331, 33N. A haptic channel includes metadata 340 to describe the channel's content, associated gain values, mixed weights, body localization information, and a reference to the haptic device specifications (defined at the perceptual level). The channel ultimately contains a list of haptic bands 351, 35N, each band defining a subset of signals within a given frequency range. For example, haptic band 351 may correspond to a frequency range from 0 to 50 Hz, and haptic band 35N may correspond to a frequency range above 2 kHz. A haptic band includes band data 360 to describe the band's frequency range, the type of encoding modality (vector or wavelet), the type of band (transient, curve, and wave), and optionally the type of curve interpolation (cubic, linear, or unknown) or window length. A haptic band is defined by a list of haptic effects 371, 37N. Finally, the haptic effect includes a list of keyframes 391, 39N and effect data 380, where the keyframes are defined by position (i.e., time reference), frequency, and amplitude. The effect data describes the type of base signal selected from Sine, Square, Triangle, SawToothUp, and SawToothDown, and provides a time reference such as a timestamp. In this case, the low-level haptic signal can be reconstructed based on keyframes in different frequency bands.

[0039] Figure 4 illustrates an example of the embodiment of the data structure of a haptic object. It represents an example of an immersive scene description 400 and shows the practical use of the elements in Figures 2 and 3. In this example, scene 410 contains three nodes. The first node 420 consists of a mesh representing a 3D object. The second node 430 contains a haptic object ("MPEG_haptic" in a standardized format) which contains a media reference for the haptic media in the MPEG_media array 450. Thus, the haptic object contains two haptic perceptions. The first perception 451 ("perception_1") relates to a temperature effect (modality is set to "temperature"), and its value is defined with an appropriate setting using transient bands. The second perception 452 ("perception_2") relates to a vibration effect (modality is set to "Vibrotactile") and contains two channels 453, 454 defined using keyframe-based frequency bands. The third node, 440, contains a mesh (in the standardized format "MPEG_material_haptic") representing a 3D object associated with a haptic texture. The haptic texture is used to define multiple haptic effects. The thermal haptic effect is defined by an array of 2D textures, each containing a single element, and the vibration haptic effect is defined by an array of 2D textures. Other modalities (only stiffness is represented) are not used. Therefore, these modalities are associated with empty arrays to specify that the haptic effects of these modalities are not defined for this mesh.

[0040] Figure 5 illustrates a simple example of the data structure for interactive haptic actions. At the highest level, behavior 510 corresponds to interaction in an immersive world and is defined by a pair of triggers 520 and actions 530. The trigger determines when to perform the interaction, and the action determines what should be done. Triggers and actions are described further below. A behavior is a combination of a trigger using logical operators and related actions that should be started sequentially or in parallel. For example, a behavior could combine a proximity trigger with a media action to start playing a specific media when the user enters a given range of a particular object in a 3D scene. This could, for example, allow the sound of a dog barking to be played when the user approaches a virtual dog in an immersive scene. In addition to conventional actions, interactive haptic actions can be defined using the ACTION_SET_HAPTIC property 531. A simple formulation of this element is proposed in Table 4. In this implementation, the ACTION_SET_HAPTIC property contains an array of indices for node 540 in the node array to initiate haptic feedback defined in the haptic object 550 associated with the node, as previously described in Figures 3 and 4.

[0041] Behavior, triggers, and actions (including interactive haptic effects) are defined at the scene level using the MPEG Scene Interactivity 203 element in Figure 2 and Table 1, and are related to virtual objects in the immersive scene. Additional data may be provided at the level of the affected glTF® node to specialize trigger activation using the MPEG Node Interactivity 204 element in Figure 2.

[0042] Table 1 describes the semantics of the MPEG_scene_interactivity element, which contains an array of triggers, actions, and behaviors. In the following table describing the element's semantics, the first column determines the element's name, the second defines the data type (here, all elements are arrays), the third defines whether the element is required (represented by "M") or optional (represented by "O"), the fourth defines the element's default value (if any), and the last column describes the element.

[0043] [Table 1]

[0044] Different types of triggers are defined, including collision, proximity, user input, and visibility, to cover the various ways in which actions are triggered. Each type of trigger is associated with a set of properties that define the interaction. For example, a proximity trigger specifies a distance range and a list of nodes associated with the trigger that define the distance or range in which the action is triggered. Table 2 describes the semantics of the triggers.

[0045] [Table 2]

[0046] Table 3 defines the types of triggers.

[0047] [Table 3]

[0048] Different types of actions are defined, and these different types of actions are covered by nodes, animations, or media. Each type of action is associated with a set of properties that are specific to the action. For example, a media action is specified via an index of media (such as an audio file) within an MPEG media array, and media control allows the selected media to be played or paused. Table 4 describes the semantics of the actions.

[0049] [Table 4-1]

[0050] [Table 4-2]

[0051] Table 5 defines the types of actions.

[0052] [Table 5]

[0053] Table 6 defines the control over animations for the "ACTION_ANIMATION" element.

[0054] [Table 6]

[0055] Table 7 defines the controls for media playback for the "ACTION_MEDIA" element.

[0056] [Table 7]

[0057] Table 8 defines the types of operation actions for the "ACTION_MANIPULATE" element.

[0058] [Table 8]

[0059] To provide at least one element of an immersive scene in the animation type, behavior links triggers and actions. Table 9 describes the semantics of triggers.

[0060] [Table 9]

[0061] Table 10 defines different trigger activation controls.

[0062] [Table 10]

[0063] Table 11 describes the semantics of the MPEG_node_interactivity element, allowing for the specialization of the behavior of a given node.

[0064] [Table 11-1]

[0065] [Table 11-2]

[0066] In at least one embodiment, interactive haptic effects are implemented in an immersive scene description (Figure 1, 191; Figure 4, 400) which includes a haptic object that uses elements defined in the action element description shown in the JSON schema of Table 12.

[0067] [Table 12-1]

[0068] [Table 12-2]

[0069] Embodiments of this specification describe different definitions of the ACTION_SET_HAPTIC property. Simple formulations of this property are proposed in Tables 4 and 12 above. However, they have the drawback that they can only specify a list of nodes. While this can be used to specify which nodes containing haptic data should be processed, it is not sufficient to determine exactly how they should be processed. In fact, a given node may be associated with multiple haptic files, each possibly containing multiple haptic perceptions, with different modalities and multiple haptic channels. A node may also contain a mesh associated with a haptic material that may reference multiple textures at different resolutions and different haptic modalities. This formulation does not allow specifying how such diverse haptic data should be processed, thus hindering the possibility of separately triggering haptic signals connected to the same object, and therefore limiting the design of haptic-enabled interactive experiences. For example, in an immersive scene, the same haptic object may be used to render different types of haptic feedback at different moments. Experience designers may want to render vibration signals using proximity triggers on nodes, while friction maps may be used in conjunction with collision triggers on the same node.

[0070] The following embodiments describe different variations of the syntax of the ACTION_SET_HAPTIC property in Table 4 to provide advanced haptic interactivity features that enable the design of precise haptic interaction experiences and the creation of precise and consistent haptic feedback through the accurate specification of interactive haptic feedback. These embodiments propose enabling haptic interaction with objects in a scene, explicitly targeting specific types of haptic signals, explicitly targeting specific types of haptic materials, explicitly targeting haptic media connected to virtual objects in a scene, restricting interaction to specific body parts, and triggering washout effects based on interaction. The proposed solutions provide interoperability between haptic devices and authoring tools and are compatible with the MPEG Haptic format (ISO / IEC 23090-31: Haptics Coding) and existing haptic rendering methods.

[0071] Figure 6 illustrates an example of the data structure for an interactive haptic action according to the first embodiment. Similar to Figure 5, the data structure includes behavior 610 defined by a trigger 620-action 630 pair. The difference from Figure 5 relates to the new definition of the ACTION_SET_HAPTIC property 631. In this embodiment, in addition to the node index 641 (in the node array) containing the haptic object 650, the ACTION_SET_HAPTIC property 631 specifies several additional elements of the haptic action 640: modality 642 (the type of haptic effect to be rendered), modality 643 for the haptic texture, mask 644 specifying where on the body the effect can be rendered, washout boolean value 645, useCollider boolean value 646, haptic control 647 controlling the rendering of the haptic data, and loop boolean value 648. Table 13 describes the semantics of the ACTION_SET_HAPTIC property according to the first embodiment.

[0072] [Table 13]

[0073] Two different fields are used to define the type of tactile sensation. The Haptic_Modality field 642 defines the type of perception to be rendered for tactile media directly connected to the node, while the Material_Haptic_Modality field 643 defines the tactile modality to use for tactile textures connected to the node's mesh, and includes only modalities that can be defined using textures.

[0074] A list of possible values ​​for Haptic_Modality is detailed in Table 14 and corresponds to the list of possible perceptions specified in ISO / IEC 23090-31: Haptics Coding standard. If no haptic modality is specified, all types of haptics connected to the specified node will be rendered.

[0075] [Table 14]

[0076] A list of possible values ​​for Material_Haptic_Modality is detailed in Table 15 and matches the list of modalities specified in the working draft of Amendment 2 of the MPEG Scene Description Standard.

[0077] [Table 15]

[0078] If Material_Haptic_Modality is not empty, only the haptic material will be rendered. If both the Haptic_Modality and Material_Haptic_Modality properties are not empty, and the haptic material references haptic media connected to the node (using a Reference texture), only the haptic modality specified by Haptic_Modality will be rendered.

[0079] The BodyPartMask field can be optionally used to restrict the areas of the body where haptic effects will be rendered. If no value is specified, the default value applies to the entire body, i.e., there are no restrictions. A complete list of body parts is detailed in Table 16.

[0080] [Table 16]

[0081] Table 17 shows examples of combinations. For example, the left leg (4th row) is made up of the left toes (0x80000000), the left instep (0x20000000), the left sole (0x08000000), the left calf (0x02000000), and the left thigh (0x00800000), resulting in a combination value of 0xAA800000.

[0082] [Table 17]

[0083] This haptic action is optional and can trigger a washout of the associated haptic device. The washout is used to reset the haptic device (e.g., a robotic arm) to its origin. Typically, this can be used when changing environments in a game or transitioning to a new scene in an immersive experience.

[0084] The useCollider syntax element can be optionally set (a boolean value equivalent to true) to indicate that haptic feedback rendering is supposed to use collision information (such as the collision location on the avatar) to render the haptic feedback. When this flag is reset (a boolean value equivalent to false), it should rely solely on the information provided in the haptic file. This field is only relevant when associated with a COLLISION trigger. For example, a haptic file may contain haptic data for the entire body. If a collision is detected between the haptic object and a specific part of the avatar, and the useCollider option is set, rendering should be played only for that part of the body.

[0085] The `haptic_control` property allows you to control the rendering of haptic data, as shown in Table 18. It can be used to start, pause, resume, or stop rendering.

[0086] [Table 18]

[0087] The `loop` property is optional and allows you to specify whether the rendering of haptic feedback should loop (when set to true) or not (when set to false). If set to false, the haptic data is played once. If set to true, rendering can continue indefinitely. It can be stopped by triggering an action with the HAPTIC_STOP control on the same data. This is typically used through behavior related to situations where the condition is no longer met (i.e., elements in Table 10 such as TRIGGER_ACTIVATE_EACH_EXIT, for example).

[0088] The JSON schema corresponding to the first embodiment is defined in Table 19, showing the updated syntax of the ACTION_SET_HAPTIC element. The other elements remain unchanged from Table 12.

[0089] [Table 19-1]

[0090] [Table 19-2]

[0091] Figure 7 illustrates an example of the data structure for an interactive haptic action according to the second embodiment. Similar to Figure 5, the data structure includes a behavior 710 defined by a pair of triggers 720 and actions 730. The difference from Figure 5 relates to the new definition of the ACTION_SET_HAPTIC property 731. This second embodiment proposes a more precise method that allows specifying which haptic media should be rendered for a given node. ACTION_SET_HAPTIC defines a list of Haptic_action_nodes 740, 74x, as shown in Table 20.

[0092] [Table 20]

[0093] The individual properties of Haptic_action_nodes are described in Table 21. They include the media_indices property 752, which corresponds to a list of media indices in the Media_reference array connected to the node via the MPEG_Haptic extension. The other properties are the same as those in the first embodiment, but are defined individually for each node identified by its nodeID (or node index), resulting in a more granular definition of the haptic effect.

[0094] [Table 21]

[0095] The JSON schema corresponding to the second embodiment is defined in Table 22 and shows the updated syntax of the ACTION_SET_HAPTIC element. The other elements remain unchanged from Table 12.

[0096] [Table 22-1]

[0097] [Table 22-2]

[0098] [Table 22-3]

[0099] Figure 8 illustrates an example of the data structure for an interactive haptic action according to a third embodiment. Similar to Figure 5, the data structure includes behavior 810 defined by a pair of triggers 820 and actions 830. The difference from Figure 5 relates to the new definition of the ACTION_SET_HAPTIC property 831. This third embodiment proposes a more precise solution that allows specifying which perception of which media should be rendered for a given list of nodes. This solution is hierarchical and, with the introduction of the HAPTIC_ACTION_MEDIA properties 840, 84x, allows for more precise selection of data from the haptic data structure. In addition to specifying the type of modality to be rendered, this solution allows for explicit specification of which perception should be rendered based on its identifier.

[0100] The nodeID, Body_Part_Mask, washout, useCollider, and Material_Haptic_Modality properties are the same as in previous embodiments. In this embodiment, the Haptic_Modality field is provided here at the media level and is optional.

[0101] For that purpose, the HAPTIC_ACTION_MEDIA list 851 provides access to one or more haptic action media elements 860, 86x. These elements include a media index 871, one or more perception indices 872, and one or more modality indices 873.

[0102] In addition to the perception_indices list, the Haptic_Modality property can also be used, as in previous embodiments, to restrict rendering to perceptions that match a specified modality and ignore others.

[0103] Tables 23, 24, and 25 show the new definitions of the ACTION_SET_HAPTIC property, HAPTIC_ACTION_NODE, and HAPTIC_ACTION_MEDIA according to the third embodiment, respectively.

[0104] [Table 23]

[0105] [Table 24]

[0106] [Table 25]

[0107] The JSON schema corresponding to the third embodiment is defined in Table 26, showing the updated syntax of the ACTION_SET_HAPTIC element. The other elements remain unchanged from Table 12.

[0108] [Table 26-1]

[0109] [Table 26-1]

[0110] [Table 26-1]

[0111] Figure 9 illustrates an example of the data structure for an interactive haptic action according to the fourth embodiment. Similar to Figure 5, the data structure includes a behavior 910 defined by a trigger 920-action 930 pair. The difference from Figure 5 relates to the new definition of the ACTION_SET_HAPTIC property 931. This third embodiment extends the hierarchical structure of the third embodiment and goes a step deeper into the haptic data structure to explicitly define which haptic channels of which perception will be rendered, with the introduction of the HAPTIC_ACTION_PERCEPTION property 980, 98x.

[0112] Tables 27, 28, 29, and 30 show the new definitions of the ACTION_SET_HAPTIC property, HAPTIC_ACTION_NODE, HAPTIC_ACTION_MEDIA, and HAPTIC_ACTION_PERCEPTION according to the third embodiment, respectively.

[0113] [Table 27]

[0114] [Table 28]

[0115] [Table 29]

[0116] [Table 30]

[0117] The JSON schema corresponding to the fourth embodiment is defined in Table 31, showing the updated syntax of the ACTION_SET_HAPTIC element. The other elements remain unchanged from Table 12.

[0118] [Table 31-1]

[0119] [Table 31-2]

[0120] [Table 31-3]

[0121] The embodiments introduced above are further described below in relation to a simple use case for enhancing a simple immersive experience by adding haptic data to elements of a scene. The immersive scene is a virtual car showroom where several car models are on display. The user can move around the virtual showroom and interact with virtual objects. For example, when the user touches a door, the door opens and the interior of the car is revealed. When the user views a car from a distance, it virtually starts the car's engine and triggers the rendering of audio content that represents the engine sound.

[0122] In addition to these conventional interactive actions, the experience can be enhanced by using different perceptual modalities; that is, virtual elements of a virtual showroom can be associated with haptic effects. The following diagrams show examples of such interactive haptic actions in four embodiments. These examples have been somewhat simplified for the sake of clarity in the diagrams. In fact, high-quality rendering of 3D objects requires far more elements than just a mesh.

[0123] Figure 10 shows an example of a scene description architecture including interactive haptic actions according to the first embodiment. In the first example, in addition to engine sounds, vibrations are triggered when the engine is virtually started (i.e., when the car becomes visible to the user), and the user feels a rise in temperature in different areas of the car. A temperature map can be connected to the car's mesh to represent the temperature distribution of the car body (hotter near the engine). Using the first embodiment, such an interactive haptic action can be defined using a scene description that includes a CAR#1 node, haptic media (e.g., a WAV file) defining the engine sound, haptic media defining the vibrations of the running engine (vibrational haptic perception), a haptic material representing a texture map of the car's temperature, and two haptic behaviors. The first Behavior#1 includes a VISIBILITY trigger that references a car node, an ACTION_SET_HAPTIC action that references a car node and has the Haptic_Modality property of the array set to "Vibrotactile", and an ACTION_MEDIA property that is set to MEDIA-PLAY and references an audio file that stores the engine sound. The second behavior "Behavior2" includes a COLLISION trigger that references a car node and an ACTION_SET_HAPTIC action that references a car node that has a Material_Haptic_Modality array containing a single element "Temperature".

[0124] In this scene description example, when the user views a haptic object, the rendering of all vibration haptic signals connected to the node is triggered. When the user collides with the object, the temperature texture connected to the node via the haptic material is rendered.

[0125] Figure 11 shows an example of a scene description architecture including interactive haptic actions according to the second embodiment. In the first example described in Figure 10, which relates to the first embodiment, if two different haptic media containing the same type of haptic perception modality are connected to a node, these two files are rendered. There is no way to control their rendering independently. For example, in the second example of an interactive experience, when the user visualizes a car, the engine starts (and idles), thereby playing a first vibration, and then when the user approaches the car and increases the engine speed, thereby playing a second vibration. This requires triggering a first vibration haptic signal for the first behavior and a second vibration haptic signal for the second behavior. Such behavior is not possible in the first embodiment; that is, when the user sees the car and approaches within a specified distance range, both vibration haptic signals are rendered. However, such a scenario can be implemented using the second embodiment, as described below for the second example shown in Figure 11.

[0126] The corresponding interactive haptic experience can be defined using a scene description that includes a CAR#1 node, a haptic media defining vib#0 for the vibration haptic perception of an idling engine, a second haptic media defining vib#1 for the second vibration haptic perception of a revving engine, a haptic material associated with the car's temperature texture map, and three haptic behaviors. The first behavior (Behavior#1) includes a VISIBILITY trigger that references the car node, an ACTION_MEDIA property set to MEDIA-PLAY and referencing an audio file that stores the engine idling sound, an ACTION_SET_HAPTIC action with an array Haptic_action_nodes containing a single element that references the car node, and an array media_indices containing a single element set to "0" to reference vib#0. The second behavior (Behavior#2) includes a PROXIMITY trigger that references a car node with a given distance range set to, for example, [0,2.0] meters, an ACTION_MEDIA property set to MEDIA-PLAY and referencing an audio file that stores engine revving sounds, an ACTION_SET_HAPTIC action with an array Haptic_action_nodes containing a single element that references the car node, and an array media_indices containing a single element set to "1" to reference vib#1. The third behavior (Behavior#3) includes a COLLISION trigger that references a car node, and an ACTION_SET_HAPTIC action with an array Haptic_action_nodes containing a single element that references the car node, along with an empty media_indices array and an array Material_Haptic_Modality containing a single element "Temperature".

[0127] Figure 12 shows an example of a scene description architecture including interactive haptic actions according to a third embodiment. In the second example described in Figure 11, which relates to the second embodiment, rendering two signals of the same modality separately is not possible if they are defined in the same media. For example, in the same use case as the previous example, if the haptic data is contained in a single haptic file with two separate perceptions, the rendering of both perceptions will be triggered simultaneously because the two perceptions are in the same media. The third embodiment allows specifying which perception should be rendered using the following configuration, as shown in the third example in Figure 12.

[0128] The corresponding interactive haptic experience can be defined using a scene description that includes a CAR#1 node, haptic media defining vib#0 for the vibration haptic perception of an idling engine and a second vib#1 for the engine revving, a haptic material associated with the car's temperature texture map, and three haptic behaviors. The first behavior (Behavior#1) includes an ACTION_SET_HAPTIC action with a VISIBILITY trigger that references the car node, an ACTION_MEDIA property set to MEDIA-PLAY and referencing an audio file that stores the engine idling sound, an array Haptic_action_nodes containing a single element that references the car node, an array media_indices containing a single element "0" to reference the haptic media file, and a perception_index set to "0" to reference vib#0. The second behavior (Behavior#2) includes a PROXIMITY trigger that references a car node with a given distance range set to, for example, [0,2.0] meters, an ACTION_MEDIA property set to MEDIA-PLAY and referencing an audio file that stores the engine sound, an ACTION_SET_HAPTIC action with an array Haptic_action_nodes containing a single element that references the car node, an array media_indices containing a single element "0" to reference the haptic media file, and a perception_index set to "1" to reference vib#1. The third behavior (Behavior#3) includes a COLLISION trigger that references a car node, an ACTION_SET_HAPTIC action with an array Haptic_action_nodes containing a single element that references the car node, along with an empty media_indices array and an array Material_Haptic_Modality containing a single element "Temperature".

[0129] An example of the fourth embodiment may be very similar to the third example, and is the same use case as the second and third examples, but can be used to go a step deeper into the data structure to address the selection of only specific channels of perception that include multiple channels. The corresponding diagram is very similar to Figure 12, with an additional "channel_indices" level in the definition of Haptic_action_node.

[0130] Figure 13 shows an example flowchart for handling interactive haptic actions according to at least one embodiment. Such a process 1300 is typically implemented in a haptic rendering device 100 and executed by a processor 101 of such a device. This process can be used in any of the embodiments described above. In step 1310, the processor obtains a description of the immersive scene (191 in Figure 1). This can be done, for example, by receiving it from a server over a communication network, by reading it from an external storage device or local memory, or by any other means. The processor analyzes the scene description file to extract interactive haptic actions and haptic effects, including at least the triggers defined, for example, in Tables 2 and 3. Examples of triggers include, for example, collisions between virtual objects and a representation of the user in the scene, approach from a virtual object, user input, or the visibility of a virtual object. In step 1320, the processor monitors the interaction between the user and the immersive scene to detect triggers, and when an expected trigger defined by an interactive haptic action is detected, in step 1330, it determines the parameters of the haptic effect defined by the element ACTION_SET_HAPTIC, for example, in Tables 18 to 29. Then, in step 1340, the corresponding haptic data is provided for rendering the haptic effect defined by the interactive haptic action.

[0131] When a device implementing process 1300 integrates a haptic actuator, haptic signals representing haptic effects are provided to the actuator. When a device implementing process 1300 uses a haptic rendering device that integrates a haptic actuator, haptic signals representing haptic effects are provided to the haptic rendering device.

[0132] As discussed above, a device that receives and decodes an immersive scene may delegate the rendering task to another device, such as a dedicated haptic rendering device, instead of performing the rendering itself. In this case, data for rendering the visual elements and / or haptic effects is prepared and sent to the device that performs the rendering. Such remote rendering may be used for audio data, video data, and haptic data and depends heavily on the capabilities built into the devices involved. In some cases, a combination of devices may be required to fully render the immersive experience. In other cases, a single device contains all the elements necessary to perform all tasks, including decoding and rendering. This is, for example, when a smartphone displays an augmented reality scene and provides vibrations as the user interacts with that scene.

[0133] Although different embodiments have been described separately, any combination of these embodiments can be carried out together, taking into account the principles of this disclosure.

[0134] While the embodiments relate to tactile effects, those skilled in the art will understand that the same principle can be applied to other effects, such as sensory effects, and therefore to smell and taste. Accordingly, appropriate syntax will determine appropriate parameters related to these effects.

[0135] Whenever the phrases "one embodiment" or "embodiment" or "one embodiment" or "embodiment," as well as any other variation thereof, refer to them, it means that certain features, structures, characteristics, etc., described in relation to that embodiment are included in at least one embodiment. Therefore, whenever the phrases "in one embodiment" or "in an embodiment" or "in one embodiment" or "in an embodiment," as well as any other variation, appear in various places throughout this specification, they do not all refer to the same embodiment.

[0136] In addition, this specification or its claims may refer to “determining” various pieces of information. Determining information may include, for example, one or more of the following: estimating information, calculating information, predicting information, or retrieving information from memory.

[0137] In addition, this specification or its claims may refer to “obtaining” various types of information. Obtaining, like accessing, is a broad term. Obtaining information may include, for example, receiving information, accessing information, or retrieving information (for example, from memory or optical media storage). Furthermore, “obtaining” is usually included in some way during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0138] For example, in the cases of "A / B", "A and / or B", and "at least one of A and B", please understand that the use of any of the following " / ", "and / or", and "at least one of" encompasses the selection of only the first listed option (A), or only the second listed option (B), or both options (A and B). As further examples, in the cases of "A, B, and / or C" and "at least one of A, B, and C", such expressions encompass the selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second options (A and B), or only the first and third options (A and C), or only the second and third options (B and C), or all three options (A, B, and C). This can be extended to the number of items listed, as will be readily apparent to those skilled in the art in this and related fields.

Claims

1. It is a method, Obtaining a data structure representing an interactive haptic action from an immersive scene description, wherein the data structure is associated with a virtual object or an immersive scene, and defines behavior based on triggers and actions, wherein the actions are associated with haptic effects. In response to an event corresponding to the trigger, the parameters of the tactile effect are determined, wherein the parameters each comprise at least the type of the tactile effect and a tactile signal for the tactile effect. To provide tactile data for rendering the tactile effect, based at least the type of the tactile effect and the tactile signal for the tactile effect, A method that includes [a certain feature].

2. The method according to claim 1, wherein the parameter of the tactile effect further comprises the identification of a tactile signal provided to a tactile actuator.

3. The method according to claim 1 or 2, wherein the parameters of the tactile effect further comprise a mask representing at least one part of the body to which the effect should be applied, and the data provided for rendering the tactile effect further comprises a selection of tactile actuators corresponding to the at least one part of the body to which the effect should be applied.

4. The method according to any one of claims 1 to 3, wherein the parameter of the tactile effect further comprises identifying the tactile texture, which determines the value of the tactile effect based on its position within the tactile texture.

5. The method according to any one of claims 1 to 4, wherein the parameters of the tactile effect further comprise a washout flag, and the data provided for rendering the tactile effect further comprises an instruction indicating that a selected tactile actuator should return to a neutral position.

6. The method according to any one of claims 1 to 5, wherein the parameter of the tactile effect further comprises control flags for controlling the rendering of the tactile data, the control flags being selected from a set of predetermined values ​​for at least starting, pausing, resuming, and stopping the rendering of the tactile data.

7. The method according to any one of claims 1 to 6, wherein the parameter of the tactile effect further comprises a flag for controlling whether the rendering of the tactile data should be in a continuous loop.

8. It is a device, From the immersive scene description, a data structure representing an interactive haptic action is obtained, the data structure is associated with a virtual object or immersive scene, defines behavior based on triggers and actions, and the actions are associated with haptic effects. In response to an event corresponding to the trigger, the parameters of the tactile object associated with the tactile effect are determined, and the parameters include at least the type of the tactile effect and a tactile signal for the tactile effect. The system provides tactile data for rendering the tactile effect, based at least the type of the tactile effect and the tactile signal for the tactile effect. A device equipped with a processor configured in such a way.

9. The apparatus according to claim 8, wherein the parameters of the tactile object further comprises the identification of tactile signals provided to a tactile actuator.

10. The apparatus according to claim 8 or 9, wherein the parameters of the tactile object further comprise a mask representing at least one part of a body to which the effect is to be applied, and the data provided for rendering the tactile effect further comprises a selection of tactile actuators corresponding to the at least one part of the body to which the effect is to be applied.

11. The apparatus according to any one of claims 8 to 10, wherein the parameters of the tactile object further comprises identifying the tactile texture, which determines the value of the tactile effect based on its position within the tactile texture.

12. The apparatus according to any one of claims 8 to 11, wherein the parameters of the tactile object further comprise a washout flag, and the data provided for rendering the tactile effect further comprises an instruction indicating that the selected tactile actuator should return to a neutral position.

13. The apparatus according to any one of claims 8 to 12, wherein the parameter of the tactile effect further comprises control flags for controlling the rendering of the tactile data, the control flags being selected from a set of predetermined values ​​for starting, pausing, resuming, or stopping the rendering of the tactile data.

14. The apparatus according to any one of claims 8 to 13, wherein the parameter of the tactile effect further comprises a flag for controlling whether the rendering of the tactile data should be in a continuous loop.

15. A computer program comprising, when executed by a processor, program code directives for implementing the method described in any one of claims 1 to 7.

16. A non-temporary computer-readable medium that stores program code instructions for implementing the method according to any one of claims 1 to 7 when executed by a processor.

Images