Hybrid haptic texture

Abstract

A data structure for immersive scene description includes information representative of a haptic effect based on a haptic texture and an additional information field used to determine how to interpret the haptic texture, allowing distinguishing whether a pixel directly represents a value of the haptic effect or whether a pixel references a haptic signal representative of the haptic effect. Additional information may also carry information to select bit depths and ranges for haptic attributes among a set of different settings.

Description

Technical Field

[0001] At least one of the present embodiments generally relates to immersive scene description, and more specifically, to haptic effects based on haptic textures. Background Art

[0002] A fully immersive user experience is provided to the user through an immersive system based on feedback and interaction. The interaction can use traditional control methods that meet the needs of the user. Current visual and auditory feedback provide a satisfying sense of realistic immersion. Haptic effects can provide additional feedback, which allows human users to perceive the virtual environment through other senses, thereby obtaining a better fully immersive experience with improved realism. However, haptics remain a potential area of ​​advancement for improving the overall user experience in immersive systems.

[0003] Traditionally, an immersive system may include a 3D scene representing a virtual environment, wherein virtual objects are located within the 3D scene. In order to improve user interaction with elements of the virtual environment, tactile feedback may be used through the stimulation of tactile actuators. This interaction is based on the concept of a "tactile object" corresponding to a physical phenomenon to be delivered to the user. In the context of an immersive scene, a tactile object allows providing tactile effects to mimic physical phenomena on a tactile rendering device by defining the stimulation of appropriate tactile actuators. Different types of tactile actuators allow the reproduction of different types of tactile feedback.

[0004] An example of a haptic object is an explosion. The explosion can be rendered with vibration and heat, thereby combining different haptic effects for the user to increase realism. An immersive scene typically includes multiple haptic objects, such as using a first haptic object associated with a global effect and a second haptic object associated with a local effect.

[0005] The principles described herein apply to any immersive environment that uses touch, such as augmented reality, virtual reality, mixed reality, or haptic-enhanced video (or omnidirectional / 360° video) rendering, and more generally to any haptic-based user experience. Therefore, the scenes used for such immersive environment examples are considered immersive scenes.

[0006] Haptics refers to the sensation of touch and includes two dimensions: tactile and kinesthetic. The first dimension is related to contact sensations, such as friction, roughness, hardness, temperature, and is felt through mechanoreceptors (Merkel cells, Ruffini endings, Meissner corpuscles, Pacinian corpuscles) and thermoreceptors in the skin. The second dimension is related to the sensation of force / torque, position, motion / velocity provided by mechanoreceptors in muscles, tendons, and joints. Haptics also involves the perception of self-motion, as it contributes to the proprioceptive system (i.e., the perception of one's own body). Therefore, the perception of acceleration, velocity, or any body model can be incorporated into the tactile effect. The frequency range is about 0-1KHz, depending on the modality type. Most existing devices capable of rendering tactile signals produce vibrations. Examples of such tactile actuators include linear resonant actuators (LRAs), eccentric rotating masses (ERMs), and voice coil linear motors. These actuators can be integrated into tactile rendering devices, such as haptic suits, or into smartphones or game controllers.

[0007] To encode haptic signals, several formats have been defined, related to high-level description using XML-like formats (e.g. MPEG-V), parametric representation using JSON-like formats (e.g. Apple Haptic Audio Pattern (AHAP) or Immersion Corporation's HAPT format), or waveform encoding (IEEE 1918.1.1 ongoing standardization of haptic and motion signals). The HAPT format has recently been incorporated into the MPEG ISOBMFF file format specification (ISO / IEC 14496 Part 12). In addition, the GL Transport Format (glTF) TM ) is a royalty-free specification for the efficient transfer and loading of 3D scenes and models by applications. This format defines an extensible, common publishing format for 3D content tools and services, streamlining authoring workflows and enabling interoperable use of content across the industry.

[0008] In addition, the MPEG standardization group is defining a new haptic file format that is related to encoded representations for haptics. The reference model for this format has not yet been published, but is referred to in this article as RM0. Using this reference model, the encoded haptic description files can be exported to a human-readable JSON interchange format (such as a .gmpg file) or a compressed binary distribution format (such as .mpg) that is particularly suitable for transmission to haptic rendering devices. The proposed format is glTF TM The format adds tactile functionality. Summary of the invention

[0009] Embodiments relate to a data structure for immersive scene description, the data structure including information representing a haptic effect, the information being based on a haptic texture and including an additional information field that determines how to interpret the haptic texture. This allows distinguishing between cases where a pixel directly represents a value of a haptic effect or where a pixel references a haptic signal representing the haptic effect. The additional information may also carry information to select a bit depth and range for a haptic attribute among a set of different settings.

[0010] A first aspect of at least one embodiment is directed to a method for decoding a tactile effect, the method comprising: obtaining information representing a tactile effect including a tactile texture and additional information; providing data of the tactile texture to a tactile actuator when the additional information corresponds to a first value; and selecting a tactile signal from a tactile signal set based on a pixel value of the texture, and providing the data of the selected tactile signal to the tactile actuator when the additional information corresponds to a second value.

[0011] A second aspect of at least one embodiment is directed to a device comprising a processor configured to: obtain information representing a tactile effect including a tactile texture and additional information; provide data of the tactile texture to a tactile actuator when the additional information corresponds to a first value; and select a tactile signal from a tactile signal set based on a pixel value of the texture when the additional information corresponds to a second value, and provide the data of the selected tactile signal to the tactile actuator.

[0012] A third aspect of at least one embodiment is directed to a non-transitory computer-readable medium comprising haptic data generated according to the first or second aspect.

[0013] A fourth aspect of at least one embodiment is directed to a computer program comprising program code instructions executable by a processor, the computer program at least implementing the steps of the method according to the first aspect.

[0014] A fifth aspect of at least one embodiment is directed to a computer program product stored on a non-transitory computer-readable medium and comprising program code instructions executable by a processor, the computer program product implementing at least the steps of the method according to the first aspect.

[0015] In a variation of the first and second methods, the first value of the additional information indicates that the texture is to be interpreted as a direct texture rendering, and wherein the data of the tactile texture is provided based on a position of an element representing the user relative to the texture.

[0016] In a variant of the first and second methods, the second value of the additional information indicates that the texture is to be interpreted as comprising a reference to the haptic signal, and wherein selecting the haptic signal is performed based on a position of the element representing the user relative to the texture.

[0017] In variations of the first and second methods, the additional information further indicates a bit depth of the texture, a range of the haptic effect, or a bit depth of the texture and a range of the haptic effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A block diagram of an example immersive system implementing various aspects and embodiments is shown.

[0019] Figure 2A An example of a tactile texture bump map according to the prior art is shown.

[0020] Figure 2B Indicates that it can be used to indicate Figure 2A One-dimensional (1D) signal of the tactile texture presented in .

[0021] Figure 2C Shows Figure 2A An example of an uncanny rendering scenario in context.

[0022] Figure 2D The use of SHO and SHT methods to render tactile textures is shown.

[0023] Figure 2E The principle of a collection of tactile pixels (taxels) providing a spatial solution for the SHT method is shown.

[0024] Figure 3 An example data structure of an immersive scene description according to at least one embodiment is shown.

[0025] Figure 4 An example of a three-dimensional (3D) object is shown in accordance with at least one embodiment.

[0026] Figure 5 A tactile texture used as a friction map for a bottle is shown.

[0027] Figure 6 An exemplary flow diagram of a process for rendering a haptic feedback description file in accordance with at least one embodiment is shown. DETAILED DESCRIPTION

[0028] Figure 1A block diagram of an example of an immersive system implementing various aspects and embodiments is shown. In the immersive system shown, user Alice uses a tactile rendering device 100 to interact with a server 180 hosting an immersive scene 190 through a communication network 170. The immersive scene 190 may include a variety of data and / or files representing different elements required for its rendering (scene description 191, audio data, video data, 3D model, and tactile description file 192). The immersive scene 190 can be generated under the control of an immersive experience editor 110, which allows different elements to be arranged together and an immersive experience to be designed. Appropriate description files and various data files representing the immersive experience are generated by an immersive scene generator 111 (also known as an encoder) and encoded in a format suitable for transmission to a tactile rendering device. The immersive experience editor 110 is typically executed on a computer that will generate the immersive scene to be hosted on the server. For simplicity, the immersive experience editor 110 is shown as being directly connected to the immersive scene 190 via a dotted line 171. In practice, immersive scene 190 is hosted on server 180 , and a computer running immersive experience editor 110 is connected to server 180 via communication network 170 .

[0029] The tactile rendering device 100 includes a processor 101. The processor 101 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor may perform data processing, such as tactile signal decoding, input / output processing, and / or any other functions that enable the device to operate in an immersive system.

[0030] The processor 101 may be coupled to an input unit 102 configured to communicate user interactions. Various types of inputs and modalities may be used for this purpose. A physical keyboard or a touch-sensitive surface is a typical example of an input suitable for this purpose, although voice control may also be used. In addition, the input unit may also include a digital camera capable of capturing static pictures or videos in a two-dimensional manner, or a more complex sensor capable of determining depth information in addition to pictures or videos and thus capable of capturing a complete 3D representation. The processor 101 may be coupled to a display unit 103 configured to output visual data to be displayed on a screen. Various types of displays may be used for this purpose, such as a liquid crystal display (LCD) or an organic light emitting diode (OLED) display unit. The processor 101 may also be coupled to an audio unit 104, which is configured to render sound data to be converted into audio waves by an adapted transducer (e.g., a speaker). The processor 101 may be coupled to a communication interface 105, which is configured to exchange data with an external device. The communication preferably uses a wireless communication standard (such as a cellular (e.g., LTE) communication, Wi-Fi communication, etc.) to provide mobility of the tactile rendering device. The processor 101 can access information in the memory 106 and store data in the memory 106, which can include multiple types of memory, including random access memory (RAM), read-only memory (ROM), hard disk, subscriber identity module (SIM) card, memory stick, secure digital (SD) memory card, any other type of storage device. In an embodiment, the processor 101 can access information in a memory that is not physically located on a device (e.g., located on a server, home computer, or other device) and store data in the memory.

[0031] The processor 101 is coupled to a haptic unit 107, which is configured to provide tactile feedback to a user, the tactile feedback being described in a tactile description file 192 associated with a scene description 191 of an immersive scene 190. The tactile description file 192 describes the type of feedback to be provided according to a syntax further described below. Such a description file is typically transmitted from a server 180 to a tactile rendering device 100. The tactile unit 107 may include a single tactile actuator or multiple tactile actuators located at multiple locations on the tactile rendering device. Different tactile units may have different numbers of actuators and / or the actuators may be positioned differently on the tactile rendering device.

[0032] In at least one embodiment, the processor 101 is configured to render a haptic signal according to the embodiments further described below, in other words, a low-level signal is applied to the haptic actuator to render a haptic effect. Such a low-level signal can be represented in different forms, such as by metadata or parameters in a description file, or by using a digital encoding of a sampled analog signal (e.g., PCM or LPCM).

[0033] The processor 101 may receive power from the power source 108 and may be configured to distribute the power to other components in the device 100 and / or control the power used for other components in the device 100. The power source may be any device suitable for powering the device. As an example, the power source may include one or more dry cells (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0034] Although the figure depicts the processor 101 and other elements 102 to 108 as separate components, it is understood that these elements can be integrated together to form an electronic package or chip. It is understood that the tactile rendering device 100 can include any sub-combination of the elements described herein while being consistent with the embodiments. The processor 101 can also be coupled to Figure 1 Other peripheral devices or units not depicted in the figure may include one or more software and / or hardware modules that provide additional features, functions and / or wired or wireless connectivity. For example, peripheral devices may include sensors such as universal serial bus (USB) ports, vibration devices, television transceivers, hands-free headsets, Module, FM radio unit, digital music player, media player, video game player module, Internet browser, etc. For example, the processor 101 can be coupled to a positioning unit configured to locate the tactile rendering device in its environment. The positioning unit can integrate a GPS chipset to provide a longitude and latitude position of the current location of the tactile rendering device, and can also integrate other motion sensors that provide positioning services, such as an accelerometer and / or an electronic compass.

[0035] Typical examples of the haptic rendering device 100 are a haptic suit, a smartphone, a game controller, a haptic glove, a haptic chair, a haptic prop, a motion platform, etc. However, any device or combination of devices providing similar functionality may be used as the haptic rendering device 100 while still complying with the principles of the present disclosure.

[0036] In at least one embodiment, the device does not include a display unit, but includes a haptic unit. In such an embodiment, the device does not render the scene visually, but only renders haptic effects. However, the device can prepare data for display so that another device (such as a screen) can perform the display. Examples of such devices are haptic suits or motion platforms.

[0037] In at least one embodiment, the device does not include a haptic unit, but includes a display unit. In such an embodiment, the device does not render haptic effects, but only renders the scene visually. However, the device can prepare data for rendering haptic effects so that another device (e.g., a haptic prop) can perform the haptic rendering. Examples of such devices are smartphones, head-mounted displays, or laptops.

[0038] In at least one embodiment, the device does not include a display unit and does not include a haptic unit. In such an embodiment, the device does not render the scene visually and does not render haptic effects. However, the device can prepare data for display so that another device (e.g., a screen) can perform the display, and can prepare data for rendering haptic effects so that another device (e.g., a haptic prop) can perform haptic rendering. Examples of such devices are computers, game consoles, optical media players, or set-top boxes.

[0039] In at least one embodiment, the immersive scene 190 and related elements are hosted directly in the memory 106 of the tactile rendering device 100, thereby allowing local rendering and interaction. In a variation of this embodiment, the device 100 also includes an immersive experience editor 110, thereby allowing completely independent operation, such as without any communication network 170 and server 180.

[0040] Although the different elements of the immersive scene 190 Figure 1 Although depicted as separate elements in the scene description, the principles described herein also apply to the case where these elements are directly integrated into the scene description rather than being separate elements. Any mix between the two alternatives is also possible, where some elements are integrated into the scene description and others are separate files.

[0041] To simplify the description, this article uses a finger touching a tactile surface as the interaction medium to describe the interaction and haptic effects. However, any other elements representing the user's position in the immersive environment (such as the user's body parts, the position provided by a force feedback device, the positioning of a head-mounted display in a virtual reality environment) can be used, still relying on the same principles.

[0042] Figure 2AAn example of a tactile texture bump map according to the prior art is shown. The proposed tactile file format allows the tactile texture information to be conveyed in a map that is an image containing tactile data rather than RGB values. Using textures to describe tactile properties allows the texture to be mapped to a 3D object using the capabilities of a 3D engine. Multiple tactile maps can be associated with a single object (friction, heat, hardness, etc.). Although these maps will be able to render tactile textures or tactile surfaces, they also bring specific problems. In fact, the tactile texture provides information at a given point in space. For example, this corresponds to the position where a finger touches a touch screen. Therefore, the tactile information is transmitted at the rate tracked by the user (finger), as shown in the figure.

[0043] Figure 2A An image 200 250 pixels wide is shown, in which three regions are associated with tactile feedback determined by a texture bump map. The bump map defines regions 201, 203, 205, 207 represented by white ("0" value) as holes, and regions 204, 204, 206 represented by black ("255" value) as bumps. Thus, this tactile texture allows a user sliding a finger 240 over the region to feel a series of bumps and pits when sliding from left to right. Haptic rendering can be performed by a vibration, electrical stimulation, or force feedback device connected to the screen. Figure 2B Indicates that it can be used to indicate Figure 2A A one-dimensional signal of the tactile texture rendered in .

[0044] For simplicity, Figure 2A In Figure 2, user tracking is set to 1Hz and the user moves across the image at 30 pixels per second (px / s). Elements 211 to 217 represent tracking (i.e., scanning, sampling) the user's finger across the image based on the finger movement and tracking rate. If the user moves faster, the expected feedback would be a faster succession of bumps and dimples. However, due to limitations of the tracking system, the sample points chosen on the texture may result in Figure 2C Exception rendering shown.

[0045] Figure 2C Shows Figure 2A An example of an unusual rendering scenario in context. In this example, the user moves a finger 250 across the image at 60px / s, much faster than in the previous image, so at the same scan rate, the user tracking only senses elements 221, 222, 223, and 224. In this case, the finger position is detected only on the portions of the texture representing the dimples 201, 203, 205, 207. Therefore, the tactile rendering will be uniform, as if the user had touched a completely flat (completely white) surface, although the black lines corresponding to the bumps have passed through.

[0046] This type of tactile rendering technique is called Surface Haptic Object (SHO) and usually relies on a discrete two-dimensional (2D) grayscale texture. For 1D textures, the principle remains the same. With this approach, the rendering of the tactile texture is based on the position of the finger on the texture and is therefore dependent on the hardware tracking rate.

[0047] To solve this problem, another approach called Surface Haptic Texture (SHT) can be used. It is based on using the velocity of the finger instead of its position. With this approach, the position of the finger is only used to re-estimate the velocity. Given the velocity, the rendering loop is no longer dependent on the tracking frequency and the tactile texture can be rendered at high frequencies with reasonable accuracy. This approach is more specifically conceived for one-dimensional periodic tactile textures such as Figure 2A and 2B This makes the solution extremely memory efficient since only a single cycle needs to be stored.

[0048] Figure 2D The figure shows the use of the SHO and SHT methods to render a tactile texture. The SHT method is limited in two ways. First, this type of signal limits the rendering to textures consisting of a single periodic element. Second, since the rendering depends only on the velocity, it does not take into account the initial finger position, which may cause signal offsets. The figure shows the use of Figure 2C 2 and 3. The input signal is shown and the same tactile texture is rendered using the SHO method 280 and the SHT method 290. The SHT begins rendering at the beginning of cycle 281, while the SHO fully renders the texture using the initial finger position 291. While this type of signal shift may not be noticeable for some high frequency tactile textures, it may be problematic for other textures.

[0049] Figure 2E The principle of a set of tactile pixels that provide a spatial solution for the SHT method is shown. The tactile pixels determine the shape area of ​​the texture associated with the tactile signal. For example, area 251 is associated with signal 261. When the user moves his finger over this area 251, he should feel the tactile effect defined by signal 261. In the rendering phase, the finger position is detected in the determined area and the corresponding effect is rendered according to the current user's speed using the SHT method. In other words, the playback rate of the tactile signal is determined by the interaction speed. For example, a higher speed will affect the playback of the tactile signal, resulting in signal 261 having a higher frequency and signal 262 having a steeper and shorter slope. This solution combines the advantages of the SHO and SHT methods by providing a spatial-based solution for rendering using speed information. This method can partially process 2D textures by using multiple 1D signals assigned to different directions (usually X and Y) (for example, transmitted through different tracks). However, this solution is only applicable to periodic signals.

[0050] The SHO, SHT, and tactile pixel-based methods described above are complementary and have their own advantages and disadvantages. The current format described below allows the use of all three rendering methods.

[0051] Figure 3 An example of a data structure for immersive scene description according to at least one embodiment is shown. This embodiment is based on glTF TM File format. glTF TM At its core is a JSON file that describes the structure and composition of a scene containing a 3D model. The diagram shows the relationship between the elements that make up the data structure of an immersive scene description 300. In this case, scene 301 is the top-level element that brings together all other elements. It contains an array of nodes. Each node 302 can contain child nodes, allowing a hierarchy to be created. A node can refer to a mesh or a camera or a skin, and a local geometric transformation can be associated with a node. Mesh 310 corresponds to the geometric data required to render the mesh. Skin 320 is used to perform vertex skinning so that the vertices of the mesh are affected by the skeleton bones according to their pose. Camera 325 determines the projection matrix. Light source 315 determines the lighting properties associated with the node. Buffer 355 contains data for 3D model geometry, animation, and skinning. BufferViews 350 adds structural information to the buffer data, while accessors 345 define the exact type and layout of BufferViews. Material 360 determines how an object should be rendered based on physical material properties. Texture 365 allows the appearance of an object to be defined. Image 370 defines the image data for the texture, while sampler 380 describes the wrapping and scaling of the texture.

[0052] The immersive scene description file also includes a haptic object 330 that describes the haptic effect to be rendered. A haptic object identified as "MPEG_Haptic" in the file format can be associated with a haptic texture map 335 identified as "MPEG_material_haptic" in the file format syntax described below. The data for the haptic texture map can be stored with the regular texture 365. Figure 2E The tactile pixels introduced in can be implemented by using a tactile texture that references a collection of tactile signals, so that different areas of the texture can be associated with different tactile effects. For example, a tactile texture attached to a node can also be rendered directly using the SHT method.

[0053] glTF TM These elements of the file allow for immersive scenes with tactile feedback.

[0054] Table 1 describes the Figure 3The MPEG_haptic extension corresponding to the element 330 of . It consists of a haptic media source (any haptic media file can be used, such as Figure 2E This extension is attached to a node and can be used, for example, to trigger a haptic effect based on a user interaction with the node.

[0055]

[0056] Table 1: MPEG_haptic extension

[0057] Table 2 describes the Figure 3 3D object. The haptic texture map is used to describe the haptic properties of a node. For example, a temperature texture map can be used to determine the temperature to be rendered in a subset of the object's surfaces (e.g., metal parts are cooler than plastic parts), and a stiffness texture map can be used to indicate that a subset of surfaces corresponding to metal parts of the object are rigid, while a subset of surfaces corresponding to plastic parts (i.e., rubber) are soft. A haptic texture map allows different parameters to be defined for different haptic properties of specific areas of a 3D object.

[0058]

[0059]

[0060] Table 2: MPEG_material_haptic extension

[0061] Table 3 describes the bit depths and value ranges of different tactile textures.

[0062] Tactile Map Format scope Resolution Stiffness 8-bit <![CDATA[0-10000N.s -1 / m.s -1 ]]> <![CDATA[40N.s -1 / m.s -1 ]]> Friction 8-bit ±5 0.04 Vibrotactile Texture 8-bit ±10 0.08mm Temperature 8-bit [-50:+75]℃ 0.5℃ Vibration 8-bit 0-255 1

[0063] Table 3: Bit depth and value range for tactile textures

[0064] However, this format does not allow to identify the type of texture being used, i.e. how it should be rendered. Therefore, the tactile rendering device cannot interpret the texture appropriately. Furthermore, the bit depth and value range of each texture type are defined as shown in Table 3. Therefore, these parameters are constant, e.g. predetermined in a common specification for interoperability purposes, and cannot be adapted to different situations.

[0065] The embodiment described below is designed in consideration of the above situation and proposes to introduce additional information fields to identify Figure 3The rendering type related to the tactile texture in the data structure of , thereby allowing to distinguish between the traditional tactile texture whose pixels directly represent the tactile effect value and the tactile texture whose texture pixels reference the corresponding tactile signal. This embodiment allows the use of a common representation to describe the tactile texture by different methods. It overcomes the limitations of different schemes, thereby providing better flexibility.

[0066] The first embodiment uses a Boolean value to distinguish the type of rendering associated with the texture in the data structure. In the second embodiment, the additional field allows for more precise specification of how the texture is interpreted. Typically, a haptic rendering system may conform to a common specification for interoperability that may define several sets of bit depths and ranges for haptic attributes, and the additional field would specify the configuration to be used. This embodiment addresses the issue with the reference above by allowing the use of textures with different representations or resolutions. Figure 2C In a third embodiment, it is proposed to use a texture array. This allows providing different configurations and resolutions (different types or different bit depths and ranges) for the same texture, thereby allowing the tactile rendering device to select the most appropriate texture according to the device capabilities.

[0067] These embodiments provide interoperability of haptic devices and authoring tools, allow the rendering of haptic textures to be adjusted according to the performance of the haptic rendering device, and are compatible with existing haptic texture representations and existing haptic rendering methods.

[0068] According to a first embodiment, Boolean information is associated with a texture and determines how the texture is interpreted. When this Boolean information is true, the associated texture should be interpreted as a reference to a haptic signal, so that each pixel value of the texture corresponds to an index in the Media Reference array of the MPEG_Haptic extension, thereby allowing the haptic signal of the haptic object to be obtained. Then the haptic signal can be obtained according to Figure 2D and 2E Otherwise, the texture is used as a traditional 2D haptic texture, so each pixel value of the texture directly corresponds to the value used to render the haptic effect.

[0069] The first embodiment describes the immersive scene ( Figure 3 300), the scene description includes haptic effects, which uses the MPEG_material_haptic description element in Table 4 that conforms to the relevant JSON schema in Table 5.

[0070]

[0071]

[0072] Table 4: First embodiment of MPEG_material_haptic description

[0073]

[0074]

[0075]

[0076]

[0077] Table 5: JSON schema of the first embodiment

[0078] According to a second embodiment, enumeration information is associated with a texture and determines how the texture is interpreted. The information specifies whether the haptic texture uses a traditional 2D haptic texture or is used to reference a haptic signal in a haptic signal set and also specifies the bit depth and value range of these haptic textures.

[0079] Two variants of this embodiment are proposed: the enumeration information can be carried as an integer or a string. In fact, these are two solutions for specifying enumeration information using the glTF format. In the proposed implementation, the variant of the second embodiment uses the same enumeration information for all tactile attributes, but the enumeration information may also be different for each type of attribute (for example, with additional bit depth and value range configurations).

[0080] A first variant of the second embodiment uses a string to describe how to interpret the tactile texture. For example, this variant embodiment is implemented in an immersive scene description that includes a tactile effect, which uses the elements of the MPEG_material_haptic description in Table 6 that conform to the relevant JSON schema in Table 7. In these tables, enumeration information is added to each tactile attribute. If set to "Reference", the associated texture should be interpreted as carrying a reference to the tactile signal to be rendered, where each pixel value corresponds to an index in the Media Reference array of the MPEG_Haptic extension. The tactile signal can then be obtained from the Media Reference. In at least one embodiment, the tactile signal obtained using the media reference is, for example, based on Figure 2E If set to "High_Resolution", the texture can be used as a traditional 2D texture with the associated bit depth and value range detailed in the specification.

[0081]

[0082]

[0083] Table 6: First variant of the second embodiment of the MPEG_material_haptic extension

[0084]

[0085]

[0086]

[0087] Table 7: JSON schema for the first variant of the second embodiment

[0088] Table 8 gives an example of a texture profile specification where the additional information is set to "Low_Resolution". In this case, the texture can be used as a traditional 2D texture, and its associated bit depth and value range will be lower than the "High_Resolution" version, and these values ​​are detailed in the specification.

[0089] Tactile Map Format scope Resolution Stiffness 8-bit <![CDATA[0-10000N.s -1 / m.s -1 ]]> <![CDATA[40N.s -1 / m.s -1 ]]> Friction 8-bit ±5 0.04 Vibrotactile Texture 8-bit ±10 0.08mm Temperature 8-bit [-50:+75]℃ 0.5℃ Custom 8-bit 0-255 1

[0090] Table 8: Bit depth and value range for low-resolution tactile textures

[0091] In addition, Table 9 gives another example of a texture profile specification, where the additional information is set to "High_Resolution".

[0092]

[0093] Table 9: Bit depth and value range for high-resolution tactile textures

[0094] Although Tables 8 and 9 regroup information related to bit depth and range, two different tables may be used for this purpose.

[0095] If the additional information is set to "Other", the texture can be used as a traditional 2D texture, where the bit depth and value range are not standard and must be provided to the haptic rendering device. This embodiment can be easily extended to support future types of haptic textures by adding more enumeration types.

[0096] The second variant of the second embodiment is similar to the first variant, except that the attribute uses an integer value instead of a string value to describe how to interpret the tactile texture. The following enumeration shows the correlation with the previous embodiment:

[0097] enum{High_Resolution=0,Low_Resolution=1,Reference=2,Other=3}

[0098] This implementation can be easily extended to support future types of tactile textures by adding more enumeration types. For example, the second variant of the second embodiment is implemented using the MPEG_material_haptic description element in Table 10 that conforms to the relevant JSON schema of Table 11, relying on the same texture configuration file as the first variant of the second embodiment (Tables 8 and 9).

[0099]

[0100] Table 10: Second variant of the second embodiment of the MPEG_material_haptic extension

[0101]

[0102]

[0103]

[0104]

[0105]

[0106]

[0107]

[0108] Table 11: JSON schema for the second variant of the second embodiment

[0109] According to a third embodiment, a texture array is used to determine how to interpret the tactile texture. With this embodiment, multiple textures, possibly with different types of tactile textures, can be specified for a single attribute and have the rendering device select the appropriate representation. This last embodiment allows the creation of a tactile experience that is compatible with different devices that provide different capabilities.

[0110] This last example is illustrated using a string enumeration, but other implementations may use an integer enumeration or a boolean value to simply differentiate between reference-based representations and traditional 2D textures. The specification for this example is in

[0111] The details are given in Table 12, and the associated JSON schema is given in Table 13.

[0112]

[0113] Table 12: Third embodiment of MPEG_material_haptic extension

[0114]

[0115]

[0116]

[0117]

[0118]

[0119] Table 13: JSON schema of the third embodiment

[0120] This implementation uses one array for each haptic attribute and one array for each associated texture type. A variant implementation uses a single array containing pairs of texture and type. The JSON schema for this variant is given in Table 14.

[0121]

[0122]

[0123]

[0124]

[0125]

[0126] Table 14: JSON schema for variant implementation of the third embodiment

[0127] When the tactile material attributes contain multiple textures with different data representation types (i.e., High Resolution, LowResolution, Reference, and Other), it is up to the tactile rendering device to decide which texture to use. For example, if the Stiffness attribute contains both a High_Resolution texture and a Low_Resolution texture, the tactile rendering device can decide which texture to use based on the performance of the rendering device. If the resolution of the rendering device is lower than the resolution defined in Table 8, the Low_Resolution texture can be used. Otherwise, if there is no information about the device performance, the tactile rendering device can use the first texture in the array as the default texture.

[0128] Figure 4 An example of a 3D object according to at least one embodiment is shown. The representation of the 3D object 400 includes a metal bottle 410 with a rubber protection 420. The object is Figure 3 The data structure of the bottle is specified in a formatted glTF file. Additional files are used to define different elements of the bottle: a binary file for the mesh, multiple texture files for different elements such as color, lighting, normals, occlusion, etc. Physically based rendering uses the data from these files to reconstruct a realistic visual effect of the bottle.

[0129] Furthermore, according to one of the above embodiments, additional textures for tactile sensation are provided to enhance the immersive experience associated with the bottle.

[0130] Figure 5 A haptic texture used as a friction map for a bottle is shown. With such a haptic texture, the portion of the bottle corresponding to the rubber protection (represented by the white area 501 at the bottom) generates more friction than the metal portion (represented by the diamond-shaped hash pattern 502, 503, 504, 505, 506, 507, 508). A haptic file (MyHapticFile.gmpg) can also be added to the scene for various interactions or for reference by the haptic texture. The following table shows the glTF syntax for describing a 3D bottle according to different embodiments.

[0131] Table 15 shows a glTF description for a 3D bottle according to a first embodiment, where the additional information is based on Boolean information. This Boolean information is inserted into the MPEG_material_haptic section. In the example of Table 15, the Boolean information is false, so each pixel value of the texture directly corresponds to the value of the haptic effect. The haptic effect is related to friction, as specified by the friction parameter of the MPEG_material_haptic section. The index is specified as 7 (the "index" parameter of the MPEG_material_haptic section), so the texture associated with this effect is the WaterBottle_friction.png file.

[0132]

[0133]

[0134]

[0135]

[0136]

[0137]

[0138]

[0139]

[0140]

[0141] Table 15: glTF description for a 3D bottle according to the third embodiment

[0142] For other embodiments, the core of the file is the same, only the MPEG_material_haptic part of the glTF description is different, as shown in Tables 16 to 19 below.

[0143] Table 16 shows the MPEG_material_haptic section of the glTF description for a 3D bottle according to the first variant of the second embodiment, in which a string is used as enumeration information to describe how to interpret the tactile texture. In this example, the string indicates High_Resolution, so that the bit depth and value range of the high-resolution tactile texture defined in Table 9 are used to render the tactile effect.

[0144]

[0145] Table 16: Example of the first variant of the second embodiment

[0146] Table 17 shows the MPEG_material_haptic part of the glTF description for a 3D bottle according to the second variant of the second embodiment, in which an integer is used as enumeration information to describe how to interpret the tactile texture. In this example, the integer represents 0, corresponding to High_Resolution listed in the enumeration below Table 9. Therefore, the bit depth and value range for the high-resolution tactile texture defined in Table 9 are used to render the tactile effect.

[0147]

[0148] Table 17: Example of the second variation of the second embodiment

[0149] Table 18 shows the MPEG_material_haptic portion of the glTF description for a 3D bottle according to the third embodiment, where a texture array based on string information is used. In this example, the friction haptic effect uses a high-resolution 2D texture.

[0150]

[0151] Table 18: Example of the first variant of the third embodiment

[0152] Table 19 shows the MPEG_material_haptic portion of the glTF description for a 3D bottle according to the second variant of the third embodiment, where a single array containing pairs of texture and type is used. In this example, the friction haptic effect uses a high-resolution 2D texture.

[0153]

[0154] Table 19: Example of the second variation of the third embodiment

[0155] Figure 6 An exemplary flow chart of a process for rendering a haptic feedback description file according to at least one embodiment is shown. Such a process 600 is typically implemented in a haptic rendering device 100 and executed by a processor 101 of the device.

[0156] In step 601, the processor obtains a description of the immersive scene ( Figure 1 Of these, 191 Figure 3 This can be done, for example, by receiving it from a server via a communication network, by reading it from an external storage device or local memory, or by any other means. The processor analyzes the scene description file to extract the tactile objects ( Figure 1 192), thereby allowing determination of parameters related to the tactile effect, more specifically including a tactile volume related to the tactile effect and additional information related to the tactile texture.

[0157] In step 602, the processor monitors the user's position within the immersive scene during interaction to detect intersections (object collisions) with the haptic volume. Collision detection may be performed by a dedicated physics engine specialized for this task.

[0158] When such an intersection is detected, additional information related to the tactile texture is checked in step 603. As described above, this information allows the tactile rendering device to determine how to interpret (and therefore render) the tactile texture.

[0159] In the first case according to the check of step 603, the additional information indicates that the texture is to be interpreted as representing values ​​for a haptic effect, i.e. conventional direct texture rendering. Therefore, in step 605, the processor provides data of the haptic texture to the haptic actuator according to the position of the user relative to the texture.

[0160] In a second case according to the check of step 603, the additional information indicates that the texture is to be interpreted as representing a reference to a haptic signal. In this case, in step 606, the processor selects from the list of haptic signals a haptic signal referenced by a pixel value of the texture, the pixel being determined according to the position of the user. For example, if the pixel value is "0", the first signal in the list will be selected.

[0161] In step 607, the processor provides data of the selected tactile signal to the tactile actuator. In this case, the tactile signal, for example, indicates that the tactile actuator is to be activated based on the tactile signal. Figures 2A to 2E The speed control signal can be rendered by any method in . Other types of haptic signals, such as haptic signals that are variable over time, can be referenced based on the same technology.

[0162] Therefore, the haptic effect is rendered according to the additional information of the haptic feedback.

[0163] As mentioned above, the device that receives and decodes the immersive scene may not perform the rendering itself, but instead delegate this task to another device, such as a dedicated haptic rendering device. In this case, the data used to render the visual elements and / or haptic effects is prepared and transmitted to the device that performs the rendering. This remote rendering can be used for audio, video, and haptic data, and is highly dependent on the built-in capabilities of the devices involved. In some cases, a combination of devices may be required to fully render the immersive experience. In other cases, the device includes all the elements required to perform all tasks, including decoding and rendering. This is the case, for example, when a smartphone displays an augmented reality scene and provides vibrations when the user interacts with the scene.

[0164] Although different embodiments have been described separately, the embodiments may be combined together arbitrarily while respecting the principles of the present disclosure.

[0165] Although the embodiments relate to haptic effects, those skilled in the art will appreciate that the same principles can be applied to other effects, such as sensory effects, thus including smell and taste. Therefore, appropriate syntax will determine the appropriate parameters associated with these effects.

[0166] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" and other variations thereof mean that a particular feature, structure, characteristic, etc. associated with the embodiment is included in at least one embodiment. Thus, the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations appearing in various places throughout the specification do not necessarily all refer to the same embodiment.

[0167] Furthermore, the application or its claims may involve "determining" various types of information. Determining information may include one or more of the following, for example, estimating information, calculating information, predicting information, or retrieving information from a memory.

[0168] Furthermore, this application or its claims may involve "obtaining" a variety of information. Obtaining, like "accessing," is intended to be a broad term. Obtaining information may include one or more of the steps of, for example, receiving information, accessing information, or retrieving information (e.g., retrieving information from a memory or optical media storage). Furthermore, "obtaining" generally involves in some way during an operation such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0169] It should be understood that the use of any of the following " / ", "and / or", and "at least one", such as in the case of "A / B", "A and / or B", and "at least one of A and B", is intended to cover selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C", such wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). It will be readily apparent to one of ordinary skill in this and related arts that this can be extended to as many items as listed.

Claims

1. A method for decoding a tactile effect, comprising: obtaining information representing the tactile effect including a tactile texture and additional information; When the additional information corresponds to a first value, providing data of the tactile texture to a tactile actuator; and When the additional information corresponds to a second value, a haptic signal is selected from a haptic signal set based on a pixel value of the haptic texture, and data of the selected haptic signal is provided to the haptic actuator. 2 . The method of claim 1 , wherein the first value of the additional information indicates that the tactile texture is interpreted as a direct texture rendering, and wherein data of the tactile texture is provided based on a position of an element representing a user relative to the tactile texture.

3. The method according to claim 1 or 2, wherein the second value of the additional information indicates that the tactile texture is interpreted as comprising a reference to a tactile signal, and wherein selecting the tactile signal is performed based on a position of an element representing a user relative to the tactile texture. The method of claim 3 , wherein the haptic signal is rendered based on a velocity of an element representing the user.

5. The method according to any one of claims 1 to 4, wherein the additional information is a Boolean value. 6 . The method according to claim 5 , wherein the first value of the additional information is false (FALSE), and the second value of the additional information is true (TRUE).

7. The method according to any one of claims 1 to 6, wherein the additional information is an enumeration value encoded as an integer value or a string value. The method of claim 7 , wherein the enumerated value further determines a bit depth of the tactile texture.

9. The method of claim 7 or 8, wherein the enumerated value further determines a range of the haptic effect.

10. The method according to any one of claims 1 to 9, further comprising a set of tactile textures and related additional information.

11. The method of any one of claims 1 to 10, further comprising selecting a texture resolution among a plurality of texture resolutions.

12. An apparatus for decoding a haptic effect, comprising a processor, the processor being configured to: obtaining information representing the tactile effect including a tactile texture and additional information; When the additional information corresponds to a first value, providing data of the tactile texture to a tactile actuator, and When the additional information corresponds to a second value, a haptic signal is selected from a haptic signal set based on a pixel value of the haptic texture, and data of the selected haptic signal is provided to the haptic actuator. 13 . The apparatus of claim 12 , wherein the first value of the additional information indicates that the tactile texture is interpreted as direct texture rendering, and wherein data of the tactile texture is provided based on a position of an element representing a user relative to the tactile texture.

14. An apparatus according to claim 12 or 13, wherein the second value of the additional information indicates that the texture is to be interpreted as comprising a reference to a haptic signal, and wherein selecting the haptic signal is performed based on a position of an element representing the user relative to the texture.

15. The apparatus of claim 14, wherein the haptic signal is rendered based on a velocity of an element representing the user.

16. The apparatus according to any one of claims 12 to 15, wherein the additional information is a Boolean value. 17 . The apparatus according to claim 16 , wherein the first value of the additional information is false, and the second value of the additional information is true.

18. The apparatus according to any one of claims 12 to 17, wherein the additional information is an enumeration value encoded as an integer value or a string value. The device of claim 18 , wherein the enumerated value further determines a bit depth of the tactile texture.

20. The apparatus of claim 18 or 19, wherein the enumerated value further determines a range of the haptic effect.

21. The apparatus of any one of claims 12 to 20, further comprising a set of tactile textures and associated additional information.

22. The method of any one of claims 12 to 21, further comprising selecting a texture resolution among a plurality of texture resolutions.

23. A signal comprising information representing a haptic effect, the information comprising a haptic texture and additional information indicating whether the haptic texture is interpreted as a direct texture rendering or a reference to a haptic signal.

24. A non-transitory computer-readable medium comprising encoded data, the encoded data comprising information representing a haptic effect, the information comprising a haptic texture and additional information indicating whether the haptic texture is interpreted as a direct texture rendering or a reference to a haptic signal.

25. A computer program comprising program code instructions for implementing the method according to any one of claims 1 to 11 when executed by a processor.

26. A non-transitory computer readable medium comprising program code instructions for implementing the method according to any one of claims 1 to 11 when executed by a processor.

Images