Multi-level texture mapping of haptic textures

Abstract

A data structure for immersive scene description includes information representing a haptic effect based on a haptic multi-level texture map, the haptic multi-level texture map including a plurality of haptic textures. At the time of rendering, a haptic texture is selected from the plurality of haptic textures based at least on the speed of the user interaction: when the user interacts at a low speed, a texture with fine details is selected to render the haptic feedback, and when the user interacts faster, a texture with less details is selected. The haptic texture may also be selected based on a tracking rate of the rendering device. This selection of haptic textures may ensure a satisfactory rendering of haptic effects.

Description

Technical Field

[0001] At least one of the present embodiments relates generally to immersive scene description, and more particularly to haptic effects using mipmap based haptic textures. Background Art

[0002] A fully immersive user experience is presented to the user through an immersive system based on feedback and interaction. The interaction may use conventional control means that meet the needs of the user. Current visual and auditory feedback provides a satisfactory level of realistic immersion. Additional feedback may be provided through haptic effects that allow the human user to perceive the virtual environment with his senses, thereby obtaining a better fully immersive experience with improved realism. However, in immersive systems, haptics remain a potential area of ​​advancement for improving the overall user experience.

[0003] Traditionally, an immersive system may include a 3D scene representing a virtual environment, wherein virtual objects are located within the 3D scene. In order to improve the user's interaction with the elements of the virtual environment, tactile feedback may be used through the excitation of tactile actuators. This interaction is based on the concept of a "tactile object", which corresponds to a physical phenomenon to be transmitted to the user. In the case of an immersive scene, the tactile objects allow to provide tactile effects by defining the excitation of appropriate tactile actuators to imitate physical phenomena on a tactile rendering device. Different types of tactile actuators allow to restore different types of tactile feedback.

[0004] An example of a haptic object is an explosion. An explosion can be presented through vibration and heat, thereby combining different haptic effects to the user to increase realism. An immersive scene often includes multiple haptic objects, such as using a first haptic object associated with a global effect and a second haptic object associated with a local effect.

[0005] The principles described here apply to any immersive environment that uses touch, such as augmented reality, virtual reality, mixed reality, or haptic-enhanced video (or omnidirectional / 360° video) rendering, and more generally to any haptic-based user experience. This example scenario of an immersive environment is therefore considered an immersive scenario.

[0006] Haptics refers to the sense of touch and includes two dimensions, tactile and kinesthetic. The first one involves tactile sensations such as friction, roughness, hardness, temperature, and is felt through the mechanoreceptors of the skin (Merkel cells, Ruffini endings, Meissner corpuscles, Pacinian corpuscles) and thermoreceptors. The second one is related to the sense of force / torque, position, motion / velocity provided by mechanoreceptors in muscles, tendons and joints. Haptics also involves the perception of self-motion, as it contributes to the proprioceptive system (i.e., the perception of one's own body). Therefore, the perception of acceleration, velocity or any body model can be assimilated into a tactile effect. The frequency range is about 0-1kHz, depending on the type of modality. Most existing devices capable of rendering tactile signals generate vibrations. Examples of such tactile actuators are linear resonant actuators (LRAs), eccentric rotating masses (ERMs), and voice coil linear motors. These actuators can be integrated into tactile rendering devices, such as haptic suits as well as smartphones or game controllers.

[0007] To encode haptic signals, several formats have been defined involving high-level descriptions using XML-like formats (e.g., MPEG-V), parametric representations using json-like formats (such as Apple Haptic Audio Profile (AHAP) or Immersive Technologies' HAPT format), or waveform encoding (ongoing standardization of IEEE 1918.1.1 for haptic and kinesthetic signals). The HAPT format has recently been included in the MPEG ISOBMFF file format specification (ISO / IEC 14496 Part 12). In addition, the GL Transport Format (glTF) TM ) is a royalty-free specification for the efficient transfer and loading of 3D scenes and models by applications. This format defines an extensible, common publishing format for 3D content tools and services, which simplifies the authoring workflow and enables interoperable use of content across industries.

[0008] In addition, a new haptic file format is being defined in the MPEG standardization group and involves an encoded representation of haptics. A reference model for this format has not yet been published, but is referred to here as RM0. Using this reference model, the encoded haptic description file can be exported as a human-readable JSON interchange format (e.g., a .gmpg file) or as a compressed binary distribution format (e.g., .mpg) that is particularly suitable for transmission to a haptic rendering device. The proposed format is a traditional glTF TM The format adds tactile capabilities. Summary of the invention

[0009] Embodiments relate to a data structure for immersive scene description, the data structure comprising information representing a tactile effect based on a tactile multi-level texture map comprising a plurality of tactile textures. At rendering time, a tactile texture is selected from the plurality of tactile textures based on the speed of the user interaction: when the user interacts at a slow speed, a texture with fine details is selected to render the tactile feedback, and when the user interacts faster, a texture with much less details is selected. The tactile texture may also be selected based on the tracking rate of the rendering device. This selection of the tactile texture ensures satisfactory and reliable rendering of the tactile effect.

[0010] A first aspect of at least one embodiment relates to a method for decoding a tactile object, the method comprising: obtaining information representing a tactile effect, comprising a tactile multi-level texture mapping, the tactile multi-level texture mapping comprising a plurality of tactile textures, each tactile texture being associated with a corresponding parameter representing an interaction speed; obtaining an interaction speed of an element representing a user; selecting a tactile texture based at least on the obtained interaction speed; and providing data of the selected tactile texture to a tactile actuator based on a position of the element representing the user relative to the tactile texture.

[0011] A second aspect of at least one embodiment relates to a device for decoding a tactile object, the device comprising a processor configured to obtain information representing a tactile effect, comprising a tactile multi-level texture map, the tactile multi-level texture map comprising a plurality of tactile textures, each tactile texture being associated with a corresponding parameter representing an interaction speed; obtaining an interaction speed of an element representing a user; selecting a tactile texture based at least on the obtained interaction speed; and providing data of the selected tactile texture to a tactile actuator based on a position of the element representing the user relative to the tactile texture.

[0012] A third aspect of at least one embodiment relates to a non-transitory computer-readable medium comprising information representing a haptic effect, comprising a haptic multi-level texture map including a plurality of haptic textures associated with a parameter representing an interaction speed.

[0013] A fourth aspect of at least one embodiment relates to a computer program comprising program code instructions executable by a processor, the computer program implementing at least the steps of the method according to the first aspect.

[0014] A fifth aspect of at least one embodiment relates to a computer program product stored on a non-transitory computer-readable medium and comprising program code instructions executable by a processor, the computer program product implementing at least the steps of the method according to the first aspect.

[0015] In variations of the first and second methods, the parameters associated with the tactile texture further represent tracking, and wherein the selection of the tactile texture is further based on a tracking rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A block diagram of an example of an immersive system in which various aspects and embodiments are implemented is shown.

[0017] Figure 2 An example of a data structure for an immersive scene description according to at least one embodiment is shown.

[0018] Figure 3A An example of a tactile texture bump map according to the prior art is shown.

[0019] Figure 3B It can be used to indicate Figure 3A 1D signal of the tactile texture presented in .

[0020] Figure 3C Shown in Figure 3A An example of an uncanny rendering scene in the context of .

[0021] Figure 3D Rendering of tactile textures with the SHO and SHT methods is shown.

[0022] Figure 3E The principle of a set of tactile pixels (taxels) is shown, which provides a spatial approach to the SHT method.

[0023] Figure 3F An example of multi-level texture mapping is shown.

[0024] Figure 4 An example of rendering of a tactile multi-level texture map is shown in accordance with at least one embodiment.

[0025] Figure 5 An example of tactile multi-level texture map generation according to a manual generation method is shown.

[0026] Figure 6 A tactile multi-level texture map generation process is shown in accordance with at least one embodiment.

[0027] Figure 7 Shown in Figure 6 Data generated during the tactile multi-level texture map generation process.

[0028] Figure 8 An example flow diagram of a process for rendering a haptic feedback description file in accordance with at least one embodiment is shown. DETAILED DESCRIPTION

[0029] Figure 1A block diagram of an example of an immersive system in which various aspects and embodiments are implemented is shown. In the depicted immersive system, user Alice uses a tactile rendering device 100 to interact with a server 180 hosting an immersive scene 190 through a communication network 170. The immersive scene 190 may include various data and / or files representing different elements required for its rendering (scene description 191, audio data, video data, 3D model, and tactile description file 192). The immersive scene 190 can be generated under the control of an immersive experience editor 110, which allows different elements to be arranged together and an immersive experience to be designed. Appropriate description files and various data files representing the immersive experience are generated by an immersive scene generator 111 (i.e., an encoder) and encoded in a format suitable for transmission to a tactile rendering device. The immersive experience editor 110 is typically executed on a computer that will generate an immersive scene to be hosted on a server. For simplicity, the immersive experience editor 110 is shown as being directly connected to the immersive scene 190 via a dotted line 171. In practice, immersive scene 190 is hosted on server 180 , and a computer running immersive experience editor 110 is connected to server 180 via communication network 170 .

[0030] The tactile rendering device 100 includes a processor 101. The processor 101 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor may perform data processing, such as tactile signal decoding, input / output processing, and / or any other function that enables the device to operate in an immersive system.

[0031] The processor 101 may be coupled to an input unit 102 configured to communicate user interactions. For this purpose, multiple types of inputs and modalities may be used. A physical keyboard or a touch-sensitive surface is a typical example of an input suitable for this purpose, although voice control may also be used. In addition, the input unit may also include a digital camera capable of capturing still pictures or videos in two dimensions, or a more complex sensor capable of determining depth information attached to a picture or video and thus capable of capturing a complete 3D representation. The processor 101 may be coupled to a display unit 103, which is configured to output visual data to be displayed on a screen. Multiple types of displays may be used for this purpose, such as a liquid crystal display (LCD) or an organic light emitting diode (OLED) display unit. The processor 101 may also be coupled to an audio unit 104, which is configured to render sound data, which will be converted into audio waves by an adapted transducer (such as a speaker). The processor 101 may be coupled to a communication interface 105 configured to exchange data with an external device. The communication preferably uses a wireless communication standard to provide mobility of the tactile rendering device, such as cellular (e.g., LTE) communication, Wi-Fi communication, etc. The processor 101 can access information from and store data in the memory 106, which can include a variety of types of memory, including random access memory (RAM), read-only memory (ROM), hard disk, subscriber identity module (SIM) card, memory stick, secure digital (SD) memory card, any other type of memory storage device. In an embodiment, the processor 101 can access information from and store data in a memory that is not physically located on the device (such as on a server, a home computer, or another device).

[0032] The processor 101 is coupled to a haptic unit 107 configured to provide tactile feedback to a user, the tactile feedback being described in a tactile description file 192 associated with a scene description 191 of an immersive scene 190. The tactile description file 192 describes the kind of feedback provided according to a syntax further described below. Such a description file is typically transmitted from a server 180 to a tactile rendering device 100. The tactile unit 107 may include a single tactile actuator or multiple tactile actuators located at multiple locations on the tactile rendering device. Different tactile units may have different numbers of actuators and / or the actuators may be positioned differently on the tactile rendering device.

[0033] In at least one embodiment, the processor 101 is configured to render a haptic signal according to the embodiments further described below, in other words, to apply a low-level signal to the haptic actuator to render a haptic effect. Such a low-level signal can be represented using different forms, such as metadata or parameters in a description file, or by using a digital encoding (e.g., PCM or LPCM) of a sampled analog signal.

[0034] The processor 101 may receive power from the power source 108 and may be configured to distribute and / or control power to other components in the device 100. The power source may be any suitable device for powering the device. As an example, the power source may include one or more dry cell batteries (e.g., nickel cadmium (NiCd), nickel zinc (NiZn), nickel metal hydride (NiMH), lithium ion (Li ion), etc.), solar cells, fuel cells, etc.

[0035] Although the figure depicts the processor 101 and other elements 102 to 108 as separate components, it should be understood that these elements can be integrated together in an electronic package or chip. It should be understood that the tactile rendering device 100 can include any sub-combination of the elements described herein while remaining consistent with the embodiments described below. The processor 101 can also be coupled to Figure 1 Other peripherals or units not shown in the figure may include one or more software and / or hardware modules that provide additional features, functions and / or wired or wireless connections. For example, peripherals may include sensors such as universal serial bus (USB) ports, vibration devices, television transceivers, hands-free headsets, Module, FM radio unit, digital music player, media player, video game player module, Internet browser, etc. For example, the processor 101 can be coupled to a positioning unit configured to locate the tactile rendering device within its environment. The positioning unit can integrate a GPS chipset that provides longitude and latitude positions of the current location of the tactile rendering device, and can also integrate other motion sensors that provide positioning services, such as accelerometers and / or electronic compasses.

[0036] Typical examples of the haptic rendering device 100 are a haptic suit, a smart phone, a game controller, a haptic glove, a haptic chair, a haptic prop, a motion platform, etc. However, any device or combination of devices providing similar functionality may be used as the haptic rendering device 100 while still conforming to the principles of the present disclosure.

[0037] In at least one embodiment, the device does not include a display unit, but includes a haptic unit. In such an embodiment, the device does not visually render the scene, but only renders haptic effects. However, the device can prepare data for display so that another device, such as a screen, can perform the display. Examples of such devices are haptic suits or motion platforms.

[0038] In at least one embodiment, the device does not include a haptic unit, but includes a display unit. In such an embodiment, the device does not render a haptic effect, but only visually renders the scene. However, the device can prepare data for rendering a haptic effect so that another device, such as a haptic prop, can perform the haptic rendering. Examples of such devices are smartphones, head-mounted displays, or laptops.

[0039] In at least one embodiment, the device does not include a display unit and does not include a haptic unit. In such an embodiment, the device does not visually render the scene and does not render haptic effects. However, the device can prepare data for display so that another device such as a screen can perform the display, and can prepare data for rendering haptic effects so that another device such as a haptic prop can perform haptic rendering. Examples of such devices are computers, game consoles, optical media players, or set-top boxes.

[0040] In at least one embodiment, the immersive scene 190 and associated elements are hosted directly in the memory 106 of the tactile rendering device 100, thereby allowing local rendering and interaction. In a variation of this embodiment, the device 100 also includes an immersive experience editor 110, which allows for completely independent operation, such as without any communication network 170 and server 180.

[0041] Although the different elements of the immersive scene 190 Figure 1 , but the principles described here also apply to the case where these elements are directly integrated in the scene description rather than being separate elements. Any mix between the two alternatives is also possible, with some elements integrated in the scene description and others being separate files.

[0042] For simplicity of description, the interaction and tactile effects are described here using a finger touching a tactile surface as the interaction medium. However, still relying on the same principle, any other element representing the user's position in the immersive environment (such as a body part of the user, the position provided by a force feedback device, the positioning of a head-mounted display in a virtual reality environment) can be used.

[0043] Figure 2 An example of a data structure for describing an immersive scene according to at least one embodiment is shown. This embodiment is based on glTFTM File format. The core of glTF is a JSON file that describes the structure and composition of a scene containing a 3D model. The figure shows the relationship between the elements that make up the data structure of an immersive scene description 200. In this case, scene 201 is the top-level element that aggregates all other elements. It includes an array of nodes. Each node 202 can contain child nodes that allow the creation of a hierarchy. A node can refer to a mesh or a camera or a skin, and a local geometric transformation can be associated with the node. Mesh 210 corresponds to the geometric data required to render the mesh. Skin 220 is used to perform vertex skinning so that the vertices of the mesh are affected by the bones of the skeleton based on its pose. Camera 225 determines the projection matrix. Light 215 determines the lighting properties associated with the node. Buffer 255 contains data for the geometry, animation, and skinning of the 3D model. Buffer view 250 adds structural information to the buffer data, while accessor 245 defines the exact type and layout of the buffer view. Material 260 determines how the object should be rendered based on physical material properties. Texture 265 allows the appearance of the object to be defined. Image 270 defines the image data for the texture, while Sampler 280 describes the wrapping and scaling of the texture. MPEG Media 205 collects various media files used by the immersive scene description.

[0044] The immersive scene description file also includes a haptic object 230 that describes the haptic effect to be rendered. A haptic object identified in the file format as "MPEG_Haptic" can be associated with a haptic material 235, which is identified in the file format as "MPEG_material_haptics" and describes the properties of the haptic material used to render the haptic effect. The material haptics use one or more haptic multi-level texture maps 236 described herein according to an embodiment and are identified as "MPEG_haptic_mipmap" in the file format syntax described below. The haptic multi-level texture map uses a texture that can be stored with a regular texture 265.

[0045] According to an embodiment, based on glTF TM These elements of an immersive scene description file in the format allow for the definition of an immersive scene with tactile feedback based on tactile multi-level texture mapping. Figure 2 The other elements shown are based on glTF TM General elements of an immersive scene description file in .

[0046] Figure 3AAn example of tactile texture bump mapping according to the prior art is shown. The proposed tactile file format allows tactile texture information to be conveyed in a mapping, which is an image that includes tactile data rather than the color value of a pixel. Using texture to describe tactile properties allows the ability of a 3D engine to be utilized to map textures to 3D objects. Multiple tactile mappings can be associated with a single object (friction, heat, hardness, etc.). Although these mappings enable rendering of tactile textures or tactile surfaces, they also bring specific problems. In fact, tactile textures provide information about a given point in space. This corresponds to, for example, the position where a finger touches a tactile screen. Therefore, as shown in the figure, tactile information is transmitted at a rate tracked by the user (finger).

[0047] Figure 3A A 250 pixel wide image 300 is shown with three regions associated with tactile feedback determined by a texture bump map. The bump map defines regions 301, 303, 305, 307 represented by white ("0" value) as holes, and regions 302, 304, 306 represented by a diagonal hash pattern ("255" value) as bumps. Thus, this tactile texture allows a user sliding his finger 340 over the region to feel a series of bumps and holes when sliding from left to right. The tactile rendering can be performed by vibration, electrical stimulation, or a force feedback device attached to the screen. Figure 3B It can be used to indicate Figure 3A 1D signal of the tactile texture presented in . It corresponds to a single bump.

[0048] For simplicity, in Figure 3A In , user tracking is set to 1Hz and the user moves at 30px / s on the image. Elements 311, 312, 313, 314, 315, 316, and 317 represent captured events (also called scans, samples) of the user's finger tracking across the image based on the finger movement and a fixed tracking rate. If the user moved faster, the expected feedback would be a series of faster bumps and holes (represented by the white areas). However, due to limitations of the tracking system, the sample points chosen on the texture may result in weird rendering, such as Figure 3C shown.

[0049] Figure 3C Shown in Figure 3A. In this example, the user moves his finger 350 across the image at 60px / s, which is much faster than the previous figure, so with the same sampling rate, the tracking only senses elements 321, 322, 323 and 324. In this case, the finger position is detected only on the parts of the texture representing holes 301, 303, 305, 307. Therefore, the tactile rendering will be uniform, similarly as if the user had touched a completely flat (completely white) surface, although the black lines corresponding to the bumps have been crossed.

[0050] This type of tactile rendering technique is called Surface Haptic Object (SHO) and usually relies on discrete 2D grayscale textures. The principle is the same for 1D textures. With this approach, the rendering of the tactile texture is based on the position of the finger on the texture and is therefore dependent on the hardware tracking rate.

[0051] To solve this problem, another approach called Surface Haptic Texture (SHT) can be used. It is based on using the velocity of the finger instead of its position. With this approach, the position of the finger is only used to re-estimate the velocity. Given the velocity, the rendering loop is no longer dependent on the tracking frequency, and it is possible to render the tactile texture at a high frequency with reasonable accuracy. This type of approach is more specifically envisioned for use with one-dimensional periodic tactile textures (such as Figure 3A and 3C ), which makes the solution memory efficient since only a single period needs to be stored.

[0052] Figure 3D The rendering of tactile textures with the SHO and SHT methods is shown. The SHT method is limited in two ways. First, this type of signal limits the rendering to textures consisting of a single periodic element. And second, since the rendering depends only on the velocity, it does not take into account the initial finger position which may cause a shift in the signal. The figure shows the rendering of tactile textures with the SHO and SHT methods. Figure 3C Input signals are shown and rendering of the same tactile texture using the SHO method 340 and the SHT method 345. When the SHT begins rendering at the beginning of period 341, the SHO uses the initial finger position 346 to fully render the texture. While this type of signal shift may not be noticeable for some tactile textures, it may be problematic for other tactile textures.

[0053] Figure 3EThe principle of a set of tactile pixels is described, providing a spatial approach to the SHT method. The tactile pixels determine the shaped area of ​​the texture associated with the tactile signal. For example, the area 351 is associated with the signal 361. When the user passes his finger over this area 351, he should feel the tactile effect defined by the signal 361. In the rendering phase, the finger position is detected in the determined area (for example, the area 351, 352, 353), and the corresponding effect (361, 362, 363, respectively) is rendered according to the current user's speed using the SHT method. In other words, the playback speed of the tactile signal is determined by the speed of the interaction. For example, a higher speed will affect the playback of the tactile signal, resulting in a signal 361 with a higher frequency and a signal 362 with a steeper and shorter slope. This solution combines the advantages of the SHO and SHT methods by providing a spatial-based method for rendering using speed information. 2D textures can be partially processed with this method by using multiple 1D signals assigned to different directions (usually X and Y), for example transmitted on different tracks. However, this solution is only applicable to periodic signals.

[0054] Figure 3F An example of multi-level texture mapping is shown. Multi-level texture mapping is another tool related to texture. Typically, virtual objects displayed in the background of a scene are smaller than when displayed in the foreground. Although a high-resolution texture is required for a close view when the virtual object is in the foreground, such details are not required when the object is far away. Therefore, a low-resolution version of the texture is sufficient. Therefore, multi-level texture mapping includes a set of textures with reduced resolutions, and is usually pre-calculated, such as by shrinking the full-resolution texture. Multi-level texture mapping allows the rendering speed to be reduced by optimizing the required calculations and the required storage space, and also reduces aliasing artifacts. The multi-level texture mapping 390 shown in the figure is a toy example for illustrative purposes, and is composed of three textures of different resolutions: a high-resolution texture 391, a low-resolution texture 393, and an intermediate texture 392. All three textures are combined in a multi-level texture mapping object. When the texture is used to display an object, the appropriate resolution will be selected based on the viewing distance of the object (or the relative size of the object).

[0055] The embodiments described below have been designed in consideration of the above, and introduce the concept of tactile multi-level texture mapping, which allows determining a tactile effect based on a tactile multi-level texture mapping including a plurality of tactile textures, from which a tactile texture is selected based at least on the speed of the user interaction: when the user interacts at a low speed (such as Figure 3A 340), selects a texture with fine details to render tactile feedback, and when the user interaction is fast (such as Figure 3C350), a texture with less detail is selected. Unlike multi-level texture mapping, different texture maps may have the same size but reduced levels of visual detail, thus representing a rougher texture. Using this technique, the rendering is closer to a real experience: in any case, the user is able to feel a series of bumps and holes. When moving slowly, he will feel a lot of details in the tactile effect, while when moving faster, the texture will become simpler. For example, a similar sensation can be felt by placing a hand on a keyboard and moving it left and right. When moving slowly, it is possible to feel different keys and the spaces between them. When moving faster, the sensation is very different.

[0056] Additionally, haptic textures can be selected based on the tracking rate of the rendering device to prevent any of the weird rendering issues mentioned above.

[0057] The proposed embodiments allow for the selection of a haptic texture, and thus a haptic effect to be rendered, based on the speed of the interaction and the tracking rate of the rendering device, thereby ensuring satisfactory and reliable rendering of the haptic effect.

[0058] The embodiments herein describe how to generate such a tactile multi-level texture map from a given tactile texture. A data encoding compatible with the MPEG tactile codec is proposed.

[0059] Figure 4 An example of rendering of a tactile multi-level texture map according to at least one embodiment is shown. It shows the effect of selecting a texture from the tactile multi-level texture map according to the user's interaction speed. For simplicity, these figures represent the tactile effects felt by the user during movement in a unidirectional horizontal movement. The first tactile effect 401 is based on a texture suitable for low speeds and allows the user to feel a series of multiple bumps (represented as a diagonal hash pattern) along the movement 411. The second tactile effect 402 is based on a texture that includes less detail but is suitable for faster user movement, thus allowing some bump feedback (represented as a diagonal hash pattern) to be felt along the movement 412. By selecting a texture from the tactile multi-level texture map according to the user's interaction speed, the expected tactile effect, here a bump effect, can be satisfactorily rendered for different interaction speeds.

[0060] Figure 5 An example of tactile multi-level texture map generation according to a manual generation method is shown. Although it looks random, the original texture 501 includes many details and is therefore suitable for low-speed interaction. From this texture, an intermediate texture 502 and a low-resolution texture 503 can be manually generated. As described above, all three textures provide some granular effects on the surface, like sandpaper, to accommodate different interaction speeds. However, manual texture generation is time consuming and is not efficient in large environments with a large number of textures.

[0061] Figure 6 A tactile multi-level texture map generation process is shown in accordance with at least one embodiment. Figure 7 Shown in Figure 6 The data generated during the tactile multi-level texture map generation process. Traditional automatic methods dedicated to visual multi-level texture mapping cannot be directly adopted because they just scale down the image, while the tactile multi-level texture map may have the same resolution as the original image and is related to the sampling rate of the tracking device. For tactile multi-level texture mapping, the goal is not to change the resolution of the image, but to take advantage of the changing interaction speed and tracking frequency to maintain the semantics behind the tactile data. Therefore, the following process 600 is proposed to automatically generate a tactile multi-level texture map from the original tactile texture.

[0062] In step 610, the original texture ( Figure 7 700) on a plausible user trajectory ( Figure 7 710). The trajectory can be arbitrarily predetermined (e.g., a straight horizontal movement) or based on an analysis of the original texture. The example shown here with a random trajectory is well suited for this random texture. For other types of textures, the trajectory must be different. For example, for Figure 3A For the texture shown in , a straight horizontal trajectory would be a better plausible trajectory. Different strategies can be defined for different types of textures. One example of determining a trajectory is to select the straight direction across the original texture that provides the most variation. Another example is to randomly select a predetermined trajectory. This step of the process can also be interactive by requiring the user to simulate a meaningful interaction with the original texture.

[0063] In step 620, a haptic signal ( ) corresponding to the plausible user trajectory on the original texture is generated by sampling the plausible user trajectory on the original texture using the sampling rate of the tracking device. Figure 7 720). This signal represents the situation where the user moves slowly, so that the obtained tactile signal contains many details.

[0064] In step 630, based on the haptic signal 720, a second haptic signal for double speed is generated ( Figure 7 731). This signal contains the same number of samples and the same samples as the initial tactile signal 720, but with twice the sampling rate. It results in a shorter version of the signal in time (half the length). It is essentially the same signal, but interpreted as having twice the sampling rate. Other techniques can be used for this step, such as keeping the same sampling rate, but removing one out of two samples.

[0065] In step 640, the second haptic signal is resampled (ie, downsampled) to accommodate the frequency rate of the tracking device, thereby generating a downsampled signal at double the speed ( Figure 7 741).

[0066] In step 650, the double-speed downsampled signal is used to generate a tactile texture ( Figure 7 751). Pixels are extrapolated from the estimated pixels. Various image processing methods can be used, such as inpainting or deep learning methods, such as generative adversarial networks (GANs).

[0067] The process 600 is described above for two different interaction speeds, but it can be applied at multiple different speeds to generate multiple textures (by Figure 7 732, 742, 752 of FIG. 1 ). For different interactive speeds, this can be accomplished by multiple iterations on steps 630, 640, 650. In addition, the generation can also take into account multiple tracking rates to accommodate multiple rendering devices. To this end, the generation process also iterates steps 620 to 650 to generate multiple textures for different tracking rates.

[0068] In step 660, the generated tactile texture (eg Figure 7 The elements 751, 752) are combined in a tactile multi-level texture map and according to Figure 2 The data structure (particularly element 236) and the syntax table described below are formatted. This step includes associating information representing tracking rate and interaction speed with the tactile texture.

[0069] While the texture in the image appears to show structure emerging from the original random dot pattern, the illustration is only a conceptual example of the generation process.

[0070] Figure 8 An example flow chart of a process for rendering a haptic feedback description file according to at least one embodiment is shown.Such a process 800 is typically implemented in a haptic rendering device 100 and executed by a processor 101 of such a device.

[0071] In step 810, the processor obtains an immersive scene ( Figure 1 Of these, 191 Figure 2 201 in the scene description file, wherein the 3D object includes a tactile object with a tactile multi-level texture mapping. This can be done, for example, by receiving it from a server via a communication network, by reading it from an external storage device or local memory, or by any other means. The processor analyzes the scene description file in order to extract the tactile object ( Figure 1 Of these, 192 Figure 2 230 in the example), the haptic object allows determination of parameters related to the haptic effect, more specifically including a haptic volume and a haptic multi-level texture map associated with the haptic effect ( Figure 2 236).

[0072] In step 820, the processor obtains the interaction speed, in other words, the movement speed of the element representing the user and controlled by the user to interact with the immersive scene. In an example implementation of a smart phone, the interaction is done through a touch screen, and the interaction speed is the speed of the finger moving on the touch screen. In another example, the interaction is done using a motorized force feedback device, and the interaction speed is the speed of the hand holding the end effector of the device.

[0073] At step 830, the processor obtains the tracking rate (the frequency rate of the tracking device). In practice, the tactile multi-level texture map may also vary based on the tracking rate, as described in the generation of the tactile multi-level texture map. This step is optional and does not exist if the tracking rate is not used.

[0074] At step 840, the processor selects a tactile texture from the tactile multilevel texture map associated with the tactile object based on the user interaction speed. The selection is done by comparing the interaction speed associated with the tactile texture with the user interaction speed and selecting a matching tactile texture, e.g., the closest (best match, smallest difference). As described above, this allows the tactile texture, and thus the tactile effect to be rendered, to be adapted to the interaction speed of the rendering device in order to ensure satisfactory and reliable rendering of the effect.

[0075] Additionally, the selection may optionally be made based on tracking rate by comparing the tracking rate associated with the tactile texture to the tracking rate of the device and selecting the matching tracking rate, such as the closest (best match, smallest difference).

[0076] At step 850 , the processor renders the haptic effect. To this end, the processor converts the selected haptic texture into a haptic signal based on the position of the user interaction relative to the haptic texture, and provides data of the haptic signal to the haptic actuator.

[0077] In at least one embodiment, a haptic effect is implemented in an immersive scene description (300 in FIG. 3 ) including a haptic object associated with a haptic material (i.e., the surface of the haptic object) using elements defined in the description of the MPEG_material_haptic element as shown in the JSON schema of Table 1. Unlike conventional haptic texture mapping, this embodiment proposes referencing a haptic multi-level texture mapping, rather than directly referencing a texture, so that the haptic effect can adapt to the speed of the interaction and the tracking rate of the rendering device to ensure satisfactory and reliable rendering of the effect.

[0078] Table 1

[0079]

[0080]

[0081]

[0082]

[0083] In this embodiment, the tactile multi-level texture mapping is implemented according to the JSON schema of Table 2, wherein the multi-level texture mapping includes a set of tactile textures, and each tactile texture is associated with information representing an interaction speed and a tracking rate.

[0084] Table 2

[0085]

[0086]

[0087] A variant embodiment proposes another implementation of tactile multi-level texture mapping, in which the tactile effect, more specifically the tactile material, directly includes multiple (i.e., an array) tactile textures (still referred to as haptic_mipmap in Tables 3 and 4), which have associated information representing the interaction speed and tracking rate. In other words, the first embodiment of Tables 1 and 2 proposes a multi-level texture mapping that groups a set of textures and associated parameters, while the variant embodiment proposes grouping a set of "multi-level texture mappings", in which each multi-level texture mapping only involves a single texture and its associated parameters.

[0088] Table 3 shows the JSON schema of the MPEG_material_haptic element according to this variant embodiment.

[0089] Table 3

[0090]

[0091]

[0092]

[0093]

[0094]

[0095] In this variant embodiment, haptic multi-level texture mapping is implemented according to the JSON schema of Table 4, where one haptic texture is associated with information representing interaction speed and tracking rate. As shown in Table 5, multiple textures are signaled at the effect layer.

[0096] Table 4

[0097]

[0098]

[0099] Table 5 shows a glTF description of an example of a 3D object (bottle) including a tactile multi-level texture map with two textures for two different speeds according to the above variant embodiment.

[0100] Table 5

[0101]

[0102]

[0103]

[0104]

[0105]

[0106]

[0107]

[0108]

[0109] Because the parameters are discrete values, it is possible to be in a situation where no tactile texture corresponds to the actual parameters of the rendering device. In this case, interpolation between two tactile textures can be performed to generate a tactile texture corresponding to the real parameters of the rendering device.

[0110] As described above, the device that receives and decodes the immersive scene may not perform the rendering itself, but instead delegate that task to another device, such as a dedicated tactile rendering device. In this case, data is prepared for the rendering of visual elements and / or tactile effects and transmitted to the device that performs the rendering. This remote rendering can be used for audio, video, and tactile data, and is highly dependent on the capabilities built into the devices involved. In some cases, a combination of devices may be required to fully render the immersive experience. In other cases, the device includes all the elements required to perform all tasks, including decoding and rendering. This is the case, for example, when a smartphone displays an augmented reality scene and provides vibrations when the user interacts with the scene.

[0111] Although different embodiments have been described separately, any combination of the embodiments may be made while adhering to the principles of the present disclosure.

[0112] Although the embodiments relate to haptic effects, those skilled in the art will appreciate that the same principles can be applied to other effects, such as, for example, sensory effects, thus including smell and taste. Therefore, appropriate syntax will determine the appropriate parameters associated with these effects.

[0113] References to "one embodiment" or "an embodiment" or "an implementation" or "implementation" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation", as well as any other variations appearing in various places throughout the specification, do not necessarily all refer to the same embodiment.

[0114] Furthermore, this application or its claims may refer to "determining" various pieces of information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0115] Furthermore, this application or its claims may refer to "obtaining" various pieces of information. As with "accessing," obtaining is a broad term. Obtaining information may include, for example, one or more of receiving information, accessing information, or retrieving information (e.g., from a memory or optical media storage device). Furthermore, during operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "obtaining" is often involved in one way or another.

[0116] It should be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," any of the following uses of " / ," "and / or," and "at least one of" are intended to include selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such wording is intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This can be extended to many of the listed items, as will be apparent to one of ordinary skill in this and related arts.

Claims

1. A method for decoding a tactile effect, comprising: Obtaining information representing a haptic effect, the information comprising a haptic multi-level texture map, the haptic multi-level texture map comprising a plurality of haptic textures, each haptic texture being associated with a corresponding parameter representing an interaction speed; Get the interaction speed of the element representing the user; selecting a haptic texture based at least on the obtained interaction speed; and Based on the position of the element representing the user relative to the tactile texture, data of the selected tactile texture is provided to the tactile actuator. 2 . The method of claim 1 , wherein the parameter associated with the tactile texture further represents a tracking rate, and wherein the selection of the tactile texture is further based on the tracking rate.

3. The method according to claim 1 or 2, wherein: The selection of the tactile texture is based on a minimum difference in a comparison between a parameter of the tactile texture representing the interaction speed and an interaction speed of the element representing the user.

4. A device for decoding a haptic effect, comprising a processor, the processor being configured to: Obtaining information representing a haptic effect, the information comprising a haptic multi-level texture map, the haptic multi-level texture map comprising a plurality of haptic textures, each haptic texture being associated with a corresponding parameter representing an interaction speed; Get the interaction speed of the element representing the user; selecting a haptic texture based at least on the obtained interaction speed; and Based on the position of the element representing the user relative to the tactile texture, data of the selected tactile texture is provided to the tactile actuator. 5 . The apparatus of claim 4 , wherein the parameter associated with the tactile texture further represents a tracking rate, and wherein the selection of the tactile texture is further based on the tracking rate.

6. The device according to claim 4 or 5, wherein: The selection of the tactile texture is based on a minimum difference in a comparison between a parameter of the tactile texture representing the interaction speed and an interaction speed of the element representing the user.

7. A non-transitory computer-readable medium comprising encoded data, the encoded data comprising information representing a haptic effect, comprising a haptic multi-level texture map, the haptic multi-level texture map comprising a plurality of haptic textures associated with a parameter representing an interaction speed.

8. A computer program comprising program code instructions for implementing the method according to any one of claims 1 to 3 when executed by a processor.

9. A non-transitory computer readable medium comprising program code instructions for implementing the method according to any one of claims 1 to 3 when executed by a processor.

10. A method for generating a tactile multi-level texture map from a first tactile texture, comprising: determining a trajectory on a first tactile texture; determining a first tactile signal corresponding to the trajectory at a first sampling rate; determining a second haptic signal that is a subset of the first haptic signal; resampling the second haptic signal at a second sampling rate; generating a second haptic texture from the resampled second haptic signal; as well as At least first and second tactile textures and corresponding information representing first and second sampling rates are combined in a data structure.

11. A signal comprising information representing a haptic effect, comprising a haptic multi-level texture map comprising a plurality of haptic textures associated with a parameter representing an interaction speed.

12. The signal of claim 11, wherein the tactile multi-level texture map is generated according to claim 10.

Images