Haptic effects including washout

Abstract

For the haptic signal, a syntax element of the haptic scene format distinguishes portions of the haptic signal associated with the haptic effect from portions associated with the scour. This allows the haptic rendering device to adapt haptic signals to particular types of haptic devices, haptic device capabilities, and target body parts in order to remain below a perceptual threshold and ensure an expected haptic experience.

Description

Technical Field

[0001] At least one of the present embodiments relates generally to haptics, and more particularly to syntax for describing haptic effects including haptic washout information, and methods and apparatus for rendering haptic effects. Background Art

[0002] A fully immersive user experience is proposed to the user through an immersive system based on feedback and interaction. The interaction can use conventional control methods that meet the needs of the user. Current visual and auditory feedback provides a satisfactory level of realistic immersion. Additional feedback can be provided through haptic effects, which allow the human user to perceive the virtual environment with his senses and thus get a better experience of full immersion with improved realism. However, haptics remains an area of ​​potential advancement to improve the overall user experience in immersive systems.

[0003] Conventionally, an immersive system may include a 3D scene representing a virtual environment, wherein virtual objects are located within the 3D scene. In order to improve the user's interaction with elements of the virtual environment, tactile feedback may be used through the excitation of tactile actuators. Such interaction is based on the concept of a "tactile object", which corresponds to a physical phenomenon to be transmitted to the user. In the context of an immersive scene, the tactile object allows to provide tactile effects by defining the excitation of appropriate tactile actuators to imitate physical phenomena on a tactile rendering device. Different types of tactile actuators allow to reproduce different types of tactile feedback.

[0004] An example of a haptic object is an explosion. An explosion can be rendered with vibration and heat, combining different haptic effects on the user to increase realism. An immersive scene typically includes multiple haptic objects, such as using a first haptic object associated with a global effect and a second haptic object associated with a local effect.

[0005] The principles described herein apply to any immersive environment that uses touch, such as, for example, augmented reality, virtual reality, mixed reality, or haptic-enhanced video (or omnidirectional / 360° video) rendering, and more generally to any haptic-based user experience. Such example scenes of immersive environments are therefore considered immersive scenes.

[0006] Haptics refers to the sense of touch and includes two dimensions: tactile and kinesthetic. The first dimension is related to the sense of touch, such as friction, roughness, hardness, temperature, and is felt through the mechanoreceptors of the skin (Merkel cells, Ruffini endings, Meissner corpuscles, Pacinian corpuscles). The second dimension is related to the sense of force / torque, position, motion / speed provided by mechanoreceptors in muscles, tendons and joints. Haptics also involves the perception of self-motion because it contributes to the proprioceptive system (i.e., the perception of one's own body). Therefore, the perception of acceleration, velocity or any body model can be assimilated into a tactile effect. The frequency range is about 0 to 1kHz, depending on the type of modality. Most existing devices capable of rendering tactile signals produce vibrations. Examples of such tactile actuators are linear resonant actuators (LRAs), eccentric rotating masses (ERMs), and voice coil linear motors. These actuators can be integrated into tactile rendering devices (such as tactile suits), as well as into smart phones or game controllers.

[0007] To encode haptic signals, several formats have been defined, related to high-level descriptions using XML-like formats (e.g., MPEG-V), parameterized representations using json-like formats (such as Apple Haptic Audio Profile (AHAP) or Immersion Corporation's HAPT format), or waveform coding (IEEE 1918.1.1 ongoing standardization of haptic and kinesthetic signals). The HAPT format has recently been included in the MPEG ISOBMFF file format specification (ISO / IEC 14496 Part 12). In addition, the GL Transmission Format (glTF) TM ) is a royalty-free specification for the efficient transmission and loading of 3D scenes and models for applications. The format defines an extensible common publishing format for 3D content tools and services that simplifies the authoring workflow and enables interoperable use of content across industries.

[0008] In addition, a new haptic file format is being defined within the MPEG standardization group, which relates to an encoded representation of haptics. The reference model for this format has not yet been published, but is referred to herein as RMO. With this reference model, the encoded haptic description file can be exported to a human-readable JSON exchange format (e.g., a .gmpg file), or to a compressed binary distribution format (e.g., .mpg) that is particularly suitable for transmission to a haptic rendering device.

[0009] In the field of tactile rendering, the main problems are related to the physical limitations associated with the rendering devices. In fact, for a rendering device that should provide a thermal sensation to the user, for example in order to be associated with an explosion in an action movie, there is some inertia of transferring heat and then cooling down again. Therefore, in some cases, it is impossible to combine the succession of some tactile effects. In the case of a tactile device, when two "forward acceleration" effects need to be rendered, the tactile device may not have enough range of movement to combine the two required movements. To overcome this problem, the tactile device will first move forward to deliver the acceleration sensation, and then it will have to return to its original position before moving forward again to deliver the second acceleration. This intermediate step of moving back to the original position between the two effects is called "wash" and should not be perceived by the user. In order not to be noticed by the user, the acceleration of the movement must be less than the threshold of the vestibular system, which is about 0.1m / s 2 . Summary of the invention

[0010] Embodiments relate at least to grammatical elements of a haptic scene format, allowing for differentiation in a haptic signal of portions associated with a washout and portions associated with a normal haptic effect. This allows a haptic rendering device to adapt the haptic signal to a specific type of haptic device, haptic device capabilities, and target body part in order to stay below a perception threshold and ensure an expected haptic experience.

[0011] A first aspect of at least one embodiment relates to a method, comprising: obtaining tactile data representing tactile feedback, the tactile data including a tactile signal for rendering the tactile feedback and information representing a flushing, when the information representing the flushing indicates that the tactile signal is a flushing effect, adapting the tactile signal to remain below a perception threshold, and providing an adapted tactile signal.

[0012] A second aspect of at least one embodiment relates to a device comprising a processor configured to obtain tactile data for tactile feedback, the tactile data comprising a tactile signal for rendering the tactile feedback and information representing a flushing, adapt the tactile signal to remain below a perception threshold when the information representing the flushing indicates that the tactile signal is a flushing effect, and provide the adapted tactile signal for rendering.

[0013] A third aspect of at least one embodiment relates to a method comprising encoding tactile data representing tactile feedback, the tactile data comprising a tactile signal and information representing a flushing effect, the information indicating that the tactile signal is a flushing effect, and providing the encoded tactile data.

[0014] A fourth aspect of at least one embodiment relates to a device comprising a processor configured to: encode tactile data representing tactile feedback, the tactile data comprising a tactile signal and information representing a flushing effect, the information indicating that the tactile signal is a flushing effect, and provide the encoded tactile data.

[0015] According to a fifth aspect of at least one embodiment, a computer program is proposed, the computer program comprising program code instructions executable by a processor, the computer program at least implementing the steps of the method according to the first aspect.

[0016] According to a sixth aspect of at least one embodiment, a computer program product is proposed, which is stored on a non-transitory computer-readable medium and includes program code instructions executable by a processor, and the computer program product at least implements the steps of the method according to the first aspect.

[0017] A seventh aspect of at least one embodiment relates to a data structure comprising tactile data for tactile feedback, the tactile data comprising a tactile signal and information representing a flushing, the information indicating that the tactile signal is a flushing effect, the tactile data being generated according to the third aspect.

[0018] An eighth aspect of at least one embodiment relates to a non-transitory computer-readable medium, comprising tactile data for tactile feedback, the tactile data comprising a tactile signal and information representing a flushing, the information indicating that the tactile signal is a flushing effect, the tactile data being generated according to the third aspect.

[0019] In embodiments of these aspects, the information indicating the flushing effect is represented by a Boolean flag or an enumeration value. TM The flushing effect information is represented using standard JSON syntax. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A block diagram illustrating an example of a system in which various aspects and embodiments are implemented is illustrated.

[0021] Figure 2 An example of a structure for describing an immersive scene according to one embodiment is illustrated.

[0022] Figure 3 An example of the structure of an interchange file format describing an immersive scene is illustrated.

[0023] Figure 4 An example of a signal encoded using two tactile bands is illustrated.

[0024] Figure 5A conventional method for encoding low-level haptic signals in two frequency bands is illustrated.

[0025] Fig. 6A An example of a haptic signal for a motion platform haptic device including three forward effects is illustrated.

[0026] Figure 6B The diagram shows the Fig. 6A Example of adapting a tactile signal to a force feedback haptic device.

[0027] Figure 7 An example flow diagram of a process for rendering a haptic feedback description file according to an embodiment is illustrated.

[0028] Fig. 8A The diagram shows the Fig. 6A A second example of adapting a tactile signal to a force feedback haptic device.

[0029] Figure 8B The diagram shows the Fig. 6A A third example of adapting a tactile signal to a force feedback haptic device. DETAILED DESCRIPTION

[0030] Figure 1 A block diagram of an example of an immersive system in which various aspects and embodiments are implemented is illustrated. In the depicted immersive system, user Alice uses a tactile rendering device 100 to interact with a server 180 hosting an immersive scene 190 through a communication network 170. The immersive scene 190 may include various data and / or files representing different elements required for its rendering (scene description 191, audio data, video data, 3D model, and tactile description file 192). The immersive scene 190 can be generated under the control of an immersive experience editor 110, which allows different elements to be arranged together and an immersive experience to be designed. Appropriate description files and various data files representing the immersive experience are generated by an immersive scene generator 111 in a format suitable for transmission to a tactile rendering device. The immersive experience editor 110 is typically executed on a computer that will generate an immersive scene to be hosted on a server. For simplicity, the immersive experience editor 110 is illustrated as being directly connected to the immersive scene 190 via a dotted line 171. In practice, immersive scene 190 is hosted on server 180 , and a computer running immersive experience editor 110 is connected to server 180 via communication network 170 .

[0031] The tactile rendering device 100 includes a processor 101. The processor 101 can be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, and the like. The processor can perform data processing, such as tactile signal decoding, input / output processing, and / or any other function that enables the device to operate in an immersive system.

[0032] The processor 101 may be coupled to an input unit 102 configured to transmit user interactions. Various types of inputs and modalities may be used for this purpose. A physical keypad or touch-sensitive surface is a typical example of an input suitable for this purpose, but voice control may also be used. In addition, the input unit may also include a digital camera capable of capturing a static picture or video in two-dimensional form, or a more complex sensor capable of determining depth information in addition to the picture or video and thus capable of capturing a complete 3D representation. The processor 101 may be coupled to a display unit 103 configured to output visual data to be displayed on a screen. Various types of displays may be used for this purpose, such as a liquid crystal display (LCD) or an organic light emitting diode (OLED) display unit. The processor 101 may also be coupled to an audio unit 104 configured to render sound data, which will be converted into audio waves by an adapted transducer such as, for example, a speaker. The processor 101 may be coupled to a communication interface 105 configured to exchange data with an external device. The communication preferably uses a wireless communication standard to provide mobility of the tactile rendering device, such as cellular (e.g., LTE) communication, Wi-Fi communication, and the like. The processor 101 can access information from and store data in the memory 106, which can include multiple types of memory, including random access memory (RAM), read-only memory (ROM), hard disk, subscriber identity module (SIM) card, memory stick, secure digital (SD) memory card, any other type of memory storage device. In an embodiment, the processor 101 can access information from and store data in a memory that is not physically located on the device (such as on a server, home computer, or other device).

[0033] The processor 101 is coupled to a haptic unit 107, which is configured to provide tactile feedback to the user, which is described in a tactile description file 192 associated with a scene description 191 of an immersive scene 190. The tactile description file 192 describes the kind of feedback to be provided according to the grammar further described below. Such a description file is usually transmitted from the server 180 to the tactile rendering device 100. The tactile unit 107 may include a single tactile actuator or multiple tactile actuators located at multiple locations on the tactile rendering device. Different tactile units may have different numbers of actuators and / or the actuators may be positioned differently on the tactile rendering device.

[0034] In at least one embodiment, the processor 101 is configured to render a haptic signal according to the embodiments further described below, in other words, to apply a low-level signal to a haptic actuator to render a haptic effect. Such a low-level signal can be represented using different forms, such as metadata or parameters in the description file, or by digital encoding using a sampled analog signal (e.g., PCM or LPCM).

[0035] The processor 101 may receive power from the power source 108 and may be configured to distribute power to and / or control power to other components in the device 100. The power source may be any suitable device for powering the device. As an example, the power source may include one or more dry cell batteries (e.g., nickel cadmium (NiCd), nickel zinc (NiZn), nickel metal hydride (NiMH), lithium ion (Li-ion), and the like), solar cells, fuel cells, and the like.

[0036] Although the figures depict the processor 101 and other elements 102 to 108 as separate components, it will be understood that these elements may be integrated together in an electronic package or chip. It will be understood that the tactile rendering device 100 may include any sub-combination of the elements described herein while remaining consistent with the embodiments. The processor 101 may be further coupled to Figure 1 Other peripheral devices or units not depicted in the figure may include one or more software and / or hardware modules that provide additional features, functions, and / or wired or wireless connections. For example, peripheral devices may include sensors such as universal serial bus (USB) ports, vibration devices, television transceivers, hands-free headsets, Modules, frequency modulation (FM) radio units, digital music players, media players, video game player modules, Internet browsers, and the like. For example, the processor 101 can be coupled to a positioning unit that is configured to locate the tactile rendering device in the environment of the tactile rendering device. The positioning unit can integrate a GPS chipset that provides longitude and latitude locations of the current location of the tactile rendering device, and can also integrate other motion sensors that provide positioning services, such as accelerometers and / or electronic compasses.

[0037] Typical examples of the haptic rendering device 100 are a haptic suit, a smart phone, a game controller, a haptic glove, a haptic chair, a haptic prop, a motion platform, etc. However, any device or combination of devices providing similar functionality may be used as the haptic rendering device 100 while still conforming to the principles of the present disclosure.

[0038] In at least one embodiment, the device does not include a display unit, but includes a haptic unit. In such an embodiment, the device does not visually render the scene, but only renders haptic effects. However, the device can prepare data for display so that another device (such as a screen) can perform the display. Examples of such devices are haptic suits or motion platforms.

[0039] In at least one embodiment, the device does not include a haptic unit, but includes a display unit. In such an embodiment, the device does not render haptic effects, but only visually renders the scene. However, the device can prepare data for rendering haptic effects so that another device (such as a haptic prop) can perform haptic rendering. Examples of such devices are smartphones, head-mounted displays, or laptop computers.

[0040] In at least one embodiment, the device does not include a display unit and does not include a haptic unit. In such an embodiment, the device does not visually render the scene and does not render a haptic effect. However, the device can prepare data for display so that another device (such as a screen) can perform display, and can prepare data for rendering a haptic effect so that another device (such as a haptic prop) can perform haptic rendering. Examples of such devices are computers, game consoles, optical media players, or set-top boxes.

[0041] In at least one embodiment, the immersive scene 190 and associated elements are hosted directly in the memory 106 of the tactile rendering device 100, allowing local rendering and interaction. In a variation of this embodiment, the device 100 also includes an immersive experience editor 110, allowing completely independent operation, such as without any communication network 170 and server 180.

[0042] Although the different elements of the immersive scene 190 Figure 1 , but the principles described herein also apply to the case where these elements are directly integrated into the scene description rather than being separate elements. Any mix between the two alternatives is also possible, where some of the elements are integrated into the scene description and others are separate files.

[0043] Figure 2 An example of the architecture of an encoder 200 for a haptic file is illustrated. The encoder 200 is implemented, for example, as a module of the immersive scene generator 111 of the immersive editor 110, and is typically executed on a computer that generates a file describing an immersive scene. The input is a metadata file 201 and at least one low-level haptic signal file 203. The metadata file 201 is, for example, based on the "OHM" haptic object file format. The signal file represents an analog signal to be applied to a haptic actuator and is typically encoded using pulse code modulation (PCM), for example based on a WAV file format. The descriptive file 202 is, for example, based on an AHAP or HAPT file format.

[0044] In step 210, metadata is extracted from metadata file 201, thereby allowing identification of descriptive files and / or signal files. In step 211, the descriptive files are analyzed and transcoded. In step 212, the signal files are processed, for example, to decompose the signal into frequency bands and key frames or wavelets, such as Figure 4 As further described in .

[0045] The formatting step 220 takes into account the transcoded and processed signals produced by steps 211 and 212 to generate an exchange file 204 that conforms to a data format according to one of the embodiments described herein. The exchange file 204 may be compressed in step 230 for distribution in a transport-friendly form (such as a distribution file 205, which is more compact than the exchange file format, but based on the same elements). Thus, the exchange file format described herein is not strictly limited to files, but may also be used for a wider range of transmission or distribution (e.g., streaming, downloading) of data.

[0046] The exchange file 204 may be, for example, based on glTF TM The distribution file 205 may be a human readable file in MPEG, XML or JSON format. The distribution file 205 may be, for example, a binary encoded file based on an MPEG file format suitable for streaming or broadcasting to a decoder device.

[0047] Figure 3An example of the structure of an exchange file format describing an immersive scene is illustrated. Data structure 300 represents an immersive scene 190. It can be decomposed into a set of layers. At the upper layer, metadata 301 describes high-level metadata information about the overall tactile experience defined in data structure 300 and a list of avatars 302 (i.e., body representations) referenced later in the file. These avatars allow the target location of tactile stimulation on the body to be specified. The tactile effect is described by a list of perceptions 310, 31N. These perceptions correspond to tactile signals associated with specific perceptual modalities (such as vibration, force, position, speed, temperature, etc.). Perception includes metadata 320 for describing the tactile content of the signal, a device 321 for describing the specification of the tactile device for which the signal is designed, and a list of haptic tracks 331, 33N. The tactile track includes metadata 340 for describing the content of the track, associated gain values, blending weights, body positioning information, and a reference to the tactile device specification (defined at the perception level). The track ultimately contains a list of haptic bands 351, 35N, each band defining a subset of the signal within a given frequency range. For example, haptic band 351 may correspond to a range of frequencies from 0 to 50 Hz, while haptic band 35N may correspond to a range of frequencies above 2 kHz. The haptic bands include band data 360 describing the frequency range of the band, the type of encoding modality (vector or wavelet), the type of band (transient, curve, and wave), and optionally the type of curve (cubic, linear, or unknown) or window length. The haptic bands are defined by a list of haptic effects 371, 37N. Finally, the haptic effects include a list of keyframes 391, 39N, defined by position (i.e., time reference) and amplitude, and effect data 380. The effect data describes the type of underlying signal selected among sine, square, triangle, sawtooth up, and sawtooth down, as well as providing a time reference such as a timestamp. The low-level haptic signal can then be reconstructed by combining the keyframes of the haptic effects in different bands, such as Figure 3 .

[0048] Figure 4 An example of a signal encoded using two tactile bands is illustrated. With this technique, a low-level tactile signal is encoded using two frequency bands (a low-frequency band 410 and a high-frequency band 420), each of which defines a portion of the signal within a given frequency range. In this example, the low-frequency band corresponds to frequencies below 72.5 Hz, while the high-frequency band corresponds to frequencies equal to or above 72.5 Hz. On the rendering side, the device combines the two portions together (i.e., adds them together) to generate a final tactile signal 440.

[0049] The data of the frequency bands can be reconstructed based on the keyframes and according to the type of tactile band selected in Transients, Curves and Bands. In addition, for the bands, two types of coding modalities can be used: vector or wavelet. Each frequency band consists of a series of effects, and each effect is defined by a list of keyframes, which are represented as points in the diagram. For different types of tactile bands and coding modalities, the effects and the data contained in the keyframes are interpreted differently.

[0050] For transient bands, each effect stores a set of keyframes defining the position, amplitude and frequency. A keyframe represents a transient event. The signal can be reconstructed using the type of periodic underlying signal specified in the effect metadata, where the amplitude is specified in the keyframe and the period is given by the keyframe's frequency. Transient events are very short signals generated over only a few periods. The number of periods generated is determined by the decoder.

[0051] For curve bands, each effect stores a set of keyframes that define the position (i.e. time reference) and amplitude. The keyframes represent the control points of the curve, and interpolation is performed to generate the curve from the control points. The type of interpolation function is either cubic or linear, and is specified in the band's metadata ( Figure 3 380 in ). The signal can be reconstructed by performing interpolation between the amplitudes of the key frames according to their temporal references.

[0052] For vector bands, the effect stores a set of keyframes defining the position (i.e. time reference), amplitude and frequency. In this case, the signal is generated using the type of periodic basis signal specified in the effect metadata, where the amplitude is specified in the keyframes and the period is given by the frequency of the keyframes. The SPIHT wavelet coding scheme (http: / / www.siliconimaging.com / SPIHT.htm) can be used for wavelet bands or types of wavelet coding. For example, for a wavelet band, the effect can store the contents of a wavelet block. It contains a keyframe for each coefficient of the wavelet transform and quantized signal indicating the amplitude value of the wavelet. The coefficients are scaled to the range [-1,1]. In addition, the original maximum amplitude as well as the maximum number of used bits are stored in the keyframes. In this case, the coefficients can be used to reconstruct the signal to perform an inverse wavelet transform.

[0053] Frequency band decomposition can use low-pass filters and high-pass filters to separate the signal into low-frequency bands and high-frequency bands. These two bands are then processed differently. Various methods can be used to encode the high-frequency part. The first solution is to divide the high-frequency signal into smaller fixed-length windows and use short-time Fourier transform (STFT) to decompose the signal in the spectrum. Another solution is to use wavelet transform to encode the high frequency. Figure 3The data structure illustrated in allows to define multiple bands with different frequency ranges. These bands are used to store the coefficients of the Fourier or wavelet transform.

[0054] For the low frequency part of the signal, the data for this frequency band is stored as a list of keyframe points defined by timestamp and amplitude. The data also contains information about the type of interpolation used to reproduce the signal for this band. The keyframes (i.e. control points) that define the low frequency band are obtained by simply extracting the local extrema of the low frequency signal.

[0055] In the example of the attached figure, a low frequency band 410 is defined as a curve band using a single effect 411. Such a representation is particularly suitable for the low frequency part of the signal. Effect 411 is defined by key frames 4111, 4112, 4113, 4114, 4115, 4116, 4117, 4118, 4119. The signal of the low frequency band is generated by cubic interpolation between these key frames. A high frequency band 420 is defined by four effects 421, 422, 423, 424. These effects are defined as vector bands. Effect 421 is based on four key frames 4211, 4212, 4213, 4214. Effect 422 is based on 11 key frames (not numbered in the attached figure). Effect 423 is based on two key frames 4231 and 4232. Effect 424 is based on two key frames 4241 and 4242.

[0056] Although the description is based on a set of two bands defining a range of low frequencies and a range of high frequencies, the principles also apply to the case where more than two frequency ranges are used. In this case, the low frequency band becomes the lowest frequency band and the high frequency band becomes the highest frequency band. The lowest frequency band can be encoded, for example, with a curved band using a single effect, such as by Figure 4 The other frequency bands may be encoded using any other type of encoding, such as using wavelet-based vector bands, as shown by Figure 4 The high frequency band is shown as 420, but multiple encoding instances are used, one encoding instance for each frequency band.

[0057] One advantage of this solution with regard to the structure is that the signal data is easy to pack and is particularly convenient for streaming purposes. In fact, with such a linear structure, the data can be easily broken down into small continuous packets and no complex data pre-fetching operations are required. The signal is easily reconstructed by patching the packets back together to ensure smooth playback of the signal. It is also possible to reconstruct it by taking only the low-frequency part and reconstructing a lower quality (but possibly adequate) signal without taking into account the high-frequency bands.

[0058] As described in detail in the following sections, other parts of this document describe the encoding of PCM waveform signals (e.g., carried by an input WAV file). In this context, the input WAV file describes a single perceptual modality, and even if the file contains multiple tracks, the encoder will process each track separately. Therefore, for clarity in the rest of this disclosure, the specification will describe the encoding and decoding of a single track.

[0059] Figure 5 A conventional method for encoding low-level haptic signals in two frequency bands is shown. This corresponds to Figure 2 The signal processing step 212 and, for example, by Figure 1 The immersive scene generator 111 of the embodiment of the present invention is implemented by an encoder. Given an input PCM signal, the encoder starts the process 500 by performing frequency band decomposition. Using a low-pass filter and a high-pass filter, the encoder separates the signal into low-frequency bands 511 and high-frequency bands 512 in step 510. In step 520, the encoder analyzes each low-frequency band and extracts data 521 representing the low-frequency band, and in step 530, the encoder analyzes each high-frequency band and extracts data 531 representing the high-frequency band. Then, in step 540, the encoder extracts the high-frequency band. Figure 2 In the formatting step 220, according to Figure 3 The structure to format the extracted data.

[0060] In a typical example implementation, there is a single low frequency band encoded using curve bands such that the LF data comprises a set of keyframes extracted in step 520, and there is a single high frequency band encoded using vector bands such that the HF data comprises a set of wavelets extracted in step 530, as described above in Figure 4 As described in.

[0061] This hybrid format combining curved bands and wavebands is interesting and allows very easy storage of low frequency signals. This is especially convenient for synthetic signals (especially kinesthetic signals) produced by haptic authoring tools.

[0062] As above Figure 2 As introduced in , information about the characteristics of the reference tactile rendering device on which the signal is best rendered can be obtained by Figure 3The metadata 321 in the data structure presented in is carried. If the tactile rendering device is different from the reference device, the signal can be adapted to better represent the intended effect. The characteristics of the reference device help for this purpose, but may not be sufficient. In fact, some information about the nature of the signal may be lost, especially in the case of kinesthetic devices and motion platforms. Some parts of the signal are not intended to be felt by the user, but to achieve a washout effect, i.e., to reposition the device to an intermediate position. In fact, returning to the center of the workspace is necessary in order to be able to render another effect. If these washout parts of the signal are not identified, transcoding from one device to another may produce inaccurate results and a poor user experience.

[0063] Fig. 6A An example of a tactile signal for a motion platform haptic device including three forward effects is illustrated. Only one degree of freedom is represented in the tactile effect 600A, and the degree of freedom represents a rotation about the X axis, corresponding to pitch. The rendering of effects 610A, 630A, and 650A causes the seat to rotate backward so that the effect of forward motion is felt by the user of the motion platform. In order to render the second effect 630A after the first effect 610A, the device must return toward the middle (or center) position to prevent the device from reaching its maximum capacity, and thus block and fail to render the subsequent effects correctly. For this purpose, in at least one embodiment, a flushing effect 620A, 640A is signaled to indicate that the device returns to the middle position in order to prepare for the next effect. There is an important difference between a tactile effect and a flushing, in that the user should not feel the flushing. The movement of the flushing is designed to remain below the perception threshold of the vestibular system (for angular motion, ~3° / s). Therefore, the flushing portion of the tactile signal is generally designed to comply with the perception threshold.

[0064] Figure 6B The diagram shows the Fig. 6A An example of adapting the tactile signal of to a force feedback tactile device. In fact, rendering on a force feedback tactile device Fig. 6A The tactile signals need some adaptation because the perception threshold of the human kinesthetic system is different from that of the vestibular system. Fig. 6A If the signal would be provided directly to the force feedback haptic device without any adaptation (or simply rescaling to the working space of the device), the user would be able to feel the device returning to its neutral position, thereby weakening the haptic experience.

[0065] In order to provide a complete haptic experience without this drawback, the flushing must be adapted according to the type of device, device capabilities, and target body part so that the haptic effect is fully felt, but the flushing is not felt. Figure 6B Shows the Fig. 6AThe signal of is adapted as a result of the signal 600B for rendering on a force feedback device held in, for example, the right hand. The portion of the signal corresponding to the flushing effect 602B, 640B is adapted to comply with the perception level of the human kinesthetic system, while the other portions 610B, 630B, 650B are unchanged.

[0066] The embodiments described below have been designed with the foregoing in mind and propose to distinguish between parts related to flushing and parts related to haptic effects in the haptic signal. This allows the haptic rendering device to adapt the haptic signal to the specific type of haptic device, haptic device capabilities, and target body part in order to stay below the perception threshold and ensure the expected haptic experience.

[0067] The embodiments described below propose to provide information for enabling implementation based on the description Figure 3 The flushing effect in the encoded haptic signal of the MPEG haptic codec is identified by using the immersive scene interchange file format introduced in . The identification can take the form of a Boolean flushing flag in the syntax of the file format.

[0068] The first embodiment presents flushing marks at the band level of the data structure of the file format. In practice, flushing usually uses a low-frequency signal, so that low-frequency bands can be used to carry flushing, while high-frequency bands will still remain. Therefore, this will result in a haptic signal that combines the curve bands identified as flushing and the bands that include the haptic effect.

[0069] The second embodiment presents the washout symbol at the effect level, allowing both haptic effect and washout to be carried within the same band.

[0070] A third embodiment may use a specific track to carry only flushes, and identify the tactile signal as flushes through a flush mark. From an authoring perspective, this embodiment may be the most straightforward implementation.

[0071] For readability, the description of the embodiments is based on a human-readable version (Exchange) of the immersive file format. However, the same principles apply to the binary version of the format.

[0072] According to a first embodiment, Boolean information (i.e., a flush flag) is associated with a tactile band and determines whether the corresponding signal of the tactile band is a flush. When the flush flag is "true", the tactile signal of the tactile band is identified as a flush and needs to be adapted by the rendering device to remain below the perception threshold. Any type of band described above (transient, curve or wave) can be used to describe the signal. In this first embodiment, the entire tactile signal of the band is dedicated to flushing, so that no tactile signal can be carried using the band.

[0073] The first embodiment describes the immersive scene ( Figure 3300), the immersive scene description includes a tactile effect using a "washout" Boolean flag of a "Haptics_band" element according to the JSON scheme of Table 1.

[0074] Table 1

[0075]

[0076] Using this information, the rendering engine of the tactile rendering device will be able to perform the correct rendering of the signal corresponding to the washout effect, thereby staying below the perception threshold.

[0077] In streaming applications, the haptic rendering device can use this information to request information related to the following effect. In fact, knowing in advance what to render may be useful for driving a flushing effect. For example, flushing the effect completely to zero may not be useful if the following effect does not start at zero, because the next start position needs to be moved again.

[0078] Table 2 shows the Fig. 6A glTF of the haptic signal of the first effect 610 and flush 620 element TM Description. Effect 610 is defined as a regular linear curve in a first band. A second band defines a second effect, which is identified as a washout by the value of the washout flag ("washout": true). This way, the haptic renderer is informed that the portion is a washout and can adapt if necessary depending on the type of haptic device, the haptic device capabilities, and the target body part.

[0079] Table 2

[0080]

[0081]

[0082] A variation of this embodiment includes using a band type instead of a Boolean flag to identify a band as a washout band. Since washout is primarily used for kinesthetic data, the band data can be interpreted as, for example, a curve band or a transient band. If the effect in a band contains multiple keyframes, it defines a washout curve, and if the effect contains a single keyframe with a start timestamp of the washout, the rendering of the washout is left to the rendering engine based on the device capabilities. Table 3 shows the JSON schema for a tactile band according to this variation.

[0083] Table 3

[0084]

[0085] According to the second embodiment, Boolean information (ie, flush flag) is associated with a haptic effect and determines whether the corresponding signal of the haptic effect corresponds to a flush effect. Defined at a lower level than in the first embodiment, this will allow for finer granularity in the definition of the effect.

[0086] The second embodiment can be described in the immersive scene ( Figure 3 300), the immersive scene description includes a tactile effect using a "washout" Boolean flag of a "Haptics_effect" element according to the JSON scheme of Table 4.

[0087] Table 4

[0088]

[0089] A flush effect can be defined as null in the sense that only a start position and an end keyframe can be specified. This indicates to the renderer that the flush effect should be performed between these two points, but the way in which the effect will be rendered is left up to the haptic renderer.

[0090] In a variant embodiment, the existing effect_type attribute is used instead of using a separate Boolean attribute to signal the flush effect. In Table 5, we provide an example of an implementation where an enumeration value, such as "Washout", is added to the existing effect_type attribute to signal that the effect is a flush effect. This solution allows the reuse of existing fields and therefore reduces the amount of data added to the immersive scene description file.

[0091] Table 5

[0092]

[0093] In a variant embodiment, additional information about the reference device can be added. The first information that can be added is the workspace center (i.e. the central point targeted by the flushing system). Although for 3DoF devices it is usually located at [0,0,0], it can also be located elsewhere. This is the case if the workspace is asymmetric or if the device has custom capabilities. The second piece of information that can be added is the maximum velocity that the device cannot overcome when flushing. It depends on the target body part that has been indicated. In most cases, flushing is performed for motor or kinesthetic devices, but the same principle can apply to thermal devices. The unit of temperature is °C / s.

[0094] These variant embodiments can be described in the immersive scene ( Figure 3300), the immersive scene description includes a tactile effect using a ""workspace_center"" element and a ""washout_max_speed"" element according to the JSON scheme of Table 6.

[0095] Table 6

[0096]

[0097]

[0098] In the MPEG haptic codec, information related to the working space center and the maximum flushing speed is related to the reference system. Similar information may be available on the rendering side, allowing adaptation to be performed based on the capabilities of the haptic device.

[0099] In another variant embodiment, an additional parameter of the flush indicates how much time is available to apply the flush. This can be determined, for example, based on the subsequent effect.

[0100] Figure 7 An example flow chart of a process for rendering a haptic feedback description file according to an embodiment is illustrated.Such a process 700 is, for example, implemented in a haptic rendering device 100 and executed by a processor 101 of such a device.

[0101] In step 710, the processor obtains a description of the immersive scene ( Figure 1 Of these, 191 Figure 3 This can be done, for example, by receiving it from a server via a communication network, by reading it from an external storage device or local memory, or by any other means. The processor analyzes the scene description file in order to extract the haptic objects ( Figure 1 192 of the haptic object, the haptic object allowing determination of parameters related to the haptic effect to be rendered, more particularly including a haptic washout flag associated with the haptic signal.

[0102] In step 720, if the tactile flushing flag indicates that the corresponding tactile signal is a flushing effect, the processor executes step 730. If this is not the case, it jumps directly to step 750. As described above, according to the use of Figure 3 In different embodiments of the file formats described in Tables 1 to 4, flush markers may be presented at different levels of immersive scene description.

[0103] In step 730 , the processor obtains information related to the type of haptic device, the capabilities of the haptic device, and the target body part in which the effect is to be rendered.

[0104] In step 740, the haptic signal is adapted to be rendered as a flush. In other words, based on the information obtained in step 730, the value of the haptic signal is modified to remain below the perception threshold. For example, Figure 6B The flushing effects 620 and 640 described in the above are generated for a specific device with a given flushing maximum speed. For other devices (e.g., different rotating platforms with different centers of rotation), the maximum speed that will not be noticed by the user may be different, such as Fig. 8A and 8B as shown in .

[0105] In step 750 , the original haptic signal or the adapted haptic signal is provided to a haptic device for rendering.

[0106] Therefore, the haptic effect is rendered according to the additional information of the haptic feedback.

[0107] Fig. 8A According to the embodiment, Fig. 6A FIG. 1 is a second example of a haptic signal adapted to a force feedback haptic device. The haptic signal includes three forward motion effects 810, 830, 850 and two flush effects 820, 840. This example illustrates a situation where the perception threshold is higher than expected, so that the flush effects 820 and 840 can be accelerated to be shorter while remaining unnoticed, thereby producing flush effects 821 and 841.

[0108] Figure 8B The diagram shows the Fig. 6A A third example of adapting a tactile signal to a force feedback tactile device. Fig. 8A Similarly, the haptic signal includes three forward motion effects 810, 830, 850 and two flush effects 820, 840. This example illustrates a situation where the perception threshold is lower than expected, such that the slope of the flush effect needs to be reduced to remain below the perception threshold, resulting in flush effects 822 and 842.

[0109] These adaptations are usually Figure 7 The rendering process is completed at step 740 and allows the flushing speed (or slope) to be adapted according to the rendering capabilities to ensure a good tactile experience.

[0110] Although the various embodiments have been described separately, any combination of the embodiments can be made while complying with the principles of this disclosure.

[0111] While the embodiments relate to haptic effects, those skilled in the art will appreciate that the same principles can be applied to other effects such as, for example, sensory effects, and thus would include smell and taste. The principles can also be applied to geolocation, such as on a virtual reality motion platform (e.g., an omnidirectional treadmill). Thus, appropriate syntax will determine the appropriate parameters associated with these effects.

[0112] References to "one embodiment" or "an embodiment" or "an implementation" or "implementation" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearance of the phrases "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" and any other variations thereof appearing in various places throughout the specification are not necessarily all referring to the same embodiment.

[0113] Furthermore, the application or its claims may refer to "determining" various pieces of information. Determining information may include, for example, one or more of: estimating information, calculating information, predicting information, or retrieving information from a memory.

[0114] Furthermore, the application or its claims may involve "obtaining" various pieces of information. As with "accessing," obtaining is intended to be a broad term. Obtaining information may include, for example, one or more of: receiving information, accessing information, or retrieving information (e.g., retrieving information from a memory or optical media storage device). Furthermore, "obtaining" is generally involved in one way or another during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0115] It is to be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," the use of any of the following " / ," "and / or," and "at least one of..." is intended to encompass selecting only the first listed option (A), or selecting only the second listed option (B), or selecting both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such wording is intended to encompass selecting only the first listed option (A), or selecting only the second listed option (B), or selecting only the third listed option (C), or selecting only the first listed option and the second listed option (A and B), or selecting only the first listed option and the third listed option (A and C), or selecting only the second listed option and the third listed option (B and C), or selecting all three options (A and B and C). As will be readily apparent to one of ordinary skill in this and related arts, this may be extended to as many items as listed.

Claims

1. A method comprising: - obtaining tactile data representing tactile feedback, the tactile data comprising a tactile signal for rendering the tactile feedback and information representing a flushing effect, - when the information representative of a flushing effect indicates that the haptic signal is a flushing effect, adapting the haptic signal to remain below a perception threshold, and - Provides adapted tactile signals.

2. The method according to claim 1, wherein the haptic signal is carried on a frequency band represented by haptic data allowing reconstruction of a portion of the signal corresponding to a selected range of frequencies, wherein The information representative of the washout effect is associated with a frequency band and is applicable to the associated frequency band.

3. The method according to claim 1, wherein the haptic signal is carried on a frequency band represented by haptic data allowing reconstruction of a portion of the signal corresponding to a selected range of frequencies, wherein The portion of the signal is further temporally divided into a group of haptic effects, and wherein information representative of the flushing effect is associated with the haptic effect and applicable to the associated haptic effect.

4. The method according to any one of claims 1 to 3, wherein the information representing the flushing effect is represented by a Boolean flag. 5 . The method according to claim 1 , wherein the information representing the flushing effect is represented by an enumeration value.

6. The method according to any one of claims 1 to 5, wherein the glTF-based TM The flushing effect information is represented using standard JSON syntax.

7. A method comprising: - encoding haptic data representing haptic feedback, the haptic data comprising a haptic signal and information representing a flushing effect, the information indicating that the haptic signal is a flushing effect, and - Provides encoded tactile data.

8. The method according to claim 7, wherein the haptic signal is carried over a frequency band represented by haptic data allowing reconstruction of a portion of the signal corresponding to a selected range of frequencies, wherein The information representative of the washout effect is associated with a frequency band and is applicable to the associated frequency band.

9. The method of claim 7, wherein the haptic signal is carried over a frequency band represented by haptic data allowing reconstruction of a portion of the signal corresponding to a selected range of frequencies, wherein The portion of the signal is further temporally divided into a group of haptic effects, and wherein information representative of the flushing effect is associated with the haptic effect and applicable to the associated haptic effect.

10. The method according to any one of claims 7 to 9, wherein the information representing the flushing effect is represented by a Boolean flag.

11. The method according to any one of claims 7 to 9, wherein the information representing the flushing effect is represented by an enumeration value.

12. The method according to any one of claims 7 to 11, wherein a glTF-based TM The flushing effect information is represented using standard JSON syntax.

13. A device comprising a processor configured to: - obtaining tactile data representing tactile feedback, the tactile data comprising a tactile signal for rendering the tactile feedback and information representing a flushing effect, - when the information representative of a flushing effect indicates that the haptic signal is a flushing effect, adapting the haptic signal to remain below a perception threshold, and - Provides adapted tactile signals.

14. The device of claim 13, wherein the haptic signal is carried over a frequency band represented by haptic data allowing reconstruction of a portion of the signal corresponding to a range of selected frequencies, wherein The information representative of the washout effect is associated with a frequency band and is applicable to the associated frequency band.

15. The device of claim 13, wherein the haptic signal is carried over a frequency band represented by haptic data allowing reconstruction of a portion of the signal corresponding to a selected range of frequencies, wherein The portion of the signal is further temporally divided into a group of haptic effects, and wherein information representative of the flushing effect is associated with the haptic effect and applicable to the associated haptic effect.

16. The device according to any one of claims 13 to 15, wherein the information representing the flushing effect is represented by a Boolean flag.

17. The apparatus according to any one of claims 13 to 15, wherein the information representing the flushing effect is represented by an enumeration value.

18. The device according to any one of claims 13 to 17, wherein the glTF-based TM The flushing effect information is represented using standard JSON syntax.

19. A device comprising a processor configured to: - encoding haptic data representing haptic feedback, the haptic data comprising a haptic signal and information representing a flushing effect, the information indicating that the haptic signal is a flushing effect, and - Provides encoded tactile data.

20. The device of claim 19, wherein the haptic signal is carried over a frequency band represented by haptic data allowing reconstruction of a portion of the signal corresponding to a range of selected frequencies, wherein The information representative of the washout effect is associated with a frequency band and is applicable to the associated frequency band.

21. The device of claim 19, wherein the haptic signal is carried over a frequency band represented by haptic data allowing reconstruction of a portion of the signal corresponding to a range of selected frequencies, wherein The portion of the signal is further temporally divided into a group of haptic effects, and wherein information representative of the flushing effect is associated with the haptic effect and applicable to the associated haptic effect.

22. The device according to any one of claims 19 to 21, wherein the information representative of the flushing effect is represented by a Boolean flag.

23. The apparatus according to any one of claims 19 to 21, wherein the information representing the flushing effect is represented by an enumeration value.

24. The apparatus according to any one of claims 19 to 23, wherein the glTF-based TM The flushing effect information is represented using standard JSON syntax.

25. A computer program comprising program code instructions for implementing the method according to any one of claims 1 to 12 when executed by a processor.

26. A non-transitory computer readable medium comprising program code instructions for implementing the method according to any one of claims 1 to 12 when executed by a processor.

27. A signal comprising haptic data representing haptic feedback, the haptic data comprising a haptic signal and information representing a flushing effect, the information indicating that the haptic signal is a flushing effect, the haptic data being encoded according to the method of any one of claims 7 to 12.

28. A non-transitory computer-readable medium storing tactile data for tactile feedback, the tactile data comprising a tactile signal and information representing a flushing effect, the information indicating that the tactile signal is a flushing effect, the tactile data being encoded according to the method of any one of claims 7 to 12.

Images