Method and apparatus of encoding / decoding haptic data

Abstract

A method and apparatus for encoding / decoding tactile signals representing tactile effects are provided. The tactile signals are encoded according to a hierarchical representation comprising a plurality of layers (11 to 14), with different levels associated with each layer. Encoding of the tactile signals involves encoding a first set of basic signals (111, 112) comprising one or more basic signals in a first layer (11). First data representing each basic signal of the first set is encoded in the first layer, each basic signal being represented in a defined space. In addition to the first data, second data representing a modulation of each basic signal is encoded in the first layer. Third data representing the defined space is then encoded in a layer different from the first layer (11), having a hierarchy higher than the first layer.

Description

Technical Field

[0001] The present invention generally relates to haptic data compression, and more particularly, to methods and apparatus for encoding / decoding data representing haptic effects. Prior Art

[0002] This section is intended to introduce the reader to various aspects of the technology that may be related to various aspects of at least one exemplary embodiment of the present invention described and / or claimed below. It is believed that this discussion will help provide the reader with background information to facilitate a better understanding of the various aspects of the present invention.

[0003] Haptics broadly refers to any technology that recreates the sense of touch in a user interface by applying force, vibration, motion, and other sensations to provide information to the end user.

[0004] Haptic feedback covers a wide range of possible stimulation implementation methods, but is mainly divided into perceptual tactile technology and kinesthetic tactile technology: Perceptual tactile feedback refers to sensations such as vibration, friction or micro-deformation, while kinesthetic tactile feedback refers to the sensation of force, which can stimulate both mechanical stimulation and stimulation related to the position and movement of the body.

[0005] Haptic feedback, or haptic effects, are achieved using haptic devices (also known as haptic actuators). For example, vibrotactile effects can be achieved using haptic devices such as eccentric rotating mass (ERM), linear resonant actuators (LRA), and wide-bandwidth actuators such as voice coil motors (VCM) or piezoelectric actuators (PZT). Kinesthetic effects can be achieved by applying a force that resists limb movement through an actuator, an effect felt only in the muscles and tendons, not the skin. Other examples of haptic devices include resistive feedback devices, active force feedback devices, and skin indentation devices.

[0006] Haptic feedback has been increasingly incorporated into portable electronic devices, known as "handheld devices" or "portables," such as cell phones, smartphones, and portable gaming devices. For example, some portable gaming applications can vibrate in a manner similar to control devices (e.g., joysticks, etc.) used with larger gaming systems configured to provide haptic feedback. Furthermore, devices such as cell phones and smartphones can provide various alerts to users through vibration. Furthermore, haptic effects can be used to simulate "real-world" dynamic events, such as the feel of a bouncing ball in a video game.

[0007] In order to present a haptic effect or haptic feedback on the end user side, data and / or signals representing the haptic effect to be presented must be encoded and then transmitted to the haptic device.

[0008] In the field of haptic effects technology, the encoding of tactile signals representing tactile effects mainly follows two different methods, namely: Encode the tactile signal in a Pulse Code Modulation (PCM) file; or Encoding a set of parameters in the form of vector data allows reconstruction of tactile signals.

[0009] In the first case (encoding haptic signals in PCM files), CSV or WAV files can be used. This solution is memory-intensive, but it can store complex haptic signals. However, such encoding has the disadvantage of being difficult to modify once encoded. Furthermore, amplitude modulation cannot be easily separated from frequency modulation, which is a problem when using different types of actuators to render haptic signals (for example, voice coils can render both amplitude and frequency modulated signals, while LRAs are limited to amplitude modulated signals).

[0010] In the second case (parametric encoding, also known as vector encoding), the haptic signal is encoded as a combination of functions to be rendered by the rendering engine. This solution has the advantage that the haptic signal can be easily modified by the application at runtime. However, the disadvantage of this solution is that it assumes a fixed rendering model is implemented in the rendering engine.

[0011] In the past few years, several solutions based on vector tactile data encoding have been developed.

[0012] For example, Apple® has developed and proposed a file format called AHAP (Apple Haptic Audio Pattern). AHAP files are dictionary representations of haptic and audio patterns in JSON (JavaScript Object Notation) format and can be written in a text editor or any program configured to export content in JSON format. The AHAP format only supports encoding vibration haptic feedback, not other haptic effects (such as kinesthetic feedback). Furthermore, AHAP is limited to rendering by Apple®'s Core Haptic library, which is limited to rendering engines running on iOS®-compatible devices.

[0013] Immersion Corporation has developed and proposed another file format, abbreviated as IVS (Immersion Vibration Source). IVS files are formatted in Extensible Markup Language (XML). The IVS format only supports encoding vibration haptic feedback, not other haptic effects (such as kinesthetic feedback).

[0014] Existing formats for encoding haptic data have many limitations and are unable to encode the various types of haptic effects that users desire to increase the user experience in an immersive manner. Summary of the Invention

[0015] The following section presents a simplified summary of at least one exemplary embodiment in order to provide a basic understanding of some aspects of the present invention. This summary is not an extensive overview of the exemplary embodiments. It is not intended to identify key or critical elements of the embodiments. The following summary merely presents some aspects of at least one of the exemplary embodiments in a simplified form as a prelude to the more detailed description provided elsewhere herein.

[0016] According to a first aspect of the present invention, a method for encoding a haptic signal representing a haptic effect in a bitstream of encoded haptic signal data is provided. To achieve this, a hierarchical representation of the haptic signal is encoded, the hierarchical representation comprising a plurality of layers, a level associated with each of the plurality of layers, wherein the encoding comprises: In a first layer of the plurality of layers, encoding first data representing modulation of at least one base signal; and Second data representing a tactile perception modality is encoded in a layer having a higher level than the first layer among the plurality of layers.

[0017] In an exemplary embodiment, the method further includes encoding, in the first layer, a first set of elementary signals including at least one elementary signal by encoding third data representing each elementary signal of the first set of elementary signals, wherein each elementary signal is represented in a determined space, the determined space being based on the tactile perception modality.

[0018] In an exemplary embodiment, the third data includes: Data indicating the type of at least one basic signal; and / or The data represents the phase of at least one basic signal, and the first data includes data representing the coordinates of a plurality of points in a certain space.

[0019] In another exemplary embodiment, the plurality of layers further include a second layer having a level higher than that of the first layer, a third layer having a level higher than that of the second layer, and a fourth layer having a level higher than that of the third layer.

[0020] In another exemplary embodiment, encoding of the tactile signal further includes: In a second layer, a second set of sequences including at least one or more sequences of modulated signals is encoded by encoding, in each sequence of the second set, first information representing a sequence of one or more modulated signals of each sequence, each modulated signal being obtained from a first set of base signals modulated from second data associated with the base signals; In the third layer, a third group of first combinations including at least one first combination of one or more sequences of the second group is encoded by encoding, for each first combination, second information indicating the number of sequences included in each first combination; and In the fourth layer, the second combination of the one or more first combinations of the third group is encoded by encoding third information indicating the number of the first combinations included in the second combination.

[0021] In another exemplary embodiment, the at least one sequence of the second set includes a plurality of modulated signals obtained from different types of base signals.

[0022] In another exemplary embodiment, the at least one basic signal corresponds to a constant signal.

[0023] In another exemplary embodiment, encoding the haptic signal further includes encoding metadata in at least one of the plurality of layers, at least a portion of the metadata including data representing an application programming interface call associated with the at least one basic signal.

[0024] According to a second aspect of the present invention, a method for decoding a tactile signal from encoded tactile signal data is provided. The method includes decoding a layered representation of the tactile signal, the layered representation including a plurality of layers, a level associated with each of the plurality of layers, the decoding comprising: decoding, from a first layer of the plurality of layers, modulated first data representing modulation of at least one elementary signal; and

[0025] Second data representing a tactile perception modality is decoded from a plurality of layers having a hierarchy level higher than that of the first layer.

[0026] In an exemplary embodiment, decoding further includes decoding a first set of elementary signals comprising at least one elementary signal from the first layer by decoding third data representing each elementary signal of the first set, wherein each elementary signal is represented in a determined space, the determined space being based on a tactile perception modality.

[0027] In an exemplary embodiment, decoding the tactile signal further includes: decoding, from the second layer, a second set of sequences comprising at least one or more sequences of modulated signals by decoding the first signal representing one or more modulated signals in each sequence of the second set, each modulated signal being obtained from a first set of base signals modulated from second data associated with the base signals; Decoding a third group of first combinations comprising at least one first combination of one or more sequences of the second group from the third layer by decoding, for each first combination, second information indicating the number of sequences comprised in each first combination; and From the fourth layer, the second combination of the one or more first combinations of the third group is decoded by decoding the third information indicating the number of the first combinations contained in the second combination.

[0028] In another exemplary embodiment, decoding the haptic signal further includes decoding metadata from at least one layer of the plurality of layers, at least a portion of the metadata including data representing an application programming interface call associated with the at least one basic signal.

[0029] According to a third aspect of the present invention, a device is provided for encoding a tactile signal representing a tactile effect in a bit stream of encoded tactile signal data, wherein the device includes a memory associated with at least one processor configured to implement the method according to the first aspect of the present invention.

[0030] According to a fourth aspect of the present invention, a device is provided for decoding a tactile signal representing a tactile effect from a bit stream of encoded tactile signal data, wherein the device includes a memory associated with at least one processor configured to implement the method according to the second aspect of the present invention.

[0031] According to a fifth aspect of the present invention, a bit stream is provided, the bit stream having data representing at least one haptic signal, the haptic signal representing at least one haptic effect, the data comprising: In a first layer of a plurality of layers, each layer having an associated hierarchy level, modulated first data representing at least one basic signal; and

[0032] In a layer of a plurality of layers having a higher level than the first layer, second data representing a tactile perception modality.

[0033] In an exemplary embodiment, the bit stream further includes third data representing each elementary signal in a group of elementary signals including at least one elementary signal, each elementary signal being represented in a determined space.

[0034] According to a sixth aspect of the present invention, there is provided a computer program product comprising instructions, which, when executed by one or more processors, causes the one or more processors to perform the method according to the first aspect of the present invention.

[0035] According to a seventh aspect of the present invention, there is provided a non-transitory storage medium carrying program code instructions for executing the method according to the first aspect of the present invention.

[0036] According to an eighth aspect of the present invention, there is provided a computer program product comprising instructions, which, when executed by one or more processors, causes the one or more processors to perform the method according to the second aspect of the present invention.

[0037] According to a ninth aspect of the present invention, there is provided a non-transitory storage medium carrying program code instructions for executing the method according to the second aspect of the present invention.

[0038] According to a tenth aspect of the present invention, a method for presenting tactile feedback is provided, the method comprising: receiving a data bit stream representing the tactile feedback; selecting a tactile rendering model based on information representing a tactile perception modality (corresponding to a type of tactile feedback), the information being included in a bit stream; generating a haptic signal by processing at least a portion of the data included in the bitstream using a haptic rendering model; Data representing the haptic signal is transmitted to at least one actuator to present haptic feedback.

[0039] In an exemplary embodiment, the presenting method further includes: receiving data representing a position of at least a portion of a body, the position being acquired at a determined frequency; Upsample the position, Among them, the generation of tactile signals is further based on the upsampling position.

[0040] In another exemplary embodiment, the upsampling is based on characteristics of at least one actuator.

[0041] In a further embodiment, processing at least a portion of the data included in the bitstream using the haptic rendering model is based on characteristics of at least one actuator.

[0042] According to an eleventh aspect of the present invention, there is provided a device for presenting tactile feedback, wherein the device comprises a memory associated with at least one processor, the processor being configured to implement the method according to the tenth aspect of the present invention.

[0043] The specific nature of at least one exemplary embodiment as well as other objects, advantages, features and uses of the at least one exemplary embodiment will become apparent from the following description of the examples taken in conjunction with the accompanying drawings. Simple diagram description

[0044] Reference will now be made, by way of example, to the accompanying drawings which show exemplary embodiments of the present application, in which: Figure 1 shows a schematic diagram of a hierarchical representation of data representing one or more tactile signals according to at least one exemplary embodiment; Figure 2 shows data representing a haptic signal encoded into a haptic effect file having a layered text format according to at least one exemplary embodiment; Figure 3 shows data representing a haptic signal encoded into a haptic effect file having a binary format according to at least one exemplary embodiment; Figure 4 shows a basic signal sequence encoded according to the hierarchical representation of Figure 1 according to at least one exemplary embodiment; Figure 5 shows an exemplary sequence of modulated base signals encoded according to the hierarchical representation of Figure 1 according to at least one exemplary embodiment; Figure 6 shows a schematic block diagram of the steps of a method for encoding a tactile signal according to the hierarchical representation of Figure 1 according to at least one exemplary embodiment; Figure 7 shows a schematic block diagram of the steps of a method of decoding tactile data representing an encoding according to the layered representation of Figure 1 according to at least one exemplary embodiment; Figure 8 shows a schematic block diagram of an example system implementing various aspects and exemplary embodiments; Figure 9 shows a schematic block diagram of an example of a rendering system and a corresponding method of rendering one or more haptic effects according to at least one exemplary embodiment; FIG10 illustrates a position upsampling method according to an exemplary embodiment; and FIG. 11 illustrates a schematic block diagram of steps of a method for presenting one or more haptic effects according to at least one example embodiment.

[0045] Like reference numerals may be used in different drawings to identify similar components. Implementation Method

[0046] At least one of the exemplary embodiments will be described more fully hereinafter with reference to the accompanying drawings, which illustrate examples of at least one of the exemplary embodiments. However, the exemplary embodiments may be embodied in many alternative forms and should not be construed as limited to the examples set forth herein. Therefore, it should be understood that there is no intention to limit the exemplary embodiments to the particular forms disclosed. On the contrary, this disclosure is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

[0047] At least one of these aspects generally relates to a method and apparatus for encoding one or more haptic signals, each haptic signal representing a haptic effect. Each haptic signal is advantageously encoded according to a hierarchical representation comprising a plurality of layers, such as two, three, four, or more layers. A specific level is associated with each layer, such as the lowest level being associated with the first layer and the level increasing with each layer. Encoding the haptic signal includes encoding first data representing a modulated at least one elementary signal in a first layer. Second data representing a haptic perception modality (e.g., a haptic perception modality corresponding to a type of haptic feedback, such as vibrotactile feedback, force feedback, stiffness feedback, texture feedback, or temperature feedback) is encoded in a layer different from the first layer and at a higher level than the first layer. The specific space used as the representation space for the elementary signals depends on the haptic perception modality.

[0048] The at least one basic signal corresponds, for example, to a constant signal stored or hard-coded in a rendering engine that receives encoded data representing the haptic signal.

[0049] According to another example, third data representing the at least one basic signal is further encoded into the first layer together with the first data.

[0050] Briefly stated, the present invention relates to encoding / decoding of haptic data to enable viewing of one or more haptic effects by a rendering engine.

[0051] The tactile perception modality is encoded in a layer whose level is higher than the level of the layer into which data representing the modulation and / or the basic signals used to represent the tactile signal are encoded, which makes it possible to optimize the decoding of the data contained in the received bit stream and representing the tactile signal.

[0052] In practice, the rendering engine can first determine which tactile modality is associated with the tactile signal encoded in the bitstream and whether it is necessary to decode the remaining data contained in the lower layers. For example, if the viewing device (also known as an actuator) communicating with the rendering engine cannot reproduce the tactile modality of the encoded tactile signal, the rendering engine does not need to decode the data contained in the bitstream and render the associated tactile signal.

[0053] This encoding format can optimize / reduce the processing time of the browsing phase based on the characteristics of the haptic device (actuator) communicatively coupled to the rendering engine.

[0054] FIG. 1 illustrates a schematic diagram of a layered structure 100 for representing one or more haptic signals according to at least one example embodiment.

[0055] Data representing haptic signals can be encoded by an encoder into a determined format according to a hierarchical structure or representation 100 described below. For example, the data can be encoded in a haptic effect file or codec, which can have one of many different formats. In some embodiments, the haptic effect file can have a hierarchical text format corresponding to, for example, the Extensible Markup Language ("XML") format, the JavaScript Object Notation ("JSON") format, or another markup language ("YMAL"). In some other embodiments, the haptic effect file can have a binary format.

[0056] The hierarchical representation 100 of the non-limiting embodiment shown in Figure 1 includes four layers 11, 12, 13, and 14, where the first layer 11 has the lowest level, the second layer 12 has a level higher than the level of the first layer 11, the third layer 13 has a level higher than the level of the second layer 12, and the fourth layer 14 has the highest level, and the level of the fourth layer 14 is higher than the level of the third layer 13.

[0057] The number of layers can be different from 4, for example, 2, 3, or 5. For example, when the number of layers is 2, the hierarchical representation of the encoded haptic signal includes only the first layer 11 and the second layer 12. When the number of layers is 2, the second layer 12 can, for example, correspond to the header of a data stream or bit stream, and the first layer can, for example, correspond to the payload data of the data stream or bit stream. Figure 3 illustrates such an example.

[0058] Layers 11 to 14 of the layered structure or representation 100 include encoded data of haptic signals representing haptic feedback or effects to be processed by a rendering engine.

[0059] The tactile signal is encoded as a tactile material or tactile file in a layered format (hereinafter referred to as the second combination), including one or more tactile perceptions or tactile tracks (hereinafter referred to as the first combination), each tactile perception (or tactile track) includes one or more tones or tactile bands (hereinafter referred to as modulation signal sequences), and each tonality (or tactile band) contains one or more sequentially arranged tactile notes or tactile waves (hereinafter referred to as modulation signals, which are examples of modulation signals obtained or encoded in the form of one or more basic signals and related modulation parameters or functions).

[0060] The first layer 11 includes first data representing modulation of one or more basic signals.

[0061] According to an exemplary embodiment, the first data represents a modulation to be applied to a single constant signal. The third data representing the single constant signal is not encoded into the layered representation 100, but is stored in the memory of the rendering engine that receives the signal. The encoded haptic signal is in the form of a bitstream of encoded data.

[0062] According to another exemplary embodiment, the first data represents a modulation to be applied to one or more elementary signals forming the first set of elementary signals 111, 112, and the third data representing each elementary signal encodes a first layer of modulation using the first data representing the modulation.

[0063] Each elementary signal is represented in a definite space, which can correspond to one of the following spaces: time space, spatial space (e.g. one-dimensional (1D), two-dimensional (2D) or three-dimensional (3D) space), angular space or any space based on a measurable dimension (such as velocity, acceleration, gravity, magnetic field, brightness, temperature, etc.).

[0064] FIG2 illustrates examples of basic signals 111 , 112 and associated modulation.

[0065] The third data representing the elementary signals encoded in the first layer 11 include, for example, for each elementary signal:

[0066] - data indicating the type of basic signal, e.g. sinusoidal, rectangular, triangular, ramp, continuous, sawtooth; the type of basic signal is identified, e.g., by an identifier encoded on 3 or 4 bits; and / or

[0067] - data representing the phase of the fundamental signal; and / or

[0068] - Data that represents basic signal characteristics, such as frequency, amplitude, and offset.

[0069] The first data associated with the basic signal corresponds to, for example, data representing the coordinates of each of a plurality of points (also referred to as a key frame) expressed in a determined space.

[0070] For example, when the determined space corresponds to a temporal space and the type of modulation to be applied to the base signal is amplitude modulation, the coordinates of a point include an abscissa corresponding to time "t" and a ordinate corresponding to the amplitude value. The modulation applied to the base signal during the rendering phase is based on first data, namely the coordinates of the points (or keyframes), and an interpolation function (e.g., a third-order polynomial function) determined using the points encoded as keypoints in the first layer, as explained in more detail below with reference to FIG. The interpolation function corresponds to a parametric function, for example, using the coordinates of the plurality of points encoded in the first layer 11 as parameters, in the form of the first data.

[0071] According to one variant, the first data associated with the base signal correspond to data representing the modulation function, which correspond to third data representing, for example, the base signal, ie its type, phase and characteristics, for example as frequency and / or amplitude.

[0072] A set of first data is advantageously associated with each elementary signal 111, 112 encoded in the first layer 11. The set of second data may, for example, represent an amplitude modulation and / or a frequency modulation.

[0073] According to an exemplary embodiment, metadata may be associated with each elementary signal 111, 112 and may be encoded in the first layer 11 along with the first data (and, when encoded in the first layer, along with the third data). For example, the metadata includes information indicating the length of each elementary signal, the length being expressed according to the determined space (e.g., when space corresponds to a temporal space or length corresponds to a distance, the length corresponding to a duration when space corresponds to a spatial space) and information indicating a starting point in the determined space, the starting point being defined by a horizontal coordinate in the determined space.

[0074] The first and optional third data can, for example, be included or transmitted in frequency bands each having a defined frequency band, such as 70, 100, 500, or 1000 Hz. The number of frequency bands can, for example, be between 2 and 9, or between 1 and any number of frequency bands. Each frequency band can, for example, include a portion of data representing a haptic effect to be rendered, with the entire haptic effect rendered, for example, by combining (e.g., summing) all frequency bands.

[0075] According to a first particular embodiment corresponding to the encoding of tactile data based on quantization, the first layer comprises:

[0076] a first frequency band comprising first data representing a representation of a plurality of points (or keyframes) for amplitude modulation of a single constant signal stored in a rendering engine, third data representing the single constant signal not being encoded in the bitstream; the length of the modulation (in representation space) corresponding to the length of the haptic signal to be viewed, for example 65 seconds; and

[0077] - one or more second frequency bands, each second frequency band comprising first data representing a modulation point (or keyframe) to be applied to a single basic signal, and third data representing a single basic signal encoded in the second frequency band (the type of the basic signal corresponding to the example is always a sinusoidal signal); the length of the basic signal (in the representation space) corresponds to the length of the tactile signal to be browsed, for example 65 seconds.

[0078] The data contained in the frequency band represents the tonality.

[0079] According to a second particular embodiment corresponding to the encoding of vector-based tactile data, the first layer comprises:

[0080] - The first band, including data representing Application Programming Interface (API) calls; and

[0081] - one or more second frequency bands, each second frequency band comprising first data representing modulation points (or keyframes) to be applied to one or more elementary signals, third data representing each elementary signal being encoded into the second frequency band, the elementary signals being, for example, of any type and any length.

[0082] The data contained in the frequency band represents the tonality.

[0083] The second layer 12 includes first information representing a second set of one or more sequences 121, 122 of modulated signals. Each sequence 121, 122 includes one or more modulated signals sequentially arranged in a certain space within the sequence.

[0084] The modulated signals correspond to the base signals 111 and 112 encoded in the first layer, to which the modulation defined by the first data has been applied, except for the example of encoding the first frequency band of quantized haptic data. The modulated signals correspond to the modulation results applied to the corresponding base signals.

[0085] The amplitude modulation is obtained by multiplying a modulation function obtained by interpolating points encoded with the first data by the basic signal.

[0086] Frequency modulation can be achieved, for example, by using linear frequency modulation.

[0087] The first information is associated with each modulated signal sequence contained in the second layer 12. The first information encoded in the second layer 12 and associated with each sequence advantageously represents that it is contained in this sequence and forms the sequence.

[0088] The first information is encoded, for example, using the syntax and / or structure of a haptic effect file or codec. The order of the modulated signals forming the sequence is obtained, for example, by encoding the elements associated with the modulated signals relative to the sequence to which they belong and in an order corresponding to the order of each modulated signal. During the decoding phase, the first information is implicitly derived from the syntax and / or structure of the haptic effect file or codec.

[0089] According to a variant, the first information corresponds to explicit data or information identifying and pointing to each modulated signal coded respectively with first and third data representing the modulation parameters and the associated elementary signal.

[0090] According to an exemplary embodiment, metadata may be associated with each sequence 121, 122 and may be encoded into the second layer 12 along with the first information. For example, the metadata includes information indicating the length of each sequence (based on the determined spatial representation), information indicating the body part to which the haptic effect represented by the sequence is to be applied, and information indicating the body part.

[0091] The third layer 13 includes a third set of second information representing one or more first combinations 131, 132 of one or more sequences 121, 122 of the modulated signal. Each first combination 131, 132 corresponds to a set of one or more sequences encoded in the second layer, which are combined according to a specific function (summation, weighted summation, filtering, average summation, compression).

[0092] The second information is associated with each first combination of sequences included in the third layer 13. The second information encoded in the third layer 13 and associated with each first combination advantageously indicates the number of sequences forming each first combination 131,132.

[0093] The second information is encoded, for example, using the syntax and / or structure of the haptic effect file or codec. The number of sequences included in the first combination is obtained, for example, by encoding elements associated with the sequences associated with the first combination to which they belong. During the decoding phase, the second information is implicitly derived from the syntax and / or structure of the haptic effect file or codec.

[0094] According to one variant, the second information corresponds to explicit data or information identifying and pointing to each sequence, each sequence being coded with data representative of the first combination.

[0095] According to an exemplary embodiment, metadata may be associated with each first combination 131, 132 and may be encoded into the third layer 13 along with the second information. For example, the metadata may include information indicating the type of addressed sensation associated with the haptic feedback (e.g., vibration, force feedback, temperature sensation, stiffness, texture), information indicating the part of the body targeted by the haptic feedback or effect represented by the first combination, and a plurality of information indicating the type of addressed sensation associated with the haptic feedback (e.g., vibration, force feedback, temperature sensation, stiffness, texture), information indicating the part of the body targeted by the haptic feedback or effect represented by the first combination, and information indicating the plurality of information indicating the addressed sensation.

[0096] The fourth layer 14 includes a third message representation representing a second combination 141 of one or more first combinations 131, 132 encoded in the third layer 13. The second combination 141 corresponds to a set of first combinations encoded in the third layer 13 that are to be encoded together to form the second combination 141.

[0097] The second combination corresponds to a haptic file representing a haptic effect or feedback encoded in a haptic effect file or codec.

[0098] The third information advantageously indicates the number of first combinations forming the second combination 141 .

[0099] The third information is encoded, for example, using the syntax and / or structure of the haptic effect file or codec. The number of first combinations included in the second combination is obtained, for example, by encoding the elements associated with the first combination relative to the second combination to which they belong. During the decoding phase, the third information is implicitly derived from the syntax and / or structure of the haptic effect file or codec.

[0100] According to one variant, the third information corresponds to explicit data or information identifying and pointing to each first combination coded with data representative of the second combination.

[0101] According to an exemplary embodiment, metadata may be associated with the second combination 141 and may be encoded into the fourth layer 14 together with the third information. For example, the metadata includes information indicating the body part to which the haptic effect represented by the second combination is to be applied and information indicating a rendering method.

[0102] Second data related to the haptic perception modality associated with the haptic effect to be rendered are advantageously encoded into the second layer 12, the third layer 13 or the fourth layer 14, for example as metadata.

[0103] The tactile perception modality corresponds to the type of tactile effect or tactile feedback to be presented. The tactile perception modality corresponds, for example, to one of the following:

[0104] -Vibration effect;

[0105] - force effect;

[0106] - Stiffness;

[0107] -texture;

[0108] -temperature.

[0109] The representation space of one or more elementary signals used to encode the tactile signal to be presented depends on the tactile perception modality. For example, vibration and force are associated with the temporal space, while stiffness and texture are associated with the spatial space.

[0110] According to an exemplary embodiment, metadata encoded into the first layer 11, the second layer 12, the third layer 13, or the fourth layer 14 includes data representing one or more application programming interface (API) calls, with a single API call associated with a single basic signal. The API call associated with the basic signal may be encoded in the following form:

[0111] - an identifier that identifies the API call stored in the rendering engine's memory; or

[0112] - A set of instructions to be executed by the rendering engine.

[0113] The use of an API rather than basic signals is determined by the rendering engine based on the existence of a communication channel (between the rendering engine and the haptic device (actuator)) that is capable of processing the API calls.

[0114] FIG. 2 illustrates data representing a haptic signal encoded into a haptic effect file 2 having a layered text format, according to at least one example embodiment.

[0115] According to an exemplary embodiment, haptic signals are encoded in the form of a JSON-compatible file or codec 2 according to a hierarchical structure or representation 100. FIG2 illustrates an example of such a JSON file 2 for illustration purposes only.

[0116] The block 21 JSON file of the elements (also called “keys” or “tags”) “m_version”, “m_description”, “m_HDFlag”, “m_time_unit”, “m_length_unit”, “m_volume”, “m_stiffness”, and “m_texture” includes metadata and associated values ​​representing a second combination of haptic signals encoded into the fourth layer 14 .

[0117] The second group includes only the first group, identified by element block 22, introduced by the element "m_vibration." The first group is encoded in the JSON file at a lower level than the second group, identified by and enclosed between the symbols "{" and "}." The JSON file elements "m_loop," "m_maximum," "m_speed," "m_volume," and "m_signalEvaluationMethod" correspond to the metadata and associated values ​​of the first group encoded in the third layer 13.

[0118] The first group, identified by the element "m_vibration," consists of only one modulated signal sequence, identified by the element "m_melodies." This sequence is encoded in the JSON file at a lower level than the first group. The blocks 23 of the JSON file's elements "m_maximum," "m_mute," and "m_volume" following the element "m_melodies" correspond to the metadata and associated values ​​of the modulated signal sequence encoded in the second layer 12.

[0119] Finally, the sequence identified by the element "m_melodies" includes two modulated signals 24 and 25, which are encoded in a hierarchical order below the hierarchical level of the sequence "m_melodies" in the JSON file.

[0120] The modulation signals 24 and 25 forming the sequence "m_melodies" are defined or encoded using the element following the element "m_notes" in the JSON file and are enclosed between the symbols "[" and "]".

[0121] Data representing the first modulated signal 24 of the sequence is first encoded or signaled in the "m_note" element block in the JSON file, followed by data representing the second modulated signal 25 (which follows the first modulated signal in sequence). According to this non-limiting example, the second information representing the order of each modulated signal of the sequence is obtained from the hierarchical structure of the JSON file itself, with elements defining the modulated signals attached to or associated with the sequence, with the second modulated signal 25 following the first modulated signal 24.

[0122] Each modulated signal 24, 25 includes metadata encoded with the first and third data in the first layer 11. The first metadata is identified or signaled by the element "m_length" corresponding to the length of the modulated signal (or similarly the length of the basic signal used to obtain the modulated signal), and the second metadata is identified by the element "m_startingPoint" corresponding to the origin of the modulated signal in a certain space (the origin or starting point of the modulated signal corresponds to, for example, a position on a time axis or a spatial line, depending on the space in which the basic signal is represented).

[0123] Each modulation signal 24, 25 is encoded or represented by a signal in a JSON file, the third data 241, 251 represents the basic signal identified by the component "m_underlyingSignal" in the JSON file, and the first data 242, 252 represents the modulation parameter identified using the component "m_amplitudeModulation" in the JSON file.

[0124] Regarding the first modulated signal 24, the basic signal is represented or signaled in the JSON file by third data 241 including the basic signal type (identified by the element "m_signalType", which has a value of 0, for example corresponding to a sine signal), the phase of the basic signal (identified by the element "m_phase"), the amplitude of the basic signal (identified by the element "m_amplitude"), the frequency of the basic signal (identified by the element "m_frequency") and the vertical offset of the basic signal (identified by the element "m_verticalOffset").

[0125] The first data 242 representing the modulation to be applied to the base signal is identified or signaled with the element "m_amplitudeModulation," which indicates that the modulation to be applied to the base signal is amplitude modulation. The metadata identified or signaled with the element "m_max" is associated with the modulation.

[0126] Then, two points or keyframes are encoded or signaled under the element "m_amplitudeModulation" in the JSON file, each of which is represented or signaled by a set of first data containing the coordinates of the point (identified by the element "m_time", corresponding to the horizontal coordinate and the element "m_value" corresponding to the vertical coordinate) and other parameters, for example, depending on the type of interpolation function to be used.

[0127] The interpolation function to be used in the decoding and / or browsing phase corresponds to, for example, a default interpolation function (e.g., a third-order polynomial function), which is known in advance by the encoder, decoder, and rendering engine, without the need to identify the default function in the JSON archive.

[0128] According to one variant, the interpolation function is a specific interpolation function identified in a JSON archive (with an identifier or parameters), for example as metadata associated with a higher-level modulation.

[0129] Regarding the second modulated signal 25 (encoded after the first modulated signal 24 in the JSON file), the basic signal is represented or signaled in the JSON file by third data 251 that is similar or identical to the third data of the first modulated signal.

[0130] The first data 252 representing the modulation to be applied to the base signal is identified or signaled by the element “m_amplitudeModulation”, which indicates that the modulation to be applied to the base signal is amplitude modulation.

[0131] In this particular example, the first data 252 of the second modulated signal 25 represents a function identified or signaled by the same elements as the base signal.

[0132] FIG. 3 illustrates data representing a haptic signal encoded into a haptic effect file 3 having a binary format, according to at least one example embodiment.

[0133] According to a second exemplary embodiment, the tactile signal is encoded in the form of one or more strings 31 , 32 of binary elements or characters according to a hierarchical structure or representation 100 .

[0134] Each binary string 31, 32 comprises, for example, a header 310, 320 and payload data 311, 321. The header advantageously has a higher level than the payload data and comprises second data representing a tactile perception modality associated with the tactile signal encoded in the string 31, 32.

[0135] According to a non-limiting example, the string includes data representing one first combination, i.e., string 31 includes data representing a first combination and string 32 includes data representing another first combination, and the two strings 31, 32 (i.e., two first combinations) form a second combination of tactile material corresponding to the tactile signal.

[0136] The header 310, 320 of each string 31, 32 includes fields, each field including a sequence of values ​​associated with variables or parameters corresponding to the third message and metadata associated with the second combination, the second message and metadata associated with the first combination, and the first message and metadata associated with the modulated signal sequence forming the first combination.

[0137] The payload 311 , 321 of each string 31 , 32 , respectively, includes fields, each field including a value associated with a variable or parameter corresponding to a modulated signal forming the first combination described by the data of each string 31 , 32 .

[0138] The data describing each modulated signal (ie, the first and second data) are encoded into the payloads 311 and 321 of the strings 31 and 32, respectively.

[0139] The end of the modulated signal sequence is signaled, for example, by a special character in the payload 311 or 321, and the subsequent sequence begins after the special character. For example, the special character is denoted as E_O_B and is encoded using the hexadecimal value "00FFFF".

[0140] According to another example, an amplitude equal to 0 may be interpreted as a special character when located at the end of a sequence.

[0141] According to certain embodiments, the end of a series of modulated signals is signaled in payloads 311, 321 using a special character labeled E_O_S and encoded with the value "00." A series of modulated signals corresponds to multiple modulated signals that share some common parameters (e.g., the same phase or the same amplitude of a base signal used to obtain the associated modulated signal), with the same parameters being encoded once for the series.

[0142] This format can encode both vector-based and quantized haptic data. When the data is a vector, this information is encoded in the header, for example, with Boolean encoding as a single bit (e.g., "0" for FALSE, "1" for TRUE). It supports encoding vector structures such as IVS and AHAP formats using a hierarchical representation 100.

[0143] This format can also encode specific signals, such as transients or API calls, with the data encoded serially in the payload and signaled in the header.

[0144] According to one embodiment, the haptic signal is encoded in the form of a single string of binary elements or characters according to a hierarchical structure or representation 100. According to this embodiment, the end of a first combination is signaled by a specific character in the stream, and the field or character following the special character corresponds to the first character or field of another first combination of one or more modulated signal sequences.

[0145] Compared with existing coding formats in the prior art, the proposed layered coding has various advantages, such as:

[0146] - being able to encode continuous and vibrotactile signals with the same encoding in the same tactile perception, corresponding to a first combination of one or more modulated signal sequences; and / or

[0147] - Capable of encoding both spatial-based and temporal-based tactile signals using the same encoding; and / or

[0148] -Ability to encode kinesthetic perception in spatial and temporal based methods;

[0149] - a modulation function capable of independently modulating different base signals, sequences of modulated signals, and a first combination of one or more sequences at runtime; and / or

[0150] - Ability to transcode AHAP format and / or IVS starting from the proposed layered coding format; and / or

[0151] - Ability to transcode IVS format starting from the recommended layered encoding format.

[0152] Convert between different file formats

[0153] The proposed layered format, along with Apple's AHAP format and Immersion's IVS format, encodes haptic signals in vectors, meaning they store the "drawing commands" to generate the signals instead of the signals themselves. This assumption allows for a one-to-one match between these three file formats to create a translator, whose goal is to convert AHAP or IVS files into files in the layered format described in this disclosure.

[0154] For example, regarding the AHAP data format, the known continuous parts are:

[0155] The sharpness between 0 and 1 is the frequency range between 65Hz and 300Hz (experimental results);

[0156] Strength is between 0 and 1, resulting in the feedback amplitude reaching the maximum value possible for the device; and

[0157] Start time and duration in seconds.

[0158] There is a direct match between these modulated signals presented in continuous and hierarchical format; they both store the same information.

[0159] Transients are stored at specific times with intensity and clarity (possibly corresponding to the signal's frequency). If the signal length is short enough, the transient can be matched with a new modulation signal containing one or more high-frequency cycles and high amplitude to represent this impulse. They can also be encoded using keyframes (representing the peak of the transient) or simply the points of modulation, applied to a continuous base signal. Transients can also be stored as API calls. The modulation function will also be a direct match between the proposed layered format and the AHAP format. The audio information stored in the file cannot be translated in the current layered format but can be managed by the application itself, which will play the two files in two different players: one for audio and the other for haptics.

[0160] Following the same principle, IVS files have a one-to-one match with the proposed layered format. Both encode the start time, duration, frequency (IVS stores window length in milliseconds, which can be directly converted to frequency: f = 1 / T), amplitude, and modulation amplitude of each modulated signal. The main difference here is that loops can be created within the timeline of an IVS file. However, this translates to simply replicating the contents of this loop N times, corresponding to the loop count. IVS also stores waveform types in a format that directly matches the proposed layered format.

[0161] The MagSweep effect can be interpreted as a normal modulated signal, but it will be driven by a constant frequency provided by the developer. During experimental testing, it was found that this effect was synthesized with a frequency of 170 Hz.

[0162] Finally, the last type is a waveform, which can be encoded like a keyframe within the proposed layered format, or a sound analysis can be performed on the archive to encode it as a frequency library within the archive. Multimedia integration can be the same as AHAP.

[0163] FIG. 4 illustrates a modulated signal sequence obtained from a base signal and encoded according to the layered representation of the present disclosure, according to at least one example embodiment.

[0164] FIG. 4 illustrates a modulated signal sequence 48 obtained from basic signals 41 , 42 , 43 modulated according to a determined modulation.

[0165] According to the non-limiting example of FIG4 , each elementary signal 41, 42, and 43 corresponds to a sine wave having a specific frequency (e.g., 87 Hz for the first elementary signal 41, 76 Hz for the second elementary signal 42 and 43, and 76 Hz for the third elementary signal 43), whose amplitude varies over time. For example, the amplitude ranges from a maximum value of 1 to a minimum value of -1. The representation space of the elementary signals corresponds to time, and the abscissa of the representation space of the elementary signals 41, 42, and 43 corresponds to the timeline.

[0166] The basic signal 41 starts at time 0, the basic signal 42 starts at time 0.025 seconds, and the basic signal 43 starts at time 0.05 seconds.

[0167] According to the exemplary embodiment of FIG. 4 , the elementary signals 41 to 45 have the same duration, with no gap between the end of a signal and the beginning of a subsequent signal.

[0168] The modulation parameters of each elementary signal 41, 42, 43 are represented by specific functions 44, 45, 46, respectively. The modulation function 44 associated with the first elementary signal 41, the modulation function 45 associated with the second elementary signal 42, and the modulation function 46 associated with the third elementary signal 43 are represented on the same timeline as the timelines of the elementary signals 41 to 43.

[0169] Each modulation function 44, 45, 46 is represented by a point or keyframe, with boldface annotations for the second modulation function 45. Each modulation function 44, 45, 46 corresponds to, for example, a parameter function whose parameters are encoded in the haptic effect file corresponding to the point of the keyframe.

[0170] According to the exemplary embodiment of FIG. 4 , four points are used to encode the parameters of the modulation function. Naturally, the number of points can be different from four, taking any value between 1 and 20, 50, or 100 points, depending on, for example, the length of the basic signal. The number of points can even be equal to 0, which would be interpreted as no modulation function.

[0171] Signal 47 represents a sequence of basic signals 41 to 43 that have been phase-adjusted and encoded into the first layer of the haptic effect file due to the phase information associated with each basic signal 41 to 43 .

[0172] Finally, signal 48 represents a sequence obtained from the sequentially arranged elementary signals 41 to 43 which have been amplitude modulated according to parametric modulation functions 44, 45, 46 represented by their corresponding points or keyframes.

[0173] The number of basic signals is not limited to 3, but is extended to any number, such as 1, 2, 5, 10 or more basic signals.

[0174] The types of basic signals are not limited to sinusoidal functions, but extend to any type of basic signal, such as constant signals (with fixed amplitude in time or space, depending on the representation space of the signal), triangular signals, and sequences of multiple basic signals can include different types of basic signals, such as a continuous signal followed by a sine signal followed by a triangle, optionally with gaps between consecutive basic signals.

[0175] FIG. 5 illustrates an exemplary sequence of modulated base signals encoded in accordance with the layered representation of the present disclosure, according to at least one exemplary embodiment.

[0176] FIG5 illustrates three different sequences of one or more modulated signals obtained by modulating one or more basic signals.

[0177] First signal 51 corresponds to a sequence obtained by applying amplitude modulation to a single continuous and constant signal (e.g., having a fixed amplitude over time). The amplitude modulation is encoded as a parametric modulation function obtained by interpolating multiple points or as keyframes used as parameters of the parametric function. The number of points or keyframes encoded as second data in the haptic effect file is, for example, 15.

[0178] The starting point of the first signal 51, shown as a black bold point, is advantageously coded with first data representing a continuous elementary signal.

[0179] The second signal 52 corresponds to the sequence obtained by applying magnitude modulation to each of the 4 basic signal sequences without phase adjustment between the different basic signals. According to that example, the first fundamental signal corresponds to a sinusoidal function with definite magnitude and frequency, the second fundamental signal corresponds to a rectangular signal with definite magnitude and frequency, the third fundamental signal corresponds to a triangular signal with definite magnitude and frequency, and the fourth fundamental signal corresponds to a sawtooth signal with definite magnitude and frequency. The starting point of each fundamental signal (and therefore the starting point of each modulated signal obtained from each fundamental signal) is indicated by a black bold dot and is advantageously encoded with the first data representing each fundamental signal.

[0180] Finally, the third signal 53 obtains a sequence of 8 basic signals from a sequence, each of which corresponds to a transient encoding as an API call associated with the basic signal encoded to the first layer. From Figure 5 , it can be seen that the sequence of eight signals is discontinuous, and a gap appears between two consecutive signals of sequence 53 . The gap is obtained directly from the starting point of each of the eight signals encoded in the first layer. Level representation. For example, API calls are represented and synthesized by sinusoidal functions with only one cycle in 22 milliseconds.

[0181] The hierarchical encoding disclosed in the present invention is capable of encoding any feedback effect in space, time, or any representation space as a sequence of one or more basic signals and associated modulation parameters.

[0182] 6 illustrates a schematic block diagram of the steps of a method of encoding a tactile signal according to a hierarchical representation of 100 pairs according to at least one exemplary embodiment.

[0183] In the first step 61 , the first data representing the modulation of at least one fundamental signal is encoded in a first layer in a plurality of layers, each with associated levels.

[0184] In the second step 62 , the second data representing the haptic perception modality associated with the haptic effect associated with the haptic signal is encoded in layers in multiple layers higher than the level of the first layer.

[0185] According to a particular embodiment, the at least one fundamental signal is included in the first set of elementary signals. In addition to the first data, the third data representing each fundamental signal of the first set is encoded in the first layer. Each fundamental signal is represented in a definite representation space that is a function of the modality of haptic perception.

[0186] FIG. 7 shows a schematic block diagram of steps of a method of decoding haptic data encoded according to the layered representation 100 , according to at least one example embodiment.

[0187] In a first step 71, first data representing a modulation of at least one elementary signal are decoded from a first layer of a plurality of layers, each layer having an associated level.

[0188] In a second step 72 , second data representing a haptic perception modality associated with a haptic effect associated with the haptic signal is decoded from a layer of the plurality of layers having a hierarchical level higher than the hierarchical level of the first layer.

[0189] According to certain embodiments, the at least one elementary signal is included in a first set of elementary signals. In addition to the first data, third data representing each elementary signal of the first set is decoded from the first layer. Each elementary signal is represented in a representation space determined as a function of a tactile perception modality.

[0190] FIG8 shows a schematic block diagram illustrating an example of a system in which various aspects and exemplary embodiments may be implemented.

[0191] System 8 can be embedded as one or more devices, including various components described below. In various embodiments, system 8 can be configured to implement one or more aspects described in this application.

[0192] Examples of devices that may form all or part of system 8 include personal computers, laptops, smartphones, tablets, digital multimedia converter boxes, digital television receivers, personal video recording systems, connected home appliances, connected vehicles and their associated processing systems, head-mounted displays (HMDs, see-through glasses), tactile sensors or actuators, "caves" (systems including multiple displays), servers, tactile encoders, tactile decoders, post-processors that process output from tactile decoders, pre-processors that provide input to tactile encoders, network servers, converter boxes, wireless (e.g., Bluetooth®) connected wearable tactile devices, and any other device or other communication device for processing tactile data or tactile signals. The elements of system 8 may be embodied, individually or in combination, in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 8 may be distributed across multiple ICs and / or discrete components. In various embodiments, system 8 may be communicatively coupled to other similar systems or other electronic devices via, for example, a communication bus or via dedicated input and / or output ports.

[0193] System 8 may include at least one processor 81 configured to execute instructions loaded therein to implement, for example, the various aspects described herein. Processor 81 may include embedded memory, input / output interfaces, and various other circuits known in the art. System 8 may include at least one memory 82 (e.g., volatile memory and / or non-volatile memory). System 8 may include a storage device 84, which may include non-volatile memory and / or volatile memory, including, but not limited to, electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, a magnetic disk drive, and / or an optical disk drive. As non-limiting examples, storage device 84 may include internal storage devices, attached storage devices, and / or network accessible storage devices.

[0194] System 8 may include an encoder / decoder module 83, which is configured to process data to provide encoded / decoded haptic signals or data, for example, and may include its own processor and memory. Encoder / decoder module 83 may represent a module that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include either or both encoding and decoding modules. Furthermore, encoder / decoder module 83 may be implemented as a separate component of system 83 or may be incorporated into processor 81 as a combination of hardware and software known to those skilled in the art.

[0195] Program code to be loaded onto processor 81 or encoder / decoder 83 to execute various aspects described herein may be stored in storage device 84 and subsequently loaded onto memory 82 for execution by processor 81. According to various embodiments, during execution of one or more of the processes described herein, processor 81, memory 82, storage device 84, and encoder / decoder module 83 may store one or more of various items. Such stored items may include, but are not limited to, point cloud frames, encoded / decoded geometry / attribute video / images or portions thereof, bitstreams, matrices, variables, and intermediate or final results of equations, formulas, operations, and arithmetic logic processing.

[0196] In some embodiments, memory within the processor 81 and / or encoder / decoder module 83 may be used to store instructions and provide working memory for processes that may be performed during encoding or decoding.

[0197] However, in other embodiments, memory external to the processing device (e.g., the processing device may be processor 81 or encoder / decoder module 83) may be used for one or more of these functions. The external memory may be memory 82 and / or storage device 84, such as dynamic volatile memory and / or non-volatile flash memory. In some embodiments, external non-volatile flash memory may be used to store the television's operating system. In at least one embodiment, fast external dynamic volatile memory, such as RAM, may be used as working memory for video encoding and decoding operations, such as for MPEG-V.

[0198] Input to the components of system 8 may be provided via various input devices, as indicated by block 85. Such input devices include, but are not limited to, (i) a RF portion of a broadcaster's broadcast that can receive, for example, an RF signal transmitted via a transmission, (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0199] In various embodiments, the input device of block 85 may have associated input processing elements, as known in the art. For example, the RF section may be associated with the following necessary elements: (i) selecting the desired frequency (also known as selecting a signal, or band-limiting the signal to a frequency band), (ii) downconverting the selected frequency, (iii) again band-limiting the frequency to a narrower frequency band to select a signal band (e.g., which may be referred to as a channel in some embodiments), (iv) demodulating the downconverted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired packet stream. The RF section of various embodiments may include one or more elements to perform these functions, such as a frequency selector, a signal selector, a frequency band limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner to perform various of these functions, including, for example, downconverting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or to baseband.

[0200] In one embodiment of a digital video converter box, the RF section and its associated input processing elements can receive RF signals transmitted via a wired medium (e.g., electrical wires). The RF section can then perform frequency selection by filtering, down-converting, and filtering again to the desired frequency band.

[0201] Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions.

[0202] Adding components can include inserting components between existing components, such as inserting an amplifier and an analog-to-digital converter. In various embodiments, the RF section can include an antenna.

[0203] Additionally, the USB and / or HDMI terminals may include corresponding interface processors for connecting system 8 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented, for example, within a separate input processing IC or, as needed, within processor 81. Similarly, various aspects of USB or HDMI interface processing may be implemented, as needed, within a separate interface IC or within processor 81. The demodulated, error-corrected, and demultiplexed streams may be provided to various processing elements, including, for example, processor 81 and encoder / decoder 83, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on a display or output device.

[0204] The various components of the system 8 may be disposed within an integrated housing. Within the integrated housing, the various components may be interconnected and data may be transferred between them using a suitable connection arrangement 85, such as an internal bus known in the art, including I2C busses, wiring, and printed circuit boards.

[0205] System 8 may include a communication interface 86 capable of communicating with other devices via a communication channel 860. Communication interface 86 may include, but is not limited to, a transceiver configured to send and receive data via communication channel 860. Communication interface 86 and communication channel 860 may include, but are not limited to, a modem or a network card, and communication channel 860 may be implemented, for example, within a wired and / or wireless medium.

[0206] In various embodiments, a Wi-Fi network such as IEEE 802.11 may be used to stream data to system 8. Wi-Fi signals for these embodiments may be received via a communication channel 860 and communication interface 86 suitable for Wi-Fi communication. Communication channel 860 for these embodiments may typically connect to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top content services to communicate.

[0207] Other embodiments may provide streaming data to system 8 using a digital video converter box that transmits data via the HDMI connection of input block 85.

[0208] Other embodiments may use the RF connection of input block 85 to provide streaming data to system 8 .

[0209] Streaming data may be used as a form of signaling messages used by system 8. The signaling messages may include data obtained from haptic effect files 2 or 3, for example, encoded in a binary stream.

[0210] It should be understood that signaling can be accomplished in a variety of ways. For example, in various embodiments, one or more syntax elements, flags, etc. may be used to signal a message to a corresponding decoder.

[0211] System 8 can provide output signals to various output devices, including display 870, speakers 880, and other peripherals 890, such as haptic devices / actuators.

[0212] In various embodiments, device-to-device control may be achieved with or without user intervention using communication protocols such as Audio / Video Link, Consumer Electronics Control, audio protocols, Universal Serial Bus, Haptic Industry Forum - Universal Haptic Protocol, or others.

[0213] Output devices may be communicatively coupled to system 8 via dedicated connections through respective interfaces 87 , 88 , and 89 .

[0214] Alternatively, the output device may be connected to the system 8 via the communication interface 86 using the communication channel 860. The display 870, the speaker 880, and / or the actuator 890 may be electronically integrated with the other components of the system 8 into a single unit, such as a television.

[0215] In various embodiments, the display interface 87 may include a display driver, such as a timing controller (T Con) chip.

[0216] Display 870, speaker 880, and / or actuator 890 may alternatively be separate from one or more other components, for example, if the RF portion of input 85 is part of a separate digital video converter box. In various embodiments where display 870, speaker 880, and / or actuator 890 are external components, the output signals may be provided via dedicated output connections, including, for example, an HDMI port, a USB port, or a COMP output.

[0217] FIG. 9 illustrates a schematic block diagram of an example of a browsing system and method for browsing one or more haptic effects, according to at least one example embodiment.

[0218] Haptic effects can be rendered as a function of space or time, as well as any independent system variable. A unified browsing solution should deliver optimal haptic experience regardless of the quality and value of the inputs and the communication capabilities of the output devices. This browsing process requires the unification of independent variable inputs and the ability to assign output values ​​to connected devices.

[0219] One of the existing approaches for representing tactile sensations in space is based on a map of expected tactile values. This approach compares these value maps with the input acquisition positions and renders the tactile sensation. The drawback of this approach is that an extremely precise and fast position acquisition system is required to match the human perceptual bandwidth. Here's an example:

[0220] V = 10 cm / s: velocity of the body part in contact with the tactile display;

[0221] F = 1000 Hz: maximum human perception bandwidth;

[0222] Minimum distance for sensing tactile elements: 0.1 ms-1 / 1000 s-1 = 0.0001 m = 100 micrometers.

[0223] The acquisition system must be able to capture finger position with an accuracy of less than 100 microns and a frequency of at least 1 kHz to enable spatial tactile texture scanning. Commercial position acquisition systems embedded in consumer devices capture body position between 30 and 120 Hz. Consequently, this scanning method is unsatisfactory for tactile signal scanning due to signal acquisition noise and low acquisition frequency.

[0224] Each block of Figure 9 will be interpreted as an element or unit of the system and as a corresponding operation implemented by the element or unit.The method 9 presenting the system and related implementations is intended to overcome at least one disadvantage of the prior art systems.

[0225] System 9 can be embodied as one or more devices, including the various components of system 8 described with reference to FIG8 . System 9 corresponds to, for example, a haptic renderer engine. System 9 is configured to render one or more haptic effects from encoded data representing the haptic effects to be rendered, the encoded data being received in the form of a data stream or bit stream. The received data stream or bit stream advantageously corresponds to the data stream obtained from the encoding process and method described above with reference to FIG1 through FIG5 .

[0226] In a first block 91, data representing the position of at least a portion of a user's body is received from a collection device or system 911. The data 911 is collected by the collection device at a determined frequency, for example at a frequency comprised between 30 and 120 Hz.

[0227] The first block 91 is configured to upsample or interpolate the position represented by the data 911. The first block 91 outputs data 912 representing the upsampled position of at least a part of the body.

[0228] The level of upsampling depends, for example, on the characteristics of one or more tactile presentation devices (actuators) controlled by the system 9. According to another example, the level of upsampling is fixed and stored as a parameter in the memory of the system 9.

[0229] Upsampling can be achieved by implementing any method known to those skilled in the art. For example, this method is described in patent document WO2018 / 202609 published on November 8, 2018.

[0230] FIG10 illustrates an exemplary embodiment of an upsampling algorithm.

[0231] Xn corresponds to the position acquisition vector obtained from the acquisition device in each application cycle.

[0232] The velocity vector Vn is calculated from the acquired position vectors Xn and Xn-1 acquired at times tn and tn-1, respectively.

[0233] Based on the position error ε calculated at each frame and fed forward to the compensation velocity integral term, the stable position vector X'n is calculated after calculating the compensation velocity integral at a series of times ti from i = 0 to k, where k is an integer. The equations used to calculate X'n, k and h correspond to the control parameters to minimize ε.

[0234] In a second block 92, the data 912 output by the first block 91 is received together with data 921 representing the processor clock and the browsing frequency.

[0235] Second block 92 also receives data 922 representing a haptic signal to be presented. Data 922 is received, for example, in the form of data encoded as one or more haptic effect files described with reference to the exemplary embodiments of FIG. 1 to FIG. 5 .

[0236] The second block 92 is configured to extract data from the received data stream or bit stream, process the extracted data according to the type of tactile effect to be presented (such information obtained from the received data stream or bit stream), and optionally according to the characteristics of the tactile presentation device (actuator) controlled by and connected to the system 9.

[0237] The layer with the highest level is decoded first.

[0238] For example, a tactile rendering model is selected from a plurality of tactile rendering models based on information indicating the type of tactile effect or feedback to be rendered (e.g., stiffness, texture, vibration, temperature), such information corresponding, for example, to data or metadata included in a data stream or bit stream, such as associated with the first combination encoded into the third layer.

[0239] The one or more characteristics of the haptic presentation device are received, for example, from a haptic presentation device communicatively coupled to the system 9. According to another example, the identifier of the haptic presentation device is received, for example, from a look-up table (LUT) stored in a memory of the system 9, wherein the LUT associates the characteristics of each haptic device with the identifier of each haptic device.

[0240] The haptic rendering model processes the bit stream of the data or only parts of the data comprised in the data stream, for example according to characteristics of a haptic rendering device intended to render the haptic effect.

[0241] For example, a broadband haptic rendering device is configured to render complex haptic feedback or effects. In this case, the entire haptic signal encoded in the bitstream data stream is rendered by the haptic rendering model.

[0242] According to another example, some haptic rendering devices are only capable of rendering simple haptic effects (e.g., vibrations that are only frequency-dependent). In this case, the haptic rendering model only scans the frequency portion of the haptic signal encoded in the bit stream or data stream.

[0243] The second data representing the haptic perception modality associated with the haptic signal / haptic effect encoded into the bitstream is parsed and decoded before the first and third data.

[0244] If an actuator coupled to the rendering engine is unable / not configured to view the haptic effect indicated by the haptic perception modality encoded with the second data, the rendering engine does not decode all of the bitstream because the actuator is not configured to view such haptic effect.

[0245] If an actuator coupled to the rendering engine is capable of / configured to browse a haptic effect indicated by the haptic perception modality encoded using the second data, the rendering engine decodes at least a portion of the first and third data encoded in the bitstream, the first and third data being decoded in part based on the browsing capability of the actuator.

[0246] Block 92 outputs data 923 representing the haptic signal to be presented to block 93 .

[0247] The block 93 is configured to process the data 923, for example according to characteristics of a haptic rendering device connected to the system 9, more specifically to the block 93. The processing may include at least one of downsampling, haptic mixing, ...

[0248] System 9 and associated processes are capable of rendering any kind of haptic effect in conjunction with a data stream or bit stream in a layered format according to the present disclosure, including use of data in the data stream or bit stream tailored to the characteristics and capabilities of the associated haptic rendering device.

[0249] FIG. 11 illustrates a schematic block diagram of steps of a method for presenting one or more haptic effects according to at least one example embodiment.

[0250] The method of FIG. 11 is implemented in the system 9 , for example.

[0251] In operation 1101 , a haptic data stream is received, for example, from a memory or from a remote device via a wired or wireless connection.

[0252] In operation 1102, it is determined whether the haptic data stream is in a hierarchical format according to the present disclosure by analyzing the received data and / or by checking the file extension and metadata. If yes, the method further proceeds to operation 1104. If not, the method further proceeds to operation 1103.

[0253] In operation 1103, when the format of the haptic data stream is different from the hierarchical structure of the present disclosure, the received data stream (eg, according to the AHAP or IVS format) is converted into the hierarchical format disclosed in the present disclosure.

[0254] At operation 1104, the haptic data is formatted according to the layered format of the present disclosure and the data is parsed or analyzed.

[0255] At operation 1105 , it is determined whether the haptic data is time-based, i.e., the representation space of the elementary signal is determined. This information is obtained, for example, from a second data layer included in a hierarchical structure that is higher in level than the first layer that transmits data representing the elementary signal and associated modulation.

[0256] If the tactile data is time-based, the method further processes operation 1117.

[0257] If the haptic data is not time-based, the method further processes operation 1106.

[0258] In operation 1106, a determination is made as to what type of haptic feedback is associated with the haptic data of the received data stream, for example, whether the type of haptic feedback corresponds to texture or stiffness. This information is obtained from the second data and / or from metadata associated with the third layer, i.e., from the first combination of one or more modulated signal sequences.

[0259] If the type of tactile feedback corresponds to texture, the method further processes operation 1108.

[0260] If the type of tactile feedback corresponds to stiffness, the method further processes operation 1107.

[0261] In operation 1107 , modulation points or keyframes (corresponding to modulation parameters) are extracted from the data stream. Such information is encoded as first data associated with each elementary signal capable of reconstructing the haptic effect.

[0262] In operation 1115, a modulation function is determined or calculated based on the points (keyframes) extracted in operation 1107. The interpolation function corresponds to, for example, a polynomial function, such as a third-order polynomial function. The interpolation function can be any other function, and a signal can be sent to the data stream to inform the rendering engine which function to use. The interpolation function is used to obtain points between two keyframes, and the keyframes are used as the starting and ending points of a segment of the function.

[0263] The function obtained by interpolating the points (key frames) encoded as the first data using the interpolation function corresponds to the modulation function used to modulate the relevant basic signal encoded with the third data in the first layer.

[0264] The point between the two keyframes is further determined based on data representing the location of a portion of the user's body experiencing the haptic effect, such data being obtained by upsampling the data representing the location of the portion in operation 1110. The body is received from the capture device in operation 1109. The portion of the body targeted by the haptic effect encoded into the data stream is obtained from the data stream, and this information is encoded as metadata in one or more layers of the data stream or bitstream.

[0265] In operation 1116, a stiffness haptic effect is calculated by modulating the base signal using the modulation function obtained in operation 1115. A sequence of modulated signals is calculated and combined according to the encoded data into the data stream and relative to the first and second combinations encoded into the third and fourth layers. The result of the calculation is a haptic signal representing a stiffness haptic effect, which will be rendered by one or more actuators associated with one or more body parts of the user for which the haptic effect is intended.

[0266] At operation 1124, a signal is transmitted to the actuator for scanning.

[0267] In operation 1108, first and third data representing the base signal and the associated modulation are extracted from the bitstream or data stream, and more specifically, from a first layer of a layered representation of data encoded into the bitstream or data stream.

[0268] At operation 1111, a determination is made as to whether the haptic presentation device (i.e., actuator) fed by the rendering engine is a broadband device based on data representing characteristics of each haptic presentation device obtained at operation 1118. This data is typically obtained from the haptic device or from a lookup table (LUT) stored in memory that associates the characteristics of each haptic device with an identifier for each haptic device.

[0269] In operation 1118, a determination is made as to whether the haptic device is wideband or amplitude-modulated only, the communication frequency, the range of signal communication, and / or the range of frequencies expressed (e.g., some actuators are configured to operate only from 65 to 300 Hz).

[0270] If the haptic device is a broadband device, the method further processes operation 1112.

[0271] If the haptic device is not a broadband device, the method further proceeds to operation 1113 .

[0272] In operation 1112, the haptic device can receive a broadband signal and process all modulated signals encoded with the first and third data into the first layer, for example in one or more of frequency bands 2 to 9, to obtain the following sequence: modulated signal.

[0273] In operation 1113, the haptic device cannot receive broadband signals, meaning it can only receive amplitude-modulated signals. In this case, a constant signal (e.g., hard-coded in the rendering engine) is modulated with specific first data representing the modulation encoded in the first layer, such as the first data encoded in the first frequency band described in the embodiments based on quantized haptic data.

[0274] Operations 1112 and 1113 can obtain the texture haptics of N application frames in operation 1114 .

[0275] In operation 1124, a signal is transmitted to the actuator for scanning.

[0276] In operation 1117, the received time-based haptic data or data stream representing the base signal and associated modulated first and third data is extracted from the bitstream or data stream, more specifically, from the first layer of the hierarchical representation of data encoded into the bitstream. This data more specifically relates to a sequence of modulated signals representing a vibration effect. In operation 1120, a determination is made as to whether the haptic presentation device (i.e., actuator) fed by the rendering engine is a wideband device based on the data representing the characteristics of each haptic presentation device obtained in operation 1118.

[0277] If the haptic device is a broadband device, the method further processes operation 1121.

[0278] If the haptic device is not a broadband device, the method further proceeds to operation 1122 .

[0279] In operation 1121, the haptic device can receive a broadband signal and process all modulated signals encoded with first and third data into a first layer, for example in one or more of frequency bands 2 to 9, to obtain the following sequence: modulated signal.

[0280] In operation 1122, the haptic device is incapable of receiving broadband signals, meaning it can only receive amplitude-modulated signals. In this case, a constant signal (e.g., hard-coded in the rendering engine) is modulated with specific first data representing the modulation encoded in the first layer, such as the first data encoded in the first frequency band described in embodiments based on quantized haptic data. According to a variant, if the haptic device is configured to implement such an API call, a transient signal encoded as an API call is presented.

[0281] Operations 1121 and 1122 can obtain the vibration haptic of N application frames in operation 1123 .

[0282] In operation 1124, the signal is transmitted to the actuator for presentation.

[0283] Such a process can adapt or adjust the browsing of haptic signals encoded into a bitstream or data stream according to the characteristics of the haptic devices and their ability to browse haptic effects according to their tactile perception modality.

[0284] Depending on the haptic device, all data encoded in the bitstream or data stream is decoded for browsing purposes, only part of the data encoded in the bitstream or data stream is decoded for browsing purposes, or only higher-level data or information is decoded (e.g., second data representing a haptic perception modality).

[0285] In Figures 1 to 11, various methods are described herein, and each method includes one or more steps or actions for implementing the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.

[0286] Some examples are described with respect to block diagrams and / or operational flow charts. Each block represents a circuit element, module, or portion of code that includes one or more executable instructions for implementing the specified logical function. It should also be noted that in other embodiments, the functions noted in the blocks may not occur in the order indicated. For example, depending on the functions involved, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order.

[0287] The embodiments and aspects described herein may be implemented in, for example, a method or process, an apparatus, a computer program, a data stream, a bit stream, or a signal. Even if discussed only in the context of a single embodiment (for example, discussed only as a method), the implementation of the features discussed may also be implemented in other forms (for example, an apparatus or a computer program).

[0288] These methods can be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. A processor also includes a communication device.

[0289] Furthermore, these methods can be implemented by instructions executed by a processor, and these instructions (and / or data values ​​resulting from the implementation) can be stored on a computer-readable storage medium. A computer-readable storage medium can take the form of a computer-readable program product embodied in one or more computer-readable media and having computer-readable program code embodied thereon that is executable by a computer. As used herein, computer-readable storage media can be considered non-transitory storage media, given their inherent ability to store information therein and provide for the inherent ability to retrieve information therefrom. A computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. It should be understood that the following, while providing more specific examples of computer-readable storage media to which the present embodiments may be applied, is illustrative only and not exhaustive, as would be readily understood by one of ordinary skill in the art: portable computer disk; hard drive; read-only memory (ROM); erasable programmable read-only memory (EPROM or flash memory); portable compact disc read-only memory (CD-ROM); optical storage device; magnetic storage device; or any suitable combination of the foregoing.

[0290] The instructions may form an application tangibly embodied on a processor-readable medium.

[0291] For example, instructions may be in hardware, firmware, software, or a combination thereof. For example, instructions may be found in an operating system, a separate application, or a combination of both. Thus, a processor may be characterized as, for example, a device configured to perform a process and a device that includes a processor-readable medium (e.g., a storage device) having instructions for performing the process. Furthermore, in addition to or in lieu of instructions, the processor-readable medium may store data values ​​generated by the implementation.

[0292] The device can be implemented, for example, with appropriate hardware, software, and firmware. Examples of such devices include personal computers, laptops, smartphones, tablets, digital multimedia converter boxes, digital television receivers, personal video recording systems, connected home appliances, head-mounted displays (HMDs, see-through glasses), projectors (projectors), "caves" (systems including multiple displays), servers, video and / or haptic encoders, video and / or haptic decoders, post-processors for processing video decoder output, pre-processors for providing input to video encoders, network servers, converter boxes, wirelessly connected wearable haptic devices, such as Bluetooth®-connected wearable haptic devices, and any other device for processing haptic data or signals representing one or more haptic feedback or effects, or other communication devices. It should be understood that the device can be mobile.

[0293] Computer software may be implemented by processor 81 or hardware, or a combination of hardware and software. As a non-limiting example, embodiments may also be implemented by one or more integrated circuits. Memory 82 may be of any type suitable for the technical environment and may be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. As a non-limiting example, processor 81 may be of any type suitable for the technical environment and may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0294] As will be apparent to one of ordinary skill in the art, implementations may generate various signals formatted to carry information that can be stored or transmitted, for example. The information may include, for example, instructions for executing a method, or data generated by one of the described implementations. For example, a signal may be formatted to carry a bit stream of the described embodiments. For example, such a signal may be formatted as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or a baseband signal. The formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0295] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that when used in this specification, the terms "include / comprise" and / or "include / comprising" may specify stated features, integers, steps, operations, elements, and / or components, for example, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Furthermore, when an element is referred to as being "responsive" or "connected" to another element, it can be directly responsive or connected to the other element, or intervening elements may be present. In contrast, when an element is referred to as being "directly responsive" or "directly connected" to another element, there are no intervening elements.

[0296] It should be understood that the use of any symbol / term " / ," "and / or," and "at least one of" in the context of, for example, "A / B," "A and / or B," and "at least one of A and B" may be intended to encompass selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the context of "A, B, and / or C" and "at least one of A, B, and C," such wording is intended to encompass selecting only the first listed option (A), or only the second option (B), or only the third option (C), or only the first and second options (A and B), or only the first and third options (A and C), or only the second and third options (B and C), or all three options (A, B, and C). As will be apparent to those of ordinary skill in this and related arts, this can be extended to as many items as listed.

[0297] Various numerical values ​​can be used in the present invention. Specific values ​​may be for illustrative purposes, and the described aspects are not limited to these specific values.

[0298] It should be understood that although the terms first, second, etc. may be used herein to describe various elements, these elements are not limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be referred to as a second element, and similarly, a second element could be referred to as a first element without departing from the teachings of the present invention. No ordering is implied between the first element and the second element.

[0299] References to "an exemplary embodiment" or "an exemplary embodiment" or "an implementation" or "an implementation" and other variations thereof are often used to convey that a particular feature, structure, characteristic, etc. (described in terms related to an embodiment / implementation) is included in at least one embodiment / implementation. Thus, the appearance of the phrase "in an exemplary embodiment" or "in an exemplary embodiment" or "in one embodiment" or "in an embodiment," and any other variations thereof, in various places throughout this disclosure are not necessarily all referring to the same embodiment.

[0300] Similarly, references herein to "according to an exemplary embodiment / example / implementation" or "in an exemplary embodiment / example / implementation" and other variations thereof are often used to convey that a particular feature, structure, or characteristic (described in connection with an exemplary embodiment / example / implementation) can be included in at least one exemplary embodiment / example / implementation. Thus, the appearances of the phrases "according to an exemplary embodiment / example / implementation" or "in an exemplary embodiment / example / implementation" in various places in the specification are not necessarily all referring to the same exemplary embodiment / example / implementation, nor are separate or alternative exemplary embodiments / examples / implementations necessarily mutually exclusive of other exemplary embodiments / examples / implementations.

[0301] The reference numbers appearing in the claims are for illustration only and have no limiting effect on the scope of the claims. Although not explicitly described, the present embodiments / examples and variations may be employed in any combination or subcombination.

[0302] When the drawings are presented in the form of flow charts, it should be understood that they also provide block diagrams of the corresponding apparatus. Similarly, when the drawings are presented in the form of block diagrams, it should be understood that they also provide flow charts of the corresponding methods / processes.

[0303] Although some figures include arrows on communication paths to show a primary direction of communication, it should be understood that communication can occur in the opposite direction of the depicted arrows.

[0304] Various implementations involve decoding. As used in this disclosure, "decoding" may include, for example, all or part of the processing performed on a received haptic signal (which may include a received bit stream encoding one or more haptic signals) to produce a suitable final output for browsing haptic effects or for further processing in reconstructed haptic feedback or effects. In various embodiments, such processes include one or more processes typically performed by a decoder. In various embodiments, such processes also or alternatively include processes performed by a decoder of the various implementations described in this disclosure, for example,

[0305] Various implementations involve encoding. Similar to the discussion above regarding "decoding," "encoding," as used in this disclosure, can encompass, for example, all or part of the processing performed on the input haptic signal to produce an encoded bitstream. In various embodiments, such processing includes one or more processes typically performed by an encoder. In various embodiments, such processing also or alternatively includes processes performed by the encoder of the various implementations described herein.

[0306] Additionally, the application may refer to "obtaining" various types of information. Obtaining information may include, for example, one or more of estimating information, calculating information, or retrieving information from memory.

[0307] Additionally, the application may refer to "accessing" various types of information. Accessing information may include one or more of receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, or estimating information.

[0308] Furthermore, the application may refer to "receiving" various types of information. Like "accessing," receiving is a broad term. Receiving information can include, for example, one or more of accessing information or retrieving information (e.g., from memory). Furthermore, operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, and calculating information often involve "receiving," determining information, or evaluating information in one way or another.

[0309] Furthermore, as used herein, the term "signal" specifically refers to indicating something to a corresponding decoder. Thus, in one embodiment, the same parameters may be used on both the encoder and decoder sides. For example, the encoder may transmit specific parameters to the decoder (explicit signaling) so that the decoder can use the same specific parameters. Conversely, if the decoder already knows the specific parameters along with other parameters, signaling may be used instead (implicit signaling) to simply allow the decoder to know and select the specific parameters. By avoiding transmitting any actual functionality, various embodiments achieve bit savings. It should be understood that signaling can be accomplished in a variety of ways. For example, in various embodiments, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder. While the verb form of the word "signal" has been used above, the term "signal" may also be used herein as a noun.

[0310] A number of implementations have been described. However, it should be understood that various modifications are possible. For example, elements of different implementations may be combined, supplemented, modified, or removed to produce other implementations. Furthermore, one of ordinary skill will understand that other structures and processes may be substituted for those disclosed, and that the resulting implementations will perform at least substantially the same functions in at least substantially the same manner to achieve at least substantially the same results as the disclosed implementations. Therefore, the present invention contemplates these and other implementations.

[0311] 2: Codec 3: Haptic Effects File 8, 9: system 11, 12, 13, 14: layer 21, 22, 23, 24: Blocks 24, 25: Modulation signal 31, 32: string 41, 42, 43: Basic signals 44, 45, 46: Modulation function 47, 48: Signal 51: First Signal 52: Second Signal 53: The Third Signal 61: First Step 62: Second Step 71: First Step 72: Second Step 81: Processor 82: Memory 83: Encoder / Decoder Module 84: Storage device 85: Block 86: Communication Interface 87, 88, 89: Interface 91: The first block 92: The second square 93: Block 100: Hierarchical representation 111, 112: Basic signal 121, 122: Sequence 131, 132: First combination 141: Second combination 241: Third Data 242: First Data 251: Third Data 252: First Data 310, 320: header 311, 321: load 860: Communication Channel 870: Display 880: Speaker 890:Actuator 911, 912, 921, 922, 923: Data 1101, 1102, 1103, 1104, 1105, 1106, 1107, 1108, 1109, 1110, 1111, 1112, 1113, 1114, 1115, 1116, 1117, 1118, 1120, 1121, 1122, 1123, 1124: Operation

Claims

1. A method for encoding a tactile signal representing a tactile effect in a bitstream of encoded tactile signal data, wherein the method includes encoding a hierarchical representation (100) of the tactile signal, the hierarchical representation comprising a plurality of layers (11, 12, 13, 14), and a hierarchy associated with each of the plurality of layers (11, 12, 13, 14), the encoding comprising: In the first layer (11) of the plurality of layers (11, 12, 13, 14), a first data encoding (61) is performed on the modulation (44, 45, 46) representing at least one basic signal (41, 42, 43), wherein the at least one basic signal (41, 42, 43) includes at least a sine signal and a triangular signal; and in one layer of the plurality of layers having a higher level than the first layer, a second data encoding (62) is performed on the tactile perception modality associated with the tactile effect.

2. The method as described in claim 1, further comprising: In the first layer (11), the first set of basic signals (41, 42, 43) containing at least one of the basic signals are encoded by third data encoding of each basic signal representing the first set of basic signals (41, 42, 43), each basic signal being represented in a defined space based on the tactile perception modality.

3. The method as described in claim 2, wherein, The third data includes: data indicating the type of the at least one basic signal (41, 42, 43); and / or data indicating the phase of the at least one basic signal (41, 42, 43), and wherein the first data includes data indicating the coordinates of a plurality of points in the defined space.

4. The method as described in claim 2 or 3, wherein, The plurality of layers (11, 12, 13, 14) further includes a second layer (12) having a level higher than the first layer, a third layer (13) having a level higher than the second layer, and a fourth layer (14) having a level higher than the third layer. The method further includes: in the second layer (12), encoding the second group of sequences including at least one or more modulated signals by encoding a first message of a sequence representing one or more modulated signals in each sequence of the second group, each modulated signal being obtained from the basic signal (41, 42, 43) of the first group that is modulated by a second data associated with the basic signal (41, 42, 43). In the third layer (13), the third group of the first group containing at least one first group of the second group is encoded by encoding a second message indicating the number of sequences contained in each first group; and in the fourth layer (14), the second group of the third group of the first group is encoded by encoding a third message indicating the number of first groups contained in the second group.

5. The method as described in request item 4, wherein, At least one sequence in the second group includes a plurality of modulation signals obtained from the basic signal (41, 42, 43) of different types.

6. The method as described in claim 5, wherein, The at least one basic signal corresponds to a constant signal.

7. The method of claim 6 further includes encoding metadata in at least one of the plurality of layers (11, 12, 13, 14), at least a portion of which includes data representing an application interface call associated with the at least one basic signal.

8. A method for decoding a tactile signal from encoded tactile signal data, wherein the method includes decoding a hierarchical representation of the tactile signal, the hierarchical representation comprising a plurality of layers, and a hierarchy associated with each of the plurality of layers, the decoding comprising: From the first layer (11) of the plurality of layers, a first data decoding (71) representing a modulation of at least one basic signal, wherein the at least one basic signal (41, 42, 43) includes at least a sine signal and a triangular signal; and from one layer of the plurality of layers having a level higher than the first layer, a second data decoding (72) representing a tactile perception modality.

9. The method as described in claim 8, further comprising: From the first layer, the first set of basic signals (41, 42, 43) containing at least one of the basic signals is decoded by decoding the third data of each basic signal representing the first set of basic signals (41, 42, 43), each of the basic signals being represented in a defined space based on the tactile perception modality.

10. The method as described in claim 9, wherein, The third data includes: data indicating the type of the at least one basic signal; and / or data indicating the phase of the at least one basic signal, and wherein the first data includes data indicating the coordinates of a plurality of points in the defined space.

11. The method as described in claim 9 or 10, wherein, The plurality of layers (11, 12, 13, 14) further includes a second layer (12) having a level higher than the first layer, a third layer (13) having a level higher than the second layer, and a fourth layer (14) having a level higher than the third layer. The method further includes: decoding the second group of sequences comprising at least one or more modulation signals from the second layer (12) by decoding a first message representing a sequence of one or more modulation signals in each sequence of the second group, each modulation signal being obtained from the basic signal of the first group modulated by a second data associated with the basic signal; decoding the third group of a first group comprising at least one first group of sequences of the second group from the third layer (13) by decoding a second message representing the number of sequences contained in each first group; and decoding the second group of one or more first groups of the third group from the fourth layer (14) by decoding a third message representing the number of first groups contained in the second group.

12. The method as described in claim 11, wherein, At least one sequence in the second group includes a plurality of modulation signals obtained from the basic signal of different types.

13. The method as described in claim 12, wherein, The at least one basic signal corresponds to a constant signal.

14. The method of claim 13, further comprising decoding metadata from at least one of the plurality of layers, at least a portion of the metadata including data representing application interface calls associated with the at least one basic signal.

15. An apparatus (8) for encoding tactile signals representing tactile effects in a bit stream of encoded tactile signal data, wherein the apparatus includes memory (82, 84) associated with at least one processor (81) configured to implement the method according to any one of claims 1 to 7.

16. An apparatus (8) for decoding tactile signals representing tactile effects from a bit stream of encoded tactile signal data, wherein the apparatus includes memory (82, 84) associated with at least one processor (81) configured to implement the method according to any one of claims 8 to 14.

Images