Representation format for associating a haptic effect to an emoji

Abstract

A haptic emoji is based on an association between an emoji represented by a graphic element and a corresponding haptic effect. This association may be stored in a haptic emoji library that may take the form of a standardized correspondence table indexed by a unique identifier, such as a Unicode, that references one haptic emoji in a set of haptic emojis of the library. The haptic effect is associated with a haptic signal that is either comprised in the data representing the haptic effect or can be generated from this data. A device displaying a text message comprising a haptic emoji renders the graphical element and the haptic effect, the haptic effect being rendered by providing the haptic signal corresponding to the haptic emoji to a haptic actuator.

Description

[0001] REPRESENTATION FORMAT FOR ASSOCIATING A HAPTIC EFFECT TO AN EMOJI

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the priority to European Application N° 23306986.3 filed 16 November 2023 which is incorporated herein by reference in its entirety.

[0004] TECHNICAL FIELD

[0005] At least one of the present embodiments generally relates to content representation and more particularly to a representation format allowing to associate a haptic effect to an emoji to form a haptic emoji.

[0006] BACKGROUND

[0007] Haptics refers to sense of touch and includes two dimensions, tactile and kinesthetic. The first relates to tactile sensations such as friction, roughness, hardness, temperature and is felt through the mechanoreceptors of the skin (Merkel cell, Ruffini ending, Meissner corpuscle, Pacinian corpuscle). The second is linked to the sensation of force / torque, position, motion / velocity provided by the muscles, tendons and the mechanoreceptors in the joints. Haptics is also involved in the perception of self-motion since it contributes to the proprioceptive system (i.e., perception of one’s own body). Thus, the perception of acceleration, speed or any body model could be assimilated as a haptic effect. The frequency range is about 0-1 kHz depending on the type of modality. Most existing devices able to render haptic signals generate vibrations. Examples of such haptic actuators are linear resonant actuator (LRA), eccentric rotating mass (ERM), and voice-coil linear motor. These actuators may be integrated into haptic rendering devices such as haptic suits but also smartphones or game controllers.

[0008] To encode haptic signals, several formats have been defined related to either a high- level description using XML-like formats (for example MPEG-V), parametric representation using j son-like formats such as Apple Haptic Audio Pattern (AHAP) or Immersion Corporation’s HAPT format, or waveform encoding (IEEE 1918.1.1 ongoing standardization for tactile and kinesthetic signals). The HAPT format has been recently included into the MPEG ISOBMFF file format specification (ISO / IEC 14496 part 12). Moreover, GL Transmission Format (glTF™) is a royalty-free specification for the efficient transmission and loading of 3D scenes and models by applications. This format defines an extensible, common publishing format for 3D content tools and services that streamlines authoring workflows and enables interoperable use of content across the industry.

[0009] Moreover, a new haptic file format is being defined within the MPEG standardization group and relates to a coded representation for haptics. Two encoding methods have been defined. The first one encodes PCM signals (regular temporal samples) using a wavelet decomposition, quantization and entropy coding similar to traditional audio and image codecs but using a dedicated perception model for the quantization. The second one using a descriptive (or parametric) representation of the input signal, either by converting the signal to a parametric representation or from a manual creation tool. Those formats are then either stored or transmitted using a human readable JSON interchange format (HJIF) or its packetized binary version ready for packet networks (MH4S for MPEG-I Haptic Stream format). MIHS is the name of the format and HMPG is the file extension. A MIHS file may be streamed and thus is a concatenation of MIHS units. The Reference Model of these formats is not yet publicly released (current version is 3.1) but is referenced herein as MPEG Reference Model. With this reference model, the encoded haptic description file can be exported either as a JSON interchange format (for example a .hj if file) that is human readable or as a compressed binary distribution format (for example a .hmpg) that is particularly adapted for transmission towards haptic rendering devices. Syntax and formats described herein are using the MPEG-I Haptic Stream format (MIHS).

[0010] An emoji is a graphical element (i.e., a pictogram, logogram, ideogram, or smiley) comprised in a textual element or container used in electronic messages and web pages. The primary function of emoji is to transport emotional cues otherwise missing from textual messages such as typed conversations or social media feeds. Emoji exist in various genres, such as facial expressions, common objects, places, types of weather, animals. They are the successors of emoticons that have the same goal but are represented using typographic approximations, generally based on a combination of characters comprising punctuation and letters. SUMMARY

[0011] Embodiments are related to haptic emojis. According to embodiments, a haptic emoji is based on an association between an emoji represented by a graphic element and a corresponding haptic effect. This association may be stored in a haptic emoji library that may take the form of a standardized correspondence table indexed by a unique identifier, such as a Unicode, that references one haptic emoji in a set of haptic emojis of the library. The haptic effect is associated with a haptic signal that is either comprised in the data representing the haptic effect or can be generated from this data. A device displaying a text message comprising a haptic emoji renders the graphical element and the haptic effect, the haptic effect being rendered by providing the haptic signal corresponding to the haptic emoji to a haptic actuator.

[0012] A first aspect of at least one embodiment is directed to a method comprising obtaining an identifier associated with a haptic emoji in a haptic emoji library comprising a plurality of haptic emojis, wherein a haptic emoji comprises at least an identifier, data representative of a graphical element representing the haptic emoji, and data representative of a haptic effect for the haptic emoji, obtaining from the haptic emoji library, based on the identifier, a graphical element and a haptic effect for the haptic emoji, rendering the graphical element, and rendering the haptic effect.

[0013] A second aspect of at least one embodiment is directed to an apparatus comprising a processor configured to obtain an identifier associated with a haptic emoji in a haptic emoji library comprising a plurality of haptic emojis, wherein a haptic emoji comprises at least an identifier, data representative of a graphical element representing the haptic emoji, and data representative of a haptic effect for the haptic emoji, obtain from the haptic emoji library, based on the identifier, a graphical element and a haptic effect for the haptic emoji, render the graphical element and render the haptic effect.

[0014] A third aspect of at least one embodiment is directed to a non-transitory computer readable medium storing a haptic emoji library comprising a plurality of haptic emojis, wherein a haptic emoji comprises at least a unique reference identifying the haptic emoji from other haptic emojis, data comprising a graphical element for the haptic emoji, and data comprising a haptic signal associated with a haptic effect for the haptic emoji.

[0015] A fourth aspect of at least one embodiment is directed to a computer program comprising program code instructions executable by a processor, the computer program implementing at least the steps of a method according to the first aspect. A fifth aspect of at least one embodiment is directed to a computer program product stored on a non-transitory computer readable medium and comprising program code instructions executable by a processor, the computer program product implementing at least the steps of a method according to the first aspect.

[0016] BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The present disclosure may be better understood by consideration of the detailed description below in conjunction with the accompanying figures in which:

[0018] Figure 1 illustrates a block diagram of an example of system in which various aspects and embodiments are implemented.

[0019] Figure 2 illustrates an example of haptic emoji library according to embodiments.

[0020] Figure 3 illustrates an example of process for rendering a haptic emoji according to embodiments.

[0021] Figures 4 A and 4B illustrate examples of usage of haptic emoji according to embodiments.

[0022] Figure 5 illustrates examples of processes for encoding and rendering a haptic effect associated with a haptic emoji.

[0023] Figure 6 illustrates an example of data structure for the interchange file format describing haptic effect associated with a haptic emoji.

[0024] Figure 7 illustrates an example of haptic signal coded using two haptic bands.

[0025] Figure 8A illustrates an example of method for encoding, in two bands of frequencies, a haptic signal for a haptic effect associated with a haptic emoji according to embodiments.

[0026] Figure 8B illustrates an example of method for decoding a haptic signal comprising two bands of frequencies for a haptic effect associated with a haptic emoji according to embodiments.

[0027] It should be understood that the drawings are for purposes of illustrating examples of various aspects, features and embodiments in accordance with the present disclosure and are not necessarily the only possible configurations. DETAILED DESCRIPTION

[0028] Figure 1 illustrates a block diagram of an example of system in which various aspects and embodiments are implemented. In the depicted system, the user Alice uses the rendering device 100 to interact with a server 180 providing a content 190 through a communication network 170. This content 190 may take many different forms and may comprise various data and / or files such as audio data, video data, text, graphics required for its rendering. It may comprise emojis and haptic effects, and more particularly haptic emojis, in other words emojis associated with haptic effects such as the ones available from the haptic emoji library 150. The content 190 may be generated under control of a content generator 110 that may take different forms according to different context of usage. For example, when the content is a written document, the content generator may be implemented for example as an edition software running on a computer or server. When the content is a flow of interpersonal interactions (E.g., SMS) or social media interactions (E.g., WhatsApp, Twitter), the content generator may be implemented for example as an interaction manager software running on a computer.

[0029] The rendering device 100 is an apparatus that comprises a processor 101. The processor 101 may be a general-purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor may perform data processing such as haptic signal decoding, input / output processing, and / or any other functionality that enables the device to operate in an immersive system.

[0030] The processor 101 may be coupled to an input unit 102 configured to convey user interactions. Multiple types of inputs and modalities can be used for that purpose. A physical keypad and a touch sensitive surface are typical examples of input units adapted to this usage although voice control could also be used. In addition, the input unit may also comprise a digital camera able to capture still pictures or video in two dimensions or a more complex sensor able to determine the depth information in addition to the picture or video and thus able to capture a complete 3D representation. The processor 101 may be coupled to a display unit 103 configured to output visual data to be displayed on a screen. Multiple types of displays can be used for that purpose such as a liquid crystal display (LCD) or organic light-emitting diode (OLED) display unit. The processor 101 may also be coupled to an audio unit 104 configured to render sound data to be converted into audio waves through an adapted transducer such as a loudspeaker for example. The processor 101 may be coupled to a communication interface 105 configured to exchange data with external devices. The communication preferably uses a wireless communication standard to provide mobility of the rendering device, such as cellular (e.g., LTE) communications, Wi-Fi communications, and the like. The processor 101 may access information from, and store data in, the memory 106, that may comprise multiple types of memory including random access memory (RAM), read-only memory (ROM), a hard disk, a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, any other type of memory storage device. In embodiments, the processor 101 may access information from, and store data in, memory that is not physically located on the device, such as on a server, a home computer, or another device.

[0031] The processor 101 is coupled to a haptic unit 107 configured to provide haptic feedback to the user (e.g.: vibrations), defined by the haptic effect associated with an emoji described in the content 190. The haptic unit 107 may comprise a single haptic actuator or a plurality of haptic actuators located at a plurality of positions on the rendering device. Different haptic units may have a different number of actuators and / or the actuators may be positioned differently on the rendering device.

[0032] In at least one embodiment, the processor 101 is configured to render a haptic signal according to embodiments described further below, in other words to provide (e.g. apply) a low-level signal to a haptic actuator to render the haptic effect associated with an emoji. Such low-level signal may be represented using different forms, for example by metadata or parameters in the description file or by using a digital encoding of a sampled analog signal (e g., PCM or LPCM).

[0033] The processor 101 may receive power from the power source 108 and may be configured to distribute and / or control the power to the other components in the device 100. The power source 108 may be any suitable device for powering the device. As examples, the power source may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickelzinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), and the like), solar cells, fuel cells, and the like.

[0034] While the figure depicts the processor 101 and the other elements 102 to 108 as separate components, it will be appreciated that these elements may be integrated in an electronic package or chip. It will be appreciated that the rendering device 100 may include any subcombination of the elements described herein while remaining consistent with an embodiment. The processor 101 may further be coupled to other peripherals or units not depicted in figure 1 which may include one or more software and / or hardware modules that provide additional features, functionality and / or wired or wireless connectivity. For example, the peripherals may include sensors such as a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, and the like.

[0035] Typical examples of a rendering device 100 are smartphones, tablets, computers, game / AR / VR controllers, headphones, or any other object with integrated haptic capabilities able to render a haptic effect associated with an emoji.

[0036] Other examples of rendering devices 100 do not comprise any haptic capabilities but are in relation with one or more device able to render a haptic effect associated with an emoji such as haptic gloves, haptic chairs, haptic props, motion platforms. In this case, the rendering device 100 may prepare data for rendering the haptic effect so that another device, such as a haptic prop, can perform the haptic rendering. Examples of such devices are television sets, head-mounted displays, gamepads, or laptops.

[0037] In at least one embodiment, the device does not include a display unit, nor does it include a haptic unit. In such embodiment, the device does not visually render the content 190 and does not render the associated haptic effects. However, the device may prepare data for display so that another device, such as a screen, can perform the display and may prepare data for rendering the haptic effect so that another device, such as a haptic prop, can perform the haptic rendering. Examples of such devices are computers, game consoles, optical media players, or set-top boxes.

[0038] In at least one embodiment, the content 190 and associated elements are directly hosted in memory 106 of the rendering device 100 allowing local rendering and interactions. In a variant of this embodiment, the device 100 also comprises the content generator 110 allowing a fully standalone operation, for example without needing any communication network 170 and server 180.

[0039] The haptic emoji library 150 may be common to a plurality of devices, for example defined in a standard specification. It comprises a set of haptic emojis and for each of them at least the following elements: a unique reference value (e.g., Unicode) allow to reference the haptic emoji, an image for the graphical representation and haptic data comprising a haptic effect for the sensorial perception. The haptic modality associated with an emoji may be for example vibrotactile, temperature, electrotactile, force or any other haptic modality. The set of haptic emojis is for example a list indexed by the Unicode value, as illustrated below in figure 2. In at least one embodiment, haptic effects of the library are encoded using the MPEG HJIF format or its binary-equivalent HMPG, as defined in the MPEG-I Haptics standard ISO / IEC 23090-31 : Haptics Coding.

[0040] In at least one embodiment, the haptic emoji library is based on the Unicode standard. In at least one embodiment, the principle is extended to alarm / alert unicodes or other haptic only emojis, which would extend the current list of unicodes. Some emojis might not have an associated haptic file. In at least one embodiment, some haptic emoji might not have an associated image and thus only the haptic effect is associated to a unicode.

[0041] In at least one embodiment, the haptic emoji library 150 is stored in the memory 106 of the rendering device 100. In at least one embodiment, the haptic emoji library 150 is obtained from a remote storage through the communication interface 105 of the rendering device 100. In at least one embodiment, the haptic emoji library 150 is pre-determined and stored in non- transitory computer readable medium of the rendering device 100.

[0042] Haptic effect may be short in time to be coherent with an emoji image goal and significance. Thus, the simplest and direct implementation is to use MPEG transient effect type, as described further below and as shown in the fourth column of the figure. However, vectorial effects may also be used, offering more flexibility in the type of waveform designed.

[0043] Figure 2 illustrates an example of haptic emoji library according to embodiments. The figure only shows an extract (200) from the haptic emoji library, illustrating a subset of four haptic emojis. A complete library may contain much more haptic emojis (i.e., dozens or hundreds). The first column (210) represents the Unicode which is a unique identifier allowing to reference one specific haptic emoji from the list. In the illustration, the Unicode uses an UTF-8 character encoding but other types of character encodings may be used alternatively, such as UTF-16, GB18030, or others. The second column (220) represents an optional textual description of an intended emotion to be represented. The third column (230) represents a graphical element to be used for visually rendering the haptic emoji and conveying the intended emotion. The fourth column (240) represents haptic data comprising a haptic effect to be used for rendering a tactile or kinesthetic sensation to be perceived by the user. The haptic data is illustrated in the figure as a temporal signal however haptic data may comprise a plurality of parameters related to the haptic effect, as illustrated in the syntax examples of table 1 to table 4. Indeed, in this document, the haptic emoji is described in a simplified manner as being an association between an emoji and a haptic effect. However, in practice, the association may not be direct since some additional parameters related to the haptic effect may be provided so that the rendering device knows how to render the haptic effect. In this context, the notion of haptic effect should be understood as a set of haptic data representing the haptic effect and other data related to the haptic effect. The fifth column (250) represents an optional textual description of haptic effect to be represented.

[0044] Independently from its representation, the haptic effect (241, 242, 243, 244) is associated with a haptic signal that is either comprised in the data representing the haptic effect or can be generated from this data. In the first case, a haptic signal associated with the haptic effect may be generated by recording an audio file (such as someone crying for the haptic effect 241, someone laughing for the haptic effect 242, someone clearing their throat for the haptic effect 243 or recording the heart beats for the haptic effect 244) and converting the recorded audio file to a haptic PCM file or vectorial MPEG file. In the second case, a haptic signal associated with the haptic effect may be authored directly using a parametric representation (either manually or through authoring tools) as illustrated in figure 7. Such effect may then be coded using a descriptive format, for example using the HJIF file format or generated as a PCM signal and encoded. In this case, a haptic signal corresponding to the parametric representation needs to be generated to be able to render the haptic effect. At the rendering stage, the haptic signal associated with the haptic effect is provided to a haptic actuator in order to render the haptic effect. Therefore, in this document, the terms haptic effect and haptic signal are used interchangeably.

[0045] Note that other formats could also be used, such as Apple AHAP or immersion corp. IVS haptic file formats.

[0046] Figure 3 illustrates an example of process for rendering a haptic emoji according to embodiments. This process 300 is for example implemented in a rendering device 100 of figure 1 and typically executed by the processor 101 of such device. Prior to the step 310, the processor obtains a content comprising a haptic emoji. This content is for example received from an electronic communication through a communication network. Examples of electronic communications comprise a web page, a message according to the Short Messaging System (SMS), a textual element in a social network interaction, or other types of electronic communications. In step 310, the processor obtains an identifier comprised in the electronic communication identifying the haptic emoji. In embodiments, the identifier is represented according to UTF-8, UTF-16 or other standard character formats. In step 320, the processor obtains the graphic element corresponding to the identifier from a haptic emoji library. In step 330, the processor obtains haptic data corresponding to the identifier from a haptic emoji library and obtains the haptic effect from this data, as well as other parameters for rendering the haptic effect. In step 340, the processor renders the visual element and the haptic effect. Rendering of the haptic effect may be done by obtaining the haptic signal associated with the haptic effect corresponding to the identifier from the haptic emoji library and providing this haptic signal to a haptic actuator. As mentioned with reference to figure 2, the haptic signal is either comprised in the data representing the haptic effect or can be generated from this data.

[0047] In some embodiments where the rendering device does not comprise display capabilities, instead of rendering the visual element, the processor provides the data needed for rendering the visual element to a device having display capabilities.

[0048] In some embodiments where the rendering device does not comprise haptic capabilities, instead of rendering the haptic effect, the processor provides the data needed for rendering the haptic effect to device having haptic capabilities.

[0049] This process 300 is based on using the haptic emoji library 200 of figure 2. Any other technique that provides an association between an identifier, a visual element and haptic data may be used.

[0050] Figures 4A and 4B illustrate examples of usage of haptic emoji according to embodiments. These figures show graphical elements corresponding to a content comprising at least one haptic emoji. A web page, a SMS message, or a social media application are examples of content that may comprise haptic emojis.

[0051] In the example of figure 4A, the content comprises a textual element (i.e., the terms “What a surprise”), and a haptic emoji of a face with stars in the eyes, identified by a specific Unicode (or reference). The graphical element corresponding to the haptic emoji is displayed through a visual representation of a corresponding image, for example as illustrated in the column 230 of figure 2. After decoding the Unicode (or reference) identifying the haptic emoji illustrated in the figure, the rendering device displays the textual element, the graphical element corresponding to the haptic emoji (i.e., a face with stars in the eyes) and the associated haptic effect (i.e., a continuous vibration at 440hz).

[0052] In at least one embodiment, the haptic effect is rendered at the same time the Unicode is decoded. In at least one embodiment, the haptic effect is rendered once at the same time the Unicode is decoded. In at least one embodiment, the haptic effect is rendered once when the graphical element corresponding to the haptic emoji is displayed. In at least one embodiment, the haptic effect is rendered as long as the graphical element corresponding to the haptic emoji is displayed. In at least one embodiment, the haptic effect is rendered several times (looping), the number of times could be specified by the application, the OS or a parameter.

[0053] In the example of figure 4B, the content comprises two textual elements and two haptic emojis. In this case, the rendering device displays the textual elements and the graphical elements corresponding to the two haptic emojis and renders the two haptic effects associated with the two haptic emojis in the same order as the graphical elements.

[0054] In at least one embodiment, the haptic effects are played one after the other a single time when displaying the content. In at least one embodiment, the effects are played one after the other continuously while the content is displayed. In at least one embodiment, the effects are played based on eye tracking information; the haptic signal associated with an emoji may be rendered when the user is looking at the visual representation of the emoji. In at least one embodiment, the effects are played based on user input; the haptic signal associated with an emoji may be rendered when the user selects the emoji, for example by clicking on its visual representation.

[0055] Figure 5 illustrates examples of processes for encoding and rendering a haptic effect associated with a haptic emoji. The encoding process 500 is for example implemented as a module of content generator 110 of figure 1 and typically performed on a computer generating the content and comprising a processor. It may also be implemented on a specific hardware platform dedicated to encoding content. The inputs are a metadata file 501 and at least one low- level haptic signal file 503. The metadata file 501 is for example based on the ‘OHM’ haptic object file format. The signal files represent analog signals to be applied to haptic actuators and are conventionally encoded using a pulse coded modulation (PCM) for example based on the WAV file format. The descriptive files 502 are for example based on the AHAP, IVS, MPEG HJIF or HAPT file formats.

[0056] In step 510, the processor extracts metadata from the metadata file 501, allowing to identify the descriptive files and / or signal files. In step 511, the processor analyzes and transcodes descriptive files 502. In step 512, signal files are processed. This process comprises decomposing the signal in frequency bands and keyframes or wavelets, as further described in figures 6 and 7.

[0057] In step 520, the processor generates an interchange file 504 in compliance with the data format as defined in the MPEG-I Haptics standard ISO / IEC 23090-31 : Haptics Coding. In step 530, the process compresses the interchange file 504 to be distributed in a transmission-friendly form such as the distribution file 505, more compact than the interchange file format.

[0058] The interchange file 504 is a human readable file for example based on glTF, XML or JSON formats. The distribution file 505 is a binary encoded file for example based on MPEG file formats adapted for streaming or broadcasting to a decoder device.

[0059] The decoding process 550 and the rendering process 580 are for example both implemented by a rendering device 100 of figure 1 and executed by the processor 101. The decoding process 550 may also be performed by a device separate from - but communicating with - the rendering device, for example by a computer, a set top box, a smartphone, a computing instance in the cloud.

[0060] In the case a distribution file 505 is used, this file needs first to be decoded. In step 550, the processor performs the binary decompression 555 and generates an interchange file 504.

[0061] For the rendering process 580, the interchange file 504 is analyzed by a synthesizer 585 to generate the appropriate haptic signals 506 as defined in the interchange file 504.

[0062] Figure 6 illustrates an example of data structure for the interchange file format describing haptic effect associated with a haptic emoji. The data structure 600 can be decomposed in a set of layers. At the upper layer, metadata 601 describe high-level metadata information regarding the overall haptic experience defined in the data structure 600 and a list of avatars 602 (i.e., body representation) later referenced in the file. These avatars allow to specify a target location of haptic stimuli on the body. The haptic effects are described through a list of perceptions 610, 6 IN. These perceptions correspond to haptic signals associated with specific perception modalities such as vibration, force, position, velocity, temperature, etc.). A perception comprises metadata 620 to describe the haptic content of the signal, information about devices 621 to describe specifications of the haptic devices for which the signal was designed and a list of haptic tracks 631, 63N. A haptic track comprises metadata 640 to describe the content of the track, the associated gain value, a mixing weight, body localization information and a reference to haptic device specification (defined at the perception level). The track finally contains a list of haptic bands 651, 65N, each band defining a subset of the signal within a given frequency range. For example, the haptic band 651 may correspond to the range of frequencies from 0 to 50 Hz while the haptic band 65N may correspond to the range of frequencies over 2 kHz. A haptic band comprises band data 660 to describe the frequency range of the band, the type of encoding modality (Vectorial or Wavelet), the type of band (Transient, Curve and Wave) and optionally the type of curve (Cubic, Linear or unknown) or the window length. A haptic band is defined by a list of haptic effects 671, 67N. Finally, a haptic effect comprises a list of keyframes 691, 69N and effect data 680, a keyframe being defined by a position (i.e. a temporal reference), a frequency and an amplitude. The effect data describes the type of base signal selected amongst Sine, Square, Triangle, SawToothUp, and SawToothDown as well as provide temporal references such as timestamps. The low-level haptic signal can then be reconstructed by combining the key frames of the haptic effects in the different bands, as illustrated in the example of figure 7.

[0063] Figure 7 illustrates an example of haptic signal coded using two haptic bands. With this technique, a low-level haptic signal is encoded using a two frequency bands, a low- frequency band 710 and a high-frequency band 720, each defining a part of the signal in a given frequency range. In this example, the low-frequency band corresponds to frequencies below 72.5 Hz while the high-frequency band corresponds to frequencies equal to or higher than 72.5 Hz. On the rendering side, the device combines the two parts (i.e., adds them) to generate the final haptic signal 740.

[0064] The data for a frequency band may be reconstructed based on keyframes and according to a type of haptic band selected amongst Transient, Curve and Wave bands. Additionally, for Wave bands, two types of encoding modalities can be used: Vectorial or Wavelet. Each band is composed of a series of Effects and each Effect is defined by a list of Keyframes that are represented as dots in the figure. The data contained in the effects and keyframes is interpreted differently for different types of haptic bands and encoding modalities.

[0065] For a Transient band, each effect stores a set of keyframes defining a position, an amplitude, and a frequency. A keyframe represents a transient event. The signal may be reconstructed using the type of periodic base signal specified in the effect metadata with the amplitude specified in the keyframe and the period given by the frequency of the keyframe. A transient event is a very short signal generated only for a few periods. The number of generated periods is determined by the decoder.

[0066] For a Curve band, each effect stores a set of keyframes defining a position (i.e., a temporal reference) and an amplitude. The keyframes represent control points of a curve and an interpolation is performed to generate the curve from the control points. The type of interpolation function is either cubic or linear and is specified in the metadata of the band. The signal may be reconstructed by performing an interpolation between the amplitudes of key frames according to their temporal references.

[0067] For Vectorial Wave bands, the effect stores a set of keyframes defining a position (i.e., a temporal reference), an amplitude and a frequency. In this case, the signal is generated using the type of periodic base signal specified in the effect metadata with the amplitude specified in the keyframe and the period given by the frequency of the keyframe. The SPIHT wavelet encoding scheme may be used for the Wavelet band or types of wavelet encoding. For example, for the Wavelet band, the effect may store the contents of one wavelet block. It contains a keyframe for every coefficient of the wavelet transformed and quantized signal, indicating the amplitude value of the wavelet. The coefficients are scaled to a range of [-1,1], Additionally, the original maximum amplitude is stored in a keyframe, as well as the maximum number of used bits. In this case, the signal may be reconstructed using the coefficients to perform an inverse wavelet transform.

[0068] The frequency band decomposition may use a low-pass filter and a high-pass filter to split the signal into a low-frequency band and a high-frequency band. The two bands are then processed differently. Various methods can be used for the encoding of the high-frequency part. A first solution is to split the high-frequency signal into smaller fixed length windows and use Short-time Fourier Transform (STFT) to decompose the signal in the frequency spectrum. Another solution is to use wavelet transforms to encode the high frequencies. The data structure allows to define multiple bands with different frequency ranges. These bands are used to store the coefficients of the Fourier or Wavelet Transforms.

[0069] For the low-frequency part of the signal, the data of this frequency band is stored through a list of keyframe points defined by a timestamp and an amplitude. The data also contains information relative to the type of interpolation used to reproduce the signal of this band. The keyframes (i.e., control points) defining the low-frequency band are obtained by simply extracting the local extrema of the low-frequency signal.

[0070] In the example of the figure, the low-frequency band 710 is defined as a Curve band using a single effect 711. Such representation is particularly adapted to the low-frequency part of the signal. The effect 711 is defined by the keyframes 7111, 7112, 7113, 7114, 7115, 7116, 7117, 7118, 7119. The signal for the low-frequency band is generated by a cubic interpolation between these keyframes. The high-frequency band 720 is defined by 4 effects 721, 722, 723, 724. The effect 721 is defined as a Vectorial band defined by 4 keyframes 7211, 7212, 7213, 7214.

[0071] While the description is based on a set of two bands defining a range for low frequencies and a range for high frequencies, the principles apply also in case more than two ranges of frequencies are used. In this case, the low-frequency band becomes the lowest frequency band, and the high-frequency band becomes the highest frequency band. The lowest frequency band may for example be encoded using a curve band using a single effect, as represented by the low-frequency band 710. Other frequency bands may be encoded with any of the other type of encoding, for example using a vectorial wave band based on wavelets, as represented by the high-frequency band 720 but using multiple instances of encoding, one for each band of frequencies.

[0072] One advantage of this solution with regards to the structure is that the signal data is easy to package and particularly convenient for streaming purposes Indeed, with such linear structure, the data can be easily broken down to small consecutive packages and does not require complicated data-pre-fetching operations. The signal is easily reconstructed by patching the packages back together to ensure a smooth playback of the signal. It may also be reconstructed by only taking the low-frequency part and reconstruct a lower quality (but potentially sufficient) signal without considering the high-frequency band.

[0073] As detailed in the following section, the further sections of this document describe the encoding of haptic signal based on a manually generated signal or of PCM waveform signals, for example carried by input WAV files. In this context, the haptic signal describes a single perception modality and even if the file contains multiple tracks, the encoder will process each track separately. Therefore, for the sake of clarity in the remainder of the disclosure, the description will describe the coding of a single track.

[0074] Figure 8A illustrates an example of method for encoding, in two bands of frequencies, a haptic signal for a haptic effect associated with a haptic emoji according to embodiments. This corresponds to the signal processing step 512 of figure 5 and is for example implemented by content generator 110 of figure 1. Given an input PCM signal, the content generator starts the process 800 by performing a frequency band decomposition. Using a low-pass filter and a high-pass filter, the encoder splits 810 the signal into low-frequency bands 811 and high- frequency bands 812. In step 820, the encoder analyses each low-frequency bands and extracts data 821 representing the low-frequency bands, and in step 830 analyses each high-frequency band and extracts data 831 representing the high-frequency bands.

[0075] The extracted data are then formatted according to the structure of figure 6 in the formatting step 520 of figure 5.

[0076] In a typical example implementation, there is a single low-frequency band that is encoded using a Curve band, so that the LF data comprises a set of keyframes extracted in step 820 and there is a single high-frequency band that is encoded using a vectorial wave band so that the HF data comprises a set of wavelets extracted in step 830, as described above.

[0077] This hybrid format combining Curve bands and Wave bands is interesting and allows to store low-frequency signals very easily. This is especially convenient for synthetic signals that were produced through Haptic authoring tools (in particular kinesthetic signals).

[0078] Figure 8B illustrates an example of method for decoding a haptic signal comprising two bands of frequencies for a haptic effect associated with a haptic emoji according to embodiments. This process 850 is for example implemented in a rendering device 100 of figure 1 and typically executed by the processor 101 of such device. The processor receives encoded data 851 generated according to figure 8 A, for example formatted according to the data structure described in figure 6. In step 860, the decoder reconstructs the signal 852 corresponding to the low-frequency band signal. In step 870, the processor reconstructs the signal 853 corresponding to the high-frequency band signal. By adding the signals 852 and 853 together, the processor will reconstruct the haptic signal 854 corresponding to the encoded data 851.

[0079] At least one embodiment uses the MPEG HIJF format for coding the haptic effect. This format provides different ways to code a haptic effect using a descriptive format based on the JSON language. The effect is described in terms of basics sinewaves (amplitude, frequency, duration). The receiver however needs to synthesize (i.e. generate) the effect to be able to provide it to the end device. The advantage of this solution is that the Tenderer can synthesize the effect for its targeted device and adapt to its capabilities. The following tables provide the HJIF implementation of the set of effects introduced in figure 2. The syntax is illustrated with the human readable interchange format as defined in the MPEG-I Haptics standard ISO / IEC 23090-31 : Haptics Coding. At least one embodiment uses the HMPG binary version of this format, based on MIHS.

[0080] The table 1 describes the syntax for the haptic effect 241 of figure 2 associated with the Unicode U+1F625, namely a simple high-frequency pic imitating someone crying. It uses a Vectorial wave coding.

[0081] Table 1 The table 2 describes the syntax for the haptic effect 242 of figure 2 associated with the Unicode U+1F602, namely a repeating high-frequency signal imitating someone laughing. It uses a transient curve coding.

[0082] Table 2

[0083] The table 3 describes the syntax for the haptic effect 243 of figure 2 associated with the

[0084] Unicode U+1F620, namely a repeating low-frequency signal imitating someone clearing the throat. It uses a vectorial wave coding.

[0085] Table 3

[0086] The table 4 describes the syntax for the haptic effect 244 of figure 2 associated with the Unicode U+1F970, namely a series of pics imitating heart beats. It uses a transient curve coding.

[0087] Table 4

[0088] The use of the MPEG HIJF format for coding the haptic effect necessitates to synthesize (i.e., generate) the signals after decoding to be able to transmit it to haptic actuators. In other words, a haptic signal associated with the haptic effect (represented by the haptic data representing the haptic effect as shown in the tables above) needs to be generated.

[0089] At least one embodiment uses the PCM format for coding the haptic effect. With such solution, the signal is directly generated at the emitter side in the form of a PCM signal (or captured with sensors) and encoded with the MPEG standard. In other words, the haptic signal associated with the haptic effect is comprised in the haptic data representing the haptic effect. This format simplifies the rendering, but it is less generic since it cannot easily adapt to different devices. Any other haptic codec can also be used, such as IEEE or proprietary codec, still using the same principles for the haptic emoji.

[0090] Although different embodiments have been described separately, any combination of the embodiments together can be done while respecting the principles of the disclosure.

[0091] Although embodiments are related to haptic effects, the person skilled in the art will appreciate that the same principles could apply to other effects such as the sensorial effects for example and thus would comprise smell, taste, temperature, emotions, intensity highlights, etc. Appropriate syntax would thus determine the appropriate parameters related to these effects.

[0092] Reference to “one embodiment” or “an embodiment” or “one implementation” or “an implementation”, as well as other variations thereof, mean that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase “in one embodiment” or “in an embodiment” or “in one implementation” or “in an implementation”, as well any other variations, appearing in various places throughout the specification are not necessarily all referring to the same embodiment.

[0093] Additionally, this application or its claims may refer to “determining” various pieces of information. Determining the information may include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory.

[0094] Additionally, this application or its claims may refer to “obtaining” various pieces of information. Obtaining is, as with “accessing”, intended to be a broad term. Obtaining the information may include one or more of, for example, receiving the information, accessing the information, or retrieving the information (for example, from memory or optical media storage). Further, “obtaining” is typically involved, in one way or another, during operations such as, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0095] It is to be appreciated that the use of any of the following “ / ”, “and / or”, and “at least one of’, for example, in the cases of “A / B”, “A and / or B” and “at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended, as readily apparent by one of ordinary skill in this and related arts, for as many items listed.

Claims

CLAIMS1. A method comprising: obtaining an identifier associated with a haptic emoji in a haptic emoji library comprising a plurality of haptic emojis, wherein a haptic emoji comprises at least an identifier, data representative of a graphical element representing the haptic emoji, and data representative of a haptic effect for the haptic emoji; obtaining from the haptic emoji library, based on the identifier, a graphical element and a haptic effect for the haptic emoji; rendering the graphical element; and rendering the haptic effect.

2. The method of claim 1, wherein rendering the haptic effect comprises obtaining a haptic signal associated with the haptic effect corresponding to the identifier from the haptic emoji library and providing this haptic signal to a haptic actuator.

3. The method of claim 1, wherein rendering the haptic effect comprises generating a haptic signal based on the data representative of the haptic effect corresponding to the identifier, the data being obtained from the haptic emoji library and providing the generated haptic signal to a haptic actuator.

4. The method of any of claim 1 to 3, wherein the haptic effect is rendered once when the graphical element is displayed.

5. The method of any of claim 1 to 3, wherein the haptic effect is rendered as long as the graphical element is displayed.

6. The method of any of claim 1 to 5, wherein the haptic effect is coded using a format defined in ISO / IEC 23090-31 : Haptics Coding.

7. An apparatus comprising a processor configured to: obtain an identifier associated with a haptic emoji in a haptic emoji library comprising a plurality of haptic emojis, wherein a haptic emoji comprises at least an identifier, data representative of a graphical element representing the haptic emoji, and data representative of a haptic effect for the haptic emoji;obtain from the haptic emoji library, based on the identifier, a graphical element and a haptic effect for the haptic emoji; render the graphical element; and render the haptic effect.

8. The apparatus of claim 7, wherein render the haptic effect comprises obtaining a haptic signal associated with the haptic effect corresponding to the identifier from the haptic emoji library and providing this haptic signal to a haptic actuator.

9. The apparatus of claim 7, wherein render the haptic effect comprises generating a haptic signal based on the data representative of the haptic effect corresponding to the identifier, the data being obtained from the haptic emoji library and providing the generated haptic signal to a haptic actuator.

10. The apparatus of any of claim 6 to 9, wherein the haptic effect is rendered once when the graphical element is displayed.

11. The apparatus of any of claim 6 to 9, wherein the haptic effect is rendered as long as the graphical element is displayed.

12. The apparatus of any of claim 6 to 11, wherein the haptic effect is coded using a format defined in ISO / IEC 23090-31 : Haptics Coding.

13. A non-transitory computer readable medium a storing a haptic emoji library comprising a plurality of haptic emojis, wherein a haptic emoji comprises at least: a unique reference identifying the haptic emoji from other haptic emojis, data comprising a graphical element for the haptic emoji, data comprising a haptic signal associated with a haptic effect for the haptic emoji.

14. A computer program comprising program code instructions for implementing the method according to any of claims 1 to 6 when executed by a processor.

15. A non-transitory computer readable medium comprising program code instructions for implementing the method according to any of claims 1 to 6 when executed by a processor.

Images