Method and apparatus for controlling a haptic actuator from position information of a source of haptic effect, and computer program product

Abstract

A method and apparatus for controlling a set of haptic actuators are provided. To achieve this, positional information, representing the position and orientation of a virtual avatar (11) relative to a source (12) of a haptic effect in a virtual environment and associated with a user (10), is obtained based on a time- and space-based description. First mapping information representing the mapping between the set of haptic actuators and parts of the set of virtual avatars (11) is obtained. The source (12) is projected onto the virtual avatar (11) based on the positional information. Control data for controlling each haptic actuator is determined based on the projection result and the first mapping information.

Description

Technical Field

[0001] This invention generally relates to the field of tactile sensing, and more particularly to the generation of control data for controlling a set of tactile actuators to produce tactile effects. The invention also relates to a method and apparatus for processing data representing tactile effects to generate control data. Background Technology

[0002] This section is intended to introduce the reader to various aspects of the art, which may relate to aspects of at least one exemplary embodiment of the invention described below and / or claimed. These descriptions are intended to help provide the reader with background information to facilitate a better understanding of the various aspects of the invention.

[0003] Haptic technology refers to any technology that, in presenting multimedia content, reproduces tactile sensations in a user interface by applying force, vibration, motion, and other senses (such as temperature), in addition to visual and audio information, to provide information to end users.

[0004] Haptic feedback encompasses a variety of possible stimulus implementations, but is mainly divided into tactile tactile technology and kinematic tactile technology: tactile tactile feedback (or tactile tactile effect) refers to sensations such as vibration, friction, or micro-deformation, while kinematic tactile feedback (or kinematic tactile effect) refers to sensations that provide mechanical stimulation and force sensations related to the body's position and movement.

[0005] Haptic feedback or tactile effects are achieved using one or more devices, which correspond to the placement of one or more tactile actuators distributed across one or more body parts of the user. For example, tactile effects can be achieved using eccentric rotating mass (ERM), linear resonant actuators (LRA), and wideband actuators such as voice coil motors (VCM), piezoelectric actuators (PZT), shape memory alloys (SMA), or pressure-type actuators. Kinematic effects can be achieved by the actuator applying a force that resists limb movement, and these effects are felt only in muscles and tendons, not on the skin. Other examples of tactile devices include resistance feedback devices, active force feedback devices, and skin indentation devices.

[0006] Haptic feedback devices can be used to enhance user immersion when watching video content associated with haptic effects. The haptic effects presented by the haptic device can be associated with various sources within the multimedia content, such as events occurring within a virtual scene of the video content. Distortion between the video content and the presentation of the haptic effects associated with it can lead to a poor user experience. For example, a mismatch between the location of the source of the haptic effect in a virtual scene and the location of the haptic actuator intended to present the haptic effect on the user's body can limit the user's experience when watching video content. Therefore, there is room for improvement to enhance user immersion when watching video / multimedia content associated with haptic effects. Summary of the Invention

[0007] The following sections present a simplified overview of at least one example embodiment to provide a basic understanding of some aspects of the invention. This overview is not a comprehensive summary of the example embodiments. It is not intended to identify key or essential elements of the embodiments. The following overview presents only some aspects of at least one example embodiment in a simplified form, as a premise for a more detailed description provided elsewhere herein.

[0008] According to a first aspect of the present invention, a method for controlling a set of tactile actuators is provided, the method comprising the steps of:

[0009] Obtain location information based on time and space, representing the location and orientation of a virtual avatar relative to the source of haptic effects in a virtual environment, and associate the virtual avatar with the user;

[0010] Obtain first mapping information representing the mapping between the set of haptic actuators and the parts of a set of virtual avatars;

[0011] Projecting the source onto the virtualization within the virtual environment based on location information;

[0012] Determine the control data for each tactile actuator in the group of tactile actuators. The control data is determined based on the projection results and the first mapping information.

[0013] In one example embodiment, the method further includes:

[0014] Receive second mapping information representing the mapping between the set of haptic actuators and body parts of the user's body, and posture information representing the user's posture;

[0015] Based on the second mapping information and posture information, spatial distribution information is determined representing the spatial distribution of the set of haptic actuators on the user's body relative to the user's body, using a set of body reference planes. This set of body reference planes includes at least one body reference plane, each dividing the body into two complementary parts.

[0016] The first mapping information is determined based on spatial distribution information and a set of virtual avatar reference planes. The set of virtual avatar reference planes includes at least one virtual avatar reference plane. Each virtual avatar reference plane corresponds to a different body reference plane in the set of body reference planes. Each virtual avatar reference plane divides the virtual avatar into two complementary parts.

[0017] In one example embodiment, the control data includes time control information for controlling each haptic actuator based on time.

[0018] In one example embodiment, the method further includes: receiving an input tactile signal representing a tactile effect, wherein control data includes tactile data for controlling each tactile actuator, the tactile data being determined based on the input tactile signal and the result of projection.

[0019] In one example embodiment, tactile data is determined by processing the input tactile signal based on the result of projection.

[0020] In one example embodiment, the processing includes amplitude modulation of the input tactile signal according to an amplitude parameter, which is the result of projection.

[0021] In another example embodiment, the amplitude parameter is based on the distance between the source and each tactile actuator.

[0022] In another example embodiment, the processing includes frequency offsetting of the input tactile signal to obtain the Doppler effect as the distance between the source and the virtual avatar increases or decreases.

[0023] In another example embodiment, the method further includes controlling each tactile actuator according to control data.

[0024] In another example embodiment, multiple haptic actuators of the group of haptic actuators are controlled based on a time sequence according to time control information.

[0025] In another example embodiment, at least two of the haptic actuators in the group are controlled simultaneously according to control data to present at least a portion of the haptic effect.

[0026] In one example embodiment, the projection is based on a conical projection, where the vertices of the cone correspond to the source of the tactile effect.

[0027] According to a second aspect of the invention, there is provided an apparatus for controlling a set of tactile actuators, wherein the apparatus includes a memory associated with at least one processor configured to implement the method according to a first aspect of the invention.

[0028] According to a third aspect of the invention, a computer program product including instructions is provided, which, when executed by one or more processors, causes the one or more processors to perform the method according to the first aspect of the invention.

[0029] According to a fourth aspect of the invention, a non-transient storage medium is provided, which carries instructions for performing program code according to a first aspect of the invention.

[0030] The specific nature of at least one exemplary embodiment, as well as other objects, advantages, features, and uses of the at least one exemplary embodiment, will become apparent from the following description of the examples in conjunction with the accompanying drawings. Attached Figure Description

[0031] The drawings of exemplary embodiments of the present invention will now be shown by way of example, wherein:

[0032] Figure 1 This diagram illustrates a user viewing video content displayed on a screen according to at least one example embodiment.

[0033] Figure 2 This illustrates communication according to at least one example embodiment. Figure 1 A diagram illustrating the communication network for the video content;

[0034] Figure 3 The following is illustrated according to at least one example embodiment. Figure 1 A schematic diagram of the video content processing scheme;

[0035] Figure 4 According to at least one example embodiment, Figure 1 users and Figure 1 A diagram illustrating the mapping between the video content and the user's virtual avatar.

[0036] Figure 5 This illustrates, according to at least one example embodiment, with Figure 1 The source of the haptic effects associated with the video content is projected onto... Figure 4 A diagram illustrating virtualization;

[0037] Figure 6 This illustrates, according to at least one example embodiment, an application to representing and Figure 5 The first instance of the result of processing the input tactile signal associated with the source of the tactile effect;

[0038] Figure 7 This illustrates, according to at least one example embodiment, an application to representing and Figure 5 The second instance is the result of processing the input tactile signal associated with the source of the tactile effect;

[0039] Figure 8 The control configuration according to at least one example embodiment is shown to present and Figure 5 A schematic block diagram illustrating the steps of a method for using a set of tactile actuators to generate tactile effects related to their sources;

[0040] Figure 9 The schematic block diagram illustrating various aspects and example embodiments of the system is shown.

[0041] In different diagrams, similar element symbols can be used to represent similar components. Detailed Implementation

[0042] At least one exemplary embodiment will be described more fully below with reference to the accompanying drawings, in which examples of at least one exemplary embodiment are illustrated. However, the exemplary embodiments may be implemented in many alternative forms and should not be construed as limited to the examples described herein. Therefore, it should be understood that this document is not intended to limit the exemplary embodiments to the specific forms disclosed. Rather, this document is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

[0043] The example embodiments described below may be combined with each other, or each example embodiment may be implemented alone, for example, as an alternative embodiment.

[0044] At least one of these aspects generally relates to including a control of a set of haptic actuators, the set of haptic actuators including one or more haptic actuators for presenting haptic effects. The haptic effects may be associated with multimedia content (e.g., video content), and data representing the haptic effects are received together with video data representing the video content (e.g., included in the video data).

[0045] Tactile actuators include, for example, LRA (multiple), VCM (multiple), ERM, PZT, SMA, and / or pressure-based actuators (e.g., soft pneumatic actuators). Pressure-based actuators are configured to provide pressure-based feedback, such as through pneumatic actuation.

[0046] Obtain location information based on time and space, representing the position and orientation of the virtual avatar relative to the source of the haptic effect to be presented. The location information can be obtained based on the location of the source of the haptic effect within the virtual environment in which the virtual avatar moves, and based on the virtual avatar's location within the virtual environment. The virtual avatar may correspond to a virtual object in the virtual environment, representing a user viewing video content representing the virtual environment. The location of the source and / or (user-controlled) location of the virtual avatar may change over time within the virtual environment.

[0047] The sources of haptic effects can correspond to visual and / or audio events occurring in the virtual environment, such as explosions, gunshots, footsteps, thunder, falling objects, etc.

[0048] Haptic effects can be represented by data representing input haptic signals received along with video data / signals representing video content. Audio data / signals representing audio content associated with the video content can further be received along with the video data / signals. The input haptic signals are intended to be provided to one or more haptic actuators for realizing haptic effects when events associated with the haptic effect are presented, i.e., displaying and / or audio presenting events, depending on the type of event.

[0049] First mapping information representing the mapping between the set of haptic actuators and a set of virtual avatar parts can be obtained, for example, received from memory or determined from other information. The virtual avatar parts may correspond to various body parts of the user associated with the virtual avatar.

[0050] Determine the control data for each haptic actuator in the group of haptic actuators. The control data is determined based on the first mapping information and the projection of the source of the haptic effect onto the virtual body in the space of the virtual environment, the projection being intended to determine which part(s) of the virtual body can come into contact with the source of the haptic effect.

[0051] This process can select one or more haptic actuators most relevant to presenting haptic effects from a set of haptic actuators available for presenting haptic effects, based on the displacement of the avatar in the source and / or virtual environment. The matching or correlation between the location of the source relative to the location of the avatar and the haptic actuators used to present haptic effects is improved.

[0052] Figure 1 This diagram illustrates a user 10 watching video content according to at least one example embodiment.

[0053] Figure 1 This illustrates user 10 viewing video content displayed on display device 100. Display device 100 may be a monitor or screen connected to a computer, a laptop screen, a screen of a mobile device such as a tablet computer or smartphone, a head-mounted display (HMD) screen, that is, any device configured to display image pixels of video content.

[0054] according to Figure 1 In a non-limiting example, user 10 is sitting in a chair. A set of tactile actuators is advantageously associated with various body parts of user 10, and the tactile actuators belong to one or more tactile devices.

[0055] Examples of equipment or devices that may correspond to haptic devices include head-mounted displays (HMDs, see-through glasses), wireless (e.g., Bluetooth®) connected wearable haptic devices, haptic suits, mouse pads, palm rests, chairs, tables, XR helmets, headphones, wristbands, head and / or lumbar supports or chairs, mice, or any other device suitable for presenting haptic effects to one or more parts of a user’s body that come into contact with the haptic device.

[0056] According to the variant example, user 10 can adopt a posture different from sitting, such as standing, lying down, reclining, lying down, or squatting.

[0057] The video content displayed on the display device 100 includes a sequence of images of a virtual environment, and the images in the video content correspond to images generated by a computer. According to a variation, the images in the video content may be mixed with images of the real-world environment and virtual objects generated by a computer, and the video content may correspond to augmented reality (AR) content or mixed reality (MR) content.

[0058] Audio content can be associated with video content and used to present sound through speakers when the video content is displayed.

[0059] Haptic effects are advantageously associated with video content, which in turn corresponds to visual-tactile content.

[0060] The video content may, for example, correspond to a video game that user 10 is playing. In the video game, the sequence of images displayed on display device 100 may depend on the instructions received by the user interface used by user 10 to interact with the content of the video game. This user interface may correspond to a mouse, keyboard, or game controller.

[0061] User 10 can use a user interface to control the movement of the virtual avatar 11 within the virtual environment of the video content displayed on the display device 100. The virtual avatar 11 corresponds to virtual objects, such as characters or roles in a video game scene.

[0062] Depending on the instructions provided by user 10, virtual avatar 11 can be displayed in the virtual environment according to different positions and orientations that change over time. In another example, the virtual avatar may not be visible in the virtual environment. In this example, the video content representing the virtual avatar moving to different scenes in the virtual environment is displayed from the virtual avatar's viewpoint, just as if the virtual avatar were moving to a scene seen within the virtual environment.

[0063] One or more sources of haptic effects are advantageously associated with the video content, such as corresponding to various events that occur as the virtual avatar 11 evolves in the virtual environment.

[0064] Events associated with tactile effects can correspond to visual events, audio events, or both audio and visual events.

[0065] An example of a source 12 for a haptic effect is represented by a star. The location of the source of the haptic effect may change in space and / or over time, depending on the type of event associated with the haptic effect. For example, an event may correspond to the sound of footsteps of a character or role in a video game walking or moving near the virtual avatar 11, whether the character is visible or invisible to the virtual avatar 11. A haptic signal may be associated with each footstep heard by the virtual avatar 11, and this haptic signal is intended to be provided to one or more haptic actuators associated with one or more body parts of the user 10.

[0066] In one embodiment, video data representing video content is stored, for example, in a storage device (e.g., a memory or hard disk) of a device used to present the video content (e.g., a computer, laptop, tablet, or smartphone). Audio data representing audio content and associated with the video content, and haptic data representing haptic effects associated with the video content, may be stored in the storage device.

[0067] In another embodiment, video data, audio data, and / or tactile data (as described below) are received via a communication network. Figure 2 (as described).

[0068] Figure 2 A schematic diagram of a communication system 2 for communicating one or more contents according to at least one example embodiment is shown.

[0069] According to one embodiment, the communication system 2 may include various elements (21, 22, 23), each element (21, 22, 23) corresponding to a system, device, or apparatus, which are interconnected via a network 20. The network 20 may correspond, for example, to a Wide Area Network (WAN), a Local Area Network (LAN), or a Wireless Local Area Network (WLAN), or a combination of one or more of these networks. According to other examples, the network 20 may correspond to a broadcast network, a cellular network, a satellite network, the Internet, or any combination thereof. The first apparatus 21 may correspond to a server, a transmitter, and / or an encoder, while the second apparatus 22 may correspond to a computing device, a rendering engine, a receiver, and / or a decoder.

[0070] According to another embodiment, network 20 may correspond to a communication and / or data bus that interconnects devices (21, 22, 23), some of which are, for example, part of the same device, such as first device 21 and second device 22. First device 21 may correspond to a storage device, which may include non-volatile memory and / or volatile memory (including, but not limited to, electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drive, and / or optical disk drive). As a non-limiting example, storage device (21) may correspond to an internal storage device (i.e., first device 21 and second device 22 are part of the same device), an attached storage device (e.g., connected to second device 22 via a USB interface), and / or a network-accessible storage device.

[0071] The devices (21, 22, 23) of communication system 2 may each correspond to a single element or device or to multiple elements or devices that are part of the system.

[0072] Network 20 is capable of transmitting data representing one or more pieces of content belonging to a set of content, which includes:

[0073] Graphical content representing a virtual environment, to obtain data representing a set of scenes of the virtual environment when rendering the content, such as data representing the color values ​​(e.g., RGB values) of the pixels of the image;

[0074] Haptic content, i.e., haptic data representing one or more haptic effects to be presented by one or more haptic devices (23); haptic data may be haptic parameters representing the haptic effects to be presented, such as parameters representing the amplitude and / or frequency of the haptic effect, parameters representing the instantaneous moment, parameters representing the location of the source in the space of the virtual environment, parameters representing the start and end positions of the haptic animation in the space of the virtual environment of the video content, parameters representing the type of haptic effect to be presented, parameters representing the target haptic device to present the haptic effect, and / or parameters representing the presentation time and / or presentation period of the haptic effect;

[0075] Audio content, for example, audio data representing sound, speech, language, and music associated with video content, presented by one or more speakers.

[0076] Some or all of the received content may be associated with or related to each other. For example, tactile data may be associated with graphic content and / or audio content to present tactile effects when playing (audio)video content, and the presented tactile effects are related to the video and / or audio content played when presenting the tactile effects.

[0077] Time or synchronization information can be associated with graphic content, haptic content, and audio content to synchronize the presentation of related content.

[0078] The content transmitted from the first device 21 to the second device 22 can be in any form. For example, the content can be transmitted as an analog signal or a digital signal representing data associated with the corresponding content.

[0079] Once received by the second device 22, the signal can be converted from analog to digital, or vice versa, as those skilled in the art will recognize.

[0080] The second device 22 is configured to process the received signal, primarily for generating or determining control data representing one or more haptic effects based on the received content (e.g., video content and haptic content). The processing performed by the second device 22 may include analog-to-digital conversion, frequency filtering (e.g., low-pass filtering, band-pass filtering), time filtering (e.g., selecting a time window), downsampling, transformation from time space to frequency space, transformation from frequency space to time space, etc., as will be apparent to those skilled in the art.

[0081] The following will refer to Figure 9 Describe the first device 21 and / or the second device 22.

[0082] The second device 22 is communicatively coupled to a group of third devices 23, which include one or more tactile devices, each including one or more tactile actuators.

[0083] Figure 3 A schematic diagram illustrates a scheme for generating control data for controlling a set of haptic actuators, according to at least one example embodiment.

[0084] Blocks 31 to 33 represent, without distinction, the operations / steps of the processing / method, and the processing unit that implements the operations / steps of the processing / method. The processing unit (31 to 33) can be implemented as a single element, or as an independent element of a combination of hardware and software as known to those skilled in the art.

[0085] At block 31, position information 312, which describes the position and orientation of the virtual avatar 11 relative to the source 12 of the haptic effect, based on time and space, is obtained in the virtual environment representing the graphic content.

[0086] Location information 312 is determined, for example, based on virtual avatar control data 311 and data 301 representing the virtual environment.

[0087] Virtual avatar control data 311 can be received from one or more peripheral devices used by user 10 to control the movement of virtual avatar 11 in a virtual environment represented by video content displayed on display device 100. Data 301 may correspond to graphic data received from first device 21, such as data representing three-dimensional (3D) objects to be presented, for displaying images representing various 3D scenes of the virtual environment.

[0088] The data representing the position and orientation of the virtual avatar 11 in the virtual environment is determined based on the displacement of the virtual avatar 11 within the virtual environment. The orientation of the virtual avatar 11 corresponds, for example, to the direction in which the virtual avatar 11 moves or the direction in which the virtual avatar 11 is looking within the virtual environment.

[0089] Data 301 may include source location information representing the location of any source of the haptic effect associated with the virtual environment. The source location information may correspond to 3D coordinates in the space of the virtual environment, and the source of the haptic effect may correspond to an event occurring at the location of the source in the virtual environment.

[0090] According to one variation, information describing the relative position of the source with respect to the virtual avatar can be stored in digital media (e.g., a file). The file can be stored in the memory of the second device 22 and / or received from the first device 21 as a data stream (e.g., a data stream representing a video including the virtual avatar 11 and haptic effects from the source). According to this variation, the virtual avatar 11 can be a character corresponding to the graphical content of the video. The position of the virtual avatar 11 in the environment and the position of the source 12 can be predetermined.

[0091] Therefore, location information 312 represents the relative positioning and orientation of the source 12 of the haptic effect with respect to the virtual avatar 11 in the virtual environment.

[0092] Location information 312 includes data describing the relative position of source 12 with respect to virtual avatar 11 in space and / or time, where source 12 and / or virtual avatar 11 are affected by movement or displacement within the virtual environment. In this respect, location information 312 describes a sequence of spatial positions of source 12 relative to virtual avatar 11. In a variant, the sequence of spatial positions of source 12 relative to virtual avatar 11 may be described over time (i.e., in real time).

[0093] The description of a sequence of positions over time can correspond to a series of locations, such as a series of 3D coordinates, each coordinate describing the position of source 12 at different times "t". Additional information representing direction (e.g., data representing vectors) can be associated with each point or location in this series.

[0094] Haptic effects are associated with each source of haptic effects included in the virtual environment. Haptic effects are indicated by input haptic signals that transmit data representing the haptic effect (e.g., data representing parameters of the haptic effect, such as amplitude and / or frequency parameters whose values ​​may change over time).

[0095] The input tactile signal can be received together with the signal that transmits data 301 representing the virtual environment.

[0096] According to one variation, the input haptic signal is based on the type of source or an event associated with the source. In this variation, the type of event associated with the haptic effect is used to determine the haptic effect to be presented. The input haptic signal representing the haptic effect to be presented is obtained, for example, from a haptic effect library through mapping information (e.g., a look-up table, LUT) that maps each type of event to a haptic effect. The type of event is included, for example, in data 301, and corresponds to a unique identifier that identifies each type of event.

[0097] At block 32, the source 12 is projected onto the virtual avatar 11 in the virtual environment based on the position information 312 received from block 31.

[0098] The projection from source 12 onto virtual avatar 11 can determine which parts of virtual avatar 11 can receive the effects associated with the events associated with source 12, and thus determine which parts can be exposed to tactile effects.

[0099] A series or sequence of projections can be performed, and the projection of each position of source 12 is described with position information 312. Advantageously, the projections can be performed according to the continuous temporal order of the positions taken by source 12 and described temporally with position information 312.

[0100] The projection can be of any form, such as a conical projection, where the vertex of the cone corresponds to a point representing the location of source 12 in the virtual environment at a given time "t".

[0101] According to at least one example embodiment, this projection is shown in Figure 5 middle.

[0102] Figure 5 This shows the temporal change in position of source 12 between its initial position 501 at initial time "t1" and its final position 502 at final time "t2", based on the displacement of source 12 relative to virtual avatar 11. The relative displacement between the initial position 501 and the final position 502 is shown in... Figure 5 The middle part is indicated by an arrow.

[0103] Figure 5The initial conical projection of source 12 at initial position 501 is shown, with the projected cone represented by a solid line, and the final conical projection of source 12 at final position 502 is shown, with the projected cone represented by a dashed line.

[0104] For example, the projection cone can be defined around a main direction. The main direction can be defined at two points, the first point corresponding to the position of source 12, and the second point corresponding to the reference point of virtual avatar 11, such as the center of gravity of virtual avatar 11 or a point on the head of virtual avatar 11 (e.g., the middle of the line segment connecting the eyes of virtual avatar 11).

[0105] The base of the projection cone can be of any shape, and its defined dimensions correspond to either the fixed parameters or the settable parameters of the system.

[0106] Each projection can determine or obtain projection data 321 representing the result of the projection, and the projection data 321 indicates the position of the source 12 relative to the various parts of the virtual avatar 11.

[0107] When virtual avatar 11 corresponds to a graphic object representing a person, the parts of virtual avatar 11 can correspond to the parts of virtual avatar 11's body, and are called virtual avatar body parts.

[0108] When haptic actuators 51, 52, 53, 54 are associated with one or more parts of the virtual avatar 11, projection data 321 indicates the position of source 12 relative to each of the set of haptic actuators (51 to 54) associated with the virtual avatar 11.

[0109] The projection data 321 obtained from the projection may include:

[0110] Data representing the visibility of each part / haptic actuator of virtual avatar 11: Visibility information can be specified for each part / haptic actuator, for example, taking two values: a first value (e.g., "1") when a part / haptic actuator is visible to source 12, and a second value ("0") when a part / haptic actuator is not visible to source 12; when the projection produces a line belonging to the cone that connects source 12 to a part / haptic actuator without traversing the virtual environment. When any object is present, that part / haptic actuator is visible to source 12; for example, haptic actuators 51, 52, and 53 are each visible to source 12 at their initial positions 501, while haptic actuator 54 is not visible to source 12 because it is concealed by the body of the virtual avatar 11 itself; haptic actuators 51, 52, and 54 are each visible to source 12 at their final positions 502, while haptic actuator 53 is not visible to source 12 because it is concealed by the body of the virtual avatar 11 itself; and / or

[0111] Data representing the first distance d between each part / haptic actuator of source 12 and virtual avatar 11; and / or

[0112] Data representing the second distance d' between the part / haptic actuator of the virtual avatar 11 and the center 500 of the bottom of the projection cone including the part / haptic actuator of the virtual avatar 11; and / or

[0113] Data representing the angle between source 12 and each part / haptic actuator of virtual avatar 11, such angle as corresponding to the angle between source 12 and each part / haptic actuator and the normal associated with each part / haptic actuator.

[0114] The association between the haptic actuators 51 to 54 and the various parts of the virtual avatar 11 can be obtained from first mapping information 331, which indicates the mapping between each haptic actuator and a part of the virtual avatar 11. The first mapping information 331 may, for example, be stored in the LUT of the memory of the second device 22.

[0115] If the first mapping information 331 is unavailable during projection processing, the projection data 321 indicates the position of the source 12 relative to the various parts of the virtual avatar 11.

[0116] At block 33, control data 332 for controlling each tactile actuator in a set of tactile actuators is determined based on the first mapping information 331 and the projection result indicated in the projection data 321 received from block 32.

[0117] The first mapping information 331 is obtained (i.e., received) for example from the memory of the second device 22 (e.g., from the first device 21). According to this example, the first mapping information 331 may be input by the user 10 via a user interface before watching video content associated with a virtual environment / playing a video game associated with a virtual environment. According to another example, the first mapping information 331 corresponds to a set of parameters of the rendering engine corresponding to the second device 22, such as hard-coded parameters.

[0118] In one example embodiment, the first mapping information 331 is obtained (i.e. determined) from spatial distribution information representing the spatial distribution of the set of haptic actuators on the body of the user 10, and information establishing the relationship between the user 10’s posture in the real world and the virtual avatar 11 in the virtual environment.

[0119] Reference Figure 4 The procedure for determining the first mapping information 331 according to at least one example embodiment is described.

[0120] In the first operation of this procedure, second mapping information representing the mapping between the set of tactile actuators (41 to 45) and body parts of the user 10 is received, as well as posture information representing the posture of the user 10.

[0121] Figure 4 Some of the tactile actuators in this group are shown, namely, tactile actuator 41 associated with the user 10's left ear (tactile actuator 41 is disposed, for example, in the left earcup of the headphones), tactile actuator 42 associated with the user 10's back, tactile actuator 43 associated with the user 10's chest, tactile actuator 44 associated with the user 10's left hand, and tactile actuator 45 associated with the back of the user 10's left thigh. Of course, Figure 4 Other tactile actuators not shown may belong to this group of tactile actuators.

[0122] The second mapping information may be received from the memory of the second device 22, for example, or stored in a LUT. The second mapping information may be input by the user 10 via a user interface.

[0123] According to a variation, second mapping information is received from each haptic device (23), for example, when each haptic device (23) is connected to a second device 22. When a haptic device (23) is connected (e.g., via USB) to the second device 22 that implements the rendering engine, the second mapping information may be a portion of the haptic device data automatically received from each haptic device (23). When a haptic device (23) is connected to the haptic engine, the haptic device (23) automatically initiates a process for transmitting haptic device data via a dedicated API. This process corresponds, for example, to a so-called "plug and play (PnP)" process, with the haptic device (23) and the haptic engine corresponding to PnP devices.

[0124] For example, when starting to watch video content associated with a virtual environment or play video games associated with a virtual environment, gesture information can be parameters input by user 10 via a dedicated user interface.

[0125] Posture information can be selected from a defined list including sitting, standing, prone, reclining, lying down, and / or squatting.

[0126] In one variation, pose information is determined by image processing of images of user 10 acquired by a camera (e.g., a camera positioned in the real world of user 10, such as on top of display device 100 or integrated into display device 100). The pose information can be determined by any method known to those skilled in the art, for example, as described in “Human Posture Recognition Based on ImagesCaptured by the Kinect Sensor” published January 1, 2016 by Wen-June Wang et al.

[0127] In the second operation, based on the second mapping information and posture information, spatial distribution information is determined representing the spatial distribution of the set of tactile actuators (41 to 45) on the body of user 10 relative to a set of body reference planes of user 10.

[0128] This set of body reference planes includes one or more reference planes, each of which divides the user 10's body into two complementary parts. The one or more reference planes are defined in the space of the user 10's real-world environment, corresponding to an orthogonal coordinate system represented by orthogonal axes X, Y, and Z.

[0129] Figure 4 Two body reference planes (401, 402) are shown: a first body reference plane 401 defined in an orthogonal coordinate system (YZ) and including the center of the user 10’s head, and a second body reference plane 402 defined in an orthogonal coordinate system (XY) and including the center of the user 10’s pelvis.

[0130] The first body reference plane 401 divides the user 10's body into two complementary parts corresponding to the front and rear of the user 10's body. The second body reference plane 402 divides the user 10's body into two complementary parts corresponding to the upper and lower parts of the user 10's body. The third body reference plane (not shown, defined in an orthogonal coordinate system plane (XZ) and including the center of the user 10's head) can divide the user 10's body into two complementary parts corresponding to the left and right sides of the user 10's body.

[0131] Spatial distribution information can define which complementary part each tactile actuator 41 to 45 belongs to, defined by one or more body reference planes, and this information can vary depending on posture.

[0132] For example, Figure 4In the standing posture shown, the tactile actuator 45 associated with the back of the user 10's thigh is located at the rear relative to the first body reference plane 401. In the sitting posture, the same tactile actuator 45 associated with the back of the user 10's thigh is located at the front of the user 10 relative to the first body reference plane 401.

[0133] In another example, Figure 4 In the standing posture shown, the haptic actuator 44 associated with the user 10's hand is located at the lower part relative to the second body reference plane 402. In the sitting posture, the same haptic actuator 44 associated with the user 10's hand is located at the upper part of the user 10 relative to the second body reference plane 402.

[0134] Then, based on the spatial distribution information and a set of virtual avatar reference planes (411, 412), the first mapping information indicating the mapping between the tactile actuators 41 to 45 and the virtual avatar 11 can be determined. Each virtual avatar reference plane (411, 412) corresponds to a different body reference plane in the set of body reference planes, and each virtual avatar reference plane (411, 412) divides the virtual avatar 11 into two complementary parts.

[0135] The set of virtual avatar reference planes (411, 412) includes one or more reference planes, each of which divides virtual avatar 11 into two complementary parts. The one or more virtual avatar reference planes (411, 412) are defined in the space of the virtual environment of virtual avatar 11 corresponding to an orthogonal coordinate system represented by orthogonal axes X', Y', and Z'.

[0136] Figure 4 Two virtual avatar reference planes (411, 412) are shown: a first virtual avatar reference plane 411, associated with and defined with the same elements as the first body reference plane 401, i.e., in plane (Y'Z'); and a second virtual avatar reference plane 412, associated with and defined with the same elements as the second body reference plane 402, i.e., in plane (X'Y'). A third virtual avatar reference plane may be further defined in a manner consistent with the third body reference plane.

[0137] The first virtual avatar reference plane 411 divides the virtual avatar 11 into two complementary parts, namely, the front and rear parts of the virtual avatar 11; the second virtual avatar reference plane 412 divides the virtual avatar 11 into two complementary parts, namely, the upper and lower parts of the virtual avatar 11; and the third virtual avatar reference plane divides the virtual avatar 11 into two complementary parts, namely, the left and right parts of the virtual avatar 11.

[0138] Using a body and avatar reference plane to determine the first mapping information can adjust the mapping between the haptic actuator and the parts of the avatar that take into account the user's posture. In fact, the user's posture can affect the haptic actuator that should present a haptic effect, thereby improving the consistency between haptic and visual feedback of the scene displayed on the display device 100.

[0139] Changes in the user 10's posture can alter the projection data 321, such as the visibility information associated with each haptic actuator. For example, depending on whether the haptic actuator 45 is in the front or rear, the haptic effect from a source located in front of the virtual avatar 11 will be perceived differently: when the haptic actuator 45 is in the front, the haptic actuator 45 is visible to the source and can be selected to present a haptic effect, while when the haptic actuator 45 is in the rear, the haptic actuator 45 is not visible to the source and can not be selected to present a haptic effect.

[0140] The control data 332 determined at block 33 for controlling the haptic actuator based on the first mapping information 331 and projection data 321 may include, for example, time control information for controlling each haptic controller 41 to 45 based on time. This time control information may include, for example, the start time and end time of each haptic actuator's activation. Therefore, when the start time equals the end time, the defined activation period is zero.

[0141] The start time can correspond to the time associated with the first projection of the haptic actuator becoming visible to the source 12 (in terms of time), while the end time can correspond to the time associated with the last projection of the haptic actuator becoming invisible to the source 12 after becoming visible (in terms of time).

[0142] The control data 332 may further include tactile data for controlling each tactile actuator, which is determined based on the input tactile signal associated with the source 12 of the tactile effect and the projection data 321.

[0143] Tactile data can be determined by processing the input tactile signal based on projection data 321.

[0144] In the first example, the process includes amplitude modulation of the input tactile signal based on an amplitude parameter, which is determined according to projection data 321. For example, the amplitude can be determined based on a transformation function that determines the amplitude based on the projection data 321.

[0145] The amplitude parameter is, for example, based on the distance between the source 12 of the haptic effect and the haptic actuator, which can vary over time with the displacement of the source 12 relative to the virtual avatar 11. For instance, the amplitude is a function of distance; the longer the distance, the smaller the amplitude, and vice versa. According to a variation, the amplitude parameter is based on the angle between the source 12 of the haptic effect and the haptic actuator, which can vary over time with the displacement of the source 12 relative to the virtual avatar 11.

[0146] According to at least one example embodiment, Figure 6 Showing the Figure 5 The amplitude modulation of the input tactile signal applied to each of the tactile actuators 51 to 54 is the result of the tactile signal.

[0147] Figure 6 This illustrates the control signals / data to be provided to the tactile actuator to produce the tactile effect indicated by the input tactile signals / data associated with the source 12, based on the displacement of the source 12 between the initial position 501 and the final position 502.

[0148] Figure 6 Each graph shows the control signals to be provided to the different tactile actuators 51 to 54. The horizontal axis represents time, and the vertical axis represents amplitude. All graphs have the same origin for time and amplitude.

[0149] The control signal 61 in the upper diagram corresponds to the control signal to be provided to the haptic actuator 53. The control signal 62 in the lower diagram corresponds to the control signal to be provided to the haptic actuator 51. The control signal 63 in the lower diagram corresponds to the control signal to be provided to the haptic actuator 52. The control signal 64 in the lower diagram corresponds to the control signal to be provided to the haptic actuator 54.

[0150] according to Figure 5 The relative motion of source 12, as shown, in time, affects the first haptic actuator corresponding to haptic actuator 53, then the second haptic actuator corresponding to haptic actuator 51, then the third haptic actuator corresponding to haptic actuator 52, and finally the last haptic actuator corresponding to haptic actuator 54. Therefore, in time, haptic actuator 53 is controlled first to produce a haptic effect, followed by haptic actuator 51, then haptic actuator 52, and finally haptic actuator 54.

[0151] Simultaneously controlling multiple haptic actuators to produce a haptic effect corresponds to the time period during which the projection of these multiple haptic actuators onto the same position of source 12 by source 12. For example, as highlighted by the dashed rectangle 600, haptic actuators 53, 51, and 52 are activated simultaneously during a defined time period, and control signals 61, 62, and 63 overlap during this defined time period.

[0152] The amplitudes of control signals 61 to 64, generated by amplitude modulation applied to the input tactile signal, vary from one control signal to another, and change in real time for each control signal 61 to 64.

[0153] like Figure 6 As shown, when source 12 moves closer to tactile actuators 51 and 52 than to tactile actuators 53 and 54, the amplitude of tactile actuators 51 and 52 (for control signals 62 and 63, respectively) is higher than that of tactile actuators 53 and 54 (for control signals 61 and 64, respectively).

[0154] The amplitude can be determined as a function of the first distance d and / or the second distance d'.

[0155] The first distance d corresponds to the distance between the position of source 12 and the location / position of the tactile actuator under consideration.

[0156] The second distance d' corresponds to the distance between the location / position of the considered haptic actuator and the center of the projection surface generated by the projection of the source 12 onto the virtual avatar 11, which includes the considered haptic actuator (or a point representing the considered haptic actuator). When the projection corresponds to a conical projection, the projection surface corresponds to, for example, the bottom of a projection cone that may correspond to a circle or an ellipse. The second distance d' is considered to determine the amplitude, enabling continuous and / or smooth changes as the source 12 and / or the virtual avatar 11 moves.

[0157] In a second example, the process includes shifting the frequency of the input tactile signal to achieve a Doppler effect as the distance between the source 12 and the virtual avatar 11 increases or decreases. For example, the frequency (and optionally the amplitude) can be determined according to a transformation function that determines the frequency (and amplitude) based on the projection data 321.

[0158] For example, when source 12 moves away from virtual avatar 11 according to the main direction, the frequency of the input haptic signal can be shifted to a lower frequency to obtain a control signal. When source 12 moves closer to virtual avatar 11 according to the main direction, the frequency of the input haptic signal can be shifted to a higher frequency to obtain a control signal.

[0159] According to at least one example embodiment, Figure 7 Showing the Figure 5The result of frequency modulation applied to the input tactile signal of some tactile actuators.

[0160] Figure 7 The diagram illustrates control signals / data to be provided to the haptic actuator to present a haptic effect indicated by the input haptic signals / data associated with the source 12, depending on the displacement of the source 12, particularly when the source 12 moves away from or closer to the virtual avatar 11 according to the main direction.

[0161] Figure 7 Each graph shows the frequency and amplitude parameters of the control signals to be provided to the different tactile actuators 51, 52, and 53. The horizontal axis represents time "t", the left vertical axis represents amplitude "A", and the right vertical axis represents frequency "F". All graphs have the same origin for time, amplitude, and frequency.

[0162] Figure 7 Each figure shows a frequency curve 70 as a function of time, indicated by a dashed line, and amplitude curves 71, 72, and 73 as a function of time, indicated by solid lines.

[0163] The upper graph applies to haptic actuator 53, the graph between the upper and lower graphs applies to haptic actuator 51, and the lower graph applies to haptic actuator 52. Frequency curve 70 is the same for all haptic actuators 51, 52, and 53, and it shows the variation of the frequency components of the control signal used to control the haptic actuator.

[0164] according to Figure 7 In a non-limiting example, frequency curve 70 includes two phases:

[0165] The first phase δ1 shifts its frequency to a lower frequency, that is, the frequency F drops from a high frequency value to a low frequency value, corresponding to the source moving away from the virtual avatar (i.e., the distance between source 12 and virtual avatar 11 increases); and

[0166] The second phase δ2 briefly follows the first phase δ1, and its frequency shifts to a higher frequency, that is, the frequency F drops from a low frequency value to a high frequency value, corresponding to the source moving closer to / back to the virtual avatar (that is, the distance between the source 12 and the virtual avatar 11 decreases).

[0167] Each amplitude curve 71, 72, and 73 shows the change in the amplitude component of the control signal used to control the haptic actuators 53, 51, and 52, respectively.

[0168] As mentioned before Figure 6 The modulation amplitude parameters are, for example, based on the relative displacement of source 12 relative to virtual avatar 11, as a function of a first distance d and / or a second distance d' and / or the angle between source 12 and each haptic actuator 53, 51, and 52.

[0169] Each tactile actuator is advantageously controlled by control data signals received by each tactile actuator from the second device 22.

[0170] Therefore, according to, for example, reference Figure 6 and Figure 7 The time sequence determined by the time control information controls multiple tactile actuators of the group of tactile actuators associated with the user 10.

[0171] This control of the haptic actuator increases the immersion of the user 10 watching video content or toy video games with haptic effects, which are presented in space (on the user's body) and in real time, and are better matched with the movement of the source of the haptic effect relative to the position and orientation of the user 10's virtual avatar 11.

[0172] Figure 8 A schematic block diagram showing the steps of a method for controlling a set of tactile actuators according to at least one example embodiment.

[0173] In the first step 81, location information based on time and space, representing the location and direction of the virtual avatar relative to the source of the haptic effect in the virtual environment, is obtained, and the virtual avatar is associated with the user.

[0174] In the second step 82, first mapping information representing the mapping between the set of haptic actuators and the parts of a set of virtual avatars is obtained.

[0175] In the third step 83, the source is projected onto the virtualization in the virtual environment based on the location information.

[0176] In the fourth step 84, control data for each tactile actuator used to control the group of tactile actuators is determined based on the projection results and the first mapping information.

[0177] Figure 9 A schematic block diagram showing an example of system 9 in which various aspects and exemplary embodiments are implemented.

[0178] System 9 can be embedded as one or more devices including the various components described below. In various embodiments, system 9 can be configured to implement one or more aspects of the invention.

[0179] Examples of devices that can form all or part of System 9 include personal computers, notebook computers, smartphones, tablet computers, digital multimedia set-top boxes and their associated processing systems, head-mounted displays (HMDs, see-through glasses), tactile sensors or actuators, "caves" (systems including multiple displays), servers, tactile encoders, post-processors that process output from tactile decoders, pre-processors that provide input to tactile encoders, web servers, set-top boxes, wirelessly (e.g., Bluetooth®) connected wearable tactile devices, game controllers, mice, mouse pads, keyboards, wrist rests, chairs, tables, XR headsets, headphones, wristbands, head and / or lumbar support devices or chairs, any other means for processing tactile data or tactile signals, or other communication devices. The elements of System 9 may be present individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoding / decoding elements of System 9 may be distributed across multiple ICs and / or discrete components. In various embodiments, system 9 may be communicatively coupled to other similar systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports.

[0180] System 9 may include at least one processor 91 configured to execute instructions loaded thereon to implement various aspects, such as those described in this invention. Processor 91 may include embedded memory, input / output interfaces, and various other circuitry known in the art. System 9 may include at least one memory 92 (e.g., volatile and / or non-volatile memory). System 9 may include a storage device 94, which may include non-volatile and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drive, and / or optical disk drive. As a non-limiting example, storage device 94 may include internal storage, attached storage, and / or network-accessible storage.

[0181] System 9 may include an encoder / decoder module 93 configured, for example, to process data to provide encoded / decoded tactile signals or data, and the encoder / decoder module 93 may include its own processor and memory. The encoder / decoder module 93 may represent a module that may be included in the device to perform encoding and / or decoding functions. It is well known that a device may include one or both an encoding module and a decoding module. Furthermore, the encoder / decoder module 93 may be implemented as a separate element of system 9 or may be incorporated into processor 91 as a combination of hardware and software known to those skilled in the art.

[0182] Program code to be loaded onto processor 91 or encoder / decoder module 93 to execute the various aspects described herein may be stored in storage device 94 and subsequently loaded into memory 92 for execution by processor 91. According to various embodiments, one or more processors 91, memory 92, storage device 94, and / or encoder / decoder module 93 may store one or more of various items during the execution of the processing described herein. These stored items may include, but are not limited to, data representing audio content, data representing video content, haptic-related data, bitstreams, matrices, variables, and intermediate or final results from equations, formulas, operations, and operational logic.

[0183] In several embodiments, the memory within the processor 91 and / or encoder / decoder module 93 may be used to store instructions and provide working memory for processes that may be performed during data processing, encoding, or decoding.

[0184] However, in other embodiments, external memory (e.g., the processing device may be a processor) may be used for one or more of these functions. External memory may be memory 92 and / or storage device 94, such as dynamically volatile memory and / or non-volatile flash memory. In at least one embodiment, a fast external dynamically volatile memory, such as RAM, may be used as working memory for data processing.

[0185] Input to the components of System 9 can be provided through various input devices as shown in block 95. Such input devices include, but are not limited to, (i) an RF section capable of receiving, for example, RF signals transmitted over the air by a broadcaster, (ii) a composite input terminal, (iii) a USB input terminal, (iv) a telephone connector (also known as a telephone jack, audio jack, headphone jack, or plug) input terminal, and / or (v) an HDMI input terminal.

[0186] In various embodiments, the input device of block 95 may have associated corresponding input processing elements known in the art. For example, the RF section may be associated with the following required elements: (i) selecting a desired frequency (also known as selecting a signal or limiting a signal band to a frequency band), (ii) down-converting the selected signal, (iii) further band-limiting the signal to a narrower band to select, for example, a signal band referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired data packet stream. The RF section in various embodiments may include one or more elements to perform these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, down-converters, demodulators, error correctors, and demultiplexers. The RF section may include tuners that perform various of these functions, such as down-converting received signals to a lower frequency (e.g., intermediate frequency or near-fundamental frequency) or fundamental frequency.

[0187] In one on-setup embodiment, the RF unit and its associated input processing elements can receive RF signals transmitted through wired (e.g., cable) media. The RF unit can then perform frequency selection by filtering, down-converting, and re-filtering to the desired frequency band.

[0188] Various embodiments rearrange the order of the above (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions.

[0189] Adding components may include inserting components between existing components, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section may include an antenna.

[0190] Furthermore, the USB and / or HDMI terminals may include their respective interface processors for connecting System 9 to other electronic devices via the USB and / or HDMI connectors. It is understood that various aspects of input processing (e.g., analog-to-digital conversion, time-domain to frequency-domain conversion, downsampling, bandpass or low-pass filtering, Reed-Solomon error correction) may be implemented as needed, for example, within a separate input processing IC or processor 91. Similarly, aspects of USB or HDMI interface processing may be implemented as needed within a separate interface IC or processor 91. The processed stream can be provided to various processing elements, including processor 91, and an encoder / decoder module 93 operating in conjunction with memory and storage elements, to process the data stream as needed for presentation on an output device.

[0191] The various components of System 9 can be housed within an integrated enclosure. Within the integrated enclosure, the various components can be interconnected and transmit data therebetween using appropriate connectivity devices (e.g., internal buses known in the art, including I2C buses, wiring, and printed circuit boards).

[0192] System 9 may include a communication interface 96 capable of communicating with other devices via a communication channel 960. The communication interface 96 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 960. The communication interface 96 may include, but is not limited to, a modem or network card, and the communication channel 960 may be implemented, for example, in wired and / or wireless media.

[0193] In various embodiments, data can be streamed to system 9 using a Wi-Fi network such as IEEE 802.11. The Wi-Fi signal in these embodiments can be received via a communication channel 960 and a communication interface 96 suitable for Wi-Fi communication. The communication channel 960 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications.

[0194] Other embodiments may use a set-top box or computer that transmits data via an HDMI connection through block 95 to provide streaming data to system 9.

[0195] Other embodiments may use the RF connection of block 95 to provide streaming data to system 9.

[0196] Streaming data can be used as a signaling information method by System 9. Signaling information may include encoded data in a container, such as a binary stream or a haptic effect file.

[0197] It should be understood that signaling can be accomplished in a variety of ways. For example, in various embodiments, one or more syntax elements, flags, etc., may be used to send signaling information to the corresponding data processing device.

[0198] System 9 can provide output signals to various output devices, including display 970, speaker 980, and other peripheral devices 990 (such as tactile devices / actuators).

[0199] In various embodiments, control signals may be used between system 9 and display 970, speaker 980, and other peripheral devices 990 using communication protocols such as AV.Link (audio / video link), Consumer Electronics Control (CEC), audio protocols, Universal Serial Bus (USB), Haptics Industry Forum - Universal Haptic Protocol (HIF UHP), or other communication protocols that enable device-to-device control with or without user intervention.

[0200] The output devices can be communicatively coupled to the system 9 via dedicated connections through their respective interfaces 97, 98, and 99.

[0201] Alternatively, the output device can be connected to the system 9 via communication interface 96 using communication channel 960. The display 970, speaker 980, and / or tactile device (actuator) can be integrated with other components of the system 9 into a single unit of the electronic device.

[0202] In various embodiments, the (display) interface 97 may include a display driver, such as a timing controller (T Con) chip.

[0203] The display 970, speaker 980, and / or haptic device (actuator) may alternatively be decoupled from one or more other components. In various embodiments where the display 970, speaker 980, and / or haptic device (actuator) can be external components, output signals may be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0204] exist Figures 1 to 9 This document describes various methods, each including one or more steps or actions for implementing the method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified or combined.

[0205] Some examples are described in relation to block diagrams and / or operation flowcharts. Each block represents the program code of a circuit element, module, or section, which includes one or more executable instructions for implementing a specific logical function. It should also be noted that in other implementations, the functions marked in the blocks may not occur in the indicated order. For example, two blocks shown consecutively may actually be executed approximately simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved.

[0206] The implementations and aspects described herein may be implemented, for example, as methods or processes, devices, computer programs, data streams, bit streams, or signals. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), implementations of the discussed features may be implemented in other forms (e.g., devices or computer programs).

[0207] These methods can be implemented, for example, in a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices.

[0208] Furthermore, these methods can be implemented via instructions executed by a processor, and these instructions (and / or data values ​​generated by the implementation) can be stored on a computer-readable storage medium. The computer-readable storage medium may take the form of a computer-readable program product embodied in one or more computer-readable media and having computer-executable computer-readable program code embodied thereon. Considering the inherent ability to store information therein and to provide the inherent ability to retrieve information therefrom, the computer-readable storage medium used herein can be considered a non-transitory storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination thereof. It should be understood that while more specific examples of computer-readable storage media to which this embodiment may be applied are provided below, they are merely illustrative and not exhaustive lists, as will be readily understood by those skilled in the art: portable computer disks; hard disks; read-only memory (ROM); erasable programmable read-only memory (EPROM or flash memory); portable optical disc read-only memory (CD-ROM); optical storage devices; magnetic storage devices; or any suitable combination thereof.

[0209] These instructions can form applications that are tangibly embodied on processor-readable media.

[0210] Instructions may be found, for example, in hardware, firmware, software, or a combination thereof. For instance, instructions may be found in an operating system, a standalone application, or a combination of both. Therefore, a processor may be embodied as, for example, a means configured to perform processing and a means including processor-readable media (such as a storage device) having instructions for performing processing. Furthermore, in addition to instructions, or in place of instructions, the processor-readable media may store data values ​​generated by the implementation.

[0211] The device can be implemented, for example, with appropriate hardware, software and firmware. Examples of this device include personal computers, notebook computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, head-mounted displays (HMDs, see-through glasses), projectors, "caves" (systems including multiple displays), servers, video and / or haptic and / or chroma encoders, video and / or haptic and / or chroma decoders, haptic engines, chroma engines, post-processors that process the output from video decoders, pre-processors that provide input to video encoders, network servers, set-top boxes, wirelessly connected wearable haptic devices, such as Bluetooth® connected wearable haptic devices, game controllers, mice, mouse pads, keyboards, wrist rests, chairs, tables, XR headsets, headphones, wristbands, head and / or lumbar supports or chairs, any light source, and any other means for processing haptic data or signals representing one or more haptic feedbacks or effects, and / or for processing light / chroma data or signals representing one or more light patterns or effects, or other communication devices. It should be understood that the device may be portable.

[0212] The computer software may be implemented by the processor 91 or hardware, or a combination of hardware and software. As a non-limiting example, this embodiment may also be implemented by one or more integrated circuits. As a non-limiting example, the memory 92 may be of any type suitable for the technical environment and may be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, the processor 91 may be of any type suitable for the technical environment and may include one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0213] It will be apparent to those skilled in the art that signals of various formats can be generated to carry, for example, information that can be stored or transmitted. For example, the information may include instructions for performing a method, or data generated by one of the described embodiments. For example, the signal may be formatted to load a bit stream into the embodiments. For example, such a signal may be formatted as, for example, electromagnetic waves (e.g., using a portion of the radio frequency spectrum) or a baseband signal. For example, formatting may include encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. The signal may be transmitted through a variety of well-known wired or wireless connections. The signal may be stored on a processor-readable medium.

[0214] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. The singular forms “a,” “an,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising / including” and / or “including / containing” as used in this specification may specify stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Furthermore, when an element is referred to as “responding” or “connected” to another element, it may directly respond to or be connected to the other element, or there may be intermediate elements. Conversely, when an element is referred to as “directly responding” or “directly connected” to another element, there are no intermediate elements.

[0215] It should be understood that, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the symbols / terms “ / ,” “and / or,” and “at least one” may be intended to cover selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” such wording is intended to cover only the selection of the first listed option (A), or only the selection of the second listed option (B), or only the selection of the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). As will be apparent to those skilled in the art and related fields, this can be extended to as many items as are listed.

[0216] Various numerical values ​​may be used in this invention. Specific values ​​may be used for exemplary purposes and the aspects described are not limited to these specific values.

[0217] It should be understood that although the terms first, second, etc., may be used herein to describe various elements, these elements are not limited to these terms. These terms are used only to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the teachings of the invention. There is no implied order between the first element and the second element.

[0218] References to "an exemplary embodiment," "an exemplary embodiment," "an implementation," or "an implementation," as well as other variations, are frequently used to convey that a particular feature, structure, characteristic, etc. (described in relation to an embodiment / implementation) is included in at least one embodiment / implementation. Therefore, the phrases "in an exemplary embodiment," "in an exemplary embodiment," "in an implementation," or "in an implementation," and any other variations appearing in various places throughout the invention, do not necessarily all refer to the same embodiment.

[0219] Similarly, references to "according to an exemplary embodiment / example / implementation" or "in an exemplary embodiment / example / implementation" and other variations are often used to convey that a particular feature, structure, or characteristic (described in conjunction with the exemplary embodiment / example / implementation) may be included in at least one exemplary embodiment / example / implementation. Therefore, expressions such as "according to an exemplary embodiment / example / implementation" or "in an exemplary embodiment / example / implementation" appearing in different places in the specification do not necessarily refer to the same exemplary embodiment / example / implementation, and individual or alternative exemplary embodiments / examples / implementations are not necessarily mutually exclusive with other exemplary embodiments / examples / implementations.

[0220] The illustrative symbols appearing in the claims are for illustrative purposes only and do not limit the scope of the claims. Although not explicitly described, embodiments / examples and variations of the invention may be employed in any combination or sub-combination.

[0221] When a diagram is presented as a flowchart, it should be understood that it also provides a block diagram of the corresponding device. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flowchart of the corresponding method / process.

[0222] Although some diagrams include arrows on communication paths to indicate the main direction of communication, it should be understood that communication can occur in the opposite direction to the arrows depicted.

[0223] Furthermore, this invention can refer to "obtaining" various types of information. Obtaining information may include, for example, receiving information, determining information, estimating information, calculating information, or retrieving information from memory, or one or more of these.

[0224] Furthermore, this invention can refer to "accessing" various types of information. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, or estimating information, or more of these.

[0225] Additionally, this invention can refer to "receiving" various types of information. Like "access," receiving is a broad term. Receiving information can include, for example, accessing information or retrieving information (e.g., from memory) or more of them. Furthermore, "receiving" generally refers in various ways to operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, or estimating information.

[0226] Various embodiments have been described herein. However, it should be understood that various modifications can be made. For example, elements of different embodiments can be combined, supplemented, modified, or removed to produce other embodiments. Furthermore, those skilled in the art will understand that other structures and processes can replace the disclosed structures and processes, and the resulting embodiments will perform at least substantially the same function in at least substantially the same manner to achieve at least substantially the same results as the disclosed embodiments. Therefore, these and other embodiments are contemplated in this invention.

Claims

1. A method for controlling a set of tactile actuators, the method comprising the following steps: Obtain (81) a time- and space-based description of the position and orientation of a virtual avatar (11) relative to the source (12) of a haptic effect in a virtual environment, the virtual avatar (11) being associated with the user (10); Obtain (82) first mapping information representing the mapping between the group of haptic actuators and the parts of the group of virtual avatars (11), second mapping information representing the mapping between the group of haptic actuators (41 to 45) and the body parts of the user (10), and posture information representing the posture of the user (10). Based on the second mapping information and the posture information, spatial distribution information is determined representing the spatial distribution of the set of tactile actuators (41 to 45) on the body of the user (10) relative to the body of the user (10). The set of body reference planes (401, 402) includes at least one body reference plane, and each body reference plane divides the body into two complementary parts. The first mapping information is determined based on the spatial distribution information and a set of virtual avatar reference planes (411, 412). The set of virtual avatar reference planes (411, 412) includes at least one virtual avatar reference plane, and each virtual avatar reference plane corresponds to a different body reference plane in the set of body reference planes. Each virtual avatar reference plane divides the virtual avatar (11) into two complementary parts. The source (12) is projected (83) onto the virtual avatar (11) in the virtual environment based on the location information; as well as Determine (84) the control data for each tactile actuator of the group of tactile actuators, the control data being determined based on the projection result and the first mapping information.

2. The method as described in claim 1, wherein, The control data includes time control information, used to control each of the haptic actuators based on time.

3. The method of claim 1 or 2, further comprising: The system receives an input tactile signal representing the tactile effect, and the control data includes tactile data for controlling each of the tactile actuators, the tactile data being determined based on the input tactile signal and the result of the projection.

4. The method of claim 3, wherein, The tactile data is determined by processing the input tactile signal based on the result of the projection.

5. The method of claim 4, wherein, The processing includes amplitude modulation of the input tactile signal according to an amplitude parameter, the amplitude parameter being the result of the projection.

6. The method of claim 5, wherein, The amplitude parameter is based on the distance between the source (12) and each of the tactile actuators.

7. The method of claim 4, wherein, The process includes shifting the frequency of the input haptic signal to a lower frequency when the distance between the source and the virtual avatar increases, and shifting the frequency of the input haptic signal to a higher frequency when the distance between the source and the virtual avatar decreases.

8. The method of any one of claims 1-7, further comprising: Each tactile actuator is controlled according to the control data.

9. The method of claim 8, wherein, The multiple tactile actuators of the group of tactile actuators are controlled based on a time sequence according to time control information. The control data includes the time control information and is used to control each tactile actuator based on time.

10. The method of claim 8 or 9, wherein, Simultaneously control at least two tactile actuators in the group of tactile actuators according to the control data to present at least a portion of the tactile effect.

11. The method according to any one of claims 1-10, wherein, The projection (83) is based on a conical projection, where the vertex of the cone corresponds to the source (12) of the tactile effect.

12. An apparatus for controlling a set of tactile actuators, wherein the apparatus includes a memory (92) associated with at least a processor (91) configured to implement the method of any one of claims 1-11.

13. A computer program product comprising instructions of program code, which, when executed on a computer, perform the method as described in any one of claims 1-11.

Images