Haptic signal processing method and apparatus, and electronic device

Abstract

The present application relates to the technical field of communications and discloses a haptic signal processing method and apparatus, and an electronic device. The haptic signal processing method of embodiments of the present application comprises: when a first element triggers a haptic feedback, an electronic device determines a haptic signal and metadata of the haptic signal; the electronic device executes a first operation on the basis of the haptic signal and the metadata of the haptic signal, wherein the first element comprises at least one of a node and a primitive, and the metadata of the haptic signal comprises at least one of the following: actuator information associated with the haptic signal, avatar information associated with the haptic signal, a perceptual modality associated with the haptic signal, and an amount of attenuation in signal strength of the haptic signal.

Description

Tactile signal processing method, device and electronic device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 202311049378.6 filed in China on August 18, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application belongs to the field of communication technology, and specifically relates to a tactile signal processing method, device and electronic device. Background Art

[0004] Extended Reality (XR) technology is used to achieve the combination of reality and virtuality, including but not limited to at least one of Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). In order to improve the interaction between users and virtual environments, tactile scenes are introduced in XR scenarios, which can provide tactile feedback to user avatars, such as generating trigger signals and playing trigger signals. However, the related art only involves generating tactile signals when tactile feedback is triggered, and there is no relevant solution for how to determine the metadata of the tactile signals, which can easily lead to poor tactile feedback effects.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a tactile signal processing method, device, and electronic device, which can provide a method for determining metadata of a tactile signal, thereby improving the effect of tactile feedback.

[0007] In a first aspect, a tactile signal processing method is provided, the method comprising:

[0008] In a case where the first element triggers haptic feedback, the electronic device determines a haptic signal and metadata of the haptic signal;

[0009] The electronic device performs a first operation according to the haptic signal and metadata of the haptic signal;

[0010] Among them, the first element includes at least one of a node and a graph element, and the metadata of the tactile signal includes at least one of the following: actuator information associated with the tactile signal, avatar information associated with the tactile signal, perceptual modality associated with the tactile signal, and attenuation of the signal strength of the tactile signal.

[0011] In a second aspect, a tactile signal processing device is provided, the device comprising:

[0012] a first determining module, configured to determine a tactile signal and metadata of the tactile signal when the first element triggers tactile feedback;

[0013] an execution module, configured to execute a first operation according to the haptic signal and metadata of the haptic signal;

[0014] Among them, the first element includes at least one of a node and a graph element, and the metadata of the tactile signal includes at least one of the following: actuator information associated with the tactile signal, avatar information associated with the tactile signal, perceptual modality associated with the tactile signal, and attenuation of the signal strength of the tactile signal.

[0015] In a third aspect, an electronic device is provided, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0016] In a fourth aspect, an electronic device is provided, comprising a processor and a communication interface, wherein the processor is used to determine a tactile signal and metadata of the tactile signal when a first element triggers tactile feedback; and perform a first operation based on the tactile signal and the metadata of the tactile signal; wherein the first element includes at least one of a node and a graph element, and the metadata of the tactile signal includes at least one of the following: actuator information associated with the tactile signal, avatar information associated with the tactile signal, perceptual modality associated with the tactile signal, and attenuation of the signal strength of the tactile signal.

[0017] In a fifth aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0018] In a sixth aspect, a chip is provided, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to run a program or instruction to implement the steps of the method described in the first aspect.

[0019] In a seventh aspect, a computer program / program product is provided, wherein the computer program / program product is stored in a storage medium and is executed by at least one processor to implement the steps of the method described in the first aspect.

[0020] In an embodiment of the present application, when the first element triggers tactile feedback, the electronic device determines the tactile signal and the metadata of the tactile signal, and performs a first operation based on the tactile signal and the metadata of the tactile signal. Since the metadata of the above-mentioned tactile signal includes at least one of the actuator information associated with the tactile signal, the avatar information associated with the tactile signal, the perception mode associated with the tactile signal, and the attenuation amount of the signal strength of the tactile signal, controlling the playback of the tactile signal based on the metadata of the above-mentioned tactile signal is conducive to improving the effect of tactile feedback. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] FIG1 is a flow chart of a tactile signal processing method provided by an embodiment of the present application;

[0022] FIG2 is a structural diagram of a tactile signal processing device provided in an embodiment of the present application;

[0023] FIG3 is a structural diagram of an electronic device provided in an embodiment of the present application;

[0024] FIG4 is a structural diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the accompanying drawings in the embodiments of this application to clearly describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0026] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and do not limit the number of objects, for example, the first object can be one or more. In addition, "or" in this application represents at least one of the connected objects. For example, "A or B" covers three options, namely, Option 1: including A but not including B; Option 2: including B but not including A; Option 3: including both A and B. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0027] The term "indication" in this application can be either a direct indication (or explicit indication) or an indirect indication (or implicit indication). A direct indication can be understood as the sender explicitly informing the receiver of specific information, the operation to be performed, or the requested result, etc. in the instruction sent; an indirect indication can be understood as the receiver determining the corresponding information based on the instruction sent by the sender, or making a judgment and determining the operation to be performed or the requested result, etc. based on the judgment result.

[0028] For ease of understanding, some of the contents involved in the embodiments of this application are described below:

[0029] 1. Graphics Language Transmission Format (glTF) Scene Language

[0030] glTF is a basic language for describing three-dimensional (3D) scenes and models. glTF uses nodes as its foundation to build a programming framework. A glTF modeling scene can be represented by a root node, and objects in the modeling scene can be described by multiple nodes / sub-nodes.

[0031] 2. Avatar Description

[0032] In the MPEG scene modeling standard, a preliminary description standard for avatars was designed, and avatar-related extensions were made at the node level. The avatar structure can be established using paths and nodes.

[0033] 3. Interactivity Description

[0034] The Moving Picture Experts Group (MPEG) scene modeling standard has initially designed a standard for describing interactions. For haptic scenes, this standard primarily describes the interaction between an avatar and a haptic source. Both the haptic source and the avatar can be modeled using elements (such as nodes or primitives). Based on this, haptic signals are triggered and generated.

[0035] Interactivity is defined based on a behavior object. A behavior object contains its trigger conditions (i.e., trigger objects) and its action objects (i.e., action objects). Triggers include collision conditions (i.e., collision triggers) and proximity conditions (i.e., proximity triggers). Action objects include haptic feedback action objects (e.g., set haptic), etc.

[0036] For the collision condition, the node array / primitive arrary is defined in the collision condition. If any node / primitive in the node array / primitive arrary collides, the collision condition is satisfied. For the proximity condition, the distance range and the node array / primitive arrary are defined in the proximity condition. If the node / primitive is within the distance range, the proximity condition is satisfied. For the tactile feedback action, several parameters for tactile signal generation and playback are defined in the tactile feedback action. For example, the meaning (Semantic) of each of the above-mentioned trigger objects can be as shown in Table 1, the meaning of the above-mentioned action object can be as shown in Table 2, and the meaning of the tactile action node object (HapticActionNode object) can be as shown in Table 3.

[0037] Table 1 Meaning of trigger objects

[0038] Table 2 Meaning of action objects

[0039] Table 3 Meaning of tactile action node objects

[0040] The tactile signal processing method provided by the embodiment of the present application is described in detail below through some embodiments and application scenarios in conjunction with the accompanying drawings.

[0041] Please refer to Figure 1, which is a flowchart of a tactile signal processing method provided in an embodiment of the present application. The method can be executed by an electronic device, wherein the electronic device can be an XR device, a cloud device, a terminal device, a rendering device, a rendering engine, a server or a computer, etc.

[0042] As shown in FIG1 , the tactile signal processing method provided in the embodiment of the present application includes the following steps:

[0043] Step 101: When a first element triggers tactile feedback, the electronic device determines a tactile signal and metadata of the tactile signal;

[0044] Step 102: The electronic device performs a first operation according to the tactile signal and metadata of the tactile signal;

[0045] Among them, the first element includes at least one of a node and a graph element, and the metadata of the tactile signal includes at least one of the following: actuator information associated with the tactile signal, avatar information associated with the tactile signal, perceptual modality associated with the tactile signal, and attenuation of the signal strength of the tactile signal.

[0046] The first element may include at least one of an avatar or user element and a tactile source element, wherein the element may include at least one of a node and a primitive.

[0047] The first element triggers tactile feedback, for example, the first element satisfies a proximity condition (i.e., the first element triggers a proximity trigger), or the first element satisfies a collision condition (i.e., the first element triggers a collision trigger).

[0048] The actuator information may include, but is not limited to, actuator position information or actuator identification. The avatar information may include, but is not limited to, an avatar ID. The perception modality may include, but is not limited to, at least one of temperature and vibration. The attenuation value represents the attenuation of the tactile signal strength over distance.

[0049] For the above step 101 , illustratively, when the first element satisfies the distance condition or the collision condition, the electronic device may generate a tactile signal and metadata of the tactile signal.

[0050] For the above step 102, exemplarily, when the above electronic device is an XR device, the above electronic device can control the playback of the above tactile signal based on the metadata of the above tactile signal; when the above electronic device is a device other than the XR device (for example, a server or a computer, etc.), the above electronic device can send the above perception signal and the metadata of the perception signal to the XR device, and then the XR device controls the playback of the above tactile signal based on the metadata of the above perception signal.

[0051] In the tactile signal processing method provided in an embodiment of the present application, when a first element triggers tactile feedback, the electronic device determines the tactile signal and the metadata of the tactile signal, and performs a first operation based on the tactile signal and the metadata of the tactile signal. Since the metadata of the above-mentioned tactile signal includes at least one of the actuator information associated with the tactile signal, the avatar information associated with the tactile signal, the perception mode associated with the tactile signal, and the attenuation amount of the signal strength of the tactile signal, controlling the playback of the tactile signal based on the metadata of the above-mentioned tactile signal is conducive to improving the effect of tactile feedback.

[0052] Optionally, the actuator information associated with the tactile signal includes actuator information associated with the target element;

[0053] Among them, the target element includes the first element or the second element, the second element is the element associated with the first action object, the first action object is the action object corresponding to the trigger condition associated with the first element, and the second element includes at least one of a node and a graph element.

[0054] In this embodiment, the action object may include, but is not limited to, a haptic action node group. Each haptic action node in the haptic action node group may include, but is not limited to, a node identifier, an action location (actionLocation), a useCollider indication, a material haptic modality (materialHapticModality) group, a haptic action media (hapticActionMedias) list, etc. For example, the action object may refer to the aforementioned related description and will not be described in detail here.

[0055] The above-mentioned first action object is the action object corresponding to the trigger condition associated with the first element (i.e., the first trigger condition), wherein the above-mentioned first element may include at least one element in the element group defined in the first trigger condition (e.g., node array / primitive arrary), and the above-mentioned first action object and the above-mentioned first trigger condition are located in the same behavior object (i.e., the first behavior object), that is, the above-mentioned first behavior object includes the above-mentioned first action object and the first trigger condition. The above-mentioned second element may include at least one element in the element group defined in the above-mentioned first action object. Among them, the above-mentioned behavior object can refer to the relevant description mentioned above, which will not be repeated here. The above-mentioned trigger condition may include at least one of a collision condition (i.e., collision trigger) and a distance condition (i.e., proximity trigger), and the above-mentioned collision condition and distance condition can refer to the relevant description mentioned above, which will not be repeated here.

[0056] The actuator information associated with the above-mentioned tactile signal includes the actuator information associated with the target element, that is, the actuator information associated with the target element is used as the actuator information associated with the above-mentioned tactile signal. For example, the actuator position information associated with the target element is used as the actuator position information associated with the above-mentioned tactile signal, or the actuator identifier associated with the target element is used as the actuator identifier associated with the above-mentioned tactile signal. This makes it easy to accurately and quickly locate the actuator used to play the above-mentioned tactile signal.

[0057] Optionally, the actuator information associated with the target element includes the following: body part target, actuator target, actuator resolution;

[0058] or,

[0059] The executor information associated with the target element includes a device identifier, and the device identifier is used to identify the executor associated with the target element.

[0060] In one embodiment, at least one of a body part target (i.e., body_part_target), an actuator target (i.e., actuator target), and an actuator resolution (i.e., actuator resolution) is defined in an element (e.g., a node or primitive of a user or avatar) to accurately locate the position information of the actuator on the humanoid avatar.

[0061] For example, taking the above elements as nodes, the newly added attributes of the nodes may be as shown in Table 4, and the meaning of the haptic actuator objects may be as shown in Table 5.

[0062] Table 4 Node attributes

[0063] Table 5 Meaning of haptic actuator objects

[0064] In another embodiment, a device ID can be defined in an element (e.g., a node or primitive of a user or avatar) to identify the actuator ID associated with the element. For example, taking the above element as a node, the newly added attributes of the node can be as shown in Table 6.

[0065] Table 6 Node attributes

[0066] Optionally, the avatar information associated with the tactile signal includes avatar information associated with the target element;

[0067] or,

[0068] The avatar information associated with the tactile signal includes avatar information associated with the tactile scene corresponding to the target element;

[0069] or,

[0070] The avatar information associated with the tactile signal includes the avatar information associated with the upper layer element of the target element;

[0071] Among them, the target element includes the first element or the second element, the second element is the element associated with the first action object, the first action object is the action object corresponding to the trigger condition associated with the first element, and the second element includes at least one of a node and a graph element.

[0072] In one embodiment, the avatar information associated with the target element can be used as the avatar information associated with the tactile signal. For example, an avatar ID can be defined in an element (for example, a node or primitive of a user or avatar). If tactile feedback is triggered, the avatar ID in the element associated with the tactile feedback is used as the avatar ID associated with the generated tactile signal, that is, the avatar ID in the metadata of the tactile signal is set to the avatar ID in the element associated with the tactile feedback.

[0073] In another embodiment, the avatar information associated with the tactile scene corresponding to the target element can be used as the avatar information associated with the tactile signal. For example, if a tactile scene corresponds to an avatar or an activated avatar, the avatar ID of the avatar or the activated avatar can be used as the avatar ID associated with the tactile signal. That is, the avatar ID field in the metadata of the tactile signal is set to the avatar ID of the avatar or the activated avatar.

[0074] In another embodiment, the avatar information associated with the upper-layer element of the target element can be used as the avatar information associated with the tactile signal, wherein the upper-layer element of the target element, for example, the target element is a node, and the upper-layer element can be the parent node of the node. Exemplarily, for an upper-layer element corresponding to an avatar or an activated avatar, the avatar ID of the avatar or the activated avatar can be used as the avatar ID associated with the tactile signal, that is, the avatar ID field in the metadata of the tactile signal is set to the avatar ID of the avatar or the activated avatar.

[0075] Optionally, the target element is an element in an activated state;

[0076] or,

[0077] The avatar information associated with the tactile signal includes the avatar information of the element in the activated state associated with the tactile scene corresponding to the target element;

[0078] or,

[0079] The avatar information associated with the tactile signal includes the avatar information of an activated element associated with an upper-layer element of the target element.

[0080] Exemplarily, the activated element associated with the tactile scene or the activated element associated with the upper-layer element can be used to represent an avatar, that is, the avatar information associated with the tactile signal is the avatar information of the activated avatar.

[0081] For example, a state identifier can be defined in an element (e.g., avatar node / primitive), which is used to identify the activation state or deactivation state, and is used to characterize whether the element is currently the execution target of the tactile signal. For example, it can be as shown in Table 7. When a tactile signal is generated, the avatar ID field in the metadata of the tactile signal can be set to the avatar ID of the activated element.

[0082] Table 7

[0083] Optionally, in the case where the first element includes N elements in an activated state, the number of the tactile signals is N, where N is a positive integer, the N tactile signals correspond one-to-one to the N activated elements, and the avatar information associated with each of the tactile signals is the avatar information of the activated element corresponding to each of the tactile signals.

[0084] For example, when multiple active avatars coexist, a haptic signal may be generated for each avatar separately, and the avatar ID field in the metadata of each haptic signal is set to the avatar ID of the corresponding avatar.

[0085] Optionally, the element being in an activated state includes at least one of the following:

[0086] The element is in a first activation state, where the first activation state is used to indicate that the element can be applied to a tactile scenario;

[0087] The element is in a second activation state, where the second activation state is used to indicate that the element can be used to trigger tactile feedback;

[0088] The element is in a third activation state, where the third activation state is used to indicate that the element is applicable to perform a tactile feedback action.

[0089] Exemplarily, at least one of a first state identifier, a second state identifier, and a third state identifier can be defined in an element (e.g., an avatar node / primitive), wherein the first state identifier, the second state identifier, and the third state identifier can each include an activation state or a deactivation state, the first state identifier can be used to indicate whether the element can be applied to a tactile scene, the second state identifier is used to indicate whether the element can be applied to trigger tactile feedback, and the third state identifier can be used to indicate whether the element can be applied to perform a tactile feedback action. For example, see Table 8, wherein the status (Status) in Table 8 can represent the first state identifier, the status (Status_Trigger) in Table 8 can represent the second state identifier, and the Status_Action in Table 8 can represent the third state identifier.

[0090] Table 8

[0091] Optionally, the state of the element is set via a second action object, which is used to activate or deactivate the element.

[0092] In this embodiment, the second action object may include functions and parameters related to activating or deactivating the element, so that the state of the element can be set through the second action object.

[0093] Optionally, the perceptual modality associated with the tactile signal includes a perceptual modality associated with the target element;

[0094] or,

[0095] The perceptual modality associated with the tactile signal includes the perceptual modality associated with the tactile scene corresponding to the target element;

[0096] or,

[0097] The perceptual modality associated with the tactile signal includes the perceptual modality associated with the upper-layer element of the target element;

[0098] Among them, the target element includes the first element or the second element, the second element is the element associated with the first action object, the first action object is the action object corresponding to the trigger condition associated with the first element, and the second element includes at least one of a node and a graph element.

[0099] In one embodiment, the perception modality associated with the target element can be used as the perception modality associated with the tactile signal. For example, a parameter perception modality is defined in the element. If tactile feedback is triggered, the corresponding perception modality in the element involved in the Trigger / Action associated with the tactile feedback is used as the perception modality associated with the generated tactile signal. In other words, the perception modality field in the metadata of the tactile signal is set to the corresponding perception modality in the element involved in the Trigger / Action associated with the tactile feedback.

[0100] In another embodiment, the perception modality associated with the tactile scene corresponding to the target element can be used as the perception modality associated with the tactile signal. For example, if a tactile scene corresponds to a perception modality, the perception modality corresponding to the tactile scene can be used as the perception modality associated with the tactile signal. That is, the perception modality field in the metadata of the tactile signal is set to the perception modality corresponding to the tactile scene.

[0101] In another embodiment, the perception modality associated with the parent element of the target element can be used as the perception modality associated with the tactile signal. For example, if a parent element corresponds to a perception modality, the perception modality corresponding to the parent element can be used as the perception modality associated with the tactile signal, that is, the perception modality field in the metadata of the tactile signal is set to the perception modality corresponding to the parent element.

[0102] Optionally, the perceptual modality associated with the tactile signal includes an activated perceptual modality associated with the target element;

[0103] or,

[0104] The perceptual modality associated with the tactile signal includes the perceptual modality in an activated state associated with the tactile scene corresponding to the target element;

[0105] or,

[0106] The perceptual modality associated with the tactile signal includes an activated perceptual modality associated with an upper-layer element of the target element.

[0107] For example, when multiple perception modalities coexist, a perception modality status flag can be introduced to indicate whether a perception modality is active. When a haptic signal is generated, the perception modality field in the haptic signal metadata can be set to the active perception modality; alternatively, a modality list can be used to indicate one or more active perception modalities.

[0108] Optionally, when there are M perceptual modes in an activated state among the perceptual modes associated with the target element, the number of the tactile signals is M, where M is a positive integer, the M tactile signals correspond one-to-one to the M perceptual modes in an activated state, and the perceptual mode associated with each of the tactile signals is the perceptual mode in an activated state corresponding to each of the tactile signals.

[0109] Exemplarily, when multiple perception modalities or multiple activated perception modalities coexist, a tactile signal can be generated for each perception modality or each activated perception modality, and the perception modality field in the metadata of each tactile signal is set to the corresponding perception modality.

[0110] Optionally, the state of the perception modality is set via a third action object, and the third action object is used to activate or deactivate the perception modality.

[0111] In this embodiment, the third action object may include functions and parameters related to activating or deactivating the perception modality, so that the state of the perception modality can be set through the third action object.

[0112] Exemplarily, as shown in Table 9, the action object for activating or deactivating the perception modality may include a perception modality or a perception modality list, which is used to indicate the perception modality that can be applied to the virtual world, that is, the perception modality in the activated state.

[0113] Table 9

[0114] Optionally, when the first element triggers tactile feedback, before the electronic device determines the tactile signal and metadata of the tactile signal, the method further includes:

[0115] The electronic device determines a state of a first behavior object, wherein the first behavior object includes a first trigger condition and a first action object, and the first trigger condition is a trigger condition associated with the first element;

[0116] The electronic device determines whether to apply or not to apply the first behavior object according to a state of the first behavior object.

[0117] In this embodiment, the state of the above-mentioned first behavior object may include an activation state or a deactivation state, wherein, when the above-mentioned first behavior object is in the activation state, it indicates that the first behavior object is valid. In this case, the electronic device can apply or execute the first behavior object, that is, the electronic device can apply the trigger condition included in the first behavior object (i.e., the first trigger condition) to detect whether tactile feedback is triggered, and can apply the action object included in the first behavior object (i.e., the first action object) to perform a tactile feedback action; when the above-mentioned first behavior object is in the deactivation state, it indicates that the first behavior object is invalid. In this case, the electronic device may not apply or execute the first behavior object, that is, the electronic device does not apply the trigger condition included in the first behavior object (i.e., the first trigger condition) to detect whether tactile feedback is triggered, or does not apply the action object included in the first behavior object (i.e., the first action object) to perform a tactile feedback action.

[0118] Exemplarily, the interaction model may include invalid behavior objects. For example, because an avatar or perception modality is deactivated, the behavior object is invalid. To avoid executing invalid commands, the electronic device can determine the status of each behavior object and only apply the behavior object in the activated state.

[0119] Optionally, the state of the first behavior object is determined according to a state identifier associated with the first behavior object, wherein the state identifier is used to identify an activation state or a deactivation state;

[0120] or,

[0121] The state of the first behavior object is determined according to at least one of a state of the first trigger condition and a state of the first action object.

[0122] In one embodiment, the state of the first behavior object can be determined based on the state identifier associated with the first behavior object. For example, if the state identifier associated with the first behavior object identifies a deactivated state, then the first behavior object is determined to be in a deactivated state; if the state identifier associated with the first behavior object identifies an activated state, then the first behavior object is determined to be in an activated state.

[0123] For example, a status flag can be set in a behavior object. For a deactivated behavior object, the behavior object does not need to be executed. For example, each behavior object includes a trigger condition, an action object, and a status flag, i.e., behavior{Trigger,Action,Status}, where Status can be used to identify the activation state or the deactivation state.

[0124] In another embodiment, the state of the first behavior object can be determined based on at least one of the state of the first trigger condition and the state of the first action object. For example, when the first trigger condition and the first action object are both in the activated state, the first behavior object can be determined to be in the activated state; otherwise, the first behavior object can be determined to be in the deactivated state.

[0125] Optionally, the state of the first trigger condition is determined according to a state identifier associated with the first trigger condition, or determined according to a state identifier associated with an element associated with the first trigger condition;

[0126] or,

[0127] The state of the first action object is determined according to the state identifier of the first action object, or according to the state identifier associated with the element associated with the first action object;

[0128] The state identifier is used to identify an activation state or a deactivation state.

[0129] The following situations are explained:

[0130] (1) Status of the first trigger condition

[0131] In one embodiment, the state of the first trigger condition can be determined based on the state identifier associated with the first trigger condition. For example, if the state identifier associated with the first trigger condition indicates a deactivated state, the first trigger condition is determined to be in a deactivated state. If the state identifier associated with the first trigger condition indicates an activated state, the first trigger condition is determined to be in an activated state. For example, a state identifier can be set in the trigger condition (i.e., trigger) to identify the activated state or deactivated state of the trigger condition. For a deactivated trigger condition, the behavior object corresponding to the trigger condition does not need to be executed.

[0132] In another embodiment, the state of the first trigger condition can be determined based on a state identifier associated with an element associated with the first trigger condition. For example, a state identifier can be set in the element to identify the activation state or deactivation state of the element, and then whether the trigger condition is activated can be determined based on the activation state or deactivation state of the element in the trigger condition. For example, if at least one element in the element list corresponding to the trigger condition is in an activated state, then the trigger condition is determined to be in an activated state.

[0133] (2) The state of the first action object

[0134] In one embodiment, the state of the first action object can be determined based on the state identifier associated with the first action object. For example, if the state identifier associated with the first action object indicates a deactivated state, the first action object is determined to be in a deactivated state. If the state identifier associated with the first action object indicates an activated state, the first action object is determined to be in an activated state. Exemplarily, a state identifier can be set in the action object to identify the activated state or deactivated state of the action object. For a deactivated action object, the behavior object corresponding to the action object does not need to be executed.

[0135] In another embodiment, the state of the first action object can be determined based on a state identifier associated with an element associated with the first action object. For example, a state identifier can be set in the element to identify the activation state or deactivation state of the element, and then whether the action object is activated can be determined based on the activation state or deactivation state of the element in the action object. For example, if at least one element in the element list corresponding to the action object is in the activation state, then the action object is determined to be in the activation state.

[0136] In some optional embodiments, there are multiple activation / deactivation forms for an element, for example, (a) whether the element is always activated / deactivated, (b) whether the element is in an activated / deactivated state when it serves as a trigger condition (i.e., trigger) element, and (c) whether the element is in an activated / deactivated state when it serves as an action element.

[0137] It should be noted that in scenarios where a humanoid avatar and a car-type avatar exist in the virtual world, when a person enters the car, the car-type avatar is responsible for triggering and generating tactile signals. The humanoid avatar will not trigger and generate tactile signals at this time, but the humanoid avatar is still in an effective state. For example, the position of the humanoid avatar can be used to determine when the person leaves the car. Therefore, it is necessary to introduce the above (b) and (c).

[0138] Exemplarily, as shown in Table 10, the status activation action (i.e., the action object used for status activation or deactivation of an element) includes setting the trigger status (Set Status_Trigger) and setting the action status (Set Status_Action), wherein Set Status_Trigger is used to indicate whether the element as the trigger condition (i.e., Trigger) is activated, and Set Status_Action is used to indicate whether the element as the action object is activated.

[0139] Table 10

[0140] Optionally, the state of the first action object is set by a fourth action object, and the fourth action object is used to activate or deactivate the behavior object;

[0141] or,

[0142] The state of the first trigger condition is set by a fifth action object, and the fifth action object is used to activate or deactivate the trigger condition;

[0143] or,

[0144] The state of the first action object is set according to a sixth action object, and the sixth action object is used to activate or deactivate the action object;

[0145] or,

[0146] The state of the element is set according to a seventh action object, and the seventh action object is used to activate or deactivate the element.

[0147] In this embodiment, the fourth action object may include functions and parameters related to activating or deactivating the behavior object, so that the state of the behavior object can be set through the fourth action object.

[0148] The fifth action object may include functions, parameters, etc. related to activation or deactivation of the trigger condition, so that the state of the first trigger condition may be set through the fifth action object.

[0149] The sixth action object may include functions, parameters, etc. related to activating or deactivating the action object, so that the state of the first action object can be set through the sixth action object.

[0150] The seventh action object may include functions and parameters related to activating or deactivating an element, so that the state of the element may be set through the seventh action object.

[0151] Optionally, the state of the first behavior object is determined according to the state of the perception modality corresponding to the first behavior object;

[0152] or,

[0153] The state of the first behavior object is determined based on the perception modality corresponding to the first trigger condition and the perception modality corresponding to the first action object, or the state of the first behavior object is determined based on the perception modality in the activated state corresponding to the first trigger condition and the perception modality in the activated state corresponding to the first action object.

[0154] In one embodiment, the state of the first behavior object can be determined based on the state of the perception modality corresponding to the first behavior object. For example, if at least one of the perception modalities corresponding to the first behavior object is in an activated state, then the first behavior object is in an activated state; otherwise, the first behavior object is in a deactivated state; or, if all the perception modalities corresponding to the first behavior object are in an activated state, then the first behavior object is in an activated state; otherwise, the first behavior object is in a deactivated state.

[0155] In another embodiment, the state of the first behavior object is determined based on the perception modality corresponding to the first trigger condition and the perception modality corresponding to the first action object. For example, if the perception modality corresponding to the first trigger condition and the perception modality corresponding to the first action object are the same, then the first behavior object is determined to be in an activated state, otherwise, the first behavior object is determined to be in a deactivated state; or, the state of the first behavior object is determined based on the perception modality in the activated state corresponding to the first trigger condition and the perception modality in the activated state corresponding to the first action object. For example, if the perception modality in the activated state corresponding to the first trigger condition and the perception modality in the activated state corresponding to the first action object are the same, then the first behavior object is determined to be in an activated state, otherwise, the first behavior object is determined to be in a deactivated state.

[0156] Optionally, the perception modality corresponding to the first behavior object is the perception modality corresponding to the tactile scene corresponding to the first behavior object.

[0157] For example, a parameter perception modality may be introduced at the tactile scene level. If a certain perception modality is deactivated in the tactile scene, the element or behavior object corresponding to the perception modality will be deactivated.

[0158] Optionally, the state of the perception modality corresponding to the first behavior object is determined according to a state identifier associated with the perception modality corresponding to the first behavior object.

[0159] In this embodiment, a state flag can be set at the perception modality level to identify the activation or deactivation state of the perception modality. For example, at least one of the perception modality and the state flag of the perception modality can be introduced at the behavior object level. If the behavior object only corresponds to a deactivated perception modality, the behavior object does not need to be executed. If the behavior object corresponds to multiple perception modalities, each perception modality can be assigned a state flag.

[0160] Optionally, when all perception modalities corresponding to the first behavior object are in a deactivated state, the first behavior object is in a deactivated state.

[0161] Optionally, when the perception modality corresponding to the first trigger condition and the perception modality corresponding to the first action object are the same, the first behavior object is in an activated state; otherwise, the first behavior object is in a deactivated state;

[0162] or,

[0163] When the perception mode in the activated state corresponding to the first trigger condition is the same as the perception mode in the activated state corresponding to the first action object, the first behavior object is in the activated state; otherwise, the first behavior object is in the deactivated state.

[0164] Optionally, the state of the perception modality corresponding to the first trigger condition is determined according to a state identifier of the perception modality corresponding to the first trigger condition;

[0165] or,

[0166] The state of the perception modality corresponding to the first trigger condition is determined according to the state identifier of the perception modality corresponding to the element corresponding to the first trigger condition.

[0167] In one embodiment, at least one of the perception modality and the state flag of the perception modality is set at the trigger condition (ie, trigger) level.

[0168] In another embodiment, at least one of the perception modality and the state flag of the perception modality is set at the element level, and the perception modality activated by the trigger condition is determined based on the perception modality of the element in the trigger condition and at least one of the state flags of the perception modality.

[0169] Optionally, the state of the perception modality corresponding to the first action object is determined according to a state identifier of the perception modality corresponding to the first action object;

[0170] or,

[0171] The state of the perception modality corresponding to the first action object is determined according to the state identifier of the perception modality corresponding to the element corresponding to the first action object.

[0172] In one embodiment, at least one of the perception modality and the state flag of the perception modality is set at the action object level.

[0173] In another embodiment, at least one of the perception modality and the state flag of the perception modality is set at the element level, and the perception modality activated by the action object is determined according to the perception modality of the element in the action object and the state flag of the perception modality.

[0174] In some optional embodiments, at least one perception modality in an activated state may be directly indicated by a perception modality or a perception modality list.

[0175] Optionally, the third element is associated with at least one of the following: a parameter related to attenuation of the tactile signal, a parameter related to propagation distance of the tactile signal;

[0176] The third element includes at least one element of a tactile source or at least one element of a tactile scene.

[0177] Exemplarily, the above-mentioned attenuation-related parameters may include an attenuation identifier, relevant parameters for attenuation calculation (attenuation formula), etc., for example, as shown in Table 11.

[0178] Table 11 Haptic source node attenuation properties

[0179] The above-mentioned propagation distance related parameters may include a maximum propagation distance of the tactile signal or may be used to indicate a propagation range of the tactile signal.

[0180] It is understood that the perceived intensity of a tactile signal is related to the distance from the tactile signal source. For example, during an explosion, the farther away from the explosion source, the weaker the perceived vibration (perception modality = vibration (vibrotactile)) and the smaller the perceived heat (perception modality = temperature); at even greater distances, the user may not even feel the vibration or heat caused by the explosion. This embodiment improves the tactile feedback effect by associating at least one of the tactile signal attenuation-related parameter and the tactile signal propagation distance-related parameter in the third element.

[0181] Optionally, the attenuation-related parameter includes an attenuation indicator, and the attenuation indicator is used to indicate that the signal strength of the tactile signal attenuates with distance.

[0182] Optionally, there is a correlation between the attenuation-related parameter and the perception modality.

[0183] For example, attenuation-related parameters are set for each perception modality (ie, per perception modality).

[0184] Optionally, the attenuation amount of the signal strength of the tactile signal is determined based on an attenuation model or an attenuation formula, wherein the attenuation model or the attenuation formula is agreed upon by a protocol.

[0185] Exemplarily, the attenuation of the signal strength of the tactile signal may be calculated according to the attenuation model or attenuation formula and the above-mentioned attenuation-related parameters.

[0186] In some optional embodiments, the attenuation of the signal strength of the tactile signal may be reflected by a channel gain in the metadata of the tactile signal, that is, the channel gain field in the metadata of the tactile signal is set to the attenuation.

[0187] Optionally, the propagation distance related parameter includes an upper limit value of the propagation distance of the tactile signal, and the upper limit value of the propagation distance is used to indicate a maximum propagation distance of the tactile signal.

[0188] Optionally, the trigger condition associated with the first element is a distance condition, the second element includes a distance usage identifier, the second element includes at least one element in a first action object, and the first action object is an action object corresponding to the trigger condition associated with the first element;

[0189] The distance usage identifier is used to indicate whether the electronic device triggers tactile feedback based on an element located within a distance range defined by the distance condition.

[0190] For example, the element associated with the action object can carry a distance usage flag (useProximity). If the value of useProximity is true, the actuator corresponding to the element within the distance specified by the distance condition corresponding to the action object can play a tactile signal, for example, as shown in Table 12.

[0191] Table 12 Haptic action node attributes

[0192] In some optional embodiments, the actuator information (eg, body part mask, body part target, actuator target) of the metadata of the haptic signal may be set to the actuator information corresponding to the element that satisfies the distance condition.

[0193] Optionally, the trigger condition associated with the first element is a collision condition, the element list corresponding to the collision condition includes at least two elements, each of the at least two elements is associated with a type identifier or a first identifier, and the first identifier associated with each element is the identifier of the upper-layer element of each element;

[0194] Wherein, it is determined that the collision condition is satisfied when elements with different associated type identifiers or first identifiers among the at least two elements collide.

[0195] The type identifier associated with the above-mentioned elements can be used to identify the type of the element. For example, the type of the above-mentioned elements may include a tactile source element and a user element (i.e., an avatar element). The type identifier associated with the above-mentioned elements may include a tactile source element identifier and a user element identifier. In this way, when different types of elements collide, it is determined that the collision condition is satisfied. For example, when a tactile source element and a user element collide, it is determined that the collision condition is satisfied, while when elements of the same type collide, it can be determined that the collision condition is not satisfied.

[0196] The first identifier associated with the element is the identifier of the element's parent element. For example, for a node, the identifier of its associated parent element can be the identifier of its parent node. In this way, when elements with different parent elements collide, it is determined that the collision condition is satisfied. For example, when a node whose parent node is parent node1 collides with a node whose parent node is parent node2, it is determined that the collision condition is satisfied. However, when elements under the same parent element collide, it can be determined that the collision condition is not satisfied. For example, as shown in Table 13 and Table 14.

[0197] Table 13 Node parent node attributes

[0198] Table 14 Meaning of triggers

[0199] Optionally, the trigger condition associated with the first element is a collision condition, and the collision condition corresponds to at least two element groups, each of which includes at least one element, wherein the collision condition is determined to be satisfied when elements between different element groups in the at least two element groups collide.

[0200] Exemplarily, the at least two element groups may include a tactile source element group and a user element group (i.e., an avatar element group), wherein the tactile source element group includes at least one tactile source element, and the user element group includes at least one user element. In this way, when elements between different element groups collide, it is determined that the collision condition is satisfied. For example, when an element of the tactile source element group collides with an element of the user element group, it is determined that the collision condition is satisfied; otherwise, it can be determined that the collision condition is not satisfied.

[0201] Optionally, the trigger condition associated with the first element is a collision condition, the collision condition corresponds to at least two elements, and the collision condition is determined to be satisfied when a collision occurs between lower-layer elements of different elements among the at least two elements.

[0202] Exemplarily, when the at least two elements are parent nodes, the lower elements of the at least two elements may be nodes or child nodes; when the at least two elements are nodes, the lower elements of the at least two elements may be child nodes.

[0203] It should be noted that in the interaction model, collisions are defined at the node level. A node list is carried in a collision trigger. If the nodes in the node list collide, the collision condition is deemed to be met. However, for high-precision collision judgment, the user node or avatar node will be virtualized into countless tiny basic geometric models, and the haptic source node will be virtualized into countless tiny basic geometric models. From the perspective of program implementation, the collision between tiny basic set models requires the repeated definition of countless trigger objects. Each trigger object contains the basic geometric model of the user node and the basic geometric model of the haptic source node. This implementation method is relatively complex. Based on this, this embodiment sets the element list corresponding to the collision condition to include at least two elements, each of the at least two elements is associated with a type identifier or a first identifier, and determines that the collision condition is satisfied when elements with different type identifiers or first identifiers associated with the at least two elements collide; or, sets the collision condition to correspond to at least two element groups, and determines that the collision condition is satisfied when elements between different element groups in the at least two element groups collide; or sets the collision condition to correspond to at least two elements, and determines that the collision condition is satisfied when a collision occurs between lower-level elements of different elements in the at least two elements. This can simplify the implementation complexity of collision condition judgment.

[0204] Optionally, the electronic device performs a first operation according to the tactile signal and metadata of the tactile signal, including:

[0205] The electronic device controls the playing of the haptic signal according to the metadata of the haptic signal.

[0206] Exemplarily, the electronic device may be an XR device. The electronic device controls the playback of the tactile signal based on the metadata of the tactile signal. For example, when the metadata of the tactile signal includes actuator information associated with the tactile signal, the electronic device may determine the actuator that plays the tactile signal based on the actuator information associated with the tactile signal, and then execute the tactile signal based on the determined actuator; or, when the metadata of the tactile signal includes the attenuation of the signal strength of the tactile signal, the signal strength of the played tactile signal may be controlled based on the attenuation of the signal strength of the tactile signal, which is conducive to improving the effect of tactile feedback.

[0207] It should be noted that the inventive concept provided in the embodiments of the present application can be applied to electronic devices to determine tactile signals, and can also be applied to electronic devices to determine other signals such as visual signals, audio signals, etc.

[0208] It should be noted that the tactile signal processing method provided in the embodiments of the present application can be executed by a tactile signal processing device, or a control module in the tactile signal processing device for executing the tactile signal processing method. In the embodiments of the present application, the tactile signal processing device provided in the embodiments of the present application is described by taking the tactile signal processing device executing the tactile signal processing method as an example.

[0209] Please refer to FIG. 2 , which is a structural diagram of a tactile signal processing device provided in an embodiment of the present application. As shown in FIG. 2 , the tactile signal processing device 200 includes:

[0210] A first determining module 201 is configured to determine a tactile signal and metadata of the tactile signal when the first element triggers tactile feedback;

[0211] An execution module 202 is configured to execute a first operation according to the haptic signal and metadata of the haptic signal;

[0212] Among them, the first element includes at least one of a node and a graph element, and the metadata of the tactile signal includes at least one of the following: actuator information associated with the tactile signal, avatar information associated with the tactile signal, perceptual modality associated with the tactile signal, and attenuation of the signal strength of the tactile signal.

[0213] Optionally, the actuator information associated with the tactile signal includes actuator information associated with the target element;

[0214] Among them, the target element includes the first element or the second element, the second element is the element associated with the first action object, the first action object is the action object corresponding to the trigger condition associated with the first element, and the second element includes at least one of a node and a graph element.

[0215] Optionally, the actuator information associated with the target element includes the following: body part target, actuator target, actuator resolution;

[0216] or,

[0217] The executor information associated with the target element includes a device identifier, and the device identifier is used to identify the executor associated with the target element.

[0218] Optionally, the avatar information associated with the tactile signal includes avatar information associated with the target element;

[0219] or,

[0220] The avatar information associated with the tactile signal includes avatar information associated with the tactile scene corresponding to the target element;

[0221] or,

[0222] The avatar information associated with the tactile signal includes the avatar information associated with the upper layer element of the target element;

[0223] Among them, the target element includes the first element or the second element, the second element is the element associated with the first action object, the first action object is the action object corresponding to the trigger condition associated with the first element, and the second element includes at least one of a node and a graph element.

[0224] Optionally, the target element is an element in an activated state;

[0225] or,

[0226] The avatar information associated with the tactile signal includes the avatar information of the element in the activated state associated with the tactile scene corresponding to the target element;

[0227] or,

[0228] The avatar information associated with the tactile signal includes the avatar information of an activated element associated with an upper-layer element of the target element.

[0229] Optionally, in the case where the first element includes N elements in an activated state, the number of the tactile signals is N, where N is a positive integer, the N tactile signals correspond one-to-one to the N activated elements, and the avatar information associated with each of the tactile signals is the avatar information of the activated element corresponding to each of the tactile signals.

[0230] Optionally, the element being in an activated state includes at least one of the following:

[0231] The element is in a first activation state, where the first activation state is used to indicate that the element can be applied to a tactile scenario;

[0232] The element is in a second activation state, where the second activation state is used to indicate that the element can be used to trigger tactile feedback;

[0233] The element is in a third activation state, where the third activation state is used to indicate that the element is applicable to perform a tactile feedback action.

[0234] Optionally, the state of the element is set via a second action object, which is used to activate or deactivate the element.

[0235] Optionally, the perceptual modality associated with the tactile signal includes a perceptual modality associated with the target element;

[0236] or,

[0237] The perceptual modality associated with the tactile signal includes the perceptual modality associated with the tactile scene corresponding to the target element;

[0238] or,

[0239] The perceptual modality associated with the tactile signal includes the perceptual modality associated with the upper-layer element of the target element;

[0240] Among them, the target element includes the first element or the second element, the second element is the element associated with the first action object, the first action object is the action object corresponding to the trigger condition associated with the first element, and the second element includes at least one of a node and a graph element.

[0241] Optionally, the perceptual modality associated with the tactile signal includes an activated perceptual modality associated with the target element;

[0242] or,

[0243] The perceptual modality associated with the tactile signal includes the perceptual modality in an activated state associated with the tactile scene corresponding to the target element;

[0244] or,

[0245] The perceptual modality associated with the tactile signal includes an activated perceptual modality associated with an upper-layer element of the target element.

[0246] Optionally, when there are M perceptual modes in an activated state among the perceptual modes associated with the target element, the number of the tactile signals is M, where M is a positive integer, the M tactile signals correspond one-to-one to the M perceptual modes in an activated state, and the perceptual mode associated with each of the tactile signals is the perceptual mode in an activated state corresponding to each of the tactile signals.

[0247] Optionally, the state of the perception modality is set via a third action object, and the third action object is used to activate or deactivate the perception modality.

[0248] Optionally, the device further comprises:

[0249] a second determining module configured to, when the first element triggers tactile feedback, determine a state of a first behavior object before determining the tactile signal and metadata of the tactile signal, wherein the first behavior object includes a first trigger condition and a first action object, and the first trigger condition is a trigger condition associated with the first element;

[0250] A third determining module is configured to determine whether to apply or not to apply the first behavior object according to a state of the first behavior object.

[0251] Optionally, the state of the first behavior object is determined according to a state identifier associated with the first behavior object, wherein the state identifier is used to identify an activation state or a deactivation state;

[0252] or,

[0253] The state of the first behavior object is determined according to at least one of a state of the first trigger condition and a state of the first action object.

[0254] Optionally, the state of the first trigger condition is determined according to a state identifier associated with the first trigger condition, or determined according to a state identifier associated with an element associated with the first trigger condition;

[0255] or,

[0256] The state of the first action object is determined according to the state identifier of the first action object, or according to the state identifier associated with the element associated with the first action object;

[0257] The state identifier is used to identify an activation state or a deactivation state.

[0258] Optionally, the state of the first action object is set by a fourth action object, and the fourth action object is used to activate or deactivate the behavior object;

[0259] or,

[0260] The state of the first trigger condition is set by a fifth action object, and the fifth action object is used to activate or deactivate the trigger condition;

[0261] or,

[0262] The state of the first action object is set according to a sixth action object, and the sixth action object is used to activate or deactivate the action object;

[0263] or,

[0264] The state of the element is set according to a seventh action object, and the seventh action object is used to activate or deactivate the element.

[0265] Optionally, the state of the first behavior object is determined according to the state of the perception modality corresponding to the first behavior object;

[0266] or,

[0267] The state of the first behavior object is determined based on the perception modality corresponding to the first trigger condition and the perception modality corresponding to the first action object, or the state of the first behavior object is determined based on the perception modality in the activated state corresponding to the first trigger condition and the perception modality in the activated state corresponding to the first action object.

[0268] Optionally, the perception modality corresponding to the first behavior object is the perception modality corresponding to the tactile scene corresponding to the first behavior object.

[0269] Optionally, the state of the perception modality corresponding to the first behavior object is determined according to a state identifier associated with the perception modality corresponding to the first behavior object.

[0270] Optionally, when all perception modalities corresponding to the first behavior object are in a deactivated state, the first behavior object is in a deactivated state.

[0271] Optionally, when the perception modality corresponding to the first trigger condition and the perception modality corresponding to the first action object are the same, the first behavior object is in an activated state; otherwise, the first behavior object is in a deactivated state;

[0272] or,

[0273] When the perception mode in the activated state corresponding to the first trigger condition is the same as the perception mode in the activated state corresponding to the first action object, the first behavior object is in the activated state; otherwise, the first behavior object is in the deactivated state.

[0274] Optionally, the state of the perception modality corresponding to the first trigger condition is determined according to a state identifier of the perception modality corresponding to the first trigger condition;

[0275] or,

[0276] The state of the perception modality corresponding to the first trigger condition is determined according to the state identifier of the perception modality corresponding to the element corresponding to the first trigger condition.

[0277] Optionally, the third element is associated with at least one of the following: a parameter related to attenuation of the tactile signal, a parameter related to propagation distance of the tactile signal;

[0278] The third element includes at least one element of a tactile source or at least one element of a tactile scene.

[0279] Optionally, the attenuation-related parameter includes an attenuation indicator, and the attenuation indicator is used to indicate that the signal strength of the tactile signal attenuates with distance.

[0280] Optionally, there is a correlation between the attenuation-related parameter and the perception modality.

[0281] Optionally, the attenuation amount of the signal strength of the tactile signal is determined based on an attenuation model or an attenuation formula, wherein the attenuation model or the attenuation formula is agreed upon by a protocol.

[0282] Optionally, the propagation distance related parameter includes an upper limit value of the propagation distance of the tactile signal, and the upper limit value of the propagation distance is used to indicate a maximum propagation distance of the tactile signal.

[0283] Optionally, the trigger condition associated with the first element is a distance condition, the second element includes a distance usage identifier, the second element includes at least one element in a first action object, and the first action object is an action object corresponding to the trigger condition associated with the first element;

[0284] The distance usage identifier is used to indicate whether the electronic device triggers tactile feedback based on an element located within a distance range defined by the distance condition.

[0285] Optionally, the trigger condition associated with the first element is a collision condition, the element list corresponding to the collision condition includes at least two elements, each of the at least two elements is associated with a type identifier or a first identifier, and the first identifier associated with each element is the identifier of the upper-layer element of each element;

[0286] Wherein, it is determined that the collision condition is satisfied when elements with different associated type identifiers or first identifiers among the at least two elements collide.

[0287] Optionally, the trigger condition associated with the first element is a collision condition, and the collision condition corresponds to at least two element groups, each of which includes at least one element, wherein the collision condition is determined to be satisfied when elements between different element groups in the at least two element groups collide.

[0288] Optionally, the trigger condition associated with the first element is a collision condition, the collision condition corresponds to at least two elements, and the collision condition is determined to be satisfied when a collision occurs between lower-layer elements of different elements among the at least two elements.

[0289] Optionally, the execution module is specifically configured to:

[0290] Playback of the haptic signal is controlled according to the metadata of the haptic signal.

[0291] The tactile signal processing device in the embodiments of the present application can be an electronic device, such as an electronic device with an operating system, or a component in an electronic device, such as an integrated circuit or chip. The electronic device can be an XR device, a terminal, a server, or a computer.

[0292] The tactile signal processing device provided in the embodiment of the present application can implement each process implemented in the method embodiment of Figure 1 and achieve the same technical effect. To avoid repetition, it will not be described here.

[0293] Optionally, as shown in Figure 3, an embodiment of the present application also provides an electronic device 300, including a processor 301 and a memory 302, and the memory 302 stores a program or instruction that can be run on the processor 301. When the program or instruction is executed by the processor 301, the various steps of the above-mentioned tactile signal processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0294] An embodiment of the present application also provides an electronic device, including a processor and a communication interface, wherein the processor is used to determine a tactile signal and metadata of the tactile signal when a first element triggers tactile feedback; and perform a first operation according to the tactile signal and metadata of the tactile signal; wherein the first element includes at least one of a node and a graph element, and the metadata of the tactile signal includes at least one of the following: actuator information associated with the tactile signal, avatar information associated with the tactile signal, perceptual modality associated with the tactile signal, and attenuation of the signal strength of the tactile signal. This electronic device embodiment corresponds to the above-mentioned electronic device side method embodiment, and each implementation process and implementation method of the above-mentioned method embodiment can be applied to this electronic device embodiment and can achieve the same technical effect. Specifically, Figure 4 is a schematic diagram of the hardware structure of an electronic device that implements an embodiment of the present application.

[0295] The electronic device 400 includes but is not limited to: a radio frequency unit 401, a network module 402, an audio output unit 403, an input unit 404, a sensor 405, a display unit 406, a user input unit 407, an interface unit 408, a memory 409 and at least some of the components of the processor 410.

[0296] Those skilled in the art will appreciate that the electronic device 400 may further include a power source (e.g., a battery) to power various components. The power source may be logically connected to the processor 410 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The electronic device structure shown in FIG4 does not limit the electronic device. The electronic device may include more or fewer components than shown, or may combine certain components or arrange the components differently, which will not be described in detail here.

[0297] It should be understood that in an embodiment of the present application, the input unit 404 may include a graphics processing unit (GPU) 4041 and a microphone 4042, and the graphics processor 4041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 406 may include a display panel 4061, and the display panel 4061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 407 includes a touch panel 4071 and at least one of other input devices 4072. The touch panel 4071 is also called a touch screen. The touch panel 4071 may include two parts: a touch detection device and a touch controller. Other input devices 4072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0298] In the embodiment of the present application, after receiving downlink data from a network-side device, the radio frequency unit 401 may transmit the data to the processor 410 for processing. Furthermore, the radio frequency unit 401 may send uplink data to the network-side device. Typically, the radio frequency unit 401 includes, but is not limited to, an antenna, an amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, and the like.

[0299] The memory 409 can be used to store software programs or instructions and various data. The memory 409 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 409 may include a volatile memory or a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 409 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0300] Processor 410 may include one or more processing units. Optionally, processor 410 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 410.

[0301] Among them, the processor 410 is used to determine the tactile signal and the metadata of the tactile signal when the first element triggers tactile feedback; perform a first operation based on the tactile signal and the metadata of the tactile signal; wherein, the first element includes at least one of a node and a graph element, and the metadata of the tactile signal includes at least one of the following: actuator information associated with the tactile signal, avatar information associated with the tactile signal, perception modality associated with the tactile signal, and attenuation of the signal strength of the tactile signal.

[0302] It can be understood that the implementation process of each implementation method mentioned in this embodiment can refer to the relevant description of the aforementioned method embodiment and achieve the same or corresponding technical effects. To avoid repetition, it will not be repeated here.

[0303] An embodiment of the present application further provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned tactile signal processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0304] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. In some examples, the readable storage medium may be a non-transitory readable storage medium.

[0305] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned tactile signal processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0306] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0307] An embodiment of the present application further provides a computer program / program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-mentioned tactile signal processing method embodiment and can achieve the same technical effect. To avoid repetition, it is not repeated here.

[0308] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0309] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of a computer software product plus a necessary general-purpose hardware platform, or of course, by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, magnetic disk, optical disk, etc.) and includes a number of instructions for enabling a terminal or network-side device to execute the methods described in each embodiment of the present application.

[0310] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms of implementation methods without departing from the purpose of this application and the scope of protection of the claims. These implementation methods are all within the protection of this application.

Claims

1. A tactile signal processing method, comprising: In the case where the first element triggers haptic feedback, the electronic device determines a haptic signal and metadata of the haptic signal; The electronic device performs a first operation according to the tactile signal and metadata of the tactile signal; The first element includes at least one of a node and a graph element, and the metadata of the tactile signal includes at least one of the following: actuator information associated with the tactile signal, avatar information associated with the tactile signal, a perception modality associated with the tactile signal, and an attenuation amount of the signal strength of the tactile signal.

2. The method according to claim 1, wherein: The actuator information associated with the tactile signal includes actuator information associated with the target element; Among them, the target element includes the first element or the second element, the second element is the element associated with the first action object, the first action object is the action object corresponding to the trigger condition associated with the first element, and the second element includes at least one of a node and a graph element.

3. The method according to claim 2, wherein: The actuator information associated with the target element includes the following: body part target, actuator target, actuator resolution; or, The actuator information associated with the target element includes a device identifier, and the device identifier is used to identify the actuator associated with the target element.

4. The method according to any one of claims 1 to 3, wherein: The avatar information associated with the tactile signal includes avatar information associated with the target element; or, The avatar information associated with the tactile signal includes avatar information associated with the tactile scene corresponding to the target element; or, The avatar information associated with the tactile signal includes the avatar information associated with the upper layer element of the target element; Among them, the target element includes the first element or the second element, the second element is the element associated with the first action object, the first action object is the action object corresponding to the trigger condition associated with the first element, and the second element includes at least one of a node and a graph element.

5. The method according to claim 4, wherein: The target element is an element in an activated state; or, The avatar information associated with the tactile signal includes the avatar information of the element in the activated state associated with the tactile scene corresponding to the target element; or, The avatar information associated with the tactile signal includes the avatar information of an activated element associated with an upper-layer element of the target element.

6. The method according to any one of claims 1 to 5, wherein: In the case where the first element includes N elements in an activated state, the number of the tactile signals is N, where N is a positive integer, the N tactile signals correspond one-to-one to the N activated elements, and the avatar information associated with each of the tactile signals is the avatar information of the activated element corresponding to each of the tactile signals.

7. The method according to claim 5 or 6, wherein: The element being in an activated state includes at least one of the following: The element is in a first activation state, and the first activation state is used to indicate that the element can be applied to a tactile scene; The element is in a second activation state, and the second activation state is used to indicate that the element can be used to trigger tactile feedback; The element is in a third activation state, and the third activation state is used to indicate that the element can be applied to perform a tactile feedback action.

8. The method according to any one of claims 5 to 7, wherein: The state of the element is set by a second action object, and the second action object is used to activate or deactivate the element.

9. The method according to any one of claims 1 to 8, wherein: The perceptual modality associated with the tactile signal includes a perceptual modality associated with the target element; or, The perceptual modality associated with the tactile signal includes the perceptual modality associated with the tactile scene corresponding to the target element; or, The perceptual modality associated with the tactile signal includes the perceptual modality associated with the upper layer element of the target element; Among them, the target element includes the first element or the second element, the second element is an element associated with the first action object, the first action object is an action object corresponding to the trigger condition associated with the first element, and the second element includes at least one of a node and a graph element.

10. The method according to claim 9, wherein: The perceptual modality associated with the tactile signal includes the perceptual modality associated with the target element in an activated state; or, The perceptual modality associated with the tactile signal includes the perceptual modality in an activated state associated with the tactile scene corresponding to the target element; or, The perceptual modality associated with the tactile signal includes an activated perceptual modality associated with an upper-layer element of the target element.

11. The method according to any one of claims 1 to 10, wherein: When there are M perceptual modalities in an activated state among the perceptual modalities associated with the target element, the number of the tactile signals is M, where M is a positive integer, the M tactile signals correspond one-to-one to the M perceptual modalities in an activated state, and the perceptual modality associated with each of the tactile signals is the perceptual modality in an activated state corresponding to each of the tactile signals.

12. The method according to claim 10 or 11, wherein: The state of the perception modality is set via a third action object, and the third action object is used to activate or deactivate the perception modality.

13. The method according to any one of claims 1 to 12, wherein: In the case where the first element triggers tactile feedback, before the electronic device determines the tactile signal and metadata of the tactile signal, the method further includes: The electronic device determines a state of a first behavior object, wherein the first behavior object includes a first trigger condition and a first action object, and the first trigger condition is a trigger condition associated with the first element; The electronic device determines whether to apply or not to apply the first behavior object according to the state of the first behavior object.

14. The method according to claim 13, wherein: The state of the first behavior object is determined according to a state identifier associated with the first behavior object, wherein the state identifier is used to identify an activation state or a deactivation state; or, The state of the first behavior object is determined according to at least one of a state of the first trigger condition and a state of the first action object.

15. The method according to claim 14, wherein: The state of the first trigger condition is determined according to a state identifier associated with the first trigger condition, or according to a state identifier associated with an element associated with the first trigger condition; or, The state of the first action object is determined according to a state identifier of the first action object, or according to a state identifier associated with an element associated with the first action object; The state identifier is used to identify an activation state or a deactivation state.

16. The method according to claim 14 or 15, wherein: The state of the first action object is set by a fourth action object, and the fourth action object is used to activate or deactivate the behavior object; or, The state of the first trigger condition is set by a fifth action object, and the fifth action object is used to activate or deactivate the trigger condition; or, The state of the first action object is set according to a sixth action object, and the sixth action object is used to activate or deactivate the action object; or, The state of the element is set according to a seventh action object, and the seventh action object is used to activate or deactivate the element.

17. The method according to claim 13, wherein: The state of the first behavior object is determined according to the state of the perception modality corresponding to the first behavior object; or, The state of the first behavior object is determined based on the perception modality corresponding to the first trigger condition and the perception modality corresponding to the first action object, or the state of the first behavior object is determined based on the perception modality in an activated state corresponding to the first trigger condition and the perception modality in an activated state corresponding to the first action object.

18. The method according to claim 17, wherein: The perception modality corresponding to the first behavior object is the perception modality corresponding to the tactile scene corresponding to the first behavior object.

19. The method according to claim 17 or 18, wherein: The state of the perception modality corresponding to the first behavior object is determined according to a state identifier associated with the perception modality corresponding to the first behavior object.

20. The method according to any one of claims 17 to 19, wherein: When all the perception modalities corresponding to the first behavior object are in a deactivated state, the first behavior object is in a deactivated state.

21. The method according to claim 17, wherein: When the perception modality corresponding to the first trigger condition and the perception modality corresponding to the first action object are the same, the first action object is in an activated state; otherwise, the first action object is in a deactivated state; or, When the perception modality in an activated state corresponding to the first trigger condition is the same as the perception modality in an activated state corresponding to the first action object, the first behavior object is in an activated state; otherwise, the first behavior object is in a deactivated state.

22. The method according to claim 21, wherein: The state of the perception modality corresponding to the first trigger condition is determined according to the state identifier of the perception modality corresponding to the first trigger condition; or, The state of the perception modality corresponding to the first trigger condition is determined according to the state identifier of the perception modality corresponding to the element corresponding to the first trigger condition.

23. The method according to any one of claims 1 to 22, wherein: The third element is associated with at least one of the following: a parameter related to the attenuation of the tactile signal, a parameter related to the propagation distance of the tactile signal; The third element includes at least one element of a tactile source or at least one element of a tactile scene.

24. The method according to claim 23, wherein: The attenuation-related parameter includes an attenuation indicator, and the attenuation indicator is used to indicate that the signal strength of the tactile signal attenuates with distance.

25. The method according to claim 23 or 24, wherein: There is a correlation between the attenuation-related parameters and the perception modality.

26. The method according to any one of claims 23 to 25, wherein: The attenuation amount of the signal strength of the tactile signal is determined based on an attenuation model or an attenuation formula, wherein the attenuation model or the attenuation formula is agreed upon by a protocol.

27. The method according to any one of claims 23 to 26, wherein: The propagation distance related parameters include an upper limit value of the propagation distance of the tactile signal, and the upper limit value of the propagation distance is used to indicate a maximum propagation distance of the tactile signal.

28. The method according to any one of claims 1 to 27, wherein: The trigger condition associated with the first element is a distance condition, the second element includes a distance usage identifier, the second element includes at least one element in the first action object, and the first action object is an action object corresponding to the trigger condition associated with the first element; The distance usage identifier is used to indicate whether the electronic device triggers tactile feedback according to an element located within a distance range defined by the distance condition.

29. The method according to any one of claims 1 to 28, wherein: The trigger condition associated with the first element is a collision condition, the element list corresponding to the collision condition includes at least two elements, each of the at least two elements is associated with a type identifier or a first identifier, and the first identifier associated with each of the elements is the identifier of the upper element of each of the elements; Wherein, it is determined that the collision condition is satisfied when elements with different associated type identifiers or first identifiers in the at least two elements collide.

30. The method according to any one of claims 1 to 28, wherein: The trigger condition associated with the first element is a collision condition, and the collision condition corresponds to at least two element groups, each of which includes at least one element, wherein the collision condition is determined to be satisfied when elements between different element groups in the at least two element groups collide.

31. The method according to any one of claims 1 to 28, wherein: The trigger condition associated with the first element is a collision condition, the collision condition corresponds to at least two elements, and the collision condition is determined to be satisfied when a collision occurs between lower-layer elements of different elements in the at least two elements.

32. The method according to any one of claims 1 to 31, wherein: The electronic device performs a first operation according to the tactile signal and metadata of the tactile signal, including: The electronic device controls the playing of the haptic signal according to the metadata of the haptic signal.

33. A tactile signal processing device, comprising: A first determining module, configured to determine a tactile signal and metadata of the tactile signal when the first element triggers tactile feedback; an execution module, configured to execute a first operation according to the tactile signal and metadata of the tactile signal; The first element includes at least one of a node and a graph element, and the metadata of the tactile signal includes at least one of the following: actuator information associated with the tactile signal, avatar information associated with the tactile signal, a perception modality associated with the tactile signal, and an attenuation amount of the signal strength of the tactile signal.

34. An electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the tactile signal processing method according to any one of claims 1 to 32 are implemented.

35. A readable storage medium storing a program or instruction, wherein the program or instruction, when executed by a processor, implements the steps of the tactile signal processing method according to any one of claims 1 to 32.

Images