Tactile signal processing method and device and electronic equipment

Abstract

The invention discloses a tactile signal processing method and device and electronic equipment, and belongs to the technical field of communication, and the tactile signal processing method comprises the steps that under the condition that a first element triggers tactile feedback, the electronic equipment determines a tactile signal and metadata of the tactile signal; the electronic equipment executes a first operation according to the tactile signal and metadata of the tactile signal; wherein the first element comprises at least one of a node and a primitive, and the metadata of the tactile signal comprises at least one of the following items: actuator information associated with the tactile signal, avatar information associated with the tactile signal, a perceptual mode associated with the tactile signal, and attenuation of signal intensity of the tactile signal.

Description

Technical Field

[0001] The present application belongs to the field of communication technology, and specifically relates to a tactile signal processing method, device and electronic device. Background Art

[0002] Extended Reality (XR) technology is used to achieve the combination of reality and virtuality, including but not limited to at least one of Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), etc. In order to improve the interaction between users and virtual environments, tactile scenes are introduced in XR scenes, which can provide tactile feedback to user avatars, such as generating trigger signals and playing trigger signals. However, the related art only involves generating tactile signals when tactile feedback is triggered, and there is no relevant solution for how to determine the metadata of the tactile signal, which can easily lead to poor tactile feedback effects. Summary of the invention

[0003] The embodiments of the present application provide a tactile signal processing method, device and electronic device, which can provide a method for determining metadata of a tactile signal, which is conducive to improving the effect of tactile feedback.

[0004] In a first aspect, a tactile signal processing method is provided, the method comprising:

[0005] In the case where the first element triggers haptic feedback, the electronic device determines a haptic signal and metadata of the haptic signal;

[0006] The electronic device performs a first operation according to the tactile signal and metadata of the tactile signal;

[0007] The first element includes at least one of a node and a graph element, and the metadata of the tactile signal includes at least one of the following: actuator information associated with the tactile signal, avatar information associated with the tactile signal, a perception modality associated with the tactile signal, and an attenuation amount of the signal strength of the tactile signal.

[0008] In a second aspect, a tactile signal processing device is provided, the device comprising:

[0009] A first determining module, configured to determine a tactile signal and metadata of the tactile signal when the first element triggers tactile feedback;

[0010] an execution module, configured to execute a first operation according to the tactile signal and metadata of the tactile signal;

[0011] The first element includes at least one of a node and a graph element, and the metadata of the tactile signal includes at least one of the following: actuator information associated with the tactile signal, avatar information associated with the tactile signal, a perception modality associated with the tactile signal, and an attenuation amount of the signal strength of the tactile signal.

[0012] According to a third aspect, an electronic device is provided, comprising a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0013] In a fourth aspect, an electronic device is provided, comprising a processor and a communication interface, wherein the processor is used to determine a tactile signal and metadata of the tactile signal when a first element triggers tactile feedback; and perform a first operation according to the tactile signal and the metadata of the tactile signal; wherein the first element includes at least one of a node and a graph element, and the metadata of the tactile signal includes at least one of the following: actuator information associated with the tactile signal, avatar information associated with the tactile signal, perception modality associated with the tactile signal, and attenuation of the signal strength of the tactile signal.

[0014] In a fifth aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0015] In a sixth aspect, a chip is provided, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the method described in the first aspect.

[0016] In a seventh aspect, a computer program / program product is provided, wherein the computer program / program product is stored in a storage medium and is executed by at least one processor to implement the steps of the method described in the first aspect.

[0017] In an embodiment of the present application, when the first element triggers tactile feedback, the electronic device determines the tactile signal and metadata of the tactile signal, and performs a first operation based on the tactile signal and the metadata of the tactile signal. Since the metadata of the tactile signal includes at least one of actuator information associated with the tactile signal, avatar information associated with the tactile signal, perceptual modality associated with the tactile signal, and attenuation of signal strength of the tactile signal, controlling the playback of the tactile signal based on the metadata of the tactile signal is beneficial to improving the effect of tactile feedback. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of a tactile signal processing method provided by an embodiment of the present application;

[0019] Figure 2 is a structural diagram of a tactile signal processing device provided in an embodiment of the present application;

[0020] Figure 3 is a structural diagram of an electronic device provided in an embodiment of the present application;

[0021] Figure 4 It is a structural diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0023] The terms "first", "second", etc. of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of one type, and the number of objects is not limited, for example, the first object can be one or more. In addition, "or" in the present application represents at least one of the connected objects. For example, "A or B" covers three schemes, namely, Scheme 1: including A but not including B; Scheme 2: including B but not including A; Scheme 3: including both A and B. The character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0024] The term "indication" in this application can be a direct indication (or explicit indication) or an indirect indication (or implicit indication). A direct indication can be understood as the sender explicitly informing the receiver of specific information, operations to be performed, or request results in the sent indication; an indirect indication can be understood as the receiver determining the corresponding information according to the indication sent by the sender, or making a judgment and determining the operation to be performed or the request result according to the judgment result.

[0025] For ease of understanding, some contents involved in the embodiments of the present application are described below:

[0026] 1. Graphics Language Transmission Format (glTF) scene language

[0027] glTF is the basic language for describing 3D scenes and models. glTF builds a program framework based on nodes. A glTF modeling scene can be represented by a root node, and objects in the modeling scene can be described by multiple nodes / sub-nodes.

[0028] 2. Avatar Description

[0029] In the MPEG scene modeling standard, a preliminary description standard for avatars was designed, and avatar-related extensions were made at the node level. The avatar structure can be established using paths and nodes.

[0030] 3. Interactivity Description

[0031] In the Moving Picture Expert Group (MPEG) scene modeling standard, a preliminary description standard for interactivity was designed. For haptic scenes, it is mainly used to describe the interaction between the avatar and the haptic source, where both the haptic source and the avatar can be modeled with elements (such as nodes or primitives). Based on this, haptic signals are triggered and generated.

[0032] Interactivity is defined based on behavior objects, which include the trigger conditions of the behavior object (i.e., trigger objects) and the action objects of the behavior object. Triggers include collision conditions (i.e., collision triggers), distance conditions (i.e., proximity triggers), etc. Action objects include haptic feedback action objects (e.g., set haptic), etc.

[0033] For collision conditions, the node array / primitive arrary is defined in the collision conditions. If any node / primitive in the node array / primitive arrary collides, the collision condition is satisfied. For proximity conditions, the distance range and node array / primitive arrary are defined in the proximity conditions. If the node / primitive is within the distance range, the proximity condition is satisfied. For tactile feedback actions, several parameters for tactile signal generation and playback are defined in the tactile feedback action. Exemplarily, the meaning (Semantic) of the above-mentioned trigger objects can be as shown in Table 1, the meaning of the above-mentioned action objects can be as shown in Table 2, and the meaning of the tactile action node object (HapticActionNode object) can be as shown in Table 3.

[0034] Table 1 Meaning of trigger objects

[0035]

[0036]

[0037] Table 2 Meaning of action objects

[0038]

[0039] Table 3 Meaning of tactile action node objects

[0040]

[0041] The tactile signal processing method provided by the embodiment of the present application is described in detail below through some embodiments and their application scenarios in combination with the accompanying drawings.

[0042] See also Figure 1 , Figure 1 It is a flowchart of a tactile signal processing method provided in an embodiment of the present application. The method can be executed by an electronic device, wherein the electronic device can be an XR device, a cloud device, a terminal device, a rendering device, a rendering engine, a server or a computer, etc.

[0043] like Figure 1 As shown, the tactile signal processing method provided in the embodiment of the present application includes the following steps:

[0044] Step 101: When a first element triggers tactile feedback, the electronic device determines a tactile signal and metadata of the tactile signal;

[0045] Step 102: the electronic device performs a first operation according to the tactile signal and metadata of the tactile signal;

[0046] The first element includes at least one of a node and a graph element, and the metadata of the tactile signal includes at least one of the following: actuator information associated with the tactile signal, avatar information associated with the tactile signal, a perception modality associated with the tactile signal, and an attenuation amount of the signal strength of the tactile signal.

[0047] The first element may include at least one of an avatar or user element and a tactile source element, wherein the element may include at least one of a node and a primitive.

[0048] The first element triggers tactile feedback, for example, the first element satisfies a proximity condition (ie, the first element triggers a proximity trigger), or the first element satisfies a collision condition (ie, the first element triggers a collision trigger).

[0049] The actuator information may include but is not limited to actuator position information or actuator identification, etc. The avatar information may include but is not limited to avatar ID. The perception modality may include but is not limited to at least one of temperature, vibration, etc. The attenuation is used to characterize the attenuation of the signal strength of the tactile signal with distance.

[0050] For the above step 101, illustratively, when the first element satisfies the distance condition or the collision condition, the electronic device may generate a tactile signal and metadata of the tactile signal.

[0051] For the above step 102, exemplarily, when the above electronic device is an XR device, the above electronic device may control the playback of the above tactile signal according to the metadata of the above tactile signal; when the above electronic device is a device other than the XR device (for example, a server or a computer, etc.), the above electronic device may send the above perception signal and the metadata of the perception signal to the XR device, and then the XR device controls the playback of the above tactile signal according to the metadata of the above perception signal.

[0052] In the tactile signal processing method provided in an embodiment of the present application, when a first element triggers tactile feedback, the electronic device determines a tactile signal and metadata of the tactile signal, and performs a first operation based on the tactile signal and the metadata of the tactile signal. Since the metadata of the tactile signal includes at least one of actuator information associated with the tactile signal, avatar information associated with the tactile signal, perception mode associated with the tactile signal, and attenuation of signal strength of the tactile signal, controlling the playback of the tactile signal based on the metadata of the tactile signal is beneficial to improving the effect of tactile feedback.

[0053] Optionally, the actuator information associated with the tactile signal includes actuator information associated with the target element;

[0054] Among them, the target element includes the first element or the second element, the second element is the element associated with the first action object, the first action object is the action object corresponding to the trigger condition associated with the first element, and the second element includes at least one of a node and a graph element.

[0055] In this embodiment, the above-mentioned action object may include but is not limited to a haptic action node (hapticActionNode) group, and each haptic action node of the haptic action node group may include but is not limited to a node identifier, an action location (actionLocation), a useCollider indication, a material haptic modality (materialHapticModality) group, a haptic action media (hapticActionMedias) list, etc. Exemplarily, the above-mentioned action object can refer to the above-mentioned related description, which will not be repeated here.

[0056] The above-mentioned first action object is the action object corresponding to the trigger condition associated with the first element (i.e., the first trigger condition), wherein the above-mentioned first element may include at least one element in the element group defined in the first trigger condition (e.g., node array / primitive arrary), and the above-mentioned first action object and the above-mentioned first trigger condition are located in the same behavior object (i.e., the first behavior object), that is, the above-mentioned first behavior object includes the above-mentioned first action object and the first trigger condition. The above-mentioned second element may include at least one element in the element group defined in the above-mentioned first action object. Among them, the above-mentioned behavior object can refer to the above-mentioned relevant description, which will not be repeated here. The above-mentioned trigger condition may include at least one of a collision condition (i.e., collision trigger) and a distance condition (i.e., proximity trigger), and the above-mentioned collision condition and distance condition can refer to the above-mentioned relevant description, which will not be repeated here.

[0057] The actuator information associated with the above-mentioned tactile signal includes the actuator information associated with the target element, that is, the actuator information associated with the target element is used as the actuator information associated with the above-mentioned tactile signal. For example, the actuator position information associated with the target element is used as the actuator position information associated with the above-mentioned tactile signal, or the actuator identifier associated with the target element is used as the actuator identifier associated with the above-mentioned tactile signal. This makes it easy to accurately and quickly locate the actuator used to play the above-mentioned tactile signal.

[0058] Optionally, the actuator information associated with the target element includes the following: body part target, actuator target, actuator resolution;

[0059] or,

[0060] The actuator information associated with the target element includes a device identifier, and the device identifier is used to identify the actuator associated with the target element.

[0061] In one embodiment, at least one of a body part target (i.e., body_part_target), an actuator target (i.e., actuator target), and an actuator resolution (i.e., actuatorresolution) is defined in an element (e.g., a node or primitive of a user or avatar) to accurately locate the position information of the actuator on the humanoid avatar.

[0062] Exemplarily, taking the above elements as nodes as an example, the newly added attributes of the nodes may be as shown in Table 4, and the meaning of the haptic actuator object may be as shown in Table 5.

[0063] Table 4 Node attributes

[0064]

[0065] Table 5 Meaning of haptic actuator objects

[0066]

[0067]

[0068] In another embodiment, a device ID may be defined in an element (eg, a node or a primitive of a user or an avatar) to identify an actuator ID associated with the element. For example, taking the above element as a node, the newly added attributes of the node may be as shown in Table 6.

[0069] Table 6 Node attributes

[0070]

[0071] Optionally, the avatar information associated with the tactile signal includes avatar information associated with the target element;

[0072] or,

[0073] The avatar information associated with the tactile signal includes avatar information associated with the tactile scene corresponding to the target element;

[0074] or,

[0075] The avatar information associated with the tactile signal includes the avatar information associated with the upper layer element of the target element;

[0076] Among them, the target element includes the first element or the second element, the second element is the element associated with the first action object, the first action object is the action object corresponding to the trigger condition associated with the first element, and the second element includes at least one of a node and a graph element.

[0077] In one embodiment, the avatar information associated with the target element can be used as the avatar information associated with the tactile signal. For example, an avatar ID can be defined in an element (e.g., a node or primitive of a user or an avatar). If tactile feedback is triggered, the avatar ID in the element associated with the tactile feedback is used as the avatar ID associated with the generated tactile signal, that is, the avatar ID in the metadata of the tactile signal is set to the avatar ID in the element associated with the tactile feedback.

[0078] In another embodiment, the avatar information associated with the tactile scene corresponding to the target element may be used as the avatar information associated with the tactile signal. For example, for a tactile scene corresponding to an avatar or an activated avatar, the avatar ID of the avatar or the activated avatar may be used as the avatar ID associated with the tactile signal, that is, the avatar ID field in the metadata of the tactile signal is set to the avatar ID of the avatar or the activated avatar.

[0079] In another embodiment, the avatar information associated with the upper element of the target element can be used as the avatar information associated with the tactile signal, wherein the upper element of the target element, for example, the target element is a node, and the upper element can be the parent node of the node. Exemplarily, for an upper element corresponding to an avatar or an activated avatar, the avatar ID of the avatar or the activated avatar can be used as the avatar ID associated with the tactile signal, that is, the avatar ID field in the metadata of the tactile signal is set to the avatar ID of the avatar or the activated avatar.

[0080] Optionally, the target element is an element in an activated state;

[0081] or,

[0082] The avatar information associated with the tactile signal includes the avatar information of the element in the activated state associated with the tactile scene corresponding to the target element;

[0083] or,

[0084] The avatar information associated with the tactile signal includes the avatar information of an activated element associated with an upper-layer element of the target element.

[0085] Exemplarily, the activated element associated with the tactile scene or the activated element associated with the upper layer element can be used to represent an avatar, that is, the avatar information associated with the tactile signal is the avatar information of the activated avatar.

[0086] Exemplarily, a state flag may be defined in an element (e.g., avatar node / primitive), and the state flag is used to identify an activation state or a deactivation state, and is used to characterize whether the element is currently the execution target of the tactile signal, for example, as shown in Table 7. When a tactile signal is generated, the avatar ID field in the metadata of the tactile signal may be set to the avatar ID of the activated element.

[0087] Table 7

[0088]

[0089] Optionally, in the case that the first element includes N elements in an activated state, the number of the tactile signals is N, wherein N is a positive integer, the N tactile signals correspond one-to-one to the N activated elements, and the avatar information associated with each of the tactile signals is the avatar information of the activated element corresponding to each of the tactile signals.

[0090] Exemplarily, when multiple activated avatars coexist, a haptic signal may be generated for each avatar, and the avatar ID field in the metadata of each haptic signal is set to the avatar ID of the corresponding avatar.

[0091] Optionally, the element being in an activated state includes at least one of the following:

[0092] The element is in a first activation state, and the first activation state is used to indicate that the element can be applied to a tactile scene;

[0093] The element is in a second activation state, and the second activation state is used to indicate that the element can be used to trigger tactile feedback;

[0094] The element is in a third activation state, and the third activation state is used to indicate that the element can be applied to perform a tactile feedback action.

[0095] Exemplarily, at least one of a first state identifier, a second state identifier, and a third state identifier may be defined in an element (e.g., an avatar node / primitive), wherein the first state identifier, the second state identifier, and the third state identifier may all include an activation state or a deactivation state, the first state identifier may be used to indicate whether the element can be applied to a tactile scene, the second state identifier may be used to indicate whether the element can be applied to trigger tactile feedback, and the third state identifier may be used to indicate that the element can be applied to perform a tactile feedback action. For example, see Table 8, wherein the status (Status) in Table 8 may represent the first state identifier, the status (Status_Trigger) in Table 8 may represent the second state identifier, and the Status_Action in Table 8 may represent the third state identifier.

[0096] Table 8

[0097]

[0098] Optionally, the state of the element is set via a second action object, and the second action object is used to activate or deactivate the element.

[0099] In this embodiment, the second action object may include functions and parameters related to activating or deactivating the element, so that the state of the element may be set through the second action object.

[0100] Optionally, the perceptual modality associated with the tactile signal includes a perceptual modality associated with the target element;

[0101] or,

[0102] The perceptual modality associated with the tactile signal includes the perceptual modality associated with the tactile scene corresponding to the target element;

[0103] or,

[0104] The perceptual modality associated with the tactile signal includes the perceptual modality associated with the upper layer element of the target element;

[0105] Among them, the target element includes the first element or the second element, the second element is an element associated with the first action object, the first action object is an action object corresponding to the trigger condition associated with the first element, and the second element includes at least one of a node and a graph element.

[0106] In one embodiment, the perception modality associated with the target element can be used as the perception modality associated with the tactile signal. Exemplarily, a parameter perception modality is defined in the element, and if tactile feedback is triggered, the corresponding perception modality in the element involved in the Trigger / Action associated with the tactile feedback is used as the perception modality associated with the generated tactile signal, that is, the perception modality field of the metadata of the tactile signal is set to the corresponding perception modality in the element involved in the Trigger / Action associated with the tactile feedback.

[0107] In another embodiment, the perception modality associated with the tactile scene corresponding to the target element can be used as the perception modality associated with the tactile signal. Exemplarily, for a tactile scene corresponding to a perception modality, the perception modality corresponding to the tactile scene can be used as the perception modality associated with the tactile signal, that is, the perception modality field in the metadata of the tactile signal is set to the perception modality corresponding to the tactile scene.

[0108] In another embodiment, the perception modality associated with the upper element of the target element can be used as the perception modality associated with the tactile signal. Exemplarily, for a perception modality corresponding to an upper element, the perception modality corresponding to the upper element can be used as the perception modality associated with the tactile signal, that is, the perception modality field in the metadata of the tactile signal is set to the perception modality corresponding to the upper element.

[0109] Optionally, the perceptual modality associated with the tactile signal includes an activated perceptual modality associated with the target element;

[0110] or,

[0111] The perceptual modality associated with the tactile signal includes the perceptual modality in an activated state associated with the tactile scene corresponding to the target element;

[0112] or,

[0113] The perceptual modality associated with the tactile signal includes an activated perceptual modality associated with an upper-layer element of the target element.

[0114] For example, when multiple perception modalities coexist, a state flag of perception modality can be introduced to indicate whether a perception modality is in an activated state. When a tactile signal is generated, the perception modality field in the metadata of the tactile signal can be set to the perception modality in the activated state; or, a modality list can be used to indicate one or more perception modalities in the activated state.

[0115] Optionally, when there are M perceptual modalities in an activated state among the perceptual modalities associated with the target element, the number of the tactile signals is M, wherein M is a positive integer, the M tactile signals correspond one-to-one to the M perceptual modalities in an activated state, and the perceptual modality associated with each of the tactile signals is the perceptual modality in an activated state corresponding to each of the tactile signals.

[0116] Exemplarily, when multiple perception modalities or multiple perception modalities in an activated state coexist, a tactile signal may be generated for each perception modality or each perception modality in an activated state, and the perception modality field in the metadata of each tactile signal is set to the corresponding perception modality.

[0117] Optionally, the state of the perception modality is set via a third action object, and the third action object is used to activate or deactivate the perception modality.

[0118] In this embodiment, the third action object may include functions and parameters related to activating or deactivating the perception modality, so that the state of the perception modality may be set through the third action object.

[0119] Exemplarily, as shown in Table 9, the action object for activating or deactivating the perception modality may include a perception modality or a perception modality list, which is used to indicate the perception modality that can be applied to the virtual world, that is, the perception modality in an activated state.

[0120] Table 9

[0121]

[0122] Optionally, in the case where the first element triggers tactile feedback, before the electronic device determines the tactile signal and metadata of the tactile signal, the method further includes:

[0123] The electronic device determines a state of a first behavior object, wherein the first behavior object includes a first trigger condition and a first action object, and the first trigger condition is a trigger condition associated with the first element;

[0124] The electronic device determines whether to apply or not to apply the first behavior object according to the state of the first behavior object.

[0125] In this embodiment, the state of the above-mentioned first behavior object may include an activation state or a deactivation state, wherein, when the above-mentioned first behavior object is in the activation state, it indicates that the first behavior object is valid. In this case, the electronic device can apply or execute the first behavior object, that is, the electronic device can apply the trigger condition included in the first behavior object (i.e., the first trigger condition) to detect whether tactile feedback is triggered, and can apply the action object included in the first behavior object (i.e., the first action object) to perform a tactile feedback action; when the above-mentioned first behavior object is in the deactivation state, it indicates that the first behavior object is invalid. In this case, the electronic device may not apply or execute the first behavior object, that is, the electronic device does not apply the trigger condition included in the first behavior object (i.e., the first trigger condition) to detect whether tactile feedback is triggered, or does not apply the action object included in the first behavior object (i.e., the first action object) to perform a tactile feedback action.

[0126] Exemplarily, an interaction model may include invalid behavior objects. For example, because an avatar or perception modality is deactivated, the behavior object is invalid. To avoid executing invalid commands, the electronic device may determine the state of each behavior object and only apply the behavior object that is in an activated state.

[0127] Optionally, the state of the first behavior object is determined according to a state identifier associated with the first behavior object, wherein the state identifier is used to identify an activation state or a deactivation state;

[0128] or,

[0129] The state of the first behavior object is determined according to at least one of a state of the first trigger condition and a state of the first action object.

[0130] In one embodiment, the state of the first behavior object can be determined based on the state identifier associated with the first behavior object. For example, if the state identifier associated with the first behavior object identifies a deactivated state, then the first behavior object is determined to be in a deactivated state; if the state identifier associated with the first behavior object identifies an activated state, then the first behavior object is determined to be in an activated state.

[0131] Exemplarily, a status flag can be set in a behavior object, and for a deactivated behavior object, the behavior object does not need to be executed. For example, each behavior object includes a trigger condition, an action object, and a status flag, that is, behavior{Trigger,Action,Status}, where Status can be used to identify an activation state or a deactivated state.

[0132] In another embodiment, the state of the first behavior object can be determined based on at least one of the state of the first trigger condition and the state of the first action object. For example, when the first trigger condition and the first action object are both in an activated state, it can be determined that the first behavior object is in an activated state; otherwise, it can be determined that the first behavior object is in a deactivated state.

[0133] Optionally, the state of the first trigger condition is determined according to a state identifier associated with the first trigger condition, or according to a state identifier associated with an element associated with the first trigger condition;

[0134] or,

[0135] The state of the first action object is determined according to a state identifier of the first action object, or according to a state identifier associated with an element associated with the first action object;

[0136] The state identifier is used to identify an activation state or a deactivation state.

[0137] The following situations are explained:

[0138] (1) Status of the first trigger condition

[0139] In one embodiment, the state of the first trigger condition can be determined according to the state identifier associated with the first trigger condition. For example, if the state identifier associated with the first trigger condition identifies a deactivated state, the first trigger condition is determined to be in a deactivated state. If the state identifier associated with the first trigger condition identifies an activated state, the first trigger condition is determined to be in an activated state. Exemplarily, a state identifier can be set in a trigger condition (i.e., trigger) to identify the activated state or deactivated state of the trigger condition. For a deactivated trigger condition, the behavior object corresponding to the trigger condition does not need to be executed.

[0140] In another embodiment, the state of the first trigger condition can be determined according to the state identifier associated with the element associated with the first trigger condition. Exemplarily, a state identifier can be set in the element to identify the activation state or deactivation state of the element, and then the activation state or deactivation state of the element in the trigger condition can be used to determine whether the trigger condition is activated. For example, if there is at least one element in the element list corresponding to the trigger condition that is in an activated state, it is determined that the trigger condition is in an activated state.

[0141] (2) The state of the first action object

[0142] In one embodiment, the state of the first action object can be determined according to the state identifier associated with the first action object. For example, if the state identifier associated with the first action object identifies a deactivated state, the first action object is determined to be in a deactivated state. If the state identifier associated with the first action object identifies an activated state, the first action object is determined to be in an activated state. Exemplarily, a state identifier can be set in the action object to identify the activated state or the deactivated state of the action object. For a deactivated action object, the behavior object corresponding to the action object does not need to be executed.

[0143] In another embodiment, the state of the first action object can be determined according to the state identifier associated with the element associated with the first action object. Exemplarily, a state identifier can be set in the element to identify the activation state or deactivation state of the element, and then the action object can be determined to be activated according to the activation state or deactivation state of the element in the action object. For example, if there is at least one element in the element list corresponding to the action object that is in the activated state, it is determined that the action object is in the activated state.

[0144] In some optional embodiments, there are multiple activation / deactivation forms for an element, for example, (a) whether the element is always activated / deactivated, (b) whether the element is in an activated / deactivated state when it is a trigger element, and (c) whether the element is in an activated / deactivated state when it is an action element.

[0145] It should be noted that, for scenarios in which a humanoid avatar and a car type avatar exist in the virtual world, when a person enters the car, the car type avatar is responsible for triggering and generating tactile signals, and the humanoid avatar will not trigger the generation of tactile signals at this time. However, the humanoid avatar is still in effect. For example, the position of the humanoid avatar can be used to determine when the person leaves the car. Therefore, it is necessary to introduce the above (b) and (c).

[0146] Exemplarily, as shown in Table 10, the state activation action (i.e., the action object used to activate or deactivate the state of an element) includes setting the trigger state (Set Status_Trigger) and setting the action state (Set Status_Action), wherein Set Status_Trigger is used to indicate whether the element as a trigger condition (i.e., Trigger) is activated, and Set Status_Action is used to indicate whether the element as an action object is activated.

[0147] Table 10

[0148]

[0149] Optionally, the state of the first action object is set by a fourth action object, and the fourth action object is used to activate or deactivate the behavior object;

[0150] or,

[0151] The state of the first trigger condition is set by a fifth action object, and the fifth action object is used to activate or deactivate the trigger condition;

[0152] or,

[0153] The state of the first action object is set according to a sixth action object, and the sixth action object is used to activate or deactivate the action object;

[0154] or,

[0155] The state of the element is set according to a seventh action object, and the seventh action object is used to activate or deactivate the element.

[0156] In this embodiment, the fourth action object may include functions and parameters related to activating or deactivating the behavior object, so that the state of the behavior object can be set through the fourth action object.

[0157] The fifth action object may include functions, parameters, etc. related to activation or deactivation of the trigger condition, so that the state of the first trigger condition may be set through the fifth action object.

[0158] The sixth action object may include functions, parameters, etc. related to activating or deactivating the action object, so that the state of the first action object may be set through the sixth action object.

[0159] The seventh action object may include functions and parameters related to activating or deactivating the element, so that the state of the element may be set through the seventh action object.

[0160] Optionally, the state of the first behavior object is determined according to the state of the perception modality corresponding to the first behavior object;

[0161] or,

[0162] The state of the first behavior object is determined based on the perception modality corresponding to the first trigger condition and the perception modality corresponding to the first action object, or the state of the first behavior object is determined based on the perception modality in an activated state corresponding to the first trigger condition and the perception modality in an activated state corresponding to the first action object.

[0163] In one embodiment, the state of the first behavior object may be determined according to the state of the perception modality corresponding to the first behavior object. For example, if at least one of the perception modalities corresponding to the first behavior object is in an activated state, the first behavior object is in an activated state, otherwise the first behavior object is in a deactivated state; or, if all the perception modalities corresponding to the first behavior object are in an activated state, the first behavior object is in an activated state, otherwise the first behavior object is in a deactivated state.

[0164] In another embodiment, the state of the first behavior object is determined based on the perception modality corresponding to the first trigger condition and the perception modality corresponding to the first action object. For example, if the perception modality corresponding to the first trigger condition and the perception modality corresponding to the first action object are the same, then the first behavior object is determined to be in an activated state, otherwise, the first behavior object is determined to be in a deactivated state; or, the state of the first behavior object is determined based on the perception modality in the activated state corresponding to the first trigger condition and the perception modality in the activated state corresponding to the first action object. For example, if the perception modality in the activated state corresponding to the first trigger condition and the perception modality in the activated state corresponding to the first action object are the same, then the first behavior object is determined to be in an activated state, otherwise, the first behavior object is determined to be in a deactivated state.

[0165] Optionally, the perception modality corresponding to the first behavior object is the perception modality corresponding to the tactile scene corresponding to the first behavior object.

[0166] Exemplarily, a parameter perception modality may be introduced at the tactile scene level. If a certain perception modality is deactivated in the tactile scene, the element or behavior object corresponding to the perception modality will be deactivated.

[0167] Optionally, the state of the perception modality corresponding to the first behavior object is determined according to a state identifier associated with the perception modality corresponding to the first behavior object.

[0168] In this embodiment, a state identifier can be set at the perception modality level to identify the activation state or deactivation state of the perception modality. Exemplarily, at least one of the perception modality and the state identifier of the perception modality can be introduced at the behavior object level. If the behavior object only corresponds to the deactivated perception modality, the behavior object does not need to be executed. If the behavior object corresponds to multiple perception modalities, each perception modality can correspond to a state identifier.

[0169] Optionally, when all perception modalities corresponding to the first behavior object are in a deactivated state, the first behavior object is in a deactivated state.

[0170] Optionally, when the perception modality corresponding to the first trigger condition and the perception modality corresponding to the first action object are the same, the first action object is in an activated state, otherwise the first action object is in a deactivated state;

[0171] or,

[0172] When the perception modality in an activated state corresponding to the first trigger condition is the same as the perception modality in an activated state corresponding to the first action object, the first behavior object is in an activated state; otherwise, the first behavior object is in a deactivated state.

[0173] Optionally, the state of the perception modality corresponding to the first trigger condition is determined according to a state identifier of the perception modality corresponding to the first trigger condition;

[0174] or,

[0175] The state of the perception modality corresponding to the first trigger condition is determined according to the state identifier of the perception modality corresponding to the element corresponding to the first trigger condition.

[0176] In one implementation, at least one of the perception modality and the state flag of the perception modality is set at the trigger condition (ie, trigger) level.

[0177] In another embodiment, at least one of the perception modality and the state flag of the perception modality is set at the element level, and the perception modality activated by the trigger condition is determined according to the perception modality of the element and the state flag of the perception modality in the trigger condition.

[0178] Optionally, the state of the perception modality corresponding to the first action object is determined according to a state identifier of the perception modality corresponding to the first action object;

[0179] or,

[0180] The state of the perception modality corresponding to the first action object is determined according to the state identifier of the perception modality corresponding to the element corresponding to the first action object.

[0181] In one embodiment, at least one of the perception modality and the state flag of the perception modality is set at the action object level.

[0182] In another embodiment, at least one of the perception modality and the state flag of the perception modality is set at the element level, and the perception modality activated by the action object is determined according to the perception modality of the element in the action object and the state flag of the perception modality.

[0183] In some optional embodiments, at least one perception modality in an activated state may be directly indicated by a perception modality or a perception modality list.

[0184] Optionally, the third element is associated with at least one of the following: an attenuation-related parameter of the tactile signal, a propagation distance-related parameter of the tactile signal;

[0185] The third element includes at least one element of a tactile source or at least one element of a tactile scene.

[0186] Exemplarily, the above-mentioned attenuation related parameters may include an attenuation identifier, related parameters for attenuation amount calculation (attenuation formula), etc., for example, as shown in Table 11.

[0187] Table 11 Haptic source node attenuation properties

[0188]

[0189] The above-mentioned propagation distance related parameters may include a maximum propagation distance of the tactile signal or may be used to indicate a propagation range of the tactile signal.

[0190] It is understandable that the perceived intensity of the tactile signal is related to the distance from the tactile signal source. For example, when there is an explosion, the farther away from the explosion source, the weaker the vibration (perception modality = vibrotactile) and the smaller the heat (perception modality = temperature) felt; when the distance is farther, the user may not even feel the vibration or heat caused by the explosion. This embodiment associates at least one of the attenuation-related parameters of the tactile signal and the propagation distance-related parameters of the tactile signal in the third element, which is conducive to improving the effect of tactile feedback.

[0191] Optionally, the attenuation-related parameter includes an attenuation indicator, and the attenuation indicator is used to indicate that the signal strength of the tactile signal attenuates with distance.

[0192] Optionally, there is a correlation between the attenuation-related parameter and the perception modality.

[0193] For example, attenuation-related parameters are set for each perception modality.

[0194] Optionally, the attenuation amount of the signal strength of the tactile signal is determined based on an attenuation model or an attenuation formula, wherein the attenuation model or the attenuation formula is agreed upon by a protocol.

[0195] Exemplarily, the attenuation amount of the signal strength of the tactile signal may be calculated according to the attenuation model or attenuation formula and the above-mentioned attenuation-related parameters.

[0196] In some optional embodiments, the attenuation of the signal strength of the tactile signal may be reflected by a channel gain in the metadata of the tactile signal, that is, the channel gain field in the metadata of the tactile signal is set to the attenuation.

[0197] Optionally, the propagation distance related parameter includes an upper limit value of the propagation distance of the tactile signal, and the upper limit value of the propagation distance is used to indicate a maximum propagation distance of the tactile signal.

[0198] Optionally, the trigger condition associated with the first element is a distance condition, the second element includes a distance usage identifier, the second element includes at least one element in the first action object, and the first action object is an action object corresponding to the trigger condition associated with the first element;

[0199] The distance usage identifier is used to indicate whether the electronic device triggers tactile feedback according to an element located within a distance range defined by the distance condition.

[0200] Exemplarily, the element associated with the action object can carry a distance usage flag (useProximity). If the value of useProximity is true, the actuator corresponding to the element within the distance specified by the distance condition corresponding to the action object can play a tactile signal, for example, as shown in Table 12.

[0201] Table 12 Haptic action node attributes

[0202]

[0203] In some optional embodiments, the actuator information (eg, body part mask, body part target, actuator target) of the metadata of the haptic signal may be set to the actuator information corresponding to the element satisfying the distance condition.

[0204] Optionally, the trigger condition associated with the first element is a collision condition, the element list corresponding to the collision condition includes at least two elements, each of the at least two elements is associated with a type identifier or a first identifier, and the first identifier associated with each of the elements is the identifier of the upper-layer element of each of the elements;

[0205] Wherein, it is determined that the collision condition is satisfied when elements with different associated type identifiers or first identifiers in the at least two elements collide.

[0206] The type identifier associated with the above-mentioned element can be used to identify the type of the element. For example, the type of the above-mentioned element may include a tactile source element and a user element (i.e., an avatar element). The type identifier associated with the above-mentioned element may include a tactile source element identifier and a user element identifier. In this way, when different types of elements collide, it is determined that the collision condition is satisfied. For example, when a tactile source element and a user element collide, it is determined that the collision condition is satisfied, and when elements of the same type collide, it can be determined that the collision condition is not satisfied.

[0207] The first identifier associated with the above element is the identifier of the upper element of the element. For example, for a node, the identifier of the upper element associated with it can be the identifier of its parent node. In this way, when elements with different upper elements collide, it is determined that the collision condition is met. For example, when a node whose parent node is parentnode1 collides with a node whose parent node is parentnode2, it is determined that the collision condition is met, and when elements under the same upper element collide, it can be determined that the collision condition is not met. For example, as shown in Table 13 and Table 14.

[0208] Table 13 Node parent node attributes

[0209]

[0210] Table 14 Meaning of triggers

[0211]

[0212] Optionally, the trigger condition associated with the first element is a collision condition, and the collision condition corresponds to at least two element groups, each of which includes at least one element, wherein the collision condition is determined to be satisfied when elements between different element groups in the at least two element groups collide.

[0213] Exemplarily, the at least two element groups may include a tactile source element group and a user element group (i.e., an avatar element group), wherein the tactile source element group includes at least one tactile source element, and the user element group includes at least one user element, so that when elements of different element groups collide, it is determined that the collision condition is satisfied. For example, when an element of the tactile source element group collides with an element of the user element group, it is determined that the collision condition is satisfied; otherwise, it can be determined that the collision condition is not satisfied.

[0214] Optionally, the trigger condition associated with the first element is a collision condition, the collision condition corresponds to at least two elements, and the collision condition is determined to be satisfied when a collision occurs between lower-layer elements of different elements in the at least two elements.

[0215] Exemplarily, when the at least two elements are parent nodes, the lower-level elements of the at least two elements may be nodes or child nodes; when the at least two elements are nodes, the lower-level elements of the at least two elements may be child nodes.

[0216] It should be noted that in the interactivity model, collisions are defined at the node level. A node list is carried in a collision trigger. If the nodes in the node list collide, the collision condition is deemed to be met. However, for high-precision collision determination, user nodes or avatar nodes are virtualized into countless tiny basic geometric models, and haptic source nodes are virtualized into countless tiny basic geometric models. From the perspective of program implementation, the collision between tiny basic set models requires the repeated definition of countless trigger objects. Each trigger object contains the basic geometric model of the user node and the basic geometric model of the haptic source node. This implementation method is relatively complex. Based on this, this embodiment sets the element list corresponding to the collision condition to include at least two elements, each of the at least two elements is associated with a type identifier or a first identifier, and determines that the collision condition is satisfied when elements with different type identifiers or first identifiers associated with the at least two elements collide; or, sets the collision condition to correspond to at least two element groups, and determines that the collision condition is satisfied when elements between different element groups in the at least two element groups collide; or sets the collision condition to correspond to at least two elements, and determines that the collision condition is satisfied when a collision occurs between lower-level elements of different elements in the at least two elements, which can simplify the implementation complexity of collision condition judgment.

[0217] Optionally, the electronic device performs a first operation according to the tactile signal and metadata of the tactile signal, including:

[0218] The electronic device controls the playing of the haptic signal according to the metadata of the haptic signal.

[0219] Exemplarily, the electronic device may be an XR device. The electronic device controls the playback of the tactile signal according to the metadata of the tactile signal. For example, when the metadata of the tactile signal includes actuator information associated with the tactile signal, the electronic device may determine the actuator that plays the tactile signal according to the actuator information associated with the tactile signal, and then execute the tactile signal based on the determined actuator; or, when the metadata of the tactile signal includes the attenuation of the signal strength of the tactile signal, the signal strength of the tactile signal played may be controlled based on the attenuation of the signal strength of the tactile signal, which is conducive to improving the effect of tactile feedback.

[0220] It should be noted that the inventive concept provided in the embodiments of the present application can be applied to electronic devices to determine tactile signals, and can also be applied to electronic devices to determine visual signals, audio signals and other signals.

[0221] It should be noted that the tactile signal processing method provided in the embodiment of the present application can be executed by a tactile signal processing device, or a control module in the tactile signal processing device for executing the tactile signal processing method. In the embodiment of the present application, the tactile signal processing device provided in the embodiment of the present application is described by taking the tactile signal processing device executing the tactile signal processing method as an example.

[0222] See also Figure 2 , Figure 2 is a structural diagram of a tactile signal processing device provided in an embodiment of the present application, such as Figure 2 As shown, the tactile signal processing device 200 includes:

[0223] A first determining module 201, configured to determine a tactile signal and metadata of the tactile signal when a first element triggers tactile feedback;

[0224] An execution module 202, configured to execute a first operation according to the haptic signal and metadata of the haptic signal;

[0225] The first element includes at least one of a node and a graph element, and the metadata of the tactile signal includes at least one of the following: actuator information associated with the tactile signal, avatar information associated with the tactile signal, a perception modality associated with the tactile signal, and an attenuation amount of the signal strength of the tactile signal.

[0226] Optionally, the actuator information associated with the tactile signal includes actuator information associated with the target element;

[0227] Among them, the target element includes the first element or the second element, the second element is the element associated with the first action object, the first action object is the action object corresponding to the trigger condition associated with the first element, and the second element includes at least one of a node and a graph element.

[0228] Optionally, the actuator information associated with the target element includes the following: body part target, actuator target, actuator resolution;

[0229] or,

[0230] The actuator information associated with the target element includes a device identifier, and the device identifier is used to identify the actuator associated with the target element.

[0231] Optionally, the avatar information associated with the tactile signal includes avatar information associated with the target element;

[0232] or,

[0233] The avatar information associated with the tactile signal includes avatar information associated with the tactile scene corresponding to the target element;

[0234] or,

[0235] The avatar information associated with the tactile signal includes the avatar information associated with the upper layer element of the target element;

[0236] Among them, the target element includes the first element or the second element, the second element is the element associated with the first action object, the first action object is the action object corresponding to the trigger condition associated with the first element, and the second element includes at least one of a node and a graph element.

[0237] Optionally, the target element is an element in an activated state;

[0238] or,

[0239] The avatar information associated with the tactile signal includes the avatar information of the element in the activated state associated with the tactile scene corresponding to the target element;

[0240] or,

[0241] The avatar information associated with the tactile signal includes the avatar information of an activated element associated with an upper-layer element of the target element.

[0242] Optionally, in the case that the first element includes N elements in an activated state, the number of the tactile signals is N, wherein N is a positive integer, the N tactile signals correspond one-to-one to the N activated elements, and the avatar information associated with each of the tactile signals is the avatar information of the activated element corresponding to each of the tactile signals.

[0243] Optionally, the element being in an activated state includes at least one of the following:

[0244] The element is in a first activation state, and the first activation state is used to indicate that the element can be applied to a tactile scene;

[0245] The element is in a second activation state, and the second activation state is used to indicate that the element can be used to trigger tactile feedback;

[0246] The element is in a third activation state, and the third activation state is used to indicate that the element can be applied to perform a tactile feedback action.

[0247] Optionally, the state of the element is set via a second action object, and the second action object is used to activate or deactivate the element.

[0248] Optionally, the perceptual modality associated with the tactile signal includes a perceptual modality associated with the target element;

[0249] or,

[0250] The perceptual modality associated with the tactile signal includes the perceptual modality associated with the tactile scene corresponding to the target element;

[0251] or,

[0252] The perceptual modality associated with the tactile signal includes the perceptual modality associated with the upper layer element of the target element;

[0253] Among them, the target element includes the first element or the second element, the second element is an element associated with the first action object, the first action object is an action object corresponding to the trigger condition associated with the first element, and the second element includes at least one of a node and a graph element.

[0254] Optionally, the perceptual modality associated with the tactile signal includes an activated perceptual modality associated with the target element;

[0255] or,

[0256] The perceptual modality associated with the tactile signal includes the perceptual modality in an activated state associated with the tactile scene corresponding to the target element;

[0257] or,

[0258] The perceptual modality associated with the tactile signal includes an activated perceptual modality associated with an upper-layer element of the target element.

[0259] Optionally, when there are M perceptual modalities in an activated state among the perceptual modalities associated with the target element, the number of the tactile signals is M, wherein M is a positive integer, the M tactile signals correspond one-to-one to the M perceptual modalities in an activated state, and the perceptual modality associated with each of the tactile signals is the perceptual modality in an activated state corresponding to each of the tactile signals.

[0260] Optionally, the state of the perception modality is set via a third action object, and the third action object is used to activate or deactivate the perception modality.

[0261] Optionally, the device further comprises:

[0262] A second determination module is configured to determine a state of a first behavior object before determining a tactile signal and metadata of the tactile signal when the first element triggers tactile feedback, wherein the first behavior object includes a first trigger condition and a first action object, and the first trigger condition is a trigger condition associated with the first element;

[0263] A third determination module is configured to determine whether to apply or not to apply the first behavior object according to a state of the first behavior object.

[0264] Optionally, the state of the first behavior object is determined according to a state identifier associated with the first behavior object, wherein the state identifier is used to identify an activation state or a deactivation state;

[0265] or,

[0266] The state of the first behavior object is determined according to at least one of a state of the first trigger condition and a state of the first action object.

[0267] Optionally, the state of the first trigger condition is determined according to a state identifier associated with the first trigger condition, or according to a state identifier associated with an element associated with the first trigger condition;

[0268] or,

[0269] The state of the first action object is determined according to a state identifier of the first action object, or according to a state identifier associated with an element associated with the first action object;

[0270] The state identifier is used to identify an activation state or a deactivation state.

[0271] Optionally, the state of the first action object is set by a fourth action object, and the fourth action object is used to activate or deactivate the behavior object;

[0272] or,

[0273] The state of the first trigger condition is set by a fifth action object, and the fifth action object is used to activate or deactivate the trigger condition;

[0274] or,

[0275] The state of the first action object is set according to a sixth action object, and the sixth action object is used to activate or deactivate the action object;

[0276] or,

[0277] The state of the element is set according to a seventh action object, and the seventh action object is used to activate or deactivate the element.

[0278] Optionally, the state of the first behavior object is determined according to the state of the perception modality corresponding to the first behavior object;

[0279] or,

[0280] The state of the first behavior object is determined based on the perception modality corresponding to the first trigger condition and the perception modality corresponding to the first action object, or the state of the first behavior object is determined based on the perception modality in an activated state corresponding to the first trigger condition and the perception modality in an activated state corresponding to the first action object.

[0281] Optionally, the perception modality corresponding to the first behavior object is the perception modality corresponding to the tactile scene corresponding to the first behavior object.

[0282] Optionally, the state of the perception modality corresponding to the first behavior object is determined according to a state identifier associated with the perception modality corresponding to the first behavior object.

[0283] Optionally, when all perception modalities corresponding to the first behavior object are in a deactivated state, the first behavior object is in a deactivated state.

[0284] Optionally, when the perception modality corresponding to the first trigger condition and the perception modality corresponding to the first action object are the same, the first action object is in an activated state, otherwise the first action object is in a deactivated state;

[0285] or,

[0286] When the perception modality in an activated state corresponding to the first trigger condition is the same as the perception modality in an activated state corresponding to the first action object, the first behavior object is in an activated state; otherwise, the first behavior object is in a deactivated state.

[0287] Optionally, the state of the perception modality corresponding to the first trigger condition is determined according to a state identifier of the perception modality corresponding to the first trigger condition;

[0288] or,

[0289] The state of the perception modality corresponding to the first trigger condition is determined according to the state identifier of the perception modality corresponding to the element corresponding to the first trigger condition.

[0290] Optionally, the third element is associated with at least one of the following: an attenuation-related parameter of the tactile signal, a propagation distance-related parameter of the tactile signal;

[0291] The third element includes at least one element of a tactile source or at least one element of a tactile scene.

[0292] Optionally, the attenuation-related parameter includes an attenuation indicator, and the attenuation indicator is used to indicate that the signal strength of the tactile signal attenuates with distance.

[0293] Optionally, there is a correlation between the attenuation-related parameter and the perception modality.

[0294] Optionally, the attenuation amount of the signal strength of the tactile signal is determined based on an attenuation model or an attenuation formula, wherein the attenuation model or the attenuation formula is agreed upon by a protocol.

[0295] Optionally, the propagation distance related parameter includes an upper limit value of the propagation distance of the tactile signal, and the upper limit value of the propagation distance is used to indicate a maximum propagation distance of the tactile signal.

[0296] Optionally, the trigger condition associated with the first element is a distance condition, the second element includes a distance usage identifier, the second element includes at least one element in the first action object, and the first action object is an action object corresponding to the trigger condition associated with the first element;

[0297] The distance usage identifier is used to indicate whether the electronic device triggers tactile feedback according to an element located within a distance range defined by the distance condition.

[0298] Optionally, the trigger condition associated with the first element is a collision condition, the element list corresponding to the collision condition includes at least two elements, each of the at least two elements is associated with a type identifier or a first identifier, and the first identifier associated with each of the elements is the identifier of the upper-layer element of each of the elements;

[0299] Wherein, it is determined that the collision condition is satisfied when elements with different associated type identifiers or first identifiers in the at least two elements collide.

[0300] Optionally, the trigger condition associated with the first element is a collision condition, and the collision condition corresponds to at least two element groups, each of which includes at least one element, wherein the collision condition is determined to be satisfied when elements between different element groups in the at least two element groups collide.

[0301] Optionally, the trigger condition associated with the first element is a collision condition, the collision condition corresponds to at least two elements, and the collision condition is determined to be satisfied when a collision occurs between lower-layer elements of different elements in the at least two elements.

[0302] Optionally, the execution module is specifically used to:

[0303] Playing of the haptic signal is controlled according to the metadata of the haptic signal.

[0304] The tactile signal processing device in the embodiment of the present application may be an electronic device, such as an electronic device with an operating system, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device may be an XR device, a terminal, a server, or a computer.

[0305] The tactile signal processing device provided in the embodiment of the present application can achieve Figure 1 The various processes implemented by the method embodiment and achieving the same technical effect are not described here to avoid repetition.

[0306] Optional, such as Figure 3 As shown, an embodiment of the present application further provides an electronic device 300, including a processor 301 and a memory 302, wherein the memory 302 stores programs or instructions that can be executed on the processor 301, and when the program or instructions are executed by the processor 301, the various steps of the above-mentioned tactile signal processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0307] An embodiment of the present application also provides an electronic device, including a processor and a communication interface, wherein the processor is used to determine a tactile signal and metadata of the tactile signal when a first element triggers tactile feedback; and perform a first operation according to the tactile signal and the metadata of the tactile signal; wherein the first element includes at least one of a node and a graph element, and the metadata of the tactile signal includes at least one of the following: actuator information associated with the tactile signal, avatar information associated with the tactile signal, perception mode associated with the tactile signal, and attenuation of the signal strength of the tactile signal. This electronic device embodiment corresponds to the above-mentioned electronic device side method embodiment, and each implementation process and implementation method of the above-mentioned method embodiment can be applied to this electronic device embodiment, and can achieve the same technical effect. Specifically, Figure 4 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0308] The electronic device 400 includes but is not limited to: a radio frequency unit 401, a network module 402, an audio output unit 403, an input unit 404, a sensor 405, a display unit 406, a user input unit 407, an interface unit 408, a memory 409 and at least some of the components of a processor 410.

[0309] Those skilled in the art will appreciate that the electronic device 400 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 410 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 4The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0310] It should be understood that in the embodiment of the present application, the input unit 404 may include a graphics processing unit (GPU) 4041 and a microphone 4042, and the graphics processor 4041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 406 may include a display panel 4061, and the display panel 4061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 407 includes a touch panel 4071 and at least one of other input devices 4072. The touch panel 4071 is also called a touch screen. The touch panel 4071 may include two parts: a touch detection device and a touch controller. Other input devices 4072 may include, but are not limited to, a physical keyboard, function keys (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0311] In the embodiment of the present application, after receiving downlink data from the network side device, the RF unit 401 can transmit the data to the processor 410 for processing; in addition, the RF unit 401 can send uplink data to the network side device. Generally, the RF unit 401 includes but is not limited to an antenna, an amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc.

[0312] The memory 409 can be used to store software programs or instructions and various data. The memory 409 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instruction required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 409 may include a volatile memory or a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 409 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0313] The processor 410 may include one or more processing units; optionally, the processor 410 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 410.

[0314] Among them, the processor 410 is used to determine the tactile signal and the metadata of the tactile signal when the first element triggers tactile feedback; perform a first operation according to the tactile signal and the metadata of the tactile signal; wherein the first element includes at least one of a node and a graph element, and the metadata of the tactile signal includes at least one of the following: actuator information associated with the tactile signal, avatar information associated with the tactile signal, perception modality associated with the tactile signal, and attenuation of the signal strength of the tactile signal.

[0315] It can be understood that the implementation process of each implementation method mentioned in this embodiment can refer to the relevant description of the aforementioned method embodiment, and achieve the same or corresponding technical effect. To avoid repetition, it will not be repeated here.

[0316] An embodiment of the present application further provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned tactile signal processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0317] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk. In some examples, the readable storage medium may be a non-transient readable storage medium.

[0318] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned tactile signal processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0319] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0320] The embodiment of the present application further provides a computer program / program product, which is stored in a storage medium, and is executed by at least one processor to implement the various processes of the above-mentioned tactile signal processing method embodiment, and can achieve the same technical effect. To avoid repetition, it is not repeated here.

[0321] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0322] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of a computer software product plus a necessary general hardware platform, and of course, can also be implemented by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, disk, CD, etc.), including several instructions to enable a terminal or a network-side device to execute the methods described in each embodiment of the present application.

[0323] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms of implementation methods without departing from the purpose of the present application and the scope of protection of the claims, and these implementation methods are all within the protection of the present application.

Claims

1. A tactile signal processing method, characterized in that: include: In the case where the first element triggers haptic feedback, the electronic device determines a haptic signal and metadata of the haptic signal; The electronic device performs a first operation according to the tactile signal and metadata of the tactile signal; The first element includes at least one of a node and a graph element, and the metadata of the tactile signal includes at least one of the following: actuator information associated with the tactile signal, avatar information associated with the tactile signal, a perception modality associated with the tactile signal, and an attenuation amount of the signal strength of the tactile signal.

2. The method according to claim 1, characterized in that The actuator information associated with the tactile signal includes actuator information associated with the target element; Among them, the target element includes the first element or the second element, the second element is the element associated with the first action object, the first action object is the action object corresponding to the trigger condition associated with the first element, and the second element includes at least one of a node and a graph element.

3. The method according to claim 2, characterized in that The actuator information associated with the target element includes the following: body part target, actuator target, actuator resolution; or, The actuator information associated with the target element includes a device identifier, and the device identifier is used to identify the actuator associated with the target element.

4. The method according to any one of claims 1 to 3, characterized in that The avatar information associated with the tactile signal includes avatar information associated with the target element; or, The avatar information associated with the tactile signal includes avatar information associated with the tactile scene corresponding to the target element; or, The avatar information associated with the tactile signal includes the avatar information associated with the upper layer element of the target element; Among them, the target element includes the first element or the second element, the second element is the element associated with the first action object, the first action object is the action object corresponding to the trigger condition associated with the first element, and the second element includes at least one of a node and a graph element.

5. The method according to claim 4, characterized in that The target element is an element in an activated state; or, The avatar information associated with the tactile signal includes the avatar information of the element in the activated state associated with the tactile scene corresponding to the target element; or, The avatar information associated with the tactile signal includes the avatar information of an activated element associated with an upper-layer element of the target element.

6. The method according to any one of claims 1 to 5, characterized in that In the case where the first element includes N elements in an activated state, the number of the tactile signals is N, where N is a positive integer, the N tactile signals correspond one-to-one to the N activated elements, and the avatar information associated with each of the tactile signals is the avatar information of the activated element corresponding to each of the tactile signals.

7. The method according to claim 5 or 6, characterized in that: The element being in an activated state includes at least one of the following: The element is in a first activation state, and the first activation state is used to indicate that the element can be applied to a tactile scene; The element is in a second activation state, and the second activation state is used to indicate that the element can be used to trigger tactile feedback; The element is in a third activation state, and the third activation state is used to indicate that the element can be applied to perform a tactile feedback action.

8. The method according to any one of claims 5 to 7, characterized in that The state of the element is set by a second action object, and the second action object is used to activate or deactivate the element.

9. The method according to any one of claims 1 to 8, characterized in that The perceptual modality associated with the tactile signal includes a perceptual modality associated with the target element; or, The perceptual modality associated with the tactile signal includes the perceptual modality associated with the tactile scene corresponding to the target element; or, The perceptual modality associated with the tactile signal includes the perceptual modality associated with the upper layer element of the target element; Among them, the target element includes the first element or the second element, the second element is an element associated with the first action object, the first action object is an action object corresponding to the trigger condition associated with the first element, and the second element includes at least one of a node and a graph element.

10. The method according to claim 9, characterized in that The perceptual modality associated with the tactile signal includes the perceptual modality associated with the target element in an activated state; or, The perceptual modality associated with the tactile signal includes the perceptual modality in an activated state associated with the tactile scene corresponding to the target element; or, The perceptual modality associated with the tactile signal includes an activated perceptual modality associated with an upper-layer element of the target element.

11. The method according to any one of claims 1 to 10, characterized in that When there are M perceptual modalities in an activated state among the perceptual modalities associated with the target element, the number of the tactile signals is M, where M is a positive integer, the M tactile signals correspond one-to-one to the M perceptual modalities in an activated state, and the perceptual modality associated with each of the tactile signals is the perceptual modality in an activated state corresponding to each of the tactile signals.

12. The method according to claim 10 or 11, characterized in that: The state of the perception modality is set via a third action object, and the third action object is used to activate or deactivate the perception modality.

13. The method according to any one of claims 1 to 12, characterized in that In the case where the first element triggers tactile feedback, before the electronic device determines the tactile signal and metadata of the tactile signal, the method further includes: The electronic device determines a state of a first behavior object, wherein the first behavior object includes a first trigger condition and a first action object, and the first trigger condition is a trigger condition associated with the first element; The electronic device determines whether to apply or not to apply the first behavior object according to the state of the first behavior object.

14. The method according to claim 13, characterized in that The state of the first behavior object is determined according to a state identifier associated with the first behavior object, wherein the state identifier is used to identify an activation state or a deactivation state; or, The state of the first behavior object is determined according to at least one of a state of the first trigger condition and a state of the first action object.

15. The method according to claim 14, characterized in that The state of the first trigger condition is determined according to a state identifier associated with the first trigger condition, or according to a state identifier associated with an element associated with the first trigger condition; or, The state of the first action object is determined according to a state identifier of the first action object, or according to a state identifier associated with an element associated with the first action object; The state identifier is used to identify an activation state or a deactivation state.

16. The method according to claim 14 or 15, characterized in that The state of the first action object is set by a fourth action object, and the fourth action object is used to activate or deactivate the behavior object; or, The state of the first trigger condition is set by a fifth action object, and the fifth action object is used to activate or deactivate the trigger condition; or, The state of the first action object is set according to a sixth action object, and the sixth action object is used to activate or deactivate the action object; or, The state of the element is set according to a seventh action object, and the seventh action object is used to activate or deactivate the element.

17. The method according to claim 13, characterized in that The state of the first behavior object is determined according to the state of the perception modality corresponding to the first behavior object; or, The state of the first behavior object is determined based on the perception modality corresponding to the first trigger condition and the perception modality corresponding to the first action object, or the state of the first behavior object is determined based on the perception modality in an activated state corresponding to the first trigger condition and the perception modality in an activated state corresponding to the first action object.

18. The method according to claim 17, characterized in that The perception modality corresponding to the first behavior object is the perception modality corresponding to the tactile scene corresponding to the first behavior object.

19. The method according to claim 17 or 18, characterized in that The state of the perception modality corresponding to the first behavior object is determined according to a state identifier associated with the perception modality corresponding to the first behavior object.

20. The method according to any one of claims 17 to 19, characterized in that When all the perception modalities corresponding to the first behavior object are in a deactivated state, the first behavior object is in a deactivated state.

21. The method according to claim 17, characterized in that When the perception modality corresponding to the first trigger condition and the perception modality corresponding to the first action object are the same, the first action object is in an activated state; otherwise, the first action object is in a deactivated state; or, When the perception modality in an activated state corresponding to the first trigger condition is the same as the perception modality in an activated state corresponding to the first action object, the first behavior object is in an activated state; otherwise, the first behavior object is in a deactivated state.

22. The method according to claim 21, characterized in that The state of the perception modality corresponding to the first trigger condition is determined according to the state identifier of the perception modality corresponding to the first trigger condition; or, The state of the perception modality corresponding to the first trigger condition is determined according to the state identifier of the perception modality corresponding to the element corresponding to the first trigger condition.

23. The method according to any one of claims 1 to 22, characterized in that The third element is associated with at least one of the following: a parameter related to the attenuation of the tactile signal, a parameter related to the propagation distance of the tactile signal; The third element includes at least one element of a tactile source or at least one element of a tactile scene.

24. The method according to claim 23, characterized in that The attenuation-related parameter includes an attenuation indicator, and the attenuation indicator is used to indicate that the signal strength of the tactile signal attenuates with distance.

25. The method according to claim 23 or 24, characterized in that There is a correlation between the attenuation-related parameters and the perception modality.

26. The method according to any one of claims 23 to 25, characterized in that The attenuation amount of the signal strength of the tactile signal is determined based on an attenuation model or an attenuation formula, wherein the attenuation model or the attenuation formula is agreed upon by a protocol.

27. The method according to any one of claims 23 to 26, characterized in that The propagation distance related parameters include an upper limit value of the propagation distance of the tactile signal, and the upper limit value of the propagation distance is used to indicate a maximum propagation distance of the tactile signal.

28. The method according to any one of claims 1 to 27, characterized in that The trigger condition associated with the first element is a distance condition, the second element includes a distance usage identifier, the second element includes at least one element in the first action object, and the first action object is an action object corresponding to the trigger condition associated with the first element; The distance usage identifier is used to indicate whether the electronic device triggers tactile feedback according to an element located within a distance range defined by the distance condition.

29. The method according to any one of claims 1 to 28, characterized in that The trigger condition associated with the first element is a collision condition, the element list corresponding to the collision condition includes at least two elements, each of the at least two elements is associated with a type identifier or a first identifier, and the first identifier associated with each of the elements is the identifier of the upper element of each of the elements; Wherein, it is determined that the collision condition is satisfied when elements with different associated type identifiers or first identifiers in the at least two elements collide.

30. The method according to any one of claims 1 to 28, characterized in that The trigger condition associated with the first element is a collision condition, and the collision condition corresponds to at least two element groups, each of which includes at least one element, wherein the collision condition is determined to be satisfied when elements between different element groups in the at least two element groups collide.

31. The method according to any one of claims 1 to 28, characterized in that The trigger condition associated with the first element is a collision condition, the collision condition corresponds to at least two elements, and the collision condition is determined to be satisfied when a collision occurs between lower-layer elements of different elements in the at least two elements.

32. The method according to any one of claims 1 to 31, characterized in that The electronic device performs a first operation according to the tactile signal and metadata of the tactile signal, including: The electronic device controls the playing of the haptic signal according to the metadata of the haptic signal.

33. A tactile signal processing device, characterized in that: include: A first determining module, configured to determine a tactile signal and metadata of the tactile signal when the first element triggers tactile feedback; an execution module, configured to execute a first operation according to the tactile signal and metadata of the tactile signal; The first element includes at least one of a node and a graph element, and the metadata of the tactile signal includes at least one of the following: actuator information associated with the tactile signal, avatar information associated with the tactile signal, a perception modality associated with the tactile signal, and an attenuation amount of the signal strength of the tactile signal.

34. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the tactile signal processing method according to any one of claims 1 to 32 are implemented.

35. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the tactile signal processing method according to any one of claims 1 to 32 are implemented.

Images