Processing method and device, equipment and readable storage medium

Abstract

The invention discloses a processing method, device and equipment and a readable storage medium, and the method comprises the steps that a rendering end carries out the rendering processing or scene synthesis of touch data according to first information associated with a first object; or, the rendering end generates second information associated with the first object, and the second information is used for controlling the first equipment.

Description

Technical Field

[0001] This application belongs to the field of communication technologies, and particularly relates to a processing method, apparatus, device, and readable storage medium. Background Art

[0002] Currently, the scene rendering based on the Graphics Language Transmission Format (glTF) lacks the description of hardware information, which may cause the mismatch between the composite rendering of the scene and the operation of external devices, affecting the working performance and user experience of external devices. Summary of the Invention

[0003] Embodiments of this application provide a processing method, apparatus, device, and readable storage medium to solve the problem that the mismatch between the composite rendering of the scene and the operation of external devices affects the working performance and user experience of external devices.

[0004] In a first aspect, a processing method is provided, including:

[0005] The rendering end renders or composes a scene for haptic data according to the first information associated with the first object;

[0006] Alternatively, the rendering end generates second information associated with the first object, and the second information is used to control the first device.

[0007] In a second aspect, a processing method is provided, including:

[0008] The first device sends the first information associated with the first object to the rendering end, and the first information is used to render or compose a scene for haptic data;

[0009] Or,

[0010] The first device receives the second information associated with the first object sent by the rendering end;

[0011] The first device performs corresponding control operations according to the second information.

[0012] In a third aspect, a processing apparatus is provided, including:

[0013] A first processing unit, configured to render or compose a scene for haptic data according to the first information associated with the first object; or configured to generate second information associated with the first object, and the second information is used to control the first device.

[0014] In a fourth aspect, a processing apparatus is provided, including:

[0015] The second transceiver unit is configured to send the first information associated with the first object to the rendering end, where the first information is used for rendering tactile data or synthesizing a scene; or, the second transceiver unit is configured to receive the second information associated with the first object sent by the rendering end; the second processing unit is configured to perform corresponding control operations according to the second information.

[0016] In a fifth aspect, an electronic device is provided, including: a processor, a memory, and a program or instruction stored on the memory and executable on the processor, where when the program or instruction is executed by the processor, the steps of the method described in the first aspect or the second aspect are implemented.

[0017] In a sixth aspect, a readable storage medium is provided, where a program or instruction is stored on the readable storage medium, and when the program or instruction is executed by a processor of a terminal, the steps of the method described in the first aspect or the second aspect are implemented.

[0018] In a seventh aspect, a chip is provided, where the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is configured to run a program or instruction to implement the steps of the method described in the first aspect or the second aspect.

[0019] In an eighth aspect, a computer program / program product is provided, where the computer program / program product is stored in a non-transitory storage medium, and the program / program product is executed by at least one processor to implement the steps of the method described in the first aspect or the second aspect.

[0020] In the embodiments of the present application, the rendering end performs rendering processing or synthesizes a scene on tactile data according to the first information associated with the first object, so that the rendering processing or scene synthesis of tactile data performed by the rendering end can match the operating conditions of the first device; or, the rendering end generates the second information associated with the first object, where the second information is used to control the first device, so that the rendering end can control the state of the first device, making the rendering processing or scene synthesis of tactile data performed by the rendering end match the operating conditions of the first device and ensuring the user experience. Description of the Drawings

[0021] Figure 1 is a schematic diagram of accessing tactile data in the embodiments of the present application;

[0022] Figure 2 is one of the schematic flowcharts of the processing method in the embodiments of the present application;

[0023] Figure 3 is the second of the schematic flowcharts of the processing method in the embodiments of the present application;

[0024] Figure 4It is one of the schematic diagrams of the modules of the processing device according to the embodiments of the present application;

[0025] Figure 5 It is the second of the schematic diagrams of the modules of the processing device according to the embodiments of the present application;

[0026] Figure 6 It is the structural block diagram of the electronic device provided by the embodiments of the present application;

[0027] Figure 7 It is the structural block diagram of the terminal provided by the embodiments of the present application. Detailed implementation manners

[0028] Next, the technical solutions in the embodiments of the present application will be clearly described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art shall fall within the protection scope of the present application.

[0029] The terms "first", "second", etc. in the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such terms can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same category, and the number of objects is not limited. For example, the first object can be one or multiple. In addition, "or" in the present application means at least one of the connected objects. For example, "A or B" covers three scenarios, namely, Scenario 1: including A and not including B; Scenario 2: including B and not including A; Scenario 3: including both A and B. The character " / " generally indicates an "or" relationship between the associated objects before and after.

[0030] It should be noted that the technology described in the embodiments of this application is not limited to Long Term Evolution (LTE) / LTE-Advanced (LTE-A) systems, and can also be used in other wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency Division Multiple Access (SC-FDMA), or other systems. The terms "system" and "network" in this application are often used interchangeably, and the described technology can be used in the above-mentioned systems and radio technologies, as well as in other systems and radio technologies. The following description describes the New Radio (NR) system for example purposes, and the NR term is used in most of the following descriptions, but these technologies can also be applied to systems other than NR system applications, such as the 6th Generation (6G) communication system. th Generation, 6G) communication system.

[0031] To enable those skilled in the art to better understand the embodiments of this application, the following explanations are provided first.

[0032] 0. Regarding the process of scene synthesis or rendering.

[0033] The rendering engine can synthesize a scene according to a glTF (3D model file format) file or render the scene according to the glTF file. By rendering the haptic scene, the rendering engine can obtain the feedback haptic data. After operations such as encoding, transmission, or decoding of the haptic data, it is played by the haptic data actuator.

[0034] The ways for the rendering engine to obtain the glTF file include: the server and the client interact with the glTF file through glTF status stream transmission, or obtain the glTF file through the program files stored on the server and the client.

[0035] 1. The Moving Pictures Experts Group (MPEG)_haptic extension in glTF.

[0036] The MPEG haptic extension provides an access method for haptic data. MPEG_haptic includes a set of haptic objects. The accessor attribute in the hapticObjects then points to the haptic data source.

[0037] Table 1: MPEG_haptic.

[0038]

[0039] Table 2: hapticObject

[0040]

[0041] The specific access method for haptic data is as Figure 1 shown. The hapticObjects in MPEG_haptic point to the accessor, and the accessor then points to the circular buffer object. The circular buffer object provides the data position information of the circular buffer on the one hand, and on the other hand, reflects the relevant information of the haptic data rendering by referring to the MPEG media object.

[0042] 2. Regarding the MPEG haptic material (MPEG_haptic_material) extension in glTF.

[0043] The MPEG_haptic_material extension provides haptic properties for the material. MPEG_haptic_material contains the materials attribute.

[0044] Table 3: MPEG_haptic_material.

[0045]

[0046] The materials attribute points to a set of material objects. Each material object points to the hapticObject in MPEG_haptic that provides the data source through the haptic attribute, and indicates the relevant material haptic properties (such as stiffness, friction, etc.) and texture coordinate related information.

[0047] Table 4: Material properties of MPEG_haptic_material.

[0048]

[0049]

[0050] 3. Regarding the Interactivity extension in glTF.

[0051] glTF is used for efficient transmission and loading of 3D scenes and models in engines and applications. glTF defines an extensible publishing format that simplifies the creation workflow and interactive services by enabling interoperable use of 3D content across the industry.

[0052] MPEG has extended glTF interactively and preliminarily designed relevant standards for interactivity. For haptic scenarios, the relevant standards for interactivity are mainly used to describe the interaction between an avatar and a haptic source, based on which haptic data is triggered and generated.

[0053] Interactivity is defined based on behavior, which includes the trigger object and the action object. Among them, the trigger object includes collision conditions, proximity conditions, and visibility conditions. Among them, the action object includes haptic feedback actions (i.e., ACTION_SET_HAPTIC), etc., and ACTION_SET_HAPTIC is implemented based on the HapticActionNode object or the HapticActionMedia object.

[0054] The hapticObject property in HapticActionNode points to the hapticObject in MPEG_haptic, and the mediaIndex property in the HapticActionMedia (haptic action media) further points to the accessor in the above hapticObject to access haptic data.

[0055] Table 5: Haptic action node.

[0056]

[0057]

[0058] Table 6: HapticActionMedia

[0059]

[0060] The objects in the embodiments of the present application include at least one of the following: scene, node, mesh, material, vertices, face, primitive, etc. The following will, with reference to the accompanying drawings, through some embodiments and their application scenarios, elaborate in detail on the data processing method and related devices provided in the embodiments of the present application.

[0061] See Figure 2 , the embodiments of the present application provide a processing method, which is applied to a rendering end. The rendering end includes at least one of the following: rendering engine, display engine, rendering device, scene synthesis device, scene synthesis engine (CompositionEngine), presentation engine (Presentation Engine), presentation device, etc. The specific steps include: Step 201.

[0062] Step 201: The rendering end renders and processes the haptic data or synthesizes a scene according to the first information associated with the first object; or, the rendering end generates the second information associated with the first object, and the second information is used to control the first device.

[0063] In this embodiment, the first object is used to instantiate the first device.

[0064] In this embodiment, synthesizing a scene means simulating or creating a virtual environment through related technologies, and the synthesized scene includes, but is not limited to, updating a 3D model.

[0065] In this embodiment, the second information can be mainly used to control the state of the first device, so that the rendering and processing of the haptic data or the synthesis of the scene performed by the rendering end can match the operation of the first device.

[0066] In this embodiment, the first object includes at least one of the following: haptic actuator-related object, sensor-related object.

[0067] The above-mentioned haptic actuator-related object can be a code unit encapsulating an execution action. The above-mentioned sensor-related object can be a code unit encapsulating sensor-related operations.

[0068] In this embodiment, the first device includes at least one of the following: a haptic actuator, a sensor.

[0069] Optionally, the first information can be used to reflect the state of the first device, the second information can be used to control the state of the first device, the first information can be associated with one or more first objects, the second information can be associated with one or more first objects, and the haptic data rendered by the rendering end or the synthesized scene can be matched with the operating conditions of the first device through the first information or the second information.

[0070] In an implementation manner of this application, the method further includes: the rendering end determines the first information associated with the first object, that is, the information associated with the first object can be generated by the rendering end.

[0071] In an implementation manner of this application, the method further includes: the rendering end obtains the first information associated with the first object from the first device through a first Application Programming Interface (API).

[0072] In an implementation manner of this application, the method further includes: the rendering end sends the second information associated with the first object to the first device through a second API;

[0073] Wherein, the first device includes at least one of the following: a haptic actuator, a sensor.

[0074] That is, the first information or the second information can be exchanged between the first device and the rendering end through the API.

[0075] In an implementation manner of this application, when the first object is a haptic actuator-related object, the first information includes at least one of the following:

[0076] 1) The perception modality associated with the haptic actuator;

[0077] For example, vibration sense, or temperature, etc.

[0078] 2) The type of the haptic actuator;

[0079] For example, a voice coil motor, an eccentric rotor motor, a piezoelectric actuator, an electromagnetic actuator, etc.

[0080] 3) The capability information of the haptic actuator;

[0081] For example, the rendering end performs compression processing on the haptic data according to the capability information of the haptic actuator.

[0082] 4) The haptic actuator identifier;

[0083] For example, a haptic actuator identifier is used to identify a device included in an object related to a haptic actuator, and the haptic actuator identifier may be a device identifier.

[0084] 5) Location information of the haptic actuator;

[0085] 6) Mixing weight information;

[0086] Optionally, the rendering end determines the mixing weight value in the metadata of the haptic data transmitted when multiple haptic data are played on a certain haptic actuator according to the mixing weight carried in the attribute information.

[0087] In one implementation, the mixing weight information represents the numerical value of the mixing weight of a set of haptic data triggered at a position near the haptic actuator when it is mixed and played on the haptic actuator.

[0088] For example, the mixing weight information may indicate the numerical value of the mixing weight of the haptic data associated with a set of vertices near the haptic actuator, and the haptic data corresponding to this set of vertices can be mixed according to the mixing weight and played on this haptic actuator.

[0089] In another implementation, the mixing weight information may also represent the numerical value of the mixing weight of the haptic data triggered at the position of the haptic actuator when it is mixed with the haptic data associated with other positions.

[0090] For example, the mixing weight information may indicate a set of haptic actuators and the mixing weight associated with each haptic actuator, and the haptic data corresponding to this set of haptic actuators can be mixed according to the mixing weight and played on a certain haptic actuator.

[0091] 7) Channel gain information.

[0092] In one implementation, the rendering end determines the channel gain value in the metadata of the haptic data transmitted when a certain path of haptic data is mapped to multiple haptic actuator devices for playing according to the channel gain.

[0093] For example, the channel gain information represents the channel gain of the haptic data played on each haptic actuator when the haptic data triggered at the position of a certain haptic actuator is played on multiple trigger actuators.

[0094] For another example, the channel gain information indicates the channel gain associated with a group of haptic actuators and each haptic actuator. The haptic data corresponding to a certain haptic actuator in this group of haptic actuators can be mapped to other haptic actuators in the group for playback according to the channel gain.

[0095] In an embodiment of the present application, the capability information of the haptic actuator includes at least one of the following:

[0096] 1) The maximum or minimum vibration frequency of the haptic actuator;

[0097] 2) The resonant frequency of the haptic actuator;

[0098] 3) The maximum or minimum amplitude of the haptic actuator.

[0099] In an embodiment of the present application, the position information of the haptic actuator includes at least one of the following:

[0100] 1) The mesh corresponding to the mapped position of the haptic actuator;

[0101] 2) The mesh vertex (mesh.vertice) corresponding to the mapped position of the haptic actuator;

[0102] 3) The primitive corresponding to the mapped position of the haptic actuator;

[0103] 4) The node corresponding to the mapped position of the haptic actuator;

[0104] 5) The body part mask corresponding to the mapped position of the haptic actuator.

[0105] In an embodiment of the present application, when the first object is a haptic actuator-related object, the second information associated with the first object includes the control information of the haptic actuator.

[0106] Optionally, the control information of the haptic actuator includes, but is not limited to, the switch information of the haptic actuator. Through this switch information, the rendering engine can selectively turn off the haptic actuator according to the scene requirements to achieve the purpose of power saving.

[0107] In an embodiment of the present application, when the first object is a sensor-related object, the first information includes at least one of the following:

[0108] 1) The type of the sensor;

[0109] Optionally, the type of the sensor includes, but is not limited to, at least one of the following: position sensor, acceleration sensor, etc.

[0110] 2) Capability information of the sensor;

[0111] 3) Sensor identifier;

[0112] 4) Location information of the sensor.

[0113] In an implementation manner of the present application, the capability information of the sensor includes at least one of the following: information related to the update frequency of sensing data, information related to the accuracy of sensing data.

[0114] In an implementation manner of the present application, when the sensor is a position sensor, the capability information of the sensor includes at least one of the following:

[0115] 1) Information related to the positioning accuracy of sensing data;

[0116] 2) Information related to the positioning accuracy of sensing data;

[0117] 3) Information related to the positioning resolution of sensing data.

[0118] In an implementation manner of the present application, the location information of the sensor includes at least one of the following:

[0119] 1) Mesh identifier (mesh ID) corresponding to the mapped location of the sensor;

[0120] 2) Mesh vertex corresponding to the mapped location of the sensor;

[0121] 3) Primitive identifier (primitive ID) corresponding to the mapped location of the sensor;

[0122] 4) Node identifier (node ID) corresponding to the mapped location of the sensor;

[0123] 5) Body part mask corresponding to the mapped location of the sensor.

[0124] In this embodiment, the location information, type, or capability information, etc. of the haptic actuator or sensor can be obtained according to the actual situation of the haptic actuator or sensor. The haptic actuator or sensor and the rendering end exchange the above relevant information through the corresponding API. Alternatively, the location information, etc. of the haptic actuator or sensor can be virtual information assumed by the rendering end.

[0125] In an implementation manner of the present application, when the first object is a sensor-related object, the second information associated with the first object includes the control information of the sensor. Through this control information, the rendering end can control the sensor according to the scene requirements.

[0126] In an implementation manner of the present application, the control information of the sensor includes at least one of the following:

[0127] 1) Switch information of the sensor;

[0128] Optionally, the switch information of the sensor can be for each sensor or a group of sensors (such as switching all position sensors of the hand).

[0129] Optionally, the switch information of the sensor can be indicated jointly with the sensor identification or the sensor position information.

[0130] In this way, the rendering end can control the sensor according to the scene requirements

[0131] 2) Update frequency of the sensing data;

[0132] The update frequency of the sensing data refers to the frequency at which the sensor collects and provides new data.

[0133] 3) Precision of the sensing data;

[0134] For example, position precision.

[0135] 4) Latency requirement of the sensing data.

[0136] The latency requirement of the sensing data refers to the requirement for how long the data needs to be processed or delivered to the target location after the sensor collects the data.

[0137] In an embodiment of the present application, the first object can be a node-level object or a scene-level object.

[0138] In another embodiment of the present application, the first object can be referenced as attribute information by objects such as nodes or scenes.

[0139] In an embodiment of the present application, when the haptic actuator-related object is referenced by a node object or a mesh object, the haptic actuator associated with the haptic actuator-related object is associated with the node object or the mesh object, that is, the haptic actuator associated with the haptic actuator-related object is mapped to the corresponding node or mesh.

[0140] In an embodiment of the present application, when the sensor-related object is referenced by a node object or a mesh object, the sensor associated with the sensor-related object is associated with the node object or the mesh object, that is, the sensor associated with the sensor-related object is mapped to the corresponding node or mesh.

[0141] Optionally, the rendering end can accurately locate the position information of the haptic actuator or the sensor on the avatar in combination with the mapping position information indicated by the haptic actuator-related object or the sensor-related object.

[0142] In an embodiment of the present application, the rendering end renders the tactile data or synthesizes a scene according to the first information associated with the first object, so that the rendering of the tactile data or the synthesis of the scene by the rendering end can match the operating conditions of the first device; alternatively, the rendering end generates second information associated with the first object, and the second information is used to control the first device, so that the rendering end can control the state of the first device, and the rendering of the tactile data or the synthesis of the scene by the rendering end can match the operating conditions of the first device, ensuring the user experience.

[0143] See Figure 3 , an embodiment of the present application provides a processing method, which is applied to a first device, and the specific steps include: Step 301.

[0144] Step 301: The first device sends the first information associated with the first object to the rendering end, and the first information is used to render the tactile data or synthesize a scene; alternatively, the first device receives the second information associated with the first object sent by the rendering end; the first device performs corresponding control operations according to the second information.

[0145] In an embodiment of the present application, the method further includes: the first device receives the rendered tactile data or the updated scene sent by the rendering end, where the rendered tactile data is obtained by the rendering end rendering the tactile data according to the first information, and the updated scene is obtained by the rendering end synthesizing the scene according to the first information.

[0146] In an embodiment of the present application, the first device sending the first information associated with the first object to the rendering end includes:

[0147] The first device sends the first information associated with the first object to the rendering end through a first API.

[0148] In an embodiment of the present application, the first device receiving the second information associated with the first object sent by the rendering end includes:

[0149] The first device receives the second information associated with the first object sent by the rendering end through a second API.

[0150] In an embodiment of the present application, the first object includes at least one of the following: a tactile actuator-related object, a sensor-related object; alternatively, the first device includes at least one of the following: a tactile actuator, a sensor.

[0151] In an implementation of the present application, when the first object is an object related to a tactile actuator, the first information includes at least one of the following: the perception modality associated with the tactile actuator, the type of the tactile actuator, the capability information of the tactile actuator, the tactile actuator identifier, the position information of the tactile actuator, the mixing weight information, and the channel gain information.

[0152] In an implementation of the present application, the capability information of the tactile actuator includes at least one of the following: the maximum or minimum vibration frequency of the tactile actuator, the resonant frequency of the tactile actuator, and the maximum or minimum amplitude of the tactile actuator.

[0153] In an implementation of the present application, the position information of the tactile actuator includes at least one of the following: the grid corresponding to the mapped position of the tactile actuator, the grid vertex corresponding to the mapped position of the tactile actuator, the primitive corresponding to the mapped position of the tactile actuator, the node corresponding to the mapped position of the tactile actuator, and the body part mask corresponding to the mapped position of the tactile actuator.

[0154] In an implementation of the present application, when the first object is an object related to a tactile actuator, the second information associated with the first object includes the control information of the tactile actuator.

[0155] In an implementation of the present application, when the first object is an object related to a sensor, the first information includes at least one of the following: the type of the sensor, the capability information of the sensor, the sensor identifier, and the position information of the sensor.

[0156] In an implementation of the present application, the capability information of the sensor includes at least one of the following: information related to the update frequency of the sensed data and information related to the accuracy of the sensed data.

[0157] In an implementation of the present application, when the sensor is a position sensor, the capability information of the sensor includes at least one of the following: information related to the positioning accuracy of the sensed data, information related to the positioning precision of the sensed data, and information related to the positioning resolution of the sensed data.

[0158] In an implementation of the present application, the position information of the sensor includes at least one of the following: the grid identifier corresponding to the mapped position of the sensor, the grid vertex corresponding to the mapped position of the sensor, the primitive identifier corresponding to the mapped position of the sensor, the node identifier corresponding to the mapped position of the sensor, and the body part mask corresponding to the mapped position of the sensor.

[0159] In an implementation of the present application, when the first object is an object related to a sensor, the second information associated with the first object includes the control information of the sensor.

[0160] In an embodiment of the present application, the control information of the sensor includes at least one of the following: the switch information of the sensor, the information related to the update frequency of the sensing data, the information related to the accuracy of the sensing data, and the latency requirement of the sensing data.

[0161] In an embodiment of the present application, the first object is a node-level object or a scene-level object.

[0162] In an embodiment of the present application, when the haptic actuator-related object is referenced by a node object or a mesh object, the haptic actuator associated with the haptic actuator-related object is associated with the node object or the mesh object; or, when the sensor-related object is referenced by a node object or a mesh object, the sensor associated with the sensor-related object is associated with the node object or the mesh object.

[0163] In an embodiment of the present application, the first device sends the first information associated with the first object to the rendering end, so that the rendering end can perform rendering processing or synthesize a scene on the haptic data according to the first information associated with the first object, so that the rendering processing or synthesis of the haptic data performed by the rendering end can match the operating conditions of the first device; or, the rendering end generates the second information associated with the first object, and the rendering end sends the second information to the first device, and the first device performs corresponding control operations according to the second information, so that the rendering end can control the state of the first device, so that the rendering processing or synthesis of the haptic data performed by the rendering end can match the operating conditions of the first device, ensuring the user experience.

[0164] See Figure 4 , an embodiment of the present application provides a processing device, which is applied to the rendering end, and the rendering end includes a rendering engine or a display engine. The device 400 includes: a first processing unit 401 and a first transceiver unit 402;

[0165] The first processing unit 401 is configured to perform rendering processing or synthesize a scene on the haptic data according to the first information associated with the first object; or, the first processing unit 401 is configured to generate the second information associated with the first object, and the second information is used to control the first device.

[0166] In this embodiment, the first object is used to instantiate the first device.

[0167] In this embodiment, the first object includes at least one of the following: a haptic actuator-related object, a sensor-related object.

[0168] In this embodiment, the first device includes at least one of the following: a haptic actuator, a sensor.

[0169] In one embodiment of the present application, the first processing unit 401 is further configured to determine first information associated with the first object.

[0170] In one embodiment of the present application, the first transceiver unit 402 is configured to obtain the first information associated with the first object from the first device through a first API; wherein, the first device includes at least one of the following: a tactile actuator, a sensor.

[0171] In one embodiment of the present application, the first transceiver unit 402 is configured to send second information associated with the first object to the first device through a second API;

[0172] wherein, the first device includes at least one of the following: a tactile actuator, a sensor.

[0173] In one embodiment of the present application, when the first object is a tactile actuator-related object, the first information includes at least one of the following:

[0174] 1) The perception modality associated with the tactile actuator;

[0175] 2) The type of the tactile actuator;

[0176] 3) The capability information of the tactile actuator;

[0177] 4) The tactile actuator identifier;

[0178] 5) The location information of the tactile actuator;

[0179] 6) The hybrid weight information;

[0180] 7) The channel gain information.

[0181] In one embodiment of the present application, the capability information of the tactile actuator includes at least one of the following:

[0182] 1) The maximum or minimum vibration frequency of the tactile actuator;

[0183] 2) The resonant frequency of the tactile actuator;

[0184] 3) The maximum or minimum amplitude of the tactile actuator.

[0185] In one embodiment of the present application, the location information of the tactile actuator includes at least one of the following:

[0186] 1) The grid corresponding to the mapped position of the tactile actuator;

[0187] 2) The grid vertex corresponding to the mapped position of the tactile actuator;

[0188] 3) The primitive corresponding to the mapped position of the tactile actuator;

[0189] 4) Nodes corresponding to the mapped positions of the tactile actuators;

[0190] 5) Body part masks corresponding to the mapped positions of the tactile actuators.

[0191] In an implementation manner of this application, when the first object is an object related to a tactile actuator, the second information associated with the first object includes control information of the tactile actuator.

[0192] In an implementation manner of this application, when the first object is an object related to a sensor, the first information includes at least one of the following:

[0193] 1) The type of the sensor;

[0194] 2) The capability information of the sensor;

[0195] 3) The sensor identifier;

[0196] 4) The position information of the sensor.

[0197] In an implementation manner of this application, the capability information of the sensor includes at least one of the following: information related to the update frequency of sensing data, information related to the accuracy of sensing data.

[0198] In an implementation manner of this application, when the sensor is a position sensor, the capability information of the sensor includes at least one of the following:

[0199] 1) Information related to the positioning accuracy of sensing data;

[0200] 2) Information related to the positioning accuracy of sensing data;

[0201] 3) Information related to the positioning resolution of sensing data.

[0202] In an implementation manner of this application, the position information of the sensor includes at least one of the following:

[0203] 1) The grid identifier corresponding to the mapped position of the sensor;

[0204] 2) The grid vertices corresponding to the mapped position of the sensor;

[0205] 3) The primitive identifier corresponding to the mapped position of the sensor;

[0206] 4) The node identifier corresponding to the mapped position of the sensor;

[0207] 5) The body part mask corresponding to the mapped position of the sensor.

[0208] In an embodiment of the present application, when the first object is a sensor-related object, the second information associated with the first object includes the control information of the sensor.

[0209] In an embodiment of the present application, the control information of the sensor includes at least one of the following:

[0210] 1) The switch information of the sensor;

[0211] 2) The update frequency of the sensing data;

[0212] 3) The accuracy of the sensing data;

[0213] 4) The latency requirement of the sensing data.

[0214] In an embodiment of the present application, the first object is a node-level object or a scene-level object.

[0215] In an embodiment of the present application, when the haptic actuator-related object is referenced by a node object or a grid object, the haptic actuator associated with the haptic actuator-related object is associated with the node object or the grid object;

[0216] In an embodiment of the present application, when the sensor-related object is referenced by a node object or a grid object, the sensor associated with the sensor-related object is associated with the node object or the grid object.

[0217] The device provided by the embodiments of the present application can implement Figure 2 the various processes implemented by the method embodiments, and achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0218] See Figure 5 , the embodiments of the present application provide a processing device, which is applied to a first device. The device 500 includes: a second processing unit 501 and a second transceiver unit 502;

[0219] The second transceiver unit 502 is configured to send the first information associated with the first object to the rendering end, and the first information is used for rendering the haptic data or synthesizing the scene; alternatively, the second transceiver unit 502 is configured to receive the second information associated with the first object sent by the rendering end; the second processing unit 501 performs corresponding control operations according to the second information.

[0220] In this embodiment, the first object is used to instantiate the first device.

[0221] In an embodiment of the present application, the first object includes at least one of the following: a haptic actuator-related object, a sensor-related object; alternatively, the first device includes at least one of the following: a haptic actuator, a sensor.

[0222] In an embodiment of the present application, the second transceiver unit 502 is further configured to send the first information associated with the first object to the rendering end through the first API.

[0223] In an embodiment of the present application, the second transceiver unit 502 is further configured to receive the second information associated with the first object sent by the rendering end through the second API.

[0224] In an embodiment of the present application, when the first object is a haptic actuator-related object, the first information includes at least one of the following: the perception modality associated with the haptic actuator, the type of the haptic actuator, the capability information of the haptic actuator, the haptic actuator identifier, the position information of the haptic actuator, the mixing weight information, and the channel gain information.

[0225] In an embodiment of the present application, the capability information of the haptic actuator includes at least one of the following: the maximum or minimum vibration frequency of the haptic actuator, the resonant frequency of the haptic actuator, and the maximum or minimum amplitude of the haptic actuator.

[0226] In an embodiment of the present application, the position information of the haptic actuator includes at least one of the following: the grid corresponding to the mapped position of the haptic actuator, the grid vertex corresponding to the mapped position of the haptic actuator, the primitive corresponding to the mapped position of the haptic actuator, the node corresponding to the mapped position of the haptic actuator, and the body part mask corresponding to the mapped position of the haptic actuator.

[0227] In an embodiment of the present application, when the first object is a haptic actuator-related object, the second information associated with the first object includes the control information of the haptic actuator.

[0228] In an embodiment of the present application, when the first object is a sensor-related object, the first information includes at least one of the following: the type of the sensor, the capability information of the sensor, the sensor identifier, and the position information of the sensor.

[0229] In an embodiment of the present application, the capability information of the sensor includes at least one of the following: information related to the update frequency of the sensed data and information related to the accuracy of the sensed data.

[0230] In an embodiment of the present application, when the sensor is a position sensor, the capability information of the sensor includes at least one of the following: information related to the positioning accuracy of the sensed data, information related to the positioning accuracy of the sensed data, and information related to the positioning resolution of the sensed data.

[0231] In an implementation manner of the present application, the position information of the sensor includes at least one of the following: the grid identifier corresponding to the sensor mapping position, the grid vertex corresponding to the sensor mapping position, the primitive identifier corresponding to the sensor mapping position, the node identifier corresponding to the sensor mapping position, and the body part mask corresponding to the sensor mapping position.

[0232] In an implementation manner of the present application, when the first object is a sensor-related object, the second information associated with the first object includes the control information of the sensor.

[0233] In an implementation manner of the present application, the control information of the sensor includes at least one of the following: the switch information of the sensor, the information related to the update frequency of the sensing data, the information related to the accuracy of the sensing data, and the latency requirement of the sensing data.

[0234] In an implementation manner of the present application, the first object is a node-level object or a scene-level object.

[0235] In an implementation manner of the present application, when the haptic actuator-related object is referenced by a node object or a grid object, the haptic actuator associated with the haptic actuator-related object is associated with the node object or the grid object; or when the sensor-related object is referenced by a node object or a grid object, the sensor associated with the sensor-related object is associated with the node object or the grid object.

[0236] The device provided by the embodiments of the present application can implement Figure 3 the various processes implemented by the method embodiments and achieve the same technical effects. To avoid repetition, they will not be elaborated here.

[0237] Optionally, as Figure 6 shown, the embodiments of the present application further provide an electronic device 600, including a processor 601 and a memory 602. A program or instruction that can run on the processor 601 is stored on the memory 602. When the program or instruction is executed by the processor 601, it implements the various steps of the above data processing method embodiments and can achieve the same technical effects. To avoid repetition, they will not be elaborated here.

[0238] The embodiments of the present application further provide an electronic device, including a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run a program or instruction to implement the steps in the method embodiments as Figure 2 or Figure 3 shown. This device embodiment corresponds to the above method embodiments. The various implementation processes and implementation manners of the above method embodiments can all be applied to this electronic device embodiment and can achieve the same technical effects.

[0239] The above-mentioned electronic device can be a terminal or other devices other than terminals, such as a server, a Network Attached Storage (NAS), etc.

[0240] Among them, the terminal can be a mobile phone, a tablet personal computer, a laptop computer, a notebook computer, a personal digital assistant (PDA), a handheld computer, a netbook, an ultra-mobile personal computer (UMPC), a mobile Internet device (MID), an augmented reality (AR) device, a virtual reality (VR) device, a mixed reality (MR) device, a robot, a wearable device, a flight vehicle, a vehicle user equipment (VUE), a shipborne device, a pedestrian user equipment (PUE), a smart home (home appliances with wireless communication functions, such as refrigerators, TVs, washing machines, or furniture, etc.), a game console, a personal computer (PC), a teller machine, or a self-service machine, etc. Wearable devices include: smart watches, smart bracelets, smart earphones, smart glasses, smart jewelry (smart bracelets, smart bracelets, smart rings, smart necklaces, smart anklets, smart ankle chains, etc.), smart wristbands, smart clothing, etc. Among them, the vehicle user equipment can also be called a vehicle terminal, a vehicle controller, a vehicle module, a vehicle component, a vehicle chip, or a vehicle unit, etc. It should be noted that the specific type of the terminal is not limited in the embodiments of the present application.

[0241] The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server. The cloud server can provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, a content delivery network (CDN), or cloud computing services based on big data and artificial intelligence platforms, etc.

[0242] Taking the electronic device as a terminal as an example, Figure 7Schematic diagram of the hardware structure of a terminal according to an embodiment of the present application. The terminal may be a first device or a second device. The terminal 700 includes, but is not limited to, at least some components such as a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, and a processor 710.

[0243] Those skilled in the art can understand that the terminal 700 may further include a power source (such as a battery) for supplying power to each component. The power source may be logically connected to the processor 710 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system. Figure 7 The terminal structure shown does not limit the terminal. The terminal may include more or fewer components than shown, or combine some components, or have different component arrangements, which will not be elaborated here.

[0244] It should be understood that in the embodiment of the present application, the input unit 704 may include a graphics processing unit (GPU) 7041 and a microphone 7042. The graphics processor 7041 processes the image data of still pictures or videos obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 706 may include a display panel 7061. The display panel 7061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 707 includes at least one of a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include two parts: a touch detection device and a touch controller. The other input devices 7072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be elaborated here.

[0245] In the embodiment of the present application, after receiving downlink data from a network-side device, the radio frequency unit 701 may transmit it to the processor 710 for processing; in addition, the radio frequency unit 701 may send uplink data to the network-side device. Generally, the radio frequency unit 701 includes, but is not limited to, an antenna, an amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, etc.

[0246] The memory 709 can be used to store software programs or instructions and various data. The memory 709 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data. Among them, the first storage area may store an operating system, application programs or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 709 may include volatile memory or non-volatile memory, or alternatively, the memory 709 may include non-transitory memory. Among them, the non-volatile memory or non-transitory memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a synchronous DRAM (SDRAM), a double data rate SDRAM (DDR SDRAM), an enhanced SDRAM (ESDRAM), a synch link DRAM (SLDRAM), and a direct rambus RAM (DRRAM). The memory 709 in the embodiments of the present application includes but is not limited to these and any other suitable types of memory.

[0247] The processor 710 may include one or more processing units; optionally, the processor 710 integrates an application processor and a modem processor. Among them, the application processor mainly processes operations related to the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above modem processor may not be integrated into the processor 710 either.

[0248] In one implementation, the processor 710 is configured to perform rendering processing or composite scene on haptic data according to the first information associated with the first object; or generate second information associated with the first object, where the second information is used to control the first device.

[0249] Optionally, the radio frequency unit 701 is configured to obtain the first information associated with the first object from the first device through the first API; or send the second information associated with the first object to the first device through the second API.

[0250] In another embodiment, the processor 710 is configured to send the first information associated with the first object to the rendering end, where the first information is used for rendering the tactile data or synthesizing the scene; or the radio frequency unit 701 receives the second information associated with the first object sent by the rendering end; the processor 710 performs corresponding control operations according to the second information. The terminal provided in the embodiments of the present application can implement Figure 2 or Figure 3 each process implemented by the method embodiments described above and achieve the same technical effects. To avoid repetition, details are not described herein again.

[0251] The embodiments of the present application further provide a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, it implements Figure 2 or Figure 3 each process of the method and the above-described embodiments and can achieve the same technical effects. To avoid repetition, details are not described herein again.

[0252] Wherein, the processor is the processor in the terminal or network-side device described in the above embodiments. The readable storage medium includes computer-readable storage media such as computer read-only memory ROM, random access memory RAM, magnetic disk or optical disc, etc. In some examples, the readable storage medium may be a non-transitory readable storage medium.

[0253] The embodiments of the present application further provide a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor, and the processor is configured to run a program or instruction to implement Figure 2 or Figure 3 each process of the method embodiments shown above and the above-described embodiments and can achieve the same technical effects. To avoid repetition, details are not described herein again.

[0254] It should be understood that the chip mentioned in the embodiments of the present application may also be referred to as a system-on-chip, system chip, chip system, or system-on-chip, etc.

[0255] The embodiments of the present application further provide a computer program / program product, which is stored in a storage medium. The computer program / program product is executed by at least one processor to implement Figure 2 or Figure 3 each process of the method embodiments shown above and the above-described embodiments and can achieve the same technical effects. To avoid repetition, details are not described herein again.

[0256] It should be noted that in this text, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also other elements not explicitly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising one..." does not exclude the presence of additional identical elements in the process, method, article or device comprising that element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0257] From the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of computer software products plus a necessary general hardware platform, and of course, can also be implemented by hardware. The computer software products are stored in a storage medium (such as ROM, RAM, magnetic disk, optical disk, etc.) and include several instructions for causing a terminal or a network-side device to execute the methods described in various embodiments of the present application.

[0258] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms of embodiments without departing from the purpose of the present application and the scope protected by the claims. These embodiments are all within the protection scope of the present application.

Claims

1. A processing method, characterized in that Including: The rendering end renders the tactile data or synthesizes a scene according to the first information associated with the first object; Alternatively, the rendering end generates second information associated with the first object, and the second information is used to control the first device.

2. The method according to claim 1, characterized in that, The method further includes: The rendering end obtains the first information associated with the first object from the first device through a first application programming interface (API); Or, The rendering end sends the second information associated with the first object to the first device through a second API.

3. The method according to claim 1, wherein The first object includes at least one of the following: a tactile actuator-related object, a sensor-related object; Or, the first device includes at least one of the following: a tactile actuator, a sensor.

4. The method according to claim 1 or 3, characterized in that, When the first object is a tactile actuator-related object, the first information includes at least one of the following: a perception modality associated with the tactile actuator, a type of the tactile actuator, capability information of the tactile actuator, a tactile actuator identifier, a position information of the tactile actuator, a mixing weight information, a channel gain information.

5. The method according to claim 4, wherein The capability information of the tactile actuator includes at least one of the following: a maximum or minimum vibration frequency of the tactile actuator, a resonance frequency of the tactile actuator, a maximum or minimum amplitude of the tactile actuator.

6. The method according to claim 4, wherein The position information of the tactile actuator includes at least one of the following: a grid corresponding to the mapped position of the tactile actuator, a grid vertex corresponding to the mapped position of the tactile actuator, a primitive corresponding to the mapped position of the tactile actuator, a node corresponding to the mapped position of the tactile actuator, a body part mask corresponding to the mapped position of the tactile actuator.

7. The method according to claim 1 or 3, characterized in that, When the first object is a tactile actuator-related object, the second information associated with the first object includes control information of the tactile actuator.

8. The method according to claim 1 or 3, characterized in that, When the first object is a sensor-related object, the first information includes at least one of the following: a type of the sensor, capability information of the sensor, a sensor identifier, a position information of the sensor.

9. The method according to claim 8, wherein The capability information of the sensor includes at least one of the following: information related to an update frequency of sensing data, information related to an accuracy of sensing data.

10. The method according to claim 8, characterized in that, In the case where the sensor is a position sensor, the capability information of the sensor includes at least one of the following: information related to a positioning accuracy of sensing data, information related to a positioning accuracy of sensing data, information related to a positioning resolution of sensing data.

11. The method according to claim 8, wherein The position information of the sensor includes at least one of the following: a grid identifier corresponding to the mapped position of the sensor, a grid vertex corresponding to the mapped position of the sensor, a primitive identifier corresponding to the mapped position of the sensor, a node identifier corresponding to the mapped position of the sensor, a body part mask corresponding to the mapped position of the sensor.

12. The method according to claim 1 or 3, characterized in that, When the first object is a sensor-related object, the second information associated with the first object includes control information of the sensor.

13. The method according to claim 12, wherein The control information of the sensor includes at least one of the following: a switch information of the sensor, information related to an update frequency of sensing data, information related to an accuracy of sensing data, a latency requirement of sensing data.

14. The method according to claim 1, wherein The first object is a node-level object or a scene-level object.

15. The method according to claim 1, characterized in that, When the first object is referenced by a node object or a mesh object, the first device associated with the first object is associated with the node object or the mesh object.

16. A processing method, characterized in that, Including: The first device sends the first information associated with the first object to the rendering end, and the first information is used for rendering the haptic data or synthesizing the scene; Or, The first device receives the second information associated with the first object sent by the rendering end; The first device performs corresponding control operations according to the second information.

17. The method according to claim 16, characterized in that, The first device sending the first information associated with the first object to the rendering end includes: The first device sends the first information associated with the first object to the rendering end through the first API; Or, The first device receiving the second information associated with the first object sent by the rendering end includes: The first device receives the second information associated with the first object sent by the rendering end through the second API.

18. The method according to claim 16, wherein The first object includes at least one of the following: an object related to a haptic actuator, an object related to a sensor; Or, the first device includes at least one of the following: a haptic actuator, a sensor.

19. The method according to claim 16 or 18, characterized in that, When the first object is an object related to a haptic actuator, the first information includes at least one of the following: a perception modality associated with the haptic actuator, a type of the haptic actuator, capability information of the haptic actuator, a haptic actuator identifier, position information of the haptic actuator, mixing weight information, channel gain information.

20. The method according to claim 19, wherein The capability information of the haptic actuator includes at least one of the following: a maximum or minimum vibration frequency of the haptic actuator, a resonant frequency of the haptic actuator, a maximum or minimum amplitude of the haptic actuator.

21. The method according to claim 19, wherein The position information of the haptic actuator includes at least one of the following: a grid corresponding to the mapped position of the haptic actuator, a grid vertex corresponding to the mapped position of the haptic actuator, a primitive corresponding to the mapped position of the haptic actuator, a node corresponding to the mapped position of the haptic actuator, a body part mask corresponding to the mapped position of the haptic actuator.

22. The method according to claim 16 or 18, characterized in that, When the first object is an object related to a haptic actuator, the second information associated with the first object includes control information of the haptic actuator.

23. The method according to claim 16 or 18, characterized in that, When the first object is an object related to a sensor, the first information includes at least one of the following: a type of the sensor, capability information of the sensor, a sensor identifier, position information of the sensor.

24. The method according to claim 23, wherein The capability information of the sensor includes at least one of the following: information related to an update frequency of sensing data, information related to an accuracy of sensing data.

25. The method according to claim 23, wherein In the case where the sensor is a position sensor, the capability information of the sensor includes at least one of the following: information related to a positioning accuracy of sensing data, information related to a positioning accuracy of sensing data, information related to a positioning resolution of sensing data.

26. The method according to claim 24, characterized in that, The position information of the sensor includes at least one of the following: a grid identifier corresponding to the mapped position of the sensor, a grid vertex corresponding to the mapped position of the sensor, a primitive identifier corresponding to the mapped position of the sensor, a node identifier corresponding to the mapped position of the sensor, a body part mask corresponding to the mapped position of the sensor.

27. The method according to claim 16 or 18, characterized in that, When the first object is an object related to a sensor, the second information associated with the first object includes control information of the sensor.

28. The method according to claim 27, wherein The control information of the sensor includes at least one of the following: switch information of the sensor, information related to an update frequency of sensing data, information related to an accuracy of sensing data, a time delay requirement of sensing data.

29. A processing device, characterized in that, Including: A first processing unit, configured to perform rendering processing or scene synthesis on tactile data according to first information associated with a first object; Or, configured to generate second information associated with the first object, where the second information includes control parameters of a first device.

30. The device according to claim 29, wherein, The apparatus further includes: a first transceiver unit, configured to obtain the first information associated with the first object from the first device through a first API; or, the first transceiver unit is configured to send the second information associated with the first object to the first device through a second API.

31. A processing device, characterized in that, Including: A second transceiver unit is configured to send the first information associated with the first object to a rendering end, where the first information is used for performing rendering processing or scene synthesis on tactile data; Or, the second transceiver unit is configured to receive the second information associated with the first object sent by the rendering end; A second processing unit is configured to perform corresponding control operations according to the second information.

32. The device according to claim 31, characterized in that, The second transceiver unit is further configured to send the first information associated with the first object to the rendering end through a first API, or, the second transceiver unit is further configured to receive the second information associated with the first object sent by the rendering end through a second API.

33. An electronic device, characterized in that, Including a processor, a memory, and a program or instruction stored on the memory and executable on the processor, where when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 28 are implemented.

34. A readable storage medium, characterized in that, A program or instruction is stored on the readable storage medium, where when the program or instruction is executed by a processor of a terminal, the steps of the method according to any one of claims 1 to 28 are implemented.

Images