Processing method and apparatus, device, and readable storage medium

Abstract

The present application discloses a processing method and apparatus, a device, and a readable storage medium. The method comprises: a rendering end performs rendering processing or scene synthesis on tactile data on the basis of first information associated with a first object; or the rendering end generates second information associated with the first object, wherein the second information is used for controlling a first device.

Description

Processing method, device, equipment and readable storage medium

[0001] Cross-references

[0002] This disclosure claims priority to Chinese patent application number 2024100418622, filed on January 10, 2024, entitled “Processing Method, Device, Equipment and Readable Storage Medium,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application belongs to the field of communication technology, and specifically relates to a processing method, apparatus, device and readable storage medium. Background Art

[0004] In the related art, scene rendering based on the Graphics Language Transmission Format (glTF) lacks description of hardware information, which causes the synthetic rendering of the scene to be mismatched with the operating status of the external device, affecting the working performance of the external device and user experience. Summary of the Invention

[0005] The embodiments of the present application provide a processing method, apparatus, device and readable storage medium to solve the problem that the synthetic rendering of the scene does not match the operating status of the external device, affecting the working performance of the external device and the user experience.

[0006] In a first aspect, a processing method is provided, comprising:

[0007] The rendering end performs rendering processing or synthesizes a scene on the tactile data according to the first information associated with the first object;

[0008] Alternatively, the rendering end generates second information associated with the first object, where the second information is used to control the first device.

[0009] A second aspect provides a processing method, comprising:

[0010] The first device sends first information associated with the first object to the rendering end, where the first information is used to render the tactile data or synthesize a scene;

[0011] or,

[0012] The first device receives second information associated with the first object sent by the rendering end;

[0013] The first device performs a corresponding control operation according to the second information.

[0014] According to a third aspect, a processing device is provided, comprising:

[0015] The first processing unit is configured to render or synthesize a scene on the tactile data according to the first information associated with the first object; or to generate second information associated with the first object, where the second information is used to control the first device.

[0016] In a fourth aspect, a processing device is provided, comprising:

[0017] The second transceiver unit is used to send first information associated with the first object to the rendering end, and the first information is used to render or synthesize the tactile data; or, the second transceiver unit is used to receive second information associated with the first object sent by the rendering end; the second processing unit is used to perform corresponding control operations based on the second information.

[0018] In a fifth aspect, an electronic device is provided, comprising: a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect or the second aspect.

[0019] In a sixth aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor of a terminal, the steps of the method described in the first aspect or the second aspect are implemented.

[0020] In the seventh aspect, a chip is provided, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the steps of the method described in the first aspect or the second aspect.

[0021] In an eighth aspect, a computer program / program product is provided, wherein the computer program / program product is stored in a non-volatile storage medium, and the program / program product is executed by at least one processor to implement the steps of the method described in the first aspect or the second aspect.

[0022] In an embodiment of the present application, the rendering end renders or synthesizes the scene on the tactile data based on the first information associated with the first object, so that the rendering or synthesis of the tactile data performed by the rendering end can match the operating status of the first device; or, the rendering end generates second information associated with the first object, and the second information is used to control the first device, so that the rendering end can control the state of the first device, so that the rendering or synthesis of the tactile data performed by the rendering end can match the operating status of the first device, thereby ensuring user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] FIG1 is a schematic diagram of accessing tactile data in an embodiment of the present application;

[0024] FIG2 is a flow chart of a processing method according to an embodiment of the present application;

[0025] FIG3 is a second flow chart of the processing method according to an embodiment of the present application;

[0026] FIG4 is a schematic diagram of a module of a processing device according to an embodiment of the present application;

[0027] FIG5 is a second schematic diagram of a module of a processing device according to an embodiment of the present application;

[0028] FIG6 is a structural block diagram of an electronic device provided in an embodiment of the present application;

[0029] FIG7 is a structural block diagram of a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the accompanying drawings in the embodiments of this application to clearly describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0031] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and do not limit the number of objects, for example, the first object can be one or more. In addition, "or" in this application represents at least one of the connected objects. For example, "A or B" covers three options, namely, Option 1: including A but not including B; Option 2: including B but not including A; Option 3: including both A and B. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0032] It is worth noting that the technology described in the embodiments of the present application is not limited to the Long Term Evolution (LTE) / LTE-Advanced (LTE-A) system, but can also be used in other wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency Division Multiple Access (SC-FDMA) or other systems. The terms "system" and "network" in this application are often used interchangeably, and the technology described can be used for the systems and radio technologies mentioned above, as well as for other systems and radio technologies. The following description describes a New Radio (NR) system for example purposes, and NR terminology is used in most of the following description, but these technologies can also be applied to systems other than NR system applications, such as 6th generation (6G) systems. th Generation, 6G) communication system.

[0033] In order to enable those skilled in the art to better understand the embodiments of the present application, the following description is first given.

[0034] 0. About the synthesis or rendering process of the scene.

[0035] The rendering engine can synthesize scenes based on glTF (3D model file format) files, or render the scenes based on glTF files. By rendering the tactile scene, the rendering engine can obtain tactile feedback data. This tactile data is then encoded, transmitted, or decoded before being played by the tactile data executor.

[0036] The rendering engine obtains glTF files in two ways: the server and client exchange glTF files through the glTF state stream, or obtain glTF files through program files stored on the server and client.

[0037] 1. Moving Pictures Experts Group (MPEG)_haptic extension in glTF.

[0038] The MPEG_haptic extension provides access methods for haptic data. MPEG_haptic includes a set of haptic objects (hapticObjects). The accessor properties in hapticObjects point to the haptic data source.

[0039] Table 1: MPEG_haptic.

[0040] Table 2: hapticObject

[0041] Figure 1 shows how haptic data is accessed. HapticObjects in MPEG_haptic point to accessors, which in turn point to circular buffer objects. The circular buffer object provides access to the circular buffer's data location information and, by referencing MPEG_media objects, reflects information related to haptic data rendering.

[0042] 2. About the MPEG haptic material (MPEG_haptic_material) extension in glTF.

[0043] MPEG_haptic_material extension provides haptic properties for materials.MPEG_haptic_material contains material properties.

[0044] Table 3: MPEG_haptic_material.

[0045] The materials property points to a set of material objects. Each material object points to the hapticObject in MPEG_haptic that provides the data source through the haptic property, and indicates the relevant material tactile properties (such as stiffness, friction, etc.) and texture coordinate information.

[0046] Table 4: Material properties of MPEG_haptic_material.

[0047] 3. About the Interactivity extension in glTF.

[0048] glTF is used for efficient transmission and loading of 3D scenes and models by engines and applications. glTF defines an extensible publishing format that simplifies authoring workflows and interactive services by enabling interoperable use of 3D content across the industry.

[0049] MPEG has extended glTF and initially designed standards for interactivity. For haptic scenarios, these standards primarily describe the interaction between avatars and haptic sources, triggering and generating haptic data based on this interaction.

[0050] Interactivity is defined based on behavior, which includes a trigger object and an action object. The trigger object includes collision conditions, proximity conditions, and visibility conditions. The action object includes haptic feedback actions (i.e., ACTION_SET_HAPTIC), which are implemented based on a HapticActionNode object or a HapticActionMedia object.

[0051] The hapticObject attribute in HapticActionNode points to the hapticObject in MPEG_haptic, and the media index (mediaIndex) attribute in the haptic action media (HapticActionMedia) further points to the accessor in the above hapticObject, thereby accessing the haptic data.

[0052] Table 5: Haptic action nodes.

[0053] Table 6: HapticActionMedia

[0054] In the embodiments of the present application, an object includes at least one of the following: a scene, a node, a mesh, a material, a vertex, a face, a primitive, etc. The data processing method and related devices provided by the embodiments of the present application are described in detail below through some embodiments and their application scenarios, in conjunction with the accompanying drawings.

[0055] Referring to Figure 2, an embodiment of the present application provides a processing method applied to a rendering end, which includes at least one of the following: a rendering engine, a display engine, a rendering device, a scene synthesis device, a scene synthesis engine (Composition Engine), a presentation engine (Presentation Engine), a presentation device, etc. The specific steps include: Step 201.

[0056] Step 201: The rendering end performs rendering processing or synthesizes a scene on the tactile data according to the first information associated with the first object; or the rendering end generates second information associated with the first object, where the second information is used to control the first device.

[0057] In this embodiment, the first object is used to instantiate the first device.

[0058] In this embodiment, the synthetic scene refers to simulating or creating a virtual environment through relevant technologies, and the synthetic scene includes but is not limited to updating the 3D model.

[0059] In this embodiment, the second information can be mainly used to control the state of the first device, so that the rendering processing or synthesis scene of the tactile data performed by the rendering end can match the operating status of the first device.

[0060] In this embodiment, the first object includes at least one of the following: a tactile actuator-related object and a sensor-related object.

[0061] The haptic actuator-related object may be a code unit that encapsulates an action execution. The sensor-related object may be a code unit that encapsulates sensor-related operations.

[0062] In this embodiment, the first device includes at least one of the following: a tactile actuator, a sensor.

[0063] Optionally, the first information can be used to reflect the state of the first device, the second information can be used to control the state of the first device, the first information can be associated with one or more first objects, and the second information can be associated with one or more first objects. Through the first information or the second information, the tactile data rendered or synthesized by the rendering end can match the operating status of the first device.

[0064] In an embodiment of the present application, the method further includes: the rendering end determining first information associated with the first object, that is, the information associated with the first object may be generated by the rendering end.

[0065] In one embodiment of the present application, the method further includes: the rendering end obtaining first information associated with the first object from a first device through a first application programming interface (API).

[0066] In one embodiment of the present application, the method further includes: the rendering end sending second information associated with the first object to the first device via a second API;

[0067] The first device includes at least one of the following: a tactile actuator and a sensor.

[0068] That is, the first device and the rendering end can exchange the first information or the second information through the API.

[0069] In one embodiment of the present application, when the first object is a haptic actuator-related object, the first information includes at least one of the following:

[0070] 1) Perception modality associated with tactile actuators;

[0071] For example, vibration sensation, or temperature, etc.

[0072] 2) Type of tactile actuator;

[0073] For example, voice coil motors, eccentric rotor motors, piezoelectric actuators, electromagnetic actuators, etc.

[0074] 3) capability information of tactile actuators;

[0075] For example, the rendering end compresses the tactile data according to the capability information of the tactile actuator.

[0076] 4) tactile actuator identification;

[0077] For example, the haptic actuator identifier is used to identify a device included in a haptic actuator-related object. The haptic actuator identifier may be a device identifier.

[0078] 5) Position information of the tactile actuator;

[0079] 6) Mixing weight information;

[0080] Optionally, the rendering end determines, based on the mixing weight carried in the attribute information, a mixing weight value in metadata of the transmitted tactile data when multiple channels of tactile data are simultaneously played on a certain tactile actuator.

[0081] In one embodiment, the mixing weight information represents a value of a mixing weight of a set of tactile data when a set of tactile data triggered at a position near the tactile actuator is mixed with the tactile actuator for playback.

[0082] For example, the mixing weight information may indicate a value of a mixing weight of tactile data associated with a group of vertices near the tactile actuator. The tactile data corresponding to the group of vertices may be mixed and played on the tactile actuator according to the mixing weight.

[0083] In another embodiment, the mixing weight information may also represent a value of a mixing weight of the tactile data when the tactile data triggered by the tactile actuator position is mixed with the tactile data associated with other positions.

[0084] For example, the mixing weight information may indicate a group of haptic actuators and a mixing weight associated with each haptic actuator. The haptic data corresponding to the group of haptic actuators may be mixed and played on a certain haptic actuator according to the mixing weight.

[0085] 7) Channel gain information.

[0086] In one embodiment, the rendering end determines, based on the channel gain, a channel gain value in metadata of the transmitted tactile data when a certain tactile data channel is mapped to multiple tactile execution devices for playback.

[0087] For example, the channel gain information represents the channel gain of the tactile data played on each tactile actuator when the tactile data triggered by a certain tactile actuator position is played on multiple trigger actuators.

[0088] For another example, the channel gain information indicates a group of tactile actuators and the channel gain associated with each tactile actuator. The tactile data corresponding to a tactile actuator in the group can be mapped to other tactile actuators in the group for playback based on the channel gain.

[0089] In one embodiment of the present application, the capability information of the haptic actuator includes at least one of the following:

[0090] 1) The maximum or minimum vibration frequency of the tactile actuator;

[0091] 2) the resonant frequency of the haptic actuator;

[0092] 3) The maximum or minimum amplitude of the tactile actuator.

[0093] In one embodiment of the present application, the position information of the tactile actuator includes at least one of the following:

[0094] 1) The mesh corresponding to the tactile actuator mapping position;

[0095] 2) The mesh vertex (mesh.vertice) corresponding to the tactile actuator mapping position;

[0096] 3) The primitive corresponding to the tactile actuator mapping position;

[0097] 4) Node corresponding to the tactile actuator mapping position;

[0098] 5) Body part mask corresponding to the tactile actuator mapping position.

[0099] In an embodiment of the present application, when the first object is a haptic actuator-related object, the second information associated with the first object includes control information of the haptic actuator.

[0100] Optionally, the control information of the haptic actuator includes but is not limited to switch information of the haptic actuator. Through the switch information, the rendering engine can selectively turn off the haptic actuator according to scene requirements to achieve the purpose of saving power.

[0101] In one embodiment of the present application, when the first object is a sensor-related object, the first information includes at least one of the following:

[0102] 1) Sensor type;

[0103] Optionally, the type of sensor includes but is not limited to at least one of the following: a position sensor, an acceleration sensor, etc.

[0104] 2) Sensor capability information;

[0105] 3) Sensor identification;

[0106] 4) Sensor location information.

[0107] In one embodiment of the present application, the capability information of the sensor includes at least one of the following: information related to the update frequency of the sensor data, and information related to the accuracy of the sensor data.

[0108] In one embodiment of the present application, when the sensor is a position sensor, the capability information of the sensor includes at least one of the following:

[0109] 1) Information related to the positioning accuracy of sensor data;

[0110] 2) Information related to the positioning accuracy of the sensor data;

[0111] 3) Information related to the positioning resolution of the sensor data.

[0112] In one embodiment of the present application, the position information of the sensor includes at least one of the following:

[0113] 1) The mesh ID corresponding to the sensor mapping location;

[0114] 2) The mesh vertex corresponding to the sensor mapping position;

[0115] 3) The primitive ID corresponding to the sensor mapping location;

[0116] 4) Node ID corresponding to the sensor mapping location;

[0117] 5) Body part mask corresponding to the sensor map location.

[0118] In this embodiment, the location, type, or capability information of the haptic actuator or sensor can be obtained based on the actual situation of the haptic actuator or sensor. The haptic actuator or sensor and the rendering end exchange such relevant information via corresponding APIs. Alternatively, the location information of the haptic actuator or sensor can be virtual information assumed by the rendering end.

[0119] In one embodiment of the present application, when the first object is a sensor-related object, the second information associated with the first object includes control information of the sensor. The rendering end can control the sensor according to scene requirements through the control information.

[0120] In one embodiment of the present application, the control information of the sensor includes at least one of the following:

[0121] 1) Sensor switch information;

[0122] Optionally, the sensor switch information may be for each sensor or a group of sensors (eg, all position sensors of the hand).

[0123] Optionally, the switch information of the sensor may be indicated together with the sensor identification or the sensor location information.

[0124] In this way, the rendering end can control the sensor according to the scene requirements

[0125] 2) Update frequency of sensor data;

[0126] The update frequency of sensor data refers to the frequency at which the sensor collects and provides new data.

[0127] 3) Accuracy of sensor data;

[0128] For example, location accuracy.

[0129] 4) Latency requirements for sensor data.

[0130] The latency requirement of sensor data refers to how long it takes for the data to be processed or delivered to the target location after the sensor collects the data.

[0131] In one embodiment of the present application, the first object may be a node-level object or a scene-level object.

[0132] In another embodiment of the present application, the first object can be referenced by an object such as a node or a scene as attribute information.

[0133] In one embodiment of the present application, when the tactile actuator-related object is referenced by a node object or a mesh object, the tactile actuator associated with the tactile actuator-related object is associated with the node object or the mesh object, that is, the tactile actuator associated with the tactile actuator-related object is mapped to the corresponding node or mesh.

[0134] In one embodiment of the present application, when the sensor-related object is referenced by a node object or a grid object, the sensor associated with the sensor-related object is associated with the node object or the grid object, that is, the sensor associated with the sensor-related object is mapped to the corresponding node or grid.

[0135] Optionally, the rendering end may combine the mapping position information indicated by the haptic actuator-related object or the sensor-related object to accurately locate the position information of the haptic actuator or the sensor on the avatar.

[0136] In an embodiment of the present application, the rendering end renders or synthesizes the scene on the tactile data based on the first information associated with the first object, so that the rendering or synthesis of the tactile data performed by the rendering end can match the operating status of the first device; or, the rendering end generates second information associated with the first object, and the second information is used to control the first device, so that the rendering end can control the state of the first device, so that the rendering or synthesis of the tactile data performed by the rendering end can match the operating status of the first device, thereby ensuring user experience.

[0137] 3 , an embodiment of the present application provides a processing method, which is applied to a first device, and the specific steps include: Step 301 .

[0138] Step 301: The first device sends first information associated with the first object to the rendering end, and the first information is used to render the tactile data or synthesize the scene; or, the first device receives second information associated with the first object sent by the rendering end; the first device performs corresponding control operations based on the second information.

[0139] In one embodiment of the present application, the method also includes: the first device receives rendered tactile data or an updated scene sent by the rendering end, wherein the rendered tactile data is obtained by the rendering end rendering the tactile data according to the first information, and the updated scene is obtained by the rendering end synthesizing the scene according to the first information.

[0140] In one embodiment of the present application, the first device sending the first information associated with the first object to the rendering end includes:

[0141] The first device sends first information associated with the first object to the rendering end through the first API.

[0142] In one embodiment of the present application, the first device receives second information associated with the first object sent by the rendering end, including:

[0143] The first device receives second information associated with the first object sent by the rendering end through the second API.

[0144] In one embodiment of the present application, the first object includes at least one of the following: a tactile actuator-related object, a sensor-related object; or the first device includes at least one of the following: a tactile actuator, a sensor.

[0145] In one embodiment of the present application, when the first object is a tactile actuator-related object, the first information includes at least one of the following: a perception modality associated with the tactile actuator, the type of the tactile actuator, capability information of the tactile actuator, a tactile actuator identifier, position information of the tactile actuator, mixing weight information, and channel gain information.

[0146] In one embodiment of the present application, the capability information of the tactile actuator includes at least one of the following: a maximum or minimum vibration frequency of the tactile actuator, a resonant frequency of the tactile actuator, and a maximum or minimum amplitude of the tactile actuator.

[0147] In one embodiment of the present application, the position information of the tactile actuator includes at least one of the following: a mesh corresponding to the tactile actuator mapping position, a mesh vertex corresponding to the tactile actuator mapping position, a basic body corresponding to the tactile actuator mapping position, a node corresponding to the tactile actuator mapping position, and a body part mask corresponding to the tactile actuator mapping position.

[0148] In an embodiment of the present application, when the first object is a haptic actuator-related object, the second information associated with the first object includes control information of the haptic actuator.

[0149] In an embodiment of the present application, when the first object is a sensor-related object, the first information includes at least one of the following: sensor type, sensor capability information, sensor identification, and sensor location information.

[0150] In one embodiment of the present application, the capability information of the sensor includes at least one of the following: information related to the update frequency of the sensor data, and information related to the accuracy of the sensor data.

[0151] In one embodiment of the present application, when the sensor is a position sensor, the capability information of the sensor includes at least one of the following: information related to the positioning accuracy of the sensing data, information related to the positioning precision of the sensing data, and information related to the positioning resolution of the sensing data.

[0152] In one embodiment of the present application, the position information of the sensor includes at least one of the following: a grid identifier corresponding to the sensor mapping position, a grid vertex corresponding to the sensor mapping position, a basic body identifier corresponding to the sensor mapping position, a node identifier corresponding to the sensor mapping position, and a body part mask corresponding to the sensor mapping position.

[0153] In an embodiment of the present application, when the first object is a sensor-related object, the second information associated with the first object includes control information of the sensor.

[0154] In one embodiment of the present application, the control information of the sensor includes at least one of the following: switch information of the sensor, information related to the update frequency of the sensor data, information related to the accuracy of the sensor data, and delay requirements of the sensor data.

[0155] In one embodiment of the present application, the first object is a node-level object or a scene-level object.

[0156] In one embodiment of the present application, when the tactile actuator-related object is referenced by a node object or a mesh object, the tactile actuator associated with the tactile actuator-related object is associated with the node object or the mesh object; or, when the sensor-related object is referenced by a node object or a mesh object, the sensor associated with the sensor-related object is associated with the node object or the mesh object.

[0157] In an embodiment of the present application, the first device sends the first information associated with the first object to the rendering end, so that the rendering end can render the tactile data or synthesize the scene according to the first information associated with the first object, so that the rendering of the tactile data or the synthesis of the scene performed by the rendering end can match the operation status of the first device; or, the rendering end generates second information associated with the first object, and the rendering end sends the second information to the first device, and the first device performs corresponding control operations according to the second information, so that the rendering end can control the state of the first device, so that the rendering of the tactile data or the synthesis of the scene performed by the rendering end can match the operation status of the first device, thereby ensuring user experience.

[0158] 4 , an embodiment of the present application provides a processing device, which is applied to a rendering end, the rendering end including a rendering engine or a display engine. The device 400 includes: a first processing unit 401 and a first transceiver unit 402;

[0159] The first processing unit 401 is used to render or synthesize a scene on the tactile data according to the first information associated with the first object; or the first processing unit 401 is used to generate second information associated with the first object, and the second information is used to control the first device.

[0160] In this embodiment, the first object is used to instantiate the first device.

[0161] In this embodiment, the first object includes at least one of the following: a tactile actuator-related object and a sensor-related object.

[0162] In this embodiment, the first device includes at least one of the following: a tactile actuator, a sensor.

[0163] In one embodiment of the present application, the first processing unit 401 is further configured to determine first information associated with the first object.

[0164] In one embodiment of the present application, the first transceiver unit 402 is configured to obtain first information associated with the first object from a first device through a first API; wherein the first device includes at least one of the following: a tactile actuator, a sensor.

[0165] In one embodiment of the present application, the first transceiver unit 402 is configured to send the second information associated with the first object to the first device through the second API;

[0166] The first device includes at least one of the following: a tactile actuator and a sensor.

[0167] In one embodiment of the present application, when the first object is a haptic actuator-related object, the first information includes at least one of the following:

[0168] 1) The perceptual modality associated with the tactile actuator;

[0169] 2) Type of tactile actuator;

[0170] 3) capability information of tactile actuators;

[0171] 4) tactile actuator identification;

[0172] 5) Position information of the tactile actuator;

[0173] 6) Mixed weight information;

[0174] 7) Channel gain information.

[0175] In one embodiment of the present application, the capability information of the haptic actuator includes at least one of the following:

[0176] 1) The maximum or minimum vibration frequency of the tactile actuator;

[0177] 2) the resonant frequency of the haptic actuator;

[0178] 3) The maximum or minimum amplitude of the tactile actuator.

[0179] In one embodiment of the present application, the position information of the tactile actuator includes at least one of the following:

[0180] 1) Grid corresponding to the tactile actuator mapping position;

[0181] 2) The mesh vertex corresponding to the tactile actuator mapping position;

[0182] 3) the primitive corresponding to the tactile actuator mapping position;

[0183] 4) Node corresponding to the tactile actuator mapping position;

[0184] 5) Body part mask corresponding to the tactile actuator mapping position.

[0185] In an embodiment of the present application, when the first object is a haptic actuator-related object, the second information associated with the first object includes control information of the haptic actuator.

[0186] In one embodiment of the present application, when the first object is a sensor-related object, the first information includes at least one of the following:

[0187] 1) Sensor type;

[0188] 2) Sensor capability information;

[0189] 3) Sensor identification;

[0190] 4) Sensor location information.

[0191] In one embodiment of the present application, the capability information of the sensor includes at least one of the following: information related to the update frequency of the sensor data, and information related to the accuracy of the sensor data.

[0192] In one embodiment of the present application, when the sensor is a position sensor, the capability information of the sensor includes at least one of the following:

[0193] 1) Information related to the positioning accuracy of sensor data;

[0194] 2) Information related to the positioning accuracy of the sensor data;

[0195] 3) Information related to the positioning resolution of the sensor data.

[0196] In one embodiment of the present application, the position information of the sensor includes at least one of the following:

[0197] 1) Grid identifier corresponding to the sensor mapping position;

[0198] 2) The mesh vertex corresponding to the sensor mapping position;

[0199] 3) The basic body identification corresponding to the sensor mapping position;

[0200] 4) Node identification corresponding to the sensor mapping location;

[0201] 5) Body part mask corresponding to the sensor map location.

[0202] In an embodiment of the present application, when the first object is a sensor-related object, the second information associated with the first object includes control information of the sensor.

[0203] In one embodiment of the present application, the control information of the sensor includes at least one of the following:

[0204] 1) Sensor switch information;

[0205] 2) Update frequency of sensor data;

[0206] 3) Accuracy of sensor data;

[0207] 4) Latency requirements for sensor data.

[0208] In one embodiment of the present application, the first object is a node-level object or a scene-level object.

[0209] In one embodiment of the present application, when the haptic actuator-related object is referenced by a node object or a mesh object, the haptic actuator associated with the haptic actuator-related object is associated with the node object or the mesh object;

[0210] In one embodiment of the present application, when the sensor-related object is referenced by a node object or a grid object, the sensor associated with the sensor-related object is associated with the node object or the grid object.

[0211] The device provided in the embodiment of the present application can implement each process implemented in the method embodiment of Figure 2 and achieve the same technical effect. To avoid repetition, it will not be described here.

[0212] 5 , an embodiment of the present application provides a processing device, which is applied to a first device. The device 500 includes: a second processing unit 501 and a second transceiver unit 502;

[0213] The second transceiver unit 502 is used to send first information associated with the first object to the rendering end, and the first information is used to render the tactile data or synthesize the scene; or, the second transceiver unit 502 is used to receive second information associated with the first object sent by the rendering end; the second processing unit 501 performs corresponding control operations based on the second information.

[0214] In this embodiment, the first object is used to instantiate the first device.

[0215] In one embodiment of the present application, the first object includes at least one of the following: a tactile actuator-related object, a sensor-related object; or the first device includes at least one of the following: a tactile actuator, a sensor.

[0216] In one embodiment of the present application, the second transceiver unit 502 is further configured to send the first information associated with the first object to the rendering end through the first API.

[0217] In one embodiment of the present application, the second transceiver unit 502 is further configured to receive, via a second API, second information associated with the first object sent by the rendering end.

[0218] In one embodiment of the present application, when the first object is a tactile actuator-related object, the first information includes at least one of the following: a perception modality associated with the tactile actuator, the type of the tactile actuator, capability information of the tactile actuator, a tactile actuator identifier, position information of the tactile actuator, mixing weight information, and channel gain information.

[0219] In one embodiment of the present application, the capability information of the tactile actuator includes at least one of the following: a maximum or minimum vibration frequency of the tactile actuator, a resonant frequency of the tactile actuator, and a maximum or minimum amplitude of the tactile actuator.

[0220] In one embodiment of the present application, the position information of the tactile actuator includes at least one of the following: a mesh corresponding to the tactile actuator mapping position, a mesh vertex corresponding to the tactile actuator mapping position, a basic body corresponding to the tactile actuator mapping position, a node corresponding to the tactile actuator mapping position, and a body part mask corresponding to the tactile actuator mapping position.

[0221] In an embodiment of the present application, when the first object is a haptic actuator-related object, the second information associated with the first object includes control information of the haptic actuator.

[0222] In an embodiment of the present application, when the first object is a sensor-related object, the first information includes at least one of the following: sensor type, sensor capability information, sensor identification, and sensor location information.

[0223] In one embodiment of the present application, the capability information of the sensor includes at least one of the following: information related to the update frequency of the sensor data, and information related to the accuracy of the sensor data.

[0224] In one embodiment of the present application, when the sensor is a position sensor, the capability information of the sensor includes at least one of the following: information related to the positioning accuracy of the sensing data, information related to the positioning precision of the sensing data, and information related to the positioning resolution of the sensing data.

[0225] In one embodiment of the present application, the position information of the sensor includes at least one of the following: a grid identifier corresponding to the sensor mapping position, a grid vertex corresponding to the sensor mapping position, a basic body identifier corresponding to the sensor mapping position, a node identifier corresponding to the sensor mapping position, and a body part mask corresponding to the sensor mapping position.

[0226] In an embodiment of the present application, when the first object is a sensor-related object, the second information associated with the first object includes control information of the sensor.

[0227] In one embodiment of the present application, the control information of the sensor includes at least one of the following: switch information of the sensor, information related to the update frequency of the sensor data, information related to the accuracy of the sensor data, and delay requirements of the sensor data.

[0228] In one embodiment of the present application, the first object is a node-level object or a scene-level object.

[0229] In one embodiment of the present application, when the tactile actuator-related object is referenced by a node object or a mesh object, the tactile actuator associated with the tactile actuator-related object is associated with the node object or the mesh object; or, when the sensor-related object is referenced by a node object or a mesh object, the sensor associated with the sensor-related object is associated with the node object or the mesh object.

[0230] The device provided in the embodiment of the present application can implement each process implemented in the method embodiment of Figure 3 and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0231] Optionally, as shown in Figure 6, an embodiment of the present application also provides an electronic device 600, including a processor 601 and a memory 602, and the memory 602 stores a program or instruction that can be run on the processor 601. When the program or instruction is executed by the processor 601, the various steps of the above-mentioned data processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0232] The present application also provides an electronic device including a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to execute a program or instruction to implement the steps in the method embodiment shown in Figure 2 or Figure 3. This device embodiment corresponds to the above-mentioned method embodiment, and each implementation process and implementation method of the above-mentioned method embodiment can be applied to this electronic device embodiment and can achieve the same technical effect.

[0233] The electronic device may be a terminal, or may be other devices other than a terminal, such as a server, a network attached storage (NAS), etc.

[0234] Among them, the terminal can be a mobile phone, tablet personal computer, laptop computer, notebook computer, personal digital assistant (PDA), handheld computer, netbook, ultra-mobile personal computer (UMPC), mobile Internet device (MID), augmented reality (AR), virtual reality (VR) equipment, mixed reality (MR) equipment, robot, wearable device, flight vehicle, vehicle user equipment (VUE), shipborne equipment, pedestrian user equipment (PUE), smart home (home appliances with wireless communication function, such as refrigerator, TV, washing machine or furniture, etc.), game console, personal computer (PC), ATM or self-service machine and other terminal-side devices. Wearable devices include: smart watches, smart bracelets, smart headphones, smart glasses, smart jewelry (smart bracelets, smart bracelets, smart rings, smart necklaces, smart anklets, smart anklets, etc.), smart wristbands, smart clothing, etc. Among them, vehicle-mounted devices can also be called vehicle-mounted terminals, vehicle-mounted controllers, vehicle-mounted modules, vehicle-mounted components, vehicle-mounted chips, or vehicle-mounted units, etc. It should be noted that the specific type of terminal is not limited in the embodiments of this application.

[0235] The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that can provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), or cloud computing services based on big data and artificial intelligence platforms.

[0236] Taking an electronic device as a terminal as an example, Figure 7 is a schematic diagram of the hardware structure of a terminal implementing an embodiment of the present application. The terminal can be a first device or a second device, and the terminal 700 includes, but is not limited to, at least some of the components of a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, and a processor 710.

[0237] Those skilled in the art will appreciate that the terminal 700 may also include a power supply (such as a battery) to power various components. The power supply may be logically connected to the processor 710 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The terminal structure shown in FIG7 does not limit the terminal. The terminal may include more or fewer components than shown, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0238] It should be understood that in an embodiment of the present application, the input unit 704 may include a graphics processing unit (GPU) 7041 and a microphone 7042, and the graphics processor 7041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 706 may include a display panel 7061, and the display panel 7061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 707 includes a touch panel 7071 and at least one of other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include two parts: a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0239] In the embodiment of the present application, after receiving downlink data from a network-side device, the RF unit 701 may transmit the data to the processor 710 for processing. Furthermore, the RF unit 701 may send uplink data to the network-side device. Typically, the RF unit 701 includes, but is not limited to, an antenna, an amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, and the like.

[0240] The memory 709 can be used to store software programs or instructions and various data. The memory 709 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instruction required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 709 may include a volatile memory or a non-volatile memory, or the memory 709 may include a non-transient memory. Among them, the non-volatile memory or non-transient memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 709 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0241] Processor 710 may include one or more processing units. Optionally, processor 710 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 710.

[0242] In one embodiment, the processor 710 is configured to render or synthesize a scene on the tactile data according to first information associated with the first object; or to generate second information associated with the first object, where the second information is used to control the first device.

[0243] Optionally, the radio frequency unit 701 is configured to obtain first information associated with the first object from the first device through a first API; or send second information associated with the first object to the first device through a second API.

[0244] In another embodiment, the processor 710 is configured to send first information associated with the first object to the rendering end, where the first information is used to render or synthesize the tactile data; alternatively, the RF unit 701 receives second information associated with the first object sent by the rendering end; and the processor 710 performs a corresponding control operation based on the second information. The terminal provided in the embodiment of the present application can implement each process implemented in the method embodiment of Figure 2 or Figure 3 and achieve the same technical effect. To avoid repetition, it will not be described here.

[0245] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the method of Figure 2 or Figure 3 and the various processes of the above-mentioned embodiments are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0246] The processor is the processor in the terminal or network-side device described in the above embodiments. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk. In some examples, the readable storage medium may be a non-transitory readable storage medium.

[0247] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes shown in Figure 2 or Figure 3 and the various method embodiments mentioned above, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0248] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0249] An embodiment of the present application further provides a computer program / program product, which is stored in a storage medium. The computer program / program product is executed by at least one processor to implement the various processes shown in Figure 2 or Figure 3 and the various method embodiments described above, and can achieve the same technical effects. To avoid repetition, they are not described here.

[0250] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0251] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of a computer software product plus a necessary general-purpose hardware platform, or of course, by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, magnetic disk, optical disk, etc.) and includes a number of instructions for enabling a terminal or network-side device to execute the methods described in each embodiment of the present application.

[0252] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms of implementation methods without departing from the purpose of this application and the scope of protection of the claims. These implementation methods are all within the protection of this application.

Claims

1. A processing method, wherein, Including: The rendering end renders the tactile data or composes a scene according to the first information associated with the first object; Alternatively, the rendering end generates second information associated with the first object, and the second information is used to control the first device.

2. The method according to claim 1, wherein, The method further includes: The rendering end obtains the first information associated with the first object from the first device through a first application programming interface (API); Or, The rendering end sends the second information associated with the first object to the first device through a second API.

3. The method according to claim 1, wherein The first object includes at least one of the following: a tactile actuator-related object, a sensor-related object; Or, the first device includes at least one of the following: a tactile actuator, a sensor.

4. The method according to claim 1 or 3, wherein When the first object is a tactile actuator-related object, the first information includes at least one of the following: a perception modality associated with the tactile actuator, a type of the tactile actuator, capability information of the tactile actuator, a tactile actuator identifier, a location information of the tactile actuator, a mixing weight information, a channel gain information.

5. The method according to claim 4, wherein, The capability information of the tactile actuator includes at least one of the following: a maximum or minimum vibration frequency of the tactile actuator, a resonance frequency of the tactile actuator, a maximum or minimum amplitude of the tactile actuator.

6. The method according to claim 4, wherein, The location information of the tactile actuator includes at least one of the following: a grid corresponding to the mapped position of the tactile actuator, a grid vertex corresponding to the mapped position of the tactile actuator, a primitive corresponding to the mapped position of the tactile actuator, a node corresponding to the mapped position of the tactile actuator, a body part mask corresponding to the mapped position of the tactile actuator.

7. The method according to claim 1 or 3, wherein When the first object is a tactile actuator-related object, the second information associated with the first object includes control information of the tactile actuator.

8. The method according to claim 1 or 3, wherein When the first object is a sensor-related object, the first information includes at least one of the following: a type of the sensor, capability information of the sensor, a sensor identifier, a location information of the sensor.

9. The method according to claim 8, wherein, The capability information of the sensor includes at least one of the following: information related to an update frequency of sensed data, information related to an accuracy of sensed data.

10. The method according to claim 8, wherein In the case where the sensor is a position sensor, the capability information of the sensor includes at least one of the following: information related to a positioning accuracy of sensed data, information related to a positioning accuracy of sensed data, information related to a positioning resolution of sensed data.

11. The method according to claim 8, wherein, The location information of the sensor includes at least one of the following: a grid identifier corresponding to the mapped position of the sensor, a grid vertex corresponding to the mapped position of the sensor, a primitive identifier corresponding to the mapped position of the sensor, a node identifier corresponding to the mapped position of the sensor, a body part mask corresponding to the mapped position of the sensor.

12. The method according to claim 1 or 3, wherein When the first object is a sensor-related object, the second information associated with the first object includes control information of the sensor.

13. The method according to claim 12, wherein, The control information of the sensor includes at least one of the following: a switch information of the sensor, information related to an update frequency of sensed data, information related to an accuracy of sensed data, a latency requirement of sensed data.

14. The method according to claim 1, wherein, The first object is a node-level object or a scene-level object.

15. The method according to claim 1, wherein When the first object is referenced by a node object or a mesh object, the first device associated with the first object is associated with the node object or the mesh object.

16. A processing method, wherein, Including: The first device sends the first information associated with the first object to the rendering end, and the first information is used for rendering tactile data or synthesizing a scene; Or, The first device receives the second information associated with the first object sent by the rendering end; The first device performs corresponding control operations according to the second information.

17. The method according to claim 16, wherein, The first device sending the first information associated with the first object to the rendering end includes: The first device sends the first information associated with the first object to the rendering end through the first API; Or, The first device receiving the second information associated with the first object sent by the rendering end includes: The first device receives the second information associated with the first object sent by the rendering end through the second API.

18. The method according to claim 16, wherein The first object includes at least one of the following: a haptic actuator-related object, a sensor-related object; Or, the first device includes at least one of the following: a haptic actuator, a sensor.

19. The method according to claim 16 or 18, wherein, When the first object is a haptic actuator-related object, the first information includes at least one of the following: a perception modality associated with the haptic actuator, a type of the haptic actuator, capability information of the haptic actuator, a haptic actuator identifier, a position information of the haptic actuator, a mixing weight information, a channel gain information.

20. The method according to claim 19, wherein, The capability information of the haptic actuator includes at least one of the following: a maximum or minimum vibration frequency of the haptic actuator, a resonant frequency of the haptic actuator, a maximum or minimum amplitude of the haptic actuator.

21. The method according to claim 19, wherein, The position information of the haptic actuator includes at least one of the following: a grid corresponding to the mapped position of the haptic actuator, a grid vertex corresponding to the mapped position of the haptic actuator, a primitive corresponding to the mapped position of the haptic actuator, a node corresponding to the mapped position of the haptic actuator, a body part mask corresponding to the mapped position of the haptic actuator.

22. The method according to claim 16 or 18, wherein When the first object is a haptic actuator-related object, the second information associated with the first object includes control information of the haptic actuator.

23. The method according to claim 16 or 18, wherein, When the first object is a sensor-related object, the first information includes at least one of the following: a type of the sensor, capability information of the sensor, a sensor identifier, a position information of the sensor.

24. The method according to claim 23, wherein, The capability information of the sensor includes at least one of the following: information related to an update frequency of sensing data, information related to an accuracy of sensing data.

25. The method according to claim 23, wherein, In the case where the sensor is a position sensor, the capability information of the sensor includes at least one of the following: information related to a positioning accuracy of sensing data, information related to a positioning accuracy of sensing data, information related to a positioning resolution of sensing data.

26. The method according to claim 24, wherein, The position information of the sensor includes at least one of the following: a grid identifier corresponding to the mapped position of the sensor, a grid vertex corresponding to the mapped position of the sensor, a primitive identifier corresponding to the mapped position of the sensor, a node identifier corresponding to the mapped position of the sensor, a body part mask corresponding to the mapped position of the sensor.

27. The method according to claim 16 or 18, wherein When the first object is a sensor-related object, the second information associated with the first object includes control information of the sensor.

28. The method according to claim 27, wherein, The control information of the sensor includes at least one of the following: switch information of the sensor, information related to an update frequency of sensing data, information related to an accuracy of sensing data, a time delay requirement of sensing data.

29. A processing device, wherein, Includes: A first processing unit, configured to perform rendering processing or scene synthesis on tactile data according to first information associated with a first object; alternatively, configured to generate second information associated with the first object, where the second information includes control parameters of a first device.

30. The apparatus according to claim 29, wherein The apparatus further includes: a first transceiver unit, where the first transceiver unit is configured to obtain the first information associated with the first object from the first device through a first API; alternatively, the first transceiver unit is configured to send the second information associated with the first object to the first device through a second API.

31. A processing device, wherein, Including: A second transceiver unit is configured to send the first information associated with the first object to a rendering end, where the first information is used to perform rendering processing or scene synthesis on tactile data; alternatively, the second transceiver unit is configured to receive the second information associated with the first object sent by the rendering end; A second processing unit is configured to perform corresponding control operations according to the second information.

32. The apparatus according to claim 31, wherein, The second transceiver unit is further configured to send the first information associated with the first object to the rendering end through a first API, or the second transceiver unit is further configured to receive the second information associated with the first object sent by the rendering end through a second API.

33. An electronic device, wherein, Including a processor, a memory, and a program or instruction stored on the memory and executable on the processor, where when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 28 are implemented.

34. A readable storage medium, wherein, A program or instruction is stored on the readable storage medium, where when the program or instruction is executed by a processor of a terminal, the steps of the method according to any one of claims 1 to 28 are implemented.

Images