Robot remote operation control methods and devices, robots, media, products

Abstract

This application provides a robot teleoperation control method and device, robot, medium, and product. The method includes: converting node pose data corresponding to multiple source motion nodes in the motion data of a source robot into skeleton pose data corresponding to an interface humanoid skeleton according to a skeleton input format; performing a first coordinate transformation on the skeleton pose data in the first coordinate system when the first coordinate system of the motion data of the source robot does not match the second coordinate system used by the motion data provided by the body tracking device received by the teleoperation platform; generating teleoperation input data based on the skeleton pose data in the second coordinate system; and inputting the teleoperation input data to the teleoperation platform so that the teleoperation platform can reuse the processing flow for the motion data provided by the body tracking device. This application can achieve motion remapping control from the source robot to the target robot with reduced modification of the teleoperation platform.

Description

Technical Field

[0001] This application relates to the field of robot teleoperation technology, and more particularly to robot teleoperation control methods and devices, robots, media, and products. Background Technology

[0002] With the development of robotics, virtual reality, and simulation training technologies, robot teleoperation is increasingly being applied to scenarios such as humanoid robot control, robot skill transfer, simulation verification, remote task execution, and embodied intelligence data acquisition. In robot R&D, simulation training, and motion transfer scenarios, the source of motion data is not necessarily a real body tracking device, making it difficult to directly reuse the existing remapping process for motion data provided by body tracking devices on the teleoperation platform.

[0003] Based on this, embodiments of this application provide a robot teleoperation control method and device, a robot, a medium, and a product to improve related technologies. Summary of the Invention

[0004] The purpose of this application is to provide a robot teleoperation control method and device, robot, medium, and product, which realizes motion remapping control from the source robot to the target robot while reducing the need for modification of the teleoperation platform.

[0005] In a first aspect, embodiments of this application provide a robot teleoperation control method, comprising: based on skeleton mapping information, converting node pose data corresponding to multiple source motion nodes in the motion data of a source robot into skeleton pose data corresponding to an interface humanoid skeleton according to a skeleton input format, wherein the data format of the interface humanoid skeleton is consistent with the skeleton input format used by the teleoperation platform for motion data provided by a body tracking device; when the first coordinate system in which the motion data of the source robot is located does not match the second coordinate system used by the motion data provided by the body tracking device received by the teleoperation platform, performing a first coordinate transformation on the skeleton pose data in the first coordinate system to obtain skeleton pose data in the second coordinate system; generating teleoperation input data based on the skeleton pose data in the second coordinate system; wherein the teleoperation input data conforms to a preset input data structure used by the teleoperation platform for motion data provided by the body tracking device; inputting the teleoperation input data to the teleoperation platform, so that the teleoperation platform performs remapping processing on the teleoperation input data for a target robot according to the processing flow for motion data provided by the body tracking device to obtain robot control data, wherein the robot control data is used to control the movement of the target robot.

[0006] Secondly, embodiments of this application provide a robot teleoperation control device, comprising: a skeleton mapping module, configured to convert node pose data corresponding to multiple source motion nodes in the motion data of a source robot into skeleton pose data corresponding to an interface humanoid skeleton based on skeleton mapping information, wherein the data format of the interface humanoid skeleton is consistent with the skeleton input format used by the teleoperation platform for motion data provided by the body tracking device; wherein the interface humanoid skeleton conforms to the skeleton input format used by the teleoperation platform for motion data provided by the body tracking device; and a coordinate adaptation module, configured to, when the first coordinate system of the motion data of the source robot is mismatched with the second coordinate system used by the motion data provided by the body tracking device received by the teleoperation platform. In the case of a configuration, a first coordinate transformation is performed on the skeleton pose data in the first coordinate system to obtain skeleton pose data in the second coordinate system; an input generation module is used to generate teleoperation input data based on the skeleton pose data in the second coordinate system; wherein, the teleoperation input data conforms to the preset input data structure adopted by the teleoperation platform for the motion data provided by the body tracking device; a platform input module is used to input the teleoperation input data to the teleoperation platform, so that the teleoperation platform performs remapping processing on the teleoperation input data for the target robot according to the processing flow for the motion data provided by the body tracking device, to obtain robot control data, which is used to control the movement of the target robot.

[0007] Thirdly, embodiments of this application provide a robot, which includes a control module for performing the steps of any of the methods described above.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0009] Fifthly, embodiments of this application provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the steps of any of the above methods.

[0010] This application provides a robot teleoperation control method and device, robot, medium, and product. It adopts the concept of converting the motion data of the source robot into a skeleton input format, coordinate system requirements, and preset input data structure that conforms to the teleoperation platform, thereby realizing motion remapping control from the source robot to the target robot while reducing the need for modification of the teleoperation platform. Attached Figure Description

[0011] The embodiments of this application are further described below with reference to the accompanying drawings and specific implementation details.

[0012] Figure 1 This is a flowchart illustrating a robot teleoperation control method provided in an embodiment of this application.

[0013] Figure 2 This is a structural block diagram of a robot teleoperation control device provided in an embodiment of this application.

[0014] Figure 3 This is a structural block diagram of a robot provided in an embodiment of this application.

[0015] Figure 4 This is a structural block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the embodiments of this application.

[0017] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0018] With the development of robotics, virtual reality, and simulation training technologies, robot teleoperation is increasingly being applied to scenarios such as humanoid robot control, robot skill transfer, simulation verification, remote task execution, and embodied intelligence data acquisition. During robot teleoperation, the teleoperation platform can receive motion data from the operator, body tracking devices, simulation devices, or other motion data sources, and generate control data based on the received motion data to control the target robot, thereby enabling the target robot to perform corresponding actions.

[0019] In some related technologies, teleoperation platforms are primarily designed for motion data provided by body tracking devices. Body tracking devices can include head-mounted displays, handles, body trackers, motion capture devices, etc. The motion data provided by these devices can include position and posture data of multiple skeletal nodes in the human skeleton, as well as input information such as controller buttons, joysticks, triggers, and timestamps. After receiving the motion data from the body tracking device, the teleoperation platform can process it according to a preset skeletal input format and input data structure, thereby generating robot control data for the target robot.

[0020] However, in robot R&D, simulation training, and motion transfer scenarios, the source of motion data is not necessarily a real body tracking device; it could also be the source robot itself. In heterogeneous robot teleoperation or motion transfer scenarios, there is a problem of structural inconsistencies between the source and target robots. The source robot could be a simulated robot or another type of robot, while the target robot could be an actual, controlled humanoid robot. Developing independent conversion logic or inverse kinematics solution logic for each combination of source and target robots would increase engineering maintenance costs and hinder the reuse of processing capabilities already configured in existing teleoperation platforms.

[0021] See Figure 1 , Figure 1 This is a flowchart illustrating a robot teleoperation control method provided in an embodiment of this application.

[0022] To achieve motion remapping control from the source robot to the target robot while reducing the need for modification to the teleoperation platform, this application provides a robot teleoperation control method, including steps S101 to S104.

[0023] Step S101: Based on the skeleton mapping information, the node pose data corresponding to multiple source motion nodes in the motion data of the source robot are converted into skeleton pose data corresponding to the interface humanoid skeleton according to the skeleton input format. The data format of the interface humanoid skeleton is consistent with the skeleton input format used by the teleoperation platform for the motion data provided by the body tracking device.

[0024] Step S102: If the first coordinate system of the motion data of the source robot does not match the second coordinate system of the motion data provided by the body tracking device received by the teleoperation platform, a first coordinate transformation is performed on the skeleton pose data in the first coordinate system to obtain the skeleton pose data in the second coordinate system.

[0025] Step S103: Generate teleoperation input data based on the skeleton pose data in the second coordinate system; wherein the teleoperation input data conforms to the preset input data structure adopted by the teleoperation platform for the motion data provided by the body tracking device.

[0026] Step S104: Input the teleoperation input data to the teleoperation platform so that the teleoperation platform performs remapping processing on the teleoperation input data for the target robot according to the processing flow of motion data provided for the body tracking device, and obtains robot control data, which is used to control the movement of the target robot.

[0027] Robot teleoperation control methods can refer to methods for converting motion data of a source robot into teleoperation input data that can be recognized and processed by a teleoperation platform, and generating robot control data for a target robot through the teleoperation platform.

[0028] The source robot can refer to the robot that provides motion data. The source robot can be a real robot or a robot in a simulated environment; it can be a humanoid robot, a bipedal robot, a robot with upper and lower limbs, or any other robot capable of outputting pose data from multiple nodes. The source robot and the target robot can have different models, different degrees of freedom, different numbers of nodes, or different skeleton topologies.

[0029] The robot teleoperation control method does not require the source robot and the target robot to have the same model, degrees of freedom, number of nodes, or skeleton topology. Instead, it enables the motion of the source robot to be transferred to the target robot through the teleoperation platform by adapting the skeleton format, coordinate system, and input data structure.

[0030] Motion data from the source robot refers to data used to characterize the motion state of the source robot. This motion data can include the pose data of multiple source motion nodes. The motion data can originate from simulation runs, trajectory playback, reinforcement learning outputs, data acquisition from real robot sensors, or third-party motion data exports.

[0031] Source motion nodes refer to nodes in the motion data of the source robot that represent the local structural motion state of the source robot. Source motion nodes can correspond to link nodes, rigid body nodes, skeleton nodes, body nodes, joint connection nodes, or other nodes in the robot model that can output position and orientation.

[0032] Node pose data can refer to the position and orientation data of the source moving node. Position data can be used to characterize the spatial position of the source moving node in the coordinate system, and orientation data can be used to characterize the orientation of the source moving node in the coordinate system. Orientation data can be represented using quaternions, rotation matrices, Euler angles, or other orientation representation methods.

[0033] Skeleton mapping information refers to the mapping basis used to convert multiple source motion nodes from the source robot side into skeleton nodes from the interface humanoid skeleton side. Skeleton mapping information can be manually configured, template-configured, or generated based on node identifiers, node topological relationships, and node geometric information in the robot model. For example, skeleton mapping information can instruct a skeleton node to directly use the node pose data of a source motion node, or it can instruct a skeleton node to be generated based on parent associated nodes and local offset information, or it can instruct a skeleton node to be generated based on the interpolation results between two source motion nodes.

[0034] The skeletal pose data corresponding to the interface humanoid skeleton refers to the skeletal input format that conforms to the motion data provided by the teleoperation platform to the body tracking device. The interface humanoid skeleton is not derived from the robot's mechanical skeleton, but rather a data skeleton constructed for the input interface of the teleoperation platform. The interface humanoid skeleton can include skeletal nodes such as the head, neck, torso, shoulders, elbows, wrists, hands, hips, knees, ankles, and feet. The interface humanoid skeleton can also be configured as a set of skeletal nodes with a specified number, order, or identifier, according to the input requirements of the teleoperation platform.

[0035] The skeleton input format refers to the data format requirements for skeleton data when a teleoperation platform receives motion data from a body tracking device. The skeleton input format can include at least one of the following: the number of skeleton nodes, the order of skeleton nodes, the identifier of skeleton nodes, the position field of skeleton nodes, the pose field of skeleton nodes, and the arrangement of skeleton data in the teleoperation input data. By setting the number and order of skeleton nodes, the teleoperation platform can read skeleton data according to the expected node set and arrangement. By setting the skeleton node identifier, the teleoperation platform can distinguish different skeleton nodes, reducing the risk of node mismatch. By setting the position and pose fields of skeleton nodes, each skeleton node can carry complete spatial pose information. By setting the arrangement of skeleton data in the teleoperation input data, the converted skeleton pose data can conform to the data parsing rules of the teleoperation platform, thereby improving the compatibility between the teleoperation input data and the teleoperation platform's body tracking input interface.

[0036] Skeleton pose data refers to the position and orientation data of each skeletal node in the interface humanoid skeleton. Skeleton pose data is obtained by mapping the node pose data from the motion data of the source robot and is used to represent the spatial orientation of the interface humanoid skeleton at a given moment.

[0037] The first coordinate system can refer to the coordinate system in which the motion data of the source robot resides. The first coordinate system can be the coordinate system used by the simulation environment, robot model, trajectory data source, or robot control environment. For example, the first coordinate system can be defined with the vertical direction as the Z-axis, or it can be other coordinate definitions agreed upon by the source robot's motion data generation side.

[0038] The second coordinate system can refer to the coordinate system used by the teleoperation platform to receive motion data provided by the body tracking device. The second coordinate system can be the coordinate system agreed upon by the body tracking device interface, or it can be the input coordinate system configured by the teleoperation platform for the motion data provided by the body tracking device. For example, the second coordinate system can be defined with the vertical direction as the Y-axis, or it can be other coordinate definitions that match the motion data provided by the body tracking device.

[0039] Mismatch between the first and second coordinate systems can refer to differences in coordinate axis directions, coordinate axis naming, coordinate axis order, attitude representation references, or attitude data arrangement. For example, the first coordinate system can use the Z-axis as the height direction, while the second coordinate system can use the Y-axis as the height direction; the attitude quaternions in the first coordinate system can be arranged in (w, x, y, z), while the attitude quaternions in the second coordinate system can be arranged in (x, y, z, w).

[0040] The first coordinate transformation refers to the coordinate transformation process performed on skeleton pose data when the first coordinate system and the second coordinate system do not match. The first coordinate transformation can be used to convert skeleton pose data in the first coordinate system to skeleton pose data in the second coordinate system. For example, the first coordinate transformation can convert the position (x, y, z) to (y, z, x) and perform a pose transformation on the pose quaternion that matches the axial permutation.

[0041] Skeleton pose data in the second coordinate system refers to skeleton position and posture data that meet the requirements of the second coordinate system after transformation by the first coordinate system. Skeleton pose data in the second coordinate system can be used to generate teleoperation input data, enabling the teleoperation platform to read and process motion data provided by the body tracking device.

[0042] Teleoperation input data refers to data input to a teleoperation platform that can be recognized and processed by the teleoperation platform according to the motion data processing flow provided by the body tracking device. The purpose of teleoperation input data is to enable the teleoperation platform to process the skeleton data corresponding to the motion data of the source robot as motion data provided by the body tracking device.

[0043] A preset input data structure refers to the pre-organized data structure adopted by the teleoperation platform for motion data provided by the body tracking device. The preset input data structure can define the skeleton data items, controller data items, and time data items in the teleoperation input data. For example, the preset input data structure may include multiple skeleton data items, each corresponding to a skeleton node and carrying the position and orientation of that skeleton node.

[0044] A teleoperation platform can refer to a platform used to generate robot control data based on motion data or teleoperation input data provided by a body tracking device. The teleoperation platform may have functions such as skeleton normalization, calibration, inverse kinematics solving, controller merging, and target robot control data generation. In this embodiment, the teleoperation platform does not need to distinguish whether the teleoperation input data comes from the body tracking device or is converted from motion data of the source robot; instead, it processes the data according to the processing flow for motion data provided by the body tracking device.

[0045] The motion data provided by the body tracking device can refer to data collected or generated by the body tracking device that characterizes the motion state of the human body or human skeleton. The body tracking device may include a head-mounted display, a controller, a body tracking device, or other data devices capable of outputting the pose of the human skeleton. The teleoperation platform could originally perform remapping processing on the motion data provided by the body tracking device. In this embodiment, by generating teleoperation input data that conforms to the motion data input requirements provided by the body tracking device, the motion data from the source robot can be reused in this processing flow.

[0046] Remapping processing refers to the process by which a teleoperation platform converts teleoperation input data into executable control data for the target robot. The purpose of remapping processing is to convert the motion represented by the humanoid skeleton interface into robot control data adapted to the target robot's structure and control interface.

[0047] Robot control data refers to data used to control the movement of a target robot. Robot control data can include the target robot's joint angles, joint velocities, joint torques, end-effector pose, motion trajectory, control commands, or other control information that can be executed by the target robot.

[0048] In this embodiment, the motion data of the source robot and the motion data originally received by the teleoperation platform from the body tracking device may differ in skeleton structure and coordinate system definition. Directly inputting the motion data of the source robot into the teleoperation platform would make it difficult for the teleoperation platform to complete the remapping process of the target robot according to the processing flow for motion data provided by the body tracking device. Based on this, this embodiment first uses skeleton mapping information to convert the node pose data corresponding to multiple source motion nodes of the source robot into skeleton pose data of an interface humanoid skeleton that conforms to the skeleton input format of the teleoperation platform, so that the motion data of the source robot is converted into a body tracking skeleton form that the teleoperation platform can recognize at the skeleton structure level. In the case of mismatch between the first coordinate system and the second coordinate system, a first coordinate transformation is performed on the skeleton pose data to convert the skeleton pose data in the first coordinate system into skeleton pose data in the second coordinate system, so that the skeleton pose data adapts to the coordinate requirements used by the teleoperation platform to receive the motion data provided by the body tracking device at the coordinate expression level. Subsequently, teleoperation input data conforming to a preset input data structure is generated based on the skeleton pose data in the second coordinate system. The teleoperation input data is then input to the teleoperation platform, which performs remapping processing on the teleoperation input data for the target robot according to the motion data processing flow provided by the body tracking device, thereby obtaining robot control data for controlling the movement of the target robot.

[0049] This embodiment converts the node pose data corresponding to multiple source motion nodes of the source robot into skeleton pose data corresponding to the interface humanoid skeleton based on skeleton mapping information. This converts the motion node structure on the source robot side into a skeleton input format that the teleoperation platform can receive, thereby improving the data access difficulties caused by the inconsistency between the source robot skeleton structure and the teleoperation platform's input skeleton structure. By performing a first coordinate transformation on the skeleton pose data when the first and second coordinate systems are mismatched, the skeleton pose data can be converted into skeleton pose data in the second coordinate system, thereby reducing axial mismatch and attitude deviation caused by the difference between the motion data coordinate system of the source robot and the motion data coordinate system provided by the body tracking device. By generating teleoperation input data conforming to a preset input data structure based on the skeleton pose data in the second coordinate system, the teleoperation platform can process the data corresponding to the source robot according to the processing flow for motion data provided by the body tracking device, thereby reducing the need for modification to the teleoperation platform's input interface and remapping process. By inputting the teleoperation input data to the teleoperation platform and having the teleoperation platform generate robot control data, the motion of the source robot can be transferred to the target robot via the teleoperation platform, thereby improving the convenience of motion reuse and motion transfer between heterogeneous robots.

[0050] This embodiment addresses the problem that motion data from the source robot is difficult to directly reuse in the remapping process of motion data provided by the body tracking device on the teleoperation platform. It adopts the idea of ​​converting the motion data of the source robot into a format that conforms to the skeleton input format, coordinate system requirements and preset input data structure of the teleoperation platform. This enables motion remapping control from the source robot to the target robot while reducing the need for modification to the teleoperation platform.

[0051] This application embodiment eliminates the need for a separate motion solution process for the target robot on the source robot side, and also eliminates the need to add a robot skeleton input branch within the teleoperation platform that is independent of the motion data processing process provided by the body tracking device. Instead, it converts the motion data of the source robot into teleoperation input data that the teleoperation platform can recognize according to the motion data processing process provided by the body tracking device through skeleton mapping information, a first coordinate transformation, and adaptation of a preset input data structure. This reduces the need for the source robot side and the teleoperation platform side to maintain multiple sets of adaptation logic, and reduces the impact of multi-level data conversion on the real-time performance of remapping and the consistency of motion semantics.

[0052] For example, the source robot is a humanoid robot, and the target robot is another humanoid robot with a different structure. The motion data of the source robot includes the pose data of 29 source motion nodes, each of which includes 3D position and quaternion pose. The teleoperation platform uses a skeleton input format for the motion data provided by the body tracking device, which includes 24 skeleton nodes. The robot teleoperation control method can convert the 29 source motion nodes in the motion data of the source robot into 24 skeleton nodes of the interface humanoid skeleton according to the skeleton input format based on the skeleton mapping information. For example, the position of the pelvic motion node in the source robot in the first coordinate system is (0.00, 0.00, 0.90), and the pose is (1, 0, 0, 0). This can be converted into the skeletal pose data of the pelvic skeleton node in the interface humanoid skeleton. In the data format of the interface humanoid skeleton, the position of this skeletal pose data is stored in the position field of the pelvic skeleton node, and the pose of this skeletal pose data is stored in the pose field of the pelvic skeleton node. Similarly, the position of the torso motion node in the source robot in the first coordinate system is (0.02, 0.00, 1.25), and the pose is (1, 0, 0, 0). This can be converted into the skeletal pose data of the torso skeleton node in the interface humanoid skeleton. In the data format of the interface humanoid skeleton, the position of this skeletal pose data is stored in the position field of the torso skeleton node, and the pose of this skeletal pose data is stored in the pose field of the torso skeleton node.

[0053] In one example, the first coordinate system uses the Z-axis as the height direction, and the second coordinate system uses the Y-axis as the height direction. When processing motion data provided by the body tracking device in the second coordinate system, the teleoperation platform can perform a second coordinate transformation, which transforms the position data from (x, y, z) to (z, x, y). In this second coordinate transformation, the y-component in the second coordinate system is converted into the z-component in the processing coordinate system, thus achieving a correspondence between the height axes from Y-up to Z-up; simultaneously, the second coordinate transformation also defines the correspondence between other coordinate axes. Based on this second coordinate transformation, the first coordinate transformation can be determined as transforming the position data from (x, y, z) to (y, z, x), so that the combination of the first and second coordinate transformations restores the position data in the first coordinate system. For example, if the position of a skeleton node in the first coordinate system is (0.30, 0.20, 1.10), after the first coordinate transformation, the position in the second coordinate system can be obtained as (0.20, 1.10, 0.30). When the teleoperation platform performs a second coordinate transformation on the position, it can obtain (0.30, 0.20, 1.10), thus restoring the position meaning in the first coordinate system. Attitude data can also undergo attitude transformations that match the position coordinate transformations, such as quaternion transformations and quaternion order adjustments.

[0054] Subsequently, the skeleton pose data of the 24 skeleton nodes in the second coordinate system can be written into the teleoperation input data according to the preset input data structure of the teleoperation platform, along with time information, such as 125000 milliseconds. After receiving the teleoperation input data, the teleoperation platform can perform remapping processing according to the processing flow for motion data provided by the body tracking device to generate target robot control data. For example, the target robot control data may include the target robot's left shoulder joint angle of 15 degrees, left elbow joint angle of 40 degrees, and left wrist joint angle of 8 degrees, thereby controlling the target robot to perform movements corresponding to the motion data of the source robot.

[0055] In some embodiments, the step of converting the node pose data corresponding to multiple source motion nodes in the motion data of the source robot into skeleton pose data corresponding to the interface humanoid skeleton according to the skeleton input format based on skeleton mapping information may include: determining the associated node information and pose generation rule corresponding to each skeleton node among the multiple skeleton nodes included in the interface humanoid skeleton based on the skeleton mapping information; determining the associated node pose data for generating the skeleton node from the node pose data corresponding to the multiple source motion nodes based on the associated node information; and generating the skeleton pose data corresponding to the skeleton node based on the pose generation rule and the associated node pose data.

[0056] Association node information can refer to information used to indicate which source motion nodes should be used when generating a certain skeleton node. For example, for a hand skeleton node, association node information can indicate the use of wrist-source motion nodes; for a spinal mid-skeleton node, association node information can indicate the use of pelvic-source motion nodes and trunk-source motion nodes.

[0057] Pose generation rules refer to the rules for generating skeleton pose data for skeleton nodes based on the pose data of associated nodes. Pose generation rules can include direct generation rules, offset generation rules, interpolation generation rules, etc. Direct generation rules can be used to directly use the pose data of a source motion node as the skeleton pose data for a skeleton node; offset generation rules can be used to generate skeleton pose data for a skeleton node based on the pose of the parent associated node and local offsets; interpolation generation rules can be used to generate skeleton pose data for a skeleton node based on position interpolation and attitude interpolation between two source motion nodes.

[0058] Associated node pose data can refer to the pose data selected from the pose data of multiple source motion nodes based on associated node information, used to generate a certain skeleton node. Associated node pose data can include the pose data of one source motion node, or it can include the pose data of two or more source motion nodes.

[0059] This embodiment determines the associated node information and pose generation rules for each skeleton node based on skeleton mapping information. This allows for different data sources and generation methods to be used for different skeleton nodes, thereby improving the adaptability of motion data from the source robot to the interface humanoid skeleton. By determining the associated node pose data from the node pose data corresponding to multiple source motion nodes based on associated node information, valid source motion node pose criteria can be selected for each skeleton node. This improves the difficulty in skeleton data generation caused by inconsistencies in the number of source robot nodes, node naming, or node topology with the interface humanoid skeleton. By generating skeleton pose data corresponding to skeleton nodes based on pose generation rules and associated node pose data, different generation methods such as direct correspondence, local offset, and node interpolation can be unified into the skeleton pose data generation process, thereby improving the stability and scalability of the interface humanoid skeleton generation process. This embodiment addresses the problem of source robot motion nodes being difficult to directly match with the teleoperation platform interface humanoid skeleton by adopting the concept of determining associated node information and pose generation rules for skeleton nodes based on skeleton mapping information. This allows for the stable conversion of source robot motion data into skeleton pose data that conforms to the teleoperation platform skeleton input format.

[0060] In some embodiments, the process of determining the skeleton mapping information may include: determining candidate association relationships between skeleton nodes in the interface humanoid skeleton and the source motion nodes based on at least one of the node identifiers, node topological relationships, and node geometric information of each source motion node in the robot model corresponding to the source robot; and determining the skeleton mapping information based on the candidate association relationships.

[0061] The robot model parameters corresponding to the source robot can refer to model data used to describe the structural information of the source robot. Robot model parameters can include node identifiers, node topological relationships, and node geometric information of multiple motion nodes in the source robot. Robot model parameters can originate from simulation models, robot description files, 3D model files, or other data files capable of representing the robot's structure. Robot model parameters provide the structural and geometric basis for the formation of skeleton mapping information.

[0062] The node identifier of a source motion node can refer to information used to identify the source motion node. The node identifier can be a node name, node number, node path, node code, or other markers that can uniquely or relatively uniquely identify the source motion node.

[0063] Node topology can refer to the structural connection relationships between multiple source motion nodes in a source robot. Node topology can include at least one of the following: upstream / downstream relationships, parent / child relationships, adjacent relationships, link connections, and branch connections between source motion nodes.

[0064] Node geometric information can refer to the geometric description information of the source motion node or the associated structure of the source motion node in the source robot. Node geometric information can include at least one of the following: node position, geometric center, collider center, inertial center, bounding box center, model size, local coordinate system orientation, and distance between nodes.

[0065] Candidate associations refer to the associations that may be used to generate skeleton nodes, determined based on the robot model parameters corresponding to the source robot during the process of forming skeleton mapping information. Candidate associations can represent a skeleton node being directly associated with a source motion node, a skeleton node being associated with a combination of multiple source motion nodes, or a skeleton node being associated with a source motion node and its geometric offset relationship.

[0066] Determining skeleton mapping information based on candidate associations can refer to confirming, filtering, adjusting, or solidifying candidate associations to obtain skeleton mapping information for subsequent skeleton pose data conversion. This process may include selecting a target association from multiple candidate associations, supplementing candidate associations with information such as node identifiers, geometric parameters, or generation methods, or saving candidate associations as mapping configurations.

[0067] This embodiment determines candidate associations based on at least one of the following: node identifier, node topology, and node geometry information of each source motion node in the robot model parameters corresponding to the source robot. It analyzes the probability of correspondence between source motion nodes and skeleton nodes from at least one dimension, such as node name, structural connection, and geometric position, thereby improving the accuracy and operability of skeleton mapping information formation. By determining skeleton mapping information based on candidate associations, the correspondence between skeleton nodes and source motion nodes, as well as the corresponding pose generation methods, can be saved as a mapping basis for subsequent use. This allows for direct determination of the source motion node and pose generation method corresponding to each skeleton node based on the skeleton mapping information each time the source robot's motion data is processed, eliminating the need to re-determine the correspondence between skeleton nodes and source motion nodes one by one based on the robot model parameters. This reduces repetitive matching calculations and improves the consistency of skeleton pose data conversion. By utilizing skeleton mapping information to support the conversion of source robot motion data to skeleton pose data corresponding to the interface humanoid skeleton, the data adaptation difficulties caused by inconsistencies between the source robot skeleton structure and the input skeleton structure of the teleoperation platform can be improved. This embodiment addresses the problem that it is difficult to directly correspond the source motion nodes in the source robot to the skeleton nodes in the interface humanoid skeleton. It adopts the concept of forming skeleton mapping information based on at least one of the node identifiers, node topological relationships, and node geometric information in the robot model parameters. This provides a stable and reusable mapping basis for converting the motion data of the source robot into skeleton pose data corresponding to the interface humanoid skeleton that can be recognized by the teleoperation platform.

[0068] For example, the robot model parameters corresponding to the source robot include 29 source motion nodes, and the interface humanoid skeleton includes 24 skeleton nodes. The robot model parameters can record the node identifier, node topology, and node geometry information of each source motion node. Node identifiers can include pelvis, torso link, left wrist link, left ankle link, etc.; node topology can indicate that pelvis and torso link are adjacent, and left shoulder link, left elbow link, and left wrist link are located in the same upper limb link; node geometry information can indicate that there is a geometric center near the left wrist link to represent the hand region.

[0069] Based on the aforementioned robot model parameters, candidate associations can be identified between the pelvic skeleton nodes and pelvic source motion nodes in the interface humanoid skeleton; between the torso skeleton nodes and torso link source motion nodes; and between the hand skeleton nodes and the left wrist link source motion node and its nearby geometric center. If the intermediate spine skeleton node in the interface humanoid skeleton is located between the pelvic skeleton nodes and the torso skeleton nodes, candidate associations can also be identified between the intermediate spine skeleton node and the pelvic and torso link source motion nodes. Subsequently, skeleton mapping information can be determined based on these candidate associations, so that the skeleton mapping information indicates the source motion nodes or combinations of source motion nodes corresponding to the 24 skeleton nodes respectively. When processing the motion data of the source robot, this skeleton mapping information can be directly called to convert the node pose data of the 29 source motion nodes into skeleton pose data of the 24 skeleton nodes.

[0070] In some embodiments, the process of determining the pose generation rule corresponding to the skeleton node may include: when there is a direct correspondence between the skeleton node and a source motion node of the source robot, determining the pose generation rule corresponding to the skeleton node as a direct generation rule.

[0071] In some embodiments, the process of determining the pose generation rule corresponding to the skeleton node may include: when the skeleton node does not directly correspond to any source motion node of the source robot and can be determined through the parent associated node and local offset information, the pose generation rule corresponding to the skeleton node is determined as an offset generation rule.

[0072] In some embodiments, the process of determining the pose generation rule corresponding to the skeleton node may include: when the skeleton node does not directly correspond to any source motion node of the source robot and is located between two source motion nodes of the source robot, determining the pose generation rule corresponding to the skeleton node as an interpolation generation rule.

[0073] Different skeleton nodes have different correspondences with the source robot. Therefore, different pose generation rules can be determined for different skeleton nodes to adapt to the differences in node structure between the source robot and the interface humanoid skeleton.

[0074] A direct correspondence means that a skeletal node in the interface humanoid skeleton can find a semantically corresponding source motion node in the source robot. For example, a pelvic skeletal node in the interface humanoid skeleton can correspond to a pelvic source motion node in the source robot, and a left elbow skeletal node in the interface humanoid skeleton can correspond to a left elbow source motion node in the source robot. When a direct correspondence exists, the skeletal pose data of the skeletal nodes can be directly generated using the node pose data of the corresponding source motion node.

[0075] Direct generation rules refer to pose generation rules used when there is a direct correspondence between a skeleton node and a source motion node. When using direct generation rules, the position data of the corresponding source motion node can be used as the position data of the skeleton node, and the pose data of the corresponding source motion node can be used as the pose data of the skeleton node. Direct generation rules are applicable when the source robot already has a source motion node with the same or similar semantics as the interface humanoid skeleton node.

[0076] A parent associated node refers to the source motion node used in the offset generation rule to provide the basic position and basic orientation. A parent associated node can be a source motion node that is adjacent to, close to, or has a structural support relationship with a skeleton node. For example, if the source robot does not have an independent hand center node, the wrist source motion node can be used as the parent associated node of the hand skeleton node; similarly, if the source robot does not have an independent foot center node, the ankle source motion node can be used as the parent associated node of the foot skeleton node.

[0077] Local offset information refers to the positional offset of a skeleton node relative to its parent node in the parent node's local coordinate system. The purpose of local offset information is to generate the skeleton nodes required for the interface humanoid skeleton based on the parent node and the local offset, when the source robot does not have a directly corresponding node.

[0078] Offset generation rules refer to pose generation rules used when a skeleton node does not directly correspond to any source motion node of the source robot, and can be determined through parent associated nodes and local offset information. Offset generation rules are applicable to skeleton nodes such as the hand center, foot center, head geometric center, and neck base point, which do not directly correspond to any source motion node of the source robot and are located between two source motion nodes of the source robot.

[0079] Interpolation generation rules refer to pose generation rules used when a skeleton node does not directly correspond to any source motion node of the source robot and is located between two source motion nodes of the source robot. Interpolation generation rules are applicable to intermediate nodes of the spine, clavicle nodes, or other skeleton nodes located between two source motion nodes.

[0080] The phrase "skeletal node located between two source motion nodes" can refer to the semantic or geometric location of a skeleton node within the interface humanoid skeleton being between two source motion nodes. This "between" can refer to spatial locations or skeleton topological links. For example, the spine mid-skeleton node in the interface humanoid skeleton could be located between a pelvic source motion node and a torso source motion node, and the clavicle skeleton node could be located between a torso source motion node and a shoulder source motion node.

[0081] In the process of determining the pose generation rules, we can first determine whether the skeleton node has a directly corresponding source motion node; if there is no directly corresponding source motion node, we can further determine whether the skeleton node can be determined through the parent associated node and local offset information; if the skeleton node is between two source motion nodes, we can determine to use the interpolation generation rule.

[0082] This embodiment determines the pose generation rule of a skeleton node as a direct generation rule when there is a direct correspondence between the skeleton node and a source motion node of the source robot. This allows for the direct generation of skeleton pose data using the semantically corresponding source motion node's node pose data, thus reducing unnecessary computation. When a skeleton node does not directly correspond to any source motion node of the source robot, but can be determined through parent associated nodes and local offset information, the pose generation rule is determined as an offset generation rule. This allows for the supplementation of skeleton nodes lacking direct source nodes in the interface humanoid skeleton based on existing source motion nodes, thereby improving the difficulty of skeleton generation caused by incomplete semantic correspondence between nodes. When a skeleton node does not directly correspond to any source motion node of the source robot, but is located between two source motion nodes of the source robot, the pose generation rule is determined as an interpolation generation rule. This allows for the generation of intermediate skeleton nodes using the position and posture transition relationship between two source motion nodes, thereby improving the continuity and rationality of the interface humanoid skeleton generation. This embodiment addresses the issue of inconsistency between the node structure of the source robot and the node structure of the interface humanoid skeleton. It adopts the concept of determining the pose generation rules based on three relationships: direct correspondence, parent node offset, and inter-node interpolation. This improves the adaptability and stability of converting the motion data of the source robot into the pose data of the interface humanoid skeleton.

[0083] In some embodiments, when the pose generation rule corresponding to the skeleton node is an offset generation rule, the skeleton pose data in the first coordinate system includes the skeleton position data and skeleton posture data of the skeleton node in the first coordinate system. The step of generating skeleton pose data corresponding to the skeleton node based on the pose generation rule and the pose data of the associated nodes may include: determining the parent node pose data corresponding to the parent associated node from the pose data of the associated nodes, wherein the parent node pose data includes the parent node position data and parent node posture data of the parent associated node in the first coordinate system; when the robot model parameters corresponding to the source robot are in the reference posture, determining a preset offset vector of the skeleton node in the local coordinate system of the parent associated node based on the positional relationship between the target geometric point corresponding to the skeleton node and the parent associated node; transforming the preset offset vector from the local coordinate system of the parent associated node to the first coordinate system based on the parent node posture data to obtain the offset of the skeleton node in the first coordinate system; generating skeleton position data of the skeleton node in the first coordinate system based on the parent node position data and the offset of the skeleton node in the first coordinate system; and determining the parent node posture data as the skeleton posture data of the skeleton node in the first coordinate system.

[0084] Parent node pose data can refer to the position and orientation data of the parent associated node. Parent node pose data includes parent node position data and parent node orientation data. Parent node position data is used to provide the starting point for offset calculation, and parent node orientation data is used to transform the preset offset vector in the local coordinate system of the parent associated node to the first coordinate system.

[0085] In some embodiments, when the robot model parameters corresponding to the source robot are in a reference posture, a preset offset vector of the skeleton node in the local coordinate system of the parent associated node can be determined based on the positional relationship between the target geometric point corresponding to the skeleton node and the parent associated node.

[0086] A reference pose can refer to the robot pose used to determine a preset offset vector. The reference pose can be a neutral standing pose, a calibration pose, a model default pose, or any other pose that facilitates determining geometric offset relationships. When the robot model parameters corresponding to the source robot are in the reference pose, the positional relationship between the target geometric point corresponding to the skeleton node and the parent associated node is relatively stable, making it easier to determine the preset offset vector in the local coordinate system of the parent associated node.

[0087] A target geometric point refers to the geometric location of a skeletal node that is desired to be represented in the humanoid skeleton interface. Target geometric points can be determined based on the robot model parameters, including the center of the geometric body, the center of the collider, the center of the inertial body, the joint origin, the bounding box center, model dimensions, manual measurements, or debugging results. For example, the target geometric point for a hand skeletal node could be the geometric center near the palm, and the target geometric point for a foot skeletal node could be the center of the sole or a location near the toes.

[0088] A local coordinate system can refer to a coordinate system established with reference to a parent node. The local coordinate system of the parent node rotates as the parent node's orientation changes. A preset offset vector, located in the parent node's local coordinate system, represents the fixed geometric offset of the target geometric point relative to the parent node. Because the preset offset vector is defined in the parent node's local coordinate system, it rotates synchronously with the parent node when the parent node rotates.

[0089] A preset offset vector can be a vector representing the positional offset of a target geometric point relative to its parent node in the local coordinate system of the parent node. The preset offset vector can be determined based on the positional relationship between the target geometric point and its parent node. For example, the preset offset vector of a hand skeleton node relative to its wrist parent node could be (0.05, 0.00, 0.00), indicating that the hand skeleton node is offset by 0.05 meters along the local coordinate axis relative to its wrist parent node.

[0090] The offset in the first coordinate system can refer to the offset obtained after transforming the preset offset vector. The offset in the first coordinate system is in the same coordinate system as the parent node position data, so it can be added to the parent node position data to obtain the skeleton position data of the skeleton node in the first coordinate system.

[0091] In the offset generation rule, skeleton position data can be obtained by adding the parent node position data to the offset in the first coordinate system. In the offset generation rule, skeleton pose data can directly use the parent node pose data. That is, the position of a skeleton node can be offset relative to its parent node, but the orientation of the skeleton node can remain consistent with its parent node, thus reducing the complexity of additional pose estimation.

[0092] This embodiment determines the parent node pose data corresponding to the parent associated node from the associated node pose data, providing a basis for position and pose calculation for skeleton nodes lacking a direct corresponding source motion node, thereby improving the problem that skeleton nodes cannot directly obtain values ​​from a single source motion node. By determining a preset offset vector based on the positional relationship between the target geometric point and the parent associated node, the geometric relationship of the skeleton node relative to the parent associated node can be fixed as a reusable parameter, thereby improving the operability of skeleton node position generation. By transforming the preset offset vector from the local coordinate system of the parent associated node to the first coordinate system based on the parent node pose data, the offset can change synchronously with the pose change of the parent associated node, thereby improving the consistency between the skeleton node position and the motion state of the source robot. By generating skeleton position data based on the parent node position data and the offset in the first coordinate system, and determining the parent node pose data as skeleton pose data, the skeleton pose data required by the interface humanoid skeleton can be generated without adding independent source motion nodes, thereby reducing the adaptation difficulty caused by the inconsistency between the source robot structure and the interface humanoid skeleton structure. This embodiment addresses the issue that skeleton nodes do not directly correspond to any source motion node in the source robot. It adopts the concept of generating skeleton pose data based on parent associated nodes and local offset information, thus supplementing the skeleton node poses in the interface humanoid skeleton that lack direct source motion nodes.

[0093] For example, the left-hand skeleton node in the interface humanoid skeleton does not directly correspond to a single source motion node in the source robot, but the source robot has a left wrist source motion node. In this case, the left wrist source motion node can be identified as the parent associated node of the left-hand skeleton node. Assuming that in the first coordinate system, the position of the left wrist source motion node is (0.32, 0.24, 1.05), and the pose of the left wrist source motion node is a unit quaternion (1, 0, 0, 0). The target geometric point corresponding to the left-hand skeleton node can be the geometric center of the palm, and the preset offset vector of the palm geometric center relative to the left wrist source motion node in the left wrist local coordinate system can be (0.05, 0.00, 0.00).

[0094] During operation, since the pose of the left wrist source motion node is a unit quaternion, the preset offset vector, after being transformed from the left wrist local coordinate system to the first coordinate system, remains (0.05, 0.00, 0.00). Adding the position of the left wrist source motion node (0.32, 0.24, 1.05) to the offset in the first coordinate system (0.05, 0.00, 0.00) yields the skeleton position data of the left hand skeleton node in the first coordinate system (0.37, 0.24, 1.05). Simultaneously, the pose of the left wrist source motion node (1, 0, 0, 0) can be determined as the skeleton pose data of the left hand skeleton node in the first coordinate system. Therefore, even if the source robot does not have an independent left hand source motion node, the left hand skeleton pose data required for the interface humanoid skeleton can still be generated.

[0095] In some embodiments, when the pose generation rule corresponding to the skeleton node is an interpolation generation rule, the skeleton pose data in the first coordinate system may include the skeleton position data and skeleton posture data of the skeleton node in the first coordinate system. Generating the skeleton pose data corresponding to the skeleton node based on the pose generation rule and the associated node pose data may include: determining, from the associated node pose data, first node pose data corresponding to a first associated node and second node pose data corresponding to a second associated node; wherein, the first node pose data includes the first node position data and first node posture data of the first associated node in the first coordinate system, and the second node pose data includes the second node position data and second node posture data of the second associated node in the first coordinate system; determining the interpolation parameter corresponding to the skeleton node based on the type of the skeleton node; performing position interpolation processing on the first node position data and the second node position data based on the interpolation parameter to obtain the skeleton position data of the skeleton node in the first coordinate system; and performing posture interpolation processing on the first node posture data and the second node posture data based on the interpolation parameter to obtain the skeleton posture data of the skeleton node in the first coordinate system.

[0096] The first and second associated nodes can refer to two source motion nodes used to jointly generate the skeleton node. The first and second associated nodes can be located on either side of the skeleton node in spatial position, or at opposite ends of the corresponding semantic location of the skeleton node in the node topology link of the source robot. For example, the first associated node can be a pelvic source motion node, and the second associated node can be a torso source motion node; both can be used to generate the intermediate skeleton node of the spine.

[0097] In some embodiments, the interpolation parameters may be the same or different for different types of skeletal nodes. For example, the types of left-hand skeletal nodes and spinal mid-skeletal nodes are different.

[0098] Interpolation parameters are parameters used to characterize the positional or pose transition ratio of a skeleton node between the first and second associated nodes. Interpolation parameters can be values ​​between 0 and 1. For example, when the interpolation parameter is 0, the skeleton node position can be close to the first associated node; when the interpolation parameter is 1, the skeleton node position can be close to the second associated node; and when the interpolation parameter is 0.5, the skeleton node position can be located at an intermediate position between the first and second associated nodes.

[0099] Position interpolation refers to the process of calculating the position transition between the first node's position data and the second node's position data based on interpolation parameters. Position interpolation can be performed using linear interpolation or a weighted average method.

[0100] Attitude interpolation refers to the process of calculating attitude transitions between the attitude data of the first node and the attitude data of the second node based on interpolation parameters. Attitude interpolation can employ spherical linear interpolation, normalized linear interpolation, or other attitude interpolation methods capable of generating intermediate attitudes.

[0101] This embodiment determines the first node pose data corresponding to the first associated node and the second node pose data corresponding to the second associated node from the associated node pose data. This provides two endpoint pose bases for skeleton nodes located between two source motion nodes, thereby improving the problem of skeleton nodes being unable to directly obtain values ​​when they lack directly corresponding source motion nodes. By determining the interpolation parameters corresponding to the skeleton nodes, the position ratio and posture transition ratio of the skeleton nodes between the first and second associated nodes can be controlled, thereby improving the adjustability of the skeleton node generation process. By performing position interpolation processing on the first and second node position data based on the interpolation parameters, skeleton position data matching the spatial relationship between the two source motion nodes can be generated, thereby improving the rationality of the intermediate skeleton node positions. By performing posture interpolation processing on the first and second node posture data based on the interpolation parameters, skeleton posture data matching the posture transition relationship between the two source motion nodes can be generated, thereby improving the continuity of the posture of the intermediate skeleton nodes. This embodiment addresses the problem that some skeleton nodes in the interface humanoid skeleton are located between two source motion nodes of the source robot and are difficult to generate directly from a single source motion node. It adopts the concept of generating skeleton pose data based on the pose data of two related nodes and interpolation parameters, thereby supplementing the pose of intermediate skeleton nodes in the interface humanoid skeleton and improving the smoothness and adaptability of the conversion of motion data from the source robot to the interface humanoid skeleton.

[0102] For example, in the humanoid skeleton interface, the spine intermediate skeleton node is located between the pelvic motion node and the trunk motion node. The pelvic motion node can be identified as the first associated node, and the trunk motion node as the second associated node. Assume that in the first coordinate system, the position of the pelvic motion node is (0.00, 0.00, 0.90), and its pose is a quaternion (1, 0, 0, 0); the position of the trunk motion node is (0.03, 0.00, 1.26), and its pose is a quaternion (1, 0, 0, 0); the interpolation parameter is 0.5.

[0103] Based on an interpolation parameter of 0.5, position interpolation processing is performed on the positions of the pelvic and torso-derived motion nodes, resulting in the position of the intermediate spinal skeleton node in the first coordinate system as (0.015, 0.00, 1.08). Since the poses of the pelvic and torso-derived motion nodes are both quaternions (1, 0, 0, 0), after pose interpolation processing based on the interpolation parameter of 0.5, the pose of the intermediate spinal skeleton node in the first coordinate system can also be a quaternion (1, 0, 0, 0). Therefore, the pose data of the intermediate spinal skeleton in the interface humanoid skeleton can be generated even when the source robot lacks an independent intermediate spinal motion node.

[0104] In some embodiments, the process of determining the first coordinate transformation may include: determining a second coordinate transformation used by the teleoperation platform to process the motion data provided by the body tracking device, the second coordinate transformation being used to transform the motion data provided by the body tracking device from the second coordinate system to the processing coordinate system of the teleoperation platform; if the processing coordinate system is consistent with the first coordinate system, determining a dual coordinate transformation that matches the second coordinate transformation based on the second coordinate transformation, and determining the dual coordinate transformation as the first coordinate transformation.

[0105] In some embodiments, the first coordinate transformation can be used to convert skeleton pose data in the first coordinate system into skeleton pose data in the second coordinate system, so that the skeleton pose data in the second coordinate system can be restored to the skeleton pose data in the first coordinate system after being processed by the second coordinate transformation.

[0106] The second coordinate transformation refers to the coordinate transformation used by the teleoperation platform when processing motion data provided by the body tracking device. The second coordinate transformation is used to convert the motion data provided by the body tracking device from the second coordinate system to the processing coordinate system of the teleoperation platform. In other words, after receiving the motion data provided by the body tracking device in the second coordinate system, the teleoperation platform can first perform the second coordinate transformation, and then perform subsequent remapping processing based on the transformed motion data provided by the body tracking device. Therefore, the first coordinate transformation needs to be determined in conjunction with the second coordinate transformation to ensure that the skeleton pose data maintains the correct coordinate meaning after entering the teleoperation platform.

[0107] The processing coordinate system refers to the coordinate system used by the teleoperation platform when performing motion data processing and remapping processing provided by the body tracking device. When the processing coordinate system is consistent with the first coordinate system, the first coordinate transformation can be determined based on the dual coordinate transformation of the second coordinate transformation. Consistency between the processing coordinate system and the first coordinate system means that the processing coordinate system used by the teleoperation platform during remapping processing has the same coordinate axis meaning, coordinate axis direction, and attitude expression reference as the first coordinate system containing the motion data of the source robot. For example, if the motion data of the source robot is located in a coordinate system with the Z-axis as the height direction, and the teleoperation platform also uses a processing coordinate system with the Z-axis as the height direction during remapping processing, then the processing coordinate system can be considered consistent with the first coordinate system.

[0108] Dual coordinate transformation refers to a coordinate transformation that complements the second coordinate transformation. It is used to transform skeleton pose data from the first coordinate system to the second coordinate system, and then, after processing by the second coordinate transformation of the teleoperation platform, restores the coordinate meaning in the first coordinate system. Dual coordinate transformation is used to realize the coordinate meaning conversion of skeleton pose data, and can include axial displacement processing of position data and quaternion transformation processing of attitude data. Quaternion component order adjustment is used to adapt to the preset input data structure of the teleoperation platform and is not part of the dual coordinate transformation itself; after obtaining the skeleton attitude data in the second coordinate system through dual coordinate transformation, the order of quaternion components can be adjusted according to the preset input data structure.

[0109] For position data, if the second coordinate transformation is used to transform the position data in the second coordinate system from (x, y, z) to (z, x, y) in the first coordinate system, then the first coordinate transformation can transform the position data in the first coordinate system from (x, y, z) to (y, z, x) in the second coordinate system. Thus, the position data is first transformed by the first coordinate transformation to obtain (y, z, x), and then transformed by the second coordinate transformation to obtain (x, y, z), thereby restoring the position data in the first coordinate system.

[0110] For attitude data, the components in the quaternion cannot be simply interchanged; a quaternion transformation method compatible with the second coordinate transformation is required. For example, if the attitude transformation quaternion corresponding to the second coordinate transformation is kPerm, the conjugate quaternion of kPerm is kPermConj, and the skeleton attitude data in the first coordinate system is q1, then the first coordinate transformation can yield the skeleton attitude data q2 in the second coordinate system according to q2 = kPermConj × q1 × kPerm. The teleoperation platform then performs the second coordinate transformation according to q1' = kPerm × q2 × kPermConj, which allows it to recover the skeleton attitude data q1 in the first coordinate system.

[0111] If the quaternion arrangement in the motion data of the source robot is (w, x, y, z), while the quaternion arrangement required by the teleoperation platform is (x, y, z, w), then after obtaining the skeleton posture data in the second coordinate system, the order of the quaternion components can be adjusted from (w, x, y, z) to (x, y, z, w) so that the teleoperation platform can read the posture data according to the preset input data structure.

[0112] The dual coordinate transformation is matched with the second coordinate transformation. It can refer to the coordinate transformation determined by the second coordinate transformation, which can restore the coordinate meaning after being combined with the second coordinate transformation.

[0113] Restoring skeleton pose data to the first coordinate system means that the skeleton pose data in the second coordinate system, after undergoing a second coordinate transformation, results in data whose coordinate meaning is consistent with the skeleton pose data in the first coordinate system. This restoration can include the restoration of position axis meaning, attitude orientation meaning, and attitude data arrangement.

[0114] This embodiment clarifies the coordinate processing method of the teleoperation platform when processing motion data provided by the body tracking device by determining the second coordinate transformation. This provides a basis for determining the first coordinate transformation. By determining a matching dual coordinate transformation based on the second coordinate transformation when the processing coordinate system is consistent with the first coordinate system, the first coordinate transformation and the second coordinate transformation on the teleoperation platform side can be matched, thereby reducing the risk of height axis, forward axis, left and right axis mismatch, and attitude direction deviation after the motion data from the source robot is input into the teleoperation platform. By determining the dual coordinate transformation as the first coordinate transformation, the skeleton pose data can be transformed from the first coordinate system to the second coordinate system. After processing by the second coordinate transformation, the skeleton pose data in the second coordinate system can be restored to the skeleton pose data in the first coordinate system, thereby reducing the need to modify the coordinate processing flow of the teleoperation platform. This embodiment addresses the issue that the coordinate system of the motion data from the source robot is inconsistent with the coordinate system of the motion data provided by the body tracking device received by the teleoperation platform, and that the second coordinate transformation of the teleoperation platform may change the motion semantics of the source robot. It adopts a mechanism to determine the dual coordinate transformation based on the second coordinate transformation and use it as the first coordinate transformation. This improves the coordinate consistency of the motion data of the source robot after it is connected to the teleoperation platform, while reusing the original processing flow of the teleoperation platform.

[0115] For example, the first coordinate system is the coordinate system where the motion data from the source robot resides, with the Z-axis as the height direction. The second coordinate system is the coordinate system used by the teleoperation platform when receiving motion data from the body tracking device, with the Y-axis as the height direction. The processing coordinate system of the teleoperation platform is consistent with the first coordinate system, also with the Z-axis as the height direction. When processing the motion data provided by the body tracking device in the second coordinate system, the teleoperation platform uses a second coordinate transformation, which transforms the position data from (x, y, z) to (z, x, y). In this second coordinate transformation, the y-component in the second coordinate system is converted into the z-component in the processing coordinate system to correspond to the height axis. Simultaneously, the second coordinate transformation also determines the correspondence between other coordinate axes.

[0116] Based on this second coordinate transformation, a matching dual coordinate transformation can be determined, and this dual coordinate transformation is used as the first coordinate transformation. For position data, the first coordinate transformation can transform the position data from (x, y, z) to (y, z, x). If the position of a skeleton node in the first coordinate system is (0.30, 0.20, 1.10), after the first coordinate transformation, the position in the second coordinate system can be obtained as (0.20, 1.10, 0.30). After receiving the position (0.20, 1.10, 0.30) in the second coordinate system, the teleoperation platform transforms the position data from (x, y, z) to (z, x, y) according to the second coordinate transformation, obtaining (0.30, 0.20, 1.10), thus restoring the position data in the first coordinate system.

[0117] For attitude data, if the attitude transformation quaternion corresponding to the second coordinate transformation is kPerm, the conjugate quaternion of kPerm is kPermConj, and the skeleton attitude data in the first coordinate system is q1, then the first coordinate transformation can be used to obtain the skeleton attitude data q2 in the second coordinate system according to q2=kPermConj×q1×kPerm. After the teleoperation platform performs the second coordinate transformation based on q1'=kPerm×q2×kPermConj, it can recover the skeleton attitude data q1 in the first coordinate system. Furthermore, if the quaternion arrangement in the motion data of the source robot is (w, x, y, z), and the preset input data structure of the teleoperation platform requires the quaternion arrangement to be (x, y, z, w), then after obtaining the skeleton attitude data in the second coordinate system, the order of the quaternion components can be adjusted from (w, x, y, z) to (x, y, z, w).

[0118] In this embodiment, the first coordinate system can be the coordinate system where the motion data of the source robot is located, the second coordinate system can be the input coordinate system used by the teleoperation platform when receiving motion data from the body tracking device, and the processing coordinate system can be the internal coordinate system used by the teleoperation platform when performing remapping processing on the motion data of the body tracking device. In other words, the skeleton pose data output by the source robot is initially located in the first coordinate system, the teleoperation input data needs to be input into the teleoperation platform in the form of the second coordinate system, and the teleoperation platform then transforms the data in the second coordinate system to the processing coordinate system through the second coordinate transformation in order to perform subsequent remapping processing.

[0119] In some embodiments, when the processing coordinate system is inconsistent with the first coordinate system, a third coordinate transformation between the processing coordinate system and the first coordinate system can be determined, and a first coordinate transformation can be determined based on the second and third coordinate transformations. The third coordinate transformation can be used to convert skeleton pose data in the processing coordinate system to skeleton pose data in the first coordinate system; the first coordinate transformation can be used to convert skeleton pose data in the first coordinate system to skeleton pose data in the second coordinate system, so that the skeleton pose data in the second coordinate system, after processing by the second and third coordinate transformations, can be restored to skeleton pose data in the first coordinate system. Therefore, even if the processing coordinate system of the teleoperation platform is not completely consistent with the first coordinate system where the source robot motion data is located, a coordinate correspondence between the processing coordinate system and the first coordinate system can be established through the third coordinate transformation, thereby maintaining the semantic consistency of the coordinates after the source robot motion data enters the teleoperation platform.

[0120] In some embodiments, the preset input data structure may include skeleton data items, controller data items, and time data items. By including skeleton data items, controller data items, and time data items in the preset input data structure, skeleton pose data, controller-compatible data, and time information can be uniformly encapsulated according to the parsing rules of the teleoperation platform, thereby improving the structural integrity of the teleoperation input data and facilitating the reuse of the processing flow for motion data of body tracking devices by the teleoperation platform.

[0121] In some embodiments, generating teleoperation input data based on skeleton pose data in the second coordinate system may include: determining skeleton data items in the preset input data structure based on skeleton node identifiers corresponding to multiple skeleton nodes in the interface humanoid skeleton; writing skeleton pose data in the second coordinate system into the corresponding skeleton data items according to the skeleton node identifiers to obtain the filled skeleton data items; filling the controller data items according to controller compatibility data, wherein the controller compatibility data corresponds to the control fields in the motion data provided by the body tracking device; obtaining time information corresponding to the motion data and writing the time information into the time data items; and generating the teleoperation input data based on the filled skeleton data items, the controller data items, and the time data items.

[0122] In some embodiments, the process of determining the controller compatible data may include: in the absence of valid controller input data corresponding to a control field, determining placeholder controller data used to characterize the existence of a controller but without control input as the controller compatible data; and in the presence of valid controller input data corresponding to a control field, determining the controller compatible data based on the valid controller input data.

[0123] Skeleton node identifiers refer to information used to identify individual skeleton nodes in the humanoid skeleton of the interface. Skeleton node identifiers can be skeleton node numbers, skeleton node names, skeleton node codes, or other markers that can distinguish different skeleton nodes. Based on the skeleton node identifiers, the skeleton pose data of different skeleton nodes can be written into the corresponding skeleton data items in a preset input data structure.

[0124] A skeleton data item refers to a data item in a preset input data structure used to carry the data of a single skeleton node. One skeleton data item can correspond to one skeleton node and can include a skeleton node identifier, a skeleton position field, and a skeleton pose field. For example, when the interface humanoid skeleton includes 24 skeleton nodes, the preset input data structure can determine 24 skeleton data items, which are used to write the position data and pose data of the 24 skeleton nodes in the second coordinate system.

[0125] Skeleton input data refers to the data formed by writing the skeleton pose data in the second coordinate system into the corresponding skeleton data items according to the skeleton node identifiers. Skeleton input data can serve as the skeleton portion of teleoperation input data, representing the overall skeleton pose of the interface humanoid skeleton at a given moment. Skeleton input data may include multiple skeleton data items.

[0126] Controller data items refer to data items in a preset input data structure used to carry control field-related data. Controller data items can correspond to left controller fields, right controller fields, handle fields, joystick fields, trigger fields, or button fields in the motion data provided by the body tracking device. By filling controller-compatible data into controller data items, the teleoperation input data can have a control field structure corresponding to the motion data provided by the body tracking device. This facilitates the teleoperation platform in reading the teleoperation input data according to the parsing rules for the motion data provided by the body tracking device. Furthermore, when the motion data from the source robot does not contain actual controller input, placeholder controller data can be used to represent the presence of a controller but without control input, thereby improving the structural integrity of the teleoperation input data.

[0127] Controller-compatible data refers to data used to populate controller data items and enable the teleoperation platform to process the motion data input structure provided by the body tracking device. Controller-compatible data can be placeholder controller data or data determined based on valid controller input data. The function of controller-compatible data is not necessarily to trigger actual controller operation, but rather to enable the teleoperation platform to recognize the existence of controller data items, thereby continuing to perform controller merging processing, debugging processing, or subsequent input expansion processing.

[0128] Valid controller input data refers to real or usable controller inputs that can be used to generate controller-compatible data. Valid controller input data can include button states, joystick values, trigger values, controller attitudes, controller identifiers, etc. Valid controller input data can come from real controllers, simulation inputs, script inputs, or other control input sources. If valid controller input data exists, controller-compatible data can be generated based on it, enabling the teleoperated input data to carry the meaning of the actual controller inputs.

[0129] Placeholder controller data refers to data used to populate controller data items when no valid controller input data exists. Placeholder controller data can be used to indicate that a controller exists but there is no control input. For example, placeholder controller data may include left and right controller data, with button fields all in an inactive state, and joystick and trigger fields both at zero values. Using placeholder controller data, the teleoperation platform can process situations where a controller exists but there is no control input, based on motion data provided by the body tracking device.

[0130] Time information refers to the time data corresponding to the motion data of the source robot, used to populate the time data items in the preset input data structure. Time information may include timestamps, sampling timestamps, motion frame timestamps, input generation timestamps, or other information used to characterize the data's temporal sequence. Time information can be expressed in units such as milliseconds, microseconds, or nanoseconds.

[0131] This embodiment generates or determines skeleton data items in a preset input data structure based on skeleton node identifiers. This allows the skeleton pose data of each skeleton node to be written to its corresponding data position, thereby improving the accuracy of skeleton data organization in teleoperation input data. By writing skeleton pose data in the second coordinate system into corresponding skeleton data items according to skeleton node identifiers, skeleton input data conforming to the teleoperation platform's reading rules can be formed, thus improving processing deviations caused by disordered skeleton data order or incorrect node correspondence. By generating controller-compatible data to fill controller data items, the integrity of the input structure used by the teleoperation platform for motion data provided by the body tracking device can be maintained, thereby reducing incomplete controller merging processing problems caused by missing controller data items. By determining placeholder controller data as controller-compatible data when no valid controller input data exists, it can be shown that the controller exists but there is no control input, allowing the teleoperation platform to process the body tracking input state as if there were no controller operation. By determining controller-compatible data based on valid controller input data when valid controller input data exists, subsequent controller input injection can be supported, thereby improving the scalability of teleoperation input data. By acquiring the time information corresponding to the motion data of the source robot, and generating teleoperation input data based on skeleton input data, controller compatibility data, and time information, the teleoperation input data can carry skeleton, controller, and timing information, thereby improving the data integrity when the teleoperation platform processes the motion data of the source robot. This embodiment addresses the problem that the skeleton pose data obtained from the motion data of the source robot cannot directly meet the preset input data structure of the teleoperation platform. It adopts the concept of writing skeleton data items according to skeleton node identifiers, filling controller data items, and encapsulating time information, so that the motion data of the source robot can be input into the teleoperation platform using the data structure adopted by the teleoperation platform for motion data provided by the body tracking device.

[0132] For example, the humanoid skeleton of the interface includes 24 skeleton nodes, with node identifiers ranging from 0 to 23. The preset input data structure includes 24 skeleton data items, 2 controller data items, and 1 time information item. For the left foot skeleton node with node identifier 10, its position in the second coordinate system can be (0.10, 0.05, 0.23), and its pose can be a quaternion (0.00, 0.00, 0.00, 1.00). The robot teleoperation control method can determine the skeleton data item corresponding to skeleton node identifier 10 in the preset input data structure and write the position (0.10, 0.05, 0.23) and pose (0.00, 0.00, 0.00, 1.00) into this skeleton data item. The remaining 23 skeleton nodes can be written into their corresponding skeleton data items according to their respective node identifiers to obtain the skeleton input data.

[0133] Furthermore, if no valid controller input data exists, placeholder controller data can be generated for the left and right controllers. For example, the buttons on both the left and right controllers may be in an untriggered state, the joystick horizontal and vertical values ​​may both be 0, and the trigger value may be 0. If the time information corresponding to the motion data of the source robot is 125,000 milliseconds, teleoperation input data can be generated based on the skeleton input data formed by 24 skeleton data items, the controller compatible data formed by 2 controller data items, and the time information of 125,000 milliseconds. After receiving this teleoperation input data, the teleoperation platform can read the skeleton data items, controller data items, and time information according to the preset input data structure for the motion data provided by the body tracking device.

[0134] In some embodiments, when continuously receiving motion data from the source robot, inputting the teleoperation input data to the teleoperation platform may include: in response to receiving motion data from the source robot this time, determining the time interval between the current reception time and the last time the teleoperation input data was input to the teleoperation platform; if the time interval is greater than or equal to a preset minimum input interval, generating the teleoperation input data based on the currently received motion data, and inputting the teleoperation input data to the teleoperation platform.

[0135] In some embodiments, the method may further include: when the time interval is less than the preset minimum input interval, skipping inputting teleoperation input data to the teleoperation platform based on the motion data received this time, skipping caching the motion data received this time as input data, and skipping adding the motion data received this time to the input queue.

[0136] Continuous reception of motion data from the source robot can refer to the robot teleoperation control method acquiring motion data from the source robot at multiple consecutive moments. The motion data from the source robot can arrive continuously according to the simulation stepping frequency, trajectory playback frequency, robot state sampling frequency, or communication reception frequency. For example, the source robot can output motion data every 5 milliseconds or every 10 milliseconds.

[0137] The motion data received from the source robot in this processing instance can refer to a set of motion data from the source robot obtained at the current processing moment. This set of motion data may include the node pose data corresponding to multiple source motion nodes. The motion data received from the source robot in this instance is the current processing object used to determine whether teleoperation input data needs to be generated and input.

[0138] The receiving time can refer to the time when the motion data from the source robot was obtained. This time can be determined by a local monotonic clock, a system timestamp, a timestamp carried in the motion data, or the communication reception time. For example, the receiving time could be 1000.00 milliseconds, 1005.00 milliseconds, or 1016.70 milliseconds.

[0139] The last time teleoperation input data was input to the teleoperation platform can refer to the time corresponding to the most recent actual input of teleoperation input data to the teleoperation platform. This time is not the time when motion data from the source robot was last received, but rather the time when the input action was last completed. By recording this time, it can be determined whether the time requirement for inputting teleoperation input data again has been met.

[0140] The time interval can refer to the difference between the current reception time and the last time the remote operation input data was input to the remote operation platform.

[0141] The preset minimum input interval refers to the minimum allowable time interval between two consecutive inputs of teleoperation data to the teleoperation platform. The preset minimum input interval can be determined based on the input frequency that the teleoperation platform can stably process, the load capacity of the communication channel, or the control cycle of the target robot.

[0142] A time interval greater than or equal to a preset minimum input interval means that the current receiving time has been separated from the previous input time by a condition that allows for the input of teleoperation input data again. In this case, teleoperation input data can be generated based on the motion data of the source robot received this time, and the teleoperation input data can be input to the teleoperation platform.

[0143] A time interval less than the preset minimum input interval can mean that the time between the current reception and the previous input has not yet met the condition for allowing the input of teleoperation data again. In this case, teleoperation input data is not input to the teleoperation platform based on the motion data of the source robot received this time, and the motion data of the source robot received this time is not cached as input data, nor is the motion data of the source robot received this time added to the input queue.

[0144] Data to be input refers to data awaiting subsequent input to the teleoperation platform. If motion data that has not reached the preset minimum input interval is cached as data to be input, the teleoperation platform may process expired motion states at later times. This embodiment does not cache the motion data received from the source robot as data to be input, thus reducing the likelihood of the teleoperation platform consuming expired motion data.

[0145] The input queue can refer to a queue structure used to store multiple sets of data waiting to be input to the teleoperation platform. If the frequency of motion data generation from the source robot is higher than the input frequency of the teleoperation platform, the input queue may continuously accumulate older motion data. In this embodiment, the motion data from the source robot received this time is not added to the input queue, which can limit the timing delay caused by queuing for the input of expired motion data.

[0146] This embodiment determines the time interval between the current reception time and the last time teleoperation input data was input to the teleoperation platform in response to the received motion data from the source robot. This allows for judgment based on the actual input timing to determine whether the conditions for re-input are met, thereby improving the controllability of the teleoperation input data input rhythm. By generating teleoperation input data based on the currently received motion data and inputting it to the teleoperation platform when the time interval is greater than or equal to a preset minimum input interval, the current motion state can be provided to the teleoperation platform in a timely manner when the input frequency requirement is met, thus improving the real-time performance of the teleoperation platform's data reception. By skipping the input of teleoperation input data based on the currently received motion data when the time interval is less than the preset minimum input interval, the input pressure when the frequency of motion data generation from the source robot exceeds the processing frequency of the teleoperation platform can be limited. By skipping the caching of the currently received motion data as input data and skipping the addition of the currently received motion data to the input queue, the situation where expired motion data is resent or queued for input at subsequent times can be reduced, thereby lowering the risk of the teleoperation platform processing lagging motion states. This embodiment addresses the issue that motion data from the source robot arrives continuously and the input frequency may exceed the stable processing capacity of the teleoperation platform. It employs a mechanism that throttles input based on a preset minimum input interval and does not buffer or queue motion data that does not meet the input conditions, thereby making the input timing of teleoperation input data more bounded and stable.

[0147] For example, the source robot outputs motion data every 5.00 milliseconds, with a preset minimum input interval of 16.67 milliseconds. Assume that the last time teleoperation input data was input to the teleoperation platform was 1000.00 milliseconds ago.

[0148] When the current reception time is 1005.00 milliseconds, the time interval is 5.00 milliseconds, which is less than 16.67 milliseconds. Therefore, no teleoperation input data is input to the teleoperation platform based on the received motion data, nor is the received motion data cached as input data or added to the input queue. When the subsequent reception time is 1010.00 milliseconds, the time interval is 10.00 milliseconds, which is still less than 16.67 milliseconds. Therefore, the input processing is skipped again.

[0149] When the subsequent reception time is 1020.00 milliseconds, the time interval is 20.00 milliseconds, which is greater than 16.67 milliseconds. Therefore, teleoperation input data can be generated based on the motion data received at 1020.00 milliseconds, and this teleoperation input data can be input to the teleoperation platform. At the same time, the previous input time is updated to 1020.00 milliseconds. Thus, even if the source robot continuously outputs motion data with a period of 5.00 milliseconds, the teleoperation platform will not receive the teleoperation input data corresponding to every piece of motion data. Instead, it will receive the teleoperation input data generated based on the current motion data only when the preset minimum input interval is met.

[0150] In some embodiments, the teleoperation platform performs remapping processing on the teleoperation input data for the target robot according to the processing flow for motion data provided by the body tracking device to obtain robot control data. This may include: the teleoperation platform identifying the teleoperation input data as motion data provided by the body tracking device; the teleoperation platform performing at least one of skeleton normalization processing, calibration processing, inverse kinematics solving processing, and controller merging processing based on the teleoperation input data, and obtaining robot control data for the target robot based on the corresponding processing results.

[0151] Recognizing teleoperation input data as motion data provided by the body tracking device means that after receiving teleoperation input data, the teleoperation platform reads the data according to the input format, field meanings, and processing entry points of the motion data provided by the body tracking device, without needing to set up a separate processing entry point for the motion data from the source robot. In other words, the teleoperation platform can process teleoperation input data as a form of motion data provided by the body tracking device, thereby reusing the processing flow configured for the motion data provided by the body tracking device.

[0152] The processing flow for motion data provided by body tracking devices can refer to the pre-set data processing flow of the teleoperation platform for the motion data provided by the body tracking devices. This processing flow may include at least one of the following: skeleton normalization processing, calibration processing, inverse kinematics solving processing, and controller merging processing. The purpose of this processing flow is to convert the input humanoid skeleton motion data into control data that the target robot can execute.

[0153] Skeleton normalization refers to the process of converting humanoid skeleton data from teleoperation input data into skeleton dimensions, proportions, or coordinate references that can be uniformly processed by the teleoperation platform. Skeleton data from different sources may have differences in height, arm length proportions, leg length proportions, or skeleton node spacing. Skeleton normalization can adjust the skeleton pose data to the standard skeleton scale preset by the teleoperation platform or the target robot's adaptation scale.

[0154] Calibration processing refers to correcting attitude deviations, position deviations, or aligning skeletons in the teleoperation input data based on the calibration parameters of the teleoperation platform. Calibration processing can be used to correct differences in reference attitude, zero-position, scale, or mounting orientation between the input skeleton and the target robot.

[0155] Inverse kinematics processing refers to the process of solving the control quantities of each joint of the target robot based on the pose of the skeleton nodes in the teleoperation input data.

[0156] Controller merging processing can refer to the process of combining controller-compatible data and skeleton input data from teleoperation input data into the teleoperation platform processing.

[0157] This embodiment enables the teleoperation platform to recognize teleoperation input data as motion data provided by the body tracking device. This allows for the reuse of the teleoperation platform's input entry points and processing flow for motion data from the body tracking device, reducing the need to develop dedicated processing flows for the source robot's motion data. By performing skeleton normalization processing based on the teleoperation input data, the teleoperation platform can unify the scale and proportion of different input skeletons, thereby improving the stability of subsequent remapping processing. By performing calibration processing based on the teleoperation input data, the teleoperation platform can correct the baseline differences between the interface humanoid skeleton and the target robot, reducing the impact of posture and position deviations on robot control data. By performing inverse kinematics solving processing based on the teleoperation input data, the teleoperation platform can convert the interface humanoid skeleton pose into joint control quantities executable by the target robot, thus achieving motion remapping for the target robot. By performing controller merging processing based on the teleoperation input data, the teleoperation platform can accommodate situations such as the presence of controller inputs, empty controller inputs, or controller inputs being placeholders, thereby improving the adaptability of the teleoperation input data to the teleoperation platform's processing flow. This embodiment addresses the problem that motion data from the source robot is difficult to directly reuse, and the teleoperation platform cannot directly remap the motion data provided by the body tracking device. It adopts a mechanism that allows the teleoperation input data to be recognized by the teleoperation platform as motion data provided by the body tracking device, and uses the same processing flow as skeleton normalization, calibration, inverse kinematics solving, and controller merging. This allows the robot control data of the target robot to be obtained with reduced modification to the teleoperation platform.

[0158] For example, the teleoperation input data includes skeleton pose data for 24 skeleton nodes, 2 controller data items, and 125,000 milliseconds of time information. After receiving the teleoperation input data, the teleoperation platform can identify it as motion data provided by the body tracking device. The teleoperation platform first performs skeleton normalization processing on the 24 skeleton nodes, for example, normalizing the input skeleton height from 1.60 meters to 1.70 meters. Then, it performs calibration processing, for example, correcting the pelvic reference height by 0.03 meters and the shoulder reference pose by 5 degrees. Next, it performs inverse kinematics solving processing, for example, based on the position (0.42, 1.20, 0.18) and pose of the left wrist skeleton node, it solves for the target robot's left shoulder joint angle of 15 degrees, left elbow joint angle of 40 degrees, and left wrist joint angle of 8 degrees. At the same time, if the controller data item indicates that both the left and right controllers exist but there is no control input, the controller fusion processing outputs a controller fusion result with no button triggering, no joystick offset, and no trigger input. The teleoperation platform can generate robot control data for the target robot based on the results of skeleton normalization, calibration, inverse kinematics solution, and controller merging, thereby controlling the target robot to move according to the actions corresponding to the teleoperation input data.

[0159] In some embodiments, the method may further include: determining the skeleton pose data in the first coordinate system as the skeleton pose data in the second coordinate system when the first coordinate system matches the second coordinate system.

[0160] Determining the skeleton pose data as skeleton pose data in the second coordinate system means that after confirming that the first coordinate system matches the second coordinate system, the first coordinate transformation is no longer performed on the skeleton pose data. Instead, the skeleton pose data is directly used as data that meets the requirements of the second coordinate system for subsequent teleoperation input data generation. This process does not change the numerical value of the skeleton pose data, but rather confirms that the skeleton pose data meets the input coordinate requirements of the teleoperation platform, thus allowing it to proceed to the subsequent data assembly process.

[0161] This embodiment determines the skeleton pose data in the second coordinate system when the first and second coordinate systems match. This reduces unnecessary coordinate transformations when the skeleton pose data already meets the input coordinate requirements of the teleoperation platform, thereby lowering additional computational overhead. By uniformly determining the skeleton pose data in the second coordinate system when they match, the subsequent teleoperation input data generation process can maintain consistency with the processing results when the coordinate systems do not match, thus improving the stability of the process flow.

[0162] See Figure 2 , Figure 2This is a structural block diagram of a robot teleoperation control device provided in an embodiment of this application.

[0163] This application also provides a robot teleoperation control device, including a skeleton mapping module, a coordinate adaptation module, an input generation module, and a platform input module.

[0164] The skeleton mapping module is used to convert the node pose data corresponding to multiple source motion nodes in the motion data of the source robot into skeleton pose data corresponding to the interface humanoid skeleton according to the skeleton input format based on the skeleton mapping information. The data format of the interface humanoid skeleton is consistent with the skeleton input format used by the teleoperation platform for the motion data provided by the body tracking device.

[0165] The coordinate adaptation module is used to perform a first coordinate transformation on the skeleton pose data in the first coordinate system to obtain the skeleton pose data in the second coordinate system when the first coordinate system in which the motion data of the source robot is located does not match the second coordinate system in which the motion data provided by the body tracking device received by the teleoperation platform is located.

[0166] The input generation module is used to generate teleoperation input data based on the skeleton pose data in the second coordinate system; wherein the teleoperation input data conforms to the preset input data structure adopted by the teleoperation platform for the motion data provided by the body tracking device.

[0167] The platform input module is used to input the teleoperation input data to the teleoperation platform, so that the teleoperation platform performs remapping processing on the teleoperation input data for the target robot according to the processing flow of motion data provided for the body tracking device, and obtains robot control data, which is used to control the movement of the target robot.

[0168] See Figure 3 , Figure 3 This is a structural block diagram of a robot provided in an embodiment of this application.

[0169] This application also provides a robot, which includes a control module for performing the steps of any of the above methods.

[0170] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0171] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0172] The computer program product may be a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the computer program product of the embodiments of this application is not limited thereto, and the computer program product may be any combination of one or more computer-readable media.

[0173] See Figure 4 , Figure 4 This is a structural block diagram of a computer device provided in an embodiment of this application.

[0174] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above methods.

[0175] The embodiments of this application do not limit the computer device, which may be, for example, a local computer device, a cloud computer device, a distributed computer device, etc.

[0176] The computer device may include: a memory 110, a processor 120, and a communication interface 130. The memory 110, the processor 120, and the communication interface 130 are connected through internal connection paths.

[0177] The memory 110 is used to store computer programs, which in some implementations may include code for implementing the methods of the embodiments of this application.

[0178] The processor 120 executes the computer program stored in the memory 110 to control the communication interface 130 to receive input data and information, and output operation results and other data. In some implementations, when the solutions of the embodiments of this application are implemented by software or firmware, the computer program used to implement the solutions of the embodiments of this application can be stored in the processor 120 and executed by the processor 120.

[0179] The memory 110 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM). It should be noted that the memory 110 described herein is intended to include, but is not limited to, any memory of these and other suitable types. As an example, the memory 110 includes random access memory (RAM), cache memory, and read-only memory (ROM). The memory 110 stores a computer program that can be executed by processor 120, causing processor 120 to implement the steps of any of the methods described above.

[0180] The processor 120 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 120 can be any conventional processor.

[0181] In implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware in the processor 120 or by instructions in software form. The method disclosed in the embodiments of this application can be directly implemented by the hardware processor, or by a combination of hardware and software modules in the processor 120. The software modules can be located in mature storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in the memory 110, and the processor 120 reads the information in the memory 110 and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are not provided here.

[0182] In some implementations, in addition to the hardware units described above, computer devices may also include software modules, such as operating systems, basic input / output systems (BIOS), and application software.

[0183] An operating system is used to manage the hardware and / or software resources of a computer device; it is the kernel and foundation of the computer. The operating system handles fundamental tasks such as managing and configuring memory, determining the priority of system resource allocation, controlling input and output devices, operating the network, and managing the file system. To facilitate user operation, most operating systems provide a user interface for interaction with the system.

[0184] The BIOS is used to perform hardware initialization during the power-on boot phase and to provide runtime services for the operating system and applications. In some implementations, the BIOS can also monitor and display processor temperature and execute temperature protection strategies.

[0185] Application software, also known as an application program, is software written for a specific user application purpose and is one of the main categories of computer software. For example, application software can be a program used to achieve purposes such as power control and temperature management.

[0186] It is understood that the specific examples in this application are only intended to help those skilled in the art better understand the implementation of this application, and are not intended to limit the scope of protection of this application.

[0187] It is understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0188] It is understood that the various implementation methods described in this application can be implemented individually or in combination, and this application does not limit them.

[0189] Unless otherwise stated, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0190] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design intervention conditions of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0191] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the embodiments described above can be referred to the corresponding processes in other embodiments, and will not be repeated here.

[0192] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0193] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the technical solution in this application, depending on actual needs.

[0194] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0195] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, essentially, or the part that contributes to related technologies, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0196] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for remotely controlling a robot, characterized in that, include: Based on skeleton mapping information, the node pose data corresponding to multiple source motion nodes in the motion data of the source robot are converted into skeleton pose data corresponding to the interface humanoid skeleton according to the skeleton input format. The data format of the interface humanoid skeleton is consistent with the skeleton input format used by the teleoperation platform for the motion data provided by the body tracking device. If the first coordinate system in which the motion data of the source robot is located does not match the second coordinate system in which the motion data provided by the body tracking device received by the teleoperation platform is used, a first coordinate transformation is performed on the skeleton pose data in the first coordinate system to obtain the skeleton pose data in the second coordinate system. Teleoperation input data is generated based on the skeleton pose data in the second coordinate system; wherein the teleoperation input data conforms to the preset input data structure adopted by the teleoperation platform for the motion data provided by the body tracking device; The teleoperation input data is input to the teleoperation platform, so that the teleoperation platform performs remapping processing on the teleoperation input data for the target robot according to the processing flow for motion data provided by the body tracking device, and obtains robot control data, which is used to control the movement of the target robot.

2. The robot teleoperation control method according to claim 1, characterized in that, The process of converting the node pose data corresponding to multiple source motion nodes in the motion data of the source robot into skeleton pose data corresponding to the interface humanoid skeleton according to the skeleton input format, based on skeleton mapping information, includes: Based on the skeleton mapping information, the associated node information and pose generation rules corresponding to each skeleton node in the multiple skeleton nodes included in the interface humanoid skeleton are determined respectively. Based on the associated node information, associated node pose data for generating the skeleton node is determined from the node pose data corresponding to the multiple source motion nodes. Based on the pose generation rules and the pose data of the associated nodes, the skeleton pose data corresponding to the skeleton node is generated.

3. The robot teleoperation control method according to claim 1, characterized in that, The process of determining the skeleton mapping information includes: Based on at least one of the node identifier, node topology relationship and node geometric information of each source motion node in the robot model parameters corresponding to the source robot, determine the candidate association relationship between the skeleton nodes in the interface humanoid skeleton and the source motion nodes; Based on the candidate associations, the skeleton mapping information is determined.

4. The robot teleoperation control method according to claim 2, characterized in that, The process of determining the pose generation rules corresponding to the skeleton nodes includes: When there is a direct correspondence between the skeleton node and a source motion node of the source robot, the pose generation rule corresponding to the skeleton node is determined as the direct generation rule; If the skeleton node does not directly correspond to any source motion node of the source robot and can be determined through the parent associated node and local offset information, the pose generation rule corresponding to the skeleton node is determined as the offset generation rule. If the skeleton node does not directly correspond to any of the source motion nodes of the source robot and is located between two source motion nodes of the source robot, the pose generation rule corresponding to the skeleton node is determined as the interpolation generation rule.

5. The robot teleoperation control method according to claim 4, characterized in that, When the pose generation rule corresponding to the skeleton node is an offset generation rule, the skeleton pose data in the first coordinate system includes the skeleton position data and skeleton posture data of the skeleton node in the first coordinate system. The step of generating skeleton pose data corresponding to the skeleton node based on the pose generation rule and the associated node pose data includes: The parent node pose data corresponding to the parent associated node is determined from the associated node pose data. The parent node pose data includes the parent node position data and parent node pose data of the parent associated node in the first coordinate system. Based on the positional relationship between the target geometric point corresponding to the skeleton node and the parent associated node, a preset offset vector of the skeleton node in the local coordinate system of the parent associated node is determined; Based on the parent node's pose data, the preset offset vector is transformed from the local coordinate system of the parent associated node to the first coordinate system to obtain the offset of the skeleton node in the first coordinate system; Based on the parent node position data and the offset of the skeleton node in the first coordinate system, the skeleton position data of the skeleton node in the first coordinate system is generated. The parent node's pose data is determined as the skeleton node's pose data in the first coordinate system.

6. The robot teleoperation control method according to claim 4, characterized in that, When the pose generation rule corresponding to the skeleton node is an interpolation generation rule, the skeleton pose data in the first coordinate system includes the skeleton position data and skeleton posture data of the skeleton node in the first coordinate system. The step of generating skeleton pose data corresponding to the skeleton node based on the pose generation rule and the associated node pose data includes: The first node pose data corresponding to the first associated node and the second node pose data corresponding to the second associated node are determined from the associated node pose data; wherein, the first node pose data includes the first node position data and the first node orientation data of the first associated node in the first coordinate system, and the second node pose data includes the second node position data and the second node orientation data of the second associated node in the first coordinate system. Based on the type of the skeleton node, determine the interpolation parameters corresponding to the skeleton node; Based on the interpolation parameters, position interpolation processing is performed on the first node position data and the second node position data to obtain the skeleton position data of the skeleton node in the first coordinate system. Based on the interpolation parameters, attitude interpolation processing is performed on the first node attitude data and the second node attitude data to obtain the skeleton attitude data of the skeleton node in the first coordinate system.

7. The robot teleoperation control method according to claim 1, characterized in that, The process of determining the first coordinate transformation includes: The second coordinate transformation used by the teleoperation platform to process the motion data provided by the body tracking device is determined. The second coordinate transformation is used to transform the motion data provided by the body tracking device from the second coordinate system to the processing coordinate system of the teleoperation platform. When the processing coordinate system is consistent with the first coordinate system, a dual coordinate transformation that matches the second coordinate transformation is determined based on the second coordinate transformation, and the dual coordinate transformation is determined as the first coordinate transformation.

8. The robot teleoperation control method according to claim 1, characterized in that, The preset input data structure includes skeleton data items, controller data items, and time data items; the generation of teleoperation input data based on the skeleton pose data in the second coordinate system includes: Based on the skeleton node identifiers corresponding to multiple skeleton nodes in the humanoid skeleton of the interface, the skeleton data items in the preset input data structure are determined. According to the skeleton node identifier, the skeleton pose data in the second coordinate system is written into the corresponding skeleton data item to obtain the filled skeleton data item; Based on the controller compatibility data, the controller data items are populated, and the controller compatibility data corresponds to the control fields in the motion data provided by the body tracking device; Obtain the time information corresponding to the motion data, and write the time information into the time data item; The teleoperation input data is generated based on the filled skeleton data item, the controller data item, and the time data item.

9. The robot teleoperation control method according to claim 8, characterized in that, The process for determining the controller compatibility data includes: In the absence of valid controller input data corresponding to a control field, placeholder controller data used to characterize the presence of a controller but without control input is determined as the controller-compatible data; If valid controller input data corresponding to a control field exists, the controller compatible data is determined based on the valid controller input data.

10. The robot teleoperation control method according to claim 1, characterized in that, When continuously receiving motion data from the source robot, the step of inputting the teleoperation input data to the teleoperation platform includes: In response to the receipt of motion data from the source robot, the time interval between the current reception time and the last time the teleoperation input data was input to the teleoperation platform is determined. If the time interval is greater than or equal to the preset minimum input interval, the teleoperation input data is generated based on the motion data received this time, and the teleoperation input data is input to the teleoperation platform; The method further includes: If the time interval is less than the preset minimum input interval, skip inputting teleoperation input data to the teleoperation platform based on the motion data received this time, skip caching the motion data received this time as input data, and skip adding the motion data received this time to the input queue.

11. A robot teleoperation control device, characterized in that, include: The skeleton mapping module is used to convert the node pose data corresponding to multiple source motion nodes in the motion data of the source robot into skeleton pose data corresponding to the interface humanoid skeleton according to the skeleton input format based on the skeleton mapping information. The data format of the interface humanoid skeleton is consistent with the skeleton input format used by the teleoperation platform for the motion data provided by the body tracking device. The coordinate adaptation module is used to perform a first coordinate transformation on the skeleton pose data in the first coordinate system to obtain the skeleton pose data in the second coordinate system when the first coordinate system in which the motion data of the source robot is located does not match the second coordinate system in which the motion data provided by the body tracking device received by the teleoperation platform is mismatched. An input generation module is used to generate teleoperation input data based on skeleton pose data in the second coordinate system; wherein the teleoperation input data conforms to the preset input data structure adopted by the teleoperation platform for motion data provided by the body tracking device; The platform input module is used to input the teleoperation input data to the teleoperation platform, so that the teleoperation platform performs remapping processing on the teleoperation input data for the target robot according to the processing flow of motion data provided for the body tracking device, and obtains robot control data, which is used to control the movement of the target robot.

12. A robot, characterized in that, The robot includes a control module for performing the steps of the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images