Unified teleoperation method, system, device and medium for heterogeneous robots

Abstract

This application relates to the field of robot control and data acquisition technology, and provides a unified telecontrol method, system, device, and medium for heterogeneous robots. The method includes: acquiring control data from an interactive device; parsing the system description information of the target robot to construct a multi-dimensional capability space adapted to the target robot; determining multi-channel mapping rules based on the characteristics of the multi-dimensional capability space and the control data from the interactive device; converting the control data into raw control commands that drive the corresponding capability dimensions based on the multi-channel mapping rules; performing kinematic processing on the raw control commands to generate joint control commands that satisfy the kinematic constraints of the target robot; and sending the joint control commands to the target robot for execution. The unified telecontrol method, system, device, and medium for heterogeneous robots provided in this application realize unified telecontrol of cross-configuration robots.

Description

Technical Field

[0001] This application relates to the field of robot control and data acquisition technology, and in particular to a unified remote control method, system, device and medium for heterogeneous robots. Background Technology

[0002] In existing robot teleoperation technologies, the correspondence between the input channels of interactive devices and the robot actuators is usually implemented using hard-coded methods embedded in the system program code. When applied to target robots with different configurations, such as different joint configurations, sensor layouts, and actuator types, existing systems require rewriting or modifying the program code to achieve matching.

[0003] This adaptation method, which relies heavily on code modification, results in high redundancy in the remote operating system's program code, making maintenance difficult and limiting the versatility between robots of different configurations. Summary of the Invention

[0004] This application provides a unified remote control method, system, device, and medium for heterogeneous robots, which solves the technical defects of existing technologies that make it impossible for robots of different configurations to be used interchangeably, and achieves universal adaptation for different heterogeneous robots.

[0005] In a first aspect, this application provides a unified remote control method for heterogeneous robots, comprising the following steps: Acquire control data from interactive devices; The system description information of the target robot is analyzed to construct a multi-dimensional capability space adapted to the target robot. The multi-dimensional capability space includes one or more of the following: operation space, movement space, posture adjustment space, or end effector space. Based on the characteristics of the multidimensional capability space and the control data of the interactive device, a multi-channel mapping rule is determined; the multi-channel mapping rule is used to structurally align the operation dimension of the interactive device with the capability dimension of the target robot. Based on multi-channel mapping rules, control data is transformed into raw control commands that drive the corresponding capability dimensions; The original control commands are kinematically processed to generate joint control commands that satisfy the kinematic constraints of the target robot. Joint control commands are sent to the target robot for execution.

[0006] According to the unified remote control method for heterogeneous robots provided in this application, multi-channel mapping rules are determined based on the characteristics of the multi-dimensional capability space and the control data of the interactive device, including: Obtain the task requirement characteristics of the current remote control task; Based on the features of the multidimensional capability space, control data, and task requirement features, a rule mapping logic is generated. The rule mapping logic is encapsulated into an instantiated configuration file to determine the multi-channel mapping rules; Among them, the multi-channel mapping rules are dynamically determined according to the characteristics of task requirements, so that the same control data can be mapped to different capability dimensions or given different control semantics under different task requirements.

[0007] According to the unified telemanipulation method for heterogeneous robots provided in this application, the multi-channel mapping rule includes at least one of the following mapping channels: The pose following mapping channel is used to map the pose increment of a handle, head-mounted display, or motion tracker in an interactive device to the pose increment of the end effector, head, or any joint of the target robot. The joystick speed mapping channel is used to map the continuous analog quantity of the joystick in the interactive device to the speed command of the target robot chassis, torso, or any joint. The trigger control mapping channel is used to map the pressing stroke of the trigger in the interactive device to the continuous control quantity or discrete control signal of the target robot gripper or dexterous hand. The button function mapping channel is used to map the digital state of buttons on an interactive device to discrete control events for the target robot.

[0008] According to the unified remote control method for heterogeneous robots provided in this application, the multi-channel mapping rule includes a multi-level button function mapping configuration. This multi-level button function mapping configuration decouples the discrete control signals of the interactive device into mutually independent functional levels according to the controlled object. The functional levels include: The body-level functional layer is used to map the underlying motion control functions of the target robot. The system-level functional layer is used to map the global state control functions of the remote control system; The view-level functional layer is used to map the display and feedback control functions from the operator's perspective.

[0009] According to the unified remote control method for heterogeneous robots provided in this application, the method further includes: Based on the system description information, generate a joint state data stream; By establishing a semantic namespace, the physical sensors of the target robot are uniformly mapped to a standard semantic interface; Synchronously acquire control data, joint status data streams, and sensor data output from various standard semantic interfaces to generate a multimodal dataset.

[0010] According to the unified remote control method for heterogeneous robots provided in this application, kinematic processing is performed on the original control commands to generate joint control commands that satisfy the kinematic constraints of the target robot, including: For subsystems involving spatial motion, extract the spatial pose increment from the original control commands; Based on the preset coordinate system mapping relationship and motion scaling ratio, the spatial pose increment is converted into the pose increment of the robot end effector, and combined with the current end effector pose of the target robot, the target pose of the robot end effector is calculated. Using a solution framework that includes multi-task priorities, the inverse kinematics of the robot end-effector pose is calculated under the kinematic constraints of the target robot to obtain the target joint state sequence, which serves as the joint control command.

[0011] According to the unified remote control method for heterogeneous robots provided in this application, after obtaining the target joint state sequence, the method further includes: Obtain the current actual state and dynamic boundary conditions of the target robot; Based on dynamic boundary conditions, taking the current actual state as the initial starting point, the target joint state sequence is subjected to online interpolation and trajectory smoothing to output a smooth command sequence and issue it for execution.

[0012] Secondly, this application also provides a unified remote control system for heterogeneous robots, including the following modules: The data acquisition module is used to acquire control data from the interactive device; The capability space construction module is used to parse the system description information of the target robot and construct a multi-dimensional capability space that adapts to the target robot. The multi-dimensional capability space includes one or more of the following: operation space, movement space, posture adjustment space, or end effector space. The rule-building module is used to determine multi-channel mapping rules based on the characteristics of the multi-dimensional capability space and the control data of the interactive device; the multi-channel mapping rules are used to structurally align the operation dimension of the interactive device with the capability dimension of the target robot. The control command generation module is used to convert control data into raw control commands that drive the corresponding capability dimensions based on multi-channel mapping rules. The kinematics processing module is used to perform kinematic processing on the original control commands to generate joint control commands that satisfy the kinematic constraints of the target robot. The command issuing module is used to issue joint control commands to the target robot for execution.

[0013] Thirdly, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described unified remote control methods for heterogeneous robots.

[0014] Fourthly, this application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described unified remote control methods for heterogeneous robots.

[0015] This application provides a unified telecontrol method, system, device, and medium for heterogeneous robots. It constructs a multi-dimensional capability space by parsing system description information, and determines multi-channel mapping rules based on the characteristics of the multi-dimensional capability space and control data from interactive devices. This allows the same input signal to be dynamically mapped to different capability dimensions under different configurations, adapting to heterogeneous robots without code modification and achieving unified telecontrol across configurations. Through multi-channel decoupling mapping, control data is converted into raw control commands that drive the corresponding capability dimensions, enabling coordinated control of various actuators and system functions, such as the robot's end effector, chassis, gripper, and torso. Kinematic processing of the raw control commands generates a smooth command sequence that satisfies kinematic constraints, ensuring execution smoothness and safety. This achieves unified telecontrol of robots across configurations. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the unified remote control method for heterogeneous robots provided in this application.

[0018] Figure 2 A flowchart illustrating the method for generating the multimodal dataset provided in this application.

[0019] Figure 3 A flowchart illustrating the trajectory smoothing method provided in this application.

[0020] Figure 4 The system architecture diagram of the unified remote control system for heterogeneous robots provided in this application.

[0021] Figure 5 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] The following is combined with Figures 1 to 5 This application describes a unified remote control method, system, device, and medium for heterogeneous robots.

[0024] Figure 1 This is a flowchart illustrating the unified remote control method for heterogeneous robots provided in this application, as shown below. Figure 1 As shown, the method includes the following: Step 101: Obtain control data from the interactive device.

[0025] Control data refers to the multi-dimensional input signals output by the interactive device under the operator's control.

[0026] Optionally, the control data includes, but is not limited to, the 6-DoF pose of the head-mounted display, the 6-DoF pose of the left and right controllers, the 6-DoF pose of the motion tracker, the analog signal of the controller joystick, the analog signal of the controller trigger, and the digital signal of the controller buttons.

[0027] In this application, interactive devices refer to external input devices that are worn or controlled by an operator and are capable of responding to the operator's limb movements and generating multi-dimensional spatial input signals, including but not limited to virtual reality headsets and controllers, augmented reality devices, motion trackers, data gloves, motion capture devices, force feedback controllers, etc.

[0028] It should be noted that the interactive device is physically independent of the target robot and is connected to the robot control system via a wired or wireless communication link.

[0029] Step 102: Analyze the system description information of the target robot and construct a multi-dimensional capability space adapted to the target robot.

[0030] The system description information refers to a collection of digital files used to describe the physical configuration, control capabilities, and data interaction capabilities of the target robot, including but not limited to Unified Robot Description Format (URDF) model files, Simulation Description Format (SDF) files, Robot Operating System / Robot Operating System 2 (ROS / ROS2) middleware interfaces, controller interfaces, communication interfaces, and sensor configurations.

[0031] Among them, the multidimensional capability space refers to the abstraction of the physical configuration and interaction requirements of the target robot into a standardized set of capability dimensions, so as to shield the underlying differences between different hardware configurations.

[0032] Specifically, the multidimensional capability space includes the following four standard capability dimensions: (1) Manipulation Capability: Responsible for adjusting the end pose of linkage mechanisms such as robotic arms in three-dimensional space, such as single-arm pose adjustment and double-arm pose adjustment.

[0033] (2) Locomotion Capability: Responsible for the displacement and posture of the robot base or the whole in physical space, such as translation and steering of wheeled chassis, and quadruped or bipedal walking.

[0034] (3) Posture Capability: Responsible for adjusting the internal configuration or overall shape of the robot body, used to change the robot's internal posture, center of gravity distribution or local shape, such as the elbow posture of the redundant robotic arm, the turning of the head gimbal, the tilting of the torso / waist, etc.

[0035] (4) Execution Capability: Responsible for direct contact and interaction with the physical environment, such as gripper opening and closing, dexterous hand grasping, suction cup suction, etc.

[0036] For example, the system loads the system description information of a wheeled dual-arm robot. The system description information includes a 16-DOF URDF model file, where each arm has 7 DDOs and the head has 2 DDOs; it also includes the CAN bus communication protocol configuration of the wheeled chassis, which defines the communication cycle for the system to send omnidirectional speed commands to the chassis as 10ms; it also includes the control interface definition of the head gimbal for receiving pitch and yaw angle commands; and it also includes configuration files for the left and right wrist cameras and the end effector force sensor.

[0037] Furthermore, the system analyzes the system description information and constructs a multi-dimensional capability space adapted to the wheeled dual-arm robot: Operational capabilities: Instantiate two independent operational capability units for the left and right arms, respectively corresponding to the end-effector pose control of the left and right robotic arms, for performing spatial operation tasks such as grasping and placing; Mobility: Instantiate the chassis omnidirectional mobility unit, map the translational speed (forward / backward / left / right) and rotational angular velocity on the two-dimensional plane into the chassis omnidirectional speed command, and output it to the CAN bus controller; Attitude adjustment capability: Instantiate the 2-DOF attitude adjustment capability unit of the head gimbal, and map the pitch and yaw angle commands to the gimbal motor control interface to adjust the observation orientation of the head camera; End-efficiency execution capability: Instantiate two end-efficiency execution capability units, left and right claws, to control the opening and closing actions of the left and right claws respectively, for grasping and releasing objects.

[0038] Step 103: Determine the multi-channel mapping rules based on the characteristics of the multi-dimensional capability space and the control data of the interactive device; wherein, the multi-channel mapping rules are used to structurally align the operation dimension of the interactive device with the capability dimension of the target robot.

[0039] Among them, the multi-channel mapping rule refers to the set of rules that divide the multi-dimensional input of the interactive device into channels according to different control semantics, and establish the mapping relationship between the input signal of each channel and the corresponding capability dimension of the target robot.

[0040] Optionally, the multi-channel mapping rules include: Decoupling mapping sub-rules are used to separate control data into multiple independent channels according to control semantics, and to establish basic independent mapping relationships between each channel and each capability dimension of the target robot; The collaborative mapping sub-rules are used to establish dynamic coupling relationships between at least two channels and each capability dimension of the target robot in order to execute multi-channel joint control or single-channel linkage control. The collaborative mapping sub-rules include at least one of the following: many-to-one collaborative mode, one-to-many collaborative mode, and context-adaptive collaborative mode.

[0041] Multi-channel mapping rules include, but are not limited to, pose following mapping channel, joystick speed mapping channel, trigger control mapping channel, and button function mapping channel.

[0042] In this application, the system determines multi-channel mapping rules based on the characteristics of the multi-dimensional capability space and the control data of the interactive device.

[0043] Specifically, the multi-channel mapping rules are determined in the following way: (1) Determination of decoupling mapping rules.

[0044] The system acquires control data from the interactive device. The control data includes the types of input channels that the interactive device can provide and the data dimensions of each channel. The input channel types include at least one or more of the following: a pose channel that provides three-dimensional spatial pose information, a joystick channel that provides two-dimensional continuous analog quantities, a trigger channel that provides one-dimensional analog quantities of press stroke, and a button channel that provides digital status signals.

[0045] Furthermore, the system identifies the instantiated capability dimension types in the multidimensional capability space. The capability dimension types include at least one or more of the following: operational capability, mobility capability, posture adjustment capability, and end-effector execution capability.

[0046] Finally, the system structurally aligns each input channel of the interactive device with each instantiated capability dimension of the target robot, assigns a corresponding target capability dimension to each valid input channel, and determines decoupling mapping logic for each assignment relationship. This decoupling mapping logic includes at least signal conversion rules, coordinate transformation rules, and control semantic mapping rules. Thus, each channel is independent and does not interfere with the others during the decoupling mapping stage.

[0047] (2) Determination of collaborative mapping rules.

[0048] Based on the decoupling mapping rules, the system also determines the collaborative mapping relationships between channels according to the target robot's capability dimension characteristics and the task requirements of the current remote control task. The collaborative mapping relationships include at least one of the following: Many-to-one collaborative relationship, which combines and maps signals from multiple input channels to jointly drive the same capability dimension, is used to realize composite control functions that a single channel cannot complete independently; One-to-many collaborative relationship, that is, when the signal of a single input channel meets the preset triggering conditions, it simultaneously drives the linkage response of multiple capability dimensions to achieve system-level linkage control; Context-adaptive collaborative relationship, that is, dynamically adjusting the combination mapping between channels according to the current task mode, to adapt to the control requirements under different operating scenarios.

[0049] The system encapsulates the determined decoupling and cooperative mapping relationships into a configuration file to complete the determination of multi-channel mapping rules.

[0050] For example, when the interaction device is a VR controller plus a motion tracker and the target robot is a wheeled dual-arm robot: The pose channel of the handle is mapped to the operational capability dimension. The mapping logic is as follows: the incremental change of the handle pose is converted into the pose increment command of the robotic arm end effector after coordinate transformation. The pose channel of the motion tracker is mapped to the posture adjustment capability dimension. The mapping logic is as follows: the incremental transformation of the motion tracker is converted into the pose increment command of the robotic arm elbow after coordinate transformation. The joystick channel is mapped to the movement capability dimension. The mapping logic is as follows: Under normal conditions, the analog quantities of the joystick's X / Y axes are linearly mapped to translation speed commands in the forward and backward and left and right directions of the chassis. When the operator presses and holds a preset function key (such as pressing the left joystick) and pushes the X / Y axes at the same time, the X-axis offset is mapped to the chassis rotation angular velocity, and the Y-axis offset is kept as translation or set to zero, so as to achieve composite control of translation and rotation.

[0051] The trigger channel is mapped to the end effector capability dimension. The mapping logic is as follows: the trigger press stroke value is linearly mapped to the opening and closing percentage command of the gripper. The key channel is mapped to system control events. The mapping logic is as follows: the key press event is mapped to a control mode switching command or a data acquisition start / stop command.

[0052] For example, when the interaction device is a VR controller and the target robot is a humanoid robot: The handle pose channel is also mapped to the operational capability dimension and is used to control the pose of the end effectors of the two arms; The head-mounted display pose channel is mapped to the posture adjustment capability dimension. The mapping logic is as follows: the head-mounted display orientation angle is mapped to the pitch and yaw angle commands of the head gimbal to achieve visual tracking. The joystick channel is mapped to the mobility dimension. The mapping logic is as follows: the analog quantity of the joystick is mapped to the direction command and step frequency command for bipedal walking. The trigger channel is mapped to the end-effector capability dimension. The mapping logic is as follows: set a press travel threshold, and trigger the dexterous hand's preset grasping posture when the press travel reaches the threshold.

[0053] Through the above structured alignment, the multi-channel mapping rule establishes a unified logical association between the operation dimension of the interactive device and the capability dimension of the target robot. The same interactive input can be automatically adapted to different capability dimensions under different robot configurations without modifying the upper-level code.

[0054] Step 104: Through multi-channel mapping rules, the control data is transformed into raw control commands that drive the corresponding capability dimensions.

[0055] The multi-channel mapping rules include decoupling mapping sub-rules and collaborative mapping sub-rules. In the decoupling mapping stage, each channel signal is independently mapped to the corresponding capability dimension according to the control semantics, without crosstalk. In the collaborative mapping stage, according to the preset coupling conditions, the multi-channel signals are combined and mapped to jointly drive the same capability dimension, or a single channel signal can simultaneously trigger the linkage response of multiple capability dimensions to achieve composite control functions.

[0056] In this context, raw control commands refer to preliminary control commands for a specific subsystem of the target robot, after mapping and transformation, without kinematic optimization. Raw control commands include, but are not limited to, end-effector pose increment commands, chassis speed commands, gripper opening and closing commands, and discrete event triggering commands.

[0057] Specifically, after receiving control data from the interactive device, the system distributes the different input dimensions of the control data to corresponding mapping channels according to the channel division method defined in the multi-channel mapping rules. Each mapping channel corresponds to one capability dimension. Within each mapping channel, the system converts the value range of the input signal into the command value range accepted by the target capability dimension according to the mapping function defined in the mapping rules, and performs coordinate system transformation when necessary. Each channel executes the above separation and transformation operations in parallel, generating multiple parallel raw control commands, each of which is used to drive the execution of the corresponding capability dimension.

[0058] Taking a wheeled dual-arm robot as an example: the system divides the control data into four data streams according to a multi-channel mapping rule. The first data stream contains the 6-DoF pose data of the left and right handles, which, after coordinate transformation and mapping function processing within the mapping channel, is converted into incremental pose commands for the left and right arm ends, used to drive the operational capability dimension. The second data stream contains the X / Y axis analog quantities of the right joystick, which, after mapping function processing, is converted into omnidirectional speed commands for the chassis, used to drive the movement capability dimension. The third data stream contains the analog quantities of the pressing strokes of the left and right triggers, which, after mapping function processing, are converted into opening and closing percentage commands for the left and right grippers, used to drive the end-effector execution capability dimension. The fourth data stream contains the digital states of each button, which, after mapping function processing, are converted into discrete event commands such as control mode switching, data acquisition start / stop, etc. Each raw control command is generated in parallel without interference and is sent to the corresponding capability dimension for execution.

[0059] Furthermore, in the collaborative mapping phase, the system achieves inter-channel linkage control based on preset coupling conditions. For example, when the operator moves the handle (handle 6-DoF pose change in the first data stream) to control the robotic arm end effector to perform basic six-degree-of-freedom pose adjustment, and simultaneously pushes the joystick (joystick analog quantity in the second data stream) to perform low-speed depth fine-tuning along a specific axis (such as the Z-axis of the tool coordinate system), the collaborative mapping rule combines and maps the joystick's low-speed fine-tuning command with the handle's pose increment command to jointly drive the robotic arm end effector's full-degree-of-freedom pose control in the operational capability dimension, achieving extremely high-precision assembly effects that require multi-axial coordination, such as high-precision docking and precision insertion of complex parts (such as shaft-hole assembly); when the operator presses the emergency stop button (button state trigger in the fourth data stream), the collaborative mapping rule simultaneously sends braking commands to the movement capability dimension, locking commands to the operational capability dimension, and release commands to the end effector execution capability dimension, achieving system-level safety linkage response.

[0060] Step 105: Perform kinematic processing on the original control commands to generate joint control commands that satisfy the kinematic constraints of the target robot.

[0061] Kinematic processing refers to the process of converting raw control commands into joint space commands that satisfy the kinematic constraints of the target robot. Kinematic processing includes, but is not limited to, inverse kinematics solving.

[0062] Kinematic constraints refer to the physical limitations that a target robot must satisfy during its movement. Kinematic constraints include, but are not limited to, joint position limits, joint velocity limits, joint acceleration limits, and self-collision avoidance constraints.

[0063] Joint control commands are sequences of instructions generated after kinematic processing and can be directly used to drive the movement of each joint of the target robot. Joint control commands include target joint position sequences, target joint velocity sequences, and target joint acceleration sequences.

[0064] Specifically, the kinematic treatment includes the following steps: Step 1051, Spatial Pose Increment Extraction: For subsystems involving spatial motion, the system extracts the corresponding spatial pose increment from the original control commands. Spatial pose increment refers to the change in position and orientation of the target robot's end effector, head, or any joint relative to its current position and orientation in three-dimensional space.

[0065] Step 1052, End-effector target pose calculation: The system converts the spatial pose increment into the pose increment of the robot end-effector based on the preset coordinate system mapping relationship and motion scaling ratio, and calculates the robot end-effector target pose by combining it with the current end-effector pose of the target robot.

[0066] The coordinate system mapping relationship is used to convert the pose changes in the coordinate system of the interactive device into the corresponding pose changes in the base coordinate system or end-effector coordinate system of the target robot; the motion scaling ratio is used to amplify or reduce the range of the operator's movements to adapt to the workspace range or task accuracy requirements of the target robot.

[0067] Step 1053, Inverse Kinematics Optimization Solution: The system uses a solution framework that includes multi-task priorities to perform inverse kinematics calculation on the target pose of the robot end effector under the kinematic constraints of the target robot, and obtains the target joint state sequence as joint control commands.

[0068] Inverse kinematics calculation refers to the process of determining the angle values ​​of each joint of a robot given the desired pose of the end effector. A multi-task priority solution framework is a computational framework capable of simultaneously handling multiple control objectives and optimizing solutions according to preset priorities. The primary task has the highest solution priority, and secondary tasks are optimized based on the completion of the primary task.

[0069] Further, in step 1053, the inverse kinematics solution is implemented using a differential inverse kinematics solver based on quadratic programming (QP). The system inputs the robot's end-effector pose error into the solver. The solver employs a weighted multi-task optimization framework, setting end-effector pose following as the primary task with the highest weight coefficient, head pose following as the secondary task with a lower weight coefficient, and minimizing joint angular velocity as the regularization term with a lower weight coefficient, thereby constructing the optimization objective function. The solver obtains the current joint angles of the target robot as initial values, and acquires the joint position limits, joint velocity limits, and self-collision avoidance constraints of the target robot, constructing these physical constraints and obstacle avoidance requirements as linear inequality constraints. The solver solves the quadratic programming problem with a high-frequency control cycle no less than a preset period, outputting a target joint angle increment sequence that satisfies all constraints. The system superimposes the current joint angles with the target joint angle increments to obtain the target joint angle sequence.

[0070] The preset high-frequency control cycle can be 50Hz or 100Hz, and this application does not impose any specific restrictions on it.

[0071] Step 106: Send the joint control commands to the target robot for execution.

[0072] This application provides a unified telecontrol method, system, device, and medium for heterogeneous robots. It constructs a multi-dimensional capability space by parsing system description information, and determines multi-channel mapping rules based on the characteristics of the multi-dimensional capability space and control data from interactive devices. This allows the same input signal to be dynamically mapped to different capability dimensions under different configurations, adapting to heterogeneous robots without code modification and achieving unified telecontrol across configurations. Through multi-channel decoupling mapping, control data is converted into raw control commands that drive the corresponding capability dimensions, enabling coordinated control of the robot's end effector, chassis, gripper, and system functions. Kinematic processing of the raw control commands generates a smooth command sequence that satisfies kinematic constraints, ensuring smooth execution and safety. This achieves unified telecontrol of robots across configurations.

[0073] In some embodiments, determining multi-channel mapping rules based on the characteristics of the multi-dimensional capability space and the control data of the interactive device includes: obtaining the task requirement characteristics of the current teleoperation task; generating rule mapping logic based on the characteristics of the multi-dimensional capability space, the control data, and the task requirement characteristics; and encapsulating the rule mapping logic into an instantiated configuration file to complete the determination of the multi-channel mapping rules. Wherein, under the condition that the control data is the same but the task requirement characteristics are different, the determined multi-channel mapping rules map the same control data to different capability dimensions or assign different control semantics.

[0074] The core of this application lies in the fact that the essence of multi-channel mapping rules is jointly determined by task requirements, robot capabilities, and the input type of the interaction device. Specifically, the system first analyzes the multi-dimensional capability space characteristics of the target robot and identifies the type of currently connected interaction device and the set of input channels it can provide. Different interaction devices have different channel configurations. For example, VR controllers can provide multiple channels such as pose, joystick, trigger, and button; data gloves provide continuous analog signal channels for the bending degree of each finger joint; and motion capture suits provide pose channels for each joint of the human body. Based on the actual channel capabilities of the currently connected device, the system dynamically maps each input channel to the corresponding capability dimension of the target robot and, combined with the control requirements of the current specific task, determines the control semantics of each channel, thereby generating multi-channel mapping rules adapted to the current device and task.

[0075] Subsequently, the system generates a configuration file that carries the multi-channel mapping rules. This configuration file, acting as the data carrier for the mapping rules, is independent of the underlying system code (e.g., in YAML or JSON format). At runtime, the system parses this configuration file and instantiates the mapping relationships defined within it into multi-channel mapping rules in memory.

[0076] This mechanism, where tasks and capabilities determine rules, and configuration files carry and instantiate those rules, ensures that the same input signal has completely different mapping targets and control semantics depending on the configuration file corresponding to different task requirements. For example, in the configuration file corresponding to a fine-tuning task, a joystick signal is mapped to a low-speed fine-tuning command for operational capabilities; while in the configuration file corresponding to a wide-range cruise task, the same joystick signal is mapped to a high-speed chassis movement command for mobility capabilities. The system does not need to modify any underlying code; it only needs to switch configuration files carrying different task rules to achieve dynamic reconfiguration of the mapping logic.

[0077] It's understandable that establishing multi-channel mapping rules essentially involves dynamically binding standard control channels within a general framework to flexible telemanipulation targets within the current configuration. For example, when a wheeled robot is connected, the continuous analog signal channel is bound to the chassis speed as a telemanipulation target; when a single-arm robotic arm is connected, the same continuous analog signal channel is rebound to the end effector speed as a telemanipulation target. This mechanism of decoupling inputs from the general framework and dynamically binding them to flexible targets allows the system to achieve unified adaptation of the same set of interactive devices to any heterogeneous robot without modifying the underlying code, simply by changing the binding relationship.

[0078] Furthermore, in scenarios involving multi-task switching or robot switching, the system re-executes the aforementioned adaptive mapping rule generation process, dynamically generating and instantiating a new configuration file, thus achieving hot updates of the mapping rules. When the target robot switches to a different configuration, the system re-parses the system description information and reconstructs the multi-dimensional capability space. The characteristics of the capability space change accordingly, and the system executes the aforementioned generation process again, generating and instantiating a new configuration file under the new capability space constraints. Throughout this process, no program code needs to be modified, thereby achieving decoupling between mapping rules and program logic.

[0079] Taking joystick signals as an example, when the target robot is a wheeled chassis robot, the system maps the joystick offset to a chassis movement speed command based on the instantiated mobility capability dimension in its multi-dimensional capability space. When the target robot is a single-arm robotic arm, the system maps the same joystick offset to an end effector movement speed command based on the instantiated operational capability dimension in its multi-dimensional capability space. The same input signal is mapped to different output targets under different configuration files, demonstrating the dynamic adaptability of configuration file-driven mapping rules.

[0080] The same input signal has different mapping targets under different configuration files. Specifically, the system stores a first configuration file and a second configuration file. The first configuration file corresponds to a wheeled dual-arm robot configuration, while the second configuration file corresponds to a single-arm robotic arm configuration. When loading the first configuration file, the system maps the right handle pose signal to a right arm end-effector pose increment command; when loading the second configuration file, the system maps the same right handle pose signal to a single-arm end-effector pose increment command. Although the mapping targets of the same input signal are the same in form under the two configuration files, they correspond to different physical actuators. For the joystick signal, it is mapped to a chassis movement speed command under the first configuration file and to an end-effector movement speed command under the second configuration file. The mapping targets and mapping semantics of the same joystick signal are different under different configuration files. It is understandable that the same input signal can produce different control effects under different configuration file driving, reflecting the dynamic adaptation capability of the configuration file-driven mapping rules in this application.

[0081] Furthermore, the mapping target of the same input signal differs depending on the control mode of the target robot. Specifically, when the target robot is currently in position control mode, the system maps the handle pose increment signal to the position increment command of the end effector; when the operator switches the target robot to force control mode by pressing a button, the system maps the same handle pose increment signal to the force / torque control command of the end effector. The physical meaning and control target of the same input signal are different in different control modes, and the operator can change the control semantics of the same operation by switching modes.

[0082] The following details the specific implementation of each mapping channel in the multi-channel mapping rule.

[0083] In some embodiments, the multi-channel mapping rule includes at least one of the following mapping channels: (1) Pose following mapping channel, used to map the pose increment of the handle, head display or motion tracker in the interactive device to the pose increment of the end effector or head or any joint of the target robot.

[0084] Specifically, the system acquires the current pose of the handle output by the interactive device in three-dimensional space in real time, and uses the handle pose of the previous moment as a reference. The difference between the current pose and the reference pose is used as the relative pose increment of the handle. The system then scales this relative pose increment and uses it as the pose increment command for the target robot's end effector. The end effector adds this increment to the current actual pose to complete incremental pose following. For head following, the system directly maps the pose of the head-mounted display to the target robot's head or torso pose target value after coordinate system transformation, achieving synchronous following of the operator's head movement and the robot's head or torso movement.

[0085] (2) Joystick speed mapping channel, used to map the continuous analog quantity of the joystick in the interactive device to the speed command of the target robot chassis or torso or any joint.

[0086] Specifically, the system reads the continuous offset of the joystick in the X and Y axes in real time, with the offset range normalized to the interval of -1 to 1. The system determines the speed mapping method according to the configuration of the target robot: for a wheeled chassis robot, the X-axis offset is mapped to the translational speed in the forward and backward direction of the chassis, and the Y-axis offset is mapped to the translational speed in the left and right direction of the chassis.

[0087] (3) Trigger control mapping channel, used to map the pressing stroke of the trigger in the interactive device to the continuous control quantity or discrete control signal of the target robot gripper or dexterous hand.

[0088] Specifically, the trigger control mapping channel supports two mapping methods: continuous control mode and discrete trigger mode. The method used is determined based on the currently loaded configuration file or operating mode. In continuous control mode, the system reads the trigger's pressing stroke value in real time. This value is normalized to a continuous range of 0 to 1, and the force value is linearly mapped to the opening and closing percentage command of the gripper, so as to realize the continuous adjustment of the opening and closing degree. In discrete trigger mode, the system reads the trigger's pressing stroke value in real time. When the force value reaches the preset threshold, it is determined that the trigger condition is met, and the corresponding discrete control signal (such as the gripper fully closing or fully opening command) is output to realize the rapid switching action.

[0089] (4) Key function mapping channel, used to map the digital state of the keys in the interactive device to discrete control events of the target robot.

[0090] Specifically, the system monitors the digital states of each physical button on the interactive device in real time, including pressed and released states. When the system detects a button's state change from released to pressed, it determines that the button has been triggered. The system then queries the button function mapping table defined in the currently loaded configuration file to find the corresponding discrete control event type and executes the appropriate event.

[0091] This application enables intuitive mapping between operator hand movements and robot end effector movements by setting up a pose-following mapping channel. By setting up a joystick speed mapping channel, chassis movement can be controlled simultaneously with intuitive joystick operation, achieving multi-tasking parallel teleoperation. By setting up a trigger control mapping channel, a single trigger element can simultaneously support both continuous fine control and discrete quick triggering modes, which can be switched naturally according to task requirements. By setting up a button function mapping channel, discrete control functions are mapped to buttons, enabling system-level control operations to be completed without leaving the operator's posture, improving operational smoothness and safety.

[0092] In some embodiments, the multi-channel mapping rule includes a multi-level button function mapping configuration, which decouples the discrete control signals of the interactive device into mutually independent functional levels according to the controlled object; the functional levels include: The body-level functional layer is used to map the underlying motion control functions of the target robot. The system-level functional layer is used to map the global state control functions of the remote control system; The view-level functional layer is used to map the display and feedback control functions from the operator's perspective.

[0093] The following sections will provide detailed explanations of the ontology-level functional layer, the system-level functional layer, and the perspective-level functional layer.

[0094] The following sections will provide a detailed explanation of each of the above functional levels.

[0095] Specifically, the multi-level button function mapping configuration defines the mapping relationship between each input channel and control function of the interactive device. Taking the button channel as an example, the digital state of this channel is mapped to discrete control events, and the specific discrete control event corresponding to each button is defined by the mapping entry of the corresponding functional level in the configuration file. Similarly, the continuous analog quantities of the joystick channel or trigger channel are also mapped by the configuration file, and this application does not limit this. The body-level function layer refers to the function mapping layer used to configure the motion and control of the target robot body, corresponding to the motion control function of the target robot. In this layer, the system maps buttons to dexterous hand preset posture switching functions, including opening, pinching, circling grasping, and pointing postures; maps buttons to control mode switching functions, used to switch between position control mode, force control mode, and force-position hybrid control mode; maps joystick pressing to chassis or walking enable function to prevent accidental touch; and maps long-pressing a specific combination key to safety return to zero function, controlling the robot to smoothly return to the preset safe posture.

[0096] The system-level functional layer refers to the functional mapping layer used to configure the global operating status of the telecontrol system, corresponding to the global control functions of the telecontrol system. At this layer, the system maps buttons to data acquisition start / stop functions to control the start, pause, and end of the recording process; it maps long-press buttons to semantic tag insertion functions to insert semantic tags during data acquisition; it maps pressing specific buttons simultaneously with both hands to emergency stop or safety freeze functions to immediately cut off control command output; and it maps specific buttons to end-effector speed adjustment functions to switch the end-effector speed between fine-tuning and large-scale movements.

[0097] The viewpoint-level functional layer refers to the display and feedback control functions used to configure and map the operator's viewpoint. At this layer, the system maps buttons to camera viewpoint switching functions, used to switch between head-view, left-wrist-view, right-wrist-view, and panoramic top-down view; maps specific buttons to virtual model display mode switching functions, used to switch between transparent mode, solid mode, and wireframe mode; and maps specific buttons to force visualization on / off functions, used to control the display or hiding of end-effector forces or torques in the head-up display (HUD) within the VR headset.

[0098] When loading the configuration file, the system establishes button function mappings for each level according to the definition of the three-layer function matrix. The configuration file is stored independently outside the program code in YAML or JSON format. When the system starts, it automatically loads the corresponding configuration file according to the configuration of the currently connected target robot, so that the same button can trigger different functions in different configurations and application scenarios.

[0099] In other embodiments, the method described above further includes a data acquisition step. Due to the current lack of a unified data abstraction layer and time synchronization mechanism, multimodal data cannot flow efficiently and achieve spatiotemporal alignment within a unified channel, leading to a break in the data acquisition link and making it difficult to directly use the acquired data for training embodied intelligence models. Therefore, this application further integrates a data acquisition step based on a unified teleoperation process to achieve standardized encapsulation and spatiotemporal alignment of multimodal data.

[0100] The process of generating multimodal datasets is explained in detail below.

[0101] Figure 2 This is a flowchart illustrating the method for generating the multimodal dataset provided in this application. Figure 2 As shown, the method includes: Step 201: Generate joint state data stream based on system description information.

[0102] Step 202: By establishing a semantic namespace, the physical sensors of the target robot are uniformly mapped to standard semantic interfaces.

[0103] Step 203: Synchronously acquire control data, joint status data streams, and sensor data output from each standard semantic interface to generate a multimodal dataset.

[0104] Semantic namespace refers to the data access path named according to the functional role of the sensor on the robot body, and unifies and abstracts sensors with different physical models and different communication interfaces into a data source with a standard semantic name.

[0105] A multimodal dataset refers to a complete set of data formed by aligning control input data and robot state data according to a unified time reference.

[0106] Specifically, the system loads the URDF model file of the target robot, parses all joint elements, dynamically generates joint state topics corresponding to the number of joints, and continuously publishes joint position, velocity, and torque data at a preset frequency, forming a joint state data stream. The system parses the installation location information of each sensor link in the URDF model, automatically classifies the sensors into semantic categories such as head vision, wrist vision, and end effector force sensing, and assigns a standardized semantic namespace path to each semantic category.

[0107] Taking a wheeled dual-arm robot as an example, the head camera is mapped to the head visual semantic namespace, the left wrist camera is mapped to the left wrist visual semantic namespace, the right wrist camera is mapped to the right wrist visual semantic namespace, and the end effector force sensor is mapped to the end effector force sensory namespace.

[0108] Therefore, the system obtains sensor data by subscribing to the semantic namespace, and at the same time subscribing to the control data topic, the joint state data stream topic, and all standard semantic interface topics. It adds a unified timestamp to each frame of data and saves it after aligning it with the timestamp, thus generating a multimodal dataset.

[0109] This application synchronously collects all data streams using a unified timestamp, ensuring that control inputs and state responses correspond one-to-one in time. The generated multimodal dataset can be directly used for training embodied intelligence models.

[0110] In other embodiments, to avoid execution jitter caused by directly sending discrete joint state sequences, the system performs trajectory smoothing processing after obtaining the target joint state sequence. The specific process of trajectory smoothing processing is described in detail below.

[0111] Figure 3 This is a flowchart illustrating the trajectory smoothing method provided in this application. Figure 3 As shown, the method includes: Step 301: Obtain the current actual state and dynamic boundary conditions of the target robot.

[0112] The current actual state refers to the actual motion state of each joint of the target robot at the current moment, including the current joint position and the current joint velocity. The dynamic boundary conditions refer to the physical constraints that each joint of the target robot must satisfy during its motion, including the position limit, maximum velocity limit, maximum acceleration limit, and maximum jerk limit of each joint.

[0113] Specifically, the system obtains the current actual position and current actual velocity of each joint through the joint state data stream of the target robot. The system reads the preset dynamic boundary condition parameters in the target robot system configuration file, including the position limit, maximum velocity limit, maximum acceleration limit, and maximum jerk limit of each joint.

[0114] The above parameters have different numerical configurations for different robot configurations. For example, the dynamic boundary conditions are lower for high-load industrial robotic arms and higher for high-speed lightweight robotic arms.

[0115] Step 302: Based on the dynamic boundary conditions, taking the current actual state as the initial starting point, perform online interpolation and trajectory smoothing on the target joint state sequence to output a smooth command sequence and issue it for execution.

[0116] Online interpolation refers to the real-time calculation of intermediate state values ​​between adjacent target states during robot operation, used to generate a continuous reference trajectory. Trajectory smoothing refers to filtering or optimizing discrete command sequences to eliminate abrupt changes and discontinuities in the commands, generating a continuous and differentiable motion trajectory.

[0117] Specifically, the system inputs the target joint state sequence into a time-optimal trajectory interpolation algorithm based on dynamic boundary constraints. This algorithm reads the current actual state of each joint of the target robot as the trajectory starting state, uses each target point in the target joint state sequence as a trajectory path point, and obtains dynamic boundary condition parameters as physical constraints for trajectory generation. Under the dynamic boundary condition constraints, a time-optimal continuous smooth trajectory is generated online between adjacent target points, outputting a smooth command sequence containing position, velocity, and acceleration commands at each interpolation moment. The system encapsulates the smooth command sequence into ROS standard control messages and sends them to the target robot's underlying controller for execution.

[0118] In this application, the time-optimal trajectory interpolation algorithm based on dynamic boundary constraints can be implemented using the Ruckig trajectory generation algorithm, without any specific restrictions. As another optional implementation, trajectory smoothing can be replaced by any one of B-spline smoothing interpolation, trajectory optimization based on model predictive control, or polynomial interpolation.

[0119] This application obtains the current actual state and dynamic boundary conditions of the target robot, and based on this, performs online interpolation and trajectory smoothing on the discrete target joint state sequence, so that the output command sequence meets the continuous smooth requirements of velocity, acceleration and jerk, effectively filtering out human hand tremors and command abrupt changes, avoiding motor overload and mechanical shock, and significantly improving the execution safety of teleoperation and the service life of the equipment.

[0120] The technical solution of this application will be described in detail below with reference to three specific application scenarios.

[0121] Example 1: Remote operation and data acquisition of a wheeled dual-arm robot.

[0122] This embodiment takes a wheeled dual-arm robot as an example to illustrate the complete implementation process of the unified remote control method, system, equipment and medium of this application.

[0123] Step S1: System initialization and data acquisition setup.

[0124] After system startup, the initialization process is executed first. The system loads the URDF model file of the wheeled dual-arm robot, which includes 14 joint degrees of freedom, 2 wrist cameras, 1 head camera, and 2 end effector force sensors. The hardware abstraction layer parses all joint elements in the URDF model file, extracts the 14 joint degrees of freedom information, and dynamically generates 14-dimensional joint state topics. The hardware abstraction layer also parses the sensor link location information in the URDF model file, mapping each sensor to a semantic namespace: the head camera is mapped to the head visual semantic namespace, the left wrist camera to the left wrist visual semantic namespace, the right wrist camera to the right wrist visual semantic namespace, the left end effector force sensor to the left end effector force sensor semantic namespace, and the right end effector force sensor to the right end effector force sensor semantic namespace. The data recording module subscribes to all semantic namespace topics and initiates a timestamp synchronization mechanism.

[0125] Step S2: VR device access and multi-channel mapping configuration.

[0126] The operator wears a PICO 4 Ultra or Meta Quest 3 headset. The data acquisition module establishes a communication connection with the VR headset via the WebXR protocol, extracting the headset's 6-DoF pose, the left and right controllers' 6-DoF pose, joystick analog signals, trigger analog signals, and button states in real time. The space construction and rule establishment module generates a remote control configuration file for the wheeled dual-arm robot based on the target robot's model identifier. The mapping rules defined in the configuration file include: the left controller pose is mapped to incremental control of the left arm's end effector, the right controller pose is mapped to incremental control of the right arm's end effector, the right joystick's X and Y axes are mapped to the chassis's forward / backward and left / right omnidirectional movement speeds, the left joystick's X-axis is mapped to the chassis's rotational angular velocity, the left and right triggers are mapped to the continuous opening and closing control of the left and right grippers, respectively, the A button is mapped to control mode switching, the B button is mapped to data acquisition start / stop, the X button is mapped to camera view cycle switching, and the Y button is mapped to safe return to zero.

[0127] Step S3: Solve the multi-objective differential inverse kinematics.

[0128] The kinematics processing module receives the left and right arm end-effector pose increments from the original control commands. It then constructs a multi-objective differential inverse kinematics optimization problem based on quadratic programming. In the objective function, left and right arm end-effector pose following is designated as the primary task with the highest weight, while minimizing joint angle changes is designated as the secondary task with a lower weight. Constraints include joint position limits, joint velocity limits, and self-collision avoidance constraints. The self-collision avoidance constraint requires that the minimum distance between the two arms is not less than a preset safety threshold. The quadratic programming solver solves the optimization problem in real-time with a preset high-frequency control cycle, outputting a 14-dimensional sequence of target joint angle increments.

[0129] Step S4: Trajectory smoothing and control distribution.

[0130] The kinematics processing module superimposes the solved target joint angle increments with the current joint state to obtain the target joint state sequence. The trajectory smoothing module uses the Ruckig algorithm for online interpolation and smoothing, setting the dynamic boundary conditions to a maximum velocity of 3 rad / s and a maximum acceleration of 5 rad / s. 2 Maximum jerk: 10 rad / s 3 The Ruckig algorithm generates smooth sequences of position, velocity, and acceleration commands under dynamic boundary constraints. The command delivery module then sends these smooth command sequences to the underlying controller of the wheeled dual-arm robot via the ROS2 Topic interface for execution.

[0131] Step S5: Data acquisition and recording.

[0132] The data recording module runs continuously during teleoperation. It synchronously acquires 14-dimensional joint state data, left and right end-effector pose data, and VR operator input data at a frequency of 100Hz; image data from the head camera and dual wrist cameras at a frequency of 30Hz; and left and right end-effector force sensor data at a preset high-frequency control cycle. The data recording module timestamps all acquired data and saves the timestamped multimodal data as an HDF5 format multimodal dataset. The operator can insert semantic tags by long-pressing the B key; a long press inserts a grasp start tag, and releasing the key inserts a grasp end tag.

[0133] This embodiment utilizes a model-driven adaptive data acquisition mechanism to complete sensor access and data acquisition configuration for a 14-DOF wheeled dual-arm robot without code modification. URDF parsing drives the dynamic generation of joint state topics, semantic namespace-driven sensor abstraction enables the upper-level system to acquire multimodal data through a unified semantic interface, and a timestamp synchronization mechanism ensures precise alignment of all data streams. The new model completes data acquisition configuration in just 10 minutes from access, improving efficiency by over 90% compared to the traditional 2-3 days. Four-channel decoupled mapping allows the operator to simultaneously control the end effectors of both arms, chassis movement, and gripper opening and closing, reducing task completion time by 40% and improving operational smoothness by 60%.

[0134] Example 2: Full-body remote operation of a humanoid robot.

[0135] This embodiment takes a humanoid robot as an example to explain in detail the implementation process of the unified remote operation method, system, equipment and medium of this application in a whole-body remote operation scenario.

[0136] Step S1: System initialization.

[0137] After system startup, the URDF model file of the humanoid robot is loaded. This humanoid robot has 27 degrees of freedom, including 12 degrees of freedom for the legs, 14 degrees of freedom for the arms, and 3 degrees of freedom for the waist. After parsing the URDF model file, the system dynamically generates 27-dimensional joint state topics. The system extracts the sensor installation location information from the URDF model file; the sensors include a head RGBD camera, an inertial measurement unit, and end effector force sensors for the arms. The system maps each sensor to its corresponding semantic namespace.

[0138] Step S2: VR mapping configuration.

[0139] The operator wears a VR headset, and the data acquisition module obtains the headset's 6-DoF pose and the left and right controllers' 6-DoF poses via the WebXR protocol. The space construction and rule establishment module loads a humanoid robot-specific function matrix configuration file. The mapping rules defined in the configuration file include: the left controller pose is mapped to left arm end-effector control, the right controller pose is mapped to right arm end-effector control, the headset pose is mapped to head camera orientation following, the right joystick is mapped to bipedal walking direction and speed, the left trigger is mapped to left dexterous hand control, the right trigger is mapped to right dexterous hand control, the A button is mapped to walking mode switching, the B button is mapped to standing and walking mode switching, and the Y button is mapped to dexterous hand preset posture switching.

[0140] Step S3: Solve the whole-body differential inverse kinematics.

[0141] The kinematics processing module receives the end-effector pose increments and head pose increments from the original control commands. It then constructs a whole-body inverse kinematics optimization problem based on quadratic programming. The objective function prioritizes end-effector pose following and head pose following, while minimizing joint angle changes is a secondary task. Optimization constraints include joint position limits, joint velocity limits, centroid projection constraints, and self-collision avoidance constraints. The centroid projection constraint requires that the projection of the robot's centroid onto the horizontal plane must lie within the polygon formed by the two foot support points when the robot is standing. The self-collision avoidance constraint requires maintaining safe distances between the arms and torso, and between the legs. A quadratic programming solver solves the 27-dimensional target joint angle sequence in real time.

[0142] Step S4: Trajectory smoothing and execution.

[0143] The trajectory smoothing module uses the Ruckig algorithm to perform online interpolation and smoothing of the 27-dimensional joint angle sequence. The command issuing module sends the smoothed command sequence to the humanoid robot's full-body controller for execution via the robot SDK interface.

[0144] This embodiment verifies the effectiveness of the system proposed in the full-body teleoperation of a high-DOF humanoid robot. The multi-task priority differential inverse kinematics solution framework based on quadratic programming can simultaneously handle multiple tasks such as end-effector following, head following, center-of-mass stabilization, and collision avoidance, completing real-time 27-dimensional high-dimensional kinematics solutions while satisfying all constraints. A single solution framework is adapted to both a 14-DOF wheeled dual-arm robot and a 27-DOF humanoid robot, verifying the versatility of the unified kinematics solution framework.

[0145] Example 3: Rapid remote operation of a single-arm robotic arm.

[0146] This embodiment takes a single-arm robotic arm as an example to illustrate in detail the implementation process of the unified remote operation method, system, equipment and medium of this application in a rapid deployment scenario.

[0147] Step S1: System initialization.

[0148] After system startup, the URDF model file of a 6-DOF single-arm robotic arm is loaded. The system then parses the URDF model file and dynamically generates 6-dimensional joint state topics. The system identifies the wrist camera of the robotic arm and maps it to the wrist visual semantic namespace.

[0149] Step S2: Simplify mapping configuration.

[0150] The operator wears a VR headset, and the data acquisition module acquires control data from the interactive device. The space construction and rule establishment module loads the robotic arm's dedicated function matrix configuration file. The mapping rules defined in the configuration file include: the right handle pose is mapped to end-effector pose incremental control, the right trigger is mapped to gripper continuous opening and closing control, the right joystick is mapped to end-effector motion speed adjustment, the A button is mapped to switching between position control mode and impedance control mode, and the B button is mapped to start and stop data acquisition.

[0151] Step S3: Solve the differential inverse kinematics.

[0152] The kinematics processing module receives the end-effector pose increment command and constructs a differential inverse kinematics optimization problem. End-effector pose following is designated as the primary task, while moving away from singularities is designated as an auxiliary task. Optimization constraints include joint position limits, joint velocity limits, and workspace boundary constraints. The QP solver solves the target joint angle sequence.

[0153] Step S4: Execute quickly.

[0154] The trajectory smoothing module uses the Ruckig algorithm for trajectory smoothing. Due to the relatively small load inertia of the single-arm robotic arm, the trajectory smoothing module is set with relatively high dynamic boundary parameters: a maximum speed of 5 rad / s and a maximum acceleration of 10 rad / s². 2 The command issuing module sends smooth commands to the robotic arm controller for execution via the ROS interface.

[0155] This embodiment verifies the rapid deployment capability of the system in this application on a low-degree-of-freedom single-arm robotic arm. The system requires no code modification; simply switching configuration files allows the same set of interactive devices to be switched from a wheeled dual-arm robot to a single-arm robotic arm. Operators do not need to relearn the operating logic after switching configurations; the same button automatically adapts to different functions in different configurations. The learning cost for operators switching between different robot configurations is reduced by 80%, and the error rate is reduced by 70%. The deployment of the single-arm robotic arm verifies the system's ability to cover all types of robots, from low to high degrees of freedom, and from single-arm to dual-arm to fully humanoid.

[0156] The three embodiments described above correspond to three typical heterogeneous configurations: a wheeled dual-arm robot, a humanoid robot, and a single-arm robotic arm, with degrees of freedom of 14, 27, and 6 dimensions, respectively, covering a complete category from low to high degrees of freedom and from single-arm to dual-arm to fully humanoid. The three embodiments share the same set of interactive devices (VR headset and controllers) and the same core control process (steps 1 to 4 / 5). The only difference lies in the loaded configuration file, thus achieving unified adaptation of the same method to robots of different configurations. The above embodiments fully verify the universality and effectiveness of the method, system, device, and medium of this application on robots of different configurations.

[0157] The following is combined with Figure 4 This application provides a detailed description of the unified remote control system for heterogeneous robots.

[0158] Figure 4 The system framework diagram of the unified remote control system for heterogeneous robots provided in this application is as follows: Figure 4 As shown, the system includes: a data acquisition module 410, a capability space construction module 420, a rule establishment module 430, a control instruction generation module 440, a kinematics processing module 450, and an instruction issuance module 460.

[0159] The data acquisition module 410 is used to acquire control data from the interactive device. Specifically, the data acquisition module establishes a communication connection with the VR device via the WebXR protocol and receives 6-DoF pose data of the head-mounted display, 6-DoF pose data of the left and right controllers, analog signal data of the joystick, analog signal data of the trigger, and digital signal data of the buttons sent by the VR device.

[0160] The capability space construction module 420 is used to parse the system description information of the target robot and construct a multi-dimensional capability space that adapts to the target robot. The multi-dimensional capability space includes one or more of the following: operation space, movement space, posture adjustment space, or end effector space.

[0161] The rule establishment module 430 is used to determine multi-channel mapping rules based on the characteristics of the multi-dimensional capability space and the control data of the interactive device; wherein, the multi-channel mapping rules are used to structurally align the operation dimension of the interactive device with the capability dimension of the target robot.

[0162] The control command generation module 440 is used to convert control data into raw control commands that drive the corresponding heterogeneous subsystems based on multi-channel mapping rules. Specifically, the control command generation module distributes the control data to each mapping channel according to the channel division method defined in the multi-channel mapping rules, and performs mapping transformation independently in each channel to complete decoupled mapping; or, the control command generation module can also perform fusion calculation or linkage distribution on the converted multi-channel commands according to preset coupling conditions to complete collaborative mapping.

[0163] The kinematics processing module 450 performs kinematic processing on the original control commands to generate joint control commands that satisfy the kinematic constraints of the target robot. Specifically, for subsystems involving spatial motion, the kinematics processing module extracts the spatial pose increment from the original control commands; based on the preset coordinate system mapping relationship and motion scaling ratio, it converts the spatial pose increment into the pose increment of the robot's end effector, and calculates the target pose of the robot's end effector by combining it with the current end effector pose of the target robot; using a solution framework that includes multi-task priority, it performs inverse kinematics calculation on the target pose of the robot's end effector under the kinematic constraints of the target robot to obtain the target joint state sequence, which serves as the joint control commands.

[0164] The command issuing module 460 is used to issue joint control commands to the target robot for execution. Specifically, the command issuing module sends the joint control commands to the target robot's underlying controller through a robot middleware communication mechanism (such as ROS, DDS, Cyber ​​RT, or a custom Socket communication protocol).

[0165] Figure 5 A schematic diagram of the structure of the electronic device provided in this application, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a unified telecontrol method for heterogeneous robots. This method includes: acquiring control data from the interactive device; parsing the system description information of the target robot to construct a multi-dimensional capability space adapted to the target robot; the multi-dimensional capability space includes one or more of the following: an operation space, a movement space, a posture adjustment space, or an end effector space; determining multi-channel mapping rules based on the characteristics of the multi-dimensional capability space and the control data from the interactive device; wherein the multi-channel mapping rules are used to structurally align the operation dimension of the interactive device with the capability dimension of the target robot; based on the multi-channel mapping rules, converting the control data into raw control instructions that drive the corresponding capability dimensions; performing kinematic processing on the raw control instructions to generate joint control instructions that satisfy the kinematic constraints of the target robot; and issuing the joint control instructions to the target robot for execution.

[0166] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0167] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the unified telecontrol method for heterogeneous robots provided by the above methods. The method includes: acquiring control data from an interactive device; parsing the system description information of the target robot to construct a multi-dimensional capability space adapted to the target robot; the multi-dimensional capability space includes one or more of an operation space, a movement space, a posture adjustment space, or an end effector space; determining multi-channel mapping rules based on the characteristics of the multi-dimensional capability space and the control data from the interactive device; wherein the multi-channel mapping rules are used to structurally align the operation dimension of the interactive device with the capability dimension of the target robot; based on the multi-channel mapping rules, converting the control data into raw control commands that drive the corresponding capability dimensions; performing kinematic processing on the raw control commands to generate joint control commands that satisfy the kinematic constraints of the target robot; and sending the joint control commands to the target robot for execution.

[0168] Furthermore, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, this computer program implements a unified telemanipulation method for heterogeneous robots provided by the methods described above. This method includes: acquiring control data from an interactive device; parsing the system description information of the target robot to construct a multi-dimensional capability space adapted to the target robot; the multi-dimensional capability space includes one or more of an operational space, a movement space, a posture adjustment space, or an end-effector space; determining multi-channel mapping rules based on the characteristics of the multi-dimensional capability space and the control data from the interactive device; wherein the multi-channel mapping rules are used to structurally align the operational dimensions of the interactive device with the capability dimensions of the target robot; based on the multi-channel mapping rules, converting the control data into raw control commands that drive the corresponding capability dimensions; performing kinematic processing on the raw control commands to generate joint control commands that satisfy the kinematic constraints of the target robot; and issuing the joint control commands to the target robot for execution.

[0169] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0170] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A unified remote control method for heterogeneous robots, characterized in that, The method includes: Acquire control data from interactive devices; The system description information of the target robot is parsed to construct a multi-dimensional capability space adapted to the target robot; the multi-dimensional capability space includes one or more of the following: operation space, movement space, posture adjustment space, or end effector space. Based on the characteristics of the multidimensional capability space and the control data of the interactive device, a multi-channel mapping rule is determined; wherein, the multi-channel mapping rule is used to structurally align the operation dimension of the interactive device with the capability dimension of the target robot. Based on the multi-channel mapping rule, the control data is transformed into raw control commands that drive the corresponding capability dimension; The original control commands are kinematically processed to generate joint control commands that satisfy the kinematic constraints of the target robot; The joint control commands are sent to the target robot for execution.

2. The unified remote control method for heterogeneous robots according to claim 1, characterized in that, The step of determining the multi-channel mapping rules based on the characteristics of the multi-dimensional capability space and the control data of the interactive device includes: Obtain the task requirement characteristics of the current remote control task; Based on the features of the multidimensional capability space, the control data, and the task requirement features, a rule mapping logic is generated. The rule mapping logic is encapsulated into an instantiated configuration file to complete the determination of the multi-channel mapping rules; The multi-channel mapping rule is dynamically determined based on the task requirement characteristics, so that the same control data is mapped to different capability dimensions or given different control semantics under different task requirements.

3. The unified remote control method for heterogeneous robots according to claim 1, characterized in that, The multi-channel mapping rule includes at least one of the following mapping channels: The pose following mapping channel is used to map the pose increment of the handle, head display or motion tracker in the interactive device to the pose increment of the end effector or head or any joint of the target robot. The joystick speed mapping channel is used to map the continuous analog quantity of the joystick in the interactive device to the speed command of the target robot chassis, torso, or any joint. The trigger control mapping channel is used to map the pressing stroke of the trigger in the interactive device to a continuous control quantity or discrete control signal of the target robot gripper or dexterous hand. A button function mapping channel is used to map the digital state of the buttons in the interactive device to discrete control events of the target robot.

4. The unified remote control method for heterogeneous robots according to claim 1, characterized in that, The multi-channel mapping rule includes a multi-level button function mapping configuration, which decouples the discrete control signals of the interactive device into mutually independent functional levels according to the controlled object. The functional hierarchy includes: The body-level functional layer is used to map the underlying motion control functions of the target robot; The system-level functional layer is used to map the global state control functions of the remote control system; The view-level functional layer is used to map the display and feedback control functions from the operator's perspective.

5. The unified remote control method for heterogeneous robots according to claim 1, characterized in that, The method further includes: Based on the system description information, a joint state data stream is generated; By establishing a semantic namespace, the physical sensors of the target robot are uniformly mapped to standard semantic interfaces; The control data, the joint state data stream, and the sensor data output from each of the standard semantic interfaces are collected synchronously to generate a multimodal dataset.

6. The unified remote control method for heterogeneous robots according to claim 1, characterized in that, The step of performing kinematic processing on the original control commands to generate joint control commands that satisfy the kinematic constraints of the target robot includes: For subsystems involving spatial motion, extract the spatial pose increment from the original control command; Based on the preset coordinate system mapping relationship and motion scaling ratio, the spatial pose increment is converted into the pose increment of the robot end effector, and combined with the current end effector pose of the target robot, the target pose of the robot end effector is calculated. Using a solution framework that includes multi-task priorities, the inverse kinematics of the robot end-effector pose is calculated under the kinematic constraints of the target robot to obtain the target joint state sequence, which serves as the joint control command.

7. The unified remote control method for heterogeneous robots according to claim 6, characterized in that, After obtaining the target joint state sequence, the method further includes: Obtain the current actual state and dynamic boundary conditions of the target robot; Based on the aforementioned dynamic boundary conditions, and taking the current actual state as the initial starting point, the target joint state sequence is subjected to online interpolation and trajectory smoothing processing to output a smooth command sequence and issue it for execution.

8. A unified remote control system for heterogeneous robots, characterized in that, include: The data acquisition module is used to acquire control data from the interactive device; The capability space construction module is used to parse the system description information of the target robot and construct a multi-dimensional capability space that adapts to the target robot. The multidimensional capability space includes one or more of the following: operation space, movement space, attitude adjustment space, or end-effector space. The rule establishment module is used to determine multi-channel mapping rules based on the characteristics of the multi-dimensional capability space and the control data of the interactive device; wherein, the multi-channel mapping rules are used to structurally align the operation dimension of the interactive device with the capability dimension of the target robot. The control command generation module is used to convert the control data into raw control commands that drive the corresponding capability dimension based on the multi-channel mapping rules. The kinematics processing module is used to perform kinematic processing on the original control commands to generate joint control commands that satisfy the kinematic constraints of the target robot. The command issuing module is used to issue the joint control commands to the target robot for execution.

9. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the unified remote control method for heterogeneous robots as described in any one of claims 1 to 7.

10. A processor-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the unified remote control method for heterogeneous robots as described in any one of claims 1 to 7.

Images