Robot teleoperation control method and electronic equipment

Abstract

The invention provides a robot teleoperation control method and electronic equipment, and the method comprises the steps: obtaining target end data from control equipment; task information and a current posture of the robot are acquired, and action intention information and configuration prior information of the robot are determined according to the target end data, the task information and the current posture; according to preset constraint conditions, the action intention information of the robot, the configuration prior information, the target end data and the current attitude iteration, inverse kinematics solving processing is carried out, and joint control information of the robot is obtained; and controlling the robot to move according to the joint control information. When a user uses the control equipment, all parts of the robot can be naturally controlled at the same time, integrated and intuitive control is achieved, and learning cost and cognitive burden are reduced. And meanwhile, the collaboration fluency and efficiency can be improved. And moreover, high robustness under various working conditions can be ensured, and the method is particularly suitable for high-risk scenes.

Description

Technical Field

[0001] This application relates to the field of robot teleoperation control technology, and more specifically, to a robot teleoperation control method and electronic device. Background Technology

[0002] With the continuous advancement of artificial intelligence, robotics, and human-computer interaction, remotely operated robot systems are showing increasingly broad application prospects in complex scenarios such as industrial automation, special operations, and medical assistance. Particularly in fields such as nuclear industry, hazardous chemical handling, disaster relief, warehousing and logistics, and surgical assistance, remotely controlled robots can effectively replace humans in performing delicate tasks in high-risk or inaccessible environments.

[0003] Existing teleoperation systems typically rely on tablets or remote controls to move automated guided vehicles (AGVs), and then use virtual reality (VR) devices to control robotic arms separately.

[0004] However, this approach requires operators to frequently switch input devices and coordinate systems, affecting operational continuity. Furthermore, the fragmented interaction and high cognitive load lead to frequent operational errors. In addition, task planning for dual-arm collaboration is complex and unnatural: the lack of ergonomic modeling results in robot postures that are often "non-human" (such as abnormal elbow rotation or wrist twisting), disrupting the operator's intuitive understanding of the VR environment and affecting the smoothness of collaboration. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing a robot teleoperation control method and electronic device, thereby solving the problems of poor operational continuity, poor collaboration, and frequent operational errors in the prior art.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, one embodiment of this application provides a robot teleoperation control method, the method comprising: Acquire target end data from a control device, the control device comprising: a plurality of handles, the target end data including at least: the position and gesture of the handles; The robot's task information and current posture are acquired, and the robot's action intention information and configuration prior information are determined based on the target end-effector data, the task information, and the current posture. Based on preset constraints, the robot's motion intention information, the configuration prior information, the target end data, and the current posture iteration, inverse kinematics solution processing is performed to obtain the robot's joint control information. The joint control information includes: the joint position and joint velocity of each joint of the robot. The constraints include at least: joint soft limit constraints and singularity constraints. The robot's movement is controlled based on the joint control information.

[0007] Secondly, another embodiment of this application provides a robot teleoperation control device, the device comprising: An acquisition module is used to acquire target end data from a control device, the control device including: multiple handles, the target end data including at least: the position and gesture of the handles; The determination module is used to acquire the robot's task information and current posture, and determine the robot's action intention information and configuration prior information based on the target end-effector data, the task information, and the current posture. The solution module is used to perform inverse kinematics solving based on preset constraints, the robot's motion intention information, the configuration prior information, the target end data, and the current posture iteration to obtain the robot's joint control information. The joint control information includes the joint position and joint velocity of each joint of the robot. The constraints include at least the joint soft limit constraints and singularity constraints. The control module is used to control the movement of the robot based on the joint control information.

[0008] Thirdly, another embodiment of this application provides an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of any of the methods described in the first aspect above.

[0009] Fourthly, another embodiment of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of any of the methods described in the first aspect above.

[0010] The beneficial effects of this application are as follows: By acquiring target end-effector data from the control device, and obtaining the robot's task information and current posture, and based on the target end-effector data, task information, and current posture, the robot's motion intention information and configuration prior information are determined. Then, based on preset constraints, the robot's motion intention information, configuration prior information, target end-effector data, and current posture, inverse kinematics is iteratively solved to obtain the robot's joint control information. Based on this joint control information, the robot's movement is controlled, enabling users to simultaneously and naturally control all parts of the robot when using the control device, achieving integrated and intuitive control, reducing learning costs and cognitive burden. Simultaneously, it improves the smoothness and efficiency of collaboration. Furthermore, it ensures high robustness under various working conditions, making it particularly suitable for high-risk scenarios such as the nuclear industry and hazardous chemicals. In addition, it not only enables the robot to complete tasks but also allows it to complete tasks in an elegant, efficient, and biomechanically sound manner, thereby improving the robot's performance in terms of energy consumption, wear, and long-term stability. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A schematic diagram of a robot teleoperation control system provided in this application embodiment; Figure 2 A schematic flowchart of a robot teleoperation control method provided in an embodiment of this application; Figure 3 This is a flowchart illustrating the process of determining the robot's motion intention information and configuration prior information in the robot teleoperation control method provided in this application embodiment. Figure 4 This is a flowchart illustrating the process of determining the robot's configuration prior information in the robot teleoperation control method provided in this application embodiment. Figure 5 This is another flowchart illustrating the process of determining the robot's configuration prior information in the robot teleoperation control method provided in this application embodiment. Figure 6 This is another flowchart illustrating the process of determining the robot's configuration prior information in the robot teleoperation control method provided in the embodiments of this application. Figure 7This is a schematic flowchart illustrating the process of obtaining joint control information of a robot in the robot teleoperation control method provided in this application embodiment. Figure 8 This is a schematic flowchart illustrating the process of obtaining candidate joint control information of a robot in the robot teleoperation control method provided in this application embodiment. Figure 9 A schematic flowchart illustrating the process of controlling robot movement in the robot teleoperation control method provided in this application embodiment; Figure 10 A schematic diagram of a robot teleoperation control device provided in an embodiment of this application; Figure 11 This is a schematic diagram of the electronic device structure provided in an embodiment of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0014] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0015] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0016] Existing teleoperation systems typically rely on tablets or remote controls to move automated guided vehicles (AGVs), and then use virtual reality (VR) devices to control robotic arms separately.

[0017] However, this fragmented control mode requires operators to frequently switch input devices and coordinate systems, affecting operational continuity. Furthermore, the fragmented interaction and high cognitive load lead to frequent operational errors. In addition, task planning for dual-arm collaboration is complex and unnatural: the lack of ergonomic modeling results in robot postures that are often "non-human" (such as abnormal elbow rotation or wrist twisting), disrupting the operator's intuitive understanding of the VR environment and affecting the smoothness of collaboration.

[0018] Based on the aforementioned problems, this application proposes a robot teleoperation control method. This method acquires target end-effector data from a control device, along with the robot's task information and current posture. Based on the target end-effector data, task information, and current posture, it determines the robot's motion intention information and configuration prior information. Then, based on preset constraints, the robot's motion intention information, configuration prior information, target end-effector data, and current posture, iteratively performs inverse kinematics solving to obtain the robot's joint control information. Based on this joint control information, the robot's movement is controlled. This method enables integrated teleoperation control of multiple entities and also possesses advantages such as low cognitive load, high robustness, and strong safety.

[0019] First, the system architecture involved in the robot teleoperation control method provided in the embodiments of this application will be described in detail.

[0020] Figure 1 This is a schematic diagram of a system architecture for a robot teleoperation control system provided in an embodiment of this application, with reference to... Figure 1 As shown, the robot teleoperation control system includes the robot body platform, VR human-computer interaction equipment, and host computer.

[0021] The robot platform includes: two sets of seven-axis dual arms (each with 7 degrees of freedom), an end effector gripper, a height-adjustable external axis (Z-axis or mast type), an AGV mobile chassis, an image acquisition device, a drive module, and a control module. The control module is connected to the drive module and the image acquisition device, and the drive module is connected to the seven-axis dual arms, the end effector gripper, the height-adjustable external axis (Z-axis or mast type), and the AGV mobile chassis.

[0022] The VR human-computer interaction device includes a VR headset and multiple controllers. The host is an edge computing unit. The host communicates with the control module and the VR human-computer interaction device.

[0023] For example, VR human-computer interaction devices may include VR headsets and wireless controllers for providing an immersive three-dimensional field of view, rendering the robot's environment and the state of its own arms in real time, and collecting the operator's head orientation and hand spatial pose. At the same time, it also supports gesture recognition (such as pinching to indicate grasping, opening to indicate releasing, and switching collaboration modes with specific postures).

[0024] Optionally, during the initialization phase of the robot teleoperation control system, joint calibration can be performed, and safe operating area, safe operating area (Soft Limit Zone), prohibited area (Keep-out Zone), and task semantic templates (such as plugging, transporting, holding, etc.) can be configured in a unified coordinate system.

[0025] Joint calibration refers to unifying the VR coordinate system, robot base coordinate system, AGV odometer coordinate system, external axis coordinate system, and visual SLAM map coordinate system.

[0026] Optionally, the robot teleoperation control system can achieve efficient decoupling and real-time data interaction between components based on the Robot Operating System 2 (ROS2) communication middleware.

[0027] For example, the host is deployed on a local server or embedded high-performance industrial control computer close to the robot body platform, undertaking core algorithm processing and safety monitoring functions, while running high-frequency tasks such as SLAM, dynamic voxel map update, and safety zone modeling, forming an "edge intelligence" closed loop, which mainly includes the following key nodes: ROS2-VR node, ROS2-Camera node, ROS2-Arm node, and robot SDK interface layer.

[0028] For example, the robot teleoperation control system builds a distributed communication framework based on ROS2, with all functional modules running as independent nodes, and asynchronous collaboration achieved through topics, services and actions.

[0029] The ROS2-VR node is used to receive raw pose data (position, rotation, button triggers, etc.) from the VR headset and controllers, perform timestamp alignment, jitter filtering and latency compensation, and publish standardized input messages in the form of ROS2 topics.

[0030] The ROS2-Camera node is used to acquire data from the depth camera or multi-view vision sensor mounted on the robot body for environmental perception, SLAM mapping, and online obstacle avoidance support.

[0031] Among them, the ROS2-Arm node is used to subscribe to VR and camera data, perform motion intent decoding, configuration prior matching, anthropomorphic inverse kinematics solving, safe projection and active obstacle avoidance, and multi-agent coordinated control.

[0032] The robot SDK interface layer is used to call the API provided by the manufacturer to complete the final instruction issuance, including: issuing joint trajectory instructions to the dual-arm controller, issuing gripper opening and closing and force control mode switching instructions to the end effector, issuing mobile chassis speed and / or path planning instructions to the AGV navigation system, and issuing lifting axis position control instructions to the external axis driver.

[0033] The following describes in detail the robot teleoperation control method provided in the embodiments of this application with reference to several examples.

[0034] Figure 2 This is a flowchart illustrating a robot teleoperation control method provided in an embodiment of this application, with reference to... Figure 2 As shown, the execution subject of this method can be any electronic device with processing capabilities, such as the aforementioned robot teleoperation control system. The method includes: S201. Obtain target terminal data from the control device.

[0035] The control device includes multiple controllers, and the target end data includes at least the controller pose and controller gestures. The control device can be a VR device.

[0036] Optionally, the target end data also includes the motion trajectory of the handle.

[0037] Specifically, the pose of a controller refers to its position and orientation in three-dimensional space. The motion trajectory of a controller refers to the path traversed by the controller during continuous movement over a period of time. The gesture of a controller refers to a specific hand movement or static posture.

[0038] For example, users can operate the control device to generate target end-user data. For instance, they can trigger operations such as starting or stopping teleoperation, switching modes, and grasping using physical buttons and gestures on the VR controller, thus generating controller gestures. Furthermore, by moving the VR controller, the user can generate the controller's motion trajectory and pose.

[0039] S202. Obtain the robot's task information and current posture, and determine the robot's action intention information and configuration prior information based on the target end data, task information and current posture.

[0040] Optionally, the robot's task information and current posture can be acquired. The task information indicates the semantics of the task the robot will perform, such as insertion, handling, or holding. The current posture refers to the robot's current orientation.

[0041] Optionally, after obtaining the task information and the current posture, the robot's action intention can be decoded and the configuration prior matched based on the target end data, task information and the current posture, thereby obtaining the robot's action intention information and configuration prior information.

[0042] Among them, the motion intention information is used to indicate the relative motion relationship between each arm, which can be characterized by the control constraint information between each arm. Specifically, the control constraint information between each arm can be the restriction conditions of the motion degrees of freedom of each arm.

[0043] The configuration prior information is used to indicate the posture relationship between the arms. It refers to the joint configuration adopted by each arm of the robot under the premise of satisfying the target pose of the end effector. Specifically, it can be a joint configuration that is as close as possible to the natural posture distribution of the human upper limb under the same task.

[0044] For example, the target end data, task information, and current posture can be encoded and input into a pre-trained action intent decoding model, which will then predict the action intent information. Alternatively, the target end data, task information, and current posture can be encoded and input into a pre-trained configuration prior matching model, which will then predict the configuration prior information.

[0045] S203. Based on the preset constraints, the robot's motion intention information, configuration prior information, target end data, and current posture, perform inverse kinematics solution processing to obtain the robot's joint control information.

[0046] Optionally, under preset constraints, with configuration prior information as a bias, the robot's motion intention information, target end-effector data, and current posture can be comprehensively considered to iteratively perform inverse kinematics solution processing to obtain the robot's joint control information.

[0047] The joint control information includes the joint position and joint velocity of each joint of the robot, and the constraints include at least the joint soft limit constraints and singularity constraints.

[0048] Specifically, soft joint constraints refer to imposing exponential costs on joints approaching their upper or lower limits, dynamically "pushing" the available space away from the limits. Singularity constraints refer to imposing penalties based on the Jacobian condition number, minimum singularity, or operability metric to move away from singularities.

[0049] S204. Control the robot's movement based on joint control information.

[0050] Optionally, after obtaining the joint control information, control commands can be generated using the joint control information to control the robot's movement.

[0051] In this embodiment, target end-effector data from the control device is acquired, along with the robot's task information and current posture. Based on the target end-effector data, task information, and current posture, the robot's motion intention information and configuration prior information are determined. Then, based on preset constraints, the robot's motion intention information, configuration prior information, target end-effector data, and current posture, inverse kinematics is iteratively solved to obtain the robot's joint control information. The robot's movement is controlled according to this joint control information, allowing users to simultaneously and naturally control all parts of the robot when using the control device. This achieves integrated and intuitive control, reducing learning costs and cognitive burden. It also improves the smoothness and efficiency of collaboration. Furthermore, it ensures high robustness under various working conditions, making it particularly suitable for high-risk scenarios such as the nuclear industry and hazardous chemicals. In addition, it not only enables the robot to complete tasks but also allows it to do so in an elegant, efficient, and biomechanically sound manner, thereby improving the robot's performance in terms of energy consumption, wear, and long-term stability.

[0052] As one possible implementation method, Figure 3 This is a flowchart illustrating the process of determining the robot's motion intention information and configuration prior information in the robot teleoperation control method provided in this application embodiment, with reference to... Figure 3 As shown, in S202 above, the robot's action intention information and configuration prior information are determined based on the target end-effector data, task information, and current posture, including: S301. Determine the relative position of the control device's handle based on the target end data.

[0053] Optionally, the relative pose of the control device's handles can be calculated based on the poses of each handle in the target end data.

[0054] For example, based on the relative positions of each handle in the target end data, the relative positions of the handles are calculated by vector subtraction, and based on the relative attitudes of each handle in the target end data, the relative attitudes of the handles are calculated by attitude inverse operation, and the relative positions and attitudes of the handles are used as the relative poses of the handles.

[0055] S302. Determine the robot's motion intention information based on the relative pose of the control device's handle and the task information.

[0056] Optionally, the robot's action intent information can be obtained by inputting the relative position of the handle of the control device and the task information into a pre-trained intent classification model, and then using the intent classification model to predict the robot's action intent information, thereby regulating the robot's behavior pattern through the action intent information.

[0057] The intent classification model can be implemented based on lightweight neural networks (such as MLP, LSTM) or conditional Gaussian mixture models.

[0058] S303. Based on the target end data, task information, and current posture, determine the robot's configuration prior information.

[0059] Optionally, configuration prior matching can be performed from a preset human posture dataset based on the target end data, task information, and current posture to obtain the robot's configuration prior information. This configuration prior information can then be used to optimize the robot's execution method, thereby reducing the risk of misjudgment and enhancing the operator's control information.

[0060] As one possible implementation method, Figure 4 This is a flowchart illustrating the process of determining the robot's configuration prior information in the robot teleoperation control method provided in this application embodiment, with reference to... Figure 4 As shown, in S303 above, the robot's configuration prior information is determined based on the target end-effector data, task information, and current posture, including: S401. Encode the target terminal data and task information into a joint condition vector.

[0061] Optionally, the target terminal data can be extracted into continuous feature vectors, the task information can be encoded into semantic embedding vectors, and the continuous feature vectors and semantic embedding vectors can be fused through an attention mechanism to obtain a joint conditional vector.

[0062] S402. Input the joint conditional vector into the pre-trained configuration prior generation model, and use the configuration prior generation model to infer and generate at least one candidate pose of the robot.

[0063] Optionally, the joint conditional vector is input into the pre-trained configuration prior generation model, and the configuration prior generation model infers and generates at least one candidate pose of the robot.

[0064] Among them, the configuration prior generation model is a conditional normalized flow model or a conditional diffusion model; the configuration prior generation model is trained based on a preset human pose dataset.

[0065] For example, based on a pre-set large-scale human upper limb posture dataset (including 3D skeleton and joint angles), the conditional distribution of "given hand pose or relative pose → elbow, shoulder and wrist configuration" can be statistically analyzed in multi-task scenarios (including grasping, holding, delivering and inserting) to achieve data modeling.

[0066] For example, given the 6D pose of the end effector, the relative pose of the contralateral arm, and task information, a configuration prior generation model is trained to output the optimal or high-probability human joint configuration μ(·) and covariance Σ(·).

[0067] In one example, taking the configuration prior generative model as a conditional Gaussian mixture model, the joint conditional vector can be modeled as a mixture distribution of K Gaussian components, and the conditional distribution can be derived using Bayes' rule, thereby taking the conditional expectation and covariance as the prior mean and uncertainty measure.

[0068] S403. Based on each candidate posture and the current posture, determine the robot's configuration prior information.

[0069] Optionally, after obtaining each candidate pose, the degree of similarity between each candidate pose and the current pose can be evaluated, and the candidate pose that is closest to the current pose can be selected. Thus, the robot's configuration prior information can be obtained based on the candidate pose that is closest to the current pose.

[0070] Optionally, after obtaining each candidate posture, the similarity between each candidate posture and the current posture can be evaluated and corrected to determine the robot's configuration prior information.

[0071] As one possible implementation method, Figure 5 This is another flowchart illustrating the determination of robot configuration prior information in the robot teleoperation control method provided in this application embodiment, referring to... Figure 5 As shown, in S403 above, the robot's configuration prior information is determined based on each candidate pose and the current pose, including: S501. Determine the robot's initial configuration prior information based on each candidate posture and the robot's current posture.

[0072] Optionally, each candidate pose can be traversed, and for the current candidate pose encountered, the similarity between the current candidate pose and the current pose can be calculated. The similarity can be the Euclidean distance.

[0073] In one example, after traversal, the candidate pose with the smallest distance can be determined from the candidate poses by using the nearest neighbor method based on the similarity of each candidate pose. Based on the candidate pose with the smallest distance, the robot's initial configuration prior information can be determined, thereby maximizing the smoothness of the motion.

[0074] In another example, after the traversal is complete, the candidate poses can be weighted and fused based on their similarity to obtain the target pose. Based on the target pose, the robot's initial configuration prior information can be determined, thus combining the advantages of multiple candidate poses and avoiding jitter caused by small deviations in a single candidate pose.

[0075] S502. Based on the robot's current posture and the preset safety space, construct a soft limit confidence matrix.

[0076] Optionally, a soft constraint confidence matrix can be constructed based on the robot's current posture and a preset safety space. This soft constraint confidence matrix can be used to characterize the hard safety constraints in the physical space, thereby guiding the inverse kinematics solution to converge in a safer direction.

[0077] Optionally, in the current posture, the relative distance between the current angle of each joint of the robot and its preset range of motion boundary can be calculated, and a confidence value for each joint can be generated based on the relative distance.

[0078] Optionally, the confidence values ​​of each joint angle can be used as diagonal elements to construct a diagonal matrix, thus obtaining the soft-limit confidence matrix.

[0079] The preset safety space includes the preset motion range boundary, obstacles or dangerous areas in the physical space, and dynamic obstacle avoidance areas. The soft limit confidence matrix is ​​a diagonal matrix dynamically constructed from the relative distances between the current angle of each joint of the robot and its preset motion range boundary, where each diagonal element represents the degree of confidence in the available space of the corresponding joint in its current posture.

[0080] S503. Based on the soft constraint confidence matrix and the initial configuration prior information, determine the robot's configuration prior information.

[0081] Optionally, the initial configuration prior information is anisotropically modulated based on the soft constraint confidence matrix to obtain the robot's configuration prior information.

[0082] Optionally, the initial configuration prior information is verified and corrected based on the soft constraint confidence matrix to obtain the robot's configuration prior information.

[0083] By considering each candidate posture and the robot's current posture, the initial configuration prior information of the robot is determined. Based on the robot's current posture and the preset safety space, a soft constraint confidence matrix is ​​constructed. Thus, based on the soft constraint confidence matrix and the initial configuration prior information, the robot's configuration prior information is determined. This ensures that the obtained configuration prior information not only conforms to human intuition but also highly adapts to the current state and strictly adheres to physical safety constraints, thereby greatly improving the smoothness, safety, and user experience of teleoperation.

[0084] As one possible implementation method, Figure 6 This is another flowchart illustrating the process of determining the robot's configuration prior information in the robot teleoperation control method provided in this application embodiment, referring to... Figure 6 As shown, in S503 above, the robot's configuration prior information is determined based on the soft constraint confidence matrix and the initial configuration prior information, including: S601. Based on the soft-limit confidence matrix, anisotropically modulate the initial configuration prior information to obtain the adjusted configuration prior information.

[0085] Optionally, the difference between the initial configuration prior information and the current attitude can be calculated, and the difference information can be multiplied element-wise with the soft constraint confidence matrix to obtain the adjusted configuration prior information.

[0086] For example, the component corresponding to each joint in the difference information is multiplied by the confidence value of that joint in the soft constraint confidence matrix to obtain the modulated component of each joint, and the modulated component of each joint is added to the corresponding joint of the current posture to obtain the adjusted configuration prior information.

[0087] By using a soft-limit confidence matrix, the initial configuration prior information is anisotropically modulated to obtain adjusted configuration prior information. This significantly suppresses the expected changes in joints near the boundaries of the motion range, while keeping the expected changes in joints with a sufficient motion range essentially constant. This allows each joint to automatically avoid dangerous directions approaching the motion limits, ensuring the safety of the robot during movement.

[0088] S602. Perform self-collision detection on the adjusted configuration prior information. If there is a self-collision conflict, project the adjusted configuration prior information onto the target collision-free configuration space to obtain the robot's configuration prior information. If there is no self-collision conflict, use the adjusted configuration prior information as the configuration prior information.

[0089] Optionally, the prior information of the adjusted configuration can be used to perform self-collision detection on the robot's three-dimensional geometric model to determine whether there is physical penetration or excessive distance between the robot's links. If so, it is determined that there is a self-collision conflict; if not, it is determined that there is no self-collision conflict.

[0090] Optionally, if there is no self-collision conflict, the adjusted configuration prior information is used as the configuration prior information.

[0091] Optionally, if self-collision conflicts exist, the adjusted configuration prior information can be projected onto the target collision-free configuration space using a preset optimization solver to obtain the robot's configuration prior information. The target collision-free configuration space is the nearest, collision-free configuration space.

[0092] For example, the adjusted configuration prior information can be projected into the target collision-free configuration space using a preset optimization solver, which can be achieved by solving a constrained optimization problem.

[0093] By performing self-collision detection on the adjusted configuration prior information, and projecting the adjusted configuration prior information onto the target collision-free configuration space when self-collision conflicts exist, the robot's configuration prior information can be obtained, which can ensure absolute physical safety, improve system robustness and reliability, and maintain the continuity and smoothness of operation.

[0094] As one possible implementation method, Figure 7 This is a flowchart illustrating the process of obtaining joint control information of a robot in the robot teleoperation control method provided in this application embodiment, with reference to... Figure 7 As shown, in step S203 above, inverse kinematics is solved based on preset constraints, robot motion intention information, configuration prior information, target end-effector data, and current posture iterations to obtain the robot's joint control information, including: S701. Using the current posture as the initial value, perform inverse kinematics solution processing based on preset constraints, target end data, robot motion intention information, and configuration prior information to obtain the robot's candidate joint control information.

[0095] Optionally, the current posture can be used as the initial value, and based on preset constraints, target end data, robot motion intention information, and configuration prior information, inverse kinematics can be solved using the damped least squares method or the damped pseudo-inverse method to obtain the robot's candidate joint control information.

[0096] S702. Based on the candidate joint control information of the robot, determine whether the robot exceeds the preset safe space under the current joint control information.

[0097] Optionally, the candidate joint control information of the robot can be verified and iterated to determine whether the robot exceeds the preset safety space under the current joint control information.

[0098] For example, the robot's future posture is derived from the robot's candidate joint control information, and it is determined whether the robot's future posture exceeds a preset safety space.

[0099] S703. If so, acquire external motion information, and based on the preset multi-objective optimization function and external motion information, perform iterative inverse kinematics solution processing on the robot's action intention information and configuration prior information to obtain the robot's candidate joint control information.

[0100] Optionally, if the robot exceeds the preset safety space under the current joint control information, external motion information can be obtained, and the robot's action intention information and configuration prior information can be iteratively solved by performing inverse kinematics on the preset multi-objective optimization function and external motion information to obtain the robot's candidate joint control information.

[0101] Among them, the external motion information can be the motion information of the liftable external axis (Z-axis or mast type) and the AGV mobile chassis.

[0102] S704. If not, the candidate joint control information will be used as the robot's joint control information.

[0103] Optionally, if the robot does not exceed the preset safety space under the current joint control information, the candidate joint control information can be used as the robot's joint control information.

[0104] By using the current posture as the initial value, and performing inverse kinematics processing based on preset constraints, target end-effector data, robot motion intent information, and configuration prior information, the robot's candidate joint control information is obtained. Based on the robot's candidate joint control information, it is determined whether the robot exceeds the preset safety space under the current joint control information. If the robot exceeds the preset safety space under the current joint control information, external motion information is acquired. Based on the preset multi-objective optimization function and external motion information, iterative inverse kinematics processing is performed on the robot's motion intent information and configuration prior information to obtain the robot's candidate joint control information. This enables dynamic and proactive obstacle avoidance, while ensuring efficient and smooth daily operation and exhibiting sufficient intelligence and robustness in complex scenarios, achieving "fast in normal times and stable in critical situations".

[0105] As one possible implementation method, Figure 8 This is a schematic flowchart illustrating the process of obtaining candidate joint control information of a robot in the robot teleoperation control method provided in this application embodiment, with reference to... Figure 8 As shown, S701 above uses the current posture as the initial value, and performs inverse kinematics solving based on preset constraints, target end-effector data, robot motion intention information, and configuration prior information to obtain the robot's candidate joint control information, including: S801. Determine the target end-effector motion information of the robot based on the target end-effector data and the robot's motion intention information.

[0106] Optionally, the target end-effector motion information of the robot can be calculated based on the target end-effector data and the robot's motion intention information.

[0107] Among them, the target end motion information refers to the target end motion velocity and the target end motion displacement.

[0108] For example, the target end data can be parsed and the pose can be extracted. The pose can be corrected by motion intention information to obtain the final target pose. Then, the target end motion velocity and target end motion displacement can be calculated based on the final target pose.

[0109] S802. Calculate the Jacobian matrix of the robot in the current posture.

[0110] Optionally, the Jacobian matrix of the robot in the current posture can be calculated based on the geometric model of the robot in the current posture, so as to represent the mapping relationship of the robot from joint space to task space through the Jacobian matrix.

[0111] S803. By using the damped pseudo-inverse method, the motion information of the target end is solved to obtain the principal solution.

[0112] Alternatively, the principal solution can be obtained by solving for the target end motion information using the damped least squares method or the damped pseudo-inverse method.

[0113] The principal solution refers to the most basic and direct motion components that the robot's joint space needs to execute in order to accurately achieve the target end-effector data specified by the user via the handle. The principal solution is a set of key velocity commands.

[0114] Obtaining the master solution ensures the responsiveness and accuracy of the system, which is a prerequisite for effective teleoperation.

[0115] S804. Based on the Jacobian matrix, construct the null space projection matrix.

[0116] Optionally, after obtaining the Jacobian matrix, a null space projection matrix can be constructed using the Jacobian matrix. Here, null space refers to the joint motion directions that do not contribute to the primary task (end-effector motion) but affect the robot's internal configuration. Using null space, other secondary objectives can be optimized while satisfying the end-effector motion.

[0117] The null-space projection matrix can identify and extract joint motion directions that do not contribute to the robot's end effector motion but can be used to optimize the robot's internal posture (such as elbow flexion direction and shoulder rotation angle). Through the null-space projection matrix, the primary and secondary tasks can be decoupled, allowing the system to actively guide the robot to adopt ergonomically natural postures while ensuring operational accuracy.

[0118] S805. Based on the master solution, preset constraints, configuration prior information, and null space projection matrix, determine the candidate joint control information of the robot.

[0119] Optionally, after obtaining the master solution, the master solution can be compensated by preset constraints, robot motion intention information, configuration prior information, and null space projection matrix to obtain candidate joint control information of the robot.

[0120] As one possible implementation, in S805 above, candidate joint control information for the robot is determined based on the master solution, preset constraints, configuration prior information, and null space projection matrix, including: Based on the preset constraints and configuration prior information, the shaping velocity is generated; based on the null space projection matrix, the shaping velocity is projected as a compensation term; based on the principal solution and the compensation term, the candidate joint control information of the robot is determined.

[0121] Optionally, a target cost function can be constructed based on preset constraints and configuration prior information, and a shaping velocity can be generated through the direction of the negative gradient of the target cost function, thereby representing the desired joint velocity vector through the shaping velocity.

[0122] The smaller the value of the objective cost function, the more ideal the desired joint velocity vector. The shaping velocity indicates how all joints should coordinate their movements to make the robot's posture closer to the "natural human posture" and better meet various safety constraints.

[0123] Optionally, a matrix-vector multiplication operation can be performed between the shaping velocity and the null space projection matrix to project the shaping velocity as a compensation term, thereby removing the components in the shaping velocity that affect the position and attitude of the end effector, and retaining only the part that is purely used to adjust the internal configuration.

[0124] Optionally, after obtaining the compensation term, the main solution is superimposed with the compensation term to obtain the candidate joint control information of the robot. This results in the candidate joint control information containing precise end-effector motion commands and optimized internal posture adjustments, which can balance task completion, operational intuition, and motion safety.

[0125] As one possible implementation method, Figure 9This is a flowchart illustrating the process of controlling robot movement in the robot teleoperation control method provided in this application embodiment, with reference to... Figure 9 As shown, in S204 above, controlling the robot's movement based on joint control information includes: S901. Projection processing is performed based on the joint control information to obtain the robot's motion posture corresponding to the joint control information.

[0126] Optionally, the joint control information can be processed by constraint optimization or projection method to obtain the robot motion posture corresponding to the joint control information.

[0127] S902. Based on the preset safety space, perform safety verification on the robot's motion posture and generate motion control commands corresponding to the joint control information.

[0128] Optionally, the robot's motion posture is verified through a preset safety space, and motion control commands corresponding to the joint control information are generated based on the results of the safety verification.

[0129] The preset safe space includes preset movement range boundaries, obstacles or dangerous areas in the physical space, and dynamic obstacle avoidance areas.

[0130] For example, safety checks include: checking whether all joint angles exceed the preset range of motion boundaries, checking whether there is self-collision between the links of the robot itself, and checking whether any part of the robot is in an obstacle or danger zone in physical space or collides with a known obstacle.

[0131] S903 controls robot movement based on motion control commands.

[0132] Optionally, motion control commands can be sent to the robot SDK interface layer.

[0133] Based on the same inventive concept, this application also provides a robot teleoperation control device corresponding to the robot teleoperation control method. Since the principle of the device in this application is similar to the robot teleoperation control method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0134] Reference Figure 10 As shown, Figure 10 This is a schematic diagram of a robot teleoperation control device provided in an embodiment of the present application. The device includes: an acquisition module 1001, a determination module 1002, a solution module 1003, and a control module 1004. The acquisition module 1001 is used to acquire target end data from the control device, the control device including: multiple handles, and the target end data including at least: the position and gesture of the handles; The determination module 1002 is used to acquire the robot's task information and current posture, and determine the robot's action intention information and configuration prior information based on the target end data, task information and current posture; The solver module 1003 is used to perform inverse kinematics solving based on preset constraints, robot motion intention information, configuration prior information, target end data and current posture iteration to obtain the robot's joint control information. The joint control information includes: the joint position and joint velocity of each joint of the robot. The constraints include at least: joint soft limit constraints and singularity constraints. The control module 1004 is used to control the robot's movement based on joint control information.

[0135] Optionally, module 1002 is specifically used for: Based on the target end data, determine the relative pose of the control device's handle; Based on the relative pose of the control device's handle and the task information, determine the robot's motion intention information; Based on the target end data, task information, and current posture, determine the robot's configuration prior information.

[0136] Optionally, module 1002 is specifically used for: Encode the target terminal data and task information into a joint condition vector; The joint conditional vector is input into the pre-trained configuration prior generation model, and the configuration prior generation model infers and generates at least one candidate pose of the robot. The configuration prior generation model is a conditional normalized flow model or a conditional diffusion model. Based on each candidate pose and the current pose, the robot's configuration prior information is determined.

[0137] Optionally, module 1002 is specifically used for: Based on each candidate pose and the robot's current pose, determine the robot's initial configuration prior information; Based on the robot's current posture and the preset safety space, a soft limit confidence matrix is ​​constructed. The soft limit confidence matrix is ​​a diagonal matrix dynamically constructed from the relative distance between the current angle of each joint of the robot and its preset motion range boundary. Each diagonal element represents the degree of confidence in the available space of the corresponding joint in its current posture. Based on the soft constraint confidence matrix and the initial configuration prior information, the configuration prior information of the robot is determined.

[0138] Optionally, module 1002 is specifically used for: Based on the soft-limit confidence matrix, the initial configuration prior information is anisotropically modulated to obtain the adjusted configuration prior information. Self-collision detection is performed on the adjusted configuration prior information. If a self-collision conflict exists, the adjusted configuration prior information is projected into the target collision-free configuration space to obtain the robot's configuration prior information. If no self-collision conflict exists, the adjusted configuration prior information is used as the configuration prior information.

[0139] Optionally, the solver module 1003 is specifically used for: Using the current posture as the initial value, inverse kinematics is performed based on preset constraints, target end data, robot motion intention information, and configuration prior information to obtain the robot's candidate joint control information. Based on the robot's candidate joint control information, determine whether the robot exceeds the preset safety space under the current joint control information; If so, external motion information is acquired, and based on the preset multi-objective optimization function and external motion information, iterative inverse kinematics processing is performed on the robot's action intention information and configuration prior information to obtain the robot's candidate joint control information. If not, the candidate joint control information will be used as the robot's joint control information.

[0140] Optionally, the solver module 1003 is specifically used for: Based on the target end-effector data and the robot's motion intent information, determine the robot's target end-effector motion information; Calculate the Jacobian matrix of the robot in the current pose; By using the damped pseudo-inverse method, the motion information of the target end is solved to obtain the principal solution; Based on the Jacobian matrix, the null space projection matrix is ​​constructed. Based on the master solution, preset constraints, configuration prior information, and null space projection matrix, the candidate joint control information of the robot is determined.

[0141] Optionally, the solver module 1003 is specifically used for: Based on preset constraints and prior configuration information, the shaping speed is generated. Based on the null space projection matrix, the shaping velocity is projected as a compensation term; Based on the master solution and the compensation term, the candidate joint control information of the robot is determined.

[0142] Optionally, the control module 1004 is specifically used for: Projection processing is performed based on the joint control information to obtain the robot's motion posture corresponding to the joint control information; Based on the preset safety space, the robot's motion posture is checked for safety, and motion control commands corresponding to the joint control information are generated. The robot's movement is controlled based on motion control commands.

[0143] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0144] This application also provides an electronic device, such as... Figure 11 As shown, Figure 11 The schematic diagram of the electronic device structure provided in this application embodiment includes: a processor 1101 and a memory 1102, and optionally, a bus 1103. The memory 1102 stores machine-readable instructions executable by the processor 1101. When the electronic device is running, the processor 1101 and the memory 1102 communicate through the bus 1103, and the processor 1101 executes the machine-readable instructions to perform the steps of the above-described robot teleoperation control method.

[0145] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described robot teleoperation control method.

[0146] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0147] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0148] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for remotely controlling a robot, characterized in that, include: Acquire target end data from a control device, the control device comprising: a plurality of handles, the target end data including at least: the position and gesture of the handles; The robot's task information and current posture are acquired, and the robot's action intention information and configuration prior information are determined based on the target end-effector data, the task information, and the current posture. Based on preset constraints, the robot's motion intention information, the configuration prior information, the target end data, and the current posture iteration, inverse kinematics solution processing is performed to obtain the robot's joint control information. The joint control information includes: the joint position and joint velocity of each joint of the robot. The constraints include at least: joint soft limit constraints and singularity constraints. The robot's movement is controlled based on the joint control information.

2. The robot teleoperation control method according to claim 1, characterized in that, The step of determining the robot's action intent information and configuration prior information based on the target end-effector data, the task information, and the current posture includes: Based on the target end data, determine the relative pose of the handle of the control device; Based on the relative pose of the handle of the control device and the task information, the robot's action intention information is determined; Based on the target end data, the task information, and the current posture, the robot's configuration prior information is determined.

3. The robot teleoperation control method according to claim 2, characterized in that, The step of determining the robot's configuration prior information based on the target end data, the task information, and the current posture includes: The target terminal data and the task information are encoded into a joint condition vector; The joint conditional vector is input into a pre-trained configuration prior generation model, and the configuration prior generation model infers and generates at least one candidate pose of the robot. The configuration prior generation model is a conditional normalized flow model or a conditional diffusion model. Based on each of the candidate poses and the current pose, the configuration prior information of the robot is determined.

4. The robot teleoperation control method according to claim 3, characterized in that, The step of determining the robot's configuration prior information based on each of the candidate poses and the current pose includes: Based on each of the candidate poses and the robot's current pose, the initial configuration prior information of the robot is determined; Based on the robot's current posture and the preset safety space, a soft limit confidence matrix is ​​constructed. The soft limit confidence matrix is ​​a diagonal matrix dynamically constructed from the relative distance between the current angle of each joint of the robot and its preset motion range boundary. Each diagonal element represents the degree of confidence in the available space of the corresponding joint in its current posture. Based on the soft constraint confidence matrix and the initial configuration prior information, the configuration prior information of the robot is determined.

5. The robot teleoperation control method according to claim 4, characterized in that, The step of determining the robot's configuration prior information based on the soft constraint confidence matrix and the initial configuration prior information includes: Based on the soft-limit confidence matrix, the initial configuration prior information is anisotropically modulated to obtain the adjusted configuration prior information; Self-collision detection is performed on the adjusted configuration prior information. If a self-collision conflict exists, the adjusted configuration prior information is projected into the target collision-free configuration space to obtain the configuration prior information of the robot. If no self-collision conflict exists, the adjusted configuration prior information is used as the configuration prior information.

6. The robot teleoperation control method according to claim 1, characterized in that, The process of performing inverse kinematics solving based on preset constraints, the robot's motion intention information, the configuration prior information, the target end effector data, and the current posture iterations to obtain the robot's joint control information includes: Using the current posture as the initial value, inverse kinematics is performed based on preset constraints, the target end data, the robot's motion intention information, and the configuration prior information to obtain the candidate joint control information of the robot. Based on the candidate joint control information of the robot, determine whether the robot exceeds the preset safe space under the current joint control information; If so, external motion information is acquired, and based on the preset multi-objective optimization function and external motion information, iterative inverse kinematics processing is performed on the robot's action intention information and configuration prior information to obtain the robot's candidate joint control information. If not, the candidate joint control information will be used as the joint control information of the robot.

7. The robot teleoperation control method according to claim 6, characterized in that, The process involves using the current posture as an initial value, and performing inverse kinematics solving based on preset constraints, the target end-effector data, the robot's motion intention information, and the configuration prior information to obtain the robot's candidate joint control information, including: Based on the target end-effector data and the robot's motion intent information, the target end-effector motion information of the robot is determined; Based on the current posture, calculate the Jacobian matrix of the robot in the current posture; The principal solution is obtained by solving the motion information of the target end effector using the damped pseudo-inverse method. Based on the Jacobian matrix, the null space projection matrix is ​​constructed. Based on the master solution, the preset constraints, the configuration prior information, and the null space projection matrix, the candidate joint control information of the robot is determined.

8. The robot teleoperation control method according to claim 7, characterized in that, The step of determining the candidate joint control information of the robot based on the master solution, the preset constraints, the configuration prior information, and the null space projection matrix includes: Based on the preset constraints and the prior configuration information, a shaping speed is generated; Based on the null space projection matrix, the shaping velocity is projected as a compensation term; Based on the master solution and the compensation term, the candidate joint control information of the robot is determined.

9. The robot teleoperation control method according to claim 1, characterized in that, The step of controlling the robot's movement based on the joint control information includes: The robot motion posture corresponding to the joint control information is obtained by projection processing based on the joint control information. Based on the preset safety space, the robot's motion posture is checked for safety, and motion control commands corresponding to the joint control information are generated. The robot's movement is controlled based on the motion control commands.

10. An electronic device, characterized in that, include: The electronic device includes a processor and a memory, the memory storing machine-readable instructions executable by the processor, which, when the electronic device is in operation, are executed by the processor to perform the steps of the robot teleoperation control method as described in any one of claims 1 to 9.

Images