Control method, model training method, robot system, and electronic device

Abstract

The invention provides a control method, a model training method, a robot system and electronic equipment. The control method comprises the steps that terminal state information and environment perception information at the current moment are acquired; the end state information comprises the pose of the end effector in the task space; inputting the end state information and the environment perception information into an action prediction model, and generating action increment prediction of the end effector; wherein the action prediction model is obtained by training based on spatial alignment training data, and the spatial alignment training data is acquired by an acquisition device; based on the current pose and action increment prediction of the end effector in the task space, obtaining a joint space trajectory of the motion execution mechanism; and controlling a motion execution mechanism to execute an action corresponding to the joint space trajectory. The control method can improve the control efficiency of the robot system.

Description

Technical Field

[0001] The present application relates to the field of robotics, and in particular to a control method, a model training method, a robotic system, an electronic device, and a storage medium. Background Art

[0002] Currently, data collection for robot imitation learning mostly relies on the robot itself. The robot's operation is controlled through master-slave manipulator teleoperation, motion capture teleoperation, and other methods, recording relevant data during the robot's operation. This recorded data is then used to reason about the robot's movements. However, these data collection solutions are costly, requiring at least one robot for each data collection session. Furthermore, the data obtained from these data collection solutions lacks generalizability, making it difficult to apply to robots of different configurations. For example, data collected from a robot with a seven-axis manipulator arm is difficult to use for reasoning about the movements of a robot with a six-axis manipulator, reducing data utilization. Summary of the Invention

[0003] The embodiments of the present application disclose a control method, a model training method, a robot system, and an electronic device, which solve the technical problem of low data utilization in the field of robotics.

[0004] The present application provides a control method, which is applied to a robot system including a motion actuator and an end effector, and the method includes the following steps: obtaining end state information and environmental perception information at a current moment; the end state information includes the position and posture of the end effector in a task space; inputting the end state information and the environmental perception information into a motion prediction model to generate a motion increment prediction of the end effector; wherein the motion prediction model is trained based on spatially aligned training data, and the spatially aligned training data is collected by an acquisition device; based on the current position and posture of the end effector in the task space and the motion increment prediction, the joint space trajectory of the motion actuator is obtained; and the motion actuator is controlled to perform the action corresponding to the joint space trajectory.

[0005] In some embodiments of the present application, the environmental perception information includes at least one of the following: RGB images and / or depth images and / or point cloud data collected by the visual sensor deployed on the end effector; six-dimensional force / torque information collected by the force sensor installed on the end effector; and joint torque sensor data set on the motion actuator.

[0006] In some embodiments of the present application, the method further includes: determining task observation information based on the environmental perception information; wherein: the task observation information includes: the task-related object posture extracted from the environmental perception information by a target detection algorithm, and the motion state prediction information of the task-related object; wherein, inputting the terminal state information and the environmental perception information into the action prediction model to generate the action incremental prediction of the end effector includes: synchronously inputting the terminal state information, the environmental perception information and the task observation information into the action prediction model to generate the action incremental prediction.

[0007] In some embodiments of the present application, the joint space trajectory of the motion actuator is obtained based on the current posture of the end effector in the task space and the action increment prediction, including: calculating the target posture P_target based on the action increment prediction, P_target = P_current ⊕ ΔP, wherein the target posture P_current is the current posture of the end effector in the task space, ΔP is the action increment prediction, and ⊕ represents the posture synthesis operator of the task space; inputting the target posture into the inverse kinematics solution module, and outputting the joint angle set of the motion actuator; based on the joint angle set, generating a joint space trajectory that satisfies the motion constraints through a trajectory planner.

[0008] In some embodiments of the present application, the trajectory planner performs the following operations: inserting intermediate points between adjacent joint angles to generate a path point sequence; calculating the timestamps of each path point in the path point sequence based on preset joint velocity limits and acceleration limits; and generating a time-parameterized joint trajectory function based on the timestamps of each path point.

[0009] In some embodiments of the present application, the method also includes: when the motion actuator is a serial robotic arm, the joint angle set is a joint angle vector; when the motion actuator is a parallel mechanism, the joint angle set is an active joint variable; when the motion actuator is a flexible continuum robot, the joint angle set is a drive length vector.

[0010] In some embodiments of the present application, the method further includes: the acquisition device and the tool center point of the end effector are defined in the same way; and the sensor installation parameters of the acquisition device are aligned with the sensor observation reference of the end effector.

[0011] In some embodiments of the present application, the end effector includes a first effector and a second effector, and the inputting of the end state information and the environmental perception information into a motion prediction model to generate a motion incremental prediction of the end effector includes: determining a posture transformation value between the first effector and the second effector based on the end state information; inputting the end state information, the posture transformation value and the environmental perception information into the motion prediction model to generate a motion incremental prediction of the first effector and a motion incremental prediction of the second effector.

[0012] In some embodiments of the present application, the motion actuator includes a first robotic arm and a second robotic arm, and the method further includes: obtaining the joint space trajectory of the first robotic arm based on the current posture of the first actuator in the task space and the motion increment prediction of the first actuator, and controlling the first robotic arm to perform the motion of the joint space trajectory of the first robotic arm; obtaining the joint space trajectory of the second robotic arm based on the current posture of the second actuator in the task space and the motion increment prediction of the second actuator, and controlling the second robotic arm to perform the motion of the joint space trajectory of the second robotic arm.

[0013] The present application also provides a model training method, which includes: obtaining first data collected by an acquisition device; spatially aligning the first data to obtain second data; and using the second data to train a preset model to obtain a motion prediction model applied to a robot system.

[0014] In some embodiments of the present application, the method further includes: determining task observation information based on the spatially aligned environmental perception information in the second data; wherein, the task observation information includes: the task-related object posture extracted from the spatially aligned environmental perception information by a target detection algorithm, and the motion state prediction information of the task-related object; wherein, using the second data to train a preset model to obtain a motion prediction model applied to the robot system includes: using the second data and the task observation information to train the preset model to obtain the motion prediction model.

[0015] In some embodiments of the present application, the acquisition device includes a first device and a second device, and the method further includes: determining a posture transformation value between the first device and the second device based on the posture of the first device and the posture of the second device in the second data; wherein, using the second data to train a preset model to obtain a motion prediction model applied to the robot system includes: using the second data and the posture transformation value to train the preset model to obtain the motion prediction model.

[0016] In some embodiments of the present application, the use of the second data to train a preset model to obtain a motion prediction model applied to the robot system includes: obtaining a candidate model when the number of training times or the preset model meets preset conditions; determining the real motion increment based on the posture of the acquisition device at the tth moment and the posture of the acquisition device at the t-1th moment, where t is a positive integer; calculating the difference between the real motion increment and the predicted motion increment prediction output by the candidate model; and using the candidate model as the motion prediction model when the difference is less than or equal to the preset difference.

[0017] In some embodiments of the present application, the first data includes: terminal state information and environmental perception information of the acquisition device; wherein, the terminal state information includes the position of the acquisition device in the task space; the environmental perception information includes at least one of the following: RGB images and / or depth images and / or point cloud data collected by the visual sensor deployed on the acquisition device; six-dimensional force / torque information collected by the force sensor installed on the acquisition device.

[0018] In some embodiments of the present application, spatially aligning the first data to obtain second data includes: spatially aligning the terminal state information and the environment perception information to obtain the second data.

[0019] In some embodiments of the present application, the method further includes: performing distortion correction on the point cloud data using a Kalman filter algorithm to obtain distortion-corrected point cloud data.

[0020] In some embodiments of the present application, the method further includes: synchronizing the RGB image and / or depth image and / or point cloud data collected by the visual sensor and the six-dimensional force / torque information collected by the force sensor based on the timestamp.

[0021] The present application also provides a robot system, which includes a processor and a memory, and the processor is used to implement the control method when executing a computer program stored in the memory.

[0022] The present application also provides an electronic device, which includes a processor and a memory, and the processor is used to implement the model training method when executing the computer program stored in the memory.

[0023] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the control method or model training method is implemented.

[0024] In the control method provided in the application embodiment, the terminal state information and environmental perception information at the current moment are obtained to provide basic data for the subsequent planning of the joint space trajectory of the motion actuator. The terminal state information and environmental perception information are input into the motion prediction model to generate the motion increment prediction of the end effector. The training data is collected by the acquisition device, and the motion prediction model obtained by training the spatially aligned training data outputs the motion increment prediction. This can avoid directly using the robot body to collect data for the training model, reducing the cost of data collection and the maintenance cost of the robot body. In addition, the method of not relying on the robot to collect training data to reproduce the robot's movements can improve the utilization rate of the motion prediction model.

[0025] Based on the motion increment prediction and combined with the current position of the end effector in the task space, the joint space trajectory of the motion actuator is obtained, and the motion actuator is controlled to perform the action corresponding to the joint space trajectory, thereby improving the control efficiency and accuracy of the robot system.

[0026] In the model training method provided in the application embodiment, first data collected by an acquisition device is obtained and spatially aligned, which can improve data accuracy and reduce interference from noisy data. A preset model is trained using the second data to obtain a motion prediction model. This motion prediction model is applied to a robotic system, reducing the cost of directly using the robotic system for training data collection and improving training accuracy to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a schematic diagram of an application scenario of the control method provided in an embodiment of the present application.

[0028] Figure 2 This is a flow chart of the control method provided in an embodiment of the present application.

[0029] Figure 3 This is a flowchart of a control method based on a first actuator and a second actuator provided in an embodiment of the present application.

[0030] Figure 4 This is a schematic diagram of generating the joint space trajectory of the first robotic arm and the joint space trajectory of the second robotic arm provided in an embodiment of the present application.

[0031] Figure 5 This is a flowchart of the model training method provided in an embodiment of the present application.

[0032] Figure 6 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] To facilitate understanding, some illustrations of concepts related to the embodiments of the present application are given for reference.

[0034] It should be noted that, in this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A alone, A and B together, and B alone, where A and B can be singular or plural. The terms "first," "second," "third," "fourth," and so on (if any) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or precedence.

[0035] Currently, data collection for robot imitation learning mostly relies on the robot itself. This is achieved through teleoperation using master-slave manipulators, motion capture, and other methods, which allows the robot to operate and record relevant data during the operation. For example, this includes recording the states of each robot's joints during the operation, as well as data from sensors such as cameras.

[0036] The recorded data is then used to infer the robot's movements. However, teleoperation data collection is limited by cost. If a robot is required for each data collection operation, the data cost is extremely high. In addition, damage to the robot and the maintenance costs of its parts are high, making large-scale data collection impossible.

[0037] Furthermore, the joint positions, torques, and velocities recorded by teleoperation are limited by the robot itself, making the recorded data difficult to apply to robots of different structures. Due to the varying structures of the manipulator arms, the data from each joint enables the end-of-arm to reach different positions. Consequently, the data obtained by this type of data collection scheme has poor generalizability and is difficult to apply to robots of different configurations. For example, the data collected from a robot with a seven-axis manipulator arm is difficult to use for motion reasoning on a robot with a six-axis manipulator arm, reducing data utilization.

[0038] Therefore, to address the technical problem of low data utilization in the field of robotics, this application provides a control method, a model training method, a robotic system, and electronic equipment. Combined with a collection device and an end effector with a consistent tool center point definition, these devices can collect data independently of the robot itself, thus addressing the technical problem of low data utilization. The following first describes the application scenarios of the control method provided by this application.

[0039] Figure 1 Schematic diagram of the application scenario of the control method provided in the embodiment of the present application. The control method is applied to a robot system, such as Figure 1The robot system 10 is communicatively connected to the electronic device 20 , and the electronic device 20 is communicatively connected to the acquisition device 30 .

[0040] Among them, the communication connection method may include a wireless communication connection method. The wireless communication connection method may include one or more wireless communication connection methods such as Wireless Fidelity (Wi-Fi), Bluetooth (BT), mobile communication network, Frequency Modulation (FM), Near Field Communication (NFC), infrared technology (IR). The communication connection method from the mobile device 10 to the second terminal device 30 may also include a wired communication connection method. The wired communication connection method may include one or more wired communication connection methods such as Universal Serial Bus (USB) and Controller Area Network (CAN).

[0041] The robotic system 10 includes at least one motion actuator 110 and an end effector 120. Each motion actuator 110 is assigned one end effector 120. For example, if the robotic system 10 includes multiple motion actuators 110, each motion actuator 110 may be assigned one end effector 120. This application does not limit the number of motion actuators 110 and end effectors 120.

[0042] The motion actuator 110 may be a robot body or a motion mechanism on the robot body. For example, when the motion actuator 110 is a robot body, the motion actuator 110 may be a flexible continuum robot. When the motion actuator 110 is a motion mechanism of the robot body, the motion actuator 110 may be a serial robotic arm, a parallel mechanism, etc. The robot body may be a lawn mower robot, a sweeping robot, a disinfecting robot, a leaf collecting robot, a snow-clearing robot, or a multifunctional robot integrating multiple functions mentioned above. The motion actuator 110 may be a lawn mower robot, a sweeping robot, a disinfecting robot, a leaf collecting robot, a snow-clearing robot, or a serial robotic arm or a parallel mechanism of the multifunctional robot.

[0043] The motion actuator 110 may include a joint torque sensor 1101 for collecting data on the joint torque of the motion actuator, which may be recorded as joint torque sensor data.

[0044] The end effector 120 may include a vision sensor 1201 and a force sensor 1202. The vision sensor 1201 is used to collect RGB images and / or depth images and / or point cloud data. The force sensor 1202 is used to collect six-dimensional force / torque information, etc.

[0045] The electronic device 20 can be a mobile phone, a tablet computer, a smart wearable device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, a netbook, a server (including a cloud server), etc. The embodiment of the present application does not impose any restrictions on the specific type of the electronic device 20.

[0046] The parameters defined by the acquisition device 30 and the end effector 110 are consistent. For example, the acquisition device 30 and the end effector 110 have tool center point alignment, sensor observation alignment, and dynamic response matching.

[0047] The above indication Figure 1 The examples are merely examples of application scenarios and do not limit the application scenarios. The robot system 10 may include more or fewer components than shown in the figure, or may combine certain components or different devices. For example, the robot system 10 may also include input and output devices, network access devices, display devices, etc.

[0048] Figure 2 is a flow chart of the control method provided in an embodiment of the present application, which is applied to a robot system including a motion actuator and an end effector (e.g. Figure 1 According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0049] Step S201: Obtain the terminal status information and environment perception information at the current moment.

[0050] In some embodiments of the present application, the current moment may be the moment when end-end state information and environmental perception information are collected. End-end state information includes state information corresponding to the end effector, for example, the end effector's position and posture in the task space. The task space may be a task-oriented operational framework that can be used to describe the position of the operational object, such as the position of the center of a screw or the position of a weld. It can also be used to describe the position and posture of the end effector, for example, the end effector's current position and posture.

[0051] The environmental perception information includes the environmental information of the task space. For example, the environmental information may include at least one of the following: one or more of RGB images, depth images, and point cloud data collected by a visual sensor deployed on the end effector; six-dimensional force / torque information collected by a force sensor installed on the end effector; and joint torque sensor data collected by a joint torque sensor installed on the motion actuator.

[0052] Step S202 : Input the terminal state information and the environment perception information into the motion prediction model to generate a motion increment prediction of the end effector.

[0053] In some embodiments of the present application, the action prediction model can be a model trained with data collected by the acquisition device as training data. Figure 5 The embodiment shown in FIG. The tool center points of the acquisition device and the end effector are defined identically. For example, the position difference between the tool center points of the acquisition device and the end effector is calculated. When the position deviation between the tool center points of the acquisition device and the end effector is less than or equal to a first preset threshold (e.g., 1 mm) and the posture deviation is less than or equal to a preset angle (e.g., 1°), the tool center points of the acquisition device and the end effector are aligned.

[0054] The sensor installation parameters of the acquisition device are aligned with the sensor observation reference of the end effector. When the field of view angle error between the acquisition device's visual sensor and the end effector's visual sensor is less than or equal to a second preset threshold (e.g., 5%), the installation pose error between the acquisition device's visual sensor and the end effector's visual sensor relative to the tool center point is less than or equal to a third preset threshold (e.g., 2 mm), and the installation pose error between the acquisition device's visual sensor and the end effector's visual sensor relative to the tool center point is less than or equal to a fourth preset threshold (e.g., 0.5°), the sensor observations of the acquisition device and the end effector are aligned.

[0055] The acquisition device and the end effector's dynamic response match. This is indicated when the ratio of the acquisition device's maximum velocity to the end effector's rated velocity is within a preset range (e.g., 0.8 m / s to 1.2 m / s).

[0056] Therefore, when the acquisition device and the end effector have a relationship such as the consistent definition of the tool center point, the motion prediction model trained using the training data collected by the acquisition device can be used to predict the motion increment of the end effector, thereby reducing the cost of data acquisition and improving the control efficiency of the robot system.

[0057] In some embodiments of the present application, a robotic system pre-deploys a motion prediction model. When acquiring end-effector state information and environmental perception information, if the robotic system is performing a static task (e.g., a fixed-position operation such as spot welding), the motion prediction model can be directly used to predict the end-effector state information and environmental perception information, thereby obtaining an incremental motion prediction for the end-effector. The incremental motion prediction represents the incremental transformation that the end-effector currently needs to move.

[0058] If the robotic system is performing a dynamic task, such as grasping a moving object or assembling an object, which requires object state recognition, for example, screwing a screw requires sensing the location of the screw hole. Before invoking the motion prediction model, the robotic system's pre-deployed object detection algorithm can be invoked. This object detection algorithm can employ a single algorithm from the R-CNN family (such as the Faster R-CNN), the YOLO family, or the Single Shot Multibox Detector (SSD). Alternatively, a combination of these algorithms can be implemented through feature fusion, multi-scale optimization, or structural improvements.

[0059] Using an object detection algorithm, task observation information is extracted from environmental perception information. This task observation information can include the pose of task-related objects and their predicted motion information. In one example, if the dynamic task is screwing, the pose of the task-related object can be the 3D pose of the screw hole. If the dynamic task is a dynamic grasping task, the predicted motion information can be the predicted trajectory of the moving object.

[0060] After receiving the task observation information from the target detection algorithm, the robot system calls the motion prediction model. The end-effector state information, environmental perception information, and task observation information are simultaneously input into the motion prediction model to generate an incremental prediction of the end-effector's motion.

[0061] Step S203 : obtaining the joint space trajectory of the motion actuator based on the current position and motion increment prediction of the end effector in the task space.

[0062] In some embodiments of the present application, for the same set of motion increment predictions, different configurations of actuators (e.g., robotic arms, robot bodies) may correspond to different joint space trajectories in the task space. For example, given a set of motion increment predictions [Δx, 0, 0, 0, 0, 0], a six-axis robotic arm may require three joint movements, while a delta robot may require parallel rod retraction. Therefore, the motion increment predictions must be combined with the end effector's current pose in the task space to calculate the required joint space trajectory for the actuator.

[0063] In some embodiments of the present application, the action incremental prediction represents the incremental transformation that the end effector currently needs to move. The target pose can be calculated based on the current pose of the end effector in the task space and the action incremental prediction. The target pose can represent the position that the end effector currently needs to reach and the pose when it reaches that position.

[0064] The target pose is calculated using the formula: P_target = P_current ⊕ ΔP, where P_target represents the target pose, P_current represents the current pose of the end effector in the task space, ⊕ represents the pose synthesis operator in the task space, and ΔP represents the action increment prediction.

[0065] If the end effector requires the assistance of a motion actuator for movement, then once the target pose of the end effector is determined, a trajectory to reach the target pose must be calculated. In one embodiment, the robotic system includes an inverse kinematics solver module. By invoking the inverse kinematics solver module to solve the target pose, a set of joint angles for the motion actuator is obtained.

[0066] When the motion actuator is a serial manipulator, the joint angle set may include a joint angle vector. When the motion actuator is a parallel mechanism, the joint angle set may include active joint variables. When the motion actuator is a flexible continuum robot, the joint angle set may include a joint parameter set, which may be a drive length vector.

[0067] Based on the joint angle set, the trajectory planner is called. The trajectory planner interpolates intermediate points between adjacent joint angles in the joint angle set to generate a pathpoint sequence {q0,q1,...,qn}. Based on the preset joint velocity and acceleration limits, the timestamps [t0,tf] of each pathpoint in the pathpoint sequence {q0,q1,...,qn} are calculated.

[0068] In one example, defining joint velocity limits and acceleration limits . Calculate the joint space distance between adjacent pairs of path points in the path point sequence {q0,q1,...,qn}, which is expressed as: , i belongs to [0, n]. Calculate the joint velocity limit First exercise time , which can be expressed as: Calculate the acceleration limit Second exercise time , which can be expressed as .

[0069] Compare the first movement time and the second movement time corresponding to each joint space distance to obtain a comparison result. Based on the comparison result, the maximum of the first movement time and the second movement time corresponding to each joint space distance is used as the target time corresponding to the corresponding joint space distance for each adjacent path point. For example, for any joint space distance, if the first movement time corresponding to the joint space distance is greater than the second movement time, the first movement time is used as the target time corresponding to the joint space distance.

[0070] Starting from the initial time t0, the target time of each segment is accumulated in sequence to obtain the timestamp of each path point [t0, tf]. For example, the timestamp corresponding to the path point q1 is t1=t0+ t0, the timestamp corresponding to the path point q2 is t2=t1+ t1, where t0 represents the target time corresponding to the joint space distance between waypoint q0 and waypoint q1; t1 represents the target time corresponding to the joint space distance between waypoint q1 and waypoint q2.

[0071] Based on the obtained timestamps of each pathpoint, a time-parameterized joint trajectory function q(t) is generated, where t∈[t0,tf]. As time changes, the joint trajectory function can be used to generate joint spatial trajectories that satisfy the motion constraints. The time parameterization represents the process of converting the spatial path into a time trajectory, which determines when the corresponding joint of the motion actuator (such as a robotic arm) reaches each pathpoint.

[0072] Step S204: controlling the motion execution mechanism to execute the action corresponding to the joint space trajectory.

[0073] In some embodiments of the present application, the robotic system sends a torque instruction to the motion actuator, and the motion actuator responds to the torque instruction, overcomes the friction, inertia, gravity and possible external loads of the joint, and moves along the joint space trajectory to complete the corresponding action.

[0074] During the movement process, in order to achieve accurate trajectory tracking, a PID controller (proportional-integral-differential controller) or a controller derived from the transformation of the PID controller (such as PI, PD, feedforward control, calculated torque control, etc.) can be set in the robot system.

[0075] For example, a PID controller compares the position error between the current target joint angle and the actual joint angle, and / or the velocity error between the target joint velocity and the actual joint velocity, to obtain error parameters. Based on these error parameters and preset control parameters (such as proportional gain, integral gain, and differential gain), the controller calculates the required joint torque or force.

[0076] The PID controller outputs a torque command based on the joint torque or force, and the motion actuator moves in response to the torque command. By setting the PID controller, the accuracy of movement along the joint space trajectory can be ensured.

[0077] Through the above-described embodiments, the current terminal state information and environmental perception information are obtained, providing basic data for the subsequent planning of the joint space trajectory of the motion actuator. The terminal state information and environmental perception information are input into the motion prediction model to generate an incremental motion prediction for the end effector. Training data is collected through an acquisition device, and the motion prediction model trained based on the spatially aligned training data outputs the incremental motion prediction. This avoids directly using the robot body to collect data for the training model, reducing the cost of data collection and the maintenance cost of the robot body. In addition, not relying on the robot to collect training data to reproduce the robot's movements can improve the utilization rate of the motion prediction model.

[0078] Based on the motion increment prediction and combined with the current position of the end effector in the task space, the joint space trajectory of the motion actuator is obtained, and the motion actuator is controlled to perform the action corresponding to the joint space trajectory, thereby improving the control efficiency and accuracy of the robot system.

[0079] In addition, the parameters of the acquisition device are consistent with those of the end effector (such as the tool center, etc.). Since the end effector is detachable, the motion prediction model can be applied to any robot body that can be mounted on the end effector. It is not restricted by the structure of the robot body and has a high data utilization rate.

[0080] Figure 3 Flowchart of the control method based on the first actuator and the second actuator provided in the embodiment of the present application. Figure 3 As shown, in the case where the end effector includes a first effector and a second effector, the control of the first robotic arm and the second robotic arm can be achieved through the following steps.

[0081] Step S301 : determining a posture transformation value between a first actuator and a second actuator based on terminal state information.

[0082] In some embodiments of the present application, the end effector that can be deployed by the robotic system may include a first effector and a second effector, and all parameters of the first effector and the second effector can be defined consistently, for example, the tool center point can be defined consistently, and the sensor installation parameters can also be defined consistently.

[0083] When all parameter definitions of the first actuator and the second actuator are consistent, a posture transformation value between the first actuator and the second actuator is calculated, and the posture change value represents the relative posture of the first actuator and the second actuator.

[0084] Step S302 : Input the terminal state information, the posture transformation value, and the environment perception information into the motion prediction model to generate a motion increment prediction of the first actuator and a motion increment prediction of the second actuator.

[0085] In some embodiments of the present application, when all parameter definitions of the first actuator and the second actuator are consistent and the task performed by the robot system is a static task, the robot system can call the motion prediction model to process the terminal state information, posture transformation value and environmental perception information to generate a motion increment prediction of the first actuator and a motion increment prediction of the second actuator.

[0086] If all parameter definitions for the first and second actuators are consistent and the robot system is performing a dynamic task, the object detection model extracts task observation information from the environmental perception information. The end-point state information, pose transformation values, environmental perception information, and task observation information are input into the motion prediction model to generate motion increment predictions for the first and second actuators.

[0087] In some embodiments of the present application, when all parameter definitions of a first actuator and a second actuator are inconsistent, the motion prediction model deployed in the robot system may include a first prediction model corresponding to the first actuator and a second prediction model corresponding to the second actuator. The first acquisition device that collects training data for the first prediction model is consistent with the definition of the tool center point of the first actuator, and the sensor installation parameters of the first acquisition device are aligned with the sensor observation datum of the first actuator. The second acquisition device that collects training data for the second prediction model is consistent with the definition of the tool center point of the second actuator, and the sensor installation parameters of the second acquisition device are aligned with the sensor observation datum of the second actuator.

[0088] If all parameter definitions of the first actuator and the second actuator are inconsistent, and the task performed by the robotic system is a static task, the robotic system can generate a motion increment prediction for the first actuator by invoking a first prediction model to process the terminal state information corresponding to the first actuator and the environmental perception information corresponding to the first actuator. Assuming that the first actuator is installed in the first robotic arm of the robotic system, the environmental perception information corresponding to the first actuator may include one or more of the following: RGB images and / or depth images and / or point cloud data collected by the visual sensor of the first actuator, six-dimensional force / torque information collected by the force sensor of the first actuator, and joint torque sensor data collected by the joint torque sensor of the first robotic arm.

[0089] The robotic system can process the terminal state information corresponding to the second actuator and the environmental perception information corresponding to the second actuator by calling the second prediction model to generate a motion increment prediction for the second actuator. Assuming that the second actuator is installed in the second robotic arm of the robotic system, the environmental perception information corresponding to the second actuator may include one or more of the following: RGB images and / or depth images and / or point cloud data collected by the second actuator's visual sensor, six-dimensional force / torque information collected by the second actuator's force sensor, and joint torque sensor data collected by the second robotic arm's joint torque sensor.

[0090] If all parameter definitions for the first and second actuators are inconsistent and the robot system is performing a dynamic task, the object detection model extracts task observation information from the environmental perception information corresponding to the first actuator, which is recorded as the first observation information. The terminal state information corresponding to the first actuator, the environmental perception information corresponding to the first actuator, and the first observation information are input into the first prediction model to generate a motion increment prediction for the first actuator.

[0091] The object detection model extracts task observation information from the environmental perception information corresponding to the second actuator, which is recorded as the second observation information. The terminal state information corresponding to the second actuator, the environmental perception information corresponding to the second actuator, and the second observation information are input into the second prediction model to generate the action increment prediction of the second actuator.

[0092] Step S303 , based on the current position of the first actuator in the task space and the motion increment prediction of the first actuator, obtain the joint space trajectory of the first manipulator, and control the first manipulator to perform the motion of the joint space trajectory of the first manipulator.

[0093] In some embodiments of the present application, a first actuator may be deployed on a first robotic arm. A target pose of the first actuator may be calculated based on the first actuator's current pose in task space and the predicted motion increment of the first actuator. The target pose of the first actuator is input into an inverse kinematics solver, which outputs a set of joint angles for the first robotic arm. Based on the set of joint angles for the first robotic arm, a trajectory planner generates a joint space trajectory that satisfies motion constraints, thereby obtaining the joint space trajectory of the first robotic arm.

[0094] After the joint space trajectory of the first robotic arm is obtained, the first robotic arm may be controlled to perform actions according to the joint space trajectory of the first robotic arm.

[0095] The calculation process of the joint space trajectory of the first manipulator can refer to the calculation process of the joint space trajectory of the motion actuator in step S203. The control process of the movement of the first manipulator can refer to the control process of the motion actuator to perform the action corresponding to the joint space trajectory in step S204, which will not be repeated here.

[0096] Step S304 , based on the current position of the second actuator in the task space and the motion increment prediction of the second actuator, obtain the joint space trajectory of the second manipulator, and control the second manipulator to perform the motion of the joint space trajectory of the second manipulator.

[0097] In some embodiments of the present application, a second actuator can be deployed on a second robotic arm. Based on the second actuator's current pose in task space and the predicted motion increments of the second actuator, a target pose of the second actuator can be calculated. The target pose of the second actuator is input into an inverse kinematics solver, which outputs a set of joint angles for the second robotic arm. Based on this set of joint angles for the second robotic arm, a trajectory planner generates a joint space trajectory that satisfies motion constraints, resulting in the joint space trajectory of the second robotic arm.

[0098] After obtaining the joint space trajectory of the second robotic arm, the second robotic arm can be controlled to perform actions according to the joint space trajectory of the second robotic arm.

[0099] The calculation process of the joint space trajectory of the second manipulator can refer to the calculation process of the joint space trajectory of the motion actuator in step S203. The control process of the movement of the second manipulator can refer to the control process of the motion actuator to perform the action corresponding to the joint space trajectory in step S204, which will not be repeated here.

[0100] Through the above embodiments, multiple robotic arms can be controlled to move simultaneously and accurately, thereby improving the control efficiency of the robotic system to a certain extent.

[0101] The following combination Figure 4Describe the process of the joint space trajectory of the first manipulator and the joint space trajectory of the second manipulator. Figure 4 As shown, the input data of the action prediction model (model) includes end state information (qpos), pose transformation value (diff), and environmental perception data obs (e.g., RGB images, depth images (depth), point cloud data (pointcloud)). The action prediction model generates incremental action predictions (action) for each moment from time 1 to time c. The current pose (end pose) of the first and second manipulators at each moment is obtained. The action is superimposed on the end pose to obtain the target pose (action pose) corresponding to each moment, forming an end pose sequence. The end pose sequence is input into the inverse kinematics solver module, which outputs the joint angle sets corresponding to the first and second manipulators. Based on the joint angle sets, the trajectory planner generates the joint space trajectory of the first and second manipulators.

[0102] Figure 5 This is a flow chart of the model training method provided in the embodiment of the present application. Figure 5 The relevant steps of the model training method shown can be performed on an electronic device (such as Figure 1 The electronic device 20 shown is executed as follows: Figure 2 as well as Figure 3 The motion prediction model used can be based on Figure 5 The steps of the embodiment shown are obtained.

[0103] Step S501: Acquire first data collected by a collection device.

[0104] In some embodiments of the present application, before using the acquisition device to collect the first data, relevant parameters of the acquisition device can be defined so that the tool center point definitions of the acquisition device and the end effector are consistent, the sensor installation parameters of the acquisition device are aligned with the sensor observation reference of the end effector, and the dynamic responses of the acquisition device and the end effector are matched.

[0105] The position difference between the tool center point of the acquisition device and the tool center point of the end effector is calculated. When the position deviation between the tool center points of the acquisition device and the end effector is less than or equal to a first preset threshold (e.g., 1mm) and the posture deviation is less than or equal to a preset angle (e.g., 1°), the tool center points of the acquisition device and the end effector are consistent. If the position deviation between the tool center points of the acquisition device and the end effector is greater than the first preset threshold (e.g., 1mm) or the posture deviation is greater than a preset angle (e.g., 1°), the tool center point definitions of the acquisition device and the end effector are inconsistent. The tool center point of the acquisition device can be adjusted until the tool center point of the acquisition device and the tool center point of the end effector are consistent.

[0106] When the field of view angle error between the acquisition device's visual sensor and the end-effector's visual sensor is less than or equal to a second preset threshold (e.g., 5%), the installation pose error between the acquisition device's visual sensor and the end-effector's visual sensor relative to the tool center point is less than or equal to a third preset threshold (e.g., 2 mm), and the installation pose error between the acquisition device's visual sensor and the end-effector's visual sensor relative to the tool center point is less than or equal to a fourth preset threshold (e.g., 0.5°), it indicates that the acquisition device and the end-effector's sensor observations are aligned. When the field of view angle error between the acquisition device's visual sensor and the end-effector's visual sensor is greater than the second preset threshold (e.g., 5%), or the installation pose error between the acquisition device's visual sensor and the end-effector's visual sensor relative to the tool center point is greater than the third preset threshold (e.g., 2 mm), or the installation pose error between the acquisition device's visual sensor and the end-effector's visual sensor relative to the tool center point is greater than the fourth preset threshold (e.g., 0.5°), it indicates that the acquisition device and the end-effector's sensor observations are not aligned, and the acquisition device's visual sensor parameters, such as the field of view angle, can be adjusted. The installation pose of the acquisition device's visual sensor, such as the installation position and angle, can also be adjusted.

[0107] The acquisition device and the end effector's dynamic response match. When the ratio of the acquisition device's maximum velocity to the end effector's rated velocity is within a preset range (e.g., 0.8 m / s to 1.2 m / s), the dynamic response of the acquisition device and the end effector match. If this ratio is outside the preset range (e.g., 0.8 m / s to 1.2 m / s), the dynamic response of the acquisition device and the end effector do not match. The acquisition device's maximum velocity can be adjusted to ensure a dynamic match between the two.

[0108] The above are only examples, and other parameters of the acquisition device may also be defined, for example, structural parameters of the acquisition device, such as geometric dimensions.

[0109] The first data collected by the acquisition device may include, but is not limited to, terminal state information of the acquisition device and environmental perception information. In one example, the terminal state information may include the position of the acquisition device in the task space, and the environmental perception information may include at least one of the following: one or more of RGB images, depth images, and point cloud data collected by a visual sensor deployed on the acquisition device; and six-dimensional force / torque information collected by a force sensor installed on the acquisition device.

[0110] In addition, the acquisition device can also embed a flexible tactile sensor (such as based on light guide or capacitance principles) on the inside of the gripper, and the first data can also include time series data of pressure distribution during the grasping process of the acquisition device. The pressure distribution time series data can enhance the model's perception of object slippage and deformation during subsequent model training.

[0111] In other embodiments of the present application, the acquisition device can set up multiple acquisition scenarios or simulate multiple scenarios in which the robot system may be used during the process of collecting the first data. For example, the lighting in the scene can be simulated, and the ambient lighting can be controlled (such as strobe light, color temperature adjustment) during the process of the data acquisition device collecting the first data to force the subsequent model training process to learn the lighting invariant features. It is also possible to set up multiple objects of different materials to simulate the end effector grasping objects of different materials. For example, a standard test suite containing objects with different friction coefficients such as metal / plastic / fabric is established to ensure the generalization of the grasping strategy through the standard test suite.

[0112] The above are only examples of simulation scenarios. It is also possible to simulate electromagnetic disturbances, temperature / humidity changes in the environment where the end effector is located, and to simulate grasping irregular objects.

[0113] Step S502: spatially align the first data to obtain second data.

[0114] In some embodiments of the present application, the first data includes terminal state information and environmental perception information, and the first data is spatially aligned, which may be the terminal state information and the environmental perception information, to obtain the second data.

[0115] In order to perform spatial alignment, a dynamic data synchronization mechanism may be set, which includes a hardware-level time synchronization mechanism and a motion compensation algorithm compensation mechanism.

[0116] In one example, PTP (Precision Time Protocol) can be used in the hardware-level time synchronization mechanism to synchronize the clocks of multiple sensors, and data synchronization can be performed on the RGB images and / or depth images and / or point cloud data collected by the visual sensor, as well as the six-dimensional force / torque information collected by the force sensor based on the timestamp, to ensure that the timestamp alignment accuracy of the data collected by devices such as the IMU, visual sensors, force sensors, and joint encoders reaches the microsecond level.

[0117] In the motion compensation algorithm, the Kalman filter algorithm can be used to correct the distortion of the point cloud data to address the jitter problem of handheld acquisition devices, thereby obtaining distortion-corrected point cloud data. In addition, the Kalman filter algorithm can be combined with IMU angular velocity data to correct the motion distortion of the point cloud data, thereby obtaining corrected point cloud data.

[0118] Step S503: Using the second data to train a preset model to obtain a motion prediction model applied to the robot system.

[0119] In some embodiments of the present application, the second data includes spatially aligned first data, for example, spatially aligned environment perception data.

[0120] To simulate the scenario where the end effector performs a static task, the preset model can be trained using spatially aligned environmental perception data and end state information to obtain a motion prediction model applied to the robotic system.

[0121] To simulate scenarios where the end effector is performing dynamic tasks, the electronic device can invoke a pre-deployed object detection algorithm to extract task observation information from spatially aligned environmental perception information. This object detection algorithm can employ a single algorithm from the Region-based Convolutional Neural Networks (R-CNN) family (such as Faster Region-based Convolutional Neural Networks (FasterR-CNN)), the YOLO family, or the Single Shot Multibox Detector (SSD), or multiple combinations can be implemented through feature fusion, multi-scale optimization, or structural improvements. This task observation information includes the pose of task-related objects and their predicted motion state.

[0122] The preset model is trained using spatially aligned environmental perception data, spatially aligned terminal state information, and task observation information to obtain a motion prediction model for the robotic system.

[0123] In other embodiments of the present application, the acquisition device may include a first device and a second device. When the first device and the second device have the same device parameters, the posture transformation value between the first device and the second device can be determined based on the posture of the first device and the posture of the second device in the second data. In the case of simulating a scenario in which the end effector performs a static task, the second data and the posture transformation value can be used to train a preset model to obtain a motion prediction model. In the case of simulating a scenario in which the end effector performs a dynamic task, the second data, task observation information and the posture transformation value can be used to train a preset model to obtain a motion prediction model.

[0124] If the first device and the second device have different device parameters, the motion prediction model may include a first prediction model corresponding to the first device and a second prediction model corresponding to the second device. In the simulation of a scenario where an end effector performs a static task, the first prediction model can be obtained by training a preset model using the second data corresponding to the first device. The second prediction model can be obtained by training the preset model using the second data corresponding to the second device.

[0125] In the case of simulating a scenario where an end effector performs a dynamic task, the first prediction model can be obtained by training a preset model using the second data corresponding to the first device and the task observation information corresponding to the first device. The second prediction model can be obtained by training a preset model using the second data corresponding to the second device and the task observation information corresponding to the second device.

[0126] In some embodiments of the present application, the electronic device may pre-set a preset number of training times and record the real-time number of training times during the training of the preset model. When the number of training times meets a preset condition, a candidate model is obtained. In one example, when the number of training times reaches a preset number of training times, a candidate model is obtained. Alternatively, not limited to the number of training times, when the preset model meets a preset condition, a candidate model is obtained. For example, when the loss function value of the preset model is less than or equal to a preset loss function threshold, it is determined that the preset model meets the preset condition, thereby obtaining a candidate model.

[0127] After obtaining the candidate model, the accuracy of the candidate model can be verified to determine whether it meets the preset requirements. The pose of the acquisition device at time t and the pose of the acquisition device at time t-1 are obtained, and the difference between the pose of the acquisition device at time t and the pose of the acquisition device at time t-1 is calculated to obtain the true motion increment, where t is a positive integer.

[0128] The difference between the actual motion increment and the predicted motion increment output by the candidate model is calculated. If the difference between the actual motion increment and the predicted motion increment is less than or equal to a preset difference, it is determined that the candidate model meets the preset accuracy requirement and the candidate model can be used as the motion prediction model. After obtaining the motion prediction model, the electronic device can send the motion prediction model to the robotic system, so that the robotic system obtains the motion increment prediction of the end effector through the motion prediction model.

[0129] Through the above embodiment, first data collected by the acquisition device is obtained and spatially aligned, thereby improving data accuracy and reducing interference from noisy data. A preset model is trained using the second data to obtain a motion prediction model. This motion prediction model is applied to the robotic system, reducing the cost of directly using the robotic system for training data collection and improving training accuracy to a certain extent.

[0130] Figure 6 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 6 As shown, the electronic device 20 may include a display device 210, a communication module 220, a memory 230, a processor 240, an input / output (I / O) interface 250, and a bus 260. The processor 240 is coupled to the display device 210, the communication module 220, the memory 230, and the I / O interface 250 via the bus 260.

[0131] The display device 210 can be a touch screen, which is an inductive touch-sensitive liquid crystal display device. Alternatively, the display device 210 can also be a non-touch screen. The display device 210 is used to display data collected by the collection device or the operating status of the robot system.

[0132] The communication module 220 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as Universal Serial Bus (USB) and Controller Area Network (CAN). The wireless communication module may provide one or more wireless communication solutions such as Wireless Fidelity (Wi-Fi), Bluetooth (BT), mobile communication networks, Frequency Modulation (FM), Near Field Communication (NFC), and Infrared (IR). Memory 230 may include one or more random access memories (RAMs) and one or more non-volatile memories (NVMs). The RAM can be directly read and written by the processor 240 and can be used to store executable programs (e.g., machine instructions) for the operating system or other running programs, as well as user and application data.

[0133] Random access memory can include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0134] The non-volatile memory can also store executable programs and user and application data, etc., and can be pre-loaded into the random access memory for direct reading and writing by the processor 240. The non-volatile memory can include disk storage devices and flash memory.

[0135] The memory 230 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 240. The one or more computer programs include multiple instructions. When the multiple instructions are executed by the processor 240, the model training method executed on the electronic device 20 can be implemented.

[0136] In other embodiments, the electronic device 20 further includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 20 .

[0137] The processor 240 may include one or more processing units, for example, the processor 240 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0138] The processor 240 provides computing and control capabilities. For example, the processor 240 is used to execute the computer program stored in the memory 230 to implement the above-mentioned model training method.

[0139] The I / O interface 250 is used to provide a channel for user input or output. For example, the I / O interface 250 can be used to connect various input and output devices, such as a mouse, keyboard, touch device, display screen, etc., so that the user can enter information or visualize information.

[0140] The bus 260 is at least used to provide a channel for mutual communication among the communication module 220 , the memory 230 , the processor 240 , and the I / O interface 250 in the electronic device 20 .

[0141] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 20. In other embodiments of the present application, the electronic device 20 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0142] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the methods in the above-mentioned embodiments of the present application.

[0143] The computer-readable storage medium may be an internal memory of the electronic device of the above-mentioned embodiment, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk equipped on the electronic device, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, etc.

[0144] In some embodiments, the computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, applications required for at least one function, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0145] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0146] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0147] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal devices and methods can be implemented in other ways. For example, the device / terminal device embodiments described above are merely illustrative. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0148] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0149] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A control method, applied to a robot system including a motion actuator and an end effector, characterized in that: The method comprises the following steps: Acquiring terminal state information and environmental perception information at the current moment; the terminal state information includes the position and posture of the end effector in the task space; Inputting the terminal state information and the environmental perception information into a motion prediction model to generate a motion increment prediction of the end effector; wherein the motion prediction model is trained based on spatially aligned training data, and the spatially aligned training data is collected by an acquisition device; Obtaining a joint space trajectory of the motion actuator based on the current position of the end effector in the task space and the motion increment prediction; Control the motion execution mechanism to execute the action corresponding to the joint space trajectory.

2. The control method according to claim 1, characterized in that: The environmental perception information includes at least one of the following: RGB images and / or depth images and / or point cloud data collected by a visual sensor deployed on the end effector; Six-dimensional force / torque information collected by a force sensor installed on the end effector; Joint torque sensor data provided on the motion actuator.

3. The control method according to claim 1, wherein: The method further comprises: Determining task observation information based on the environmental perception information; wherein: the task observation information includes: the pose of the task-related object extracted from the environmental perception information by the target detection algorithm, and the motion state prediction information of the task-related object; The step of inputting the terminal state information and the environmental perception information into a motion prediction model to generate a motion increment prediction of the end effector includes: The terminal state information, the environment perception information, and the task observation information are synchronously input into the action prediction model to generate the action increment prediction.

4. The control method according to claim 1, wherein: The obtaining of the joint space trajectory of the motion actuator based on the current posture of the end effector in the task space and the motion increment prediction includes: Calculate the target pose P_target based on the action increment prediction, P_target = P_current ⊕ ΔP, where the target pose P_current is the current pose of the end effector in the task space, ΔP is the action increment prediction, and ⊕ represents the pose synthesis operator in the task space; Input the target posture into the inverse kinematics solution module, and output the joint angle set of the motion actuator; Based on the joint angle set, a joint space trajectory that satisfies the motion constraints is generated by a trajectory planner.

5. The method according to claim 4, characterized in that The trajectory planner performs the following operations: Interpolate intermediate points between adjacent joint angles to generate a sequence of waypoints; Calculating a timestamp for each waypoint in the waypoint sequence based on preset joint velocity limits and acceleration limits; Based on the timestamps of the various path points, a time-parameterized joint trajectory function is generated.

6. The method according to claim 4, characterized in that The method further comprises: When the motion actuator is a serial robot arm, the joint angle set is a joint angle vector; When the motion actuator is a parallel mechanism, the joint angle set is an active joint variable; When the motion actuator is a flexible continuum robot, the joint angle set is a driving length vector.

7. The control method according to claim 1, characterized in that: The method further comprises: The acquisition device is defined in accordance with the tool center point of the end effector; The sensor installation parameters of the acquisition device are aligned with the sensor observation reference of the end effector.

8. The control method according to claim 1, characterized in that: The end effector includes a first effector and a second effector, and inputting the end state information and the environment perception information into a motion prediction model to generate a motion increment prediction of the end effector includes: determining a posture transformation value between the first actuator and the second actuator based on the terminal state information; The terminal state information, the posture transformation value and the environmental perception information are input into the motion prediction model to generate a motion increment prediction of the first actuator and a motion increment prediction of the second actuator.

9. The control method according to claim 8, characterized in that: The motion execution mechanism includes a first robotic arm and a second robotic arm, and the method further includes: Obtaining a joint space trajectory of the first robotic arm based on the current position of the first actuator in the task space and a motion increment prediction of the first actuator, and controlling the first robotic arm to perform a motion along the joint space trajectory of the first robotic arm; Based on the current position of the second actuator in the task space and the motion increment prediction of the second actuator, the joint space trajectory of the second robotic arm is obtained, and the second robotic arm is controlled to perform the motion of the joint space trajectory of the second robotic arm.

10. A model training method, characterized in that: The method comprises: Acquire first data collected by a collection device; performing spatial alignment on the first data to obtain second data; The preset model is trained using the second data to obtain a motion prediction model applied to the robot system.

11. The model training method according to claim 10, characterized in that: The method further comprises: Determining task observation information based on the spatially aligned environmental perception information in the second data; wherein the task observation information includes: task-related object poses extracted from the spatially aligned environmental perception information by a target detection algorithm, and motion state prediction information of the task-related objects; The method of using the second data to train a preset model to obtain a motion prediction model applied to the robot system includes: The preset model is trained using the second data and the task observation information to obtain the action prediction model.

12. The model training method according to claim 10, characterized in that: The acquisition device includes a first device and a second device, and the method further includes: determining a posture transformation value between the first device and the second device based on the posture of the first device and the posture of the second device in the second data; The method of using the second data to train a preset model to obtain a motion prediction model applied to the robot system includes: The preset model is trained using the second data and the posture transformation value to obtain the action prediction model.

13. The model training method according to any one of claims 10 to 12, characterized in that: The method of using the second data to train a preset model to obtain a motion prediction model applied to the robot system includes: When the number of training times or the preset model meets the preset conditions, a candidate model is obtained; Determining a real motion increment based on the posture of the acquisition device at time t and the posture of the acquisition device at time t-1, where t is a positive integer; Calculating the difference between the true action increment and the predicted action increment prediction output by the candidate model; When the difference is less than or equal to a preset difference, the candidate model is used as the action prediction model.

14. The model training method according to claim 10, characterized in that: The first data includes: terminal status information and environment perception information of the acquisition device; Wherein, the terminal state information includes the position and posture of the acquisition device in the task space; The environmental perception information includes at least one of the following: RGB images and / or depth images and / or point cloud data collected by the visual sensor deployed on the acquisition device; The six-dimensional force / torque information is collected by the force sensor installed on the collection device.

15. The model training method according to claim 14, characterized in that: The spatially aligning the first data to obtain second data includes: The terminal state information and the environment perception information are spatially aligned to obtain the second data.

16. The model training method according to claim 14, characterized in that: The method further comprises: The point cloud data is subjected to distortion correction using a Kalman filter algorithm to obtain distortion-corrected point cloud data.

17. The model training method according to claim 14, characterized in that: The method further comprises: Data synchronization is performed on the RGB image and / or depth image and / or point cloud data collected by the visual sensor and the six-dimensional force / torque information collected by the force sensor based on the timestamp.

18. A robot system, characterized in that: The robot system includes a processor and a memory, wherein the memory stores a computer program, and the processor implements the control method according to any one of claims 1 to 9 when executing the computer program.

19. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory stores a computer program, and the processor implements the model training method as described in any one of claims 10 to 17 when executing the computer program.

20. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by the processor, it implements the control method as described in any one of claims 1 to 9, or the model training method as described in any one of claims 10 to 17.

Images