Composite robot arm control method, device, equipment and storage medium

Abstract

The present application discloses a method, device, equipment, and storage medium for controlling the manipulator arm of a composite robot, relating to the technical field of composite robots. The method comprises: obtaining motion state information of the manipulator arm of the composite robot, workpiece point cloud data and target point cloud data of a workpiece clamped by the manipulator arm in a first coordinate system, where the first coordinate system is the coordinate system of a depth camera; determining the position error information of the workpiece in a real environment based on the workpiece point cloud data and the target point cloud data; and determining the manipulator arm position information in a second coordinate system based on the motion state information of the manipulator arm, where the second coordinate system is the coordinate system of a mobile chassis; and calling a trained neural network model based on the motion state information of the manipulator arm, the manipulator arm position information, and the workpiece position error information, determining the manipulator arm's position adjustment parameters, and controlling the manipulator arm to execute the position adjustment parameters. This method can improve workpiece assembly efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of composite robots, and in particular to a method, device, equipment and storage medium for controlling a manipulator arm of a composite robot. Background Art

[0002] Hybrid robots integrate a mobile chassis with a robotic arm, achieving a perfect fusion of "mobility" and "manipulation," greatly expanding the robot's application capabilities in complex and dynamic environments. The robotic arm, mounted on a mobile chassis, can perform specific manipulation tasks such as grasping, placing, assembly, and inspection.

[0003] In related technologies, a hybrid robot can grasp a workpiece (for example, a server memory card) and assemble it onto the server. During the grasping and assembly process, specialized fixtures and external visual calibration technology can be used to achieve high-precision positioning in static scenarios.

[0004] However, during the movement of the robotic arm after grasping the workpiece, the workpiece's posture may shift due to the gravity of the workpiece, the contact force and friction between the robotic arm and the workpiece. At this time, it is difficult to compensate and adjust the workpiece's posture shift through special tooling and external visual calibration technology, resulting in low accuracy of workpiece assembly. Summary of the Invention

[0005] The present application provides a method, device, equipment and storage medium for controlling a manipulator arm of a composite robot, so as to at least solve the problem of low accuracy in workpiece assembly in the related art.

[0006] In one aspect, the present application provides a method for controlling a manipulator arm of a composite robot, comprising:

[0007] Acquire motion state information of a manipulator arm of the composite robot, workpiece point cloud data of a workpiece held by the manipulator arm, and target point cloud data in a first coordinate system, wherein the first coordinate system is a coordinate system of a depth camera used to acquire workpiece point cloud data;

[0008] Determine the pose error information of the workpiece in a real environment based on the workpiece point cloud data and the target point cloud data; and determine the pose information of the manipulator in a second coordinate system based on the motion state information of the manipulator; wherein the second coordinate system is the coordinate system of the mobile chassis used to fix the manipulator;

[0009] According to the motion state information of the robot arm, the posture information of the robot arm and the posture error information of the workpiece, the trained neural network model is called to determine the posture adjustment parameters of the robot arm, and the robot arm is controlled to execute the posture adjustment parameters.

[0010] On the other hand, the present application provides a manipulator control device for a composite robot, comprising:

[0011] an acquisition unit, configured to acquire motion state information of a manipulator arm of the composite robot, workpiece point cloud data and target point cloud data of a workpiece clamped by the manipulator arm in a first coordinate system, wherein the first coordinate system is a coordinate system of a depth camera used to acquire workpiece point cloud data;

[0012] a determination unit, configured to determine, based on the workpiece point cloud data and the target point cloud data, position error information of the workpiece in a real environment; and, based on the motion state information of the robotic arm, determine the robotic arm position information in a second coordinate system; wherein the second coordinate system is a coordinate system of a mobile chassis for fixing the robotic arm;

[0013] The control unit is used to call the trained neural network model according to the motion state information of the robot arm, the posture information of the robot arm and the posture error information of the workpiece, determine the posture adjustment parameters of the robot arm, and control the robot arm to execute the posture adjustment parameters.

[0014] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned methods for controlling the manipulator arm of a composite robot when executing the computer program.

[0015] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods for controlling the manipulator arm of a composite robot are implemented.

[0016] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned methods for controlling the manipulator arm of a composite robot when the computer program is executed by a processor.

[0017] The present application provides a method, device, equipment and storage medium for controlling a manipulator arm of a composite robot. The method comprises: obtaining motion state information of a manipulator arm of the composite robot, workpiece point cloud data and target point cloud data of a workpiece clamped by the manipulator arm in a first coordinate system, wherein the first coordinate system is the coordinate system of a depth camera used to obtain the workpiece point cloud data; determining the position error information of the workpiece in a real environment based on the workpiece point cloud data and the target point cloud data; and determining the manipulator arm position information of the manipulator arm in a second coordinate system based on the motion state information of the manipulator arm; wherein the second coordinate system is the coordinate system of a mobile chassis used to fix the manipulator arm; based on the motion state information of the manipulator arm, the manipulator arm position information and the position error information of the workpiece, calling a trained neural network model to determine the manipulator arm's position adjustment parameters, and controlling the manipulator arm to execute the position adjustment parameters. In an embodiment of the present application, since the manipulator arm's position adjustment parameters can be determined by calling a trained neural network model based on the motion state information of the manipulator arm, the manipulator arm position information and the position error information of the workpiece, the assembly process of the workpiece can be completed without multiple adjustments to the manipulator arm, thereby improving the assembly efficiency of the workpiece. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 A schematic diagram of an application scenario of the manipulator arm control method of a composite robot provided in an embodiment of the present application;

[0020] Figure 2 The process of the manipulator arm control method of the composite robot provided in the embodiment of the present application Figure 1 ;

[0021] Figure 3 The process of training the neural network model provided in the embodiment of the present application Figure 1 ;

[0022] Figure 4 A schematic structural diagram of a manipulator arm control device for a composite robot provided in an embodiment of the present application;

[0023] Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0024] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0025] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0026] Hybrid robots integrate a mobile chassis with a robotic arm, achieving a perfect fusion of "mobility" and "manipulation," greatly expanding the robot's application capabilities in complex and dynamic environments. The robotic arm, mounted on a mobile chassis, can perform specific manipulation tasks such as grasping, placing, assembly, and inspection.

[0027] In related technologies, a hybrid robot can grasp a workpiece (for example, a server memory card) and assemble it onto the server. During the grasping and assembly process, specialized fixtures and external visual calibration technology can be used to achieve high-precision positioning in static scenarios.

[0028] However, during the movement of the robotic arm after grasping the workpiece, the workpiece's posture may shift due to the gravity of the workpiece, the contact force and friction between the robotic arm and the workpiece. At this time, it is difficult to compensate and adjust the workpiece's posture shift through special tooling and external visual calibration technology, resulting in low accuracy of workpiece assembly.

[0029] Therefore, how to determine the posture adjustment parameters of the robot arm and compensate for the posture deviation of the workpiece to improve the accuracy of workpiece assembly is a technical problem that needs to be solved urgently.

[0030] Among them, the external visual calibration technology can include three stages: template calibration-coarse positioning adjustment-iterative fine correction. (1) First, the operator manually controls the robot arm to grasp the workpiece and adjust it to the ideal assembly posture. The fixed depth camera is used to collect the complete point cloud of the workpiece, which is stored as a template point cloud after noise reduction filtering. (2) Then, it enters the automatic assembly stage. The robot arm uses the hand-eye camera to grasp the new workpiece, and then moves to the coarse positioning area so that the workpiece is roughly within the visual area of ​​the fixed depth camera. The workpiece is processed by the fixed depth camera to obtain the workpiece posture information with large errors. The end of the robot arm is preliminarily adjusted to roughly align the workpiece and template posture. (3) Then, the fixed depth camera captures the local point cloud of the workpiece, calculates its posture deviation with the template point cloud based on the iterative closest point algorithm (ICP), drives the end of the robot arm to move, minimizes the ICP registration error, and thus achieves end-to-end fine alignment. (4) During this process, the system goes through multiple closed-loop iterations of “point cloud acquisition-registration calculation-pose correction” until the measured pose error converges to the preset threshold, and finally completes the precision assembly without fixtures and model dependence.

[0031] First, the above solution has high requirements for the placement and field of view of the binocular camera. In addition, it is necessary to obtain the transformation matrix from the camera coordinate system to the base coordinate system to achieve posture adjustment guided by posture error across coordinate systems. In order to achieve real-time calibration, it is necessary to install markers on the composite robot body, and the external fixed binocular camera is required to clearly capture the markers on the composite robot body. Secondly, the above solution requires a lot of calibration and hyperparameter adjustment, such as the six-degree-of-freedom error gain matrix, etc., which requires a lot of manpower to adjust and debug. Moreover, after the system structure is changed, it is difficult to ensure that the set hyperparameters still meet the application requirements. Therefore, the deployment and debugging process is complicated.

[0032] In response to the above technical problems, the inventor's technical conception is as follows: abandon the physical fixture and offline calibration process, directly model the mapping relationship between the end-effector posture error and the joint motion through the reinforcement learning strategy network, and use the quaternion dot product reward function to drive the robotic arm to autonomously generate real-time posture compensation actions, thereby realizing dynamic posture self-correction without fixture constraints.

[0033] Accordingly, the specific steps include: first, obtaining the motion state information of the composite robot's manipulator arm, the workpiece point cloud data of the workpiece held by the manipulator arm, and the target point cloud data in a first coordinate system, where the first coordinate system is the coordinate system of the depth camera used to obtain the workpiece point cloud data. Then, based on the workpiece point cloud data and the target point cloud data, determining the workpiece's pose error information in a real environment; and, based on the manipulator arm's motion state information, determining the manipulator arm's pose information in a second coordinate system, where the second coordinate system is the coordinate system of the mobile chassis used to secure the manipulator arm. Finally, based on the manipulator arm's motion state information, the manipulator arm's pose information, and the workpiece's pose error information, calling a trained neural network model, determining the manipulator arm's pose adjustment parameters, and controlling the manipulator arm to execute the pose adjustment parameters.

[0034] In an embodiment of the present application, the posture adjustment parameters of the robot arm can be determined by calling a trained neural network model through the motion state information of the robot arm, the posture information of the robot arm and the posture error information of the workpiece. The assembly process of the workpiece can be completed without multiple adjustments to the robot arm, thereby improving the assembly efficiency of the workpiece.

[0035] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0036] In combination with the specific application environment architecture or specific hardware architecture on which the execution of the manipulator arm control method of the composite robot depends, the specific application environment architecture or specific hardware architecture is described here. Figure 1 Schematic diagram of the application scenario of the manipulator control method of the composite robot provided in the embodiment of the present application. Figure 1 As shown, the compound robot 101 includes a mobile chassis and a multi-degree-of-freedom robotic arm mounted on the chassis, with a gripper and a hand-eye camera at the end of the robotic arm. A depth camera 102 is set at a fixed position on the workbench to collect workpiece point cloud data. The industrial computer 103 is connected to the depth camera 102 by wire and is responsible for point cloud processing and posture solution. Data can be exchanged between the industrial computer 103 and the controller of the compound robot 101 via a wireless network. Among them, the controller of the compound robot 101 is built into the body of the compound robot 101, and is used to determine the posture adjustment parameters of the robotic arm through the robotic arm control method of the compound robot provided in the embodiment of the present application, and control the robotic arm to execute the posture adjustment parameters.

[0037] Figure 2 The process of the manipulator arm control method of the composite robot provided in the embodiment of the present application Figure 1 The execution subject of the control method of the manipulator arm of the composite robot can be a terminal device or a server. Figure 2 As shown, the method includes:

[0038] S201, obtaining motion state information of a manipulator arm of a composite robot, workpiece point cloud data and target point cloud data of a workpiece clamped by the manipulator arm in a first coordinate system, wherein the first coordinate system is a coordinate system of a depth camera used to obtain workpiece point cloud data.

[0039] In the embodiment of the present disclosure, the robotic arm may include multiple joints and a gripper. In this case, the motion state information of the robotic arm includes the angles of each joint and the opening and closing states of the gripper, as well as the angular velocities of each joint and the opening and closing velocities of the gripper.

[0040] In some embodiments, the robot arm's motion state information (joint angles, gripper opening and closing states, joint angular velocities, and gripper opening and closing velocities) is stored in the robot system. Alternatively, the robot arm's motion state information can be obtained through the robot system's application programming interface (API).

[0041] Among them, the workpiece point cloud data of the workpiece clamped by the robot arm in the first coordinate system can be obtained by the depth camera. Figure 1 As shown, the depth camera is located at a fixed position on the workbench and can collect workpiece point cloud data.

[0042] S202. Determine the posture error information of the workpiece in the real environment based on the workpiece point cloud data and the target point cloud data; and determine the posture information of the robotic arm in a second coordinate system based on the motion state information of the robotic arm; wherein the second coordinate system is the coordinate system of the mobile chassis used to fix the robotic arm.

[0043] In the embodiment of the present disclosure, determining the pose error information of the workpiece in the real environment based on the workpiece point cloud data and the target point cloud data may include the following steps (1) to (2):

[0044] (1) According to the workpiece point cloud data and the target point cloud data, the rotation and translation matrices of the workpiece position and the target position are determined. The rotation and translation matrices include rotation transformation parameters and translation vector parameters.

[0045] Optionally, the rotation and translation matrix is ​​calculated using the point cloud registration algorithm:

[0046]

[0047] in, represents the rotation transformation parameters, Represents the translation vector parameter.

[0048] (2) Determine the Euclidean distance between the workpiece position and the target position based on the translation vector parameters; and determine the difference in the three-axis orientation angle between the workpiece position and the target position based on the rotation transformation parameters.

[0049] Optionally, based on the translation vector parameters, the Euclidean distance between the workpiece position and the target position is determined as: .

[0050] Optionally, based on the rotation transformation parameters, the difference in the three-axis orientation angle between the workpiece position and the target position is determined as:

[0051]

[0052] It should be noted that the process of acquiring target point cloud data may include: manually controlling the composite robot to grab the workpiece to be assembled, and adjusting the posture parameters of the robotic arm so that the position of the workpiece meets the assembly process requirements of the workpiece; triggering the depth camera fixed on the workbench to shoot the positioned workpiece, obtaining the point cloud data of the workpiece, and storing the point cloud data as target point cloud data.

[0053] S203. Based on the motion state information of the robot arm, the posture information of the robot arm, and the posture error information of the workpiece, the trained neural network model is called to determine the posture adjustment parameters of the robot arm, and the robot arm is controlled to execute the posture adjustment parameters.

[0054] In some embodiments, the obtained state data can be used as an input vector and input into a neural network model to obtain the mean of the robot's motion posture. Then, motion sampling is performed according to a Gaussian distribution to obtain the robot's motion. Accordingly, this step includes: converting the robot's motion state information, the robot's posture information, and the workpiece's posture error information into one-dimensional vector information, which is then input into the neural network model to output the mean of the robot's posture parameters; and sampling the posture parameters according to a Gaussian distribution based on the mean of the robot's posture parameters to obtain the robot's posture adjustment parameters.

[0055] Among them, the posture adjustment parameters include the angle adjustment parameters of each joint.

[0056] The present invention provides a method for controlling a manipulator arm of a composite robot. The method comprises obtaining motion state information of a manipulator arm of the composite robot, workpiece point cloud data and target point cloud data of a workpiece held by the manipulator arm in a first coordinate system, wherein the first coordinate system is the coordinate system of a depth camera used to obtain the workpiece point cloud data; determining pose error information of the workpiece in a real environment based on the workpiece point cloud data and the target point cloud data; and determining manipulator arm pose information in a second coordinate system based on the motion state information of the manipulator arm; wherein the second coordinate system is the coordinate system of a mobile chassis used to fix the manipulator arm; and calling a trained neural network model based on the motion state information of the manipulator arm, the manipulator arm pose information, and the pose error information of the workpiece, determining the manipulator arm pose adjustment parameters, and controlling the manipulator arm to execute the pose adjustment parameters. In the present invention, since the manipulator arm pose adjustment parameters can be determined by calling a trained neural network model based on the motion state information of the manipulator arm, the manipulator arm pose information, and the pose error information of the workpiece, the workpiece assembly process can be completed without multiple adjustments to the manipulator arm, thereby improving the assembly efficiency of the workpiece.

[0057] The training process of the neural network model in step S203 is described in detail below. In the embodiment of the present application, based on deep reinforcement learning and adaptive feature extraction mechanism, through posture perturbation training in a simulation environment, the policy network can learn a universal alignment pattern from quaternion posture error and three-dimensional space Euclidean distance error, without the need to design a matching algorithm for a specific workpiece, which significantly improves the ability to quickly adapt to unknown workpieces. Figure 3 As shown in Figure 2, the training process of the neural network model includes:

[0058] S301 , obtaining motion state information of a manipulator arm of a compound robot, first workpiece pose information and target pose information of a workpiece clamped by the manipulator arm in a first coordinate system, wherein the first coordinate system is a coordinate system of a depth camera.

[0059] In some embodiments, the robot arm's motion state information (the robot arm's joint angles, the gripper opening and closing states, and the joint angular velocities and gripper opening and closing velocities of the joints), and the first workpiece pose information and target pose information of the workpiece gripped by the robot arm in the first coordinate system are stored in the robot system. Alternatively, the robot arm's motion state information and the first workpiece pose information and target pose information of the workpiece gripped by the robot arm in the first coordinate system can be obtained through an API query of the robot system.

[0060] For example, the first workpiece pose information of the workpiece in the world coordinate system is:

[0061]

[0062] The target pose information of the template position is:

[0063]

[0064] The posture information of the depth camera's coordinate system in the world coordinate system is:

[0065]

[0066] in, Indicates spatial location, represents the attitude quaternion, and the subscripts c, t, and d represent the workpiece, template, and depth camera, respectively.

[0067] It should be noted that if Figure 1 As shown, the template position is: manually controlling the composite robot to grab the workpiece to be assembled, and adjusting the posture parameters of the robot arm so that the position of the workpiece meets the assembly process requirements of the workpiece.

[0068] S302. Determine the first pose error information of the workpiece in the simulation environment based on the first workpiece pose information and the target pose information; and determine the manipulator pose information of the manipulator in a second coordinate system based on the motion state information of the manipulator; wherein the second coordinate system is the coordinate system of the mobile chassis.

[0069] In some embodiments, the first workpiece pose information includes first workpiece position information and first workpiece three-axis orientation angle information, and the target pose information includes target position information and target three-axis orientation angle information; accordingly, determining the first pose error information of the workpiece in a simulation environment based on the first workpiece pose information and the target pose information includes the following steps (1) to (3):

[0070] (1) Determine the first Euclidean distance between the first workpiece position information and the target position information.

[0071] For example, the Euclidean distance between the workpiece and the template in the grasping state in the depth camera coordinate system is:

[0072]

[0073] (2) Determine a first difference between the three-axis orientation angle information of the first workpiece and the three-axis orientation angle information of the target.

[0074] For example, the workpiece posture in the world coordinate system is converted to the depth camera coordinate system as follows:

[0075]

[0076]

[0077] Similarly, the posture of the template position in the depth camera coordinate system is:

[0078]

[0079]

[0080] In this case, the difference in the three-axis orientation angles of the workpiece and template in the grasping state in the depth camera coordinate system under the simulation environment is:

[0081]

[0082] Wherein, arctan2 represents the four-quadrant inverse tangent function.

[0083] (3) The first Euclidean distance and the first difference are determined as the first position error information of the workpiece in the simulation environment.

[0084] S303: Based on the motion state information, the position and posture information of the manipulator, and the first position error information of the manipulator, the primary neural network model is called to determine the first position adjustment parameters of the manipulator, and the manipulator is controlled to execute the first position adjustment parameters.

[0085] In some embodiments, the primary neural network model is a multi-layer perceptron network. Exemplarily, the multi-layer perceptron network includes three hidden layers with dimensions of 256, 128, and 64, respectively.

[0086] Optionally, the motion state information, posture information and first posture error information of the robot arm are converted into a one-dimensional vector and input into the primary neural network model to obtain the mean of the robot's motion posture, and then the action sampling is performed according to the Gaussian distribution to obtain the first posture adjustment parameter of the robot arm.

[0087] For example, the robot arm's motion state information includes: the angles of each joint and the gripper's opening and closing state; the angular velocities of each joint and the gripper's opening and closing speed. The robot arm's position information includes: the posture of the gripper in the base coordinate system of the hybrid robot arm. The first position error information includes: the Euclidean distance between the workpiece and the template in the gripping state in the depth camera coordinate system; and the difference in the three-axis orientation angles between the workpiece and the template in the gripping state in the depth camera coordinate system.

[0088] It should be noted that, in some embodiments, in order to increase the speed of training, the last historical posture adjustment parameters can also be input into the primary neural network model to improve the accuracy of the output first posture adjustment parameters.

[0089] S304, after determining the execution of the first pose adjustment parameter, the second workpiece pose information of the workpiece clamped by the robot arm in the first coordinate system is determined, and the second pose error information of the workpiece in the simulation environment is determined according to the second workpiece pose information and the target pose information.

[0090] In this step, the method for determining the second pose error information of the workpiece in the simulation environment is the same as the method for determining the first pose error information of the workpiece in the simulation environment in step S302, and will not be repeated here.

[0091] S305. If the second posture error information is greater than the preset error threshold, the model parameters of the primary neural network model are updated until the obtained posture error information is less than the preset error threshold, thereby obtaining a trained neural network model.

[0092] In some embodiments, the model parameters of the primary neural network model can be updated according to the reward value of performing the first pose adjustment parameter.

[0093] Optionally, the reward function for the first pose adjustment parameter is as follows:

[0094]

[0095] Among them, R represents the reward value, and Represents the weight coefficient, q sim Represents the dot product of the workpiece and the template's attitude quaternion in the depth camera coordinate system, that is, D represents the Euclidean distance between the workpiece and the template in the depth camera coordinate system, that is, .

[0096] It should be noted that after controlling the robotic arm to execute the posture adjustment parameters in step S203, it can also be determined whether the adjusted posture error information is less than the preset error threshold. If the adjusted posture error information is greater than the preset error threshold, the model parameters of the primary neural network model are updated according to the reward value of executing the posture adjustment parameters.

[0097] In real environment, .

[0098] when season ,but

[0099]

[0100] when and season ,but

[0101]

[0102] when and season ,but

[0103]

[0104] when and season ,but

[0105]

[0106] It should be noted that in a dynamic and complex environment without pre-calibration, the positions of the workpiece, template, composite robot, and depth camera all contain uncertainty. Although the composite robot and depth camera can exchange data via an industrial computer and the on-site wireless network, the position information they describe is based on their own coordinate system. Therefore, to best simulate the complex dynamic environment of this scenario, domain randomization is performed during system initialization. In a real-world environment, the system's posture uncertainty is primarily manifested in the following aspects: the uncertainty of the workpiece's posture in the grasping state, the uncertainty of the template's position in the depth camera's coordinate system, the uncertainty of the composite robot's arm base position, and the uncertainty of the depth camera's coordinate system.

[0107] In some embodiments, domain randomization training can be combined to enable the network to extract effective signals from noisy sparse observations and maintain stable attitude alignment accuracy under complex working conditions.

[0108] For example, b -x b y b z b Represents the base coordinate system of the composite robot arm, O c -x c y c z c The coordinate system representing the workpiece's posture in the gripping state, O t -x t y t z t Represents the template position coordinate system in the depth camera coordinate system, O d -x d y d z d Represents the coordinate system of the depth camera. b -x b y b z b There are three degrees of freedom, namely, plane movement and rotation around the axis; c -x c y c z c There are four degrees of freedom, namely translation along three axes and rotation around the axes; t -x ty t z t There are six degrees of freedom, namely translation along three axes and rotation around three axes; d -x d y d z d There are six degrees of freedom: translation along three axes and rotation around three axes.

[0109] In the simulation environment, domain randomization is performed on the uncertainty of the above postures during the system initialization phase, and a certain position error and three-axis orientation error are imposed respectively. d -x d y d z d For example, the observation position P of the depth camera when the system is initialized d for

[0110]

[0111] in, is a three-dimensional rotation matrix generated by Euler angles, satisfying: , the subscript is the rotation axis,

[0112] ,

[0113] , × represents the Cartesian product, is a hypercube region in three-dimensional space, and P0 represents the initialization center position of the random field. Similarly, the pose uncertainty of other coordinate systems can be set according to the above content, and the missing degrees of freedom can be set to 0.

[0114] Figure 4 This is a schematic diagram of the structure of the manipulator control device of the composite robot provided in the embodiment of the present application. Figure 4 As shown, an embodiment of the present application further provides a manipulator arm control device for a composite robot, the device comprising:

[0115] An acquisition unit 401 is configured to acquire motion state information of a manipulator arm of the composite robot, workpiece point cloud data of a workpiece held by the manipulator arm, and target point cloud data in a first coordinate system, wherein the first coordinate system is a coordinate system of a depth camera used to acquire the workpiece point cloud data;

[0116] a determination unit 402 for determining, based on the workpiece point cloud data and the target point cloud data, position error information of the workpiece in a real environment; and, based on the motion state information of the robotic arm, determining the robotic arm position information in a second coordinate system; wherein the second coordinate system is a coordinate system of a mobile chassis for fixing the robotic arm;

[0117] The control unit 403 is used to call the trained neural network model according to the motion state information of the robot arm, the posture information of the robot arm and the posture error information of the workpiece, determine the posture adjustment parameters of the robot arm, and control the robot arm to execute the posture adjustment parameters.

[0118] In some embodiments, the control unit 403 calls a trained neural network model based on the motion state information of the robotic arm, the posture information of the robotic arm, and the posture error information of the workpiece to determine the posture adjustment parameters of the robotic arm, including: converting the motion state information of the robotic arm, the posture information of the robotic arm, and the posture error information of the workpiece into one-dimensional vector information and inputting the information into the neural network model to output the mean value of the posture parameters of the robotic arm; sampling the posture parameters according to the Gaussian distribution based on the mean value of the posture parameters of the robotic arm to obtain the posture adjustment parameters of the robotic arm.

[0119] In some embodiments, the robotic arm includes multiple joints and a gripper; the motion state information of the robotic arm includes the angle of each joint and the opening and closing state of the gripper, as well as the angular velocity of each joint and the opening and closing velocity of the gripper; the posture adjustment parameters include the angle adjustment parameters of each joint.

[0120] In some embodiments, the determination unit 402 determines the posture error information of the workpiece in a real environment based on the workpiece point cloud data and the target point cloud data, including: determining the rotation and translation matrix of the workpiece position and the target position based on the workpiece point cloud data and the target point cloud data, the rotation and translation matrix including rotation transformation parameters and translation vector parameters; determining the Euclidean distance between the workpiece position and the target position based on the translation vector parameters; and determining the difference in the three-axis orientation angles between the workpiece position and the target position based on the rotation transformation parameters.

[0121] In some embodiments, the device also includes: a storage unit; the storage unit is used to manually control the composite robot to grab the workpiece to be assembled, and adjust the posture parameters of the robotic arm so that the position of the workpiece meets the assembly process requirements of the workpiece; trigger the depth camera fixed on the workbench to shoot the positioned workpiece, obtain point cloud data of the workpiece, and store the point cloud data as target point cloud data.

[0122] In some embodiments, the device further includes: a model training unit; the training process of the neural network model trained by the model training unit includes: obtaining the motion state information of the manipulator arm of the composite robot, the first workpiece posture information and the target posture information of the workpiece clamped by the manipulator arm in the first coordinate system, wherein the first coordinate system is the coordinate system of the depth camera; determining the first posture error information of the workpiece in the simulation environment according to the first workpiece posture information and the target posture information; and determining the manipulator arm posture information in the second coordinate system according to the motion state information of the manipulator arm; wherein the second coordinate system is the coordinate system of the mobile chassis; according to the manipulator arm The motion state information, the robot arm posture information and the first posture error information are used to call the primary neural network model, determine the first posture adjustment parameters of the robot arm, and control the robot arm to execute the first posture adjustment parameters; after determining to execute the first posture adjustment parameters, the second workpiece posture information of the workpiece clamped by the robot arm in the first coordinate system is used, and the second posture error information of the workpiece in the simulation environment is determined according to the second workpiece posture information and the target posture information; if the second posture error information is greater than the preset error threshold, the model parameters of the primary neural network model are updated until the posture error information obtained is less than the preset error threshold, and a trained neural network model is obtained.

[0123] In some embodiments, the first workpiece posture information includes first workpiece position information and first workpiece three-axis orientation angle information, and the target posture information includes target position information and target three-axis orientation angle information; accordingly, the model training unit determines the first posture error information of the workpiece in the simulation environment based on the first workpiece posture information and the target posture information, including: determining the first Euclidean distance between the first workpiece position information and the target position information, and determining the first difference between the first workpiece three-axis orientation angle information and the target three-axis orientation angle information; the first Euclidean distance and the first difference are determined as the first posture error information of the workpiece in the simulation environment.

[0124] For the description of the features in the embodiments corresponding to the manipulator arm control device of the composite robot provided in the embodiments of the present application, please refer to the relevant description of the embodiments corresponding to the manipulator arm control method of the composite robot, which will not be repeated here.

[0125] Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 further includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus.

[0126] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502 , so that the at least one processor 501 executes the above-mentioned embodiment of the method for controlling the manipulator arm of the composite robot.

[0127] The specific implementation process of the processor 501 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0128] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the application may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0129] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage.

[0130] A bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0131] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned embodiments of the method for controlling the manipulator arm of a composite robot when running.

[0132] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0133] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned embodiments of the method for controlling the manipulator arm of a composite robot are implemented.

[0134] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned embodiments of the method for controlling the manipulator arm of a composite robot.

[0135] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0136] The above is a detailed introduction to the manipulator arm control method, device, equipment and storage medium of a composite robot provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for controlling a manipulator arm of a composite robot, characterized in that: include: Acquire motion state information of a manipulator arm of the composite robot, workpiece point cloud data and target point cloud data of a workpiece clamped by the manipulator arm in a first coordinate system, wherein the first coordinate system is a coordinate system of a depth camera used to acquire workpiece point cloud data; Determining position error information of the workpiece in a real environment based on the workpiece point cloud data and the target point cloud data; and, determining the manipulator position information of the manipulator in a second coordinate system according to the motion state information of the manipulator; The second coordinate system is a coordinate system of a mobile chassis for fixing the robotic arm; According to the motion state information of the robotic arm, the posture information of the robotic arm and the posture error information of the workpiece, calling a trained neural network model, determining the posture adjustment parameters of the robotic arm, and controlling the robotic arm to execute the posture adjustment parameters; The training process of the neural network model includes: obtaining the motion state information of the manipulator arm of the composite robot, the first workpiece posture information and the target posture information of the workpiece clamped by the manipulator arm in the first coordinate system, wherein the first coordinate system is the coordinate system of the depth camera; determining the first posture error information of the workpiece in the simulation environment according to the first workpiece posture information and the target posture information; and determining the manipulator arm posture information of the manipulator arm in the second coordinate system according to the motion state information of the manipulator arm; wherein the second coordinate system is the coordinate system of the mobile chassis; determining the first posture error information of the workpiece in the simulation environment according to the first workpiece posture information and the target posture information; and determining the manipulator arm posture information of the manipulator arm in the second coordinate system according to the motion state information of the manipulator arm, wherein the second coordinate system is the coordinate system of the mobile chassis; and determining the first posture error information of the workpiece in the simulation environment according to the first workpiece posture information and the target posture information. The first posture error information calls the primary neural network model to determine the first posture adjustment parameter of the robot arm, and controls the robot arm to execute the first posture adjustment parameter; after determining the execution of the first posture adjustment parameter, the second workpiece posture information of the workpiece clamped by the robot arm in the first coordinate system is determined, and the second posture error information of the workpiece in the simulation environment is determined according to the second workpiece posture information and the target posture information; if the second posture error information is greater than the preset error threshold, the model parameters of the primary neural network model are updated until the obtained posture error information is less than the preset error threshold, and a trained neural network model is obtained.

2. The robotic arm control method according to claim 1, wherein: The method of calling a trained neural network model to determine the posture adjustment parameters of the robotic arm according to the motion state information of the robotic arm, the posture information of the robotic arm, and the posture error information of the workpiece includes: Converting the motion state information of the robotic arm, the posture information of the robotic arm, and the posture error information of the workpiece into one-dimensional vector information and inputting the information into the neural network model, thereby outputting the mean value of the posture parameters of the robotic arm; According to the mean value of the posture parameters of the robotic arm, the posture parameters are sampled according to Gaussian distribution to obtain the posture adjustment parameters of the robotic arm.

3. The robot arm control method according to claim 2, characterized in that: The robotic arm includes multiple joints and a gripper; the motion state information of the robotic arm includes the angle of each joint and the opening and closing state of the gripper, as well as the angular velocity of each joint and the opening and closing velocity of the gripper; the posture adjustment parameters include the angle adjustment parameters of each joint.

4. The robot arm control method according to claim 1, characterized in that: Determining the pose error information of the workpiece in a real environment based on the workpiece point cloud data and the target point cloud data includes: Determine a rotation and translation matrix of a workpiece position and a target position according to the workpiece point cloud data and the target point cloud data, wherein the rotation and translation matrix includes rotation transformation parameters and translation vector parameters; The Euclidean distance between the workpiece position and the target position is determined according to the translation vector parameter; and the difference in three-axis orientation angle between the workpiece position and the target position is determined according to the rotation transformation parameter.

5. The robot arm control method according to any one of claims 1 to 4, characterized in that: The method further comprises: Manually controlling the composite robot to grab the workpiece to be assembled, and adjusting the posture parameters of the robotic arm so that the position of the workpiece meets the assembly process requirements of the workpiece; The depth camera fixed on the workbench is triggered to shoot the located workpiece, obtain point cloud data of the workpiece, and store the point cloud data as target point cloud data.

6. The robot arm control method according to claim 1, characterized in that: The first workpiece posture information includes first workpiece position information and first workpiece three-axis orientation angle information, and the target posture information includes target position information and target three-axis orientation angle information; Accordingly, determining the first pose error information of the workpiece in the simulation environment according to the first workpiece pose information and the target pose information includes: Determining a first Euclidean distance between the first workpiece position information and the target position information, and determining a first difference between the first workpiece three-axis orientation angle information and the target three-axis orientation angle information; The first Euclidean distance and the first difference are determined as first position error information of the workpiece in a simulation environment.

7. A manipulator arm control device for a composite robot, characterized in that: include: an acquisition unit, configured to acquire motion state information of a manipulator arm of the composite robot, workpiece point cloud data and target point cloud data of a workpiece clamped by the manipulator arm in a first coordinate system, wherein the first coordinate system is a coordinate system of a depth camera used to acquire workpiece point cloud data; a determining unit, configured to determine position error information of the workpiece in a real environment based on the workpiece point cloud data and the target point cloud data; and, determining the manipulator position information of the manipulator in a second coordinate system according to the motion state information of the manipulator; The second coordinate system is a coordinate system of a mobile chassis for fixing the robotic arm; A control unit is used to call a trained neural network model based on the motion state information of the robotic arm, the robotic arm posture information and the posture error information of the workpiece, determine the posture adjustment parameters of the robotic arm, and control the robotic arm to execute the posture adjustment parameters; wherein the training process of the neural network model includes: obtaining the motion state information of the robotic arm of the compound robot, the first workpiece posture information and the target posture information of the workpiece clamped by the robotic arm in a first coordinate system, wherein the first coordinate system is the coordinate system of the depth camera; determining the first posture error information of the workpiece in a simulated environment based on the first workpiece posture information and the target posture information; and determining the robotic arm posture information of the robotic arm in a second coordinate system based on the motion state information of the robotic arm. ; wherein the second coordinate system is the coordinate system of the mobile chassis; according to the motion state information of the robotic arm, the robotic arm posture information and the first posture error information, the primary neural network model is called to determine the first posture adjustment parameter of the robotic arm, and the robotic arm is controlled to execute the first posture adjustment parameter; after determining the execution of the first posture adjustment parameter, the second workpiece posture information of the workpiece clamped by the robotic arm in the first coordinate system, according to the second workpiece posture information and the target posture information, the second posture error information of the workpiece in the simulation environment is determined; if the second posture error information is greater than the preset error threshold, the model parameters of the primary neural network model are updated until the obtained posture error information is less than the preset error threshold, and a trained neural network model is obtained.

8. An electronic device, characterized in that: include: memory for storing computer programs; A processor is configured to implement the steps of the method for controlling the manipulator arm of a composite robot as claimed in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method for controlling the manipulator arm of the composite robot according to any one of claims 1 to 6 are implemented.

Images