Double-arm robot satellite assembly method and device based on multi-agent reinforcement learning

Abstract

The invention relates to the field of robots and artificial intelligence, in particular to a double-arm robot satellite assembly method and device based on multi-agent reinforcement learning. The method comprises the steps that a double-arm robot satellite assembly system is built; according to the double-arm robot satellite assembly system, an MDP model of double-mechanical-arm satellite collaborative assembly is established; based on an MADDPG algorithm, the MDP model of the double-mechanical-arm satellite collaborative assembly is trained, and the trained MDP model of the double-mechanical-arm satellite collaborative assembly is obtained; and adjusting admittance control parameters of the double-arm robot by using the trained MDP model for cooperative assembly of the double-arm satellite, and completing a satellite assembly task of the double-arm robot. Compared with a traditional assembling method, when satellite parts are assembled, the contact force between the assembling parts is more stable, and the technical problems that automatic assembling adaptability of a double-arm robot is poor, and autonomous learning cannot be achieved can be solved.

Description

Technical Field

[0001] The present invention relates to the field of robots and artificial intelligence technology, and in particular to a dual-arm robot satellite assembly method and device based on multi-agent reinforcement learning. Background Art

[0002] In satellite engineering, satellite assembly is one of the important links. At present, the field of satellite assembly is in rapid development and change. With the continuous development of automation technology, automated assembly lines and robotic arms can be used for the assembly and connection of satellite components, greatly reducing the need for manual operation. However, with the continuous expansion of satellite application scenarios and the continuous advancement of space technology in various countries, satellite models are becoming more and more diverse. Automated production line assembly has begun to show its weakness because it only assembles a single or one type of satellite. Satellite assembly has begun to develop in the direction of efficiency and intelligence. Dual-arm robots have left and right arms similar to humans, and can coordinate operations. They can coordinate with each other to complete more complex tasks and perform precise assembly operations.

[0003] In recent years, robot control methods based on reinforcement learning have achieved initial results in improving system autonomy and robustness. Through reinforcement learning, robots can learn the optimal action strategy through interaction with the environment, thereby achieving autonomous decision-making and action. Reinforcement learning can also be used to adjust model parameters and dynamically optimize robot motion control when the task environment changes, which can further improve the robustness of robot control methods. For the satellite assembly task of the dual-arm robot, the two agents represented by the two robotic arms achieve the best overall performance during the task through collaborative learning and decision-making, which is more suitable for multi-agent reinforcement learning algorithms. Summary of the invention

[0004] In order to solve the technical problems of poor adaptability and inability to autonomous learning of dual-arm robot automated assembly in the prior art, the present invention provides a dual-arm robot satellite assembly method and device based on multi-agent reinforcement learning. The technical solution is as follows:

[0005] On the one hand, a dual-arm robot satellite assembly method based on multi-agent reinforcement learning is provided, the method is implemented by a dual-arm robot satellite assembly device, and the method includes:

[0006] S1. Build a dual-arm robot satellite assembly system.

[0007] S2. Based on the dual-arm robot satellite assembly system, an MDP model for dual-arm satellite collaborative assembly is established.

[0008] S3. Based on the MADDPG algorithm, the MDP model of the collaborative assembly of dual-robotic-arm satellites is trained to obtain a trained MDP model of the collaborative assembly of dual-robotic-arm satellites.

[0009] S4. Use the trained MDP model of dual-arm satellite collaborative assembly to adjust the admittance control parameters of the dual-arm robot to complete the dual-arm robot satellite assembly task.

[0010] Optionally, the dual-arm robot satellite assembly system in S1 includes: a multi-degree-of-freedom dual robotic arm subsystem, a six-dimensional force sensor subsystem, a visual sensor subsystem, an end tool subsystem, a mobile chassis subsystem, and a multi-agent reinforcement learning subsystem.

[0011] Optionally, the state of the MDP model of the dual-manipulator satellite collaborative assembly in S2 is as shown in the following equation (1):

[0012] (1)

[0013] In the formula, Represents the state of the MDP model for collaborative assembly of dual-manipulator satellites; Indicates the contact point set for the left arm part , , Force information in three directions; Indicates the contact point set for the right arm part , , Force information in three directions; Indicates the contact point set for the left arm part , Torque information in two directions; Indicates the contact point set for the right arm part , Torque information in two directions.

[0014] Optionally, the actions of the MDP model of the dual-manipulator satellite collaborative assembly in S2 are as shown in the following equation (2):

[0015] (2)

[0016] In the formula, Represents the left arm motion of the MDP model for collaborative assembly of dual-manipulator satellites; Represents the right arm action of the MDP model for collaborative assembly of dual-manipulator satellites; Represents the left arm admittance controller , , Three degrees of freedom of movement and , Inertia parameters in two rotational degrees of freedom; Represents the right arm admittance controller , , Three degrees of freedom of movement and , Inertia parameters in two rotational degrees of freedom; Represents the left arm admittance controller , , Three degrees of freedom of movement and , Damping parameters on two rotational degrees of freedom; Represents the right arm admittance controller , , Three degrees of freedom of movement and , Damping parameters for the two rotational degrees of freedom.

[0017] Optionally, the reward function of the MDP model for the dual-manipulator satellite collaborative assembly in S2 is as shown in the following equation (3):

[0018] (3)

[0019] In the formula, Represents the left arm reward function of the MDP model for collaborative assembly of dual-manipulator satellites; Represents the right arm reward function of the MDP model for collaborative assembly of dual-manipulator satellites; Indicates the reward obtained when the jack is successfully inserted; Indicates the reward when the jack fails; Indicates the penalty when the contact force is too high; Indicates the penalty when the contact force is out of range; Indicates the number of steps in a jack round; Indicates the total number of steps in the current round; Indicates the maximum torque value; Indicates the minimum torque value; It represents the resultant force in the y-axis and z-axis directions; Represents the force in the x-axis direction.

[0020] Optionally, the MDP model of the satellite collaborative assembly of dual robotic arms is trained based on the MADDPG algorithm in S3 to obtain the trained MDP model of the satellite collaborative assembly of dual robotic arms, including:

[0021] S31. Build an experience pool.

[0022] S32: Obtain any experience in the experience pool, and use the first strategy network to determine the next state of the acquired experience. Make predictions and get the next action ; The next state of the acquired experience through the second strategy network Make predictions and get the next action .

[0023] S33. Predict the current action and state through the first value network to obtain the first action value; predict the next action and state through the second value network to obtain the second action value.

[0024] S34. Calculate the TD target and TD error according to the first action value, the second action value and the discounted reward.

[0025] S35. Update the first value network and the second value network according to the TD error, and update the first strategy network and the second strategy network according to the updated first value network and the second value network.

[0026] Optionally, the method of using the trained MDP model for satellite collaborative assembly of dual-arm manipulators in S4 to adjust the admittance control parameters of the dual-arm robot to complete the dual-arm robot satellite assembly task includes:

[0027] S41, obtaining left arm force sensor data and right arm force sensor data.

[0028] S42, performing gravity compensation and rigid body internal force transformation on the left arm force sensor data and the right arm force sensor data respectively to obtain the left arm end contact force and the right arm end contact force.

[0029] S43, inputting the contact force at the end of the left arm into the first strategy network to obtain the parameters of the left arm admittance controller; inputting the contact force at the end of the right arm into the second strategy network to obtain the parameters of the right arm admittance controller.

[0030] S44. Design the desired force of the left arm admittance controller and the desired force of the right arm admittance controller.

[0031] S45. Input the desired force of the left arm admittance controller and the contact force at the end of the left arm into the left arm admittance controller to obtain the end movement speed of the left arm; input the desired force of the right arm admittance controller and the contact force at the end of the right arm into the right arm admittance controller to obtain the end movement speed of the right arm.

[0032] S46, sending the end motion speed of the left arm and the end motion speed of the right arm to the dual-arm robot to complete the satellite assembly task.

[0033] On the other hand, a dual-arm robot satellite assembly device based on multi-agent reinforcement learning is provided, and the device is applied to a dual-arm robot satellite assembly method based on multi-agent reinforcement learning, and the device comprises:

[0034] System building module, used to build a dual-arm robot satellite assembly system.

[0035] The model building module is used to build the MDP model of dual-arm robot satellite collaborative assembly based on the dual-arm robot satellite assembly system.

[0036] The model training module is used to train the MDP model of the collaborative assembly of dual-robotic-arm satellites based on the MADDPG algorithm to obtain a trained MDP model of the collaborative assembly of dual-robotic-arm satellites.

[0037] The output module is used to adjust the admittance control parameters of the dual-arm robot using the trained MDP model of dual-arm satellite collaborative assembly to complete the dual-arm robot satellite assembly task.

[0038] Optionally, the dual-arm robot satellite assembly system includes: a multi-degree-of-freedom dual robotic arm subsystem, a six-dimensional force sensor subsystem, a visual sensor subsystem, an end tool subsystem, a mobile chassis subsystem and a multi-agent reinforcement learning subsystem.

[0039] Optionally, the state of the MDP model for the collaborative assembly of dual-manipulator satellites is as shown in the following equation (1):

[0040] (1)

[0041] In the formula, Represents the state of the MDP model for collaborative assembly of dual-manipulator satellites; Indicates the contact point set for the left arm part , , Force information in three directions; Indicates the contact point set for the right arm part , , Force information in three directions; Indicates the contact point set for the left arm part , Torque information in two directions; Indicates the contact point set for the right arm part , Torque information in two directions.

[0042] Optionally, the actions of the MDP model for the collaborative assembly of dual-manipulator satellites are as shown in the following equation (2):

[0043] (2)

[0044] In the formula, Represents the left arm motion of the MDP model for collaborative assembly of dual-manipulator satellites; Represents the right arm action of the MDP model for collaborative assembly of dual-manipulator satellites; Represents the left arm admittance controller , , Three degrees of freedom of movement and , Inertia parameters in two rotational degrees of freedom; Represents the right arm admittance controller , , Three degrees of freedom of movement and , Inertia parameters in two rotational degrees of freedom; Represents the left arm admittance controller , , Three degrees of freedom of movement and , Damping parameters on two rotational degrees of freedom; Represents the right arm admittance controller , , Three degrees of freedom of movement and , Damping parameters for the two rotational degrees of freedom.

[0045] Optionally, the reward function of the MDP model for the collaborative assembly of dual-robotic-arm satellites is as shown in the following equation (3):

[0046] (3)

[0047] In the formula, Represents the left arm reward function of the MDP model for collaborative assembly of dual-manipulator satellites; Represents the right arm reward function of the MDP model for collaborative assembly of dual-manipulator satellites; Indicates the reward obtained when the jack is successfully inserted; Indicates the reward when the jack fails; Indicates the penalty when the contact force is too high; Indicates the penalty when the contact force is out of range; Indicates the number of steps in a jack round; Indicates the total number of steps in the current round; Indicates the maximum torque value; Indicates the minimum torque value; It represents the resultant force in the y-axis and z-axis directions; Represents the force in the x-axis direction.

[0048] Optionally, the model training module is further used to:

[0049] S31. Build an experience pool.

[0050] S32: Obtain any experience in the experience pool, and use the first strategy network to determine the next state of the acquired experience. Make predictions and get the next action ; The next state of the acquired experience through the second strategy network Make predictions and get the next action .

[0051] S33. Predict the current action and state through the first value network to obtain the first action value; predict the next action and state through the second value network to obtain the second action value.

[0052] S34. Calculate the TD target and TD error according to the first action value, the second action value and the discounted reward.

[0053] S35. Update the first value network and the second value network according to the TD error, and update the first strategy network and the second strategy network according to the updated first value network and the second value network.

[0054] Optionally, the output module is further configured to:

[0055] S41, obtaining left arm force sensor data and right arm force sensor data.

[0056] S42, performing gravity compensation and rigid body internal force transformation on the left arm force sensor data and the right arm force sensor data respectively to obtain the left arm end contact force and the right arm end contact force.

[0057] S43, inputting the contact force at the end of the left arm into the first strategy network to obtain the parameters of the left arm admittance controller; inputting the contact force at the end of the right arm into the second strategy network to obtain the parameters of the right arm admittance controller.

[0058] S44. Design the desired force of the left arm admittance controller and the desired force of the right arm admittance controller.

[0059] S45. Input the desired force of the left arm admittance controller and the contact force at the end of the left arm into the left arm admittance controller to obtain the end movement speed of the left arm; input the desired force of the right arm admittance controller and the contact force at the end of the right arm into the right arm admittance controller to obtain the end movement speed of the right arm.

[0060] S46, sending the end motion speed of the left arm and the end motion speed of the right arm to the dual-arm robot to complete the satellite assembly task.

[0061] On the other hand, a dual-arm robot satellite assembly device is provided, which includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned dual-arm robot satellite assembly methods based on multi-agent reinforcement learning is implemented.

[0062] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned dual-arm robot satellite assembly methods based on multi-agent reinforcement learning.

[0063] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0064] In the embodiment of the present invention, the dual-arm robot can realize four typical tasks in the process of assembling a micro-satellite: dual-arm plugging, dual-arm twisting, dual-arm carrying, and flexible control algorithms in dual-arm plugging, so as to realize the dual-arm robot to perform flexible and coordinated assembly of the satellite wings and satellite shell parts of the micro-satellite, and make further explorations for the efficiency and intelligence of satellite assembly. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0066] Figure 1 It is a flow chart of a dual-arm robot satellite assembly method based on multi-agent reinforcement learning provided by an embodiment of the present invention;

[0067] Figure 2 is a schematic diagram of the system composition provided by an embodiment of the present invention;

[0068] Figure 3 is a dual-arm overall control strategy diagram provided by an embodiment of the present invention;

[0069] Figure 4 is a diagram of a reinforcement learning simulation training environment provided by an embodiment of the present invention;

[0070] Figure 5 It is a training block diagram of a dual-arm variable admittance control algorithm based on a MADDPG algorithm provided by an embodiment of the present invention;

[0071] Figure 6 It is an execution block diagram of a dual-arm variable admittance control algorithm under a strategy network provided by an embodiment of the present invention;

[0072] Figure 7 It is a block diagram of a dual-arm robot satellite assembly device based on multi-agent reinforcement learning provided by an embodiment of the present invention;

[0073] Figure 8 It is a structural schematic diagram of a dual-arm robot satellite assembly device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0075] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0076] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0077] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are consistent.

[0078] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0079] The embodiment of the present invention provides a dual-arm robot satellite assembly method based on multi-agent reinforcement learning, which can be implemented by a dual-arm robot satellite assembly device, which can be a terminal or a server. Figure 1 The flowchart of the dual-arm robot satellite assembly method based on multi-agent reinforcement learning is shown in the figure. The processing flow of the method may include the following steps:

[0080] S1. Build a dual-arm robot satellite assembly system.

[0081] Alternatively, if Figure 2As shown, the dual-arm robot satellite assembly system in S1 includes: a multi-degree-of-freedom dual robotic arm subsystem, a six-dimensional force sensor subsystem, a visual sensor subsystem, an end tool subsystem, a mobile chassis subsystem, and a multi-agent reinforcement learning subsystem.

[0082] In a feasible implementation, the multi-degree-of-freedom dual-arm subsystem can mainly follow a preset trajectory or make the end of the robotic arm reach a desired position, which needs to be achieved through robot inverse kinematics, and the corresponding joint angles are inversely solved through the motion trajectory or the known end posture.

[0083] The six-dimensional force sensor subsystem detects displacement changes and converts them into real-time force and torque using relevant algorithms, and has six-dimensional force / torque detection capabilities.

[0084] The vision sensor subsystem consists of stereo vision cameras that provide accurate depth perception when objects are moving or the device is in motion.

[0085] The end tool subsystem is mainly responsible for grabbing and fixing the satellite parts to be operated.

[0086] The mobile chassis subsystem is mainly responsible for driving the dual-arm robot to move to the designated position.

[0087] The multi-agent reinforcement learning subsystem adopts a force-based admittance control deep reinforcement learning strategy to model and generate strategies for each assembly subtask.

[0088] S2. Based on the dual-arm robot satellite assembly system, an MDP model for dual-arm satellite collaborative assembly is established.

[0089] In a feasible implementation, Figure 3 , Figure 4 As shown, a reinforcement learning visualization simulation environment is established. The simulation environment realizes information interaction between the simulation environment and the intelligent agent, and trains the decision model of the assembly task.

[0090] Specifically, a Markov decision process (MDP) model for the collaborative assembly of dual-manipulator satellites is established, where the state is as shown in the following formula (1):

[0091] (1)

[0092] In the formula, Represents the state of the MDP model for collaborative assembly of dual-manipulator satellites; Indicates the contact point set for the left arm part , , Force information in three directions; Indicates the contact point set for the right arm part , , Force information in three directions; Indicates the contact point set for the left arm part , Torque information in two directions; Indicates the contact point set for the right arm part , Torque information in two directions.

[0093] Optionally, the motion of the MDP model of the dual-manipulator satellite collaborative assembly in S2: inertial parameters on the five degrees of freedom , and the damping parameters on the five degrees of freedom , as shown in the following formula (2):

[0094] (2)

[0095] In the formula, Represents the left arm motion of the MDP model for collaborative assembly of dual-manipulator satellites; Represents the right arm action of the MDP model for collaborative assembly of dual-manipulator satellites; Represents the left arm admittance controller , , Three degrees of freedom of movement and , Inertia parameters in two rotational degrees of freedom; Represents the right arm admittance controller , , Three degrees of freedom of movement and , Inertia parameters in two rotational degrees of freedom; Represents the left arm admittance controller , , Three degrees of freedom of movement and , Damping parameters on two rotational degrees of freedom; Represents the right arm admittance controller , , Three degrees of freedom of movement and , Damping parameters for the two rotational degrees of freedom.

[0096] Optionally, define the reward function as follows:

[0097] (3)

[0098] in, , , Indicates the total number of steps in the current round. A jack round is set to Steps, within 450 steps is a successful jack, and more than 450 steps is a failed jack. Indicates the reward obtained when the jack is successfully inserted. Indicates the reward when the jack fails. It is the resultant force in the y-axis and z-axis directions, that is, the contact force between the axis and the side wall of the hole. is the force in the x-axis direction, that is, the contact force between the front end of the shaft and the surface of the hole part or the bottom of the hole. When the contact force between the shaft and hole is too large, This is the penalty for excessive contact force. When the contact force between the shaft and hole assembly exceeds the range, Penalty for contact force exceeding the range. Indicates the maximum torque value; Indicates the minimum torque value.

[0099] S3. Based on the MADDPG algorithm, the MDP model of the collaborative assembly of dual-robotic-arm satellites is trained to obtain a trained MDP model of the collaborative assembly of dual-robotic-arm satellites.

[0100] Optionally, the above step S3 may include the following steps S31-S35:

[0101] S31. Build an experience pool.

[0102] S32: Obtain any experience in the experience pool, and use the first strategy network to determine the next state of the acquired experience. Make predictions and get the next action ; The next state of the acquired experience through the second strategy network Make predictions and get the next action .

[0103] S33. Predict the current action and state through the first value network to obtain the first action value; predict the next action and state through the second value network to obtain the second action value.

[0104] S34. Calculate the TD target and TD error according to the first action value, the second action value and the discounted reward.

[0105] S35. Update the first value network and the second value network according to the TD error, and update the first strategy network and the second strategy network according to the updated first value network and the second value network.

[0106] In a feasible implementation, Figure 5As shown, a multi-agent reinforcement learning framework is established based on the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm and dual-manipulator satellite collaborative assembly training is performed. Here, the present invention adopts a centralized training distributed application framework, that is, the input states of the value network and the policy network are both global states, and the outputs are local values ​​and local policies.

[0107] The following describes the update of the real value network and the real policy network. When the algorithm is running, the experience pool is first collected through the execution part. When the experience pool has accumulated a certain amount of experience, the training and update of the policy network and the value network begins. For an experience in the experience pool, let the two policy networks update the next state. and Make predictions and get the next action and ; Secondly, let the value network predict the current action and state and the next action and state to obtain two action values. Then use the discounted reward and action value to calculate the TD target and TD error. Finally, update the value network Q1 and Q2 according to the TD error, and then use the value network to guide the update of the policy network.

[0108] S4. Use the trained MDP model of dual-arm satellite collaborative assembly to adjust the admittance control parameters of the dual-arm robot to complete the dual-arm robot satellite assembly task.

[0109] Optionally, the above step S4 may include the following steps S41-S46:

[0110] S41, obtaining left arm force sensor data and right arm force sensor data.

[0111] S42, performing gravity compensation and rigid body internal force transformation on the left arm force sensor data and the right arm force sensor data respectively to obtain the left arm end contact force and the right arm end contact force.

[0112] S43, inputting the contact force at the end of the left arm into the first strategy network to obtain the parameters of the left arm admittance controller; inputting the contact force at the end of the right arm into the second strategy network to obtain the parameters of the right arm admittance controller.

[0113] S44. Design the desired force of the left arm admittance controller and the desired force of the right arm admittance controller.

[0114] S45. Input the desired force of the left arm admittance controller and the contact force at the end of the left arm into the left arm admittance controller to obtain the end movement speed of the left arm; input the desired force of the right arm admittance controller and the contact force at the end of the right arm into the right arm admittance controller to obtain the end movement speed of the right arm.

[0115] S46, sending the end motion speed of the left arm and the end motion speed of the right arm to the dual-arm robot to complete the satellite assembly task.

[0116] In a feasible implementation, the admittance parameters of the dual-arm robot are adjusted by using a strategy network so that the dual-arm robot can perform variable admittance compliant control. Figure 6 As shown in the figure, first, the force sensor data of the left arm and the force sensor data of the right arm are obtained from the robot operation platform; secondly, the data of the left and right arm force sensors are used to compensate for the gravity of the fixture and the object it holds, and transform the internal force of the rigid body, so as to obtain the end contact force of the left arm and the end contact force of the right arm. Then, the end contact force of the left arm is input into the strategy network , get the parameters of the left arm admittance controller and The end contact force of the right arm is input into the strategy network , get the parameters of the right arm admittance controller and Finally, the expected forces of the left and right arm admittance controllers are designed according to the task conditions, and the contact forces at the ends of the left and right arms are input into the left and right arm admittance controllers respectively, so that the end motion velocities of the left and right arms can be obtained and sent to the robot for variable admittance compliance operation.

[0117] Furthermore, the trained decision model is transferred to the physical system of the dual-arm robot satellite assembly system, thereby realizing the dual-arm collaborative satellite assembly task.

[0118] The present invention adopts a multi-agent reinforcement learning algorithm to dynamically optimize the admittance control parameters under dual-arm coordination, so that each contact force in the assembly process meets the assembly requirements, including the loose coordination task of screwing the sockets of satellite wing components and the tight coordination task of carrying and plugging satellite shell components.

[0119] In the embodiment of the present invention, the dual-arm robot can realize four typical tasks in the process of assembling a micro-satellite: dual-arm plugging, dual-arm twisting, dual-arm carrying, and flexible control algorithms in dual-arm plugging, so as to realize the dual-arm robot to perform flexible and coordinated assembly of the satellite wings and satellite shell parts of the micro-satellite, and make further explorations for the efficiency and intelligence of satellite assembly.

[0120] Figure 7 The present invention is a block diagram of a dual-arm robot satellite assembly device based on multi-agent reinforcement learning according to an exemplary embodiment. The device is used in a dual-arm robot satellite assembly method based on multi-agent reinforcement learning. Figure 7 The device includes a system building module 310, a model building module 320, a model training module 330 and an output module 340. Among them:

[0121] The system building module 310 is used to build a dual-arm robot satellite assembly system.

[0122] The model building module 320 is used to build an MDP model of dual-arm robotic satellite collaborative assembly according to the dual-arm robotic satellite assembly system.

[0123] The model training module 330 is used to train the MDP model of the collaborative assembly of satellites with dual robotic arms based on the MADDPG algorithm to obtain a trained MDP model of the collaborative assembly of satellites with dual robotic arms.

[0124] The output module 340 is used to adjust the admittance control parameters of the dual-arm robot using the trained dual-arm satellite collaborative assembly MDP model to complete the dual-arm robot satellite assembly task.

[0125] Optionally, the dual-arm robot satellite assembly system includes: a multi-degree-of-freedom dual robotic arm subsystem, a six-dimensional force sensor subsystem, a visual sensor subsystem, an end tool subsystem, a mobile chassis subsystem and a multi-agent reinforcement learning subsystem.

[0126] Optionally, the state of the MDP model for the collaborative assembly of dual-manipulator satellites is as shown in the following equation (1):

[0127] (1)

[0128] In the formula, Represents the state of the MDP model for collaborative assembly of dual-manipulator satellites; Indicates the contact point set for the left arm part , , Force information in three directions; Indicates the contact point set for the right arm part , , Force information in three directions; Indicates the contact point set for the left arm part , Torque information in two directions; Indicates the contact point set for the right arm part , Torque information in two directions.

[0129] Optionally, the actions of the MDP model for the collaborative assembly of dual-manipulator satellites are as shown in the following equation (2):

[0130] (2)

[0131] In the formula, Represents the left arm motion of the MDP model for collaborative assembly of dual-manipulator satellites; Represents the right arm action of the MDP model for collaborative assembly of dual-manipulator satellites; Represents the left arm admittance controller , , Three degrees of freedom of movement and , Inertia parameters in two rotational degrees of freedom; Represents the right arm admittance controller , , Three degrees of freedom of movement and , Inertia parameters in two rotational degrees of freedom; Represents the left arm admittance controller , , Three degrees of freedom of movement and , Damping parameters on two rotational degrees of freedom; Represents the right arm admittance controller , , Three degrees of freedom of movement and , Damping parameters for the two rotational degrees of freedom.

[0132] Optionally, the reward function of the MDP model for the collaborative assembly of dual-robotic-arm satellites is as shown in the following equation (3):

[0133] (3)

[0134] In the formula, Represents the left arm reward function of the MDP model for collaborative assembly of dual-manipulator satellites; Represents the right arm reward function of the MDP model for collaborative assembly of dual-manipulator satellites; Indicates the reward obtained when the jack is successfully inserted; Indicates the reward when the jack fails; Indicates the penalty when the contact force is too high; Indicates the penalty when the contact force is out of range; Indicates the number of steps in a jack round; Indicates the total number of steps in the current round; Indicates the maximum torque value; Indicates the minimum torque value; It represents the resultant force in the y-axis and z-axis directions; Represents the force in the x-axis direction.

[0135] Optionally, the model training module 330 is further used to:

[0136] S31. Build an experience pool.

[0137] S32: Obtain any experience in the experience pool, and use the first strategy network to determine the next state of the acquired experience. Make predictions and get the next action ; The next state of the acquired experience through the second strategy network Make predictions and get the next action .

[0138] S33. Predict the current action and state through the first value network to obtain the first action value; predict the next action and state through the second value network to obtain the second action value.

[0139] S34. Calculate the TD target and TD error according to the first action value, the second action value and the discounted reward.

[0140] S35. Update the first value network and the second value network according to the TD error, and update the first strategy network and the second strategy network according to the updated first value network and the second value network.

[0141] Optionally, the output module 340 is further configured to:

[0142] S41, obtaining left arm force sensor data and right arm force sensor data.

[0143] S42, performing gravity compensation and rigid body internal force transformation on the left arm force sensor data and the right arm force sensor data respectively to obtain the left arm end contact force and the right arm end contact force.

[0144] S43, inputting the contact force at the end of the left arm into the first strategy network to obtain the parameters of the left arm admittance controller; inputting the contact force at the end of the right arm into the second strategy network to obtain the parameters of the right arm admittance controller.

[0145] S44. Design the desired force of the left arm admittance controller and the desired force of the right arm admittance controller.

[0146] S45. Input the desired force of the left arm admittance controller and the contact force at the end of the left arm into the left arm admittance controller to obtain the end movement speed of the left arm; input the desired force of the right arm admittance controller and the contact force at the end of the right arm into the right arm admittance controller to obtain the end movement speed of the right arm.

[0147] S46, sending the end motion speed of the left arm and the end motion speed of the right arm to the dual-arm robot to complete the satellite assembly task.

[0148] In the embodiment of the present invention, the dual-arm robot can realize four typical tasks in the process of assembling a micro-satellite: dual-arm plugging, dual-arm twisting, dual-arm carrying, and flexible control algorithms in dual-arm plugging, so as to realize the dual-arm robot to perform flexible and coordinated assembly of the satellite wings and satellite shell parts of the micro-satellite, and make further explorations for the efficiency and intelligence of satellite assembly.

[0149] Figure 8 FIG. 1 is a schematic diagram of the structure of a dual-arm robot satellite assembly device provided by an embodiment of the present invention. Figure 8 As shown, the dual-arm robot satellite assembly equipment may include the above Figure 7 The dual-arm robot satellite assembly device based on multi-agent reinforcement learning is shown. Optionally, the dual-arm robot satellite assembly device 410 may include a first processor 2001.

[0150] Optionally, the dual-arm robotic satellite assembly equipment 410 may also include a memory 2002 and a transceiver 2003 .

[0151] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0152] Combine the following Figure 8 The components of the dual-arm robot satellite assembly equipment 410 are specifically introduced:

[0153] The first processor 2001 is the control center of the dual-arm robot satellite assembly device 410, and can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0154] Optionally, the first processor 2001 can perform various functions of the dual-arm robotic satellite assembly equipment 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0155] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 8CPU0 and CPU1 are shown in FIG.

[0156] In a specific implementation, as an embodiment, the dual-arm robot satellite assembly device 410 may also include multiple processors, such as Figure 8 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0157] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0158] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently and access the first processor 2001 through the interface circuit ( Figure 8 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0159] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0160] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 8 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0161] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently and communicate with the first processor 2001 through the interface circuit ( Figure 8 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0162] It should be noted that Figure 8 The structure of the dual-arm robot satellite assembly device 410 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0163] In addition, the technical effects of the dual-arm robot satellite assembly equipment 410 can refer to the technical effects of the dual-arm robot satellite assembly method based on multi-agent reinforcement learning described in the above method embodiment, and will not be repeated here.

[0164] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0165] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0166] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0167] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0168] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0169] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0170] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0171] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0172] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0173] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0175] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0176] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A dual-arm robot satellite assembly method based on multi-agent reinforcement learning, characterized in that: The method comprises: S1. Build a dual-arm robot satellite assembly system; S2, establishing an MDP model of dual-arm robot satellite collaborative assembly according to the dual-arm robot satellite assembly system; S3, training the MDP model of the dual-manipulator satellite collaborative assembly based on the MADDPG algorithm to obtain a trained MDP model of the dual-manipulator satellite collaborative assembly; S4. Use the trained MDP model of dual-arm satellite collaborative assembly to adjust the admittance control parameters of the dual-arm robot to complete the dual-arm robot satellite assembly task.

2. The dual-arm robot satellite assembly method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The dual-arm robot satellite assembly system in S1 includes: a multi-degree-of-freedom dual-manipulator subsystem, a six-dimensional force sensor subsystem, a visual sensor subsystem, an end tool subsystem, a mobile chassis subsystem and a multi-agent reinforcement learning subsystem.

3. The dual-arm robot satellite assembly method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The state of the MDP model of the dual-manipulator satellite collaborative assembly in S2 is shown in the following formula (1): （1） In the formula, Represents the state of the MDP model for collaborative assembly of dual-manipulator satellites; Indicates the contact point set for the left arm part , , Force information in three directions; Indicates the contact point set for the right arm part , , Force information in three directions; Indicates the contact point set for the left arm part , Torque information in two directions; Indicates the contact point set for the right arm part , Torque information in two directions.

4. The dual-arm robot satellite assembly method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The actions of the MDP model of the dual-manipulator satellite collaborative assembly in S2 are shown in the following formula (2): （2） In the formula, Represents the left arm motion of the MDP model for collaborative assembly of dual-manipulator satellites; Represents the right arm action of the MDP model for collaborative assembly of dual-manipulator satellites; Represents the left arm admittance controller , , Three degrees of freedom of movement and , Inertia parameters in two rotational degrees of freedom; Represents the right arm admittance controller , , Three degrees of freedom of movement and , Inertia parameters in two rotational degrees of freedom; Represents the left arm admittance controller , , Three degrees of freedom of movement and , Damping parameters on two rotational degrees of freedom; Represents the right arm admittance controller , , Three degrees of freedom of movement and , Damping parameters for the two rotational degrees of freedom.

5. The dual-arm robot satellite assembly method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The reward function of the MDP model for the dual-manipulator satellite collaborative assembly in S2 is shown in the following formula (3): （3） In the formula, Represents the left arm reward function of the MDP model for collaborative assembly of dual-manipulator satellites; Represents the right arm reward function of the MDP model for collaborative assembly of dual-manipulator satellites; Indicates the reward obtained when the jack is successfully inserted; Indicates the reward when the jack fails; Indicates the penalty when the contact force is too high; Indicates the penalty when the contact force is out of range; Indicates the number of steps in a jack round; Indicates the total number of steps in the current round; Indicates the maximum torque value; Indicates the minimum torque value; It represents the resultant force in the y-axis and z-axis directions; Represents the force in the x-axis direction.

6. The dual-arm robot satellite assembly method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The step of training the MDP model of the satellite collaborative assembly of dual robotic arms based on the MADDPG algorithm in S3 to obtain a trained MDP model of the satellite collaborative assembly of dual robotic arms includes: S31. Build experience pool; S32: Obtain any experience in the experience pool, and use the first strategy network to determine the next state of the acquired experience. Make predictions and get the next action ; The next state of the acquired experience through the second strategy network Make predictions and get the next action ; S33, predicting the current action and state through the first value network to obtain a first action value; predicting the next action and state through the second value network to obtain a second action value; S34, calculating a TD target and a TD error according to the first action value, the second action value and the discounted reward; S35. Update the first value network and the second value network according to the TD error, and update the first strategy network and the second strategy network according to the updated first value network and the second value network.

7. The dual-arm robot satellite assembly method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The step S4 uses the trained MDP model for satellite collaborative assembly of dual-arm robotic arms to adjust the admittance control parameters of the dual-arm robot to complete the dual-arm robot satellite assembly task, including: S41, obtaining left arm force sensor data and right arm force sensor data; S42, performing gravity compensation and rigid body force transformation on the left arm force sensor data and the right arm force sensor data respectively to obtain the left arm end contact force and the right arm end contact force; S43, inputting the contact force of the left arm end into the first strategy network to obtain parameters of the left arm admittance controller; inputting the contact force of the right arm end into the second strategy network to obtain parameters of the right arm admittance controller; S44. Designing the expected force of the left arm admittance controller and the expected force of the right arm admittance controller; S45, inputting the desired force of the left arm admittance controller and the contact force of the left arm end into the left arm admittance controller to obtain the end motion speed of the left arm; inputting the desired force of the right arm admittance controller and the contact force of the right arm end into the right arm admittance controller to obtain the end motion speed of the right arm; S46, sending the end movement speed of the left arm and the end movement speed of the right arm to the dual-arm robot to complete the satellite assembly task.

8. A dual-arm robot satellite assembly device based on multi-agent reinforcement learning, wherein the dual-arm robot satellite assembly device based on multi-agent reinforcement learning is used to implement the dual-arm robot satellite assembly method based on multi-agent reinforcement learning as described in any one of claims 1 to 7, characterized in that: The device comprises: System building module, used to build a dual-arm robot satellite assembly system; A model building module, used to build an MDP model of dual-arm robotic satellite collaborative assembly according to the dual-arm robotic satellite assembly system; A model training module is used to train the MDP model of the dual-manipulator satellite collaborative assembly based on the MADDPG algorithm to obtain a trained MDP model of the dual-manipulator satellite collaborative assembly; The output module is used to adjust the admittance control parameters of the dual-arm robot using the trained dual-arm satellite collaborative assembly MDP model to complete the dual-arm robot satellite assembly task.

9. A dual-arm robot satellite assembly device, characterized in that: The dual-arm robot satellite assembly equipment comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images