Dual-arm robot satellite assembly method and device based on multi-agent reinforcement learning

Abstract

The present application relates to the field of robots and artificial intelligence, and particularly relates to a dual-arm robot satellite assembly method and device based on multi-agent reinforcement learning. The method comprises the following steps: building a dual-arm robot satellite assembly system; establishing a MDP model of dual-arm robot satellite cooperative assembly according to the dual-arm robot satellite assembly system; training the MDP model of dual-arm robot satellite cooperative assembly based on a MADDPG algorithm to obtain a trained MDP model of dual-arm robot satellite cooperative assembly; and adjusting the admittance control parameters of the dual-arm robot by using the trained MDP model of dual-arm robot satellite cooperative assembly to complete the dual-arm robot satellite assembly task. Compared with the traditional assembly method, the contact force between each assembly part is more stable when the satellite parts are assembled, and the technical problems of poor adaptability and inability to learn autonomously of the dual-arm robot automatic assembly can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robots and artificial intelligence, and particularly relates to a dual-arm robot satellite assembly method and device based on multi-agent reinforcement learning. BACKGROUND

[0002] In satellite engineering, satellite assembly is an important link. At present, the field of satellite assembly is in rapid development and change. With the continuous development of automation technology, automated assembly lines and robot arms can be used for the assembly and connection of satellite components, greatly reducing the demand for manual operation. However, with the continuous expansion of satellite application scenarios and the continuous progress of space technology, satellite models are becoming more and more diversified. The automated production line assembly begins to show its weaknesses as it only assembles a single or a type of satellite. Satellite assembly is developing towards high efficiency and intelligence. Dual-arm robots, similar to humans, have left and right arms that can coordinate operations, enabling them to coordinate with each other to complete more complex tasks and perform precise assembly operations.

[0003] In recent years, robot control methods based on reinforcement learning have shown initial success in improving system autonomy and robustness. Through reinforcement learning, robots can learn optimal action strategies through interaction with the environment, enabling autonomous decision-making and action. Reinforcement learning can also be used to adjust model parameters, dynamically optimizing robot motion control when the task environment changes, which can further improve the robustness of the robot control method. For the satellite assembly task of dual-arm robots, two agents represented by two robot arms learn and make decisions in coordination to achieve optimal overall performance during the task, making them more suitable for multi-agent reinforcement learning algorithms. SUMMARY

[0004] To solve the technical problems of poor adaptability and inability to learn autonomously of existing dual-arm robot automation assembly, the embodiments of the present application provide a dual-arm robot satellite assembly method and device based on multi-agent reinforcement learning. The technical solution is as follows:

[0005] On the one hand, a dual-arm robot satellite assembly method based on multi-agent reinforcement learning is provided, which is implemented by a dual-arm robot satellite assembly device. The method comprises the following steps:

[0006] S1, a dual-arm robot satellite assembly system is built.

[0007] S2, a dual-robot satellite collaborative assembly MDP model is established according to the dual-arm robot satellite assembly system.

[0008] S3, training the MDP model of the dual-arm satellite cooperative assembly based on the MADDPG algorithm to obtain the trained MDP model of the dual-arm satellite cooperative assembly.

[0009] S4, adjusting the admittance control parameters of the dual-arm robot by using the trained MDP model of the dual-arm satellite cooperative assembly to complete the satellite assembly task of the dual-arm robot.

[0010] Optionally, the dual-arm robot satellite assembly system in S1 comprises a multi-degree-of-freedom dual-arm subsystem, a six-dimensional force sensor subsystem, a vision sensor subsystem, an end tool subsystem, a mobile chassis subsystem and a multi-agent reinforcement learning subsystem.

[0011] Optionally, the state of the MDP model of the dual-arm satellite cooperative assembly in S2 is shown in the following formula (1):

[0012] (1)

[0013] In the formula, represents the state of the MDP model of the dual-arm satellite cooperative assembly; represents the force information in three directions of the set contact point of the left arm part , , represents the force information in three directions of the set contact point of the right arm part , , represents the torque information in two directions of the set contact point of the left arm part , represents the torque information in two directions of the set contact point of the right arm part ,

[0014] Optionally, the action of the MDP model of the dual-arm satellite cooperative assembly in S2 is shown in the following formula (2):

[0015] (2)

[0016] In the formula, represents the left arm action of the MDP model of the dual-arm satellite cooperative assembly; represents the right arm action of the MDP model of the dual-arm satellite cooperative assembly; represents the three movement degrees of freedom of the left arm admittance controller , , , ​​​​​Inertial parameters on two rotational degrees of freedom; Indicates the right arm admittance controller , , Three degrees of freedom of movement and , Inertial parameters on two rotational degrees of freedom; Indicates left arm admittance controller , , Three degrees of freedom of movement and , Damping parameters for two rotational degrees of freedom; Indicates the right arm admittance controller , , Three degrees of freedom of movement and , Damping parameters for the two rotational degrees of freedom.

[0017] Optionally, the reward function of the MDP model for dual-manipulator satellite collaborative assembly in S2 is shown in equation (3) below:

[0018] (3)

[0019] In the formula, The left arm reward function represents the MDP model of dual-arm satellite collaborative assembly; The reward function for the right arm of the MDP model representing dual-arm satellite collaborative assembly; This indicates the reward received when the socket is successfully connected; Indicates a reward when the socket fails to connect; This indicates a penalty for applying excessive contact force. Indicates the penalty for exceeding the contact force limit; Indicates the number of steps in a socket round; This represents the total number of moves in the current round; Indicates the maximum torque value; Indicates the minimum torque value; This represents the resultant force along the y-axis and z-axis. This represents the force along the x-axis.

[0020] Optionally, the MADDPG-based algorithm in S3 is used to train the MDP model for dual-robotic arm satellite collaborative assembly, resulting in a trained MDP model for dual-robotic arm satellite collaborative assembly, including:

[0021] S31. Construct an experience pool.

[0022] S32. Obtain any experience from the experience pool, and determine the next state of the obtained experience through the first policy network. performing prediction to obtain the next action ; performing prediction to obtain the next state of the acquired experience by the second policy network performing prediction to obtain the next action .

[0023] S33, predicting the current action and state by the first value network to obtain a first action value; predicting the next action and state by the second value network to obtain a second action value.

[0024] S34, calculating a TD target and a TD error according to the first action value, the second action value and the discounted reward.

[0025] S35, updating the first value network and the second value network according to the TD error, and guiding the first policy network and the second policy network to update according to the updated first value network and the second value network.

[0026] Optionally, the adjusting of the admittance control parameters of the dual-arm robot by using the trained MDP model of the dual-arm robot satellite cooperative assembly in S4 is used to complete the satellite assembly task of the dual-arm robot, and the method comprises the following steps:

[0027] S41, acquiring left arm force sensor data and right arm force sensor data.

[0028] S42, performing gravity compensation and rigid body internal force transformation on the left arm force sensor data and the right arm force sensor data respectively to obtain left arm end contact force and right arm end contact force.

[0029] S43, inputting the left arm end contact force into the first policy network to obtain parameters of a left arm admittance controller; and inputting the right arm end contact force into the second policy network to obtain parameters of a right arm admittance controller.

[0030] S44, designing expected force of the left arm admittance controller and expected force of the right arm admittance controller.

[0031] S45, inputting the expected force of the left arm admittance controller and the left arm end contact force into the left arm admittance controller to obtain end motion speed of the left arm; and inputting the expected force of the right arm admittance controller and the right arm end contact force into the right arm admittance controller to obtain end motion speed of the right arm.

[0032] S46, sending the end motion speed of the left arm and the end motion speed of the right arm to the dual-arm robot to complete the satellite assembly task.

[0033] In another aspect, a dual-arm robot satellite assembly device based on multi-agent reinforcement learning is provided, which is applied to the dual-arm robot satellite assembly method based on multi-agent reinforcement learning, and the device comprises:

[0034] The system building module is configured to build a dual-arm robot satellite assembly system.

[0035] The model building module is configured to build an MDP model of dual-arm robot satellite assembly system.

[0036] The model training module is configured to train the MDP model of dual-arm robot satellite assembly system based on the MADDPG algorithm to obtain a trained MDP model of dual-arm robot satellite assembly system.

[0037] The output module is configured to adjust the admittance control parameters of the dual-arm robot using the trained MDP model of dual-arm robot satellite assembly system to complete the dual-arm robot satellite assembly task.

[0038] Optionally, the dual-arm robot satellite assembly system comprises a multi-degree-of-freedom dual-arm subsystem, a six-dimensional force sensor subsystem, a visual sensor subsystem, an end tool subsystem, a mobile chassis subsystem, and a multi-agent reinforcement learning subsystem.

[0039] Optionally, the state of the MDP model of dual-arm robot satellite assembly system is shown in the following formula (1):

[0040] (1)

[0041] In the formula, represents the state of the MDP model of dual-arm robot satellite assembly system; represents the force information in three directions of the set contact point of the left arm part , , represents the force information in three directions of the set contact point of the right arm part , , represents the torque information in two directions of the set contact point of the left arm part , represents the torque information in two directions of the set contact point of the right arm part ,

[0042] Optionally, the action of the MDP model of dual-arm robot satellite assembly system is shown in the following formula (2):

[0043] (2)

[0044] In the formula, ​​​​Left arm action of the MDP model representing dual-arm satellite cooperative assembly Right arm action of the MDP model representing dual-arm satellite cooperative assembly Left arm admittance controller , , Inertia parameters in three translational degrees of freedom , Inertia parameters in two rotational degrees of freedom Right arm admittance controller , , Inertia parameters in three translational degrees of freedom , Inertia parameters in two rotational degrees of freedom Left arm admittance controller , , Inertia parameters in three translational degrees of freedom , Damping parameters in two rotational degrees of freedom Right arm admittance controller , , Inertia parameters in three translational degrees of freedom , Damping parameters in two rotational degrees of freedom

[0045] Optionally, the reward function of the MDP model representing dual-arm satellite cooperative assembly is as shown in the following formula (3):

[0046] (3)

[0047] In the formula, Left arm reward function of the MDP model representing dual-arm satellite cooperative assembly Right arm reward function of the MDP model representing dual-arm satellite cooperative assembly Reward obtained when the insertion is successful Reward when the insertion fails Penalty when the contact force is too large Penalty when the contact force is out of range Step number of one insertion round Total step number of the current round Maximum torque value Minimum torque value Resultant force in y and z directions Force in x direction

[0048] Optionally, the model training module is further configured to:

[0049] S31, constructing an experience pool.

[0050] S32, obtaining any experience in the experience pool, predicting a next step state of the obtained experience by a first strategy network to obtain a next step action ; predicting a next step state of the obtained experience by a second strategy network to obtain a next step action .

[0051] S33, predicting a first action value by a first value network for the current action and state; predicting a second action value by a second value network for the next step action and state.

[0052] S34, calculating a TD target and a TD error according to the first action value, the second action value and the discounted reward.

[0053] S35, updating the first value network and the second value network according to the TD error, and updating the first strategy network and the second strategy network according to the updated first value network and the second value network.

[0054] Optionally, the output module is further configured to:

[0055] S41, obtaining left arm force sensor data and right arm force sensor data.

[0056] S42, respectively performing gravity compensation and rigid body internal force transformation on the left arm force sensor data and the right arm force sensor data to obtain left arm end contact force and right arm end contact force.

[0057] S43, inputting the left arm end contact force into the first strategy network to obtain parameters of a left arm admittance controller; inputting the right arm end contact force into the second strategy network to obtain parameters of a right arm admittance controller.

[0058] S44, designing expected force of the left arm admittance controller and expected force of the right arm admittance controller.

[0059] S45, inputting the expected force of the left arm admittance controller and the left arm end contact force into the left arm admittance controller to obtain end motion speed of the left arm; inputting the expected force of the right arm admittance controller and the right arm end contact force into the right arm admittance controller to obtain end motion speed of the right arm.

[0060] S46, sending the end motion speed of the left arm and the end motion speed of the right arm to the dual-arm robot to complete the satellite assembly task.

[0061] ​​In another aspect, a dual-arm robot satellite assembly device is provided, which comprises a processor, and a memory having computer readable instructions stored thereon, which, when executed by the processor, implement any one of the above-mentioned dual-arm robot satellite assembly methods based on multi-agent reinforcement learning.

[0062] In another aspect, a computer readable storage medium is provided, which stores at least one instruction loaded and executed by a processor to implement any one of the above-mentioned dual-arm robot satellite assembly methods based on multi-agent reinforcement learning.

[0063] The technical solutions provided by the embodiments of the present application have at least the following beneficial effects:

[0064] In the embodiments of the present application, four typical tasks in the dual-arm robot satellite assembly process can be implemented, i.e., a compliant control algorithm in dual-arm insertion, dual-arm screwing, dual-arm carrying, and dual-arm insertion, to realize compliant and coordinated assembly of satellite wings and satellite shell parts of a micro-satellite by the dual-arm robot, and to make further exploration for the efficiency and intelligence of satellite assembly. BRIEF DESCRIPTION OF DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0066] Figure 1 is a flow chart of a dual-arm robot satellite assembly method based on multi-agent reinforcement learning provided by the embodiments of the present application;

[0067] Figure 2 is a system composition schematic diagram provided by the embodiments of the present application;

[0068] Figure 3 is a dual-arm overall control strategy diagram provided by the embodiments of the present application;

[0069] Figure 4 is a reinforcement learning simulation training environment diagram provided by the embodiments of the present application;

[0070] Figure 5 is a dual-arm variable admittance control algorithm training block diagram based on the MADDPG algorithm provided by the embodiments of the present application;

[0071] Figure 6 is an execution block diagram of a dual-arm variable admittance control algorithm under a policy network provided by the embodiments of the present application;

[0072] Figure 7 is a double-arm robot satellite assembly device block diagram based on multi-agent reinforcement learning provided by an embodiment of the application.

[0073] Figure 8 is a structural schematic diagram of a double-arm robot satellite assembly device provided by an embodiment of the application. DETAILED DESCRIPTION

[0074] The technical solutions in the application will be described below with reference to the drawings.

[0075] In the embodiments of the application, the words such as “example”, “for example” and the like are used to represent an example, illustration or description. Any embodiment or design scheme described as “example” in the application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word “example” is intended to present the concept in a specific manner. In addition, in the embodiments of the application, the meaning expressed by “and / or” can be both, or can be one of the two.

[0076] In the embodiments of the application, “image” and “picture” can be used interchangeably at times, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized. “Of”, “corresponding” and “corresponding” can be used interchangeably at times, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized.

[0077] In the embodiments of the application, sometimes the subscript such as W1 can be written in the form of non-subscript such as W1, and the meanings expressed are consistent when the distinction is not emphasized.

[0078] To make the technical problems, technical solutions and advantages to be solved by the application more clear, specific embodiments will be described in detail below with reference to the drawings.

[0079] The embodiments of the application provide a double-arm robot satellite assembly method based on multi-agent reinforcement learning, which can be implemented by a double-arm robot satellite assembly device, and the double-arm robot satellite assembly device can be a terminal or a server. Figure 1 As shown in the flowchart of the double-arm robot satellite assembly method based on multi-agent reinforcement learning, the processing flow of the method can include the following steps:

[0080] S1, a double-arm robot satellite assembly system is built.

[0081] Optionally, as shown in Figure 2As shown in S1, the dual-arm robot satellite assembly system includes: a multi-degree-of-freedom dual-arm subsystem, a six-dimensional force sensor subsystem, a visual sensor subsystem, an end tool subsystem, a mobile chassis subsystem, and a multi-agent reinforcement learning subsystem.

[0082] In a feasible implementation, the multi-degree-of-freedom dual-arm subsystem can mainly follow a preset trajectory or make the end of the mechanical arm reach a desired position, which needs to be realized through robot inverse kinematics, and the corresponding joint angle is solved through inverse solution of the motion trajectory or known end pose.

[0083] The six-dimensional force sensor subsystem converts the displacement change into real-time force and torque through related algorithms, and has six-dimensional force / torque detection capability.

[0084] The visual sensor subsystem is composed of a stereo vision camera and can provide accurate depth perception when the object moves or the device moves.

[0085] The end tool subsystem is mainly responsible for grasping and fixing the satellite parts to be operated.

[0086] The mobile chassis subsystem is mainly responsible for moving the dual-arm robot to the specified position.

[0087] The multi-agent reinforcement learning subsystem adopts a force-based admittance control deep reinforcement learning strategy to model and generate strategies for each assembly subtask.

[0088] S2, establish a dual-arm satellite cooperative assembly MDP model based on the dual-arm robot satellite assembly system.

[0089] In a feasible implementation, as shown in S1, a reinforcement learning visualization simulation environment is established, the simulation environment realizes information interaction between the simulation environment and the agent, and a decision model of the assembly task is trained. Figure 3 , Figure 4 As shown in S1, a reinforcement learning visualization simulation environment is established, the simulation environment realizes information interaction between the simulation environment and the agent, and a decision model of the assembly task is trained.

[0090] Specifically, a Markov Decision Process (MDP) model of dual-arm satellite cooperative assembly is established, wherein the state is as shown in the following formula (1):

[0091] (1)

[0092] In the formula, represents the state of the MDP model of the dual-arm satellite cooperative assembly; represents the force information in three directions of the set contact point , , of the left arm part; represents the force information in three directions of the set contact point , , force information in three directions; represents a contact point set by the left arm part , torque information in two directions; represents a contact point set by the right arm part , torque information in two directions.

[0093] Optionally, the action of the dual-robot satellite cooperative assembly MDP model in S2: inertia parameters in five degrees of freedom , and damping parameters in five degrees of freedom , as shown in the following formula (2):

[0094] (2)

[0095] In the formula, represents the left arm action of the dual-robot satellite cooperative assembly MDP model; represents the right arm action of the dual-robot satellite cooperative assembly MDP model; represents the left arm admittance controller , , three translational degrees of freedom and , inertia parameters in two rotational degrees of freedom; represents the right arm admittance controller , , three translational degrees of freedom and , inertia parameters in two rotational degrees of freedom; represents the left arm admittance controller , , three translational degrees of freedom and , damping parameters in two rotational degrees of freedom; represents the right arm admittance controller , , three translational degrees of freedom and , damping parameters in two rotational degrees of freedom.

[0096] Optionally, the reward function is defined as follows:

[0097] (3)

[0098] wherein, , , represents the total number of steps in the current round, and a plug-in round is set to steps, and the plug-in is successful within 450 steps, and the plug-in fails beyond 450 steps, represents the reward obtained when the plug-in is successful, represents the reward when the plug-in fails. is the resultant force in the y-axis and z-axis directions, that is, the contact force between the shaft and the hole side wall. is the force in the x-axis direction, that is, the contact force between the front end surface of the shaft and the surface of the hole part or the bottom of the hole. When , the contact force between the shaft and the hole is too large, is the penalty when the contact force is too large. When , the contact force between the shaft and the hole is out of range, is the penalty when the contact force is out of range, represents the maximum torque value; represents the minimum torque value.

[0099] S3, training the MDP model of the dual-arm satellite cooperative assembly based on the MADDPG algorithm to obtain the trained MDP model of the dual-arm satellite cooperative assembly.

[0100] Optionally, the above step S3 can include the following steps S31-S35:

[0101] S31, constructing an experience pool.

[0102] S32, obtaining any experience in the experience pool, predicting the next step state of the obtained experience through a first policy network to obtain a next step action ; predicting the next step state of the obtained experience through a second policy network to obtain a next step action .

[0103] S33, predicting the current action and state through a first value network to obtain a first action value; predicting the next step action and state through a second value network to obtain a second action value.

[0104] S34, calculating the TD target and the TD error according to the first action value, the second action value, and the discount reward.

[0105] S35, updating the first value network and the second value network according to the TD error, and guiding the update of the first policy network and the second policy network according to the updated first value network and the second value network.

[0106] In a feasible implementation manner, as Figure 5As shown, a multi-agent reinforcement learning framework is established based on a Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm and a double-robot satellite cooperative assembly training is performed, where a centralized training and distributed application framework is adopted, that is, the input state of the value network and the policy network is a global state, and the output is a local value and a local policy.

[0107] The following describes the update of the real value network and the real policy network. During the algorithm running, the experience pool is collected through the executed part, and when the experience storage in the experience pool reaches a certain amount, the training and update of the policy network and the value network are started. For an experience in the experience pool, two policy networks are used to predict the next state and , to obtain the next action and ; secondly, the value network is used to predict the current action and state and the next action and state to obtain two action values. Then, the TD target and the TD error are calculated using the discount reward and the action value. Finally, the value network Q1 and Q2 are updated according to the TD error, and the policy network is updated using the value network.

[0108] S4, adjusting the admittance control parameters of the double-robot satellite cooperative assembly MDP model to complete the satellite assembly task of the double-robot.

[0109] Optionally, the above step S4 can include the following steps S41-S46:

[0110] S41, acquiring left arm force sensor data and right arm force sensor data.

[0111] S42, respectively performing gravity compensation and rigid body internal force transformation on the left arm force sensor data and the right arm force sensor data to obtain left arm end contact force and right arm end contact force.

[0112] S43, inputting the left arm end contact force into a first policy network to obtain parameters of a left arm admittance controller, and inputting the right arm end contact force into a second policy network to obtain parameters of a right arm admittance controller.

[0113] S44, designing expected force of the left arm admittance controller and expected force of the right arm admittance controller.

[0114] S45, inputting the expected force of the left arm admittance controller and the left arm end contact force into the left arm admittance controller to obtain the end motion speed of the left arm, and inputting the expected force of the right arm admittance controller and the right arm end contact force into the right arm admittance controller to obtain the end motion speed of the right arm.

[0115] S46, the end motion speed of the left arm and the end motion speed of the right arm are sent to the dual-arm robot, and the satellite assembly task is completed.

[0116] In a feasible implementation, the admittance parameters of the dual-arm robot are adjusted by using the policy network to enable the dual-arm robot to perform variable admittance compliant control. Figure 6 As shown in the figure, first, the left arm force sensor data and the right arm force sensor data are obtained from the robot operating platform; second, the data of the left and right arm force sensors are respectively subjected to gravity compensation of the clamp and the object clamped thereby, and rigid body internal force transformation, so as to obtain the left arm end contact force and the right arm end contact force. Then, the left arm end contact force is input to the policy network , to obtain the parameters and of the left arm admittance controller. The right arm end contact force is input to the policy network , to obtain the parameters and of the right arm admittance controller. Finally, the expected forces of the left and right arm admittance controllers are respectively designed according to the task conditions, and the left and right arm end contact forces are respectively input to the left and right arm admittance controllers, so that the end motion speeds of the left and right arms are obtained and sent to the robot to perform variable admittance compliant operation.

[0117] Further, the trained decision model is migrated to a dual-arm robot satellite assembly system physical system, so that the dual-arm cooperative satellite assembly task is realized.

[0118] The present application adopts a multi-agent reinforcement learning algorithm to dynamically optimize the admittance control parameters under the coordination of the dual arms, so that each contact force in the assembly process meets the assembly requirements, including the loose coordination task of satellite wing part jack plug screwing and the tight coordination task of satellite shell part carrying and plugging.

[0119] In the embodiment of the present application, the compliant control algorithm in the four typical tasks of the dual-arm robot in the micro-satellite assembly process, i.e., dual-arm jack plug, dual-arm screwing, dual-arm carrying, and dual-arm plugging, can be realized, so as to realize the compliant and coordinated assembly of the satellite wing and satellite shell parts of the micro-satellite by the dual-arm robot, and further explore the efficiency and intelligence of satellite assembly.

[0120] Figure 7 is a dual-arm robot satellite assembly device based on multi-agent reinforcement learning according to an exemplary embodiment, which is used for a dual-arm robot satellite assembly method based on multi-agent reinforcement learning. Referring to Figure 7 , the device includes a system building module 310, a model establishing module 320, a model training module 330, and an output module 340. Among them:

[0121] System building module 310 is used to build a dual-arm robot satellite assembly system.

[0122] Model building module 320 is used to build an MDP model of dual-arm satellite collaborative assembly based on the dual-arm robot satellite assembly system.

[0123] The model training module 330 is used to train the MDP model of dual-robotic arm satellite collaborative assembly based on the MADDPG algorithm to obtain the trained MDP model of dual-robotic arm satellite collaborative assembly.

[0124] Output module 340 is used to adjust the admittance control parameters of the dual-arm robot using the trained MDP model of dual-arm satellite collaborative assembly, so as to complete the dual-arm robot satellite assembly task.

[0125] Optionally, the dual-arm robot satellite assembly system includes: a multi-degree-of-freedom dual-manipulator subsystem, a six-dimensional force sensor subsystem, a vision sensor subsystem, an end-effector subsystem, a mobile chassis system, and a multi-agent reinforcement learning subsystem.

[0126] Optionally, the state of the MDP model for dual-robotic-arm satellite collaborative assembly is shown in equation (1) below:

[0127] (1)

[0128] In the formula, This indicates the state of the MDP model for collaborative satellite assembly with dual robotic arms. Indicates the contact point set for the left arm component. , , Force information in three directions; Indicates the contact point set for the right arm component. , , Force information in three directions; Indicates the contact point set for the left arm component. , Torque information in both directions; Indicates the contact point set for the right arm component. , Torque information in both directions.

[0129] Optionally, the actions of the MDP model for dual-arm satellite collaborative assembly are as shown in equation (2):

[0130] (2)

[0131] In the formula, Left arm action of the MDP model representing dual-arm satellite cooperative assembly Right arm action of the MDP model representing dual-arm satellite cooperative assembly Left arm admittance controller , , Inertia parameters in three translational degrees of freedom , Inertia parameters in two rotational degrees of freedom Right arm admittance controller , , Inertia parameters in three translational degrees of freedom , Inertia parameters in two rotational degrees of freedom Left arm admittance controller , , Inertia parameters in three translational degrees of freedom , Damping parameters in two rotational degrees of freedom Right arm admittance controller , , Inertia parameters in three translational degrees of freedom , Damping parameters in two rotational degrees of freedom

[0132] Optionally, the reward function of the MDP model representing dual-arm satellite cooperative assembly is as shown in the following formula (3):

[0133] (3)

[0134] In the formula, Left arm reward function of the MDP model representing dual-arm satellite cooperative assembly Right arm reward function of the MDP model representing dual-arm satellite cooperative assembly Reward obtained when the insertion is successful Reward when the insertion fails Penalty when the contact force is too large Penalty when the contact force is out of range Step number of one insertion round Total step number of the current round Maximum torque value Minimum torque value Resultant force in y and z directions Force in x direction

[0135] Optionally, the model training module 330 is further configured to:

[0136] S31, construct an experience pool.

[0137] S32, obtain any experience in the experience pool, predict the next step state of the obtained experience by the first strategy network to obtain the next step action ; predict the next step state of the obtained experience by the second strategy network to obtain the next step action .

[0138] S33, predict the current action and state by the first value network to obtain the first action value; predict the next step action and state by the second value network to obtain the second action value.

[0139] S34, calculate the TD target and the TD error according to the first action value, the second action value and the discount reward.

[0140] S35, update the first value network and the second value network according to the TD error, and guide the update of the first strategy network and the second strategy network according to the updated first value network and the second value network.

[0141] Optionally, the output module 340 is further configured to:

[0142] S41, obtain left arm force sensor data and right arm force sensor data.

[0143] S42, respectively perform gravity compensation and rigid body internal force transformation on the left arm force sensor data and the right arm force sensor data to obtain left arm end contact force and right arm end contact force.

[0144] S43, input the left arm end contact force to the first strategy network to obtain the parameters of the left arm admittance controller; input the right arm end contact force to the second strategy network to obtain the parameters of the right arm admittance controller.

[0145] S44, design the expected force of the left arm admittance controller and the expected force of the right arm admittance controller.

[0146] S45, input the expected force of the left arm admittance controller and the left arm end contact force to the left arm admittance controller to obtain the end motion speed of the left arm; input the expected force of the right arm admittance controller and the right arm end contact force to the right arm admittance controller to obtain the end motion speed of the right arm.

[0147] S46, send the end motion speed of the left arm and the end motion speed of the right arm to the dual-arm robot to complete the satellite assembly task.

[0148] In the embodiment of the present application, four typical tasks of the dual-arm robot in the micro-satellite assembly process can be realized: dual-arm insertion, dual-arm screwing, dual-arm carrying, and compliant control algorithm in dual-arm insertion, to realize compliant and coordinated assembly of the satellite wing and satellite shell parts of the micro-satellite by the dual-arm robot, and to make further exploration for the efficiency and intelligence of satellite assembly.

[0149] Figure 8 is a structural schematic diagram of a dual-arm robot satellite assembly device provided by the embodiment of the present application, as shown in Figure 8 The dual-arm robot satellite assembly device can include the dual-arm robot satellite assembly device based on multi-agent reinforcement learning shown in Figure 7 Optionally, the dual-arm robot satellite assembly device 410 can include the first processor 2001.

[0150] Optionally, the dual-arm robot satellite assembly device 410 can further include the memory 2002 and the transceiver 2003.

[0151] The first processor 2001, the memory 2002, and the transceiver 2003 can be connected through a communication bus.

[0152] The various constituent components of the dual-arm robot satellite assembly device 410 will be specifically introduced below: Figure 8

[0153] The first processor 2001 is the control center of the dual-arm robot satellite assembly device 410, and can be one processor or a plurality of processing elements. For example, the first processor 2001 is one or more central processing units (CPU), or is an application specific integrated circuit (ASIC), or is one or more integrated circuits configured to implement the embodiment of the present application, such as one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA).

[0154] Optionally, the first processor 2001 can execute various functions of the dual-arm robot satellite assembly device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0155] In a specific implementation, as an embodiment, the first processor 2001 can include one or more CPUs, for example Figure 8 ​CPU0 and CPU1 shown in FIG. 1.

[0156] In a specific implementation, as an embodiment, the dual-arm robot satellite assembly device 410 can also include multiple processors, for example Figure 8 the first processor 2001 and the second processor 2004 shown in FIG. 2. Each of these processors can be a single-CPU or a multi-CPU. The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (for example, computer program instructions).

[0157] The memory 2002 is configured to store a software program for implementing the scheme of the present application, and the first processor 2001 is configured to control the execution. For specific implementation, refer to the above-mentioned method embodiments, which will not be repeated here.

[0158] Optionally, the memory 2002 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, an optical disk storage (including a compact disk, a laser disk, an optical disk, a digital versatile disk, a Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but not limited to this. The memory 2002 can be integrated with the first processor 2001, or can exist independently and be coupled with the first processor 2001 through an interface circuit (not shown in FIG. 2) of the dual-arm robot satellite assembly device 410, and the embodiments of the present application are not limited in this regard. Figure 8

[0159] The transceiver 2003 is configured to communicate with a network device or a terminal device.

[0160] Optionally, the transceiver 2003 can include a receiver and a transmitter (not shown separately in FIG. 2). The receiver is configured to implement a receiving function, and the transmitter is configured to implement a transmitting function. Figure 8

[0161] ​​Optionally, the transceiver 2003 can be integrated with the first processor 2001, or exist independently, and is coupled with the first processor 2001 through an interface circuit (not shown in the figure) of the dual-arm robot satellite assembly device 410, and the embodiments of the present application do not make a specific limitation thereon. Figure 8

[0162] It should be noted that, Figure 8 The structure of the dual-arm robot satellite assembly device 410 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure identification device can include more or fewer components than those shown in the figure, or combine certain components, or different component arrangements.

[0163] In addition, the technical effects of the dual-arm robot satellite assembly device 410 can refer to the technical effects of the dual-arm robot satellite assembly method based on multi-agent reinforcement learning described in the above method embodiments, which will not be repeated here.

[0164] It should be understood that the first processor 2001 in the embodiments of the present application can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), ready programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0165] ​It should also be understood that the memory in the embodiments of the present application can be volatile or nonvolatile memory, or can include both volatile and nonvolatile memory. The nonvolatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), or flash memory. The volatile memory can be random access memory (RAM) used as external cache. By way of example, and not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0166] The above-described embodiments can be implemented in whole or in part by software, hardware (such as a circuit), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through a wired (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state disk.

[0167] It should be understood that the term "and / or" herein merely describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In addition, the character " / " herein generally represents that the associated objects before and after it are in an "or" relationship, but it can also represent an "and / or" relationship, which can be understood according to the context before and after it.

[0168] In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or the like means any combination of the items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0169] It should be understood that in various embodiments of the present application, the size of the sequence number of the above-described processes does not mean the order of execution, and the execution order of the processes should be determined according to their functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0170] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0171] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the devices, apparatuses and units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0172] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0173] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0174] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit.

[0175] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0176] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A dual-arm robot satellite assembly method based on multi-agent reinforcement learning, characterized by, The method comprises: S1, building a dual-arm robot satellite assembly system; S2, establishing a dual-robot satellite cooperative assembly MDP model according to the dual-arm robot satellite assembly system; S3, training the dual-robot satellite cooperative assembly MDP model based on the MADDPG algorithm to obtain a trained dual-robot satellite cooperative assembly MDP model; S4, adjusting the admittance control parameters of the dual-arm robot using the trained dual-robot satellite cooperative assembly MDP model to complete the dual-arm robot satellite assembly task. The state of the dual-robot satellite cooperative assembly MDP model in S2 is shown in the following formula (1): In the formula, S represents the state of the MDP model of the dual-arm satellite cooperative assembly; represents the force information in the x, y, z directions of the contact point set for the left-arm part; represents the force information in the x, y, z directions of the contact point set for the right-arm part; represents the moment information in the y, z directions of the contact point set for the left-arm part; represents the moment information in the y, z directions of the contact point set for the right-arm part; The action of the dual-robot satellite cooperative assembly MDP model in S2 is shown in the following formula (2): In the formula, A L represents the left arm action of the MDP model of the dual-robot satellite cooperative assembly; A R represents the right arm action of the MDP model of the dual-robot satellite cooperative assembly; represents the inertia parameters of the left arm admittance controller in the x, y, z three movement degrees of freedom and the y, z two rotation degrees of freedom; represents the inertia parameters of the right arm admittance controller in the x, y, z three movement degrees of freedom and the y, z two rotation degrees of freedom; represents the damping parameters of the left arm admittance controller in the x, y, z three movement degrees of freedom and the y, z two rotation degrees of freedom; represents the damping parameters of the right arm admittance controller in the x, y, z three movement degrees of freedom and the y, z two rotation degrees of freedom.

2. The dual-arm robot satellite assembly method based on multi-agent reinforcement learning according to claim 1, wherein, The dual-arm robot satellite assembly system in S1 comprises a multi-degree-of-freedom dual-arm subsystem, a six-dimensional force sensor subsystem, a visual sensor subsystem, an end tool subsystem, a mobile chassis subsystem, and a multi-agent reinforcement learning subsystem.

3. The dual-arm robot satellite assembly method based on multi-agent reinforcement learning according to claim 1, wherein, The reward function of the dual-robot satellite cooperative assembly MDP model in S2 is shown in the following formula (3): In the formula, R L The left arm reward function of the MDP model of the dual-arm satellite cooperative assembly; R R The right arm reward function of the MDP model of the dual-arm satellite cooperative assembly; R done The reward obtained when the insertion is successful; R fail The reward when the insertion fails; R max The penalty when the contact force is too large; R out The penalty when the contact force exceeds the range; K represents the number of steps of an insertion round; k represents the total number of steps of the current round; F max The maximum torque value; F limit The minimum torque value; F edge The resultant force in the y-axis and z-axis directions; F x The force in the x-axis direction.

4. The dual-arm robot satellite assembly method based on multi-agent reinforcement learning according to claim 1, wherein, The training of the dual-robot satellite cooperative assembly MDP model based on the MADDPG algorithm in S3 to obtain a trained dual-robot satellite cooperative assembly MDP model comprises: S31, constructing an experience pool; S32, obtaining any experience in the experience pool, predicting the next state s1 of the obtained experience through a first policy network to obtain the next action a1', and predicting the next state s2 of the obtained experience through a second policy network to obtain the next action a2'; S33, predicting the current action and state through a first value network to obtain a first action value, and predicting the next action and state through a second value network to obtain a second action value; S34, calculating a TD target and a TD error according to the first action value, the second action value, and the discounted reward; S35, updating the first value network and the second value network according to the TD error, and guiding the update of the first policy network and the second policy network according to the updated first value network and the second value network.

5. The dual-arm robot satellite assembly method based on multi-agent reinforcement learning of claim 1, wherein, The use of the trained dual-robot satellite cooperative assembly MDP model to adjust the admittance control parameters of the dual-arm robot to complete the dual-arm robot satellite assembly task in S4 comprises: S41, obtaining left arm force sensor data and right arm force sensor data; S42, performing gravity compensation and rigid body internal force transformation on the left arm force sensor data and the right arm force sensor data respectively to obtain left arm end contact force and right arm end contact force; S43, inputting the left arm end contact force into a first policy network to obtain the parameters of a left arm admittance controller, and inputting the right arm end contact force into a second policy network to obtain the parameters of a right arm admittance controller; S44, designing the expected force of the left arm admittance controller and the expected force of the right arm admittance controller; S45, input the expected force of the left arm admittance controller and the left arm end contact force into the left arm admittance controller to obtain the end motion speed of the left arm; input the expected force of the right arm admittance controller and the right arm end contact force into the right arm admittance controller to obtain the end motion speed of the right arm; S46, send the end motion speed of the left arm and the end motion speed of the right arm to the dual-arm robot to complete the satellite assembly task.

6. A multi-agent reinforcement learning based dual-arm robot satellite assembly apparatus for implementing the multi-agent reinforcement learning based dual-arm robot satellite assembly method according to any one of claims 1-5, characterized in that, The device comprises: a system building module, configured to build a dual-arm robot satellite assembly system; a model establishing module, configured to establish a dual-robot satellite collaborative assembly MDP model according to the dual-arm robot satellite assembly system; a model training module, configured to train the dual-robot satellite collaborative assembly MDP model based on a MADDPG algorithm to obtain a trained dual-robot satellite collaborative assembly MDP model; an output module, configured to adjust the admittance control parameters of the dual-arm robot by using the trained dual-robot satellite collaborative assembly MDP model to complete a dual-arm robot satellite assembly task.

7. A dual-arm robotic satellite assembly apparatus, characterized by, The dual-arm robot satellite assembly device comprises: a processor; a memory, wherein the memory stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the method in any one of claims 1 to 5.

8. A computer readable storage medium, characterized in that, The computer readable storage medium stores program codes, and the program codes can be called and executed by the processor to implement the method in any one of claims 1 to 5.

Images