An intelligent cooperative control method and system for a spatial multi-arm robot to capture non-cooperative targets

Abstract

An intelligent cooperative control method and system for a spatial multi-arm robot to capture non-cooperative targets, which relates to the field of intelligent capture control of spatial dual-arm robots. It solves the problem that classical control methods show insufficient adaptability when dealing with complex robot nonlinear systems and uncertain or dynamically changing task environments. The method includes: constructing the state space and action space of the spatial dual-arm robot; constructing a reward function for the spatial dual-arm robot to capture the target along the desired trajectory according to the state space and action space of the spatial dual-arm robot; establishing a neural network model based on DDPG according to the reward function; training the neural network model based on DDPG to obtain an optimized model; and controlling the spatial dual-arm robot to capture the non-cooperative target according to the optimized model. The present invention is applied to the aerospace field.

Description

Technical Field

[0001] The present invention relates to the field of intelligent capture control of space dual-arm robots, and particularly to an intelligent cooperative control method for a space multi-arm robot to capture a non-cooperative target. Background Art

[0002] During the process of a space multi-arm robot executing a task, it is necessary to plan the motion trajectory of each arm according to the requirements of the task, so as to reach the desired and required pose. Since the tasks undertaken by space multi-arm robots are generally more complex and usually involve the mutual cooperation between each arm, and interference will also occur between multiple manipulators during cooperative operation. Therefore, compared with traditional single-arm robots, the performance requirements for the controller of space multi-arm robots are higher. On the other hand, due to the more complex dynamic model of space multi-arm robots, there are more external interferences during the execution of tasks. For example, a closed-loop system is easily formed during operation, and the collision and coupling problems between arms also pose challenges to the design of the controller.

[0003] In the field of manipulator control, traditional control methods include PD control, computed torque method, sliding mode control, and adaptive control, etc. These methods have been widely studied and have achieved remarkable results in the application of manipulators.

[0004] PD control refers to proportional-derivative control. The PD controller provides a control signal according to the difference value between the desired state and the current state, and has a wide range of applications. This control method essentially belongs to feedback regulation and has hysteresis. It is suitable for control systems with low speed, low precision, and small-range motion, and has been widely used in the fields of manipulators and robot control. The PD control law can be used to achieve the fixed-point control or trajectory tracking control of a robot. Its advantages are simple and easy to implement, and it does not depend on dynamic modeling. Its disadvantages are that it cannot eliminate the steady-state error, is sensitive to external interferences, and requires repeated parameter adjustment when dealing with different models or unknown models.

[0005] The computed torque method is a control method that considers the dynamic model of the controlled object. It calculates the desired joint torque according to the dynamic equation through the error between the desired and the actual to make the robot track the given trajectory or position. However, the computed torque method is based on the known dynamic model or the case where the error is small. When the dynamic model is unknown or the dynamic model is a time-varying system, the control effect of the computed torque method will be very poor.

[0006] With the expansion of the application field of robots and the complexity of the structure design of robots themselves, the limitations of classical control theory have gradually emerged. Classical control methods show insufficient adaptability when dealing with complex robot nonlinear systems and uncertain or dynamically changing task environments. Summary of the Invention

[0007] In view of the problem that classical control methods show insufficient adaptability when dealing with complex robot nonlinear systems and uncertain or dynamically changing task environments, an intelligent cooperative control method for a space multi-arm robot to capture non-cooperative targets is proposed. The specific solution is as follows:

[0008] An intelligent cooperative control method for a space multi-arm robot to capture non-cooperative targets, the method comprising:

[0009] Construct the state space and action space of the space two-arm robot;

[0010] Construct a reward function for the space two-arm robot to capture the target along the desired trajectory according to the state space and action space of the space two-arm robot;

[0011] Establish a neural network model based on DDPG according to the reward function;

[0012] Train the neural network model based on DDPG to obtain an optimized model;

[0013] Control the space two-arm robot to capture the non-cooperative target according to the optimized model.

[0014] Furthermore, a preferred method is also provided. The construction of the state space and action space of the space two-arm robot includes:

[0015] The dimension of the state quantity of the state space and the dimension of the action value of the action space:

[0016] The dimension of the state quantity of the state space is 39, including: the position, attitude angle, velocity and angular velocity of the base on the x, y, and z axes, the joint angles and joint angular velocities of the robotic arms, and the velocity and angular velocity of the non-cooperative target;

[0017] The dimension of the action value of the action space is 18, including: the control forces and control torques of the base on the x, y, and z axes, and 12 joint control torques.

[0018] Furthermore, a preferred method is also provided. The construction of the reward function for the space two-arm robot to capture the target along the desired trajectory according to the state space and action space of the space two-arm robot includes:

[0019] R = -e tv -e tω ,

[0020] where e represents the error between the expected value and the true value, e tv is the velocity error of the non-cooperative target, and e tω is the angular velocity error.

[0021] Furthermore, a preferred method is also provided. Establishing a neural network model based on DDPG according to the reward function includes:

[0022] S1: Construct a policy network and a value function network;

[0023] S2: Use an experience buffer to store the experiences of the neural network model. The experiences include the robot state, action, reward, and next state information during environment interaction;

[0024] S3: Use the policy network to generate actions and improve the policy according to the evaluation results of the value function network;

[0025] S4: Use the value function network to calculate the gradient of the loss function and use the gradient descent algorithm to update the parameters of the policy network and the value function network;

[0026] S5: Repeat steps S2 to S4, use the experiences in the experience buffer and the gradients to update the neural network model, and obtain the neural network model based on DDPG.

[0027] Furthermore, a preferred method is also provided. Using the gradient descent algorithm to update the parameters of the policy network and the value function network includes:

[0028] The loss function L of the policy network a is:

[0029] L a =-q π (s,a),

[0030] where (s,a) represents the current state value and action value, and q π is the value function;

[0031] The loss function L of the value function network c is:

[0032] L c =(R+γq π (s',a')-q π (s,a)) 2 ,

[0033] where R represents the reward value at the current moment, (s',a') represents the next state value and action value, and γ represents the discount factor;

[0034] Calculate the gradient of the loss function with respect to the parameters of the policy network and the gradient of the value function network

[0035]

[0036]

[0037] Among them, is the gradient of the policy network parameters, is the gradient of the value function network.

[0038] Update the parameters using the gradient descent strategy:

[0039]

[0040] Among them, α is the learning rate, is the gradient of the network parameters, and θ represents the network parameters.

[0041] Furthermore, a preferred method is also provided. The method of controlling the space dual-arm robot to capture a non-cooperative target according to the optimization model includes:

[0042] The space dual-arm robot obtains the motion state of the non-cooperative target,

[0043] The optimization model generates a control strategy for the space dual-arm robot according to the motion state of the non-cooperative target;

[0044] The space dual-arm robot executes corresponding actions according to the control strategy, interacts with the non-cooperative target and the environment, and obtains environmental information;

[0045] Adjust the control strategy of the optimization model according to the environmental information to form closed-loop training.

[0046] Furthermore, a preferred method is also provided. The termination conditions of the single closed-loop training include:

[0047] The total control duration exceeds 9 s or the non-cooperative target collides with the space dual-arm robot.

[0048] Based on the same inventive concept, the present invention also provides an intelligent cooperative control system for a space multi-arm robot to capture a non-cooperative target. The system includes:

[0049] A space construction unit for constructing the state space and action space of the space dual-arm robot;

[0050] A reward function construction unit for constructing a reward function for the space dual-arm robot to capture a target along an expected trajectory according to the state space and action space of the space dual-arm robot;

[0051] A neural network model establishment unit for establishing a neural network model based on DDPG according to the reward function;

[0052] A training unit for training the neural network model based on DDPG to obtain an optimization model;

[0053] A capture unit for controlling a space dual-arm robot to capture a non-cooperative target according to the optimization model.

[0054] Based on the same inventive concept, the present invention also provides a computer-readable storage medium for storing a computer program, and the computer program executes the intelligent cooperative control method for a space multi-arm robot to capture a non-cooperative target described in any one of the above.

[0055] Based on the same inventive concept, the present invention also provides a computer device including a memory and a processor. A computer program is stored in the memory, and when the processor runs the computer program stored in the memory, the processor executes the intelligent cooperative control method for a space multi-arm robot to capture a non-cooperative target described in any one of the above.

[0056] The beneficial effects of the present invention are as follows:

[0057] The present invention solves the problem that the classical control method shows insufficient adaptability when dealing with complex robot nonlinear systems and uncertain or dynamically changing task environments.

[0058] The intelligent cooperative control method for a space multi-arm robot to capture a non-cooperative target according to the present invention can better describe the states of the robot and the environment and the actions that the robot can take by constructing a state space and an action space, thereby enhancing the modeling ability for complex nonlinear systems and uncertain or dynamically changing task environments; in reinforcement learning, the reward function plays a role in guiding the behavior of the robot. By constructing a reward function according to the desired trajectory, the present invention can formalize the task objective into an optimization problem, provide a more flexible and more adaptable reward signal for uncertain or dynamically changing environments, and this adaptive reward function can provide more accurate feedback to help the robot adjust its strategy to adapt to complex nonlinear systems and dynamic environments; traditional classical control methods are usually based on linear or nonlinear models and require accurate knowledge of system dynamics. However, in complex robot nonlinear systems, it is difficult to accurately model. The neural network model based on DDPG of the present invention uses a deep neural network to approximate the deterministic policy and value function, can better adapt to complex nonlinear systems, and reduce the dependence on the system model, which enables the robot to learn from experience and flexibly adjust its strategy according to changes in the environment; the present invention uses the DDPG algorithm to train and optimize the neural network model, and continuously collects sample data for learning through interaction with the environment. This process is iterative, and the robot can continuously improve its strategy through techniques such as experience replay and gradient descent. Compared with classical control methods, this training and optimization process is more adaptable to uncertain or dynamically changing task environments and can adaptively improve the control strategy of the robot.

[0059] An intelligent cooperative control method for a spatial multi-arm robot to capture non-cooperative targets. By using the DDPG algorithm based on deep reinforcement learning, it can effectively solve the problem of insufficient adaptability of classical control methods in dealing with complex robot nonlinear systems and uncertain or dynamically changing task environments. The present invention provides more flexible and adaptable modeling capabilities and allows the robot to learn from experience and optimize its strategies, thus better coping with the challenges of complex robot systems and dynamic environments.

[0060] The present invention provides an intelligent cooperative control method for a spatial multi-arm robot to capture non-cooperative targets, introducing a spatial two-arm robot controller based on DDPG. A dynamic environment model for a spatial two-arm robot to capture non-cooperative targets is constructed through a dynamic machine learning training platform. In the case where the motion state of the non-cooperative target is random, the reinforcement learning DDPG algorithm is used to solve the two-arm cooperative control problem, reducing the jitter problem generated during the two-arm cooperative capture of non-cooperatives and improving the control accuracy.

[0061] The present invention is applied to the aerospace field. Description of the Drawings

[0062] Figure 1 Flowchart of an intelligent cooperative control method for a spatial multi-arm robot to capture non-cooperative targets as described in Embodiment 1;

[0063] Figure 2 Frame structure diagram of the reinforcement learning as described in Embodiment 11;

[0064] Figure 3 Flowchart of the DDPG algorithm as described in Embodiment 11;

[0065] Figure 4 Schematic diagram of the training process as described in Embodiment 11;

[0066] Figure 5 Schematic diagram of the reward value during the training process of the original reward function as described in Embodiment 11;

[0067] Figure 6 Schematic diagram of the reward value during the training process of the improved reward function as described in Embodiment 11;

[0068] Figure 7 Schematic diagram of the comparison between the improved and unimproved reward functions as described in Embodiment 11;

[0069] Figure 8 Schematic diagram of the target motion state as described in Embodiment 11, where Figure 8 (a) is the target velocity, Figure 8 (b) is the target angular velocity. Detailed Description of the Invention

[0070] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention.

[0071] Embodiment 1. Refer to Figure 1 This embodiment will be described. An intelligent cooperative control method for a spatial multi-arm robot to capture a non-cooperative target according to this embodiment, the method includes:

[0072] Construct the state space and action space of the spatial two-arm robot;

[0073] Construct a reward function for the spatial two-arm robot to capture the target along the desired trajectory according to the state space and action space of the spatial two-arm robot;

[0074] Establish a neural network model based on DDPG according to the reward function;

[0075] Train the neural network model based on DDPG to obtain an optimized model;

[0076] Control the spatial two-arm robot to capture the non-cooperative target according to the optimized model.

[0077] In this embodiment, by constructing the state space and action space, the spatial two-arm robot can adapt to various uncertain or dynamically changing situations. The state space can include information such as the positions, speeds, and accelerations of the robot and the target, while the action space can define the motion instructions of the robot. This enables the robot to flexibly adjust and optimize the control strategy according to the changes in the environment.

[0078] In this embodiment, by establishing a reward function for capturing the target according to the state space and the action space, the task objective can be formalized as an optimization problem. The reward function can be adjusted according to the uncertainty and dynamic changes of the task, enabling the robot to find the optimal action strategy in different environments. The use of this adaptive reward function enables the robot to quickly adapt and optimize its behavior. By establishing a DDPG model, a neural network is used to approximate the policy and value function, and training is performed to obtain an optimized model. The DDPG algorithm combines the deterministic policy gradient method and the experience replay technique. For an uncertain or dynamically changing task environment, it can learn a more adaptable policy and can handle the challenges of continuous action spaces. By training a neural network model based on DDPG, iterative optimization and gradual improvement of the robot's control strategy can be carried out. In each iteration, the robot can collect data by interacting with the environment, and then use techniques such as experience replay and gradient descent to train the network. This iterative optimization process enables the robot to gradually improve and adapt to the dynamic changes of the task environment.

[0079] The method described in this embodiment has the advantages of flexibility, adaptability, an adaptive reward function, DDPG-based training, and iterative optimization when the space dual-arm robot is tracking in an uncertain or dynamically changing task environment, enabling the robot to effectively respond to changes in the task environment and achieve the capture of non-cooperative targets.

[0080] Embodiment 2: This embodiment further defines an intelligent cooperative control method for a space multi-arm robot to capture non-cooperative targets described in Embodiment 1. The construction of the state space and action space of the space dual-arm robot includes:

[0081] The dimension of the state quantity of the state space and the dimension of the action value of the action space:

[0082] The dimension of the state quantity of the state space is 39, including: the position, attitude angle, velocity, and angular velocity of the base on the x, y, and z axes, the joint angles and joint angular velocities of the manipulator, and the velocity and angular velocity of the non-cooperative target;

[0083] The dimension of the action value of the action space is 18, including: the control force and control torque of the base on the x, y, and z axes, and 12 joint control torques.

[0084] By increasing the dimension of the state space and including more state quantities, more detailed and accurate state information can be provided, which helps to enhance the robot's modeling ability for complex nonlinear systems and dynamic environments. In this way, the robot can better perceive environmental changes and its own state changes, and make more accurate decision-making and control operations.

[0085] By expanding the dimension of the action space, more control options and flexibility are provided for the robot. The robot can apply appropriate forces and torques on different degrees of freedom as needed to achieve fine control and motion performance. Such a design of the action space helps to improve the mobility and adaptability of the robot, enabling it to flexibly respond to various task requirements in complex non-linear systems and dynamic environments.

[0086] The design of the state space and the action space aims to provide more comprehensive and accurate state information, and to provide more control options and flexibility. Such a design helps to solve the problem of insufficient adaptability shown by classical control methods when dealing with complex non-linear robot systems and uncertain or dynamically changing task environments, and improves the robot's autonomous decision-making and control capabilities.

[0087] Embodiment 3: This embodiment further limits the intelligent cooperative control method for a spatial multi-arm robot to capture a non-cooperative target described in Embodiment 1. The construction of the reward function for the spatial two-arm robot to capture the target along the desired trajectory according to the state space and action space of the spatial two-arm robot includes:

[0088] R = -e tv -e tω ,

[0089] where e represents the error between the expected value and the true value, e tv is the velocity error of the non-cooperative target, and e tω is the angular velocity error.

[0090] The purpose of the reward function in this embodiment is to guide the robot to optimize its behavior during the task execution, enabling it to approach or reach the desired target trajectory as closely as possible.

[0091] Embodiment 4: This embodiment further limits the intelligent cooperative control method for a spatial multi-arm robot to capture a non-cooperative target described in Embodiment 1. The establishment of the neural network model based on DDPG according to the reward function includes:

[0092] S1: Construct a policy network and a value function network;

[0093] S2: Use an experience buffer to store the experiences of the neural network model. The experiences include the robot state, action, reward, and next state information during the environmental interaction;

[0094] S3: Generate actions using the policy network and improve the policy according to the evaluation results of the value function network;

[0095] S4: Calculate the loss function gradient using the value function network and use the gradient descent algorithm to update the parameters of the policy network and the value function network;

[0096] S5: Repeat steps S2 to S4, update the neural network model using the experiences and gradients in the experience buffer, and obtain the neural network model based on DDPG.

[0097] This embodiment can use a deep neural network to model high-dimensional state and action spaces, providing a more powerful function fitting ability to handle complex tasks and environments. The policy network is used to generate actions, and the value function network is used to evaluate the value of actions. The combination of these two networks can provide the direction and guidance for policy improvement and effectively optimize the behavior of the robot. By storing experiences, including information such as state, action, reward, and next state, the diversity and efficiency of training samples can be increased, avoiding real-time calculation for each sample. With the evaluation results of the value function network, the policy network can select better actions according to the current state. Through policy improvement, the performance of the policy can be gradually improved during training. By calculating the gradient of the loss function of the value function network and using the gradient descent algorithm, the parameters of the policy network and the value function network can be updated, continuously optimizing the network model and gradually improving the performance.

[0098] This embodiment uses a neural network model based on DDPG to learn and optimize the robot's policy, enabling it to continuously improve action selection according to the reward signal during the interaction with the environment. By using an experience buffer to store and replay experiences, the utilization efficiency of samples and the stability of training can be increased. The collaborative training of the policy network and the value function network can provide the direction and guidance for policy optimization.

[0099] The advantage of this step is to use a deep neural network and the DDPG algorithm for reinforcement learning. Through the optimization of the policy network and the value function network, the policy performance of the neural network model based on DDPG can be gradually improved, enabling it to better adapt to and optimize its behavior during the interaction with the environment.

[0100] Embodiment 5. This embodiment further limits an intelligent cooperative control method for a spatial multi-arm robot to capture a non-cooperative target described in Embodiment 4. The use of the gradient descent algorithm to update the parameters of the policy network and the value function network includes:

[0101] The loss function L of the policy network a is:

[0102] L a = -q π (s,a),

[0103] where (s,a) represents the current state value and action value, and q π is the value function;

[0104] The loss function L of the value function network c is as follows:

[0105] L c = (R + γq π (s', a') - q π (s, a)) 2 ,

[0106] where R represents the reward value at the current moment, (s', a') represents the next state value and action value, and γ represents the discount factor;

[0107] Calculate the gradient of the loss function with respect to the parameters of the policy network and the gradient of the value function network

[0108]

[0109]

[0110] where, is the gradient of the policy network parameters, is the gradient of the value function network.

[0111] Update the parameters using the gradient descent strategy:

[0112]

[0113] where α is the learning rate, is the gradient of the network parameters, and θ represents the network parameters.

[0114] This embodiment mainly updates the parameters of the policy network and the value function network through the gradient descent algorithm, enabling the policy network to better select actions and maximize the return, while the value function network can more accurately estimate the value of the state. This can improve the effect of the reinforcement learning algorithm.

[0115] Embodiment Six. This embodiment further limits the intelligent cooperative control method for a spatial multi-arm robot to capture a non-cooperative target described in Embodiment One. Controlling the spatial two-arm robot to capture the non-cooperative target according to the optimization model includes:

[0116] The spatial two-arm robot obtains the motion state of the non-cooperative target,

[0117] The optimization model generates a control strategy for the spatial two-arm robot according to the motion state of the non-cooperative target;

[0118] The spatial two-arm robot executes corresponding actions according to the control strategy and interacts with the non-cooperative target and the environment to obtain environmental information;

[0119] Adjust the control strategy of the optimization model according to the environmental information to form closed-loop training.

[0120] In this embodiment, by obtaining the motion state of the non-cooperative target, the optimization model can generate the control strategy of the space dual-arm robot according to the real-time motion state of the target. This enables the robot to adjust its own actions in real time according to the motion trajectory and changes of the target, so as to better track and capture the target and improve the success rate of capture.

[0121] In this embodiment, by interacting with the non-cooperative target and the environment and obtaining environmental information, the robot can perceive and understand the dynamic changes of the environment in real time. According to the environmental information, the optimization model can adjust the control strategy, enabling the robot to better cope with environmental changes and the behavior of the target, and improving the efficiency and accuracy of capture.

[0122] In this embodiment, by adjusting the control strategy of the optimization model according to the environmental information, closed-loop training is formed. This means that in the interaction between the robot and the non-cooperative target and the environment, the robot continuously updates its own control strategy and obtains feedback information to optimize the model. Such closed-loop training enables the robot to gradually improve its action strategy and adapt to the changes in the target behavior in different scenarios, and improve the capture ability for non-cooperative targets.

[0123] This embodiment enables the robot to adjust the control strategy in real time according to the motion state of the non-cooperative target, and continuously optimize the model through the interaction with the target and the environment, thereby enhancing the capture ability for non-cooperative targets. This provides better adaptability and flexibility for the robot to cope with uncertain and complex environments in practical applications.

[0124] Embodiment 7: This embodiment further limits a method for intelligent cooperative control of a space multi-arm robot to capture a non-cooperative target described in Embodiment 6. The termination conditions for the single closed-loop training include:

[0125] The total control duration exceeds 9 s or the non-cooperative target collides with the space dual-arm robot.

[0126] In this embodiment, by setting an upper limit on the total control duration, the time of each closed-loop training can be limited. This is to ensure the efficiency and real-time nature of the training process. In practical applications, there may be time requirements and task deadlines, so limiting the training duration enables the robot to quickly learn and adapt to the motion state of the target within the specified time and improve the capture efficiency. By setting collision detection as the termination condition, the safety of the training process can be ensured. When the non-cooperative target collides with the robot, it may cause damage to the robot or injury to the non-cooperative target, or even damage to the environment. Therefore, setting collision detection as the termination condition can avoid the occurrence of potential dangerous situations and protect the safety of the robot, the target and the environment.

[0127] By setting these termination conditions, the duration and safety of the training process can be controlled, and it can be ensured that the robot learns an effective capture strategy within a limited time. At the same time, these termination conditions also help to combat problems such as overfitting, avoiding excessive prolongation or unreasonableness of the training process. Ultimately, this helps to improve the effect of the non-cooperative target capture task and the performance of the robot.

[0128] Embodiment 8. An intelligent cooperative control system for a spatial multi-arm robot to capture a non-cooperative target, the system comprising:

[0129] A space construction unit for constructing the state space and action space of the spatial two-arm robot;

[0130] A reward function construction unit for constructing a reward function for the spatial two-arm robot to capture a target along a desired trajectory according to the state space and action space of the spatial two-arm robot;

[0131] A neural network model establishment unit for establishing a neural network model based on DDPG according to the reward function;

[0132] A training unit for training the neural network model based on DDPG to obtain an optimized model;

[0133] A capture unit for controlling the spatial two-arm robot to capture a non-cooperative target according to the optimized model.

[0134] Embodiment 9. A computer-readable storage medium for storing a computer program, and the computer program executes an intelligent cooperative control method for a spatial multi-arm robot to capture a non-cooperative target according to any one of Embodiments 1 to 7.

[0135] Embodiment 10. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, and when the processor runs the computer program stored in the memory, the processor executes an intelligent cooperative control method for a spatial multi-arm robot to capture a non-cooperative target according to any one of Embodiments 1 to 7.

[0136] Embodiment 11. Refer to Figures 2 to 8 Describe this embodiment. This embodiment provides a specific example for an intelligent cooperative control method for a spatial multi-arm robot to capture a non-cooperative target according to Embodiment 1, and is also used to explain Embodiments 2 to 7. Specifically:

[0137] This embodiment models and designs the reinforcement learning environment into the following five basic elements: the neural network model based on DDPG, the state quantity s, the environment, the reward value r, and the action value a. During the interaction with the environment, the neural network model based on DDPG needs to select appropriate actions to achieve the established goals. After each action selection, the environment changes according to the decision of the neural network model based on DDPG, and at the same time, a state signal at the current moment is generated, including the state quantity and the reward function at the current moment. These signals will be fed back to the neural network model based on DDPG in a timely manner to help it further optimize the selection strategy. The neural network model based on DDPG will optimize the model parameters and select the next action according to the current state and the reinforcement signal to maximize the reward value. During the process of action selection, the neural network model based on DDPG not only needs to consider the magnitude of the immediate reward value, but also needs to consider the impact of the selected action on the next state of the environment and the final reward value. Therefore, the agent needs to continuously trial and error and learn to improve its selection and decision-making abilities to adapt to different environmental disturbances and task requirements. Their relationship is as Figure 2 shown, where the agent represents the neural network model based on DDPG.

[0138] In this embodiment, the capture target training environment of the space dual-arm robot consists of a space dual-arm robot and a non-cooperative target. The space dual-arm robot is a floating-base dual 6-degree-of-freedom robotic arm space robot. The model of the non-cooperative target is constructed as a rotating target dynamics model, and its discrete form of the state equation and measurement equation are formed:

[0139]

[0140] where, x k represents the state quantity [q0, q1, q2, q3, ω1, ω2, ω3] of the system, including a total of seven values of attitude quaternion and angular velocity; the system noise w k in the state equation is Gaussian white noise with a mean of 0

[46] ; the noise v k in the measurement equation is also Gaussian white noise with a mean of 0. In addition to these quantities, the initial covariance matrix also needs to be defined: P0 is the initial covariance matrix, representing the degree of confidence in the initial estimate value x0; the covariance matrix Q k represents the degree of modeling uncertainty (errors introduced by linearization and discretization); the covariance matrix R k represents the degree of observer uncertainty. In this embodiment, the initial rotation speed value of the non-cooperative target is set as a random value during training.

[0141] In this embodiment, the dimension of the state quantity is 39, including respectively: the positions, attitude angles, velocities and angular velocities of the base on the x, y, and z axes, the joint angles and joint angular velocities of the robotic arm, and the velocities and angular velocities of the non-cooperative target; the dimension of the action value is 18, including respectively: the control forces and control torques of the base on the x, y, and z axes, and 12 joint control torques.

[0142] Let e represent the error between the expected value and the true value. The velocity error of the non-cooperative target is expressed as e tv , and the angular velocity error is e tω , then the reward value in the training process is described as follows:

[0143] R = -e tv -e tω

[0144] The reward value in the training process is negative, and the smaller the error between the expected velocity and angular velocity of the non-cooperative target and the true value, the higher the reward value.

[0145] The action and state spaces of the space dual-arm robot are continuous values. The DDPG algorithm is used for training. Specifically:

[0146] 1. State Representation:

[0147] Determine the state information of the environment and use the state information of the environment as the input of the neural network.

[0148] 2. Neural Network Construction:

[0149] Construct two neural networks respectively, one for constructing the policy and one for constructing the value function. Usually, a deep neural network is used. Among them, the policy network (Actor Network) is used to select actions, that is, π(a|s), and the value function network (Critic Network) is used to evaluate the value of actions, that is, q π (s,a).

[0150] 3. Experience Replay:

[0151] Use an experience buffer (Replay Buffer), that is, an experience pool, to store the experiences of the agent. Each time it interacts with the environment, information such as the state, action, reward, and next state is stored in the experience pool. The significance of the experience pool is to enhance the efficiency and stability of the samples. During the training process, instead of immediately using the current experience for parameter update, a batch of data is randomly sampled from the experience buffer for training.

[0152] 4. Policy Evaluation:

[0153] Use the value function network to evaluate the value of the selected action. Here, select the state-action value function q with better Markov property π (s,a). For a given state and action, the state-action value function represents the expected return that the agent can obtain from the current moment after taking a certain action in that state. It measures the overall value of the agent taking a certain action, and this value function is estimated to guide policy improvement subsequently.

[0154] 5. Policy Improvement:

[0155] Guided by the action values obtained using the policy evaluation function, the agent can select a better action (such as the greedy policy) based on the current policy network and value function network to expect better long-term returns.

[0156] 6. Gradient Update:

[0157] When performing gradient update, it is necessary to calculate the gradient of the loss function according to the output of the value function network (Critic Network), and use the gradient descent algorithm to update the parameters of the policy network and the value function network.

[0158] Specifically, for the policy network, we hope to maximize the estimated value of the value function network for the next state. The loss function of the policy network can be defined as:

[0159] L a =-q π (s,a)

[0160] For the value function network, this embodiment hopes to minimize the TD error (Temporal Difference Error), that is, the gap between the value function estimate of the current state and the value function estimate of the next state, which is defined as:

[0161] L c =(R+γq π (s',a')-q π (s,a)) 2

[0162] Where R represents the reward value at the current moment, (s,a) represents the current state value and action value, (s',a') represents the next state value and action value, γ represents the discount factor, and then calculate the gradients of the loss function with respect to the parameters of the policy network and the value function network respectively

[0163]

[0164]

[0165] After that, the gradient descent strategy is used to update the parameters, and the specific rules are as follows:

[0166]

[0167] The parameters are updated by subtracting the gradient multiplied by the learning rate from the parameters, so as to move in the direction of the loss function decreasing.

[0168] Repeat steps 3 to 6, continuously interact with the environment and update the neural network parameters until the predetermined stop condition or learning goal is reached. The DDPG algorithm structure is as Figure 3 shown.

[0169] To sum up, the pseudocode of the DDPG algorithm applied to the capture of a space dual-arm robot is shown in Table 1:

[0170] Table 1 DDPG Pseudocode

[0171]

[0172]

[0173] The space dual-arm robot obtains the control quantity according to the motion state of the non-cooperative target and the control strategy provided by the current agent, and then adjusts the control strategy using the feedback of the environment to form a closed-loop training process. Finally, what is obtained through reinforcement learning is a highly robust controller.

[0174] Assume that under the condition of zero gravity, the single training termination conditions include the following two points:

[0175] (1) The total control duration exceeds 9 s.

[0176] (2) A collision occurs between the non-cooperative target and the space dual-arm robot.

[0177] If either of the above two situations occurs, stop the training immediately. Through the above two points, the agent is respectively urged to complete the capture task as soon as possible and avoid a collision between the non-cooperative target and the space dual-arm robot.

[0178] The simulation experiment is based on the following hardware environment: CPU 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30 GHz, RAM 16.0 GB, GPU Nvidia GeForce RTX 3060 Laptop. Both the actor and critic networks are 3 layers, and the number of units in each layer is 128, 64, and 32 respectively. The training parameter table of the experiment constructed in this embodiment is shown in Table 2. A total of 5000 training times are carried out, and the training process is as Figure 4 shown.

[0179] Table 2 Training Parameter Table

[0180]

[0181]

[0182] Training was carried out respectively using the construction forms of the reward functions in Equation and Equation. The reward functions during the training process are respectively as Figure 5 、 Figure 6 shown. The comparison of the average reward values for the two trainings is as Figure 7 shown. Aiming at the problem that it is impossible to extract a state representation with good Markov property and independent of the observable reconstruction, a neural network model for extracting low-dimensional state representations from high-dimensional observables and realizing the mutual information estimation between state representations was established, and an objective function optimization method based on kinematic prior knowledge was introduced. As Figure 7 shown, the reward function of the objective function optimization method based on kinematic prior knowledge has an average reward value increased by 7% under the same training episodes compared with the original reward function, and the convergence speed and training stability have both been improved.

[0183] The trained DDPG controller was used for simulation verification during capture, and the speed and angular velocity of the target during capture were obtained as Figure 8 shown.

[0184] The above simulation results prove that the DDPG agent constructed in this paper can effectively solve the problem of a space dual-arm robot capturing a non-cooperative target. As Figure 8 shown, the controller obtained by DDPG training has very little disturbance to the target during capture, can quickly converge the speed of the rotating target, and effectively suppresses the jitter problem during capture. By designing the reward function through the objective function optimization method based on the dynamic model prediction error, the training stability is improved and the training speed is accelerated.

[0185] The above further describes the technical solutions provided by the present invention in conjunction with the accompanying drawings to highlight the advantages and beneficial effects, and is not used as a limitation to the present invention. Any modifications, combinations of implementation manners, improvements, and equivalent replacements of the present invention within the scope of the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An intelligent cooperative control method for a spatial multi-arm robot to capture non-cooperative targets, characterized in that, The method includes: Constructing the state space and action space of the space dual-arm robot; Constructing a reward function for the space dual-arm robot to capture the target along a desired trajectory according to the state space and action space of the space dual-arm robot; Establishing a neural network model based on DDPG according to the reward function; Training the neural network model based on DDPG to obtain an optimized model; Controlling the space dual-arm robot to capture the non-cooperative target according to the optimized model; The constructing the reward function for the space dual-arm robot to capture the target along a desired trajectory according to the state space and action space of the space dual-arm robot includes: ， Among them, represents the error between the expected value and the true value, is the velocity error of the non-cooperative target, is the angular velocity error of the non-cooperative target.

2. The intelligent cooperative control method for a spatial multi-arm robot to capture a non-cooperative target according to claim 1, characterized in that The constructing the state space and action space of the space dual-arm robot includes: The dimension of the state variables in the state space and the dimension of the action values in the action space: The dimension of the state variables in the state space is 39, including: the position, attitude angles, velocities and angular velocities of the base on the x, y, and z axes, the joint angles and joint angular velocities of the manipulator, and the velocities and angular velocities of the non-cooperative target; The dimension of the action values in the action space is 18, including: the control forces and control torques of the base on the x, y, and z axes, and 12 joint control torques.

3. The intelligent cooperative control method for a spatial multi-arm robot to capture a non-cooperative target according to claim 1, wherein, The establishing a neural network model based on DDPG according to the reward function includes: S1: Constructing a policy network and a value function network; S2: Using an experience buffer to store the experiences of the neural network model, where the experiences include the robot state, action, reward, and next state information during the environmental interaction; S3: Generating actions using the policy network and improving the policy according to the evaluation results of the value function network; S4: Calculating the loss function gradient using the value function network and using the gradient descent algorithm to update the parameters of the policy network and the value function network; S5: Repeating steps S2 to S4, using the experiences and gradients in the experience buffer to update the neural network model to obtain a neural network model based on DDPG.

4. The intelligent cooperative control method for a spatial multi-arm robot to capture a non-cooperative target according to claim 3, characterized in that, The using the gradient descent algorithm to update the parameters of the policy network and the value function network includes: The loss function of the policy network is as follows: ， Among them, represents the current state value and action value, is the value function; The loss function of the value function network is as follows: ， Among them, represents the reward value at the current moment, represents the next state value and action value, represents the discount factor; Calculate the gradients of the loss function with respect to the parameters of the policy network and the gradients of the value function network : ， ， Among them, is the gradient of the policy network parameters, is the gradient of the value function network; Updating the parameters using the gradient descent strategy: Among them, is the learning rate, is the gradient of the network parameters, represents the network parameters.

5. An intelligent cooperative control method for a spatial multi-arm robot to capture a non-cooperative target according to claim 1, characterized in that The controlling the space dual-arm robot to capture the non-cooperative target according to the optimized model includes: The space dual-arm robot obtaining the motion state of the non-cooperative target; The optimized model generating a control strategy for the space dual-arm robot according to the motion state of the non-cooperative target; The space dual-arm robot performing corresponding actions according to the control strategy and interacting with the non-cooperative target and the environment to obtain environmental information; Adjusting the control strategy of the optimized model according to the environmental information to form a closed-loop training.

6. The intelligent cooperative control method for a spatial multi-arm robot to capture a non-cooperative target according to claim 5, characterized in that, The termination conditions of the closed-loop training include: The total control duration exceeding 9 s or the non-cooperative target colliding with the space dual-arm robot.

7. An intelligent cooperative control system for a spatial multi-arm robot to capture non-cooperative targets, characterized in that, The system includes: A space construction unit for constructing the state space and action space of the space dual-arm robot; A reward function construction unit for constructing a reward function for the space dual-arm robot to capture the target along a desired trajectory according to the state space and action space of the space dual-arm robot; A neural network model establishment unit for establishing a neural network model based on DDPG according to the reward function; A training unit for training the neural network model based on DDPG to obtain an optimized model; A capture unit for controlling a space dual-arm robot to capture a non-cooperative target according to the optimization model; The construction of a reward function for the space dual-arm robot to capture a target along a desired trajectory based on the state space and action space of the space dual-arm robot, including: ， wherein, represents the error between the expected value and the true value, is the velocity error of the non-cooperative target, is the angular velocity error of the non-cooperative target.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, and the computer program executes an intelligent cooperative control method for a space multi-arm robot to capture a non-cooperative target according to any one of claims 1-6.

9. A computer device, characterized in that: It includes a memory and a processor. A computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes an intelligent cooperative control method for a space multi-arm robot to capture a non-cooperative target according to any one of claims 1-6.

Images