A Non-Cooperative Target Capture Method Based on Deep Reinforcement Learning
By adopting deep reinforcement learning in the space robot arm combined with pre-training and main task training methods, the reward function and network parameters are optimized, and the deep reinforcement learning algorithm is solved. The high training time cost and success rate of deep reinforcement learning algorithm in the non-cooperative target capture task of space robot arm is solved. The low problem achieves the optimization of energy and time loss, and provides a general control method suitable for high-dimensional space robotic arms.
Patent Information
- Application Number
- CN202311655229.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-05
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2043-12-05
AI Technical Summary
Existing deep reinforcement learning algorithms have problems with high training time cost and low success rate in non-cooperative target capture tasks of space robotic arms, making it difficult to optimize energy and time losses.
Using a non-cooperative target capture method based on deep reinforcement learning, the pre-training reward function and optimization reward function are designed through the combination of pre-training and main task training, and the network parameters are optimized using soft updates and Lagrange methods to build an end-to-end control strategy, directly from System state quantity generation control strategy to avoid complex dynamic modeling.
The optimization of energy and time loss in non-cooperative target capture tasks of space robot arm is achieved, the training success rate and efficiency are improved, and a general method is provided for high-dimensional space robot arm systems.
Smart Images

Figure CN117464687B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of space manipulator control and non-cooperative target capture, and specifically relates to a non-cooperative target capture method based on deep reinforcement learning with energy and time loss as optimization targets under output constraints. Background Art
[0002] As human beings continue to explore space, space missions are becoming more and more complex and diverse, which makes space robots play an increasingly important role in space missions. As a powerful space robot, the space manipulator can be used for spacecraft fuel replenishment, on-orbit maintenance, equipment assembly and disassembly, and various experimental tasks. Among them, non-cooperative target capture is one of the most basic and important capabilities of the space manipulator, and it is also the basis for realizing many other space missions. Since the energy of the space manipulator is very precious, optimizing the energy loss of the task can help extend the on-orbit service time of the space manipulator. At the same time, reducing the time for the space manipulator to perform the target capture task can improve the work efficiency of the manipulator, which is conducive to the subsequent tasks. Therefore, it is of great significance to optimize the time and energy loss of the space floating manipulator during non-cooperative target capture. The traditional optimization method needs to establish an analytical dynamic model of the space manipulator system. The space manipulator system is in a zero gravity environment. There is a coupling relationship between the kinematics and dynamics of the manipulator and the base, which makes the dynamic model of the high-dimensional space manipulator very complex. The problems of fuel consumption and dynamic parameter errors further increase the difficulty of obtaining an accurate dynamic model of the space manipulator, making it difficult for the traditional optimization method to be applied to the high-dimensional space manipulator. As a sampling-based optimization control algorithm, deep reinforcement learning obtains data through the interaction between the agent and the environment, and optimizes the control strategy under the guidance of the reward function. Using deep reinforcement learning for space manipulator control, there is no need to perform complex dynamic modeling while optimizing the strategy, and the optimization target can be directly expressed in the reward function. Compared with traditional optimization methods, it is simpler and more intuitive, and the algorithm framework is versatile and can be used for manipulator systems of any configuration. However, the non-cooperative target capture task with optimal time and energy loss requires the optimization of energy and time loss under the premise of ensuring successful capture. The overall objective function is very complex, and the success rate of direct deep reinforcement learning training is not high. It also requires a huge amount of samples for training, and the time cost is extremely high. Therefore, the existing reinforcement learning algorithm cannot be directly used for manipulator target capture and optimization tasks.
[0003] In summary, traditional optimization algorithms require an analytical dynamics model of the space robot, which makes them difficult to apply to high-dimensional space robot systems. When faced with complex objective functions that require both target capture and energy and time loss performance optimization, existing deep reinforcement learning algorithms have problems such as high training time costs and low target capture success rates. Summary of the invention
[0004] The purpose of the present invention is to solve the problems of high time cost of training existing deep reinforcement learning algorithms and low success rate of target capture when target capture and energy and time loss performance optimization are required at the same time, and to propose a non-cooperative target capture method based on deep reinforcement learning.
[0005] The technical solution adopted by the present invention to solve the above technical problems is: a non-cooperative target capture method based on deep reinforcement learning, the method specifically comprising the following steps:
[0006] Step 1: Determine the objective function C to be optimized and the state vector and action vector of the reinforcement learning control strategy;
[0007] Step 2: Determine the control strategy function π according to the motion vector dimension of the floating base space manipulator θ , the first value function Second value function And the loss function to be optimized C ψ The network structure is as follows, where θ is the network parameter of the control strategy function, φ1 is the network parameter of the first value function, φ2 is the network parameter of the second value function, and ψ is the network parameter of the loss function to be optimized;
[0008] Step 3: Set the number of pre-training steps to T pre , the number of training steps for the main task is T main , the length of each task episode, the batch size during policy update, the capacity of the buffer, the frequency of policy update, and the hyperparameters during training;
[0009] The hyperparameters include the control strategy function learning rate λ θ , value function learning rate λ φ , learning rate λ of the loss function to be optimized ψ , temperature coefficient learning rate λ α and soft update coefficient τ;
[0010] Step 4: Design pre-training reward function and optimize reward function;
[0011] Step 5: Initialize the control strategy function π θ , the first value function Second value function And the loss function C to be optimized ψ The network parameters of the first objective value function are set. Second objective value function And the target loss function to be optimized First objective value function Second objective value function And the target loss function to be optimized The parameters of the soft update coefficient τ and the first value function Second value function And the loss function to be optimized C ψ Soft update of network parameters;
[0012] Step 6: Initialize the pre-training step number t=1;
[0013] Step 7: Reset the training environment and set the floating base space manipulator and the non-cooperative target to the initial state;
[0014] Step 8. In the tth step of the pre-training phase, obtain the current state s from the environment t After that, the current state s t Input control strategy function π θ The network, control strategy function π θ The mean μ of the network output action distribution t and variance σ t , and then sample from the action distribution to obtain the action to be performed by the floating-based space manipulator system in the current state is the action distribution, which is a normal distribution;
[0015] Floating-based space manipulator system performs action a t , and get reward r after interacting with the environment t pre and the reward to be optimized r t opt , and enter the next state s t+1 ;Tuple (s t ,a t ,r t pre ,r t opt ,s t+1 ) into a buffer;
[0016] Step 9: Determine whether the task of the current scene is completed, that is, determine whether the number of steps reaches the length of the current scene;
[0017] If the task of the current scene is not finished, set t = t + 1 and use state s t+1 Return to step eight;
[0018] Otherwise, if the task of the current scene is completed, determine whether the strategy needs to be updated based on the strategy update frequency;
[0019] If the policy update frequency is reached, a batch of tuples is sampled from the buffer to update the control policy function π θNetwork parameters, first value function The network parameters and the second value function The network parameters and the loss function to be optimized C ψ The network parameters and temperature coefficient α are updated using the first value function The network parameters and the second value function Q φ2 The network parameters and loss function to be optimized C ψ The network parameters have a great impact on the first objective value function. The network parameters and the second objective value function The network parameters and target loss function to be optimized Perform soft update on the network parameters, and then execute step 10;
[0020] Otherwise, if the policy update frequency is not reached, directly execute step 10;
[0021] Step 10: Determine whether the set number of pre-training steps T has been reached pre ;
[0022] If the set number of pre-training steps is reached, the pre-training phase ends and step 11 is executed;
[0023] If the pre-training step number is not reached, set t=t+1 and return to step 7;
[0024] Step 11: Set the number of training steps to the number of training steps for the main task T main , update the objective function of the task; and initialize the number of training steps t = 1;
[0025] Step 12: reset the training environment and set the space manipulator and the non-cooperative target to the initial state;
[0026] Step 13: In the tth step of the training process, get the current state s′ from the environment t , and then the current state s′ t Input control strategy function π θ The network, control strategy function π θ The network output action distribution The mean μ′ t and variance σ′ t , from the action distribution The action to be performed by the space manipulator system in the current state is obtained by sampling The space manipulator system performs action a′ t , and get reward r after interacting with the environment t pre and the reward to be optimized r t opt , and enter the next state s′ t+1 ;
[0027] The tuple (s′ t ,a′ t ,r t pre ,r t opt ,s′ t+1 ) into a buffer;
[0028] Step 14: Determine whether the task of the current scene is completed, that is, determine whether the number of steps reaches the length of the current scene;
[0029] If the task of the current scene is not finished, set t = t + 1 and use state s' t+1 Return to step 13;
[0030] Otherwise, when the current task is completed, determine whether the strategy needs to be updated based on the strategy update frequency;
[0031] If the strategy update frequency is reached, a batch of tuples is sampled from the buffer and the control strategy function π is updated using the Lagrange method. θ , value function Q φ , loss function C to be optimized ψ The network parameters and Lagrange multiplier β, where
[0032]
[0033] And use the updated value function Q respectively φ The network parameters and loss function to be optimized C ψ The network parameters have an effect on the target value function The network parameters and target loss function to be optimized Perform soft update on the network parameters, and then execute step 15;
[0034] Otherwise, if the policy update frequency is not reached, directly execute step 15;
[0035] Step 15: Determine whether the set number of training steps has been reached;
[0036] If the set number of training steps is reached, the training phase ends and step 16 is executed;
[0037] If the set number of training steps is not reached, set t=t+1 and return to step 12;
[0038] Step 16: The network parameters of the control strategy function obtained at the end of the training are used as the trained network parameters, and the floating-based space manipulator system is controlled in real time according to the trained network parameters.
[0039] The beneficial effects of the present invention are:
[0040] 1. The present invention proposes an end-to-end control method based on model-free deep reinforcement learning for the non-cooperative target capture task of a space manipulator under the optimal constraints of energy and time loss. The method is not affected by the configuration of the space manipulator and the motion state of the non-cooperative target, and is a general method for such tasks. In addition, the method of the present invention has an end-to-end structure, and the control strategy is directly generated through the system state quantity, without the need to model the complex space manipulator system.
[0041] 2. The present invention estimates the cumulative expectation of integral losses such as time and energy loss during training through the bootstrap method, incorporates the optimization problem of the loss function into the actor-critic structure of the deep reinforcement learning algorithm, and optimizes it with the deep reinforcement learning algorithm, obtaining a general algorithm structure for solving such integral losses.
[0042] 3. Aiming at complex high-dimensional control problems such as non-cooperative target capture by space manipulators, the present invention proposes a method of pre-training plus main task training. The target capture task is completed in the pre-training process, and the objective function to be optimized during capture is optimized in the main task training process. This solves the problems of low training success rate and difficult algorithm convergence when capture and objective function optimization are performed simultaneously during training. In addition, the training time cost is low and the success rate of target capture using the trained control strategy is high.
[0043] 4. The present invention uses KL divergence to constrain the update step size of the action value function during the main task training process, ensuring that the target capture capability learned by the control strategy in pre-training will not be excessively affected by the optimization of the objective function, thereby ensuring that the capture process is optimized under the premise of successfully capturing the target. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 is a flow chart of the method of the present invention;
[0045] Figure 2 It is the energy loss curve of the three methods after the same number of training steps in the target grasping task;
[0046] Our method indicates the method of the present invention, pre-training indicates that only the pre-training method is used, and withoutpre-training indicates that no pre-training method is used (i.e., only the main training method is used);
[0047] Figure 3 It is the curve of the joint energy and time loss of the three methods after the same number of training steps in the target grasping task. DETAILED DESCRIPTION
[0048] Specific implementation method 1: Combination Figure 1 The present embodiment describes a non-cooperative target capture method based on deep reinforcement learning, and the method specifically comprises the following steps:
[0049] Step 1: Determine the objective function C to be optimized and the state vector and action vector of the reinforcement learning control strategy;
[0050] Step 2: Determine the control strategy function π according to the motion vector dimension of the floating base space manipulator θ , the first value function Second value function And the loss function C to be optimized ψ The network structure is as follows, where θ is the network parameter of the control strategy function, φ1 is the network parameter of the first value function, φ2 is the network parameter of the second value function, and ψ is the network parameter of the loss function to be optimized;
[0051] Step 3: Set the number of pre-training steps to T pre , the number of training steps for the main task is T main , the length of each episode of the task (an episode refers to the process from the beginning to the end of the task, and the maximum number of steps contained in each episode is called its length), the batch size when the policy is updated, the capacity of the buffer, the policy update frequency (the update frequency specifies how many steps in the training process the policy and value function are updated once), and the hyperparameters during training;
[0052] The hyperparameters include the control strategy function learning rate λ θ , value function learning rate λ φ , learning rate λ of the loss function to be optimized ψ , temperature coefficient learning rate λ α and soft update coefficient τ;
[0053] Step 4: Design pre-training reward function and optimize reward function;
[0054] Step 5: Initialize the control strategy function π θ , the first value function Second value function And the loss function C to be optimized ψ The network parameters of the first objective value function are set. Second objective value function And the target loss function to be optimized First objective value function Second objective value function And the target loss function to be optimized The parameters of the soft update coefficient τ and the first value function Second value function And the loss function to be optimized C ψ Soft update of network parameters;
[0055] Step 6: Initialize the pre-training step number t=1;
[0056] Step 7: Reset the training environment and set the floating base space manipulator and the non-cooperative target to the initial state;
[0057] Step 8. In the tth step of the pre-training phase, obtain the current state s from the environment t After that, the current state s t Input control strategy function π θ The network, control strategy function π θ The mean μ of the network output action distribution t and variance σ t , and then sample from the action distribution to obtain the action to be performed by the floating-based space manipulator system in the current state is the action distribution, which is a normal distribution;
[0058] The floating space robot system performs action a t , and get reward r after interacting with the environment t pre and the reward to be optimized r t opt , and enter the next state s t+1 ;Tuple (s t ,a t ,r t pre ,r t opt ,s t+1 ) into a buffer;
[0059] Step 9: Determine whether the task of the current scene is completed, that is, determine whether the number of steps reaches the length of the current scene;
[0060] If the task of the current scene is not finished, set t = t + 1 and use state s t+1 Return to step eight;
[0061] Otherwise, if the task of the current scene is completed, determine whether the strategy needs to be updated based on the strategy update frequency;
[0062] If the policy update frequency is reached, a batch of tuples is sampled from the buffer to update the control policy function π θ Network parameters, first value function The network parameters and the second value function The network parameters and the loss function to be optimized C ψThe network parameters and temperature coefficient α are updated using the first value function The network parameters and the second value function The network parameters and loss function to be optimized C ψ The network parameters have a great impact on the first objective value function. The network parameters and the second objective value function The network parameters and target loss function to be optimized Perform soft update on the network parameters, and then execute step 10;
[0063] Otherwise, if the policy update frequency is not reached, directly execute step 10;
[0064] Step 10: Determine whether the set number of pre-training steps T has been reached pre ;
[0065] If the set number of pre-training steps is reached, the pre-training phase ends and step 11 is executed;
[0066] If the pre-training step number is not reached, set t=t+1 and return to step 7;
[0067] Step 11: Set the number of training steps to the number of training steps for the main task T main , update the objective function of the task; and initialize the number of training steps t = 1;
[0068] Step 12: reset the training environment and set the space manipulator and the non-cooperative target to the initial state;
[0069] Step 13: In the tth step of the training process, get the current state s′ from the environment t , and then the current state s′ t Input control strategy function π θ The network, control strategy function π θ The network output action distribution The mean μ′ t and variance σ′ t , from the action distribution The action to be performed by the space manipulator system in the current state is obtained by sampling The space manipulator system performs action a′ t , and get reward r after interacting with the environment t pre and the reward to be optimized r t opt , and enter the next state s′ t+1 ;
[0070] The tuple (s′ t ,a′ t ,r tpre ,r t opt ,s′ t+1 ) into a buffer;
[0071] Step 14: Determine whether the task of the current scene is completed, that is, determine whether the number of steps reaches the length of the current scene;
[0072] If the task of the current scene is not finished, set t = t + 1 and use state s' t+1 Return to step 13;
[0073] Otherwise, when the current task is completed, determine whether the strategy needs to be updated based on the strategy update frequency;
[0074] If the strategy update frequency is reached, a batch of tuples is sampled from the buffer and the control strategy function π is updated using the Lagrange method. θ , value function Q φ , loss function C to be optimized ψ The network parameters and Lagrange multiplier β, where Q φ Select a value function network with a smaller output;
[0075] And use the updated value function Q respectively φ The network parameters and loss function to be optimized C ψ The network parameters have an effect on the target value function The network parameters and target loss function to be optimized Perform soft update on the network parameters, and then execute step 15;
[0076] Otherwise, if the policy update frequency is not reached, directly execute step 15;
[0077] Step 15: Determine whether the set number of training steps has been reached;
[0078] If the set number of training steps is reached, the training phase ends and step 16 is executed;
[0079] If the set number of training steps is not reached, set t=t+1 and return to step 12;
[0080] Step 16: The network parameters of the control strategy function obtained at the end of the training are used as the trained network parameters, and the floating-based space robot system is controlled in real time according to the trained network parameters, and the action vector to be executed by the floating-based space robot system is output in real time.
[0081] Specific implementation method 2: This implementation method is different from the specific implementation method 1 in that: the objective function to be optimized is C=k1c t +k2ceng , where c t =T is the total time taken for the task, c eng is the energy loss, k1 is the time weight, and k2 is the energy loss weight;
[0082] c eng Specifically:
[0083]
[0084] Where n is the number of controlled joints of the floating-based space manipulator, i = 1, 2, ..., n, τ mi is the torque of the ith controlled joint of the floating-based space manipulator, is the angular velocity of the ith controlled joint of the floating-based space manipulator, t is the time, τ bj is the control torque of the jth base, is the angular velocity of the jth base, j = 1, 2, 3, F bj is the jth base control force, is the linear velocity of the jth base, and |·| represents the absolute value.
[0085] The three items in formula (1) represent the energy loss of the robot arm, base rotation and base translation respectively.
[0086] The other steps and parameters are the same as those in the first embodiment.
[0087] Specific implementation method three: This implementation method is different from specific implementation method one or two in that: the state vector is Among them, b p is the position of the base, b o For the posture of the base, is the linear velocity of the base, is the angular velocity of the base, q is the joint position of the floating base space manipulator, is the joint velocity vector of the floating base space manipulator, τ b is the base control torque, τ m is the joint control torque of the floating space manipulator, F b is the control force of the base, e p is the position vector of the end gripper, t p is the target position vector.
[0088] The other steps and parameters are the same as those in the first or second embodiment.
[0089] Specific implementation method 4: This implementation method is different from any one of the specific implementation methods 1 to 3 in that: the motion vector is [τ b ,τ m ].
[0090] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.
[0091] Specific implementation method 5: This implementation method is different from any one of the specific implementation methods 1 to 4 in that: the control strategy function π θ , the first value function and the second value function All are represented using MLP neural network.
[0092] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.
[0093] Specific implementation method 6: This implementation method is different from any one of the specific implementation methods 1 to 5 in that: the pre-training reward function is:
[0094]
[0095] Among them, r t pre is the pre-trained reward function, v t is the target linear velocity, ||·||2 represents the 2-norm, and ||·|| represents the 1-norm.
[0096] The other steps and parameters are the same as those in Specific Implementations 1 to 5.
[0097] Specific implementation method 7: This implementation method is different from any one of the specific implementation methods 1 to 6 in that: the optimization reward function is:
[0098]
[0099] Among them, r t opt To optimize the reward function.
[0100] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.
[0101] Specific implementation eight: This implementation differs from any one of specific implementations one to seven in that: in the pre-training stage, the first value function The network and the second value function The loss function of the network is:
[0102]
[0103] in, is the loss function of the network with the i-th value function, i = 1, 2, Ε represents the expectation, B represents the set of tuples sampled during update, y(s t+1 ,r t pre ,d t ) is the objective function of the value function;
[0104]
[0105] Among them, γ represents the discount rate of cumulative rewards, d t Is a Boolean value indicating whether the task is completed (when the tth step task is completed, d t =1, otherwise d t =0), α is the temperature coefficient, π θ (a t+1 |s t+1 ) represents the control strategy π θ In status t+1 Next output action a t+1 , ln is the natural logarithm;
[0106] The i-th value function The network parameters are updated by formula (6):
[0107]
[0108] Among them, ▽ represents the gradient, |B| represents the number of tuples contained in set B, and λ φ is the value function learning rate;
[0109] Control strategy function π θ The network parameters θ are updated by equation (7):
[0110]
[0111] in, By π θ After reparameterization, λ θ is the learning rate of the control strategy function;
[0112] π θ (s)=[μ,σ],define but for:
[0113]
[0114] Loss function C to be optimized ψ The network parameter ψ is updated by formula (9):
[0115]
[0116] Among them, λ ψ is the learning rate of the loss function to be optimized;
[0117] The temperature coefficient α is updated by equation (10):
[0118]
[0119] Among them, λ α is the temperature coefficient learning rate, H(π θ (s t ))=-∫π θ (s t )ln(π θ (s t ))ds t is the control strategy function π θ The information entropy of is the target entropy.
[0120] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.
[0121] Specific implementation nine: This implementation differs from any one of specific implementations one to eight in that the capacity of the buffer is not less than three times the batch during policy update. If the buffer capacity limit is reached, the earliest tuple stored in the current buffer is deleted and the new tuple is saved in the buffer.
[0122] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.
[0123] Specific implementation method 10: This implementation method is different from any one of specific implementation methods 1 to 9 in that: in step 11, the specific process of updating the objective function of the task is:
[0124] The main task training problem can be expressed as:
[0125]
[0126] Among them, D KL [·] is the KL divergence, is the value function Q φ The corresponding pre-training result is The parameter is the value function Q φ The parameters after pre-training, d is the threshold of KL divergence;
[0127] This problem can be optimized using the Lagrange method, and the updated objective function to be optimized is equivalent to:
[0128]
[0129] Among them, L m is the updated objective function to be optimized, and β is the Lagrange multiplier;
[0130] L m The gradient with respect to θ is:
[0131]
[0132] in, It's Q φ The corresponding target value function;
[0133] Control strategy function π during training θ The network parameters are updated by formula (14):
[0134] θ←θ-λ θ ▽ θ L m (14)
[0135] The loss function C to be optimized during training ψ The network parameters are updated by formula (15):
[0136]
[0137] The Lagrange multiplier β is updated by equation (16):
[0138]
[0139] Among them, λ β is the learning rate of the Lagrange multiplier.
[0140] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.
[0141] Example
[0142] This embodiment proposes a non-cooperative target capture method for a floating-based space manipulator under the optimal joint constraints of energy and time based on deep reinforcement learning. The method is based on model-free reinforcement learning and can be used for a space manipulator system of any structure. It can simultaneously optimize energy and time losses in a target capture task. The method specifically includes the following steps:
[0143] Step 1: Determine the function form and correlation coefficient of the target C to be optimized. The most commonly used optimization target form is the linear combination of energy and time loss k1C eng +k2C T In this embodiment, k1=k2=1;
[0144] Determine the state space and action space of the reinforcement learning control policy:
[0145] To ensure that the control strategy obtains enough information, the state vector should contain at least the subset Among them, b p ,b o Indicates the position and attitude of the base, are the linear and angular velocities of the base, q, is the robot joint position and velocity vector, τ b ,τ m are the base and robot joint control torques, F b is the base control force, e p ,t p are the position vectors of the end gripper and the target, respectively. The action vector of the task can be selected as the velocity or force and moment [F b ,τ b ,τ m ]. If the robot base is not controllable, the state vector should contain the subset The action vector is or τ m ;
[0146] A simulation environment is built in CoppeliaSim. In this embodiment, a base-controllable seven-degree-of-freedom dexterous manipulator is used as the controlled object to capture the target in uniform linear motion. The state vector of the task is The action vector is the base and robot joint torque [τ b ,τ m ], where the seventh joint has no effect on the capture task, so it keeps its initial position, and only the first six joints are controlled;
[0147] Step 2: Determine the control strategy function π according to the action space dimension of the space manipulator θ , the first value function Second value function And the loss function C to be optimized ψ The network structure of , where θ is the network parameter of the control strategy function, φ1 is the network parameter of the first value function, φ2 is the network parameter of the second value function, and ψ is the network parameter of the loss function to be optimized;
[0148] The above control strategy function π θ , the first value function Second value function And the loss function to be optimized C ψ are all represented by MLP neural networks, where the subscripts represent neural network parameters. The network structure can be adjusted according to the specific task. In this embodiment, the strategy function π θ The network structure is [42, 128, 256, 256, 128, 18], where 42 is the neural network input dimension, which is equal to the state vector dimension; 18 is the neural network output dimension, which is twice the action vector dimension. Its output can be recorded as [μ, σ], where μ and σ are both 9-dimensional vectors, and the two form a Gaussian distribution. Where Σ=diag(σ), the control amount of the space manipulator is obtained by sampling from the Gaussian distribution [128,256,256,128] is the network structure of the hidden layer. The first value function Second value function The network structure is [51,128,256,256,128,64,1], and the loss function to be optimized is C ψ The network structure of is [51,128,256,256,128,64,1], the network input of both is the combination of state and action vector [s,a], and the output is the corresponding scalar, that is, the value of the state-action pair and the value of the loss function to be optimized;
[0149] Step 4: Set training related parameters, including the number of pre-training steps T pre , the number of training steps for the main task is T main , the length of each episode of the task (an episode refers to the process from the beginning to the end of the task, and the maximum number of steps contained in each episode is called its length), the size of the batch when the policy is updated, the capacity of the buffer, the policy update frequency (the update frequency specifies how many steps in the training process the policy and value function are updated once), and the hyperparameters during training (including the policy learning rate λ θ , value function learning rate λ φ , learning rate λ of the loss function to be optimized ψ , temperature coefficient learning rate λ α and soft update coefficient τ);
[0150] In this embodiment, the number of pre-training steps T pre = 2000000, number of training steps for the main task T main = 2000000, the length of each episode of the task is l = 2000, the batch size is set to 256 during policy update, the buffer capacity is 5000, the policy update frequency is 50000 (i.e., update once every 50000 steps), and the policy learning rate λ θ , value function learning rate λ φ , learning rate λ of the loss function to be optimized ψ and temperature coefficient learning rate λ α are all set to 0.003, the soft update coefficient τ=0.995, and the entropy target value
[0151] Step 5: Design pre-training reward function r pre And optimize the reward function r opt , where the pre-trained reward function has the following form:
[0152]
[0153] where v b Indicates the linear velocity of the base, v t represents the target linear velocity, tp represents the target position vector, e p represents the position vector of the end gripper of the robot arm, represents the base angular velocity, ||·||2 represents the 2-norm;
[0154] The optimization reward function has the following form:
[0155]
[0156] Where, the superscript t represents the tth step, n is the number of controlled joints of the robot arm, is the joint torque of the robot arm at step t, is the angular velocity of the i-th controlled joint of the robot at the t-th step, is the control torque of the base at step t, is the angular velocity of the base at step t, is the base control force at step t, is the linear velocity of the base at step t;
[0157] Step 6: Set the number of training steps to the number of pre-training steps T = T pre , set the reward function to the training reward function R = R pre , initialize the control strategy function π θ , the first value function Second value function And the loss function C to be optimized ψ The network parameters and set the first objective value function Second objective value function And the target loss function to be optimized And the first value function Second value function And the loss function to be optimized C ψ The network parameters are copied to the first objective value function Second objective value function And the target loss function to be optimized
[0158] Step 7: Reset the training environment and set the space manipulator and the target to be captured to the initial state;
[0159] Step 8. In step t, get the current state s from the environment t , the current state s t Input policy network π θ , the mean μ of the action distribution output by the policy network t and σ t , sample from this distribution to obtain the action to be performed by the space manipulator system in the current state Space manipulator system performs action at , and get reward r after interacting with the environment t pre and the reward to be optimized r t opt , and enter the next state s t+1 And the tuple (s t ,a t ,r t pre ,r t opt ,s t+1 ) into a buffer;
[0160] Step 9: Determine whether the task of the current scene is completed, that is, determine whether the number of steps reaches the length of the current scene;
[0161] If the task of the current scene is not completed, use the state s t+1 Return to step eight;
[0162] Otherwise, when the current task is completed, determine whether the strategy needs to be updated based on the strategy update frequency;
[0163] If the policy update frequency is reached, a batch of tuples is sampled from the buffer to update the control policy function π θ , the first value function Second value function Temperature coefficient α and loss function to be optimized C ψ , and use the updated first value function Second value function Loss function C to be optimized ψ For the first objective value function Second objective value function And the target loss function to be optimized Perform a soft update and then go to step 10;
[0164] Otherwise, if the policy update frequency is not reached, directly execute step 10;
[0165] Step 10: Determine whether the set number of training steps has been reached;
[0166] If the set number of training steps is reached, the pre-training phase ends and step 11 is executed;
[0167] If the set number of training steps is not reached, return to step 7;
[0168] Step 11: Set the number of training steps to the number of training steps for the main task T = T main , update the objective function of the task;
[0169] Step 12: Reset the training environment and set the space manipulator and the target to be captured to the initial state;
[0170] Step 13: In step t, get the current state s from the environment t , the current state s t Input policy network π θ , the mean μ of the action distribution output by the policy network t and σ t , sample from this distribution to obtain the action to be performed by the space manipulator system in the current state Space manipulator system performs action a t , and get reward r after interacting with the environment t pre and the reward to be optimized r t opt , and enter the next state s t+1 And the tuple (s t ,a t ,r t pre ,r t opt ,s t+1 ) into a buffer;
[0171] Step 14: Determine whether the task of the current scene is completed, that is, determine whether the number of steps reaches the length of the current scene;
[0172] If the task of the current scene is not completed, use the state s t+1 Return to step 13;
[0173] Otherwise, when the current task is completed, determine whether the strategy needs to be updated based on the strategy update frequency;
[0174] If the strategy update frequency is reached, a batch of tuples is sampled from the buffer and the control strategy function π is updated using the Lagrange method. θ , value function Q φ , loss function C to be optimized ψ and Lagrange multiplier β, and use the updated value function Q φ And the loss function to be optimized C ψ The target value function Target loss function to be optimized Perform soft update and then return to step 15;
[0175] Step 15: Determine whether the set number of training steps has been reached;
[0176] If the set number of training steps is reached, the pre-training phase ends and step 16 is executed;
[0177] If the set number of training steps is not reached, return to step 12;
[0178] Step 16: The network parameters of the control strategy function obtained at the end of the training are used as the trained network parameters, and the floating-based space manipulator system is controlled in real time according to the trained network parameters.
[0179] The specific update process of the algorithm is as follows:
[0180] In the pre-training phase, the first value function The network and the second value function The loss function of the network is:
[0181]
[0182] Where B represents the batch sampled during update, y(s t+1 ,r t pre ,d t ) is the objective function of the value function and has the following form:
[0183]
[0184] where d t Is a Boolean value indicating whether the task is completed. When the tth step task is completed, d t =1, otherwise d t = 0. Value function The parameters of are updated by the following formula:
[0185]
[0186] Control strategy π θ Update via:
[0187]
[0188] in By π θ Obtained by reparameterization. θ (s)=[μ,σ],define but
[0189]
[0190] C ψ Update it with the following formula:
[0191]
[0192] The temperature coefficient α is updated by the following formula:
[0193]
[0194] Where H(π θ )=-∫π θ (s t )ln(π θ (s t ))ds t is strategy π θ The information entropy of is the target entropy.
[0195] Furthermore, after the pre-training process is completed, the main task training process is carried out. The main task training problem can be expressed as:
[0196]
[0197] Where C is the optimization target, D KL [·] is the KL divergence, is the value function after pre-training, and d is the threshold of KL divergence. This problem can be optimized using the Lagrange method, and the optimization target is equivalent to:
[0198]
[0199] Where β is the Lagrange multiplier. m The gradient with respect to θ is:
[0200]
[0201] π θ Update via:
[0202] θ←θ-λ θ ▽ θ L m (29)
[0203] C ψ Update it with the following formula:
[0204]
[0205] where r opt Has the following form:
[0206]
[0207] β is updated by:
[0208]
[0209] Experimental Section
[0210] The algorithm of the present invention is trained and tested in a simulation environment. The algorithm of the present invention can be divided into two parts: pre-training and performance optimization. To verify the effectiveness of the algorithm, the pre-training and performance optimization parts of the algorithm are ablated respectively, and the same training parameters are used for training, and then tested in the capture task. Figure 2 is the energy loss of the three methods in the capture task after training, Figure 3 is the energy-time joint loss of the three methods in the capture task after training. Figure 2 and Figure 3 It can be seen that the algorithm of the present invention has the minimum energy and time loss while successfully capturing the target. Although the capture can be achieved with only pre-training, it will consume more time and energy, and the target cannot be captured without pre-training. The present invention effectively reduces the time and energy loss of the task on the premise of successfully capturing non-cooperative targets, and has a high practical value.
[0211] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.
Claims
1. A non-cooperative target capture method based on deep reinforcement learning, characterized in that: The method specifically comprises the following steps: Step 1: Determine the objective function C to be optimized and the state vector and action vector of the reinforcement learning control strategy; The objective function to be optimized is C=k1c t +k2c eng , where c t =T is the total time taken for the task, c eng is the energy loss, k1 is the time weight, and k2 is the energy loss weight; c eng Specifically: Where n is the number of controlled joints of the floating-based space manipulator, i = 1, 2, ..., n, τ mi is the torque of the ith controlled joint of the floating-based space manipulator, is the angular velocity of the ith controlled joint of the floating-based space manipulator, t is the time, τ bj is the control torque of the jth base, is the angular velocity of the jth base, j = 1, 2, 3, F bj is the jth base control force, is the linear velocity of the jth base, and |·| represents the absolute value; The state vector is Among them, b p is the position of the base, b o For the posture of the base, is the linear velocity of the base, is the angular velocity of the base, q is the joint position of the floating base space manipulator, is the joint velocity vector of the floating base space manipulator, τ b is the base control torque, τ m is the joint control torque of the floating space manipulator, F b is the control force of the base, e p is the position vector of the end gripper, t p is the position vector of the target; The action vector is [τ b ,τ m ]; Step 2: Determine the control strategy function π according to the motion vector dimension of the floating base space manipulator θ , the first value function Second value function And the loss function to be optimized C ψ The network structure of , where θ is the network parameter of the control strategy function, φ1 is the network parameter of the first value function, φ2 is the network parameter of the second value function, and ψ is the network parameter of the loss function to be optimized; Step 3: Set the number of pre-training steps to T pre , the number of training steps for the main task is T main , the length of each task episode, the batch size during policy update, the capacity of the buffer, the frequency of policy update, and the hyperparameters during training; The hyperparameters include the control strategy function learning rate λ θ , value function learning rate λ φ , learning rate λ of the loss function to be optimized ψ , temperature coefficient learning rate λ α and soft update coefficient τ; Step 4: Design pre-training reward function and optimize reward function; Step 5: Initialize the control strategy function π θ , the first value function Second value function And the loss function to be optimized C ψ The network parameters of the first objective value function are set. Second objective value function And the target loss function to be optimized First objective value function Second objective value function And the target loss function to be optimized The parameters of the soft update coefficient τ and the first value function Second value function And the loss function to be optimized C ψ Soft update of network parameters; Step 6: Initialize the pre-training step number t=1; Step 7: Reset the training environment and set the floating base space manipulator and the non-cooperative target to the initial state; Step 8. In the tth step of the pre-training phase, obtain the current state s from the environment t After that, the current state s t Input control strategy function π θ The network, control strategy function π θ The mean μ of the network output action distribution t and variance σ t , and then sample from the action distribution to obtain the action to be performed by the floating-based space manipulator system in the current state is the action distribution, which is a normal distribution; Floating-based space manipulator system performs action a t , and get reward r after interacting with the environment t pre and the reward to be optimized r t opt , and enter the next state s t+1 ;Tuple (s t ,a t ,r t pre ,r t opt ,s t+1 ) into a buffer; Step 9: Determine whether the task of the current scene is completed, that is, determine whether the number of steps reaches the length of the current scene; If the task of the current scene is not finished, set t = t + 1 and use state s t+1 Return to step eight; Otherwise, if the task of the current scene is completed, determine whether the strategy needs to be updated based on the strategy update frequency; If the policy update frequency is reached, a batch of tuples is sampled from the buffer to update the control policy function π θ Network parameters, first value function The network parameters and the second value function The network parameters and the loss function to be optimized C ψ The network parameters and temperature coefficient α are updated using the first value function The network parameters and the second value function The network parameters and loss function to be optimized C ψ The network parameters have a great impact on the first objective value function. The network parameters and the second objective value function The network parameters and target loss function to be optimized Perform soft update on the network parameters, and then execute step 10; Otherwise, if the policy update frequency is not reached, directly execute step 10; Step 10: Determine whether the set number of pre-training steps T has been reached pre ; If the set number of pre-training steps is reached, the pre-training phase ends and step 11 is executed; If the pre-training step number is not reached, set t=t+1 and return to step 7; Step 11: Set the number of training steps to the number of training steps for the main task T main , update the objective function of the task; and initialize the number of training steps t = 1; Step 12: reset the training environment and set the space manipulator and the non-cooperative target to the initial state; Step 13: In the tth step of the training process, get the current state s from the environment t ′, and then the current state s t ′ Input control strategy function π θ The network, control strategy function π θ The network output action distribution The mean μ t ′ and variance σ t ′, from the action distribution The action to be performed by the space manipulator system in the current state is obtained by sampling Space manipulator system performs action a t ′, and obtain reward r after interacting with the environment t pre and the reward to be optimized r t opt , and enter the next state s t ' +1 ; The tuple (s t ′,a t ′,r t pre ,r t opt ,s t ' +1 ) into a buffer; Step 14: Determine whether the task of the current scene is completed, that is, determine whether the number of steps reaches the length of the current scene; If the task of the current scene is not finished, set t = t + 1 and use state s t ' +1 Return to step 13; Otherwise, when the current task is completed, determine whether the strategy needs to be updated based on the strategy update frequency; If the strategy update frequency is reached, a batch of tuples is sampled from the buffer and the control strategy function π is updated using the Lagrange method. θ , value function Q φ , loss function C to be optimized ψ The network parameters and Lagrange multiplier β, where i=1,2; And use the updated value function Q respectively φ The network parameters and loss function to be optimized C ψ The network parameters have an effect on the target value function The network parameters and target loss function to be optimized Perform soft update on the network parameters, and then execute step 15; Otherwise, if the policy update frequency is not reached, directly execute step 15; Step 15: Determine whether the set number of training steps has been reached; If the set number of training steps is reached, the training phase ends and step 16 is executed; If the set number of training steps is not reached, set t=t+1 and return to step 12; Step 16: The network parameters of the control strategy function obtained at the end of the training are used as the trained network parameters, and the floating-based space manipulator system is controlled in real time according to the trained network parameters.
2. According to claim 1, a non-cooperative target capture method based on deep reinforcement learning is characterized in that: The control strategy function π θ , the first value function and the second value function All are represented using MLP neural network.
3. A non-cooperative target capture method based on deep reinforcement learning according to claim 2, characterized in that: The pre-training reward function is: Among them, r t pre is the pre-trained reward function, v t is the target linear velocity, ||·||2 represents the 2-norm, and ||·|| represents the 1-norm.
4. The non-cooperative target capture method based on deep reinforcement learning according to claim 3, characterized in that: The optimization reward function is: Among them, r t opt To optimize the reward function.
5. The non-cooperative target capture method based on deep reinforcement learning according to claim 4, characterized in that: In the pre-training stage, the first value function The network and the second value function The loss function of the network is: in, is the loss function of the network with the i-th value function, i = 1, 2, Ε represents the expectation, B represents the set of tuples sampled during update, y(s t+1 ,r t pre ,d t ) is the objective function of the value function; Among them, γ represents the discount rate of cumulative rewards, d t is a Boolean value indicating whether the task is finished, α is the temperature coefficient, π θ (a t+1 |s t+1 ) represents the control strategy π θ In status t+1 Next output action a t+1 , ln is the natural logarithm; The i-th value function The network parameters are updated by formula (6): in, represents the gradient, |B| represents the number of tuples contained in set B, and λ φ is the value function learning rate; Control strategy function π θ The network parameters θ are updated by equation (7): in, By π θ After reparameterization, λ θ is the learning rate of the control strategy function; Loss function C to be optimized ψ The network parameter ψ is updated by equation (9): Among them, λ ψ is the learning rate of the loss function to be optimized; The temperature coefficient α is updated by equation (10): Among them, λ α is the temperature coefficient learning rate, H(π θ (s t ))=-∫π θ (s t )ln(π θ (s t ))ds t is the control strategy function π θ The information entropy of is the target entropy.
6. A non-cooperative target capture method based on deep reinforcement learning according to claim 5, characterized in that: The capacity of the buffer is not less than 3 times of the batch during the strategy update. If the buffer capacity limit is reached, the earliest tuple stored in the current buffer is deleted and the new tuple is saved in the buffer.
7. The non-cooperative target capture method based on deep reinforcement learning according to claim 6, characterized in that: In step 11, the specific process of updating the objective function of the task is: Among them, D KL [·] is the KL divergence, is the value function Q φ The corresponding pre-training result, d is the threshold of KL divergence; Then the updated objective function to be optimized is equivalent to: Among them, L m is the updated objective function to be optimized, and β is the Lagrange multiplier; L m The gradient with respect to θ is: in, It's Q φ The corresponding target value function; Control strategy function π during training θ The network parameters are updated by formula (14): The loss function C to be optimized during training ψ The network parameters are updated by formula (15): The Lagrange multiplier β is updated by equation (16): Among them, λ β is the learning rate of the Lagrange multiplier.
