A target capture method for space robots based on model-assisted reinforcement learning

Abstract

This invention proposes a model-assisted reinforcement learning-based method for target capture in space robots, comprising: Step 1: deriving the dynamic constraints of a free-flying space robot system and building an environment model; Step 2: setting appropriate parameters and objective functions for model-free reinforcement learning training; Step 3: designing a deep Lagrangian neural network framework and training the system dynamic model, and deriving a state transition model; Step 4: using the parameters from Step 2 as a reference, incorporating the state transition model obtained in Step 3 to generate virtual experience to assist in completing the reinforcement learning training process. This invention can improve the sample efficiency, convergence speed, and speed and accuracy of training a controller model for grasping moving targets in reinforcement learning; the overall logic and principle are clear and simple, applicable to different systems and various working conditions, and have strong generalization ability; the use of a deep Lagrangian neural network ensures the reliability of the state transition model.

Description

Technical Field

[0001] This invention belongs to the field of space free-flying robot control, specifically involving a dynamic capture method for space flying robots based on reinforcement learning of a deep Lagrange neural network-assisted model. Background Technology

[0002] Free-flying space robots are a type of robotic system capable of operating autonomously or semi-autonomously in microgravity environments. As the core execution unit in space missions, they have wide applications in satellite maintenance, space station auxiliary operations, and deep space exploration.

[0003] In the design of controllers for free-flying robots to grasp moving targets, traditional methods mainly rely on precise mathematical models, typically employing Newtonian classical mechanics and mechanics of materials to derive dynamic equations, such as the Lagrange equation and Hamiltonian equation. However, these methods face numerous challenges in practical applications, primarily due to the uncertainty of model parameters and the difficulty of modeling in complex environments.

[0004] With the rapid development of artificial intelligence technology in the field of robotics, machine learning methods have been gradually introduced into the controller design of free-flying robots. These methods mainly fall into two categories: The first is data-driven modeling, which uses deep learning techniques (including deep Lagrangian neural networks and Manhattan neural networks) to obtain the parameters or equations of the robot system, given the robot's dynamic constraints and equation structure. Subsequently, traditional control algorithms such as PID and MPC are used to design the controller, combined with the trained system model. However, the challenge of this method lies in obtaining an accurate and reliable system model. The second category utilizes reinforcement learning to train the robot's controller end-to-end to perform specific tasks. However, pure model-free reinforcement learning methods suffer from long training cycles, difficulties in parameter tuning, and insufficient controller accuracy.

[0005] In recent years, to improve the efficiency of training samples and the accuracy of models in reinforcement learning, scholars have proposed a model-based reinforcement learning method. This method mainly generates virtual experience by having the robot learn an environment model (primarily including a state transition model and an environment reward model), and then samples this virtual experience in combination with real and virtual experience to update the policy network. The key to this model-based reinforcement learning algorithm lies in the accuracy of the robot's learning of the environment model.

[0006] In summary, traditional modeling and control methods struggle to analyze dynamic models, while methods combining machine learning suffer from high training costs, low success rates in training complex models, and inaccuracies. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention proposes a moving target capture method based on reinforcement learning using a deep Lagrange neural network-assisted model. The method specifically includes the following steps:

[0008] Step 1: Derive the dynamic constraints of the space-free-flying robot system and build an environmental model;

[0009] Step 2: Set appropriate parameters and objective function to perform model-free reinforcement learning training;

[0010] Step 3: Design a deep Lagrange neural network framework and train the system dynamics model, and derive the state transition model;

[0011] Step 4: Using the parameters from Step 2 as a reference, add the state transition model obtained in Step 3 to generate virtual experience to assist in completing the reinforcement learning training process.

[0012] The specific steps of step 1 are as follows:

[0013] Step 1.1: The general expression for the dynamic model of a space-free-flying robot system is as follows:

[0014]

[0015] Where q is the generalized state; H(q), G(q) represents the system's inertia matrix, Coriolis matrix, and torque due to gravity, respectively; τ is the generalized force or torque.

[0016] Step 1.2: Build an environment model, which includes the position and motion equation of the moving target, the location of the robot system, time and time step interval, and the number of training rounds;

[0017] Step 1.3: Determine the state space, action space, and reward function. The experience pool set is defined as follows:

[0018]

[0019] Where, max t represents the maximum step size in each round; p represents the maximum number of rounds of training.

[0020] s represents the state variable in the state space, denoted as s. 'a' represents the action variable in the action space, denoted as a = τ.

[0021] In step 1.3, t, δt, and ep represent the time after t time steps in each round, the time step interval, and the round number in which the training takes place, respectively; the ordered pair (t, ep) represents the system's current time step t in round ep; P eef Pbase , These represent the spatial coordinates of the end effector, base, base target, and moving target in the inertial coordinate system, respectively; the reward function includes sparse reward and distance reward, etc.

[0022] The specific steps of step 2 are as follows:

[0023] Step 2.1: Determine the control policy function and reinforcement learning neural network of the robot system, set the training parameters for reinforcement learning, and perform model-free offline reinforcement learning training. The policy function is updated based on the empirical data recorded in the data, according to the objective function J(π).

[0024] Step 2.2: Record the reward obtained in each round of training and select a reward threshold r. m After m rounds of training, the reward reaches r. m Then, the controller model trained in the m-th round is used to drive the robot system to grasp the moving target, and the empirical information of ep=m is collected and made into a dataset:

[0025]

[0026] In this context, variables with the subscript 't' represent the values ​​assigned to those variables at time step 't', and q... t , and τ t The state variable s can be obtained from the experience pool t or action variable a t Obtain directly from the middle, Defined as:

[0027]

[0028] The reinforcement learning neural network mentioned in step 2.1 is an MLP neural network, and the policy function is π. θ Where θ is the network parameter controlling the policy function; reinforcement learning training parameters include the number of network layers L and the number of neurons in each layer n. L The maximum number of experience records that the experience pool can hold (batch), and the learning rate α. learn The maximum number of training rounds p1, the discount rate γ, and the extraction of i2 empirical data points for updating every i1 steps, etc.

[0029] The specific steps of step 3 are as follows:

[0030] Step 3.1: Set the loss function according to the Lagrange equation After setting the training parameters, including the number of network layers l and the maximum number of training epochs, a deep Lagrange neural network is trained using a subset of S to obtain the system dynamics model.

[0031] Step 3.2: Based on the dynamic model Obtain the state transition model This leads to a virtual experience.

[0032] Step 3.2 State transition model Represented as (s t ,a t )→(r t f ,s t+1 f That is, based on the current state and action (s) t ,a t Predict the state at the next moment and the reward value (r) obtained. t f ,s t+1 f The virtual experience includes s t ,a t ,r t f ,s t+1 f ;

[0033] The process of generating a virtual experience is as follows:

[0034] According to (s) t ,a t ) to get q t , P eef,t ,P base,t , τ t ;according to q t , τ t Predicted According to the prediction Get q t+1 , Use the following formula:

[0035]

[0036] According to q t+1 Obtain P eef,t+1 and P base,t+1 ; All of these can be obtained by sensors; according to s t+1 Calculate r t .

[0037] Where, according to q t+1 Obtain P eef,t+1 and Pbase,t+1 There are two methods: one is to derive it from the kinematic model of the robot system, that is, to use the Jacobian matrix of the spatial robot system to obtain the pose of the end effector and the base based on the joint spatial position, and thus obtain P. eef,t+1 P base,t+1 Secondly, a supervised learning method is used, first recording a large amount of data. t q in t and [P] eef,t ,P base,t Then, the Jacobian matrix is ​​trained using a BP neural network. Since the relationship between the two is linear and relatively simple, it is very easy to predict using a BP neural network.

[0038] The specific steps of step 4 are as follows:

[0039] Step 4.1: Generate virtual experience: Assume the robot system is in state s t The virtual experience begins to be generated at this time. It can be determined by the policy function π θ Obtained by The conclusion is Generate a virtual experience get Reuse Generate the next virtual experience And so on to generate n f Virtual experience, using sets Let it be represented, and the definition is as follows:

[0040]

[0041] Step 4.2: Using the parameters set in Step 2 as a reference, perform reinforcement learning training on the model. After training, a controller that can accurately grasp moving targets is obtained.

[0042] Step 4.2 The specific parameter settings are as follows:

[0043] n f The settings can balance the ratio of virtual experience to real experience in the experience pool; an excessively large n f This will lead to a longer training time per round, n f A value close to i1 can be set; the number of training rounds p2 can be appropriately reduced based on p1, specifically, its selection can be based on the following process:

[0044] Use the formula time MF ×p1=time MF ×m+time MB ×P2+time DeLaNCalculate the number of retraining rounds P2 required for model reinforcement learning with a time consumption similar to traditional model-free reinforcement learning methods, where time... MF and time MB These represent model-free reinforcement learning and adding [the model], respectively. The time required for each round in the subsequent model reinforcement learning process; time DeLaN This represents the time spent training the deep Lagrange neural network. Then, a suitable p2 is chosen based on the following constraints:

[0045] P2 < p2 + m < p1, p2 ∈ Z;

[0046] All other parameters can be the same as those set in step 2.

[0047] Step 4.2 The parameter θ of the policy function is updated in the following way during the training process:

[0048] In each round, the robot system generates n every i1 steps. f Virtual experience Then Integration In the middle, then the robot system in Extract i2 empirical data points and update the parameters θ of the policy function based on the optimization objective function J(π).

[0049] Beneficial effects:

[0050] This invention improves the sample efficiency, convergence speed, and speed and accuracy of training a controller model for grasping moving targets by adding a state transition model trained by a deep Lagrange neural network to the traditional model-free reinforcement learning.

[0051] The overall logic and principle of this invention are clear and simple, applicable to a large number of different systems and various working conditions, and have strong generalization ability;

[0052] This invention addresses the problem of using offline reinforcement learning to train a controller for grasping moving targets in a space-free-flying robot, and uses a deep Lagrange neural network to make the state transition model reliable. Attached Figure Description

[0053] Figure 1 This is a flowchart of the method of the present invention;

[0054] Figure 2 This is a model diagram for simulation testing;

[0055] Figure 3 The acceleration error curves of each joint in the deep Lagrangian training dynamics model for simulation testing are shown.

[0056] Figure 4 The graph shows the change in training reward between this method and traditional model-free reinforcement learning in simulation tests.

[0057] Figure 5 This is a graph showing the error changes in the grasping process compared to the traditional model-free reinforcement learning method during simulation testing. Detailed Implementation

[0058] The present invention will be described in detail below with reference to specific embodiments.

[0059] This embodiment presents an example of a 12-DOF free-space flight robot (hereinafter referred to as the space robot) based on a deep Lagrange neural network-assisted SAC reinforcement learning algorithm, capturing a moving target at a constant speed. The space robot system is configured to ignore gravity, and the 12 degrees of freedom include 3 rotational degrees of freedom and 3 translational degrees of freedom of the base, and 6 degrees of freedom of the robotic arm (e.g., ...). Figure 2 (As shown). Figure 1 The flowchart of the method of the present invention is shown below, and the specific steps of the method are as follows:

[0060] Step 1: For a space-based free-flying robot system that ignores gravity, has a fully controllable base, and is equipped with a 6-DOF robotic arm, the Lagrange equations are:

[0061]

[0062] Where q is defined as:

[0063] q = [q m1 ,q m2 ,...,q m6 ,q b1 ,q b2 ,...,q b6 ]

[0064] Where [q] m1 ,q m2 ,...,q m6 ] represents the rotation angle of the 6 joints of a 6-DOF robotic arm; [q b1 ,q b2 ,q b3 [1] represents the Euler angle of the ball joint of the controllable base; q b4 ,q b5 ,q b6 H(q) and represent the distances the controllable base moves along the X, Y, and Z axes, respectively. τ represents the system's inertia matrix and Coriolis matrix, respectively; τ is defined as a vector composed of the forces or moments of the system's 12 joints.

[0065] Step 2: Set δt = 0.002s, the maximum number of training rounds p1 = 100 (to ensure convergence during training and relative accuracy of the trained controller model), and the maximum step size for each round max. t =1500.

[0066] Furthermore, the coordinates of the target position of the base are set as follows:

[0067]

[0068] The trajectory equation for the moving target is set as follows:

[0069]

[0070] The initial position of the base is (0,0,0), and the initial position of the end effector is approximately (0.17(m),0.1(m),0.1(m)).

[0071] Step 3: Define the state variable s and the action variable a as follows:

[0072]

[0073] a=τ

[0074] Where [A,B,...] represents a new matrix obtained by concatenating the columns of matrices A, B, etc., with the same number of rows. eef ,P base ∈R 1 ×3 Defined as:

[0075]

[0076] The matrix with the superscript T indicates the transpose of the matrix, (x eef ,y eef ,z eef ) and (x base ,y base ,z base These represent the position coordinates of the end effector and the base in a three-dimensional inertial coordinate system, respectively.

[0077] Let be the spatial coordinates of the base target and the moving target in the inertial coordinate system, respectively; i is defined as:

[0078]

[0079] Where d(m) represents the tracking error, which is defined as follows:

[0080]

[0081] Where ||2| represents the 2-norm of the matrix. t All variables involved can be acquired by the system's sensors or derived through simple calculations.

[0082] Define the reward function as follows:

[0083] r = reward sparse +reward distance +penalty action

[0084] in:

[0085]

[0086] penalty action =-5‖a‖2

[0087] Then, the clamping force of the actuator is set to increase as d decreases in order to capture the moving target.

[0088] Define the experience pool set:

[0089]

[0090] Step 4: Determine the control strategy function π based on the motion space dimension of the space robot. θ θ represents the network parameters of the control policy function. The networks involved in reinforcement learning are all represented by MLP neural networks. The training parameters for SAC reinforcement learning are set for training.

[0091] Furthermore, let i1 = 50, i2 = 32, L = 4, n L =128, learning rate α learn =0.001, entropy regularization coefficient α = 0.1, discount rate γ = 0.99. The optimization objective is:

[0092]

[0093] in: It is the estimated expected return, r j It is the immediate return obtained at the time of index j.

[0094] Step 5: In this case, after 18 rounds of training, the reward reaches the set r. m =Reward value of 10000. Then collect experience data from the capture process at ep=18 and create a dataset:

[0095]

[0096] The following steps 6, 7, and 8 are all processes of training the system dynamics model using a deep Lagrange neural network.

[0097] Step 6: Design the architecture of the deep Lagrange neural network. Based on the Lagrange equation (1) for a space-free-flying robot, transform H(q) into:

[0098] H(q)=L(q,φ)L(q,φ) T

[0099] Where, L(q,φ) T The transpose of matrix L(q,φ); φ is the parameter of the MLP architecture neural network, which can be obtained by minimizing violations of physical laws, and its optimization objective is:

[0100]

[0101] Among them, the loss function Defined as mean squared error, i.e.:

[0102]

[0103] Where f and f -1 The inverse kinematic function of the system is represented as:

[0104]

[0105] The estimated value of the inverse kinematics output containing the parameter φ is defined as follows:

[0106]

[0107]

[0108] Then the objective function is defined as:

[0109]

[0110] Here, Ω(φ) represents the L2 norm of the network weights. λ is the regularization parameter, which adjusts the model complexity and the strength of the regularization. Deep Lagrange neural networks can be trained by optimizing this objective function.

[0111] Step 7: Randomly select 750 sets of data from dataset S in Step 4 and name them training set S1. Use the data in S1 to train the dynamic model according to Step 5.

[0112] Step 8: Set the network parameters, including l=3, epoch=30000, λ=0.5. We can then train an estimated dynamic model.

[0113] The following steps 9-12 are... State transition model The process.

[0114] Step 9: According to (s) t ,a t ) can be used to obtain q t , P eef,t ,P base,t , i t , τ t ,according to q t , τ t Predicted

[0115] Step 10: Based on the prediction Get q t+1 , The formula is as follows:

[0116]

[0117] Step 11: Based on q t+1 Obtain P eef,t+1 and P base,t+1 There are two methods: one is to derive it based on the kinematic model of the robot system, that is, to obtain the pose of the end effector and the base by using the Jacobian matrix of the spatial robot system based on the joint spatial positions, and thus obtain P. eef,t+1 P base,t+1 Secondly, a supervised learning method is used, first recording a large amount of data. t q in t and [P] eef,t ,P base,t Then, a BP neural network is used to train the Jacobian matrix. Since the relationship between the two is linear and relatively simple, prediction using a BP neural network is also very easy to achieve. Based on the training results and q t+1 Then P can be derived. eef,t+1 and P base,t+1 Due to the deterministic nature of this dynamic model, this case uses the first method to complete this step.

[0118] Step 12: All of these can be obtained from sensors, and i t+1 and r t This can be derived from step 2. Thus, a state transition model can be obtained.

[0119] Steps 13 and 14 are based on and state s at time t t The process of generating 20 virtual experiences.

[0120] Step 13: Assume the space robot is in state s t The virtual experience begins to be generated at this time. It can be determined by the policy function π θ Obtained by The conclusion is Generate a virtual experience

[0121] Step 14: Obtain Then return to step 13 to generate the next virtual experience. This process is repeated to generate 20 virtual experiences. (Using a set...) To represent this n f A set of virtual experiences, defined as follows:

[0122]

[0123] Step 15: Based on time MF ×p1=time MF ×m+time MB ×P2+time DeLaN Find P2. Furthermore, time... MF =3min,time MB =4min,time DeLaN =12min, P2=59. Let p2=72.

[0124] Step 16: Based on steps 4 and 14, perform model-based reinforcement learning training. Set up a process where 20 virtual experiences are generated every 50 steps during training. Then Integration In the middle, and then in 32 data points were extracted for updating.

[0125] Step 17: After training, create a graph showing the change in reward for each round of the two reinforcement learning training processes. (For easier comparison, add...) To demonstrate the superiority of assisted reinforcement learning, we selected the number of rounds in which model-assisted reinforcement learning significantly improved rewards in a short period of time (v1 = 32), and the number of rounds in which model-free reinforcement learning took slightly longer to reach v1 but yielded higher rewards during training (v2 = 55). We then selected the number of rounds in which model-assisted and model-free reinforcement learning yielded the highest rewards during training (v3 = 65 and v4 = 88, respectively).

[0126] Step 18: Test the ability of the controller model obtained from model-free reinforcement learning to grasp moving targets in rounds m, v2, and v4. Then test the addition of... The model reinforcement learning yielded the controller model's ability to grasp moving targets in rounds v1 and v3. Controller performance was analyzed using d as a function of t, with the time step when the error first fell below 0.05m representing the controller model's speed and the time step when the error reached its minimum representing the controller model's accuracy.

[0127] Simulation experiment:

[0128] To verify the effectiveness of the present invention, the algorithm of the present invention was tested in the MuJoCo simulation environment using the methods described in the above specific embodiments. The simulation involved training and testing a controller for grasping and moving targets on a space-based free-flying robot with a 6-DOF robotic arm and a fully controllable 6-DOF base. Figure 2 The model diagram is shown below. Figure 3 The graph shows the change of acceleration error over time steps for 12 joints of the dynamic model trained by the deep Lagrange neural network (1200 sets of data were extracted from 1500 time steps for testing). Figure 4 The graph shows the change in rewards during the training process of the object model and the model-based method in step 17. Figure 5 The graph shows the error variation over time for the five models trained in step 18 during the grasping process. It can be seen that the model-based method can train a controller that can quickly control the error below 0.05m, with a minimum error of around 0.02m. In contrast, the traditional model-free method is slightly inferior and, after reaching the minimum error, shows a tendency for the error to increase, indicating the instability of the controller. Experiments show that the model-based method... It can optimize the training process of reinforcement learning, making the trained controller model more accurate.

[0129] Therefore, it can be seen that the deep Lagrange neural network of the present invention can effectively assist reinforcement learning in training the controller of a space-free-flying robot for grasping moving targets.

[0130] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

[0131] This invention may have other embodiments. Without departing from the spirit and essence of this invention, those skilled in the art can make various corresponding changes and modifications according to this invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A method for target acquisition of a space robot based on model-assisted reinforcement learning, the method specifically including the following steps: Step 1: Derive the dynamic constraints of the space-free-flying robot system and build an environmental model; Step 2: Set appropriate parameters and objective function to perform model-free reinforcement learning training; Step 3: Design a deep Lagrange neural network framework and train the system dynamics model, and derive the state transition model; Step 4: Using the parameters from Step 2 as a reference, add the state transition model obtained in Step 3 to generate virtual experience to assist in completing the reinforcement learning training process; The specific steps of step 2 are as follows: Step 2.1: Determine the control policy function and reinforcement learning neural network of the robot system, set the training parameters for reinforcement learning, and perform model-free offline reinforcement learning training. The policy function is updated based on the recorded empirical data according to the objective function J(π); Step 2.2: Record the reward obtained in each round of training and select a reward threshold r. m After m rounds of training, the reward reaches r. m Then, the controller model trained in the m-th round is used to drive the robot system to grasp the moving target, and the empirical information of ep=m is collected and made into a dataset: In this context, variables with the subscript 't' represent the values ​​assigned to those variables at time step 't', and q... t , and τ t The state variable s can be obtained from the experience pool t or action variable a t Obtain directly from the middle, Defined as: The specific steps of step 3 are as follows: Step 3.1: Set the loss function l according to the Lagrange equation and set the training parameters, including the number of network layers l and the maximum number of training epochs, etc. Train the deep Lagrange neural network using a subset of S to obtain the system dynamics model. Step 3.2: Based on the dynamic model Obtain the state transition model This leads to a virtual experience.

2. The space robot target acquisition method based on model-assisted reinforcement learning according to claim 1, characterized in that, The specific steps of step 1 are as follows: Step 1.1: The general expression for the dynamic model of a space-free-flying robot system is as follows: Where q is the generalized state; H(q), G(q) represents the system's inertia matrix, Coriolis matrix, and torque due to gravity, respectively; τ is the generalized force or torque. Step 1.2: Build an environment model, which includes the position and motion equation of the moving target, the location of the robot system, time and time step interval, and the number of training rounds; Step 1.3: Determine the state space, action space, and reward function. The experience pool set is defined as follows: Where, max t represents the maximum step size in each round; p represents the maximum number of rounds of training. s represents the state variable in the state space, denoted as s. 'a' represents the action variable in the action space, denoted as a = τ.

3. The space robot target acquisition method based on model-assisted reinforcement learning according to claim 2, characterized in that, In step 1.3, t, δt, and ep represent the time after t time steps in each round, the time step interval, and the round number in which the training takes place, respectively; the ordered pair (t, ep) represents the time step t of the ep-th round. These represent the spatial coordinates of the end effector, base, base target, and moving target in the inertial coordinate system, respectively; the reward function includes sparse reward and distance reward, etc.

4. The space robot target acquisition method based on model-assisted reinforcement learning according to claim 1, characterized in that, The reinforcement learning neural network mentioned in step 2.1 is an MLP neural network, and the policy function is π. θ Where θ is the network parameter controlling the policy function; reinforcement learning training parameters include the number of network layers L and the number of neurons in each layer n. L The maximum number of experience records that the experience pool can hold (batch), and the learning rate α. learn The maximum number of training rounds p1, the discount rate γ, and the extraction of i2 empirical data points for updating every i1 steps, etc.

5. The space robot target acquisition method based on model-assisted reinforcement learning according to claim 1, characterized in that, Step 3.2 State transition model Represented as That is, based on the current state and action (s) t ,a t Predict the state at the next moment and the reward value (r) obtained. t f ,s t+1 f The virtual experience includes s t ,a t ,r t f ,s t+1 f .

6. The space robot target acquisition method based on model-assisted reinforcement learning according to claim 1, characterized in that, The specific steps of step 4 are as follows: Step 4.1: Generate virtual experience: Assume the robot system is in state s t The virtual experience begins to be generated at this time. It can be determined by the policy function π θ Obtained by The conclusion is Generate a virtual experience get Reuse Generate the next virtual experience And so on to generate n f Virtual experience, using sets Let it be represented, and the definition is as follows: Step 4.2: Using the parameters set in Step 2 as a reference, perform reinforcement learning training on the model. After training, a controller that can accurately grasp moving targets is obtained.

7. The space robot target acquisition method based on model-assisted reinforcement learning according to claim 6, characterized in that, Step 4.2 The specific parameter settings are as follows: n f Let it be a value close to i1; the number of training rounds p2 is appropriately reduced from p1, and the selection process is as follows: Use the formula time MF ×p1=time MF ×m+time MB ×P2+time DeLaN Calculate the number of retraining rounds P2 required for model reinforcement learning with a time consumption similar to traditional model-free reinforcement learning methods, where time... MF and time MB These represent model-free reinforcement learning and adding [the model], respectively. The time required for each round in the subsequent model reinforcement learning process; time DeLaN Let p2 represent the time spent training the deep Lagrange neural network, and then choose a suitable p2 according to the following constraints: P2 < p2 + m < p1, p2 ∈ Z; All other parameters can be the same as those set in step 2.

8. The space robot target acquisition method based on model-assisted reinforcement learning according to claim 6, characterized in that, Step 4.2 The parameter θ of the policy function is updated in the following way during the training process: In each round, the robot system generates n every i1 steps. f Virtual experience Then Integration In the middle, then the robot system in Extract i2 empirical data points and update the parameters θ of the policy function based on the optimization objective function J(π).

Images