Robot reinforcement learning training method based on staged sampling and progress trend reward

Abstract

The invention provides a robot reinforcement learning training method based on staged sampling and progress trend reward. The method comprises the following steps: firstly, introducing a memory bank structure of an improved three-memory bank, combining characteristics of different stages in robot motion planning skill training, and performing targeted experience playback in a mode of combining and sampling every two memory banks; and then, in the later training stage, an exploration memory bank experience migration mode is adopted, and high-quality exploration experience obtained after screening is used for replacing original expert experience, so that the purposes of self-learning and over-fitting avoiding are achieved. And finally, performing training evaluation on the small-range experience continuously stored in the exploration memory bank by using a progress trend evaluation method, and representing an evaluation result as a progress reward of the current robot, thereby improving the task skill learning efficiency of the robot on the premise of not influencing the learning quality. According to the method, effective and rapid motion planning can be realized in a complex obstacle scene, and principle support is provided for generalization of robot skills in different scenes.

Description

Technical Field

[0001] The invention relates to a robot reinforcement learning training method based on phased sampling and progressive trend reward, belonging to the technical field of robot motion control. Background Art

[0002] With the rapid development of artificial intelligence technology, the integration of automation technology and intelligent AI has become closer, and the demand for applying it to intelligent production and disaster relief has become more urgent. At the same time, it also brings high requirements for the strategy execution, trajectory planning, obstacle recognition, energy optimization and other technical aspects of the robot control system. A robot reinforcement learning training method based on phased sampling and progressive trend reward came into being under this demand background. Deep reinforcement learning (DRL), which uses a deep neural network model to approximate the value function or strategy function in reinforcement learning, plays the role of a medium between artificial intelligence and traditional robot control. This enables the robot to learn the goals and strategies that meet the system design requirements in the continuous interaction and feedback with the environment, and continuously conduct self-learning to achieve the task objectives. This effectively solves the problem that the system cannot use existing fixed codes to perform tasks in unstructured environments in the past. This enables the robot to learn strategies that meet the task requirements according to environmental changes and the existing algorithm framework, and perform tasks, which significantly increases flexibility and applicability. In addition, the obstacle avoidance system and simulation system in the algorithm framework can effectively increase the safety of operations and reduce the operating losses of robots, providing technical prerequisites for future robot application fields such as intelligent production, logistics sorting and transmission, disaster relief, medical surgery, etc., and promoting the development of social artificial intelligence towards a smarter, more flexible and safer future. Summary of the invention

[0003] The technical problem to be solved by the present invention is that traditional robots use a set of inherent memory library sampling methods in different training stages during the self-ability iteration process of updating neural network parameters, resulting in low robot learning efficiency and too long training cycle.

[0004] In order to solve the above technical problems, the technical solution of the present invention is to disclose a robot reinforcement learning training method based on phased sampling and progressive trend reward, which is characterized by comprising the following steps:

[0005] Step 1: Build an improved three-memory base infrastructure including a tutor memory base, a feature memory base, and an exploration memory base, and further confirm the storage capacity of each of the tutor memory base, the feature memory base, and the exploration memory base. The feature memory base is further divided into a tutor sub-base and an exploration sub-base in terms of structure.

[0006] Step 2: Screen the original tutor data obtained, and fill the screened high-quality tutor data into the tutor memory bank;

[0007] Step 3: Based on the deep reinforcement learning algorithm, the robot's exploration data is obtained through interactive training between the robot and the virtual simulation environment;

[0008] Step 4: Write the exploration data obtained in step 3 into the exploration memory in real time. At the same time, switch the indicator I in the middle of the training process. pm Switching indicator I ma , confirm the training stage, and then write the exploration data into the tutor memory bank or feature memory bank according to the training stage, where the training stage includes the early training stage, the middle training stage, and the late training stage;

[0009] Step 5: Determine the phased combined sampling mode of the memory bank and the target sampling data number of the corresponding memory bank according to the phase determination result in step 4, so as to obtain batch combined sampling data;

[0010] Step 6: Integrate the combined sampling data in step 5 to update the physical parameters of the training neural network.

[0011] Preferably, in step 1, the memory capacity of the tutor memory bank, the feature memory bank and the exploration memory bank are V T 、V F With V E , and a data set is represented as:

[0012] {s t ,a t ,s t+1 ,r t}

[0013] Where: s t With s t+1 is the state of the robot at the current and next time step; a t is the action output by the robot at the current time step; r t The environmental reward value obtained by the robot at the current time step.

[0014] Preferably, the state s includes the robot joint angle, the end effector center point posture, the distance and posture difference between the end effector and the target point, the distance between the end effector and the obstacle, and the robot and obstacle collision flag. Then a state is expressed as:

[0015] s t =[I0,I1,I2,I3,I4,I5,I6,x f ,y f ,z f ,θ,γ,β,Dis tg , Po ,Dis tc ,F col ]

[0016] Among them: [I0, I1, I2, I3, I4, I5, I6] are the angle values ​​of the robot's seven joint angles; [x f ,y f , z f ] is the Cartesian coordinate of the center point of the gripper; [θ, γ, β] is the posture of the center point of the gripper; Dis tg is the distance between the center point of the gripper and the center of mass of the target object; P o Dis is the attitude difference between the center point of the gripper and the center of mass of the target object; tc is the surface distance between the center point of the gripper and the obstacle; F col It is the collision mark between the robot and the obstacle.

[0017] Preferably, in step 2, the original tutor data obtained is derived from the result data of ordinary deep reinforcement learning algorithm training that does not include the deep reinforcement learning algorithm method used by the robot reinforcement learning training method or manual teaching data, and the original tutor data is screened based on round rewards and trajectory end conditions to obtain high-quality tutor data for filling the tutor memory bank data.

[0018] Preferably, step 3 comprises the following steps:

[0019] Step 301, initialization of simulation environment:

[0020] In the physical scene platform, a depth camera is used to identify the robot's initial point position, the robot's target point position, the obstacle geometry and posture, and other environmental states, and the state s t Enter the simulation environment;

[0021] Step 302: Initialize the reinforcement learning algorithm:

[0022] For the policy network π(a|s; θ) and the value network q(a, s; w) in the reinforcement learning algorithm, two deep neural networks, an online network and a target network, are constructed respectively to avoid the bootstrap overestimation problem caused by using the temporal difference algorithm to perform gradient descent to update the value network during training, and synchronize the parameters to the target network at the same time;

[0023] Step 303: Perform a random exploration process according to the algorithm neural network parameters to obtain the current time step action a t , where the online network selection action formula is as follows:

[0024]

[0025] Among them: at is the action taken by the robot at the current time step; μ o is the online network strategy function; s t is the state of the environment at the current time step; θ is the online strategy network parameter; is OU noise;

[0026] Step 304: Simulation environment status update:

[0027] The simulation environment is based on the s obtained in step 301 and step 303. t with a t Update the environment to obtain the next time step environment state s t+1 At the same time, based on the inclusion of the basic reward r bas Reward with range-based improvement trend rag The improved reward function obtains the environment reward r of the current time step t ;

[0028] Step 305: Integrate data to obtain an exploration data set for the current exploration process. t , a t ,s t+1 , r t}.

[0029] Preferably, in step 304, the reward function is specifically:

[0030] For each time step t, obtain the distance set D between the center point of the gripper and the center of mass of the target object within four time steps based on the current time step t list =[Dis t-3 ,Dis t-2 ,Dis t-1 ,Dis t ], select action a according to the deep reinforcement learning algorithm t Interact with the environment to get the next environment state s t+1 Rewards with robots sum , then the total reward r of the agent at time step t sum The formula is as follows:

[0031]

[0032] Where: D idx is the range distance improvement index; η is the range discount rate; μ is the range improvement reward weight; r rag With r bas They are respectively range-based improvement trend rewards and basic rewards.

[0033] Preferably, writing the exploration data into the tutor memory or feature memory in step 4 specifically comprises the following steps:

[0034] Step 401: Calculate mid-term switching index I pm Switching indicator I ma , the indicator calculation formula is as follows:

[0035]

[0036] in: Indicates that every T time step falls on the tutor sub-library F T Number of experience points; C eachT Indicates the total number of experiences per T time steps; C done represents the number of rounds in which the robot touches the target point in each E training round; C eachE represents the total number of E training rounds;

[0037] Step 402: Perform a training phase switching determination, and confirm the training phase of the current time step based on the determination result:

[0038] When I pm ≥μ, switch the current training phase to the mid-term;

[0039] When I ma ≥λ, switch the current training stage to the later stage;

[0040] In other cases, no stage state switching is performed, where μ and λ are both hyperparameters;

[0041] Step 403: According to the training phase status obtained in step 402, the exploration data obtained in step 3 is written into the tutor memory or feature memory after collision screening:

[0042] If the training is in the early and middle stages of training, the filtered exploration data will be based on the state s t Dis tg With Dis tc The approaching state is respectively related to the approaching target distance threshold N tg and the adjacent obstacle threshold N tc Perform feature classification: When Dis tg ≤N tg When writing feature memory F T Submodule; when Dis tc ≤N tc When writing feature memory F E Submodules;

[0043] If the training is in the late stage of training, the screened exploration data will replace the tutor data in the tutor memory bank accordingly to update the tutor data and avoid overfitting.

[0044] Preferably, the details of the phased combined sampling mode of the memory bank and the target sampling data number of the corresponding memory bank in step 5 are:

[0045] Determine the number of target data items B for sampling in the three memory banks T , B F With B E , the expression of the number of sampled data between each memory bank in the early stage of training is as follows:

[0046]

[0047] The expression for the number of sampled data between each memory bank in the middle and late stages of training is as follows:

[0048]

[0049] in: and are the number of sampled data in the tutor memory bank, the tutor sub-bank in the feature memory bank, the exploration sub-bank in the feature memory bank, and the exploration memory bank at the current time step; V′ FT is the amount of data stored in the mentor sub-library in the feature memory library; V FT is the total capacity of the tutor sub-library in the feature memory library; N is the total amount of data sampled in the batch.

[0050] In the whole training process, the traditional robot trajectory planning adopts a single uniform sampling or non-uniform sampling mode for the inherent memory bank, ignoring the robot learning ability and progress status contained in the experience of different stages, and not performing adaptive sampling and rewards according to the latest exploration information, which leads to problems such as low experience utilization, large fluctuations in training results, and repeated execution of bad behaviors. The present invention proposes a robot reinforcement learning training method based on phased sampling and progressive trend rewards. By transforming the memory bank structure, the scene characteristics and task objectives are comprehensively considered, and the scene characteristics at different time steps are classified and special experience is preferentially replayed. In addition, the present invention performs mode switching and progressive trend rewards according to the state information contained in the latest exploration experience, so that the robot can conduct targeted learning and replay of the acquired experience, has stronger learning ability and robustness, significantly shortens the training cycle, improves the success rate of training results, overcomes the limitations of low experience utilization and unstable training results in traditional methods, and has broad application prospects and important technical significance.

[0051] The present invention takes a robot with seven degrees of freedom as the research object, transforms the memory bank structure and introduces an adaptive sampling mode and a range-based progressive reward function based on the characteristics of different training stages, and proposes a set of training method strategies that can achieve effective experience learning in trajectory planning scenarios, shorten training time, and improve planning success rate. Compared with the traditional robot memory bank sampling method based on reinforcement learning, the present invention adopts an adaptive sampling strategy according to different training stages and performs progressive evaluation on the experience acquired within a certain range. It has significant advantages in improving the efficiency of experience learning, enhancing the utilization rate of experience, shortening the training cycle, etc., can avoid the defects brought by existing methods, and has broad application prospects and technical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is the actual deployment diagram of the present invention;

[0053] Figure 2 It is a schematic diagram of the overall framework of the present invention;

[0054] Figure 3 It is a detailed schematic flow chart of the present invention. DETAILED DESCRIPTION

[0055] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0056] like Figure 1 As shown, the example of the present invention uses a simulation environment built based on CoppeliaSim simulation software, including: a robot arm ①, a two-finger robot gripper ②, an obstacle ③, a base ④, a target grasping point ⑤, and an operating table ⑥. The robot arm ① is located directly above the base ④ to perform operations. The obstacle ③ is located in the middle area between the robot arm ① and the target grasping point ⑤, which is used to simulate the situation in which there are obstacles in the planned path of the robot arm ① in a real scene and the path is unreachable. The target grasping point ⑤ is located in the area above the operating table ⑥ and is the target point of the trajectory planning task of the robot arm ①. In the embodiment of the present invention, the robot arm ① uses KUKA LBR iiwa 7R800, and the two-finger robot gripper ② uses Robotiq 140.

[0057] like Figure 2 As shown, the present invention provides a robot reinforcement learning training method based on phased sampling and progressive trend reward. The details of this method are shown in Figure 3 , the actual deployment is as follows Figure 1 As shown, the following steps are included:

[0058] Step 1: Build the improved three-memory TFE (Tutor Memory, TM; Feature Memory, FM; Explore Memory, EM) infrastructure, and then confirm the storage capacity of each of the three memories. Among them, the feature memory FM is structurally divided into the tutor sub-library F T and explore sub-library F E .

[0059] The memory capacity of the three memory banks is V T 、V F With V E , and a data set can be represented as:

[0060] {s t , a t ,s t+1 ,r t}

[0061] Where: s t With s t+1 is the state of the robot at the current and next time step; a t is the action output by the robot at the current time step; r t The environmental reward value obtained by the robot at the current time step.

[0062] In the embodiment of the present invention, the state s includes the robot joint angle, the end effector center point posture, the distance and posture difference between the end effector and the target point, the distance between the end effector and the obstacle, and the robot and obstacle collision flag. A state can be expressed as:

[0063] s t =[I0,I1,I2,I3,I4,I5,I6,x f ,y f ,z f ,θ,γ,β,Dis tg ,P o ,Dis tc ,F col ]

[0064] Among them: [I0, I1, I2, I3, I4, I5, I6] are the angle values ​​of the robot's seven joint angles; [x f ,y f , z f ] is the Cartesian coordinate of the center point of the gripper; [θ, γ, β] is the posture of the center point of the gripper; Dis tg is the distance between the center point of the gripper and the center of mass of the target object; P o Dis is the attitude difference between the center point of the gripper and the center of mass of the target object; tcis the surface distance between the center point of the gripper and the obstacle; F col It is the collision mark between the robot and the obstacle.

[0065] Step 2: Screen the original tutor data obtained, and fill the screened high-quality tutor data into the tutor memory bankTM.

[0066] In the embodiment of the present invention, the original tutor data obtained is derived from the result data of training of a common deep reinforcement learning algorithm (not including the method of the embodiment of the present invention) or artificial teaching data. In addition, by screening the original tutor data based on the round reward and the trajectory end condition, high-quality tutor data is obtained to fill the tutor memory bank TM data.

[0067] Step 3: Based on the deep reinforcement learning algorithm (SAC is used in the embodiment of the present invention), the robot's exploration data is obtained by interactive training between the robot and the virtual simulation environment (CoppeliaSim is used in the embodiment of the present invention), which specifically includes the following steps:

[0068] Step 301: Initialize the simulation environment.

[0069] In the physical scene platform, a depth camera is used to identify the robot's initial point position, the robot's target point position, the obstacle geometry and posture, and other environmental states, and the state s t into the simulation environment.

[0070] Step 302: Initialize the reinforcement learning algorithm.

[0071] For the policy network π(a|s; θ) (actor network) and the value network q(a, s; w) (critic network) in the algorithm, two deep neural networks, online network and target network, are constructed respectively, and four neural networks are established to avoid the bootstrap overestimation problem caused by using the temporal difference algorithm (TD algorithm) to perform gradient descent to update the value network during training. Initialize the online policy network and the online value network, and synchronize their parameters to the corresponding target network.

[0072] Step 303: Perform a random exploration process based on the algorithm neural network parameters to obtain the current time step action a t The online network selection action formula is as follows:

[0073]

[0074] Among them, a t is the action taken by the robot at the current time step; μ o is the online network strategy function; s t is the state of the environment at the current time step; θ is the online strategy network parameter; is OU noise.

[0075] Step 304: Simulation environment status is updated.

[0076] The simulation environment is based on the obtained s t with a t Update the environment to obtain the next time step environment state s t+1 At the same time, based on the improved reward function (including the basic reward r bas Reward with range-based improvement trend rag ) Get the current time step environment reward r t , where the specific details of the range improvement trend reward function are:

[0077] For each time step t, obtain the distance set D between the center point of the gripper and the center of mass of the target object within four time steps based on the current time step t list =[Dis t-3 ,Dis t-2 ,Dis t-1 ,Dis t ], select action a according to the algorithm SAC t Interact with the environment to get the next environment state s t+1 Rewards with robots sum , then the total reward r of the agent at time step t sum The formula is as follows:

[0078]

[0079] Where: D idx is the range distance improvement index; η is the range discount rate; μ is the range improvement reward weight; r rag With r bas They are respectively range-based improvement trend rewards and basic rewards.

[0080] Step 305. Integrate the data to obtain an exploration data set for the current exploration process. t , a t ,s t+1 , r t}.

[0081] Step 4: Write the exploration data obtained in step S3 into the exploration memory in real time. At the same time, according to the mid-term switching indicator I pm Switching indicator I ma , confirm the training stage (early training, mid-training, late training), and then write the exploration data into the tutor memory TM or the feature memory FM according to the training stage, wherein writing the exploration data into the tutor memory TM or the feature memory FM specifically includes the following steps:

[0082] Step 401. Calculate mid-term switching index I pm Switching indicator I ma The indicator calculation formula is as follows:

[0083]

[0084] in: Indicates that every T time step falls on the tutor sub-library F T Number of experience points; C eachT Indicates the total number of experiences per T time steps; C done represents the number of rounds in which the robot touches the target point in each E training round; C eachE I represents the total number of E training rounds. pm It indicates the current robot’s absorption degree of the expert memory experience and the recognition status of the target point of the current training task. ma It indicates the current robot's ability to reach the target point based on the instructor's experience, guidance and exploration.

[0085] Step 402: Perform a training phase switching decision, and confirm the training phase of the current time step based on the decision result. pm ≥μ, switch the current training phase to the mid-term; when I ma ≥λ, the current training stage is switched to the later stage. In other cases, the stage state is not switched. μ and λ are both hyperparameters.

[0086] Step 403. According to the training phase status obtained in step 402, the exploration data obtained in step 3 is written into the tutor memory TM or feature memory FM after collision screening. If the training is in the prophase and metaphase, the screened exploration data is written into the tutor memory TM or feature memory FM according to the state s t Dis tg With Dis tc The approaching state is respectively related to the approaching target distance threshold N tg and the adjacent obstacle threshold N tc Perform feature classification. tg ≤N tg When , it indicates that the exploration data is near the target point, which is a high-quality result data with high result value, which can speed up the robot's recognition of the task target and write it into the feature memory F. T Submodule. When Dis tc ≤N tc , it indicates that the exploration data is near the obstacle surface and has a high exploration potential value. It can provide an experience reference for the robot to find the global optimal path and is written into the feature memory F. ESubmodule. If the training is in the late stage of training (Anaphase), the screened exploration data will replace the tutor data in the tutor memory bank, and the tutor data will be updated to avoid overfitting.

[0087] Step 5: Determine the phased combined sampling mode of the memory bank and the target sampling data number of the corresponding memory bank according to the phase determination result in step S4, so as to obtain batch combined sampling data, specifically:

[0088] Determine the combined sampling mode of the memory bank at different training stages and the number of target data items B for sampling the three memory banks T , B F With B E The expression of the number of sampled data between each memory bank in the early stage of training is as follows:

[0089]

[0090] The expression for the number of sampled data between each memory bank in the middle and late stages of training is as follows:

[0091]

[0092]

[0093] in, and They are respectively the tutor memory TM and the tutor sub-library F in the feature memory FM at the current time step T , the exploration sub-library F in the feature memory FM E The number of sampled data in the exploration memory EM; V′ FT is the tutor sub-bank F in the feature memory bank FM T The amount of data stored; V FT is the tutor sub-bank F in the feature memory bank FM T The total capacity of; N is the total amount of data sampled in batches.

Claims

1. A robot reinforcement learning training method based on phased sampling and progressive trend reward, characterized in that: The following steps are involved: Step 1: Build an improved three-memory base infrastructure including a tutor memory base, a feature memory base, and an exploration memory base, and further confirm the storage capacity of each of the tutor memory base, the feature memory base, and the exploration memory base. The feature memory base is further divided into a tutor sub-base and an exploration sub-base in terms of structure. Step 2: Screen the original tutor data obtained, and fill the screened high-quality tutor data into the tutor memory bank; Step 3: Based on the deep reinforcement learning algorithm, the robot's exploration data is obtained through interactive training between the robot and the virtual simulation environment; Step 4: Write the exploration data obtained in step 3 into the exploration memory in real time. At the same time, switch the indicator I in the middle of the training process. pm Switching indicator I ma , confirm the training stage, and then write the exploration data into the tutor memory bank or feature memory bank according to the training stage, where the training stage includes the early training stage, the middle training stage, and the late training stage; Step 5: Determine the phased combined sampling mode of the memory bank and the target sampling data number of the corresponding memory bank according to the phase determination result in step 4, so as to obtain batch combined sampling data; Step 6: Integrate the combined sampling data in step 5 to update the physical parameters of the training neural network.

2. A robot reinforcement learning training method based on phased sampling and progressive trend reward as claimed in claim 1, characterized in that: In step 1, the memory capacity of the tutor memory bank, feature memory bank and exploration memory bank are V T 、V F With V E , and a data set is represented as: {s t ,a t ,s t+1 ,r t } Where: s t With s t+1 is the state of the robot at the current and next time step; a t is the action output by the robot at the current time step; r t The environmental reward value obtained by the robot at the current time step.

3. A robot reinforcement learning training method based on phased sampling and progressive trend reward as claimed in claim 1, characterized in that: The state s includes the robot joint angle, the end effector center point posture, the distance and posture difference between the end effector and the target point, the distance between the end effector and the obstacle, and the robot and obstacle collision flag. A state is represented as: s t =[I0,I1,I2,I3,I4,I5,I6,x f ,y f ,z f ,θ,γ,β,Dis tg ,P o ,Dis tc ,F col ] Among them: [I0, I1, I2, I3, I4, I5, I6] are the angle values ​​of the robot's seven joint angles; [x f ,y f , z f ] is the Cartesian coordinate of the center point of the gripper; [θ, γ, β] is the posture of the center point of the gripper; Dis tg is the distance between the center point of the gripper and the center of mass of the target object; P o Dis is the attitude difference between the center point of the gripper and the center of mass of the target object; tc is the surface distance between the center point of the gripper and the obstacle; F col It is the collision mark between the robot and the obstacle.

4. A robot reinforcement learning training method based on phased sampling and progressive trend reward as claimed in claim 1, characterized in that: In step 2, the original tutor data obtained comes from the result data of ordinary deep reinforcement learning algorithm training that does not include the deep reinforcement learning algorithm method used by the robot reinforcement learning training method or manual teaching data, and the original tutor data is screened based on round rewards and trajectory end conditions to obtain high-quality tutor data for filling the tutor memory bank data.

5. A robot reinforcement learning training method based on phased sampling and progressive trend reward as claimed in claim 1, characterized in that: The step 3 comprises the following steps: Step 301, initialization of simulation environment: In the physical scene platform, a depth camera is used to identify the robot's initial point position, the robot's target point position, the obstacle geometry and posture, and other environmental states, and the state s t Enter the simulation environment; Step 302: Initialize the reinforcement learning algorithm: For the policy network π(a|s; θ) and the value network q(a, s; w) in the reinforcement learning algorithm, two deep neural networks, an online network and a target network, are constructed respectively to avoid the bootstrap overestimation problem caused by using the temporal difference algorithm to perform gradient descent to update the value network during training, and synchronize the parameters to the target network at the same time; Step 303: Perform a random exploration process according to the algorithm neural network parameters to obtain the current time step action a t , where the online network selection action formula is as follows: Among them: a t is the action taken by the robot at the current time step; μ o is the online network strategy function; s t is the environment state at the current time step; θ is the online strategy network parameter; is OU noise; Step 304: Simulation environment status update: The simulation environment is based on the s obtained in step 301 and step 303. t with a t Update the environment to obtain the next time step environment state s t+1 At the same time, based on the inclusion of the basic reward r bas Reward with range-based improvement trend rag The improved reward function obtains the environment reward r of the current time step t ; Step 305: Integrate data to obtain an exploration data set for the current exploration process. t , a t ,s t+1 , r t }.

6. A robot reinforcement learning training method based on phased sampling and progressive trend reward as claimed in claim 5, characterized in that: In step 304, the reward function is specifically: For each time step t, obtain the distance set D between the center point of the gripper and the center of mass of the target object within four time steps based on the current time step t list =[Dis t-3 ,Dis t-2 ,Dis t-1 ,Dis t ], select action a according to the deep reinforcement learning algorithm t Interact with the environment to get the next environment state s t+1 Rewards with robots sum , then the total reward r of the agent at time step t sum The formula is as follows: Where: D idx is the range distance improvement index; η is the range discount rate; μ is the range improvement reward weight; r rag With r bas They are respectively range-based improvement trend rewards and basic rewards.

7. A robot reinforcement learning training method based on phased sampling and progressive trend reward as claimed in claim 1, characterized in that: Writing the exploration data into the tutor memory or feature memory in step 4 specifically includes the following steps: Step 401: Calculate the mid-term switching index I pm Switching indicator I ma , the indicator calculation formula is as follows: in: Indicates that every T time step falls on the tutor sub-library F T Number of experience points; C eachT Indicates the total number of experiences per T time steps; C done represents the number of rounds in which the robot touches the target point in each E training round; C eachE represents the total number of E training rounds; Step 402: Perform a training phase switching determination, and confirm the training phase of the current time step based on the determination result: When I pm ≥μ, switch the current training phase to the mid-term; When I ma ≥λ, switch the current training stage to the later stage; In other cases, no stage state switching is performed, where μ and λ are both hyperparameters; Step 403: According to the training phase status obtained in step 402, the exploration data obtained in step 3 is written into the tutor memory or feature memory after collision screening: If the training is in the early and middle stages of training, the filtered exploration data will be based on the state s t Dis tg With Dis tc The approaching state is respectively related to the approaching target distance threshold N tg and the adjacent obstacle threshold N tc Perform feature classification: When Dis tg ≤N tg When writing feature memory F T Submodule; when Dis tc ≤N tc When writing feature memory F E Submodules; If the training is in the late stage of training, the screened exploration data will replace the tutor data in the tutor memory bank accordingly to update the tutor data and avoid overfitting.

8. A robot reinforcement learning training method based on phased sampling and progressive trend reward as claimed in claim 1, characterized in that: The details of the phased combined sampling mode of the memory bank and the target sampling data number of the corresponding memory bank in step 5 are as follows: Determine the number of target data items B for sampling in the three memory banks T , B F With B E , the expression of the number of sampled data between each memory bank in the early stage of training is as follows: The expression for the number of sampled data between each memory bank in the middle and late stages of training is as follows: in: and are the number of sampled data in the tutor memory bank, the tutor sub-bank in the feature memory bank, the exploration sub-bank in the feature memory bank, and the exploration memory bank at the current time step; V′ FT is the amount of data stored in the mentor sub-library in the feature memory library; V FT is the total capacity of the tutor sub-library in the feature memory library; N is the total amount of data sampled in the batch.

Images