A robot reinforcement learning training method based on staged sampling and progress trend reward

Abstract

The application provides a robot reinforcement learning training method based on phased sampling and progress trend reward. First, an improved three-memory bank memory bank structure is introduced, the characteristics of different stages in robot motion planning skill training are combined, and the memory bank two-by-two combination sampling mode is used for targeted experience playback. Subsequently, in the later stage of training, the exploration memory bank experience migration method is used to replace the original expert experience with the high-quality exploration experience obtained after screening, so as to achieve the purposes of self-learning and avoiding overfitting. Finally, the progress trend evaluation method is used for training evaluation on the small range of experience continuously stored in the exploration memory bank, and the evaluation result is expressed as the progress reward of the current robot, so as to improve the task skill learning efficiency of the robot without affecting the learning quality. The method can realize effective and rapid motion planning in a complex obstacle scene, and provides principle support for the generalization of robot skills in different scenes.

Description

Technical Field

[0001] The present invention relates to a robot reinforcement learning training method based on phased sampling and progressive trend reward, belonging to the technical field of robot motion control. Background Art

[0002] With the rapid development of artificial intelligence (AI), the integration of automation technology and intelligent AI has become increasingly close, and the demand for its application in areas such as intelligent production and disaster relief has become more urgent. This has brought with it high demands on robot control systems in areas such as policy execution, trajectory planning, obstacle recognition, and energy optimization. A robot reinforcement learning training method based on phased sampling and progressive reward has emerged precisely in response to these demands. Deep reinforcement learning (DRL), which uses deep neural network models to approximate the value function or policy function in reinforcement learning, serves as an intermediary between AI and traditional robot control. This enables robots to learn goals and strategies that meet system design requirements through continuous interaction and feedback with the environment, continuously self-learning to achieve their mission objectives. This effectively addresses the previous problem of systems being unable to perform tasks using existing fixed coding in unstructured environments. This allows robots to learn and execute tasks based on environmental changes and existing algorithmic frameworks, significantly increasing their flexibility and applicability. In addition, the obstacle avoidance system and simulation system in the algorithm framework can effectively increase the safety of operations and reduce the operating losses of robots, providing technical prerequisites for future robot application fields such as intelligent production, logistics sorting and transmission, disaster relief, medical surgery, etc., and promoting the development of social artificial intelligence towards a smarter, more flexible and safer future. Summary of the Invention

[0003] The technical problem to be solved by the present invention is that traditional robots use a set of inherent memory library sampling methods in different training stages during the self-ability iteration process of updating neural network parameters, resulting in low robot learning efficiency and excessively long training cycles.

[0004] In order to solve the above technical problems, the technical solution of the present invention is to disclose a robot reinforcement learning training method based on phased sampling and progressive trend rewards, which is characterized by comprising the following steps:

[0005] Step 1: Build an improved three-memory bank infrastructure including a tutor memory bank, a feature memory bank, and an exploration memory bank, and further confirm the storage capacity of each of the tutor memory bank, the feature memory bank, and the exploration memory bank. The feature memory bank is further divided into a tutor sub-bank and an exploration sub-bank.

[0006] Step 2: Screen the original tutor data obtained, and fill the screened high-quality tutor data into the tutor memory bank;

[0007] Step 3: Based on the deep reinforcement learning algorithm, the robot's exploration data is obtained through interactive training between the robot and the virtual simulation environment;

[0008] Step 4: Write the exploration data obtained in step 3 into the exploration memory in real time. At the same time, switch the indicator I in the middle of the training process. pm With the later switching indicator I ma , confirm the training stage, and then write the exploration data into the tutor memory bank or feature memory bank according to the training stage, where the training stage includes early training, mid-training, and late training;

[0009] Step 5: Determine the phased combined sampling mode of the memory bank and the target sampling data number of the corresponding memory bank according to the phase determination result in step 4, thereby obtaining batch combined sampling data;

[0010] Step 6: Integrate the combined sampling data in step 5 to update the physical parameters of the training neural network.

[0011] Preferably, in step 1, the memory capacity of the tutor memory bank, the feature memory bank and the exploration memory bank are V T 、V F With V E , and a data set is represented as:

[0012] {s t ,a t ,s t+1 ,r t}

[0013] Where: s t With s t+1 is the state of the robot at the current and next time step; a t The action output by the robot at the current time step; r t The environment reward value obtained by the robot at the current time step.

[0014] Preferably, the state s includes the robot joint angle, the end effector center point posture, the distance and posture difference between the end effector and the target point, the distance between the end effector and the obstacle, and the robot and obstacle collision flag. Then a state is expressed as:

[0015] s t =[I0,I1,I2,I3,I4,I5,I6,x f ,y f ,z f ,θ,γ,β,Dis tg , Po ,Dis tc ,F col ]

[0016] Among them: [I0, I1, I2, I3, I4, I5, I6] are the angle values ​​of the robot's seven joint angles; [x f ,y f , z f ] is the Cartesian coordinate of the center point of the gripper; [θ, γ, β] is the posture of the center point of the gripper; Dis tg is the distance between the center point of the gripper and the center of mass of the target object; P o Dis is the posture difference between the center point of the gripper and the center of mass of the target object; tc F is the distance between the center point of the gripper and the surface of the obstacle; col It is the collision mark between the robot and the obstacle.

[0017] Preferably, in step 2, the original tutor data obtained is derived from the result data of ordinary deep reinforcement learning algorithm training that does not include the deep reinforcement learning algorithm method used by the robot reinforcement learning training method or manual teaching data, and the original tutor data is screened based on round rewards and trajectory end conditions to obtain high-quality tutor data for filling the tutor memory bank data.

[0018] Preferably, the step 3 comprises the following steps:

[0019] Step 301: Initialize the simulation environment:

[0020] In the physical scene platform, a depth camera is used to identify the robot's initial point position, the robot's target point position, the obstacle geometry and posture, and other environmental states, and the state s t Enter the simulation environment;

[0021] Step 302: Initialize the reinforcement learning algorithm:

[0022] For the policy network π(a|s;θ) and the value network q(a,s;w) in the reinforcement learning algorithm, two deep neural networks, an online network and a target network, are constructed respectively to avoid the bootstrap overestimation problem caused by using the temporal difference algorithm to perform gradient descent to update the value network during training, while synchronizing the parameters to the target network.

[0023] Step 303: Perform random exploration based on the algorithm neural network parameters to obtain the current time step action a t , where the online network action selection formula is as follows:

[0024]

[0025] Among them: at is the action taken by the robot at the current time step; μ o is the online network strategy function; s t is the state of the environment at the current time step; θ is the online policy network parameter; is OU noise;

[0026] Step 304: Simulation environment status update:

[0027] The simulation environment is based on the s obtained in step 301 and step 303. t with a t Update the environment to obtain the next time step environment state s t+1 , at the same time, based on the inclusion of the basic reward r bas Reward with a range of progressive trends rag The improved reward function obtains the environmental reward r of the current time step t ;

[0028] Step 305: Integrate data to obtain an exploration data set for the current exploration process. t , a t , s t+1 , r t}.

[0029] Preferably, in step 304, the reward function is specifically:

[0030] For each time step t, obtain the distance set D between the center point of the gripper and the center of mass of the target object within four time steps based on the current time step t list =[Dis t-3 ,Dis t-2 ,Dis t-1 ,Dis t ], select action a according to the deep reinforcement learning algorithm t Interact with the environment to get the next environment state s t+1 Rewards with robots sum , then the total reward r of the agent at time step t sum The formula is as follows:

[0031]

[0032] Where: D idx is the range distance improvement index; η is the range discount rate; μ is the range improvement reward weight; r rag With r bas They are respectively the range progress trend reward and the basic reward.

[0033] Preferably, writing the exploration data into the tutor memory or feature memory in step 4 specifically includes the following steps:

[0034] Step 401: Calculate mid-term switching index I pm With the later switching indicator I ma , the indicator calculation formula is as follows:

[0035]

[0036] in: Indicates that every T time step falls on the mentor sub-library F T Number of experience points; C eachT Indicates the total number of experiences per T time steps; C done Indicates the number of rounds in which the robot touches the target point in each E training round; C eachE represents the total number of E training rounds;

[0037] Step 402: Perform a training phase switching decision, and confirm the training phase of the current time step based on the decision result:

[0038] When I pm When ≥μ, the current training phase is switched to the mid-term phase;

[0039] When I ma When ≥λ, the current training phase is switched to the later phase;

[0040] In other cases, no phase state switching is performed, where μ and λ are both hyperparameters;

[0041] Step 403: Based on the training phase status obtained in step 402, the exploration data obtained in step 3 is written into the tutor memory or feature memory after collision screening:

[0042] If the training is in the early and middle stages of training, the filtered exploration data will be t Dis tg With Dis tc Approaching state, respectively with the approaching target distance threshold N tg and the adjacent obstacle threshold N tc Perform feature classification: When Dis tg ≤N tg When writing feature memory F T Submodule; when Dis tc ≤N tc When writing feature memory F E Submodules;

[0043] If the training is in the late stage of training, the screened exploration data will replace the tutor data in the tutor memory bank to update the tutor data and avoid overfitting.

[0044] Preferably, the details of the phased combined sampling mode of the memory bank and the target sampling data number of the corresponding memory bank in step 5 are:

[0045] Determine the number of target data items B for sampling in the three memory banks T 、B F With B E , the expression of the number of sampled data between each memory bank in the early stage of training is as follows:

[0046]

[0047] The expression for the number of sampled data between each memory bank in the middle and late stages of training is as follows:

[0048]

[0049] in: and are the number of sampled data in the tutor memory bank, the tutor sub-bank in the feature memory bank, the exploration sub-bank in the feature memory bank, and the exploration memory bank at the current time step; V′ FT is the amount of data stored in the mentor sub-library in the feature memory library; V FT is the total capacity of the mentor sub-library in the feature memory library; N is the total amount of data sampled in the batch.

[0050] Traditional robot trajectory planning adopts a single uniform sampling or non-uniform sampling mode for the inherent memory bank throughout the training process, ignoring the robot's learning ability and progress status contained in the experience of different stages, and failing to perform adaptive sampling and rewards based on the latest exploration information, resulting in problems such as low experience utilization, large fluctuations in training results, and repeated execution of bad behaviors. The present invention proposes a robot reinforcement learning training method based on staged sampling and progress trend rewards. By transforming the memory bank structure, it comprehensively considers scene characteristics and task objectives, classifies scene characteristics at different time steps, and preferentially replays special experience. In addition, the present invention performs mode switching and progress trend rewards based on the status information contained in the latest exploration experience, allowing the robot to conduct targeted learning and replay of the acquired experience, with stronger learning ability and robustness, significantly shortening the training cycle, and improving the success rate of training results. It overcomes the limitations of traditional methods such as low experience utilization and unstable training results, and has broad application prospects and important technical significance.

[0051] This paper, focusing on a seven-degree-of-freedom robot, transforms the memory bank structure and introduces an adaptive sampling pattern and range-based progress reward function based on the characteristics of different training stages. This paper proposes a set of training methods and strategies that can achieve effective experience learning in trajectory planning scenarios, shorten training time, and improve planning success rates. Compared with traditional reinforcement learning-based robot memory bank sampling methods, this paper adopts an adaptive sampling strategy based on different training stages and performs progress evaluation on experience gained within a certain range. This method offers significant advantages in improving the efficiency of experience learning, enhancing experience utilization, and shortening the training cycle. It can avoid the shortcomings of existing methods and has broad application prospects and technical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is the actual deployment diagram of the present invention;

[0053] Figure 2 It is a schematic diagram of the overall framework of the present invention;

[0054] Figure 3 It is a detailed schematic flow chart of the present invention. DETAILED DESCRIPTION

[0055] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall equally within the scope limited by the appended claims of the application.

[0056] like Figure 1 As shown, the example of the present invention uses a simulation environment built based on CoppeliaSim simulation software, including: a robotic arm ①, a two-finger robotic gripper ②, an obstacle ③, a base ④, a target grasping point ⑤, and an operating table ⑥. The robotic arm ① is located directly above the base ④ and performs operations. The obstacle ③ is located in the middle area between the robotic arm ① and the target grasping point ⑤, which is used to simulate the situation in real scenes where there are obstacles in the planned path of the robotic arm ① and it is unreachable. The target grasping point ⑤ is located in the area above the operating table ⑥ and is the target point of the trajectory planning task of the robotic arm ①. In the embodiment of the present invention, the robotic arm ① uses the KUKA LBR iiwa 7R800, and the two-finger robotic gripper ② uses the Robotiq 140.

[0057] like Figure 2 As shown, the present invention provides a robot reinforcement learning training method based on phased sampling and progressive trend reward. The details of this method are shown in Figure 3 , the actual deployment is as follows Figure 1 As shown, the following steps are included:

[0058] Step 1: Build the improved three-memory TFE (Tutor Memory, TM; Feature Memory, FM; Explore Memory, EM) infrastructure, and then confirm the storage capacity of each of the three memory banks. Among them, the feature memory bank FM is structurally divided into the tutor sub-bank F T and explore sub-library F E .

[0059] The memory capacity of the three memory banks is V T 、V F With V E , and a data set can be represented as:

[0060] {s t , a t ,s t+1 ,r t}

[0061] Where: s t With s t+1 is the state of the robot at the current and next time step; a t The action output by the robot at the current time step; r t The environment reward value obtained by the robot at the current time step.

[0062] In the embodiment of the present invention, the state s includes the robot joint angles, the end effector center point pose, the distance and pose difference between the end effector and the target point, the distance between the end effector and the obstacle, and the robot-obstacle collision flag. A state can be expressed as:

[0063] s t =[I0,I1,I2,I3,I4,I5,I6,x f ,y f ,z f ,θ,γ,β,Dis tg ,P o ,Dis tc ,F col ]

[0064] Among them: [I0, I1, I2, I3, I4, I5, I6] are the angle values ​​of the robot's seven joint angles; [x f ,y f , z f ] is the Cartesian coordinate of the center point of the gripper; [θ, γ, β] is the posture of the center point of the gripper; Dis tg is the distance between the center point of the gripper and the center of mass of the target object; P o Dis is the posture difference between the center point of the gripper and the center of mass of the target object; tcis the surface distance between the center point of the gripper and the obstacle; F col It is the collision mark between the robot and the obstacle.

[0065] Step 2: Screen the original tutor data obtained and fill the screened high-quality tutor data into the tutor memory bankTM.

[0066] In the embodiments of the present invention, the original tutor data obtained is derived from training results of a common deep reinforcement learning algorithm (not including the method of the embodiments of the present invention) or manual teaching data. Furthermore, by filtering the original tutor data based on round rewards and trajectory endpoint conditions, high-quality tutor data is obtained to populate the Tutor Memory™ data.

[0067] Step 3: Based on a deep reinforcement learning algorithm (SAC is used in this embodiment of the present invention), the robot's exploration data is obtained through interactive training between the robot and a virtual simulation environment (CoppeliaSim is used in this embodiment of the present invention). Specifically, the following steps are included:

[0068] Step 301: Initialize the simulation environment.

[0069] In the physical scene platform, a depth camera is used to identify the robot's initial point position, the robot's target point position, the obstacle geometry and posture, and other environmental states, and the state s t into the simulation environment.

[0070] Step 302: Initialize the reinforcement learning algorithm.

[0071] For the policy network π(a|s;θ) (actor network) and the value network q(a, s; w) (critic network) in the algorithm, two deep neural networks, an online network and a target network, are constructed. This creates four neural networks to avoid the bootstrap overestimation problem caused by using the temporal difference algorithm (TD algorithm) to perform gradient descent updates on the value network during training. Initialize the online policy network and the online value network, and synchronize their parameters with the corresponding target network.

[0072] Step 303: Perform random exploration based on the algorithm neural network parameters to obtain the current time step action a t The online network action selection formula is as follows:

[0073]

[0074] Among them, a t is the action taken by the robot at the current time step; μ o is the online network strategy function; s t is the state of the environment at the current time step; θ is the online policy network parameter; is OU noise.

[0075] Step 304: Simulation environment status is updated.

[0076] The simulation environment is based on the obtained s t with a t Update the environment to obtain the next time step environment state s t+1 At the same time, based on the improved reward function (including the basic reward r bas Reward with a range of progressive trends rag ) Get the current time step environment reward r t , where the specific details of the range improvement trend reward function are:

[0077] For each time step t, obtain the distance set D between the center point of the gripper and the center of mass of the target object within four time steps based on the current time step t list =[Dis t-3 ,Dis t-2 ,Dis t-1 ,Dis t ], select action a according to the algorithm SAC t Interact with the environment to get the next environment state s t+1 Rewards with robots sum , then the total reward r of the agent at time step t sum The formula is as follows:

[0078]

[0079] Where: D idx is the range distance improvement index; η is the range discount rate; μ is the range improvement reward weight; r rag With r bas They are respectively the range progress trend reward and the basic reward.

[0080] Step 305. Integrate data to obtain an exploration data set for the current exploration process. t , a t , s t+1 , r t}.

[0081] Step 4: Write the exploration data obtained in step S3 into the exploration memory in real time. At the same time, according to the mid-term switching indicator I pm With the later switching indicator I ma , confirm the training stage (early training, mid-training, late training), and then write the exploration data into the tutor memory TM or feature memory FM according to the training stage, wherein writing the exploration data into the tutor memory TM or feature memory FM specifically includes the following steps:

[0082] Step 401. Calculate mid-term switching index I pm With the later switching indicator I ma The indicator calculation formula is as follows:

[0083]

[0084] in: Indicates that every T time step falls on the mentor sub-library F T Number of experience points; C eachT Indicates the total number of experiences per T time steps; C done Indicates the number of rounds in which the robot touches the target point in each E training round; C eachE Indicates the total number of E training rounds. pm It indicates the degree to which the current robot absorbs the experience of the expert memory bank and the recognition status of the target point of the current training task. ma It indicates the current robot's ability to reach the target point based on the instructor's experience, guidance and exploration.

[0085] Step 402: Perform a training phase switching decision, and confirm the training phase of the current time step based on the decision result. pm ≥μ, switch the current training phase to the mid-term; when I ma When μ ≥ λ, the current training phase is switched to the later phase. In other cases, the phase state is not switched. μ and λ are both hyperparameters.

[0086] Step 403. According to the training phase status obtained in step 402, the exploration data obtained in step 3 is written into the tutor memory TM or feature memory FM after collision screening. If the training is in the early training phase (Prophase) and the middle training phase (Metaphase), the filtered exploration data is written into the tutor memory TM or feature memory FM according to the status s t Dis tg With Dis tc Approaching state, respectively with the approaching target distance threshold N tg and the adjacent obstacle threshold N tc Perform feature classification. tg ≤N tg When , it indicates that the exploration data is near the target point, which is a high-quality result data with high result value, which can speed up the robot's recognition of the task target and write it into the feature memory F T Submodule. When Dis tc ≤N tc When , it indicates that the exploration data is exploring near the obstacle surface, has a high exploration potential value, and can provide an experience reference for the robot to find the global optimal path, and is written into the feature memory F EIf the training is in the late training phase (Anaphase), the filtered exploration data will be used to replace the tutor data in the tutor memory bank to update the tutor data and avoid overfitting.

[0087] Step 5: Determine the phased combined sampling mode of the memory bank and the target number of sampling data of the corresponding memory bank according to the phase determination result in step S4, so as to obtain batch combined sampling data, specifically:

[0088] Determine the combined sampling mode of the memory bank at different training stages and the number of target data items B for the three memory banks T 、B F With B E The expression for the number of sampled data between each memory bank in the early stage of training is as follows:

[0089]

[0090] The expression for the number of sampled data between each memory bank in the middle and late stages of training is as follows:

[0091]

[0092]

[0093] in, and They are respectively the tutor memory TM of the current time step and the tutor sub-library F in the feature memory FM T , the exploration sub-library F in the feature memory library FM E The number of sampled data in the exploration memory EM; V′ FT is the tutor sub-library F in the feature memory FM T The amount of data stored; V FT is the tutor sub-library F in the feature memory FM T The total capacity of N is the total amount of data sampled in a batch.

Claims

1. A robot reinforcement learning training method based on phased sampling and progressive trend reward, characterized in that: The following steps are involved: Step 1: Build an improved three-memory bank infrastructure including a tutor memory bank, a feature memory bank, and an exploration memory bank, and further confirm the storage capacity of each of the tutor memory bank, the feature memory bank, and the exploration memory bank. The feature memory bank is further divided into a tutor sub-bank and an exploration sub-bank. Step 2: Screen the original tutor data obtained, and fill the screened high-quality tutor data into the tutor memory bank; Step 3: Based on the deep reinforcement learning algorithm, the robot's exploration data is obtained through interactive training between the robot and the virtual simulation environment; Step 4: Write the exploration data obtained in step 3 into the exploration memory in real time. At the same time, switch the indicator I in the middle of the training process. pm With the later switching indicator I ma , confirm the training stage, and then write the exploration data into the tutor memory bank or the feature memory bank according to the training stage, wherein the training stage includes the early training stage, the middle training stage, and the late training stage, wherein writing the exploration data into the tutor memory bank or the feature memory bank specifically includes the following steps: Step 401: Calculate mid-term switching index I pm With the later switching indicator I ma , the indicator calculation formula is as follows: in: Indicates that every T time step falls on the mentor sub-library F T Number of experience points; C eachT Indicates the total number of experiences per T time steps; C done Indicates the number of rounds in which the robot touches the target point in each E training round; C eachE represents the total number of E training rounds; Step 402: Perform a training phase switching decision, and confirm the training phase of the current time step based on the decision result: When I pm When ≥μ, the current training phase is switched to the mid-term phase; When I ma When ≥λ, the current training phase is switched to the later phase; In other cases, no phase state switching is performed, where μ and λ are both hyperparameters; Step 403: Based on the training phase status obtained in step 402, the exploration data obtained in step 3 is written into the tutor memory or feature memory after collision screening: If the training is in the early and middle stages of training, the filtered exploration data will be t Dis tg With Dis tc Approaching state, respectively with the approaching target distance threshold N tg and the adjacent obstacle threshold N tc Perform feature classification: When Dis tg ≤N tg When writing feature memory F T Submodule; when Dis tc ≤N tc When writing feature memory F E Submodules; If the training is in the late stage, the screened exploration data will replace the tutor data in the tutor memory bank to update the tutor data and avoid overfitting; Step 5: Determine the phased combined sampling mode of the memory bank and the target sampling data number of the corresponding memory bank according to the phase determination result in step 4, thereby obtaining batch combined sampling data; Step 6: Integrate the combined sampling data in step 5 to update the physical parameters of the training neural network.

2. A robot reinforcement learning training method based on phased sampling and progressive trend reward as claimed in claim 1, characterized in that: In step 1, the memory capacity of the tutor memory bank, feature memory bank and exploration memory bank are V T 、V F With V E , and a data set is represented as: {s t ,a t ,s t+1 ,r t } Where: s t With s t+1 is the state of the robot at the current and next time step; a t The action output by the robot at the current time step; r t The environment reward value obtained by the robot at the current time step.

3. A robot reinforcement learning training method based on phased sampling and progressive trend reward as claimed in claim 1, characterized in that: The state s includes the robot joint angle, the end effector center point pose, the distance and pose difference between the end effector and the target point, the distance between the end effector and the obstacle, and the robot and obstacle collision flag. A state is represented as: s t =[I0,I1,I2,I3,I4,I5,I6,x f ,y f ,z f ,θ,γ,β,Dis tg P o ,Dis tc ,F col ] Among them: [I0, I1, I2, I3, I4, I5, I6] are the angle values ​​of the robot's seven joint angles; [x f ,y f , z f ] is the Cartesian coordinate of the center point of the gripper; [θ, γ, β] is the posture of the center point of the gripper; Dis tg is the distance between the center point of the gripper and the center of mass of the target object; P o Dis is the posture difference between the center point of the gripper and the center of mass of the target object; tc F is the distance between the center point of the gripper and the surface of the obstacle; col It is the collision mark between the robot and the obstacle.

4. A robot reinforcement learning training method based on phased sampling and progressive trend reward as claimed in claim 1, characterized in that: In step 2, the original tutor data obtained comes from the result data of ordinary deep reinforcement learning algorithm training that does not include the deep reinforcement learning algorithm method used by the robot reinforcement learning training method or manual teaching data, and the original tutor data is screened based on the round reward and trajectory end condition to obtain high-quality tutor data for filling the tutor memory bank data.

5. The robot reinforcement learning training method based on phased sampling and progressive trend reward as claimed in claim 1, characterized in that: The step 3 comprises the following steps: Step 301: Initialize the simulation environment: In the physical scene platform, a depth camera is used to identify the robot's initial point position, the robot's target point position, the obstacle geometry and posture, and other environmental states, and the state s t Enter the simulation environment; Step 302: Initialize the reinforcement learning algorithm: For the policy network π(a|s;θ) and the value network q(a,s;w) in the reinforcement learning algorithm, two deep neural networks, an online network and a target network, are constructed respectively to avoid the bootstrap overestimation problem caused by using the temporal difference algorithm to perform gradient descent to update the value network during training, while synchronizing the parameters to the target network. Step 303: Perform random exploration based on the algorithm neural network parameters to obtain the current time step action a t , where the online network action selection formula is as follows: Among them: a t is the action made by the robot in the current time step; μ o is the online network strategy function; s t is the state of the environment at the current time step; θ is the online policy network parameter; is OU noise; Step 304: Simulation environment status update: The simulation environment is based on the s obtained in step 301 and step 303. t with a t Update the environment to obtain the next time step environment state s t+1 , at the same time, based on the inclusion of the basic reward r bas Reward with a range of progressive trends rag The improved reward function obtains the environmental reward r of the current time step t ; Step 305: Integrate data to obtain an exploration data set for the current exploration process. t , a t , s t+1 , r t }.

6. A robot reinforcement learning training method based on phased sampling and progressive trend reward as claimed in claim 5, characterized in that: In step 304, the reward function is specifically: For each time step t, obtain the distance set D between the center point of the gripper and the center of mass of the target object within four time steps based on the current time step t list =[Dis t-3 ,Dis t-2 ,Dis t-1 ,Dis t ], select action a according to the deep reinforcement learning algorithm t Interact with the environment to get the next environment state s t+1 Rewards with robots sum , then the total reward r of the agent at time step t sum The formula is as follows: Where: D idx is the range distance improvement index; η is the range discount rate; μ is the range improvement reward weight; r rag With r bas They are respectively the range progress trend reward and the basic reward.

7. A robot reinforcement learning training method based on phased sampling and progressive trend reward as claimed in claim 1, characterized in that: The details of the phased combined sampling mode of the memory bank and the target sampling data number of the corresponding memory bank in step 5 are as follows: Determine the number of target data items B for sampling in the three memory banks T 、B F With B E , the expression of the number of sampled data between each memory bank in the early stage of training is as follows: The expression for the number of sampled data between each memory bank in the middle and late stages of training is as follows: in: and are the number of sampled data in the tutor memory bank, the tutor sub-bank in the feature memory bank, the exploration sub-bank in the feature memory bank, and the exploration memory bank at the current time step; V′ FT is the amount of data stored in the mentor sub-library in the feature memory library; V FT is the total capacity of the mentor sub-library in the feature memory library; N is the total amount of data sampled in the batch.

Images