Mechanical arm grabbing method and system based on improved multi-thread PPO reinforcement learning

Abstract

The invention discloses a mechanical arm grabbing method and system based on improved multi-thread PPO reinforcement learning. The method comprises the steps that the environment state is initialized; the multiple working sub-threads start to work in parallel, initial state information of the mechanical arm is obtained and input into an Actor strategy network, and the Actor strategy network adopts an improved PPO reinforcement learning algorithm to conduct data collection; putting the collected data into an experience pool, and calculating all advantage values; performing regularization processing on the advantage value, and performing scaling processing on the reward; the Actor strategy network and the Critic evaluation network are updated; and according to the latest action instruction output by the Actor strategy network, joint movement and clamping jaw action of the mechanical arm are controlled. According to the method, the PPO algorithm is improved by improving the reward function, the advantage value regularization and the reward scaling method, and the training efficiency of the algorithm and the grabbing performance of the mechanical arm are improved through multi-thread parallel computing, so that the mechanical arm can rapidly and accurately grab the target object in a complex environment.

Description

Technical Field

[0001] The present invention belongs to the field of mechanical control, relates to a robot arm control technology, and specifically relates to a robot arm grasping method and system based on improved multi-threaded PPO reinforcement learning. Background Art

[0002] Modern robotic arms are widely used in industrial manufacturing and medical fields as operating equipment, such as parts grasping and assembly in industrial production, material handling and sorting, and surgical assistance in medical processes. Completing such a series of complex operations requires more efficient and stable robotic arm control. At present, in the field of robotic arm control, traditional control methods such as adaptive PD control and fuzzy adaptive control are mostly based on mathematical models, and these methods are often accompanied by nonlinear, strong coupling, time-varying and poor adaptability, which affects the working efficiency of robotic arms. With the development of deep reinforcement learning, its application in the field of robot control has gradually increased. Deep reinforcement learning combines the perception ability of deep learning and the decision-making ability of reinforcement learning, and has achieved breakthrough results in many high-dimensional actual continuous control problems, providing new ideas and methods for solving robotic arm grasping problems. Compared with traditional robotic arm control, deep reinforcement learning can make the robotic arm have higher flexibility in control tasks, is easy to apply to various tasks, and has better robustness and autonomous learning ability.

[0003] Zach et al. used the representative algorithm DDPG (Deep Deterministic Policy Gradient) in deep reinforcement learning to complete a series of experiments. They found that an important feature of the DDPG algorithm is to solve the control problem in continuous space, and the learning process is more stable and efficient. At the same time, they also noticed that the DDPG algorithm has problems such as low sample utilization efficiency. Similarly, typical deep reinforcement learning algorithms such as DQN and Sarsa have problems such as low learning efficiency, difficulty in setting rewards, and poor control effects in continuous high-dimensional action spaces. The PPO algorithm, as a deep reinforcement learning algorithm based on policy gradient, has the characteristics of simplicity, efficiency, easy implementation, and high stability. It has application potential in robotic arm control, but there are also some limitations that need further research and improvement.

[0004] The PPO (Proximal Policy Optimization) algorithm is a policy gradient-based reinforcement learning algorithm used to optimize the policy network to maximize the cumulative reward. It is relatively insensitive to the selection of hyperparameters, which reduces the difficulty of parameter adjustment to a certain extent, has good robustness, and has high training efficiency after optimization, and has the characteristics of high efficiency and good adaptability.

[0005] There are still some problems when applying the existing PPO algorithm to robot arm control. The existing PPO robot arm control often has low sample processing efficiency. It requires a large number of samples to train in order to obtain an effective control strategy, which leads to a long training time and increases the time consumption of data collection. In addition, there are multiple hyperparameters in the PPO algorithm, such as learning rate, discount factor, etc. Determining the appropriate hyperparameters also requires a large number of experimental samples and parameter adjustment experience, which requires improving the processing efficiency of samples. In addition, there is also the problem of difficulty in setting the reward function, which often leads to the phenomenon of sparse reward function. There are few or even almost no rewards in the middle process of completing the entire task, which affects the learning efficiency and effect. Summary of the invention

[0006] Purpose of the invention: In order to overcome the deficiencies in the prior art, a robotic arm grasping method and system based on improved multi-threaded PPO reinforcement learning is provided, the PPO algorithm is improved by improving the reward function, advantage value regularization and reward scaling method, and multi-threaded parallel computing is used to improve the training efficiency of the algorithm and the grasping performance of the robotic arm, so that the robotic arm can quickly and accurately grasp the target object in a complex environment.

[0007] Technical solution: To achieve the above purpose, the present invention provides a robotic arm grasping method based on improved multi-threaded PPO reinforcement learning, comprising the following steps:

[0008] S1: Initialize the environment state, including the number of working sub-threads, Actor strategy network and Critic evaluation network;

[0009] S2: Multiple working sub-threads start working in parallel, obtain the initial state information of the robot arm, and input it into the Actor strategy network. The Actor strategy network uses the improved PPO reinforcement learning algorithm to collect data;

[0010] S3: Put the collected data into the experience pool and use the generalized advantage estimation (GAE) algorithm to calculate all advantage values;

[0011] S4: Regularize the advantage value and scale the reward;

[0012] S5: Update the Actor strategy network and Critic evaluation network based on the regularized advantage value and scaled reward;

[0013] S6: Control the joint movement and gripper action of the robot arm according to the latest action instructions output by the Actor strategy network.

[0014] Furthermore, the state information in step S2 includes the joint angle of the robot arm, the state of the gripper, and the position and distance information of the target object relative to the robot arm.

[0015] Furthermore, the data collection method of the Actor strategy network in step S2 includes: introducing multiple working sub-threads into the PPO reinforcement learning algorithm, setting up a shared network, the Actor strategy network gives weights to the shared network, and distributes them to each working sub-thread through the shared network, and each sub-thread interacts with the environment.

[0016] Furthermore, a replay buffer is set in the experience pool of step S3, and the four-tuple data (state, action, reward, next state) sampled from the environment each time is stored in the replay buffer. During training, some data are randomly sampled from the buffer for training to improve sample efficiency. Each sample can be used multiple times, which is suitable for gradient learning of neural networks.

[0017] Furthermore, the calculation formula of the advantage value in step S3 is:

[0018]

[0019] Among them, γ is the discount factor, λ is a constant between 0 and 1, V represents the state value, r t+l is the total reward of a single step at the t+lth moment, l is the number of steps experienced, s t+l is the state at time t+l, s t+l+1 It is the state at time t+l+1.

[0020] Advantage value reflects the pros and cons of each action in the same state. The state value is the average of the action values, and the advantage value is the action value minus the state value. The higher the advantage value, the higher the reward that the action can bring.

[0021] Furthermore, the regularization process in step S4 includes:

[0022] Calculate the mean and standard deviation of all advantage values ​​in the experience pool;

[0023] Each advantage value is regularized according to the mean and standard deviation, as follows:

[0024]

[0025] Among them A t is the original advantage value, μ A is the mean of all advantage values ​​in the experience pool, σ A is the standard deviation of all advantage values ​​in the experience pool, ∈ is a small constant to prevent the denominator from being zero, is the advantage value after regularization.

[0026] Furthermore, the scaling process in step S4 includes:

[0027] A1: Design reward function:

[0028] R=γr d +I success r success

[0029]

[0030] Calculate the Euclidean distance d between the center of the robot gripper and the center of the target object in three-dimensional space; give a smaller reward when the distance is far, and increase the reward as the distance decreases; use a simple nonlinear function to implement this relationship; where k d is a scaling factor that determines the effect of distance on the reward, d max is a preset maximum distance threshold; when d max =d, r d = 0; when d = 0 (i.e. the gripper successfully grasps the target object), r d =k d ; This encourages the robot arm to gradually approach the target object; I success is the indicator function. When the capture is successful, I success =1; when the crawl fails, I success =0;

[0031] A2: Calculate the total discount of a reward in a rolling manner. The formula is R t =γR t-1 +r t , where R t is the cumulative reward calculated at each step, γ is the discount factor, r t is the reward of the current step;

[0032] Dynamically calculate the standard deviation σ of the reward at each step R ;

[0033] The reward is obtained by scaling the standard deviation, and the formula is where r t ' is the scaled reward.

[0034] Furthermore, in step S5, the Actor strategy network is updated using the clip function, as follows:

[0035]

[0036] Among them, θ is the model parameter, ∈ is the cropping coefficient, is the policy function before update, π θ is the updated policy function, a t is the action being performed at time t, is the expectation of the objective function at time t. The clip function can be used to implement small batch updates in multiple training steps, solving the problem of difficult step size determination in the policy gradient algorithm.

[0037] Furthermore, in step S5, the improved loss function is used to update the critic evaluation network, as follows:

[0038]

[0039] Among them, rt' is the reward at time t', t is the current time, V is the state value function, is the model parameter, and st is the state at time t.

[0040] The present invention also provides a robotic arm grasping system based on improved multi-threaded PPO reinforcement learning, comprising:

[0041] A state acquisition module is used to acquire state information of the robot arm, including the joint angle of the robot arm, the state of the gripper, and the position and distance information of the target object relative to the robot arm;

[0042] A multi-threaded computing module, including multiple parallel computing threads, is used to execute a computing process based on an improved PPO reinforcement learning algorithm;

[0043] An action execution module is used to control the robot arm to perform corresponding actions according to the action instructions output by the multi-threaded computing module;

[0044] The reward feedback module is used to calculate the reward value based on the action results of the robot arm and the grasping task objectives, and feed the reward value back to the multi-threaded computing module.

[0045] The state and reward data received in the present invention, as well as the outputs of the Actor and Critic, are all put into the experience pool as an experience. When the experience pool is full, it means that the data collection for this round is completed and the training phase is entered. A batch of data is sampled from the experience playback buffer, and the advantage value is calculated using the GAE algorithm. The GAE algorithm more accurately evaluates the degree of advantage of each action by considering the reward information of multiple time steps in the future. The Actor network updates the parameters according to the advantage value and the sampled data by minimizing the objective function PP0-clip function of the PPO algorithm. The Critic network updates the parameters according to the state and reward information through the loss function. During the update process, the steps of data collection, advantage value calculation and network update are repeated until the network converges, that is, the grasping strategy of the robot arm reaches a stable state and can effectively perform grasping tasks under different circumstances.

[0046] The present invention improves the PPO algorithm and applies it to the robot grasping control. The improvements include:

[0047] Advantage value regularization: In the PPO algorithm, the advantage value reflects the degree of advantage of taking a certain action in a certain state relative to the average level. By calculating the mean and standard deviation of the advantage value and regularizing each advantage value, the distribution of the advantage value can be made more stable, avoiding the occurrence of too large or too small advantage values ​​that have too much impact on the strategy update. This helps to improve the convergence speed of the algorithm, reduce fluctuations during training, and enable the robot arm to learn effective grasping strategies more quickly.

[0048] Reward scaling: Reward scaling adjusts the reward by rolling the discounted sum of the reward and dynamically calculating the standard deviation. This can avoid gradient explosion or gradient vanishing problems caused by too large or too small reward values, making the training of the value function more stable. At the same time, reasonable reward scaling can enable the robot arm to better perceive the difference in reward signals, improve learning efficiency, and find the optimal grasping strategy faster.

[0049] Multi-threaded parallel work: Multiple threads work simultaneously, and parallel processing is achieved in various links such as data collection, advantage value calculation, regularization and scaling, and model update, which greatly shortens the calculation time and improves the training efficiency. In the data collection stage, multiple data collection threads can interact with the environment at the same time to obtain more sample data. At the same time, the buffer experience playback buffer is used to obtain sample data and put it into the buffer. When there are enough sample data in the buffer, samples are taken from the buffer to train the update strategy. In the model update stage, multiple model update threads can simultaneously calculate the gradients of different parts and update network parameters to accelerate the convergence of the model. This parallel computing method makes full use of the multi-core processor resources of modern computers, enabling the robotic arm grasping system to complete training and optimization in a shorter time, improve the real-time performance and response speed of the system, and adapt to complex and changeable grasping task environments.

[0050] Beneficial effects: Compared with the prior art, the present invention improves the PPO algorithm by introducing a multi-threaded method combined with reward function improvement, advantage value regularization and reward scaling, and applies it to the robot arm grasping control. By setting a reasonable state space, action space and reward function, the robot arm can obtain sufficient information to understand the environment and state, thereby obtaining a more reasonable Actor model and Critic model, so that the robot arm can learn strategies more efficiently and accurately. Then, the grasping success rate evaluation curve and reward evaluation curve are used to conduct comparative experiments with the original PPO algorithm, which improves the stability and convergence speed of the algorithm, enables the robot arm to learn effective grasping strategies faster, and improves the grasping success rate. The application of multi-threaded parallel computing technology and the use of experience playback buffer greatly improve the computing efficiency of the system, shorten the training time, enhance the real-time performance and response speed of the system, and can better adapt to the complex and changeable robot arm grasping task environment. The system can automatically adjust the grasping strategy according to the characteristics of the grasping task and environmental changes, has strong adaptability and robustness, and has broad application prospects in industrial manufacturing, logistics and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flow chart of the robotic arm grasping system based on the multi-threaded PPO algorithm;

[0052] Figure 2 It is a multi-threaded PPO algorithm flow chart;

[0053] Figure 3 This is the specific application framework diagram of the PPO algorithm;

[0054] Figure 4 It is the flow chart of the robot arm grasping experiment;

[0055] Figure 5 is a schematic diagram of the simulation environment;

[0056] Figure 6 It is the training reward graph of the robot grasping in the simulation environment;

[0057] Figure 7 It is a graph of the training success rate of the robot grasping in a simulation environment;

[0058] Figure 8 This is a comparison chart of the original PPO algorithm and the improved multi-threaded PPO algorithm training. DETAILED DESCRIPTION

[0059] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0060] Embodiment 1:

[0061] like Figure 1 to Figure 3 As shown, this embodiment provides a robotic arm grasping method based on improved multi-threaded PPO reinforcement learning, comprising the following steps:

[0062] S1: Initialize the environment state, including the number of working sub-threads, Actor strategy network and Critic evaluation network;

[0063] When the system starts, the sensors in the state acquisition module are first calibrated and initialized to ensure that the state information of the robot arm can be accurately obtained. At the same time, each thread in the multi-threaded computing module is initialized, memory space is allocated, thread parameters are set, and the number of threads is set to 5 working sub-threads. The Actor and Critic networks are initialized, the network structure and initial parameters are set, and the learning rates of the Actor and Critic networks are set to 0.0003, such as the number of network layers, the number of neurons in each layer, and the learning rate.

[0064] S2: Multiple working sub-threads start working in parallel, obtain the initial state information of the robot arm, and input it into the Actor strategy network. The Actor strategy network uses the improved PPO reinforcement learning algorithm to collect data;

[0065] The status information includes the robot arm joint angle, the gripper state, and the position and distance information of the target object relative to the robot arm;

[0066] The data structure of the state space of the robot grasping is determined by the type of environmental information obtained by the robot. Since the end-to-end design is adopted, the data obtained by the agent from the environment are the position of the target object and the changes in joint angles and angular velocity. Therefore, the state space is designed as S = (S arm , S env ,S ops , S d ), where S arm =(J1, J2, J3, J4, J5, J6), J represents the angle vector of the joint, S d represents the Euclidean distance between the gripper and the object, S ops represents the relative position between the gripper and the target object, S env Indicates the open and closed state of the gripper, with a value of 1 for closed and 0 for open. The robot motion space is the position of the end of the robot, the state of the parallel angle between the two fingers, and the joints, that is, the four-dimensional vector [x, y, z, θ] T and T=(t i), x represents the coordinate of the end of the robot arm on the x-axis, y represents the coordinate of the end of the robot arm on the y-axis, z represents the coordinate of the end of the robot arm on the z-axis, θ represents the two-finger gripper state, i.e., open and closed, and t represents the angle at the i joint.

[0067] The data collection method of the Actor policy network includes: the Actor policy grid input layer clearly marks the received robot arm state information, including six joint angles, six joint angular velocities, the three-dimensional position coordinates of the target object, and the distance between the gripper and the target object, with a total of 16 input neurons. The input layer is connected to the first hidden layer, which has 64 neurons. The lines between neurons represent the transmission of data, and each neuron applies the ReLU activation function. The output of the first hidden layer is used as the input of the second hidden layer. The second hidden layer has 32 neurons and also uses the ReLU activation function. The output of the second hidden layer is connected to the output layer, which has 7 neurons, corresponding to the torque instructions of the six joints and one gripper control instruction. Multiple working sub-threads are introduced into the PPO reinforcement learning algorithm, and a shared network is set. The Actor policy network and the shared network usually have the same network structure. When the Actor policy network completes a round of training and updates its own weights, it will send a signal (a Boolean variable) to indicate that the weights are ready to update the shared network. The weights represent the mapping relationship between the input features (state information) and the output actions. The above shared network is a fully connected layer, and each element w of the weight matrix W ij Represents the connection strength from the i-th input neuron to the j-th output neuron. During the training process, the weight W is continuously updated. Therefore, the Actor strategy network gives the weight to the shared network, which is then distributed to each working sub-thread through the shared network, and each sub-thread interacts with the environment.

[0068] S3: Put the collected data into the experience pool and use the generalized advantage estimation (GAE) algorithm to calculate all advantage values;

[0069] A replay buffer is set in the experience pool to store the four-tuple data (state, action, reward, next state) sampled from the environment each time in the replay buffer. During training, some data are randomly sampled from the buffer for training to improve sample efficiency. Each sample can be used multiple times, which is suitable for gradient learning of neural networks.

[0070] The data collection thread outputs the action to the environment, and the environment returns the next state and reward information based on the action of the robot arm. These interaction data are stored in the experience pool, which manages data in a first-in-first-out (FIFO) manner. When the experience pool is full, the oldest data is automatically discarded.

[0071] The calculation formula of advantage value is:

[0072]

[0073] Among them, γ is the discount factor, λ is a constant between 0 and 1, V represents the state value, r t+l is the total reward of a single step at the t+lth moment, l is the number of steps experienced, s t+l is the state at time t+l, s t+l+1 It is the state at time t+l+1.

[0074] The advantage value reflects the pros and cons of each action in the same state. The state value is the average of the action value, and the advantage value is the action value minus the state value. The higher the advantage value, the higher the reward the action can bring. In this process, it is necessary to pay attention to the data synchronization between multiple threads to ensure the consistency and correctness of the advantage value and reward data.

[0075] S4: Regularize the advantage value and scale the reward;

[0076] Regularization processing includes:

[0077] Calculate the mean and standard deviation of all advantage values ​​in the experience pool;

[0078] Each advantage value is regularized according to the mean and standard deviation, as follows:

[0079]

[0080] Among them A t is the original advantage value, μ A is the mean of all advantage values ​​in the experience pool, σ A is the standard deviation of all advantage values ​​in the experience pool, ∈ is a small constant to prevent the denominator from being zero, is the advantage value after regularization.

[0081] Scaling processing includes:

[0082] A1: Design reward function:

[0083] R=γr d +I success r success

[0084]

[0085] Calculate the Euclidean distance d between the center of the robot gripper and the center of the target object in three-dimensional space; give a smaller reward when the distance is far, and increase the reward as the distance decreases; use a simple nonlinear function to implement this relationship; where k dis a scaling factor that determines the effect of distance on the reward, d max is a preset maximum distance threshold; when d max =d, r d = 0; when d = 0 (i.e. the gripper successfully grasps the target object), r d =k d ; This encourages the robot arm to gradually approach the target object; I success is the indicator function. When the capture is successful, I success =1; when the crawl fails, I success =0;

[0086] A2: Calculate the total discount of a reward in a rolling manner. The formula is R t =γR t-1 +r t , where R t is the cumulative reward calculated at each step, γ is the discount factor, r t is the reward of the current step;

[0087] Dynamically calculate the standard deviation σ of the reward at each step R ;

[0088] The reward is obtained by scaling the standard deviation, and the formula is where r t ' is the scaled reward.

[0089] S5: Update the Actor strategy network and Critic evaluation network based on the regularized advantage value and scaled reward;

[0090] The Actor strategy network is updated using the clip function as follows:

[0091]

[0092] Among them, θ is the model parameter, ∈ is the cropping coefficient, is the policy function before update, π θ is the updated policy function, a t is the action being performed at time t, is the expectation of the objective function at time t. The clip function can be used to implement small batch updates in multiple training steps, solving the problem of difficult step size determination in the policy gradient algorithm.

[0093] The improved loss function is used to update the Critic evaluation network as follows:

[0094]

[0095] Among them, rt' is the reward at time t', t is the current time, V is the state value function, is the model parameter, and st is the state at time t.

[0096] S6: Control the joint movement and gripper action of the robot arm according to the latest action instructions output by the Actor strategy network.

[0097] Embodiment 2:

[0098] Based on the robotic arm grasping method of Example 1, this embodiment provides a robotic arm grasping system based on improved multi-threaded PPO reinforcement learning, including:

[0099] A state acquisition module is used to acquire state information of the robot arm, including the joint angle of the robot arm, the state of the gripper, and the position and distance information of the target object relative to the robot arm;

[0100] A multi-threaded computing module, including multiple parallel computing threads, is used to execute a computing process based on an improved PPO reinforcement learning algorithm;

[0101] An action execution module is used to control the robot arm to perform corresponding actions according to the action instructions output by the multi-threaded computing module;

[0102] The reward feedback module is used to calculate the reward value based on the action results of the robot arm and the grasping task objectives, and feed the reward value back to the multi-threaded computing module.

[0103] Embodiment 3:

[0104] In this embodiment, the method of embodiment 1 and the system of embodiment 2 are simulated. The process is as follows Figure 4 The environment of the simulation experiment in this embodiment is as follows Figure 5 As shown. Figure 6 The training reward return data shown in Figure 7 The training success rate data shown is from Figure 6 and Figure 7 It can be seen that the improved PPO algorithm combined with multi-threading in the reward function graph of the first 1000 episodes can achieve good learning effects in this environment, and the grasping success rate gradually increases in a smaller number of episodes, approaches 1 more closely, and converges more stably.

[0105] This embodiment also obtains Figure 8 The training comparison data shown is from Figure 8 It can be seen that the original PPO algorithm robotic arm grasping success rate (0_1) is relatively unstable after gradually increasing, while the improved PPO algorithm robotic arm grasping success rate (1_1) is higher in the same number of episodes and gradually increases and tends to be stable.

Claims

1. A robotic arm grasping method based on improved multi-threaded PPO reinforcement learning, characterized in that: The steps include: S1: Initialize the environment state, including the number of working sub-threads, Actor strategy network and Critic evaluation network; S2: Multiple working sub-threads start working in parallel, obtain the initial state information of the robot arm, and input it into the Actor strategy network. The Actor strategy network uses the improved PPO reinforcement learning algorithm to collect data; S3: Put the collected data into the experience pool and use the generalized advantage estimation (GAE) algorithm to calculate all advantage values; S4: Regularize the advantage value and scale the reward; S5: Update the Actor strategy network and Critic evaluation network based on the regularized advantage value and scaled reward; S6: Control the joint movement and gripper action of the robot arm according to the latest action instructions output by the Actor strategy network.

2. According to claim 1, a robotic arm grasping method based on improved multi-threaded PPO reinforcement learning is characterized in that: The state information in step S2 includes the joint angle of the robot arm, the state of the gripper, and the position and distance information of the target object relative to the robot arm.

3. According to claim 1, a robotic arm grasping method based on improved multi-threaded PPO reinforcement learning is characterized in that: The data collection method of the Actor strategy network in step S2 includes: introducing multiple working sub-threads into the PPO reinforcement learning algorithm, setting up a shared network, the Actor strategy network gives weights to the shared network, and distributes them to each working sub-thread through the shared network, and each sub-thread interacts with the environment.

4. According to claim 1, a robotic arm grasping method based on improved multi-threaded PPO reinforcement learning is characterized in that: A playback buffer is set in the experience pool of step S3, and the four-tuple data sampled from the environment each time is stored in the playback buffer. During training, some data are randomly sampled from the buffer for training.

5. The robotic arm grasping method based on improved multi-threaded PPO reinforcement learning according to claim 1 is characterized in that: The calculation formula of the advantage value in step S3 is: Among them, γ is the discount factor, λ is a constant between 0 and 1, V represents the state value, r t+l is the total reward of a single step at the t+lth moment, l is the number of steps experienced, s t+l is the state at time t+l, s t+l+1 It is the state at time t+l+1.

6. The robotic arm grasping method based on improved multi-threaded PPO reinforcement learning according to claim 1 is characterized in that: The regularization process in step S4 includes: Calculate the mean and standard deviation of all advantage values ​​in the experience pool; Each advantage value is regularized according to the mean and standard deviation, as follows: Among them A t is the original advantage value, μ A is the mean of all advantage values ​​in the experience pool, σ A is the standard deviation of all advantage values ​​in the experience pool, ∈ is a small constant to prevent the denominator from being zero, is the advantage value after regularization.

7. The robotic arm grasping method based on improved multi-threaded PPO reinforcement learning according to claim 1 is characterized in that: The scaling process in step S4 includes: A1: Design reward function: R=γr d +I success r success Calculate the Euclidean distance d between the center of the robot gripper and the center of the target object in three-dimensional space; give a smaller reward when the distance is far, and increase the reward as the distance decreases; use a simple nonlinear function to implement this relationship; where k d is a scaling factor that determines the effect of distance on the reward, d max is a preset maximum distance threshold; when d max =d, r d =0; when d = 0, r d =k d ; This encourages the robot arm to gradually approach the target object; I success is the indicator function. When the capture is successful, I success =1; when the crawl fails, I success =0; A2: Calculate the total discount of a reward in a rolling manner. The formula is R t =γR t-1 +r t , where R t is the cumulative reward calculated at each step, γ is the discount factor, r t is the reward of the current step; Dynamically calculate the standard deviation σ of the reward at each step R ; The reward is obtained by scaling the standard deviation, and the formula is where r t ' is the scaled reward.

8. The robotic arm grasping method based on improved multi-threaded PPO reinforcement learning according to claim 1 is characterized in that: In step S5, the Actor strategy network is updated using the clip function, as follows: Among them, θ is the model parameter, ∈ is the cropping coefficient, is the policy function before update, π θ is the updated policy function, a t is the action being performed at time t, is the expectation of the objective function at time t.

9. The robotic arm grasping method based on improved multi-threaded PPO reinforcement learning according to claim 1, characterized in that: In step S5, the improved loss function is used to update the Critic evaluation network, as follows: Among them, rt' is the reward at time t', t is the current time, V is the state value function, is the model parameter, and st is the state at time t.

10. A robotic arm grasping system based on improved multi-threaded PPO reinforcement learning, characterized in that: include: A state acquisition module is used to acquire state information of the robot arm, including the joint angle of the robot arm, the state of the gripper, and the position and distance information of the target object relative to the robot arm; A multi-threaded computing module, including multiple parallel computing threads, is used to execute a computing process based on an improved PPO reinforcement learning algorithm; An action execution module is used to control the robot arm to perform corresponding actions according to the action instructions output by the multi-threaded computing module; The reward feedback module is used to calculate the reward value based on the action results of the robot arm and the grasping task objectives, and feed the reward value back to the multi-threaded computing module.

Images