A mechanical arm grabbing method and system based on improved multi-thread PPO reinforcement learning

Abstract

The application discloses a mechanical arm grabbing method and system based on improved multi-thread PPO reinforcement learning, which comprises the following steps: initializing an environment state; a plurality of working sub-threads start parallel work, obtain initial state information of the mechanical arm, and input the initial state information into an Actor policy network; the Actor policy network adopts an improved PPO reinforcement learning algorithm to collect data; the collected data is put into an experience pool, and all advantage values are calculated; the advantage values are subjected to regularization processing, and rewards are subjected to scaling processing; the Actor policy network and a Critic evaluation network are updated; and the latest action instruction output by the Actor policy network is used to control joint movement and gripper action of the mechanical arm. The PPO algorithm is improved by improving a reward function, advantage value regularization and reward scaling method, and the training efficiency of the algorithm and the grabbing performance of the mechanical arm are improved by using multi-thread parallel calculation, so that the mechanical arm can quickly and accurately grab a target object in a complex environment.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of mechanical control, and relates to mechanical arm control technology, in particular to a mechanical arm grasping method and system based on improved multi-thread PPO reinforcement learning. BACKGROUND

[0002] Modern mechanical arms are widely used in industrial manufacturing and medical fields as operating devices, such as part grasping and assembly in industrial production, material handling and sorting, surgical assistance in medical processes, etc. The mechanical arm control needs to be more efficient and stable to complete such a series of complex operations. At present, in the field of mechanical arm control, traditional control methods such as adaptive PD control and fuzzy adaptive control are mostly based on mathematical models, and these methods are often accompanied by characteristics such as nonlinearity, strong coupling, time-varying and poor adaptive ability, thereby affecting the working efficiency of the mechanical arm. With the development of deep reinforcement learning, its application in the field of robot control has gradually increased. Deep reinforcement learning combines the perception ability of deep learning and the decision-making ability of reinforcement learning, and has made breakthroughs in many high-dimensional actual continuous control problems, providing new ideas and methods for solving the mechanical arm grasping problem. Compared with traditional mechanical arm control, deep reinforcement learning can make the mechanical arm have higher flexibility in control tasks, can be easily applied to various different tasks, and has better robustness and autonomous learning ability.

[0003] Zach et al. used the representative algorithm DDPG (Deep Deterministic Policy Gradient) in deep reinforcement learning to complete a series of experiments, and found that an important feature of the DDPG algorithm is to solve the control problem in continuous space, and the learning process is more stable and efficient. At the same time, they also noticed that the DDPG algorithm has problems such as low sample utilization efficiency. Similarly, typical deep reinforcement learning algorithms such as DQN and Sarsa also have problems such as low learning efficiency, difficulty in setting reward, poor control effect, etc. in continuous high-dimensional action space. The PPO algorithm, as a deep reinforcement learning algorithm based on policy gradient, has the characteristics of simplicity, efficiency, easy implementation, stability, etc., and has application potential in mechanical arm control, but also has some limitations that need to be further researched and improved.

[0004] The PPO (Proximal Policy Optimization) algorithm, i.e. the proximal policy optimization algorithm, is a reinforcement learning algorithm based on policy gradient, which is used to optimize the policy network to obtain the maximum cumulative reward. It is relatively not very sensitive to the selection of hyperparameters, to some extent, reducing the difficulty of parameter tuning, having good robustness, and having high efficiency, adaptability, etc. after optimization.

[0005] There are still some problems in the application of existing PPO algorithm to robot arm control. The existing PPO robot arm control often has low sample processing efficiency, and a large number of samples are needed to train to obtain an effective control strategy, which leads to a long training time and increases the time consumption of data acquisition. In addition, there are multiple hyperparameters in the PPO algorithm, such as learning rate, discount factor, etc., and it also requires a large number of experimental samples and parameter tuning experience to determine the appropriate hyperparameters, which all require to improve the sample processing efficiency. In addition, there is a problem of difficulty in setting the reward function, and the reward function often appears to be sparse, with little or even no reward in the middle process of completing the task, thereby affecting the learning efficiency and effect. SUMMARY

[0006] The present application aims to overcome the deficiencies in the prior art and provide an improved multi-thread PPO reinforcement learning-based robot arm grasping method and system. The PPO algorithm is improved by improving the reward function, advantage value regularization and reward scaling method, and the training efficiency of the algorithm and the grasping performance of the robot arm are improved by using multi-thread parallel computing, so that the robot arm can quickly and accurately grasp the target object in a complex environment.

[0007] Technical solution: To achieve the above purpose, the present application provides an improved multi-thread PPO reinforcement learning-based robot arm grasping method, comprising the following steps:

[0008] S1: initialize the environment state, including the number of working sub-threads, the Actor policy network and the Critic evaluation network;

[0009] S2: multiple working sub-threads start parallel work, obtain the initial state information of the robot arm, and input it to the Actor policy network, which uses an improved PPO reinforcement learning algorithm for data collection;

[0010] S3: put the collected data into the experience pool, and calculate all the advantage values using the generalized advantage estimation GAE algorithm;

[0011] S4: regularize the advantage values and scale the rewards;

[0012] S5: update the Actor policy network and the Critic evaluation network according to the regularized advantage values and the scaled rewards;

[0013] S6: control the joint movement and gripper action of the robot arm according to the latest action instruction output by the Actor policy network.

[0014] Further, the state information in step S2 includes the joint angle of the robot arm, the state of the gripper, and the position and distance information of the target object relative to the robot arm.

[0015] Further, the data collection method of the Actor policy network in step S2 includes introducing multiple working sub-threads into the PPO reinforcement learning algorithm, setting a shared network, the Actor policy network giving weights to the shared network, and each working sub-thread interacting with the environment through the shared network.

[0016] Further, a replay buffer is set in the experience pool in step S3, and the four-tuple data (state, action, reward, next state) obtained from the environment each time is stored in the replay buffer, and a plurality of data are randomly sampled from the buffer for training to improve sample efficiency, and each sample can be used multiple times, which is suitable for gradient learning of the neural network.

[0017] Further, the calculation formula of the advantage value in step S3 is:

[0018]

[0019] wherein γ is a discount factor, λ is a constant between 0 and 1, V represents the state value, r t+l is the single-step total reward at the t+l moment, l is the number of steps experienced, s t+l is the state at the t+l moment, s t+l+1 is the state at the t+l+1 moment.

[0020] The advantage value reflects the advantages and disadvantages of each action in the same state. The state value is the average of the action value, and the advantage value is the action value minus the state value. The higher the advantage value, the higher the return brought by the action.

[0021] Further, the regularization processing in step S4 includes:

[0022] calculating the mean and standard deviation of all advantage values in the experience pool;

[0023] regularizing each advantage value according to the mean and standard deviation, and the formula is as follows:

[0024]

[0025] wherein A t is the original advantage value, μ A is the mean of all advantage values in the experience pool, σ A is the standard deviation of all advantage values in the experience pool, ∈ is a small constant to prevent the denominator from being zero, is the advantage value after regularization.

[0026] Further, the scaling processing in step S4 includes:

[0027] A1: Design reward function:

[0028] R=γr d +I success r success

[0029]

[0030] Calculate the Euclidean distance d between the center of the robot arm gripper and the center of the target object in three-dimensional space; give a smaller reward when the distance is far, and gradually increase the reward as the distance gradually decreases; use a simple nonlinear function to achieve this relationship; where k d is a scaling factor that determines the degree of influence of distance on reward, d max is a pre-set maximum distance threshold; when d max = d, r d = 0; when d = 0 (i.e., the gripper successfully grasps the target object), r d = k d ; this can encourage the robot arm to gradually approach the target object; I success is an indicator function, when successfully grasped, I success = 1; when grasping fails, I success = 0;

[0031] A2: Roll a discount sum of rewards, the formula is R t = γR t-1 + r t , where R t is the cumulative reward calculated at each step, γ is the discount factor, and r t is the reward of the current step;

[0032] Dynamically calculate the standard deviation σ R of the reward of each step;

[0033] Scale the reward using the standard deviation, the formula is where r t ' is the scaled reward.

[0034] Further, the step S5 updates the Actor policy network using the clip function, as follows:

[0035]

[0036] where θ is the model parameter, ∈ is the clipping coefficient, is the policy function before updating, π θ is the policy function after updating, a t is the action being executed at time t, is the expectation of the target function at time t. The clip function is used to achieve small batch updates at multiple training steps, solving the problem of step length determination in the policy gradient algorithm.

[0037] Further, the improved loss function is used to update the Critic evaluation network in the step S5, and the specific process is as follows:

[0038]

[0039] Wherein, rt' is the reward at time t', t is the current time, V is the state value function, is the model parameter, and st is the state at time t.

[0040] The application also provides a mechanical arm grabbing system based on improved multi-thread PPO reinforcement learning, comprising:

[0041] A state acquisition module is configured to acquire state information of the mechanical arm, and the state information includes joint angle of the mechanical arm, gripper state, and position and distance information of the target object relative to the mechanical arm.

[0042] A multi-thread calculation module includes a plurality of parallel calculation threads and is configured to execute a calculation process based on the improved PPO reinforcement learning algorithm.

[0043] An action execution module is configured to control the mechanical arm to perform corresponding actions according to action instructions output by the multi-thread calculation module.

[0044] A reward feedback module is configured to calculate a reward value according to the action result of the mechanical arm and the grabbing task target, and feed back the reward value to the multi-thread calculation module.

[0045] In the application, the received state and reward data, as well as the outputs of the Actor and Critic, are all put into the experience pool as an experience. When the experience pool is full, it represents that the data collection of this round is completed, and the training phase is entered. A batch of data is sampled from the experience replay buffer, and the advantage value is calculated using the GAE algorithm. The GAE algorithm more accurately evaluates the advantage degree of each action by considering the reward information of multiple future time steps. The Actor network updates the parameters by minimizing the PPO-clip function of the PPO algorithm according to the advantage value and the sampled data. The Critic network updates the parameters by using the loss function according to the state and reward information. In the updating process, the data collection, advantage value calculation and network updating steps are repeated constantly until the network converges, i.e. the grabbing strategy of the mechanical arm reaches a stable state and can effectively perform the grabbing task under different conditions.

[0046] In the application, the PPO algorithm is improved and applied to the mechanical arm grabbing control. The improvements include:

[0047] Advantage value regularization: In the PPO algorithm, the advantage value reflects the degree of advantage of taking a certain action relative to the average level in a given state. By calculating the mean and standard deviation of the advantage values ​​and regularizing each advantage value, the distribution of advantage values ​​can be made more stable, avoiding excessively large or small advantage values ​​from having too much influence on policy updates. This helps improve the convergence speed of the algorithm, reduces fluctuations during training, and enables the robotic arm to learn effective grasping strategies more quickly.

[0048] Reward scaling: Reward scaling adjusts the reward by continuously calculating the sum of the discounts and dynamically calculating the standard deviation. This avoids gradient explosion or vanishing gradient problems caused by excessively large or small reward values, making the training of the value function more stable. At the same time, reasonable reward scaling enables the robotic arm to better perceive differences in reward signals, improving learning efficiency and finding the optimal grasping strategy more quickly.

[0049] Multi-threaded parallel operation: Multiple threads work simultaneously, achieving parallel processing in various stages such as data acquisition, advantage calculation, regularization and scaling, and model updates, significantly reducing computation time and improving training efficiency. During the data acquisition phase, multiple data acquisition threads can interact with the environment simultaneously to acquire more sample data. They also use a buffer for experience replay, placing sample data into the buffer. When there is enough sample data in the buffer, samples are retrieved to train and update the strategy. During the model update phase, multiple model update threads can simultaneously calculate gradients for different parts and update network parameters, accelerating model convergence. This parallel computing method fully utilizes the multi-core processor resources of modern computers, enabling the robotic arm grasping system to complete training and optimization in a shorter time, improving the system's real-time performance and response speed, and adapting to complex and ever-changing grasping task environments.

[0050] Beneficial effects: Compared with the prior art, the PPO algorithm is improved by introducing the method of multi-threading combined with reward function improvement, advantage value regularization and reward scaling, and is applied to the mechanical arm grasping control. By setting reasonable state space, action space and reward function, the mechanical arm can obtain sufficient information to understand the environment and state, so as to obtain more reasonable Actor model and Critic model, so that the mechanical arm can learn the strategy more efficiently and accurately, and then the grasping success rate evaluation curve and the reward evaluation curve are compared with the original PPO algorithm, the stability and convergence speed of the algorithm are improved, the mechanical arm can learn the effective grasping strategy faster, and the grasping success rate is improved. The application of multi-thread parallel computing technology and the use experience replay buffer greatly improve the computing efficiency of the system, shorten the training time, enhance the real-time performance and response speed of the system, and can better adapt to the complex and changeable mechanical arm grasping task environment. The system can automatically adjust the grasping strategy according to the characteristics of the grasping task and the change of the environment, has strong adaptability and robustness, and has wide application prospect in the fields of industrial manufacturing and logistics. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a flow chart of the mechanical arm grasping system based on the multi-thread PPO algorithm;

[0052] Figure 2 is a flow chart of the multi-thread PPO algorithm;

[0053] Figure 3 is a specific application framework diagram of the PPO algorithm;

[0054] Figure 4 is a flow chart of the mechanical arm grasping experiment;

[0055] Figure 5 is a schematic diagram of the simulation environment;

[0056] Figure 6 is a training reward return diagram of the mechanical arm grasping in the simulation environment;

[0057] Figure 7 is a training success rate diagram of the mechanical arm grasping in the simulation environment;

[0058] Figure 8 is a comparison diagram of the original PPO algorithm and the improved multi-thread PPO algorithm training. DETAILED DESCRIPTION

[0059] The present application will be further illustrated below in conjunction with the drawings and specific embodiments, and it should be understood that these embodiments are only used to illustrate the present application and not to limit the scope of the present application, and after reading the present application, those skilled in the art can make various equivalent modifications of the present application, which fall within the scope defined by the appended claims.

[0060] Embodiment 1:

[0061] As Figures 1-3 shown, the embodiment provides a mechanical arm grasping method based on improved multi-thread PPO reinforcement learning, comprising the following steps:

[0062] S1: initialize the environment state, including the number of working sub-threads, Actor policy network and Critic evaluation network;

[0063] When the system starts, first calibrate and initialize the sensors in the state acquisition module to ensure that the state information of the mechanical arm can be accurately acquired. At the same time, initialize each thread in the multi-thread computing module, allocate memory space, set thread parameters, and set the number of threads to 5 working sub-threads. Initialize the Actor and Critic networks, set the network structure and initial parameters, and set the learning rate of the Actor and Critic networks to 0.0003, such as network layers, number of neurons per layer, learning rate, etc.

[0064] S2: multiple working sub-threads start parallel work, acquire the initial state information of the mechanical arm, and input it to the Actor policy network, which uses an improved PPO reinforcement learning algorithm for data collection;

[0065] The state information includes the joint angle of the mechanical arm, the state of the gripper, and the position and distance information of the target object relative to the mechanical arm;

[0066] The data structure of the state space of the mechanical arm grasping is determined by the type of environment information acquired by the mechanical arm. Since an end-to-end design is used, the data obtained by the agent from the environment is the position of the target object and the joint angle, angular velocity change, so the state space is designed as S=(S arm , S env , S ops , S d ), where S arm =(J1, J2, J3, J4, J5, J6), J represents the angle vector of the joint, S d represents the Euclidean distance between the gripper and the object, S ops represents the relative position of the gripper and the target object, and S env represents the open and close state of the gripper, with a value of 1 for closed and 0 for open state. The action space of the mechanical arm is the position of the end of the mechanical arm, the state of the parallel angle of the two fingers, and the joint, i.e. four-dimensional vector [x, y, z, θ] T and T=(t i), x represents the coordinate of the end of the mechanical arm on the x-axis, y represents the coordinate of the end of the mechanical arm on the y-axis, z represents the coordinate of the end of the mechanical arm on the z-axis, and theta represents the two-finger clamping state, i.e., opening and closing, and t represents the angle at the i joint.

[0067] The data collection method of the actor policy network includes that the actor policy network input layer explicitly labels the received mechanical arm state information, including six joint angles, six joint angular velocities, three-dimensional position coordinates of the target object, and the distance between the gripper and the target object, a total of 16 input neurons, the input layer is connected to the first layer of hidden layer, the hidden layer has 64 neurons, the connection between the neurons represents the transmission of data, and each neuron applies a ReLU activation function. The output of the first layer of hidden layer is used as the input of the second layer of hidden layer, the second layer of hidden layer has 32 neurons, and the ReLU activation function is also used. The output of the second layer of hidden layer is connected to the output layer, and the output layer has 7 neurons, respectively corresponding to the torque instructions of the six joints and a gripper control instruction. A plurality of working sub-threads are introduced into the PPO reinforcement learning algorithm, a shared network is set, and the actor policy network and the shared network usually have the same network structure. When the actor policy network completes a round of training and updates its own weights, it will send a signal (a Boolean variable) indicating that the weights are ready to update the shared network, and the weights represent the mapping relationship between the input features (state information) and the output actions. The above shared network is a fully connected layer, and each element w ij of the weight matrix W represents the connection strength from the i-th input neuron to the j-th output neuron. During the training process, the weight W is constantly updated. Therefore, the actor policy network gives the weights to the shared network, and each working sub-thread is allocated through the shared network. Each sub-thread interacts with the environment.

[0068] S3: Put the collected data into the experience pool, and calculate all the advantage values by using the generalized advantage estimation (GAE) algorithm.

[0069] A playback buffer is set in the experience pool, and the four-tuple data (state, action, reward, next state) obtained from the environment each time is stored in the playback buffer. During training, a certain amount of data is randomly sampled from the buffer for training, improving sample efficiency, and each sample can be used multiple times, which is suitable for gradient learning of neural networks.

[0070] The data collection thread outputs the action to the environment, and the environment returns the next state and reward information according to the action of the mechanical arm. These interaction data are stored in the experience pool, and the experience pool uses a first-in, first-out (FIFO) method to manage data. When the experience pool is full, the earliest data is automatically discarded.

[0071] The calculation formula of the advantage value is:

[0072]

[0073] where γ is a discount factor, λ is a constant between 0 and 1, V represents the state value, r t+l is the total reward of a single step at the t+l moment, l is the number of steps experienced, s t+l is the state at the t+l moment, s t+l+1 is the state at the t+l+1 moment.

[0074] The advantage value reflects the advantages and disadvantages of each action in the same state. The state value is the average of the action value, while the advantage value is the action value minus the state value. The higher the advantage value, the higher the return the action can bring. In this process, attention needs to be paid to the data synchronization between multiple threads to ensure the consistency and correctness of the advantage value and the reward data.

[0075] S4: Regularize the advantage value and scale the reward at the same time;

[0076] The regularization process includes:

[0077] Calculate the mean and standard deviation of all advantage values in the experience pool;

[0078] According to the mean and standard deviation, each advantage value is regularized, and the formula is as follows:

[0079]

[0080] where A t is the original advantage value, μ A is the mean of all advantage values in the experience pool, σ A is the standard deviation of all advantage values in the experience pool, ∈ is a small constant to prevent the denominator from being zero, is the advantage value after regularization.

[0081] The scaling process includes:

[0082] A1: Design a reward function:

[0083] R=γr d +I success r success

[0084]

[0085] Calculate the Euclidean distance d between the center of the robot arm gripper and the center of the target object in three-dimensional space; give a smaller reward when the distance is far, and gradually increase the reward as the distance gradually decreases; use a simple nonlinear function to achieve this relationship; where k dis a scaling factor that determines the degree of influence of distance on the reward, d max is a pre-set maximum distance threshold; when d max = d, r d = 0; when d = 0 (i.e., the gripper successfully grasps the target object), r d = k d ; this can encourage the robot arm to gradually approach the target object; I success is an indicator function, when successfully grasped, I success = 1; when grasping fails, I success = 0;

[0086] A2: Roll out a discounted sum of rewards, formula R t = γR t-1 + r t , where R t is the cumulative reward calculated at each step, γ is the discount factor, and r t is the reward of the current step;

[0087] The standard deviation σ R of the reward at each step is dynamically calculated;

[0088] The reward is scaled using the standard deviation, formula where r t ' is the scaled reward.

[0089] S5: Update the Actor policy network and Critic evaluation network according to the regularized advantage value and the scaled reward;

[0090] The Actor policy network is updated using the clip function, as follows:

[0091]

[0092] where θ is the model parameter, ∈ is the clipping coefficient, is the policy function before updating, π θ is the policy function after updating, a t is the action being executed at time t, is the expectation of the target function at time t. The clip function allows for small batch updates over multiple training steps, solving the problem of step size determination in policy gradient algorithms.

[0093] The Critic evaluation network is updated using an improved loss function, as follows:

[0094]

[0095] Wherein, rt' is the reward at t' time, t is the current time, V is the state value function, is a model parameter, and st is the state at t time.

[0096] S6: According to the latest action instruction output by the Actor policy network, the joint movement and gripper action of the robot arm are controlled.

[0097] Embodiment 2:

[0098] Based on the robot arm grasping method of embodiment 1, this embodiment provides a robot arm grasping system based on improved multi-thread PPO reinforcement learning, comprising:

[0099] A state acquisition module is configured to acquire state information of the robot arm, the state information including joint angle of the robot arm, gripper state, and position and distance information of the target object relative to the robot arm.

[0100] A multi-thread calculation module includes multiple parallel calculation threads and is configured to execute a calculation process based on the improved PPO reinforcement learning algorithm.

[0101] An action execution module is configured to control the robot arm to perform corresponding actions according to the action instruction output by the multi-thread calculation module.

[0102] A reward feedback module is configured to calculate a reward value according to the action result of the robot arm and the grasping task target, and feed back the reward value to the multi-thread calculation module.

[0103] Embodiment 3:

[0104] In this embodiment, the method of embodiment 1 and the system of embodiment 2 are simulated, and the flow is as shown in Figure 4 The simulation environment in this embodiment is as shown in Figure 5 The training reward return data as shown in Figure 6 is obtained, and the training success rate data as shown in Figure 7 is obtained. Figure 6 and Figure 7 It can be seen that the improved PPO algorithm combined with multi-threading can achieve good learning effect in this environment, and the grasping success rate gradually increases in smaller set numbers, more close to 1 and more stable convergence.

[0105] This embodiment also obtains the training comparison data as shown in Figure 8 , and Figure 8 It can be seen that the original PPO algorithm robot arm grasping success rate (0_1) gradually improves and is relatively unstable, while the improved PPO algorithm robot arm grasping success rate (1_1) is higher in the same set number and gradually improves to be stable.

Claims

1. A mechanical arm grasping method based on improved multi-thread PPO reinforcement learning, characterized in that, Comprising the following steps: S1: initializing the environment state, including the number of working sub-threads, the Actor policy network and the Critic evaluation network; S2: multiple working sub-threads start parallel work, obtain the initial state information of the robot arm, and input it to the Actor policy network, which uses an improved PPO reinforcement learning algorithm for data collection; S3: put the collected data into the experience pool, and calculate all the advantage values using the generalized advantage estimation GAE algorithm; S4: regularize the advantage values and scale the rewards; S5: update the Actor policy network and the Critic evaluation network according to the regularized advantage values and the scaled rewards; S6: control the joint movement and gripper action of the robot arm according to the latest action instructions output by the Actor policy network. The regularization processing in step S4 includes: Calculate the mean and standard deviation of all advantage values in the experience pool; Regularize each advantage value according to the mean and standard deviation, the formula is as follows: ； wherein is the original advantage value, is the mean of all advantage values in the experience pool, is the standard deviation of all advantage values in the experience pool, is a small constant to prevent the denominator from being zero, is the advantage value after regularization; The scaling processing in step S4 includes: A1: design a reward function: ； ； wherein is a discount factor; Calculate the Euclidean distance between the center of the robot arm gripper and the center of the target object in three-dimensional space ; give a smaller reward when the distance is far, and gradually increase the reward as the distance gradually decreases; use a simple nonlinear function to achieve this relationship; wherein is a scaling factor that determines the degree of influence of distance on reward, is a pre-set maximum distance threshold; when , ; when , ; this can encourage the robot arm to gradually approach the target object; is an indicator function, when successfully grasped, ; when the grasp fails, ; A2: Roll out a discounted sum of rewards, formula is where is the cumulative reward computed at each step, is the discount factor, is the reward for the current step; Dynamic calculation of standard deviation of reward for each step ; The rewards are scaled by the standard deviation, with the formula where is the scaled reward.

2. The method of claim 1, wherein, The state information in step S2 includes the joint angle of the robot arm, the gripper state, and the position and distance information of the target object relative to the robot arm.

3. The method of claim 1, wherein, The data collection method of the Actor policy network in step S2 includes: introducing multiple working sub-threads into the PPO reinforcement learning algorithm, setting a shared network, and giving the weights of the Actor policy network to the shared network. Each working sub-thread is assigned to the shared network, and each sub-thread interacts with the environment.

4. The method of claim 1, wherein, A playback buffer is set in the experience pool of step S3, and the four-tuple data sampled from the environment each time is stored in the playback buffer. When training, a certain number of data is randomly sampled from the buffer for training.

5. The method of claim 1, wherein, The calculation formula of the advantage value in step S3 is: ； in, As a discount factor, It is a constant between 0 and 1. Represents state value. For the first Total reward per step at any given moment The number of steps taken. for The state at any given moment, for The state at any given moment.

6. The method of claim 1, wherein, In step S5, the Actor policy network is updated using the clip function, as follows: ； ； ； wherein, is a model parameter, is a clipping coefficient, is the policy function before update, is the policy function after update, is the action being performed at time t, is the expectation of the objective function at time t.

7. The method of claim 1, wherein, In step S5, the Critic evaluation network is updated using an improved loss function, as follows: ； where, is the reward at time t is the current time, is the state value function, is the model parameter, is the state at time t.

8. A mechanical arm grasping system based on improved multi-thread PPO reinforcement learning, characterized in that, A system for implementing the robot arm grasping method of any one of claims 1-7, comprising: A state acquisition module for acquiring state information of the robot arm, including joint angle of the robot arm, gripper state, and position and distance information of the target object relative to the robot arm; A multi-thread calculation module including multiple parallel computing threads for executing a calculation process based on an improved PPO reinforcement learning algorithm; An action execution module for controlling the robot arm to perform corresponding actions according to the action instructions output by the multi-thread calculation module; A reward feedback module for calculating the reward value according to the action result of the robot arm and the grasping task target, and feeding back the reward value to the multi-thread calculation module.

Images