Mechanical arm grabbing method and system based on action branch architecture value advantage learning

Abstract

The invention discloses a mechanical arm grabbing method and system based on action branch architecture value advantage learning. The method comprises the steps that a mechanical arm grabbing task is modeled as a Markov decision problem; based on the action space, a VA learning algorithm is introduced, and a main network and a target network of an action branch structure are built; based on a greedy strategy, training the main network and the target network, and accumulating an experience playback pool; and performing setting according to an actual scene, and combining a greedy strategy and the experience playback pool to complete a target capture task. The system comprises a modeling module, a network construction module and a mechanical arm training module. According to the method, the deep learning algorithm based on the action branch architecture is used for exploring the deterministic strategy, the discrete state action space of the mechanical arm can be better processed, and then the control performance is improved. The method can be widely applied to the field of mechanical arm control.

Description

Technical Field

[0001] The present invention relates to the field of robotic arm control, and in particular to a robotic arm grasping method and system based on action branch architecture value advantage learning. Background Art

[0002] Traditional control design methods for robotic arms include adaptive control, proportional differential control (PD) control, nonlinear feedback control, robust control, etc. However, in complex nonlinear systems such as robotic arms, these methods have certain limitations in control performance. For example, these methods are difficult to fully capture and process the behavior of the system, and the nonlinear terms of the robotic arm may lead to a decrease in control performance; traditional control methods may be sensitive to parameter changes in the robotic arm system, and non-ideal factors such as friction and vibration of the robotic arm joints may introduce uncertainty, resulting in a decrease in control performance; therefore, in the face of the complexity and uncertainty of the robotic arm, traditional control design methods are insufficient in control performance, especially in dealing with complex unknown environments and requiring high-precision trajectory control. This has prompted the study of new and more advanced control methods to improve the performance and flexibility of robotic arms in various working environments. Summary of the invention

[0003] In view of this, in order to solve the technical problem that the existing robot arm control method does not consider uncertainty factors, thereby resulting in limitations in control performance, the present invention proposes a robot arm grasping method based on action branch architecture value advantage learning, the method comprising the following steps:

[0004] Analyze the characteristics of the robot grasping task and model it as a Markov decision problem;

[0005] Design a VA learning algorithm based on the action branch architecture, and build the main network and target network of the action branch structure;

[0006] Based on the priority experience replay mechanism, a greedy strategy is used to generate a grasping strategy for grasping and accumulate an experience replay pool;

[0007] The settings are made according to the actual scenario, the robot arm model and the scenario information are built, and the greedy strategy and the experience replay pool are combined to complete the target grasping task.

[0008] In some embodiments, the step of generating a grasping strategy based on a priority experience replay mechanism using a greedy strategy to grasp and accumulate an experience replay pool further includes:

[0009] Input the observed state of the robot arm into the Q network to obtain the action set;

[0010] Based on the greedy strategy, select the corresponding action;

[0011] Based on the selected action, the system enters a new state and obtains rewards and constraints;

[0012] Put the sample sequence of the new state back into the experience replay pool;

[0013] The above steps are repeated for a preset number of times to accumulate an experience replay pool.

[0014] In some embodiments, the step of setting up according to the actual scene, building a robot arm model and scene information, combining the greedy strategy and the experience replay pool, and completing the target grasping task specifically includes:

[0015] Set the environment coordinate system, set the initial state of the robot arm, determine the robot arm workspace, set scene information such as the desktop and target object, determine the observation state information and action information of the robot arm, and determine the update time interval of the robot arm;

[0016] The VA learning algorithm based on the action branch architecture interacts with the environment and accumulates the experience replay pool using the experience pool processing method in the previous step;

[0017] Based on the target grasping task, the state is input into the Q network, the action based on the greedy strategy output is obtained, and the main network parameters are updated;

[0018] The parameters of the target network are soft updated using the parameters of the main network.

[0019] The present invention also proposes a robotic arm grasping system based on action branch architecture value advantage learning, the system comprising:

[0020] Modeling module, which analyzes the characteristics of the robot grasping task and models it as a Markov decision problem;

[0021] Network construction module, designing VA learning algorithm based on action branch architecture, building main network and target network of action branch structure;

[0022] The training accumulation module trains the main network and the target network based on the greedy strategy and accumulates the experience replay pool; the application module is set according to the actual scenario, combining the greedy strategy and the experience replay pool to complete the target grasping task.

[0023] Based on the above scheme, the present invention provides a robotic arm grasping method and system based on value advantage learning of action branch architecture. The action branch architecture is used as the computing network, which can better process the high-dimensional discrete state action space of the robotic arm, and add action random exploration parameters to enhance the exploration ability of the robotic arm; VA-learning is used as a more efficient alternative to Q-learning, aiming to directly learn the value function and advantage function instead of indirect learning through the Q function, thereby improving sample efficiency and learning speed, simplifying the learning problem, and further improving the overall efficiency of the grasping control method. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of steps of a robot arm grasping method based on action branch architecture value advantage learning of the present invention;

[0025] Figure 2 is a network diagram of an action branch architecture according to a specific embodiment of the present invention;

[0026] Figure 3 It is a schematic diagram of the framework of an encoder according to a specific embodiment of the present invention;

[0027] Figure 4 It is a schematic diagram of the process of updating the priority experience pool of the present invention;

[0028] Figure 5 It is a structural block diagram of a robotic arm grasping system based on action branch architecture value advantage learning of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] It should be noted that, for the convenience of description, only the parts related to the relevant invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0031] It should be understood that the "system", "device", "unit" and / or "module" used in this application is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the word can be replaced by other expressions.

[0032] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements. The elements defined by the sentence "includes a..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.

[0033] In the description of the embodiments of the present application, "plurality" means two or more than two. The following terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0034] In addition, flow charts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, the various steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0035] Reference Figure 1 , which is a flow chart of an optional example of a robot arm grasping method based on action branch architecture value advantage learning proposed in the present invention, which method can be applied to computer equipment. The grasping control method proposed in this embodiment may include but is not limited to the following steps:

[0036] Step S1, modeling the robot grasping task as a Markov decision problem;

[0037] Step S2: Based on the action space, the VA learning algorithm is introduced to build the main network and target network of the action branch structure;

[0038] Step S3: Based on the greedy strategy, the main network and the target network are trained, and an experience replay pool is accumulated;

[0039] Step S4: Set according to the actual scenario, combine the greedy strategy and the experience replay pool to complete the target grasping task.

[0040] In some feasible embodiments, the step S1 specifically includes:

[0041] Use PyBullet to build a simulation scene, and use a 7-DOF KUKA LBR iiwa 7R800 robotic arm equipped with a two-finger gripper to perform object grasping tasks.

[0042] According to the characteristics of the robot arm grasping task, the four-tuple is defined as: (S t ,s a ,r t ,γ);

[0043] S t is the state space, which represents the set of information observed by the agent, specifically represented by the depth image captured by the camera;

[0044] s a is the action space, which represents the set of operations performed by the agent, specifically represented by the three-dimensional coordinate increment of the end effector, the rotation angle of the mechanical claw, and the opening and closing state of the gripper s a =(x,y,z,θ,α);

[0045] r t is a reward function used to evaluate the effectiveness of the agent in executing actions, so as to guide the agent to learn and achieve the expected goal. The reward function adopts the following binary sparse reward function:

[0046]

[0047] The reward is 1 when the target object is successfully grasped, otherwise it is 0;

[0048] Through this optimization step, the reward function signal uses a binary sparse reward form, which reduces the complexity of reward function design, simplifies learning problems, reduces design costs, and has strong interpretability.

[0049] In some feasible embodiments, the step S2 specifically includes:

[0050] Since the action space of the robot arm in this system is five dimensions s a =(x, y, z, θ, α), so the action network in S2 includes five branches, and the RGB-D image is feature extracted through the five branch network trunks to obtain action values ​​in five dimensions.

[0051] The action branch architecture network contains a shared decision module and multiple network branches, each of which is responsible for controlling an independent action dimension. This design makes the number of network outputs grow linearly with the increase in degrees of freedom, rather than combinatorially, thereby improving scalability in high-dimensional action spaces. The shared network module calculates the latent representation of the input state and then passes it to several action branches. Each action branch is responsible for controlling a single degree of freedom, and the connection of the selected sub-actions produces a joint action tuple. Each dimension uses the duel network architecture to divide the representation of state value and (state-related) action advantage into two independent branches, while sharing a common feature learning module between them. The two branches are combined through a special aggregation layer to produce an estimate of the action value function. In the presence of many similar-valued (or redundant) actions, the duel network architecture can lead to better policy evaluation, thereby achieving faster generalization in large action spaces. Action branch architecture network such as Figure 2 shown.

[0052] In this system, the state image of the network passes through the encoder and then through a 512-unit fully connected layer to reach a 256-unit fully connected layer 2, and then passes through 128-unit fully connected layers 3 to 14 and the Relu activation function to output A values ​​of five dimensions and a shared state value respectively. The number of A values ​​is n, that is, n discrete actions in each dimension. Then each A value is subtracted from the mean in each dimension and added to the state value to finally obtain the Q value of each action. Finally, the index of the maximum value is taken to obtain the action selected in each dimension.

[0053] The action branch architecture network separates its own network branch for each action dimension, while each dimension shares a common convolutional neural network θ with parameters to obtain the same state representation vector s, which is then input into different action branches to calculate the action advantage estimate for each dimension. Here, d represents the d-th action branch. At the same time, a general value function estimator V is used π (s; θ, β) is used to approximate the state value V(s), which is then used to calculate the state-action value estimate for each dimension d. The Q value of each action branch of the VA algorithm based on the action branch architecture is calculated as follows:

[0054]

[0055] Wherein, s represents the state vector, a d represents the action taken by the d-th action branch, V(s) represents the state value; A d Indicates that in state s, action a d The advantage function of a' d Represents any sub-action under the action dimension d; Represents the set of all possible sub-actions under the action dimension d.

[0056] The loss function is different from traditional Q learning, which directly approximates the network to the Q value. VA learning decomposes Q(x,a)=V(x)+A(x,a) to converge the value function V and advantage function A output by the network to the correctly defined value function and advantage function respectively. The loss function is as follows:

[0057]

[0058] Where N is the number of action branches, is the sampling importance weight, where N is the number of action branches, ω j represents the weight coefficient, N represents the size of the experience pool, β represents the non-uniform probability compensation coefficient, V d (x t ) means that in x t Next, the estimation of the state value function of the action dimension d is: Indicates that in x t The target value of the state value function of action dimension d is A d (x t ,a t ) means in state x t and action a t Next, the estimation of the action advantage function of action dimension d is: In state x t and action a t Under this condition, the target value of the action advantage function of action dimension d, the value function and advantage function of each branch are:

[0059]

[0060] in In state x t and action a t Under the estimated value of the action value function of the target policy π, A φ -(x t+1 ,μ) is in state x t+1 Under this condition, the average value of the advantage function of all possible actions is weighted by the behavior strategy μ, where μ represents the behavior strategy, which is a t For all possible actions a t Distribution of assignment probabilities, V θ -(x t ) means in state x t The estimated value of the state value function, φ - ,θ - represents the parameters of the target network, which are slowly updated toward φ and θ.

[0061] The structure of the target network is exactly the same as the main network, only the network parameters are different.

[0062] Specifically, the encoder network structure is as follows Figure 3 As shown in the figure, the input is a 64×64-sized depth image. The first 7×7 Cov2d reduces the feature map of the 64×64 channel to 32×32×32 channels. The second 5×5 Cov2d reduces the feature map of the 32×32×32 channel to 16×16×32 channels. The third 3×3 Cov2d reduces the feature map of the 16×16×32 channel to 8×8×32 channels. Each convolutional layer is followed by a LeakyReLU activation function to introduce nonlinearity and prevent the gradient from disappearing. After a series of convolutional layers, a fully connected layer is connected to flatten all features, and finally a latent space feature vector with a dimension of 101 is obtained.

[0063] In some feasible embodiments, the step S3 specifically includes:

[0064] First, create an experience replay pool. The data stored in the experience pool is in the form of five tuples (S t ,A t ,R t ,S t+1 , done), train the main network and the target network according to batch sampling in the experience pool, where S t is the state of the agent at time t, A t represents the action taken by the agent at time t, R t Indicates that the agent is taking action A t The reward obtained after t+1 Indicates that the agent is in state S t Take action A t The state transferred at time t+1, done represents the next state S of the agent t+1 , if it is the terminal state, done = 1, otherwise done = 0. Then the observed state S of the robot at time t is t Input to the Q network to get the action set. According to the following ε-greedy strategy, select action A t .

[0065]

[0066] When the random probability is 1-ε, the maximum predicted value Q value index in each action branch result is selected and converted into a motion primitive, including the action execution position offset point (x, y, z) and the rotation angle of the robot claw and the opening and closing state of the gripper, which is the grasping strategy of the current state. When the random probability is 1-ε, a grasping action is randomly performed to enhance the exploration ability of the robot arm.

[0067] Then execute action A t , the system enters a new state S t+1 , and receive reward R t and constraint d, the sample sequence of the new state (S t ,A t ,R t ,S t+1 , done) is stored in the experience pool D;

[0068] Repeat step S3 N times in total.

[0069] Through this optimization step, the experience pool is processed based on priority experience playback, the experience utilization rate is improved, the capture success rate is improved, and the network convergence speed and training efficiency are enhanced.

[0070] In some feasible embodiments, the step S4 specifically includes:

[0071] First, set the environment coordinate system O-XYZ, set the initial state of the robot, and determine the robot's working space;

[0072] Set scene information such as desktop and target object; determine the observation status information and action information of the robot arm, and determine the update time interval of the robot arm;

[0073] The VA learning algorithm based on the action branch architecture interacts with the environment and uses the experience pool processing method in step 3 to accumulate the experience replay pool; a minibatch β of size N is sampled from the experience replay pool R, and the sampling is combined with the random sampling method to sample N state-action transition processes (s, a, r, s', done). The sampling follows The specification of p(j) represents the probability that the jth transition tuple is sampled. j represents the priority of the jth transfer tuple, which is determined by the absolute value of its VA error. α is a hyperparameter that adjusts the impact of priority, and the denominator is a normalization factor to ensure that the sum of the sampling probabilities of all transition tuples is 1, and then the sampling importance weight is calculated Among them, ω j represents the weight coefficient, N represents the size of the experience pool, and β represents the non-uniform probability compensation coefficient.

[0074] Input the state s into the Q network and get the output action A based on the greedy strategy π(s; θ) t , θ is the Q network parameter, which is updated by gradient descent. Use the optimizer Adam to update the current Q network parameter θ, and the update formula is:

[0075]

[0076] Soft update the parameters of the target network based on the parameters of the main network:

[0077] θ Q′ ←τθ Q +(1-τ)θ Q′

[0078] where θ Q is the network parameter of the Q network, θ Q′ is the network parameter of the Target network in the target network, τ is the soft update amplitude of the network parameter, τ∈(0,1);

[0079] like Figure 4 The experience replay pool is updated as shown. This experiment uses a binary tree-based priority experience replay method (Sumtree) to update the priority. Specifically: first generate a random number in the interval [0, sum], and then start from the root node, select the left child node or the right child node according to the size of the random number, until reaching the leaf node, the leaf node is the sampled experience; when updating, only the affected nodes are updated, that is, all nodes on the path from the leaf node to the root node, and the update amount of each node is equal to the sum of the update amounts of its left and right child nodes. Discard the oldest experience trajectory in the experience pool and accumulate a set of new experiences;

[0080] The training cycle ends when the set number of rounds is reached.

[0081] like Figure 5 As shown, a robotic arm grasping system based on action branch architecture value advantage learning includes:

[0082] Modeling module, which models the robot grasping task as a Markov decision problem;

[0083] The network construction module introduces the VA learning algorithm based on the action space to build the main network and target network of the action branch structure;

[0084] The robotic arm training module is based on a greedy strategy, an accumulated experience replay pool, and is set according to actual scenarios. It combines the greedy strategy and the experience replay pool to train the main network and the target network and complete the target grasping task.

[0085] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0086] A robotic arm grasping device based on action branch architecture value advantage learning:

[0087] at least one processor;

[0088] at least one memory for storing at least one program;

[0089] When the at least one program is executed by the at least one processor, the at least one processor implements the robotic arm grasping method based on action branch architecture value advantage learning as described above.

[0090] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0091] A storage medium stores processor-executable instructions, which are used to implement a robotic arm grasping method based on action branch architecture value advantage learning when executed by the processor.

[0092] The contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0093] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A robotic arm grasping method based on action branch architecture value advantage learning, characterized in that: The following steps are involved: Model the robotic grasping task as a Markov decision problem; Based on the action space, the VA learning algorithm is introduced to build the main network and target network of the action branch structure; Based on a greedy strategy, the main network and the target network are trained, and an experience replay pool is accumulated; The settings are made according to the actual scenario, and the greedy strategy and the experience replay pool are combined to complete the target grasping task.

2. According to claim 1, a robotic arm grasping method based on action branch architecture value advantage learning is characterized in that: The step of modeling the robot grasping task as a Markov decision problem specifically includes: According to the characteristics of the robot grasping task, the four-tuple is defined as: (S t ,s a ,r t ,γ); Among them, S t is the state space, which represents the information set observed by the agent; s a is the action space, which represents the set of operations performed by the agent; r t is a reward function used to evaluate the effectiveness of the agent in executing actions; γ is a discount factor; The reward function formula is as follows:

3. According to claim 2, a robot arm grasping method based on action branch architecture value advantage learning is characterized in that: The step of introducing the VA learning algorithm based on the action space and building the main network and the target network of the action branch structure specifically includes: According to the dimension of the action space, a corresponding number of action branches are set, each of which includes a main network and a target network; The Q value calculation formula for each action branch is as follows: Wherein, s represents the state vector, a d represents the action taken by the d-th action branch, V(s) represents the state value; A d Indicates that in state s, action a d The advantage function of a' d Represents any sub-action under the action dimension d; Represents the set of all possible sub-actions under the action dimension d.

4. According to claim 3, a robot arm grasping method based on action branch architecture value advantage learning is characterized in that: The loss function of each action branch is specifically expressed as follows: Where N is the number of action branches, ω j represents the weight coefficient, N represents the size of the experience pool, β represents the non-uniform probability compensation coefficient, V d (x t ) means that in x t Next, the estimation of the state value function of the action dimension d is: Indicates that in x t The target value of the state value function of action dimension d is A d (x t ,a t ) means in state x t and action a t Next, the estimation of the action advantage function of action dimension d is: In state x t and action a t , the objective value of the action advantage function of action dimension d.

5. According to claim 2, a robot arm grasping method based on action branch architecture value advantage learning is characterized in that: The value function and advantage function of each action branch are specifically expressed as follows: in In state x t and action a t Under this condition, according to the estimated value of the action value function of the target policy π, For state x t+1 Under this condition, the average value of the advantage function of all possible actions weighted by the behavior strategy μ is: In state x t The estimated value of the state value function, φ - ,θ - Represents the parameters of the target network.

6. According to claim 1, a robot arm grasping method based on action branch architecture value advantage learning is characterized in that: The step of setting according to the actual scenario, combining the greedy strategy and the experience replay pool, and completing the target grasping task specifically includes: Set the environment coordinate system, the initial state of the robot, the robot workspace and scene information; Based on the experience replay pool, sampling is performed in combination with random sampling to obtain the state-action transition process; Based on the target grasping task, the state is input into the Q network, the action based on the greedy strategy output is obtained, and the main network parameters are updated; The parameters of the target network are soft updated using the parameters of the main network.

7. A robotic arm grasping method based on action branch architecture value advantage learning according to claim 6, characterized in that: The step of setting according to the actual scenario, combining the greedy strategy and the experience replay pool to complete the target grasping task also includes: The priority of the experience replay pool is updated based on a binary tree priority experience replay method.

8. A robotic arm grasping system based on action branch architecture value advantage learning, characterized in that: include: Modeling module, which models the robot grasping task as a Markov decision problem; The network construction module introduces the VA learning algorithm based on the action space to build the main network and target network of the action branch structure; A training accumulation module, based on a greedy strategy, trains the main network and the target network and accumulates an experience replay pool; The application module is set according to the actual scenario, combining the greedy strategy and the experience replay pool to complete the target grasping task.

9. A robotic arm grasping device based on action branch architecture value advantage learning, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements a robotic arm grasping method based on action branch architecture value advantage learning as described in any one of claims 1 to 7.

Images