A robotic arm grasping method and system based on value advantage learning of action branch architecture

Abstract

This invention discloses a robotic arm grasping method and system based on action branch architecture value advantage learning. The method includes: modeling the robotic arm grasping task as a Markov decision problem; based on the action space, introducing a VA learning algorithm to build a main network and a target network with an action branch structure; training the main network and target network based on a greedy policy and accumulating an experience replay pool; and, according to the actual scenario, combining the greedy policy and the experience replay pool to complete the target grasping task. The system includes: a modeling module, a network construction module, and a robotic arm training module. By using this invention, and employing a deep learning algorithm based on action branch architecture to explore deterministic strategies, the discrete state action space of the robotic arm can be better handled, thereby improving control performance. This invention can be widely applied in the field of robotic arm control.

Description

Technical Field

[0001] This invention relates to the field of robotic arm control, and in particular to a robotic arm grasping method and system based on value advantage learning of motion branch architecture. Background Technology

[0002] Traditional robotic arm control design methods include adaptive control, proportional-derivative (PD) control, nonlinear feedback control, and robust control. However, in complex nonlinear systems such as robotic arms, these methods have certain limitations in control performance. For example, they cannot fully capture and handle the system's behavior, and the nonlinear terms of the robotic arm may lead to a decrease in control performance. Traditional control methods may be sensitive to changes in the parameters of the robotic arm system, and non-ideal factors such as friction and chatter at the robotic arm joints may introduce uncertainties, leading to a decrease in control performance. Therefore, facing the complexity and uncertainty of robotic arms, traditional control design methods are insufficient in terms of control performance, especially when dealing with complex and unknown environments and requiring high-precision trajectory control. This has prompted research into new and more advanced control methods to improve the performance and flexibility of robotic arms in various working environments. Summary of the Invention

[0003] In view of this, in order to solve the technical problem that existing robotic arm control methods do not consider uncertainties, thus leading to limitations in control performance, this invention proposes a robotic arm grasping method based on the value advantage learning of action branch architecture. The method includes the following steps:

[0004] Analyze the characteristics of robotic arm grasping tasks and model them as Markov decision problems;

[0005] Design an action branching architecture-based VA learning algorithm and build a main network and a target network with an action branching structure;

[0006] Based on the priority experience replay mechanism, a greedy strategy is used to generate a crawling strategy for crawling, and an experience replay pool is accumulated.

[0007] Based on the actual scenario, a robotic arm model and scenario information are built. Combining a greedy strategy and the experience replay pool, the target grasping task is completed.

[0008] In some embodiments, the step of generating a crawling strategy using a greedy strategy based on a priority experience replay mechanism and accumulating an experience replay pool further includes:

[0009] The observed state of the robotic arm is input into the Q network to obtain a set of actions;

[0010] Based on a greedy strategy, select the corresponding action;

[0011] Based on the selected action, the system enters a new state and acquires rewards and constraints.

[0012] Put the sample sequence of the new state back into the experience replay pool;

[0013] The above steps are repeated a preset number of times to accumulate experience replay pool.

[0014] In some embodiments, the step of setting up a robotic arm model and scene information according to the actual scenario, and combining a greedy strategy and the experience replay pool to complete the target grasping task specifically includes:

[0015] Set the environmental coordinate system, set the initial state of the robotic arm, determine the workspace of the robotic arm, set scene information such as the desktop and target object, determine the observation state information and motion information of the robotic arm, and determine the update time interval of the robotic arm.

[0016] The VA learning algorithm based on the action branch architecture interacts with the environment and accumulates the experience replay pool using the experience pool processing method in the previous step.

[0017] Based on the target grasping task, the state is input into the Q network to obtain the action output based on the greedy policy, and the main network parameters are updated.

[0018] The parameters of the target network are softly updated using the parameters of the master network.

[0019] This invention also proposes a robotic arm grasping system based on value advantage learning of action branch architecture, the system comprising:

[0020] The modeling module analyzes the characteristics of the robotic arm's grasping task and models it as a Markov decision problem.

[0021] The network construction module designs a VA learning algorithm based on an action branch architecture and builds the main network and target network with the action branch structure.

[0022] The training and accumulation module trains the main network and the target network based on a greedy strategy and accumulates an experience replay pool; the application module is configured according to the actual scenario and combines the greedy strategy and the experience replay pool to complete the target capture task.

[0023] Based on the above scheme, this invention provides a robotic arm grasping method and system based on action branch architecture value advantage learning. Using action branch architecture as the computational network, it can better handle the high-dimensional discrete state action space of the robotic arm, and adds action random exploration parameters to enhance the robotic arm's exploration capability. Using VA-learning as a more efficient alternative to Q-learning, it aims to directly learn the value function and advantage function, rather than indirectly learning through the Q function, thereby improving sample efficiency and learning speed, simplifying the learning problem, and further improving the overall efficiency of the grasping control method. Attached Figure Description

[0024] Figure 1 This is a flowchart of the steps of a robotic arm grasping method based on the value advantage learning of action branch architecture according to the present invention;

[0025] Figure 2 This is a network diagram of the action branch architecture in a specific embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of the encoder frame according to a specific embodiment of the present invention;

[0027] Figure 4 This is a schematic diagram of the priority experience pool update process of the present invention;

[0028] Figure 5 This is a structural block diagram of a robotic arm grasping system based on value advantage learning of action branch architecture according to the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] It should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0031] It should be understood that the terms "system," "apparatus," "unit," and / or "module" used in this application are a method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0032] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "a," and / or "the" are not specifically singular and may include the plural. Generally, the terms "comprising" and "including" only indicate the inclusion of expressly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements. An element defined by the phrase "comprising an..." does not exclude the presence of other identical elements in the process, method, product, or apparatus that includes the element.

[0033] In the description of the embodiments of this application, "a plurality of" refers to two or more. The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0034] Furthermore, flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, the steps can be processed in reverse order or simultaneously. Additionally, other operations can be added to these processes, or one or more steps can be removed from them.

[0035] Reference Figure 1 This is a flowchart illustrating an optional example of the robotic arm grasping method based on action branch architecture value advantage learning proposed in this invention. This method can be applied to computer devices, and the grasping control method proposed in this embodiment may include, but is not limited to, the following steps:

[0036] Step S1: Model the robotic arm grasping task as a Markov decision problem;

[0037] Step S2: Based on the action space, introduce the VA learning algorithm to build the main network and target network with action branch structure;

[0038] Step S3: Based on a greedy strategy, train the main network and the target network, and accumulate an experience replay pool;

[0039] Step S4: Configure the settings according to the actual scenario, combining the greedy strategy and the experience replay pool to complete the target capture task.

[0040] In some feasible embodiments, step S1 specifically includes:

[0041] A simulation scene was built using PyBullet, and a 7-DOF robotic arm, model KUKA LBR iiwa 7R800, equipped with a two-finger gripper, was used to perform the task of grasping objects.

[0042] Based on the characteristics of robotic arm grasping tasks, the quadruple is defined as: (S t ,s a ,r t ,γ);

[0043] S t The state space represents the set of information observed by the agent, specifically the depth image captured by the camera;

[0044] s a The action space represents the set of operations performed by the agent, specifically the three-dimensional coordinate increments of the end effector, the rotation angle of the gripper, and the opening and closing states of the grippers. a = (x,y,z,θ,α);

[0045] r t The reward function is used to evaluate the effectiveness of the agent in performing actions, thereby guiding the agent to learn and achieve the expected goal. The reward function adopts the following binary sparse reward function:

[0046]

[0047] The reward is 1 when the target object is successfully grabbed, and 0 otherwise.

[0048] Through this optimization step, the reward function signal uses a binary sparse reward form, which reduces the complexity of reward function design, simplifies the learning problem, reduces design costs, and enhances interpretability.

[0049] In some feasible embodiments, step S2 specifically includes:

[0050] Because the robotic arm's motion space in this system has five dimensions s a = (x,y,z,θ,α), therefore the action network described in S2 contains five branches. The five-branch network backbone is used to extract features from the RGB-D image to obtain action values ​​in five dimensions.

[0051] The Action Branch Architecture network comprises a shared decision module and multiple network branches, each controlling an independent action dimension. This design allows the number of network outputs to grow linearly with increasing degrees of freedom, rather than combinatorially, thus improving scalability in high-dimensional action spaces. The shared network module computes a latent representation of the input state and then passes it to several action branches. Each action branch controls a single degree of freedom, and the concatenation of selected sub-actions produces a joint action tuple. Each dimension utilizes a duel network architecture to separate the representations of state values ​​and (state-related) action advantages into two independent branches, sharing a common feature learning module between them. These two branches are combined through a special aggregation layer to produce an estimate of the action-value function. In the presence of many similar (or redundant) actions, the duel network architecture can lead to better policy evaluation, resulting in faster generalization in large action spaces. The Action Branch Architecture network is as follows: Figure 2 As shown.

[0052] In this system, the state image of the network passes through an encoder, then through a 512-unit fully connected layer to a 256-unit fully connected layer 2. Next, it passes through 128-unit fully connected layers 3 to 14 and a ReLU activation function, which outputs A values ​​for five dimensions and a shared state value. The number of A values ​​is n, which represents n discrete actions in each dimension. Then, each A value is subtracted from the mean of its respective dimension and added to the state value to obtain the Q value for each action. Finally, the index of the maximum value is taken to obtain the action selected for each dimension.

[0053] The action branch architecture separates individual network branches for each action dimension, while each dimension shares a common convolutional neural network θ with parameters to obtain the same state representation vector s. This vector is then input into different action branches to compute action advantage estimates for each dimension. Here, d represents the d-th action branch. A general value function estimator V is also used. π The state value V(s) is approximated by (s; θ, β), and then used to calculate the state-action value estimate for each dimension d. The Q-value of each action branch in the VA algorithm based on the action branch architecture is calculated as follows:

[0054]

[0055] Where s represents the state vector, a d V(s) represents the action taken by the d-th action branch, and V(s) represents the state value; A d This indicates that in state s, action a d The dominant function; a' d This represents any sub-action under action dimension d; It represents the set of all possible sub-actions under action dimension d.

[0056] Unlike traditional Q-learning, which directly approximates the network to the Q-value, VA-learning decomposes Q(x,a) = V(x) + A(x,a) to converge the network's output value function V and advantage function A to the correctly defined value function and advantage function, respectively. The loss function is as follows:

[0057]

[0058] Where N is the number of action branches. The importance weights are for sampling, where N is the number of action branches, and ω j V represents the weighting coefficient, N represents the size of the experience pool, β represents the non-uniform probability compensation coefficient, and V represents the weighting coefficient. d (x t ) indicates that in x t Below, the estimation of the state-value function of the action dimension d, Indicates in x t Below, the objective value of the state-value function in action dimension d, A d (x t ,a t ) indicates that in state x t and action a t Below, the estimation of the action advantage function for action dimension d, Indicates that in state x t and action a t Below, the objective value of the action advantage function for action dimension d, and the value function and advantage function for each branch are:

[0059]

[0060] in Indicates that in state x t and action a t Below, based on the estimated value of the action value function of the target strategy π, A φ -(x t+1 μ) represents the state x t+1 Under the given condition, the advantage function of all possible actions is weighted according to the behavioral policy μ, where μ represents the behavioral policy, which is a factor in state x. t For all possible actions a t The distribution of the probability of allocation, V θ -(x t ) indicates that in state x t The estimated value of the lower state value function, φ - θ - This represents the parameters of the target network, which are slowly updated towards φ and θ.

[0061] The target network has the same structure as the main network, except for the network parameters.

[0062] Specifically, the encoder network structure is as follows: Figure 3 As shown, the input is a 64×64 depth image. The first 7×7 Cov2d layer reduces the dimensionality of the 64×64-channel feature map to 32×32×32 channels. The second 5×5 Cov2d layer reduces the dimensionality of the 32×32×32-channel feature map to 16×16×32 channels. The third 3×3 Cov2d layer reduces the dimensionality of the 16×16×32-channel feature map to 8×8×32 channels. Each convolutional layer is followed by a LeakyReLU activation function to introduce non-linearity and prevent the gradient vanishing problem. After a series of convolutional layers, a fully connected layer is connected to flatten all the features, finally obtaining a latent space feature vector with a dimension of 101.

[0063] In some feasible embodiments, step S3 specifically includes:

[0064] First, create an experience replay pool. The experience pool stores data in the form of quintuples (S). t A t ,R t ,S t+1 ,done), train the main network and the target network in the experience pool according to batch sampling, where S t Let A be the state of the agent at time t. t R represents the action taken by the agent at time t. t This indicates that the agent is taking action A. t The reward obtained later, S t+1 This indicates that the agent is in state S. t Take action A t The state transitioned to at time t+1, 'done' represents the agent's next state S. t+1 If it is a terminated state, then done = 1; otherwise, done = 0. Next, the observed state S of the robotic arm at time t is recorded. t The input is fed into the Q-network to obtain the action set. Action A is selected according to the ε-greedy greedy policy. t .

[0065]

[0066] When the random probability is 1-ε, the index of the maximum predicted value Q in each action branch result is selected and converted into a motion primitive, which includes the action execution position offset point (x,y,z) and the rotation angle and gripper opening / closing state of the robotic gripper, thus forming the grasping strategy for the current state. When the random probability is 1-ε, a grasping action is executed randomly once to enhance the robotic arm's exploration capabilities.

[0067] Then execute action A. t The system enters a new state S. t+1 And receive reward R t Given the constraint d, the sample sequence (S) of the new state will be... t A t ,R t ,S t+1 (done) is stored in experience pool D;

[0068] Repeat step S3 N times.

[0069] This optimization step processes the experience pool based on prioritized experience replay, improving experience utilization, increasing capture success rate, and enhancing network convergence speed and training efficiency.

[0070] In some feasible embodiments, step S4 specifically includes:

[0071] First, set the environmental coordinate system O-XYZ, set the initial state of the robotic arm, and determine the workspace of the robotic arm;

[0072] Set scene information such as desktop and target object; determine the observation status and motion information of the robotic arm, and determine the update time interval of the robotic arm;

[0073] The VA learning algorithm based on the action branching architecture interacts with the environment, accumulating an experience replay pool using the experience pool processing method in step 3; it samples a minibatch β of size N from the experience replay pool R, and samples N state-action transition processes (s, a, r, s', done) using a sampling method combined with random sampling, following the sampling principle. The specification is given by p(j), where p(j) represents the probability that the j-th transition tuple is sampled. j The priority of the j-th transition tuple is determined by the absolute value of its VA error. α is a hyperparameter that adjusts the influence of priority, while the denominator... It is a normalization factor that ensures the sum of the sampling probabilities of all transition tuples is 1. Then, the sampling importance weights are calculated. Where, ω j β represents the weighting coefficient, N represents the size of the experience pool, and β represents the non-uniform probability compensation coefficient.

[0074] Inputting state s into the Q network yields output action A based on the greedy policy π(s; θ). t θ is the Q-network parameter, updated via gradient descent. The current Q-network parameter θ is updated using the optimizer Adam, with the update formula:

[0075]

[0076] Perform soft updates to the target network parameters based on the main network parameters:

[0077] θ Q′ ←τθ Q +(1-τ)θ Q′

[0078] Where θ Q These are the network parameters of the Q-network, θ Q′ Let τ be the network parameters of the Target network in the target network, and τ be the soft update magnitude of the network parameters, where τ∈(0,1).

[0079] like Figure 4 The experiment uses a priority-based experience replay pool updated with a binary tree-based method (Sumtree). Specifically, a random number within the range [0, sum] is generated. Then, starting from the root node, the left or right child node is selected based on the size of the random number, continuing until a leaf node is reached. The leaf node represents the sampled experience. During the update, only the affected nodes are updated, i.e., all nodes on the path from the leaf node to the root node. The update amount for each node is equal to the sum of the update amounts for its left and right child nodes. The oldest experience trajectory in the experience pool is discarded, and a new set of experiences is accumulated.

[0080] The training continues in a loop until the set number of rounds is reached, at which point the training ends.

[0081] like Figure 5 As shown, a robotic arm grasping system based on action branch architecture value advantage learning includes:

[0082] The modeling module models the robotic arm grasping task as a Markov decision problem.

[0083] The network construction module, based on the action space, introduces the VA learning algorithm to build the main network and target network with action branch structure;

[0084] The robotic arm training module, based on a greedy strategy, accumulates an experience replay pool, which is set according to the actual scenario. Combining the greedy strategy and the experience replay pool, it trains the main network and the target network and completes the target grasping task.

[0085] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0086] A robotic arm grasping device based on value advantage learning of action branch architecture:

[0087] At least one processor;

[0088] At least one memory for storing at least one program;

[0089] When the at least one program is executed by the at least one processor, the at least one processor implements a robotic arm grasping method based on the value advantage learning of an action branch architecture as described above.

[0090] The content of the above method embodiments is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0091] A storage medium storing processor-executable instructions, which, when executed by a processor, are used to implement a robotic arm grasping method based on action branch architecture value advantage learning as described above.

[0092] The content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0093] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for robot arm grasping based on action branch architecture value advantage learning, characterized in that, The method comprises the following steps: modeling a robotic arm grasping task as a Markov decision problem; based on the action space, introducing a VA learning algorithm, building a main network and a target network of the action branch structure; based on the greedy strategy, training the main network and the target network, and accumulating an experience replay pool; according to the actual scene, combining the greedy strategy and the experience replay pool, completing the target grasping task; the step of modeling the robotic arm grasping task as a Markov decision problem specifically comprises: According to the task characteristics of the robot arm grasping task, a four-tuple is defined, which is specifically: ; wherein, is a state space, representing a set of information observed by the agent; is an action space, representing a set of operations performed by the agent; is a reward function, used to evaluate the effect of the agent performing an action; γ is a discount factor; The reward function formula is represented as follows: ； the step of based on the action space, introducing a VA learning algorithm, building a main network and a target network of the action branch structure specifically comprises: According to the dimension of the action space, a corresponding number of action branches are set, and each action branch includes a main network and a target network; The Q value calculation formula of each action branch is as follows: wherein the state vector, represents the action taken by the state value; represents the advantage function for action represents an arbitrary sub-action in action dimension d; represents the set of all possible sub-actions in action dimension d; The loss function of each action branch is specifically represented as follows: wherein, is the number of action branches, denotes a weight coefficient, represents the experience pool size, denotes a non-uniform probability compensation coefficient, denotes the state value function of action dimension d at the estimate of the action advantage function of action dimension d at state the target value of the state value function of action dimension d at the estimate of the action advantage function of action dimension d at state the target value of the action advantage function of action dimension d at state and action the estimate of the action advantage function of action dimension d at state the target value of the action advantage function of action dimension d at state and action the target value of the action advantage function of action dimension d at state 2. The robot arm grasping method based on action branch architecture value advantage learning according to claim 1, characterized in that, The value function and the advantage function of each action branch are specifically represented as follows: where represents the estimated value of the state-value function at state and action under the target policy π, is the average of the advantage function for all possible actions, weighted by the behavior policy μ, at state under the behavior policy μ, represents the estimated value of the state-value function at state under the target policy π, , represents the parameters of the target network.

3. The robot arm grasping method based on action branch architecture value advantage learning according to claim 1, characterized in that, The step of according to the actual scene, combining the greedy strategy and the experience replay pool, completing the target grasping task specifically comprises: Set the environment coordinate system, the initial state of the robotic arm, the working space of the robotic arm and the scene information; Based on the experience replay pool, sample and combine the random sampling method to obtain the state-action conversion process; Based on the target grasping task, input the state into the Q network to obtain the action output based on the greedy strategy, and update the main network parameters; The main network parameters are used to perform soft update on the parameters of the target network.

4. The robot arm grasping method based on action branch architecture value advantage learning according to claim 3, characterized in that, The step of according to the actual scene, combining the greedy strategy and the experience replay pool, completing the target grasping task also comprises: The priority experience replay method based on the binary tree is used to update the priority of the experience replay pool.

5. A mechanical arm grabbing system based on action branch architecture value advantage learning, characterized in that, A robotic arm grasping method based on action branch architecture value advantage learning, comprising: a modeling module, modeling a robotic arm grasping task as a Markov decision problem; a network building module, based on the action space, introducing a VA learning algorithm, building a main network and a target network of the action branch structure; a training and accumulation module, based on the greedy strategy, training the main network and the target network, and accumulating an experience replay pool; an application module, according to the actual scene, combining the greedy strategy and the experience replay pool, completing the target grasping task.

6. A mechanical arm grabbing device based on action branch architecture value advantage learning, characterized in that, comprise: at least one processor; at least one memory for storing at least one program; when the at least one program is executed by the at least one processor, the at least one processor implements the robotic arm grasping method based on the action branch architecture value advantage learning according to any one of claims 1-4.

Images