Robotic arm control method and device
By constructing a simulated environment and a symbolic network learning method, symbolic policies are generated, solving the safety and deployment difficulties of deep neural networks in robotic arm control, and achieving a significant improvement in the precision of robotic arm control.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INST OF COMPUTING TECH CHINESE ACAD OF SCI
- Filing Date
- 2023-12-06
- Publication Date
- 2026-05-19
AI Technical Summary
The complexity of deep neural networks in robotic arm control leads to safety risks and deployment difficulties under computational resource constraints, and existing symbolic policy learning methods are inefficient.
A simulated environment for robotic arm control is constructed, and symbolic networks are used for learning. Symbolic strategies are generated through multi-layer feedforward neural networks, dense connections, symbolic operators, and binary masks. The symbolic network and path probabilities are optimized by combining soft actor criticism algorithms to improve learning efficiency.
Efficiently learn symbolic strategies for multiple continuous motion space control tasks, reduce environmental interaction data requirements by 200 times, and improve the control accuracy of robotic arms.
Smart Images
Figure CN117532610B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of automatic control technology, and more specifically, to a robotic arm control method and apparatus based on symbolic policy learning. Background Technology
[0002] In the field of robotic arm control, reinforcement learning strategies can be used to control robotic arm movements by constructing simulated environments. However, in deep reinforcement learning, strategies are typically implemented using deep neural networks. Deep neural networks contain a large number of parameters and thousands of nested nonlinear operators. Although deep neural networks have excellent representation capabilities for relationships, their complexity makes it difficult to understand predicted behavior, leading to additional safety risks in robotic arm control. Furthermore, complex neural network strategies are difficult to deploy with limited computing resources. Summary of the Invention
[0003] The purpose of this invention is to provide a robotic arm control method and apparatus. This method can improve the efficiency of symbolic policy learning, thereby learning robotic arm control with less interactive data and improving the accuracy of robotic arm control.
[0004] To achieve the above objectives, the present invention provides a robotic arm control method, comprising:
[0005] A simulated environment for controlling the robotic arm is constructed, and environmental state information is obtained, including the pose and speed of the robotic arm and the position of the object to be operated.
[0006] The environmental state information is used as input information and input into the symbol network. The output of the symbol network is the robot arm motion value, which includes the joint velocity of the robot arm.
[0007] A symbol generation strategy is generated by selecting an appropriate path from the symbol network.
[0008] The control tasks of the robotic arm are deployed according to the symbol strategy.
[0009] In some embodiments, the symbol network employs a multilayer feedforward neural network.
[0010] Based on the movement dimensions of the robotic arm, one layer of a symbolic network is constructed for each movement dimension. The input information of each layer of the symbolic network includes the environmental state information, and the output is the robotic arm movement value corresponding to that movement dimension.
[0011] In some embodiments, dense connections are introduced in the symbolic network, wherein the input of each layer of the symbolic network is connected to all subsequent layers.
[0012] The first layer of the symbolic network will receive the environmental state information and the robotic arm motion values output by all the preceding layers.
[0013] In some embodiments, in the symbolic network, a symbolic operator is used instead of an activation function, and the symbolic operator is normalized and a penalty term is used to keep the input from the “forbidden” region.
[0014] In some embodiments, the final weights of the symbolic network are obtained by multiplying the initial weights of the symbolic network by the binary mask.
[0015] The symbol strategy is determined based on the final weights;
[0016] The final weights of the symbolic network are expressed as follows:
[0017]
[0018] in, It is an element-wise multiplication operation, where w represents the initial weights of the symbolic network and m represents the binary mask.
[0019] In some embodiments, the binary mask is sampled from a Bernoulli distribution.
[0020] The binary mask for the i-th path is represented as: m i ~Bern(p i ), p i ∈[0,1] represents the path probability, and Bernoulli represents the Bernoulli distribution;
[0021] The sampling function is expressed as:
[0022]
[0023] Where g1 and g0 represent sampled samples, and the Sigmoid activation function; p = ∑p i , where is the probability sum; τ represents the parameter variable.
[0024] The obtained binary mask is represented as follows:
[0025]
[0026] in, It is an indicator function, x = m gs The overline indicates that the gradient has stopped.
[0027] In some embodiments, a first loss function is constructed to regularize the probability sum, wherein the first loss function is expressed as:
[0028]
[0029] Among them, l min It is the minimum L_0 specification.
[0030] In some embodiments, it further includes:
[0031] Construct a value network relating environmental state information to the robotic arm's motion values;
[0032] Construct a standard deviation network for the environmental state information;
[0033] The symbol network and path probabilities, as well as the value network and standard deviation network, are trained and updated.
[0034] The robotic arm motion values are determined based on the symbol strategy and the standard deviation network.
[0035] In some embodiments, training and updating the symbolic network and path probabilities, as well as the value network and standard deviation network using the SAC soft actor criticism algorithm includes:
[0036] The value network is updated using a second loss function, which directly reuses the critic loss function from the SAC algorithm. The second loss function is expressed as follows: in This represents the critical loss function;
[0037] The standard deviation network, symbolic network, and path probabilities are updated using a third loss function, which is a combination of the actor loss function and penalty term in the SAC algorithm, and the first loss function.
[0038] Where α1 and α2 represent hyperparameters, This represents the actor's loss function and penalty term.
[0039] In some embodiments, the robotic arm motion value is represented as:
[0040]
[0041] Where a represents the robotic arm's motion value, s represents the environmental state information, and π sym Let F(s) represent the symbolic strategy, F(s) represent the standard deviation network, and N represent the Gaussian distribution.
[0042] In another aspect, the present invention provides a robotic arm control device, employing the above-described robotic arm control method, comprising at least:
[0043] The simulation environment construction module is used to construct a simulation environment for the control of the robotic arm and acquire environmental state information, which includes the pose and speed of the robotic arm and the position of the object to be operated.
[0044] The symbol network generation module is used to input the environmental state information into the symbol network, and the output of the symbol network is the robot arm motion value, including the joint speed of the robot arm;
[0045] The path selection module is used to select a suitable path from the symbol network to generate a symbol strategy.
[0046] The application module is used to deploy the control tasks of the robotic arm according to the symbol strategy.
[0047] As can be seen from the above solutions, the advantages of the present invention are:
[0048] This invention provides a robotic arm control method based on symbolic policy learning. It constructs a simulated environment for robotic arm control, acquires environmental state information, and uses this information as input to a symbolic network. The output of the symbolic network is the robotic arm's motion values. A symbolic policy is generated by selecting a suitable path from the symbolic network. Finally, the control task of the robotic arm is deployed according to the symbolic policy. This method can efficiently learn symbolic policies in a test environment consisting of multiple continuous motion space control tasks. The learned symbolic policies have superior performance, thus requiring less interaction data to learn robotic arm control and improving the accuracy of robotic arm control. Attached Figure Description
[0049] Figure 1 This is a schematic diagram of the robotic arm control method provided by the present invention;
[0050] Figure 2 An example of a structure incorporating dense connections in a symbolic network is shown;
[0051] Figure 3 This is a block diagram of the robotic arm control device provided by the present invention;
[0052] in:
[0053] 300 - Robotic arm control device;
[0054] 301 - Simulation Environment Construction Module;
[0055] 302 - Symbolic Network Generation Module;
[0056] 303 - Path Selection Module;
[0057] 304 - Application Module;
[0058] Steps S1-S4. Detailed Implementation
[0059] To make the above features and effects of the present invention clearer and easier to understand, specific embodiments are described below, and detailed descriptions are provided in conjunction with the accompanying drawings.
[0060] As mentioned above, in existing technologies, methods for controlling robotic arms using deep neural networks suffer from additional safety risks due to the complexity of deep neural networks, which make it difficult to understand and predict behavior. Furthermore, complex neural network strategies are difficult to deploy with limited computing resources. Additionally, existing methods for learning symbolic policies indirectly suffer from low data efficiency. To address these issues, this invention specifically optimizes symbolic policies through a gradient-based learning method and applies it to robotic arm control. By improving the efficiency of symbolic policy learning, it learns strategies for controlling robotic arm movements using less interactive data.
[0061] like Figure 1 As shown, Figure 1 The diagram shows the overall flow of the robotic arm control method based on symbolic policy learning provided by the present invention.
[0062] A robotic arm control method, comprising:
[0063] S1. Construct a simulation environment for the control of the robotic arm and obtain environmental state information, which includes the pose and speed of the robotic arm and the position of the object to be operated.
[0064] In this embodiment, it is first necessary to construct a corresponding simulated environment for robotic arm control based on the mechanical control task, and then obtain relevant environmental state information s, specifically including the pose and speed of the robotic arm, as well as the position of the object to be operated.
[0065] S2. The environmental state information is used as input information and input into the symbol network. The output of the symbol network is the robot arm motion value, including the joint speed of the robot arm.
[0066] In this embodiment, after obtaining the environmental state information s, the environmental state information s is used as input information and input into the symbolic network. By training and updating the symbolic network, the output of the symbolic network can obtain the robotic arm motion value a, which specifically includes the joint speed of the robotic arm.
[0067] In its implementation, the symbolic network employs a multi-layer feedforward network. Specifically, based on the movement dimensions of the robotic arm, one layer of the symbolic network is constructed for each movement dimension. The input information of each layer in the symbolic network includes the environmental state information, and the output is the robotic arm movement value corresponding to that movement dimension. For example, for a robotic arm with n joints, the dimension of the movement, i.e., the three-dimensional velocity of the robotic arm joints, is n*3. An n*3-layer symbolic network is used, where the input information of each layer includes the environmental state information, and the output is the robotic arm movement value corresponding to that movement dimension.
[0068] Furthermore, unlike traditional neural networks, this embodiment uses symbolic operators instead of activation functions, such as trigonometric and exponential functions, including sin, cos, exp, log, ×, ÷, etc. For the first layer of the symbolic network, the input information is represented as x. l-1 The parameter is represented as w l ,b l These parameters are used as constants in symbolic expressions. Assume the l-th... th The layer contains m unary functions and n binary functions First, the l th The input of the layer will be linearly transformed by a fully connected layer, y = F l (x)=w l x+b l The fully connected layer implements addition and subtraction in the symbolic expression, generating m+2n outputs. These outputs are then concatenated using symbolic operators to form the layer output, yielding the corresponding robotic arm motion values.
[0069]
[0070] Then the lth symbolic network th A layer can be represented as S l :x l =G l (F l ( x l -1 After the last layer of the symbolic network, there will be a fully connected layer to produce a single output.
[0071] Furthermore, the objective of this embodiment is to find a sign policy with gradients. Therefore, ensuring the numerical stability of the system is crucial. However, this is not natural in sign networks. For example, the division and logarithmic operators will create an extreme point when the input becomes zero, while the exponential function may produce a large output. Therefore, this embodiment maintains the input from the "forbidden" region by normalizing the operators and using a penalty term. For example, the logarithmic operator y = log(x) is valid for x > bound. log and log(bound) log Otherwise, return log(x), with the penalty term defined as L. log =max(bound) log -x,0). The division operator c = a / b for b > bound. div Returns a / b, otherwise returns 0. The penalty term is defined as L. div =max(bound) div -b,0).
[0072] Furthermore, in this embodiment, dense connections are introduced into the symbol network, such as... Figure 2 As shown, the input of each layer of the symbolic network is connected to all subsequent layers; the first layer of the symbolic network will receive the environmental state information s and the robot arm motion values from the outputs of all preceding layers, i.e., x1,…,x l-1 :x l =G l (F l ([s,x1,…,x l-1 This embodiment introduces dense connections into the symbolic network. On the one hand, dense connections improve the information flow between layers, which is beneficial to the training process. On the other hand, dense skip connections across layers can more flexibly control the complexity of symbolic expressions with parameters. In practice, each layer in the symbolic network may contain different operators. Through dense connectivity, the position of operations can be flexibly arranged, such as... Figure 2 As shown, it can be assumed that multiplication and division operations are more likely to occur in the shallow layers of the symbolic network, while more complex operations (such as sine and cosine) are more likely to occur in the deep layers of the symbolic network.
[0073] S3. Select a suitable path from the symbol network to generate a symbol strategy.
[0074] The symbolic policy is generated by selecting appropriate paths from the symbolic network, reducing the number of paths involved in the final policy. Then, suitable paths are retained while redundant paths are removed. This is naturally achieved by minimizing the L_0 norm of the symbolic network weights. In this embodiment, a probabilistic path selector is provided, which obtains the final weights of the symbolic network by multiplying the initial weights by a binary mask, and determines the symbolic policy based on these final weights.
[0075] In the specific implementation, the binary mask is sampled from the Bernoulli distribution, and the binary mask of the i-th path is represented as: m i ~Bern(p i ), p i ∈[0,1] represents the path probability, and Bernoulli denotes the Bernoulli distribution. For the training process, the gumbel-softmax technique is used to construct the sampling function, which is expressed as:
[0076]
[0077] Where g1 and g0 represent sampled samples, and the Sigmoid activation function; p = ∑p i , where p represents the probability sum; τ represents the parameter variable. i ∈[0,1] represents the path probability.
[0078] To obtain a binary mask while preserving the gradient, this embodiment employs the pass-through (ST) technique, and the resulting binary mask m is represented as:
[0079]
[0080] in, It is an indicator function, x = m gs The overline indicates that the gradient has stopped.
[0081] The final weights of the symbolic network are represented as follows:
[0082]
[0083] in, It is an element-wise multiplication operation, where w represents the initial weights of the symbolic network and m represents the binary mask.
[0084] During training, if paths with zero weights are not removed, a symbolic network can be used directly. As a symbolic strategy π sym .
[0085] Furthermore, in this embodiment, a first loss function regularization probability sum is constructed, where the probability sum p is the minimized binary mask. The expectation of the L_0 norm, to constrain the minimum complexity of the sign policy, is expressed as follows:
[0086]
[0087] Among them, l min It is the minimum L_0 specification.
[0088] Minimize binary mask The expectation of the L_0 norm, without penalizing the magnitude of the weights.
[0089] Furthermore, in this embodiment, during steps S3 and S4, when running the symbolic policy in the constructed simulation environment, it is necessary to further train and update the symbolic network and path probabilities to determine convergence. Specifically, this embodiment builds our non-policy learning framework based on the Soft Actor Critique Algorithm (SAC), constructing a value network Q(s,a) about the environmental state information and the robotic arm's action values, and a standard deviation network F(s) about the environmental state information. Q(s,a) and F(s) are only used during symbolic network training. Then, the Soft Actor Critique Algorithm (SAC) is used to train and update the symbolic network and path probabilities, as well as the value network and standard deviation network. Specifically, the value network is updated using a second loss function, which directly reuses the critic loss function from the SAC algorithm. The second loss function is expressed as: in The first loss function represents the critical loss function. The standard deviation network, symbolic network, and path probabilities are updated using a third loss function, which is a combination of the actor loss function and penalty term in the SAC algorithm, and the first loss function. Where α1 and α2 represent hyperparameters, This represents the actor's loss function and penalty term.
[0090] S4. Deploy the control tasks of the robotic arm according to the symbol strategy.
[0091] In this embodiment, after convergence is determined, a symbolic policy π in the form of a symbolic expression is obtained by sampling from the symbolic network based on the path probability. sym Deployed on a robotic arm, it can then be used according to the symbol strategy π sym The standard deviation network F(s) is used to determine the motion value of the robotic arm, which is expressed as:
[0092]
[0093] Where a represents the robotic arm's motion value, s represents the environmental state information, and π sym Let F(s) represent the symbolic strategy, F(s) represent the standard deviation network, and N represent the Gaussian distribution.
[0094] In summary, the robotic arm control method based on symbolic policy learning provided by this invention constructs a simulated environment for robotic arm control, acquires environmental state information, and uses this environmental state information as input to a symbolic network. The output of the symbolic network is the robotic arm's motion values. A symbolic policy is generated by selecting a suitable path from the symbolic network. Finally, the control task of the robotic arm is deployed according to this symbolic policy. This method can efficiently learn symbolic policies in a test environment consisting of multiple continuous motion space control tasks. The learned symbolic policies have superior performance, and the number of environmental interaction trajectories required for learning the symbolic policy is reduced by more than 200 times. Therefore, robotic arm control is learned using less interaction data, improving the accuracy of robotic arm control.
[0095] Furthermore, the above embodiments of the present invention can be applied to terminal devices for robotic arm control methods. These terminal devices may include personal terminals and host computer terminals, etc., and the embodiments of the present invention do not impose limitations on them. The terminal can support operating systems such as Windows, Android, iOS, and Windows Phone.
[0096] Reference Figure 3 , Figure 3 A robotic arm control device 300 is shown, which can realize control via, for example Figure 1The robotic arm control method shown in this application and the robotic arm control device provided in this embodiment can realize each process of the above method.
[0097] A robotic arm control device 300, employing the robotic arm control method of the above embodiments, includes at least:
[0098] The simulation environment construction module 301 is used to construct a simulation environment for the control of the robotic arm and obtain environmental state information, which includes the pose and speed of the robotic arm and the position of the object to be operated.
[0099] The symbol network generation module 302 is used to input the environmental state information into the symbol network, and the output of the symbol network is the robot arm motion value, including the joint speed of the robot arm;
[0100] The path selection module 303 is used to select a suitable path from the symbol network to generate a symbol strategy.
[0101] Application module 304 is used to deploy the control tasks of the robotic arm according to the symbol strategy.
[0102] It should be understood that the descriptions of the robotic arm control methods in this application also apply to the robotic arm control device 300 according to the embodiments of this application, and will not be described in detail again to avoid repetition.
[0103] Furthermore, it should be understood that the robotic arm control device 300 according to the embodiments of this application is only illustrated by the division of the above-described functional modules. In actual applications, the above functions can be assigned to different functional modules as needed. That is, the device can be divided into functional modules different from the modules illustrated above to complete all or part of the functions described above.
[0104] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be applied, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
[0105] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
Claims
1. A robotic arm control method, characterized in that, Include: A simulated environment for controlling the robotic arm is constructed, and environmental state information is obtained, including the pose and speed of the robotic arm and the position of the object to be operated. The environmental state information is used as input to a symbolic network. The output of the symbolic network is the robotic arm motion value, including the joint velocities of the robotic arm. The symbolic network employs a multi-layer feedforward neural network. Based on the robotic arm's motion dimensions, a layer is constructed for each motion dimension. The input information of each layer in the symbolic network includes the environmental state information, and the output is the robotic arm motion value corresponding to that motion dimension. Dense connections are introduced into the symbolic network, where the input of each layer is connected to all subsequent layers. The first layer of the symbolic network receives the environmental state information and the robotic arm motion values output from all preceding layers. A symbol generation strategy is generated by selecting an appropriate path from the symbol network. The control tasks of the robotic arm are deployed according to the symbol strategy.
2. The method according to claim 1, characterized in that, In the symbolic network, symbolic operators are used instead of activation functions, and the symbolic operators are normalized and a penalty term is used to keep the input from the "forbidden" region.
3. The method according to claim 1, characterized in that, The final weights of the symbolic network are obtained by multiplying the initial weights of the symbolic network by the binary mask, and the symbolic strategy is determined based on the final weights. The final weights of the symbolic network are expressed as follows: ， in, It is an element-wise multiplication operation, where w represents the initial weights of the symbolic network and m represents the binary mask.
4. The method according to claim 3, characterized in that, The binary mask is sampled from a Bernoulli distribution. The binary mask for the i-th path is represented as: , Here, Bernoulli represents the path probability; The sampling function is expressed as: Where g1 and g0 represent the sampled samples, and the Sigmoid activation function; , where is the sum of probabilities; Indicates parameters; The obtained binary mask is represented as follows: ， in, It is an indicator function, x=m gs The overline indicates that the gradient has stopped.
5. The method according to claim 4, characterized in that, Also includes: Construct the first loss function to regularize the probability sum, and the first loss function is expressed as: Among them, l min It is the minimum L_0 specification.
6. The method according to claim 5, characterized in that, Also includes: Construct a value network relating environmental state information to the robotic arm's motion values; Construct a standard deviation network for the environmental state information; The symbol network and path probabilities, as well as the value network and standard deviation network, are trained and updated. The robotic arm motion values are determined based on the symbol strategy and the standard deviation network.
7. The method according to claim 6, characterized in that, The symbolic network and path probabilities, as well as the value network and standard deviation network, are trained and updated using the SAC soft actor criticism algorithm, including: The value network is updated using a second loss function, which directly reuses the critic loss function from the SAC algorithm. The second loss function is expressed as follows: ,in This represents the critical loss function; The standard deviation network, symbolic network, and path probabilities are updated using a third loss function, which is a combination of the actor loss function and penalty term in the SAC algorithm, and the first loss function. in, , Indicates hyperparameters, , This represents the actor's loss function and penalty term.
8. The method according to claim 6, characterized in that, The robotic arm motion value is expressed as: Where 'a' represents the robotic arm's motion value, and 's' represents the environmental state information. Let F(s) represent the symbolic strategy, F(s) represent the standard deviation network, and N represent the Gaussian distribution.
9. A robotic arm control device, characterized in that, The robotic arm control method according to any one of claims 1-8 includes at least: The simulation environment construction module is used to construct a simulation environment for the control of the robotic arm and acquire environmental state information, which includes the pose and speed of the robotic arm and the position of the object to be operated. The symbol network generation module is used to input the environmental state information into the symbol network, and the output of the symbol network is the robot arm motion value, including the joint speed of the robot arm; The path selection module is used to select a suitable path from the symbol network to generate a symbol strategy. The application module is used to deploy the control tasks of the robotic arm according to the symbol strategy.