A control method for a robotic arm to perform tasks based on model fusion
By fusing the parameters of multiple reinforcement learning models and constructing a fusion model using weight networks and reward function networks, the problem of high computational load for robotic arms in multi-task environments is solved, enabling efficient task execution in systems with limited computing power.
Patent Information
- Application Number
- CN202410560557.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-08
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-05-08
AI Technical Summary
Existing robotic arms perform poorly in multi-task environments, and existing multi-task reinforcement learning methods are computationally intensive and cannot be deployed in robotic arm control systems with limited computing power.
A model fusion-based control method is adopted, which integrates the parameters of multiple reinforcement learning models and uses weight networks and reward function networks for weighting to construct a fusion model, thereby reducing the amount of computation and adapting to different tasks.
It achieves good control performance even when the task changes, reduces the amount of computation, is suitable for robotic arm control systems with limited computing power, and improves model performance and learning and training speed.
Smart Images

Figure CN118493379B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of robot control technology, and in particular to a control method for a robot arm to perform tasks based on model fusion. Background Art
[0002] With the rapid development of robotic arm technology, robotic arms have been widely used in industrial tasks such as handling, stacking, cutting, and welding, which not only liberates manpower but also improves the efficiency and quality of industrial production.
[0003] In the traditional control field, the control and motion planning of mobile robotic arms usually involve kinematic modeling of the robotic arm and solving the end position and angle values of each joint. As the complexity of the actual application environment continues to increase, the traditional model- and rule-based control methods will become more and more complex in modeling the environment and solving the kinematics of the robotic arm.
[0004] As an important technology of artificial intelligence, reinforcement learning is increasingly being used in the control of robotic arm movements. The essence of reinforcement learning is a machine learning method that uses environmental feedback as input and adapts to the environment. It imitates the basic way humans or animals learn in nature, and uses evaluative feedback signals to optimize the behavioral decisions taken through continuous interaction with the environment and trial and error.
[0005] However, many current studies focus on a single task for the robotic arm. When the properties of the target and the object change, the model that trains the robotic arm to push the object to the target to complete the task cannot complete the task of grabbing the object to the target. In other words, the model trained using a single task cannot achieve the same effect as during training, limiting the applicability of the robotic arm in certain scenarios. Some existing multi-task reinforcement learning focuses on achieving knowledge sharing through changes in model structure, which requires setting too many parameters for multiple tasks, resulting in a large amount of calculation and cannot be deployed in a robotic arm control system with limited computing power. Summary of the invention
[0006] In view of the above analysis, an embodiment of the present invention aims to provide a control method for a robot arm to perform tasks based on model fusion, so as to solve the problem that the existing robot arm has poor performance when performing multiple tasks.
[0007] An embodiment of the present invention provides a control method for a robot arm to perform a task based on model fusion, comprising the following steps:
[0008] Deploy the fusion model to the control system of the robot arm. The fusion model is obtained by weighting the parameters of multiple trained reinforcement learning models according to the weights.
[0009] Initialize the working environment of the robotic arm, collect information about objects and targets based on the task, and obtain the state space; pass the state space into the fusion model, and the policy network of the fusion model outputs the action space. After controlling the robotic arm to perform corresponding actions according to the action space, update the state space, and then pass the updated state space into the fusion model until the robotic arm is controlled according to the output action space to complete the task.
[0010] Based on the further improvement of the above method, the robotic arm includes: a base, an arm, a joint, an end effector and a depth camera. The base is the base of the robotic arm and is fixed during movement; the arm is the support of the robotic arm; the joint is the rotating joint of the robotic arm and is used to locate the position of the end effector; the end effector is used to perform tasks; the depth camera is used to collect the position of the end effector, as well as the position of targets and objects.
[0011] Based on the further improvement of the above method, the reinforcement learning model includes a policy network and a value network, which are trained in the simulation environment using the policy gradient algorithm in the following two ways: one is to train for the same task without setting a random seed, with the target and object attributes, action space, state space, and reward function being exactly the same; the second is to train for different tasks without setting a random seed, with the action space, state space, and reward function being the same but the target and object attributes being different.
[0012] Based on the further improvement of the above method, the weights are obtained by the following steps:
[0013] Based on each reinforcement learning model, each piece of data generated by the multiple interactions between the robot arm and the simulation environment is constructed into a sample set; the sample set is used to train a neural network whose output is a reward value to obtain a reward function network;
[0014] A weight layer is added before the input layer of the reward function network to form a weight network. Based on each reinforcement learning model, the weight network is trained using data generated by interacting once with the simulation environment in the same state space to obtain the weight of each reinforcement learning model corresponding to the weight layer.
[0015] Based on the further improvement of the above method, the data generated by the multiple interactions between the robot arm and the simulation environment are constructed into a sample set, including: taking the state space at the current moment in each data, as well as the mean and variance of each dimension of the action space as a sample, and taking the reward value fed back by the simulation environment in each data as the sample label of the corresponding sample, and putting them into the sample set.
[0016] Based on the further improvement of the above method, the reward value fed back by the simulation environment is the negative of the distance between the object and the target at the current moment.
[0017] Based on the further improvement of the above method, the sample set is used to train the neural network whose output is the reward value to obtain the reward function network, which is obtained by concatenating the mean and variance of each dimension of the state space and action space in each sample at the current moment, and passing them into the neural network. The mean square error between the neural network output value and the sample label is used as the loss function, and the neural network parameters are updated by the gradient descent method to obtain the trained reward function network.
[0018] Based on the further improvement of the above method, the weight layer is a linear weighted connection layer, the input is the mean and variance of each dimension of the action space output by the policy network of each reinforcement learning model, and the output is the weighted mean and variance of each dimension of the action space.
[0019] Based on the further improvement of the above method, based on each reinforcement learning model, the weight network is trained using the data generated by interacting once with the simulation environment in the same state space to obtain the weight of each reinforcement learning model corresponding to the weight layer, including:
[0020] The same state space is passed into each reinforcement learning model to output different action spaces. According to the action spaces, each model interacts with the simulation environment once to obtain different reward values fed back by the simulation environment.
[0021] The means and variances of each dimension in different action spaces are passed into the weight layer, and the weight parameters in the weight layer are used to linearly weight the means and variances of the same dimension respectively. The weighted means and variances of each dimension are output, and then concatenated with the state space at the current moment and passed into the reward function network. The parameters of the reward function network remain unchanged, and the mean square error between the output value of the reward function network and the maximum reward value fed back by the simulation environment is used as the loss function. The weight parameters of the weight layer are updated by the gradient descent method. After the training, the weight parameters of the weight layer are used as the weights of the corresponding reinforcement learning model.
[0022] Based on the further improvement of the above method, the state space includes: the observation value at the current moment, the observation value at the previous moment and the target position at the current moment, wherein the observation value includes: the end effector position, the normalized opening and closing degree of the end effector, the object position and the object moving speed; the action space includes: the end effector position and the normalized opening and closing degree of the end effector.
[0023] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:
[0024] 1. By integrating multiple reinforcement learning models, it avoids setting too many parameters to adapt to different tasks and reduces the amount of calculation. It is suitable for deployment in the control system of a robotic arm with limited computing power.
[0025] 2. The reinforcement learning model is learned multiple times by setting multiple different tasks; the state is constructed by training the reward function network, and the output of the policy network of the reinforcement learning model is mapped to the reward value, and then the weights are obtained by training the weight network to fuse multiple reinforcement learning models. The resulting fusion model improves the model performance and supports multiple tasks, achieving good control effects even when the tasks change, and successfully completing the tasks.
[0026] 3. The proposed model fusion method based on gradient descent can adapt to all reinforcement learning paradigms, is suitable for reinforcement learning models trained by the same algorithm, and obtains reward values through the reward function network, which speeds up the learning and training speed of multi-agent systems when facing multiple tasks.
[0027] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.
[0029] Figure 1 This is a flow chart of a control method for a robotic arm to perform a task based on model fusion in an embodiment of the present invention;
[0030] Figure 2 This is a schematic diagram of the structure of the reward function network in an embodiment of the present invention;
[0031] Figure 3 Schematic diagram of the structure of the weight network in an embodiment of the present invention. DETAILED DESCRIPTION
[0032] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.
[0033] A specific embodiment of the present invention discloses a control method for a robot arm to perform a task based on model fusion, such as Figure 1 As shown, the following steps are included:
[0034] S1. Deploy the fusion model to the control system of the robot arm. The fusion model is obtained by weighting the parameters of multiple trained reinforcement learning models according to the weights;
[0035] S2. Initialize the working environment of the robot arm, collect information about objects and targets based on the task, and obtain the state space; pass the state space into the fusion model, and the strategy network of the fusion model outputs the action space. After controlling the robot arm to perform corresponding actions according to the action space, update the state space, and then pass the updated state space into the fusion model until the robot arm is controlled according to the output action space to complete the task.
[0036] During implementation, the robot arm includes: a base, an arm, a joint, an end effector and a depth camera. The base is the base of the robot arm, which is fixed during movement; the arm is the bracket of the robot arm; the joint is the rotating joint of the robot arm, including a servo motor, a reducer and an encoder, which are used to control the movement of the robot arm and locate the position of the end effector; the end effector is used to perform tasks; the depth camera is used to collect the position of the end effector, as well as the position of the target and the object. The tasks in this embodiment include pushing objects to the target and grabbing objects to the target. Within the range of the object and the target, based on the action space output by the fusion model, the robot arm interacts with the working environment multiple times to complete the set tasks.
[0037] The state space constructed in this embodiment includes: the observation value at the current moment, the observation value at the previous moment and the target position at the current moment, wherein the observation value includes: the end effector position, the normalized opening and closing degree of the end effector, the object position and the object moving speed; the action space includes: the end effector position and the normalized opening and closing degree of the end effector.
[0038] It should be noted that the normalized opening and closing degree of the end effector is 1D, the position includes the 3D Cartesian coordinate value, and the object movement speed includes the speed value on the 3D Cartesian coordinate system, which is calculated based on the position difference and time interval of the object at adjacent moments. Therefore, in this embodiment, the state space is a 23D continuous state space, and the action space is a 4D continuous action space.
[0039] The reinforcement learning model in step S1 includes a policy network and a value network. This embodiment does not limit the model structures of the two networks, but each reinforcement learning model has a policy network with the same model structure and a value network with the same model structure. The model structures of the policy network and the value network can be the same or different. For example, both can use a two-layer fully connected network.
[0040] It should be noted that the input of the policy network is the state space, and the output is the mean and variance of each dimension of the action space. The corresponding action is obtained by randomly selecting the value of each dimension according to the Gaussian probability distribution of the mean and variance; the input of the value network is the state space, and the output is the value of the state space under the current action.
[0041] Furthermore, based on the simulation environment, the policy gradient algorithm is used to train the reinforcement learning model in the following two ways: one is to train for the same task without setting a random seed, with the target and object attributes, action space, state space, and reward function being exactly the same; the second is to train for different tasks without setting a random seed, with the action space, state space, and reward function being the same but the target and object attributes being different.
[0042] In reinforcement learning, the policy gradient algorithm is used to solve the reinforcement learning problem of model-free, continuous state and action space. Unlike the value function-based method, the policy gradient algorithm directly optimizes the policy function so that the robot arm can learn the optimal policy. Specifically, the policy gradient algorithm calculates the probability distribution of taking actions in each state. Then, the policy parameters are updated by the gradient ascent method, so that the probability of selecting high-reward actions increases, while the probability of selecting low-reward actions decreases. In this way, the strategy is improved step by step until the optimal policy function is found.
[0043] It should be noted that setting the same random seed value during training can make the training process repeatable and obtain the same network parameters. Conversely, not setting the random seed can obtain different network parameters. Thus, multiple reinforcement learning models with the same model structure but different network parameters can be obtained.
[0044] After the reinforcement learning model is trained, the parameters of multiple reinforcement learning models are weighted according to the weights to obtain a fusion model. The fusion model is obtained by constructing a fusion model with the same structure as the reinforcement learning model, and taking the sum of the products of the network parameters at the same position of each reinforcement learning model and the weight of the reinforcement learning model as the network parameters at the same position of the fusion model.
[0045] Compared with the prior art, this embodiment uses weights to fuse multiple reinforcement learning models to obtain a fusion model that has the same structure as the reinforcement learning model and integrates the characteristics of all reinforcement learning models, avoiding setting too many parameters to adapt to different target tracking tasks, reducing the amount of calculation, and improving model performance.
[0046] Specifically, the weights of each reinforcement learning model are obtained through the following steps:
[0047] ① Based on each reinforcement learning model, the data generated by the multiple interactions between the robot arm and the simulation environment are constructed into a sample set; the sample set is used to train a neural network whose output is a reward value to obtain a reward function network.
[0048] ② Add a weight layer before the input layer of the reward function network to form a weight network. Based on each reinforcement learning model, the weight network is trained using the data generated by interacting once with the simulation environment in the same state space to obtain the weight of each reinforcement learning model corresponding to the weight layer.
[0049] That is, this embodiment constructs a mapping between the output of the policy network of the state space and the reinforcement learning model and the reward value by training the reward function network, and then obtains the weight by training the weight network.
[0050] It should be noted that since the reward value is an indicator for evaluating the quality of a strategy or an action in reinforcement learning, the reward value of the current action can only be determined by the action and the state after the action is executed, but the policy network does not directly output the action or reward value. Therefore, in step ①, the weighted fusion method based on gradient descent is used to construct a mapping from the state and the corresponding output of the policy network to the reward value based on the gradient descent of the neural network.
[0051] The process of interaction between the robotic arm and the simulation environment is as follows: based on the state space of the simulation environment at the current moment, the robotic arm outputs the mean and variance of each dimension of the action space under the control of the policy network, and randomly selects the value of each dimension according to the Gaussian probability distribution of the mean and variance to obtain the action. The robotic arm executes the action to interact with the simulation environment once, generates the state space at the next moment, continues to select actions according to the policy network, executes the action to interact with the simulation environment once, and completes the task.
[0052] A sample set is constructed based on each piece of data generated by multiple interactions, including: taking the state space at the current moment in each piece of data, and the mean and variance of each dimension of the action space output by the policy network of the reinforcement learning model as a sample, calculating the negative of the distance between the object and the target at the current moment, obtaining the reward value of the simulation environment, and taking the reward value of the simulation environment as the sample label of the corresponding sample and putting it into the sample set.
[0053] like Figure 2 As shown in the figure, the sample set is used to train a neural network whose output is a reward value to obtain a reward function network. The mean and variance of each dimension of the state space and action space at the current moment in each sample are concatenated and passed into the neural network. The mean square error between the neural network output value and the sample label is used as the loss function. The neural network parameters are updated through the gradient descent method to obtain a trained reward function network.
[0054] It should be noted that Figure 2 There are n reinforcement learning models in , and the sample data of model 1 (i.e. the first reinforcement learning model) is in, Represents the current state space; since the action space is 4-dimensional, each dimension corresponds to a mean and variance, the 4 means are abbreviated as The four variances are abbreviated as
[0055] Step ② adds a weight layer before the input layer of the reward function network to form a weight network, and trains the mapping from the output of the policy network of different reinforcement learning models in the same state space to the reward value. The weight parameters to be trained in the weight layer are the weights of each reinforcement learning model that is expected to be obtained. The weight layer is a linear weighted connection layer. The input is the mean and variance of each dimension of the action space output by the policy network of each reinforcement learning model, and the output is the weighted mean and variance of each dimension of the action space.
[0056] like Figure 3 As shown, based on each reinforcement learning model, the weight network is trained using the data generated by interacting once with the simulation environment in the same state space to obtain the weight of each reinforcement learning model corresponding to the weight layer, including:
[0057] The same state space is passed into each reinforcement learning model to output different action spaces. According to the action spaces, each model interacts with the simulation environment once to obtain different reward values fed back by the simulation environment.
[0058] The means and variances of each dimension in different action spaces are passed into the weight layer, and the weight parameters in the weight layer are used to linearly weight the means and variances of the same dimension respectively. The weighted means and variances of each dimension are output, and then concatenated with the state space at the current moment and passed into the reward function network. The parameters of the reward function network remain unchanged, and the mean square error between the output value of the reward function network and the maximum reward value fed back by the simulation environment is used as the loss function. The weight parameters of the weight layer are updated by the gradient descent method. After the training, the weight parameters of the weight layer are used as the weights of the corresponding reinforcement learning model.
[0059] Exemplarily, the state space in each sample is taken out from the sample set, passed into each reinforcement learning model, and the respective different action spaces are output.
[0060] It is understandable that for reinforcement learning models with the same model structure obtained by training with the same algorithm, including but not limited to the PPO algorithm, DDPG algorithm, SAC algorithm, etc., when the reward function of the task is not sparse and the strategy action time is independent, the reward function network is first trained based on gradient descent, and then the parameters of the reward function network are solidified, and the weight parameters are trained to speed up the learning and training speed for multiple tasks.
[0061] In step S2, the working environment of the robot arm is initialized, including: taking the base of the robot arm as the world coordinate system, constructing the action area of the robot arm according to the appearance range of the object to be pushed / grasped and the range of the target; using the depth camera to collect the position of the end effector, as well as the position of the target and the object, and converting them to the world coordinate system.
[0062] In the process of executing the task, the policy network based on the fusion model controls the robotic arm to perform corresponding actions and iteratively interacts with the real environment until the task is completed.
[0063] Compared with the prior art, the control method of a robot arm performing tasks based on model fusion provided in this embodiment is to learn the reinforcement learning model multiple times by setting multiple different tasks; to construct the state and output of the policy network of the reinforcement learning model to the reward value by training the reward function network, and then to fuse multiple reinforcement learning models after obtaining the weights by training the weight network. The obtained fusion model improves the model performance and supports multiple tasks, so that a good control effect can be achieved even when the task changes, and the task is completed smoothly. By fusing multiple reinforcement learning models, it is avoided to set too many parameters to adapt to different tasks, and the amount of calculation is reduced, which is suitable for deployment in the control system of a robot arm with limited computing power.
[0064] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.
[0065] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.
Claims
1. A control method for a robotic arm to perform tasks based on model fusion, It is characterized in that The following steps are involved: Deploying a fusion model to a control system of a robotic arm, wherein the fusion model is obtained by weighting parameters of multiple trained reinforcement learning models according to weights; Initialize the working environment of the robot arm, collect information about objects and targets based on the task, and obtain the state space; pass the state space into the fusion model, the policy network of the fusion model outputs the action space, and after controlling the robot arm to perform corresponding actions according to the action space, update the state space, and then pass the updated state space into the fusion model until the robot arm is controlled to complete the task according to the output action space; The reinforcement learning model includes a policy network and a value network, which are trained by the policy gradient algorithm based on a simulation environment in the following two ways: one is to train the same task without setting a random seed under the condition that the target and object attributes, action space, state space, and reward function are exactly the same; the second is to train the same action space, state space, and reward function under the condition that the target and object attributes are different but the random seed is not set for different tasks; The weight is obtained by the following steps: based on each reinforcement learning model, each piece of data generated by the interaction between the robot arm and the simulation environment for multiple times is constructed into a sample set; the sample set is used to train a neural network whose output is a reward value to obtain a reward function network; a weight layer is added before the input layer of the reward function network to form a weight network, and based on each reinforcement learning model, the weight network is trained using data generated by interacting with the simulation environment once in the same state space to obtain the weight of each reinforcement learning model corresponding to the weight layer; The data generated by the multiple interactions between the robot arm and the simulation environment are constructed into a sample set, including: taking the state space at the current moment in each data, and the mean and variance of each dimension of the action space as a sample, taking the reward value fed back by the simulation environment in each data as the sample label of the corresponding sample, and putting them into the sample set; the reward value fed back by the simulation environment is the negative of the distance between the object and the target at the current moment; The state space includes: the observation value at the current moment, the observation value at the previous moment and the target position at the current moment, wherein the observation value includes: the end effector position, the normalized opening and closing degree of the end effector, the object position and the object moving speed; the action space includes: the end effector position and the normalized opening and closing degree of the end effector.
2. The control method for a robot arm to perform a task based on model fusion according to claim 1, It is characterized in that The robotic arm includes: a base, an arm, a joint, an end effector and a depth camera. The base is the base of the robotic arm and is fixed during movement; the arm is the support of the robotic arm; the joint is the rotating joint of the robotic arm and is used to locate the position of the end effector; the end effector is used to perform tasks; and the depth camera is used to capture the position of the end effector, as well as the position of targets and objects.
3. The control method for a robot arm to perform a task based on model fusion according to claim 2, It is characterized in that The method of using the sample set to train a neural network whose output is a reward value to obtain a reward function network is to concatenate the means and variances of each dimension of the state space and action space at the current moment in each sample, and pass them into the neural network. The mean square error between the output value of the neural network and the sample label is used as the loss function, and the neural network parameters are updated by the gradient descent method to obtain a trained reward function network.
4. The control method for a robot arm to perform a task based on model fusion according to claim 2, It is characterized in that The weight layer is a linear weighted connection layer, the input is the mean and variance of each dimension of the action space output by the policy network of each reinforcement learning model, and the output is the weighted mean and variance of each dimension of the action space.
5. The control method for a robot arm to perform a task based on model fusion according to claim 2, It is characterized in that The method of training a weight network based on each reinforcement learning model by using data generated by interacting once with the simulation environment in the same state space to obtain the weight of each reinforcement learning model corresponding to the weight layer includes: The same state space is passed into each reinforcement learning model to output different action spaces. According to the action spaces, each model interacts with the simulation environment once to obtain different reward values fed back by the simulation environment. The means and variances of each dimension in different action spaces are passed into the weight layer, and the weight parameters in the weight layer are used to linearly weight the means and variances of the same dimension respectively. The weighted means and variances of each dimension are output, and then concatenated with the state space at the current moment and passed into the reward function network. The parameters of the reward function network remain unchanged, and the mean square error between the output value of the reward function network and the maximum reward value fed back by the simulation environment is used as the loss function. The weight parameters of the weight layer are updated by the gradient descent method. After the training, the weight parameters of the weight layer are used as the weights of the corresponding reinforcement learning model.
