A robotic arm control method based on guided DQN control

Abstract

The present invention discloses a robotic arm control method based on guided DQN control, comprising the following steps: Markov modeling of the robotic arm control strategy, 1) establishing a perception environment model, 2) designing the robotic arm action space, 3) establishing a state model, 4) designing a reward function, 5) a data enhancement method, 6) designing a guided DQN strategy, and 7) designing a robotic arm control program structure; the robotic arm control method based on guided DQN control designed by the present invention can realize deep exploration, improve the instability caused by the random sampling strategy of the original DQN, has better performance, and improves the success rate of the robotic arm grasping objects.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a robot arm control method based on guided DQN control. Background Art

[0002] As we all know, intelligent robots are a hot topic of research at present and in the future. With the continuous deepening of research on robotics technology, people gradually realize that the essence of robotics technology is the combination of perception, decision-making, action and interaction technology. In the process of developing intelligent robots, people have mastered some key technologies, such as multi-sensor information processing technology, path planning technology, navigation and positioning technology, intelligent control and human-machine interface technology.

[0003] However, with the development of artificial intelligence technology, the shortcomings of traditional intelligent robot control theory have gradually emerged, such as the inability to accurately analyze and model physical objects and the inability to solve pathological processes with insufficient information. The automation of the manufacturing industry is inseparable from the robot arm. Traditional robot arm control is mainly based on single-chip microcomputers, sensors or embedded systems. The traditional robot arm is modified by adding sensors to the traditional robot arm, so that the robot arm has external perception functions, which are similar to some external functions of humans, and its flexibility is effectively improved, but the information obtained by the sensor often has a large error with the environment.

[0004] In recent years, deep reinforcement learning has rapidly become an important frontier in the field of artificial intelligence. Algorithms based on deep reinforcement learning have made breakthrough progress in various fields, especially in the field of intelligent robot control. Deep reinforcement learning can interact well with the surrounding environment and continuously learn from previous experience, which overcomes the problem that the robot arm cannot learn well. Deep learning makes it possible to extract high-level features from raw input data. Although it performs very well in perception, it is not satisfactory in decision-making. In contrast, reinforcement learning performs well in decision-making, but has no outstanding ability in perception. Therefore, combining deep learning with reinforcement learning to form a deep reinforcement learning algorithm, the two complement each other's advantages, and can provide an effective method for solving the perception and decision-making problems of complex systems. Deep reinforcement learning combines the perception ability of deep learning with the decision-making ability of reinforcement learning, and can be controlled directly according to the input image. It is an artificial intelligence method that is closer to human thinking.

[0005] The DQN algorithm, which combines convolutional neural networks with the Q-learning algorithm in traditional reinforcement learning, is a pioneering work in the field of deep reinforcement. It solves the problem that traditional intelligent robots cannot perceive the environment when making behavioral decisions. Therefore, it is very important to use DQN and its improved algorithm in a real environment to enable robots to learn the optimal strategy directly based on high-dimensional perception input. Based on the guided DQN manipulator control strategy algorithm, a self-service sampling method is adopted, and multiple shunt networks are used to randomize the value function, temporarily expand the exploration range of the state space, and realize deep exploration. Through this distributed deep exploration method, the exploration of different strategies by the intelligent agent is fully guaranteed, and diversified samples are generated, so that the dynamic information of the environment can be better generalized to the state space of the position. Summary of the invention

[0006] In view of the problems existing in the prior art, the object of the present invention is to provide a robot arm control method based on guided DQN control.

[0007] The technical solution adopted by the present invention to solve the technical problem is: a robot arm control method based on guided DQN control, comprising the following steps:

[0008] 1) Establish a perception environment model: The camera is used to obtain the state of the environment in which the robot is located in real time, and all information used for decision-making is obtained from the environment. The action selected by the decision is used as a control signal to act on the environment. At the same time, the system perception environment model can monitor the dynamic changes of the environment in real time;

[0009] 2) Robotic arm motion space design: Choose a method to control the robot arm using feedback, that is, specify the spatial position of the robot arm, and then feed it back to the robot arm system, which will independently calculate the relative movement angle of each joint point;

[0010] 3) Establishing the state model: Using a camera to shoot the robot arm environment as the state information, the robot arm system preprocesses the collected state image. For the robot arm strategy control system, color information is redundant information. In the first step of preprocessing, the 8-bit true color image is converted into a grayscale image, and different weights are given during the conversion;

[0011] 4) Reward function design: The reward function is a key issue in policy control. The optimal control strategy must be considered, which has environmental universality. Three factors that affect the reward function should be considered: time factor, distance factor, and environmental robustness. Considering the time factor allows the robot arm to grasp the target as quickly as possible, and considering the distance factor is determined by the task of grasping the target.

[0012] 5) Data enhancement method: Adopting autonomous collection mechanism and action transposition mechanism, first collect a small data set through VR / AR equipment, use this small-scale data set to initialize the reinforcement learning algorithm experience pool, pre-train the reinforcement learning algorithm, and then use the pre-trained model to strategically control the robot arm. At the same time, record the control process, update the experience pool, and then use the updated experience pool to train the reinforcement learning algorithm, and repeat this cycle; adopting the action transposition mechanism, the previous state and the next state are interchangeable, and only the action and loss function need to be changed;

[0013] 6) Guided DQN strategy design: After being processed by a shared network structure, the algorithm generates k head nodes online and in parallel, representing the estimated values ​​of k Q-value functions Q1, Q2, ..., Qk, respectively, and temporarily expands the uncertainty estimate of the quantized value function generated by their respective TD errors. The higher the uncertainty of the value function corresponding to an action, the higher the reward value brought by exploring the action, which can adapt to changing dynamic environments.

[0014] 7) Design of the robot arm control program structure: The main purpose is to establish an effective control system to coordinate the different modules of the program and all subsystems of the robot layer. The robot arm control system consists of two parts: the main program and the auxiliary program.

[0015] Specifically, the robot arm motion space design in step 2 discretizes the robot arm motion space, does not continuously control the robot arm, and defines control commands with different motion steps, so that the robot arm can approach the target object with a certain step length.

[0016] Specifically, in step 3, the robot arm environment is photographed by a camera as an 8-bit true color image, and its resolution of 320×240 is relatively high for the deep Q value network.

[0017] Specifically, the true color image in step 3 is converted into a grayscale image, and after the grayscale processing, the image is unified into a size of 84×84, and the pre-processed 84×84 grayscale image is used as the state S in the Markov process.

[0018] Specifically, the environmental robustness in step 4 is that when the environment in which the robot is located undergoes a significant change, the designed reward function is still applicable.

[0019] Specifically, the main program in step 7 includes a deep reinforcement learning algorithm module, a communication module and a sampling data processing module, and the auxiliary program serves as a bridge between the main program and the robot layer and runs independently of the main program in the experiment.

[0020] The present invention has the following beneficial effects:

[0021] The robotic arm control method based on guided DQN control designed by the present invention can achieve deep exploration, improve the instability caused by the random sampling strategy of the original DQN, has better performance, and improves the success rate of the robotic arm grasping objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is the block diagram of the robotic arm control system architecture.

[0023] Figure 2 It is a block diagram of the communication method of the robotic arm control platform.

[0024] Figure 3 It is a schematic diagram of the guided DQN model structure.

[0025] Figure 4 This is the structure diagram of the robot arm control program. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely further describe the technical solutions in the embodiments of the present invention. The following will be clearly and completely further describe the technical solutions in the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0027] like Figure 1-4 As shown, a robotic arm control method based on guided DQN control includes the following steps: 1) establishing a perception environment model, 2) designing the robotic arm action space, 3) establishing a state model, 4) designing a reward function, 5) data enhancement method, 6) designing a guided DQN strategy, and 7) designing a robotic arm control program structure.

[0028] 1) Establish a perception environment model: The state of the environment in which the robot is located is obtained in real time through the camera. The entire experimental scene is simulated into a deep reinforcement learning environment. All information used for decision-making is obtained from the environment. The decision-making action is used as a control signal to act on the environment. At the same time, the system perception environment model can monitor the dynamic changes of the environment in real time.

[0029] 2) Design of the robot motion space: Choose a feedback method to control the robot, that is, specify the spatial position of the robot, and then feed it back to the robot system, which will then autonomously calculate the relative movement angle of each joint point. At the same time, discretize the robot motion space, do not continuously control the robot, and define control commands with different motion steps, so that the robot can approach the target object with a certain step length.

[0030] Taking the end effector of the robot as the coordinate origin, for the three-dimensional coordinate system, each dimension has three possible changes (-1: negative direction movement, 0: static, 1: positive direction movement), so there are 27 optional actions in the action space. At state t and state t+1, the robot coordinate space has the following corresponding relationship:

[0031]

[0032] (x t+1 ,y t+1 ,z t+1 ),(x t ,y t ,z t ) represent the spatial coordinates of the end effector of the robot at state t+1 and time t respectively, coordinatestep represents the discretized spatial step, dx,dy,dz∈{-1,0,1}.

[0033] 3) Establish state model: Use a camera to shoot the robot arm environment as state information. Since the captured environment images are all 8-bit true color images, the resolution of 320x240 is relatively high for the deep Q value network. If the image with a resolution of 320x240 is directly used as the input of the deep neural network for training and learning, it will become very time-consuming, increase the complexity of the algorithm, and cause the network training to be difficult to converge. Therefore, the collected state image will be preprocessed. For the robot arm strategy control system, color information is redundant information. In the first step of preprocessing, the 8-bit true color image is converted into a grayscale image. Because the human eye has different perception of REB color, different weights need to be given during conversion. After grayscale processing, the image is unified into 84x84 size, and the preprocessed 84x84 grayscale image is used as the state S in the Markov process.

[0034] 4) Reward function design: The reward function is a key issue in policy control. The optimal control strategy must be considered and have environmental universality. When designing the reward function, the environmental robustness of the reward function must be considered. When the environment in which the robot is located changes significantly, the designed reward function is still applicable. Therefore, three factors that affect the reward function are considered: time factor, distance factor and environmental robustness. Considering the time factor allows the robot to grasp the target as quickly as possible, and considering the distance factor is determined by the task of grasping the target.

[0035] 5) Data enhancement method: adopt autonomous collection mechanism and action transposition mechanism. First, collect a small data set through VR / AR devices, use the small-scale data set to initialize the reinforcement learning algorithm experience pool, pre-train the reinforcement learning algorithm, and then use the pre-trained model to control the robot arm strategically, record the control process, and update the experience pool; then use the updated experience pool to train the reinforcement learning algorithm, and repeat this cycle. Since the action space distribution of manually collected data is uneven, if it is not processed, it is easy to cause network overfitting or underfitting. The action transposition mechanism is adopted. Its core idea is to use the characteristics of the previous state and the next state to exchange states. The previous state and the next state are exchanged with each other. Only the action and loss function need to be changed. Through experiments, it is found that without action transposition, no matter where the object is, the control strategy of the algorithm is always concentrated in the place where the action is densely distributed; after action transposition, the robot arm effectively locates the approximate position of the object and approaches the target object.

[0036] 6) Guided DQN strategy design: After being processed by a shared network structure, the algorithm generates k head nodes online and in parallel, representing the estimated values ​​of k Q-value functions, Q1, Q2, ..., Qk, respectively, and temporarily expands the uncertainty estimate of the quantized value function generated by the respective TD errors. The higher the uncertainty of the value function corresponding to an action, the higher the reward value brought by exploring the action. Through this distributed deep exploration method, the agent's exploration of different strategies is fully guaranteed, and diversified samples are generated, so that the dynamic information of the environment is better generalized to the state space of the position. Let each head node have a strategy, so that it can adapt to the changing dynamic environment.

[0037] 7) Design of the robot control program structure: The main purpose is to establish an effective control system to coordinate the different modules of the program and all subsystems of the robot layer. The robot control system consists of two parts: the main program and the auxiliary program. The main program includes a deep reinforcement learning algorithm module, a communication module, and a sampling data processing module. The auxiliary program is a bridge between the main program and the robot layer and runs independently of the main program in the experiment.

[0038] Figure 1 The following is a block diagram of a robot control system based on deep reinforcement learning. If the robot wants to complete the task of grasping a target based on deep reinforcement learning, it first obtains environmental image information through the camera, and after visual processing, it serves as sample information for deep learning. After deep reinforcement learning neural network training, it sends action command information to the robot controller to complete the grasping task of the robot.

[0039] Figure 2Block diagram of the communication mode of the robot control platform. When the deep reinforcement learning algorithm controls the robot, the control command issued by the host computer is first transmitted to the robot control box via Ethernet. Then the control box uses the internal motor control algorithm to move each joint of the robot to the corresponding angle. At the same time, the joint information of the robot is published to the robot control box at a frequency of 150Hz and received by the host computer via Ethernet. In addition, every time the robot is started, the robot load needs to be initialized through the control panel. In addition to the necessary initialization operations, some auxiliary work needs to be performed through the control panel, such as emergency stop and reset of the robot and simple movement of the robot. Finally, when training and testing the convolutional neural network strategy, it is also necessary to obtain image information at each moment through the camera. The RealSense camera produced by Intel is used to communicate directly with the host computer through the USB interface.

[0040] The guided DQN algorithm uses multiple shunt networks to randomize the value function and temporarily expand the scope of exploration of the state space. Figure 3 The structure of the guided DQN model is briefly described. A structure with k head nodes is added at the end of the network. Each head node predicts the Q value of each action in the environment. Some data is selected through sampling for the head node to train separately. Each head node is trained using a shared deep convolutional network. In addition, flag values ​​are set to record the training data corresponding to these head nodes.

[0041] Guided DQN uses a self-sampling algorithm to approximate the distribution of Q values. The randomly initialized self-estimates produce reasonable uncertainty estimates for the neural network at a low computational cost, allowing the trained strategy to fully explore the environment at each step, enhancing the algorithm's robustness to the environment, and Guided DQN can use these uncertainty estimates for efficient exploration.

[0042] When testing guided DQN deep learning, an effective control system is established to coordinate the different modules of the program and all subsystems at the robot level. Figure 4The figure is a block diagram of the robot control program. In the main program, the deep reinforcement learning algorithm module is the core part of the main program, which includes the entire process of the guided DQN deep learning algorithm and all the training settings of the convolutional neural network; the main function of the communication module is to communicate with the auxiliary program. When resetting the robot, sampling the robot, and testing the controller and convolutional neural network strategy, the deep reinforcement learning algorithm sends the joint control amount at each moment to the auxiliary program, and obtains the state data of the robot at the next control moment and the camera image data; after obtaining the above data, the sampling data processing module first processes and classifies all the data, then stores the important types of data, and finally encapsulates all the sampling data in a standard format to ensure the uniformity of the data format during the execution of the algorithm. In addition, a log file will be generated each time the main program runs to record the setting method of important parameters in the algorithm.

[0043] In the robot control system, the auxiliary program is a bridge between the main program and the robot layer and runs independently of the main program in the experiment. First, after receiving the joint control quantity sent by the main program, the auxiliary program makes the robot move smoothly to the corresponding joint position by performing cosine interpolation in the joint space; secondly, since the RGB image data released by the camera in the standard ROS message format cannot be directly used for the training of convolutional neural networks, the original data of the camera is converted in the format in the auxiliary program; finally, since the robot state information required by the deep reinforcement learning algorithm cannot be completely obtained through the actual robot, the actual joint information of the robot needs to be used in the auxiliary program to solve other state information. By establishing the above robot control system structure, different modules of the program and the subsystems of the robot layer can be well coordinated. At the same time, it is also convenient to test the deep reinforcement learning algorithm and effectively record the experimental data in the experiment.

[0044] The present invention is not limited to the above-mentioned implementation modes, and anyone should be aware that any structural changes made under the enlightenment of the present invention, and any technical solutions having the same or similar technical solutions as the present invention, shall fall within the protection scope of the present invention.

[0045] The techniques, shapes, and structural parts not described in detail in the present invention are all well-known techniques.

Claims

1. A robotic arm control method based on guided DQN control, It is characterized in that The following steps are involved: 1) Establish a perception environment model: The camera is used to obtain the state of the environment in which the robot is located in real time, and all information used for decision-making is obtained from the environment. The action selected by the decision is used as a control signal to act on the environment. At the same time, the system perception environment model can monitor the dynamic changes of the environment in real time; 2) Robotic arm motion space design: Choose a method to control the robot arm using feedback, that is, specify the spatial position of the robot arm, and then feed it back to the robot arm system, which will independently calculate the relative movement angle of each joint point; 3) Establishing the state model: Using a camera to shoot the robot arm environment as the state information, the robot arm system preprocesses the collected state image. For the robot arm strategy control system, color information is redundant information. In the first step of preprocessing, the 8-bit true color image is converted into a grayscale image, and different weights are given during the conversion; 4) Reward function design: The reward function is a key issue in policy control. The optimal control strategy must be considered, which has environmental universality. Three factors that affect the reward function should be considered: time factor, distance factor, and environmental robustness. Considering the time factor allows the robot arm to grasp the target as quickly as possible, and considering the distance factor is determined by the task of grasping the target. 5) Data enhancement method: Adopting autonomous collection mechanism and action transposition mechanism, first collect a small data set through VR / AR equipment, use the small data set to initialize the reinforcement learning algorithm experience pool, pre-train the reinforcement learning algorithm, and then use the pre-trained model to strategically control the robot arm. At the same time, record the control process, update the experience pool, and then use the updated experience pool to train the reinforcement learning algorithm, and repeat this cycle; adopting the action transposition mechanism, the previous state and the next state are interchanged, and only the action and loss function need to be changed; 6) Guided DQN strategy design: After being processed by a shared network structure, the algorithm generates k head nodes online and in parallel, representing the estimated values ​​of k Q-value functions Q1, Q2, ..., Qk, respectively, and temporarily expands the uncertainty estimate of the quantized value function generated by the respective TD errors. The higher the uncertainty of the value function corresponding to an action, the higher the reward value brought by exploring the action, which can adapt to the changing dynamic environment; 7) Design of the robot arm control program structure: The main purpose is to establish an effective control system to coordinate the different modules of the program and all subsystems of the robot layer. The robot arm control system consists of two parts: the main program and the auxiliary program.

2. The robotic arm control method based on guided DQN control according to claim 1, It is characterized in that In step 2, the motion space design of the robot arm discretizes the motion space of the robot arm, does not continuously control the robot arm, and defines control commands with different motion steps, so that the robot arm can approach the target object with a certain step length.

3. The robotic arm control method based on guided DQN control according to claim 1, It is characterized in that In step 3, the robot arm environment is photographed by a camera as an 8-bit true color image, and its resolution of 320 x 240 is relatively high for the deep Q value network.

4. The robotic arm control method based on guided DQN control according to claim 1, It is characterized in that The true color image in step 3 is converted into a grayscale image, and after the grayscale processing, the image is unified into a size of 84 x 84, and the pre-processed 84 x 84 grayscale image is used as the state S in the Markov process.

5. The robotic arm control method based on guided DQN control according to claim 1, It is characterized in that The environmental robustness in step 4 means that when the environment in which the robot is located undergoes significant changes, the designed reward function is still applicable.

6. The robotic arm control method based on guided DQN control according to claim 1, It is characterized in that The main program in step 7 includes a deep reinforcement learning algorithm module, a communication module, and a sampling data processing module. The auxiliary program serves as a bridge between the main program and the robot layer and runs independently of the main program in the experiment.

Images