Robot moving target grabbing and tracking method and system based on enhanced deep learning

Abstract

The invention provides a robot moving target grabbing and tracking method and system based on enhanced deep learning, and relates to the technical field of artificial intelligence, and the method comprises the steps: constructing a robot simulation environment, and carrying out the initialization; constructing a control decision network model based on deep reinforcement learning; obtaining robot state observation values, and constructing a state observation value sequence; according to the state observation value sequence, through a control decision network model of deep reinforcement learning, corresponding actions of the robot are obtained; after corresponding actions of the robot are executed, a new state observation value sequence is obtained; on the basis of an improved reward function rule, calculating a reward value, accumulating the reward value and judging whether the target is successfully grabbed or not; the next robot action is repeatedly trained until the grabbing requirement is met, and information in the simulation environment is sent to a robot controller in real time; therefore, the robot is controlled to move coordinately. According to the invention, the real robot can grab and track the moving target.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a robot moving target grasping and tracking method and system based on enhanced deep learning. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] On the automated production line, the robot system needs to perform extensive calculations to track the trajectory and grasp objects. At present, some basic grasping, fixing, and handling tasks of the robot still require the intervention of programmers. The intelligence level of the robot cannot meet the actual production needs. Even some highly intelligent robot manufacturers still need manual teaching by technical workers before completing tasks on the production line; or precise offline programming by control engineers in advance. The low level of robot intelligence is hindering the development of industries such as automation.

[0004] The current offline programming of robots still lacks generalizability. As long as the relative position and posture of the robot and the grasped object change, or even the working environment changes, it needs to be re-taught or re-programmed. At the same time, it also brings heavy labor to the operating engineers and requires them to have high professional knowledge. It has a great impact on productivity and wastes manpower, resulting in increased production costs.

[0005] At present, although domestic humanoid robots have a certain industrial foundation, they still have shortcomings in key basic components, dedicated operating systems, large AI models, general complete machine products and industrial ecology. Humanoid robot dual-arm grasping and tracking is an important direction of artificial intelligence technology. It can autonomously make the best decision based on information such as the environment, tasks and goals to achieve intelligent grasping or tracking of dynamic targets, autonomous navigation, task execution, human-computer interaction and other functions.

[0006] Reinforcement learning is also widely used in the field of robotics, such as training robots to complete specific tasks, such as autonomous operation in factories. If robots want to be truly intelligent, they must achieve fast, accurate, stable, and adaptive grasping and operation functions. The intelligent operation of robots can adopt the path of imitation learning and reinforcement learning.

[0007] At present, the intelligent operation of robots has made significant progress in the direction of deep reinforcement learning. The theory of deep reinforcement learning can be applied to the intelligent grasping application of robots. The robot makes continuous action decisions and realizes grasping by observing the current real-time working environment scene offline, and can complete continuous and intelligent decision-making tasks.

[0008] The training process of reinforcement learning requires continuous trial and error in the environment. The robot needs to continuously grasp objects in the grasping environment and hope to receive rewards from the environment for each grasping attempt, thereby optimizing the intelligent agent and making it more and more intelligent.

[0009] However, there are also the following problems with robot intelligent operation based on reinforcement learning: In terms of the interaction between reinforcement learning and real environments, using real robots to interact in real environments leads to problems such as high resource consumption, high costs, damage to robot hardware, difficulty in real-time movement of grasping targets, and impact on user experience.

[0010] In terms of reward function construction, a good reward function is often more important than the choice of algorithm. However, real-world problems are complex and reward functions are difficult to design. Current reward functions generally use sparse rewards, that is, rewards are obtained when the task is completed, and no rewards are obtained when the task is not completed. However, learning from such a reward function takes a lot of time and may not even be successful. It usually requires additional heuristic exploration mechanisms to help the agent discover this sparse positive reward. Therefore, it is impossible to learn quickly or converge quickly.

[0011] In terms of state environment extraction, the current state sequence generally uses the robot's joint angles and grasping result marks. The extracted content is not detailed enough, and it is impossible to make corresponding adjustments based on environmental changes and feedback, and it is also impossible to improve the learning speed and convergence efficiency. Summary of the invention

[0012] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a robot mobile target grasping and tracking method and system based on reinforced deep learning. It adopts a control decision network model based on deep reinforcement learning, uses self-learning decision-making ability, analyzes the real-time status during the grasping and tracking process, automatically generates and executes the optimal action for controlling the robot, enables serial port communication and data analysis, and can realize the grasping and tracking of mobile targets by a real robot.

[0013] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: A first aspect of the present invention provides a robot moving target grasping and tracking method based on enhanced deep learning.

[0014] A robot moving target grasping and tracking method based on reinforcement deep learning, comprising: Step 1: Build a robot simulation environment and initialize it; Step 2: Use the deep deterministic policy gradient algorithm to build a control decision network model based on deep reinforcement learning; Step 3, obtain the robot state observation value and construct a state observation value sequence; wherein the state observation value sequence is a vector sequence composed of the coordinates of each link arm end point, the vector coordinates of each link arm end point to the moving target point B, and the grasping result flag; Step 4: According to the state observation value sequence, the corresponding action of the robot is obtained through the control decision network model of deep reinforcement learning; Step 5: After executing the corresponding action of the robot, a new state observation value sequence is obtained, and based on the improved reward function rule, its reward value is calculated, the accumulated reward value is added, and whether the target is successfully grasped is determined; Step 6: Repeat the training of the next robot action until the grasping requirements are met and the state observation value in the simulation environment is obtained; Step 7: After the grasping requirements are met, the state observation value joint angle information in the simulation environment is sent to the robot controller in real time; the robot controller controls the coordinated movement of the robot.

[0015] A second aspect of the present invention provides a robot moving target grasping and tracking system based on enhanced deep learning.

[0016] A robot moving target grasping and tracking system based on reinforcement deep learning, comprising: The robot and mobile target programming module is configured to construct a robot simulation environment and initialize it; The deep deterministic policy gradient algorithm DDPG programming module is configured to use the deep deterministic policy gradient algorithm to build a control decision network model based on deep reinforcement learning; The main program programming module of the enhanced deep learning is configured to obtain the robot state observation value and construct a state observation value sequence; wherein the state observation value sequence is a vector sequence composed of the coordinates of each link arm end point, the vector coordinates of each link arm end point to the moving target point B, and the grasping result flag; According to the state observation value sequence, the corresponding action of the robot is obtained through the control decision network model of deep reinforcement learning; After executing the corresponding action of the robot, a new state observation value sequence is obtained; and based on the improved reward function rule, its reward value is calculated, the cumulative reward value is added, and whether the target is successfully grasped is determined; Repeat the training of the next robot action until the grasping requirements are met and the state observation value in the simulation environment is obtained; The real control robot scene module is configured to send the state observation value joint angle information in the simulation environment to the robot controller in real time after meeting the grasping requirements; the robot controller controls the coordinated movement of the robot.

[0017] The third aspect of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in a method as described in the first aspect of the present invention are implemented.

[0018] A fourth aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in a method as described in the first aspect of the present invention.

[0019] A fifth aspect of the present invention provides a computer program product comprising instructions, which, when running on a computer, enables the computer program to implement the steps in a method as described in the second aspect of the present invention when executed by a processor.

[0020] One or more of the above technical solutions have the following beneficial effects: The present invention improves the state observation value sequence. On the basis of the robot joint angle, it adds the coordinates of each link arm endpoint, the vector coordinates of each link arm endpoint to the moving target point, and a set of vector sequences of successful grasping marks. The reinforcement learning platform can make corresponding adjustments according to environmental changes and feedback, making deep neural network learning faster and easier to converge, and the learning speed is significantly improved.

[0021] The present invention improves the reward function strategy in the reinforcement learning algorithm and designs a reward function based on the distance between the robot's fingertips and the end points of each connecting rod and the moving target. As long as the fingertips and the end points of the connecting rod are close to the moving target, the reward is large, and the farther away, the smaller the reward is. In addition, the importance of the distance from each end point of the connecting rod to the target is lower than the distance from the fingertips to the moving target. The reward signal is designed to be continuous, which can improve the convergence to a higher level and will be much easier to learn than the discrete one. When the moving target is successfully grasped, the reward signal is designed to be 1. It can solve the sparse reward problem, adaptively adjust the reward function according to the distance, do not require expert experience to conduct a comprehensive analysis and judgment of the environment, can consider long-term rewards to make current decisions, and has the ability to plan for the long term.

[0022] The present invention uses deep reinforcement learning methods to explore in unknown environments, help users discover new and effective strategies, avoid actual robot and personnel damage, and save a lot of online operation resources.

[0023] The present invention can be used for real robot control, establish a robot system that is the same as the real environment, let the robot train in the simulation environment, load the saved network model file on the reinforcement learning platform, and start the computer serial port communication, and send the joint angle information of the state observation value through inverse kinematics analysis to the robot controller in real time. The robot will automatically grasp and track the moving target according to the trained model. The time overhead of the robot movement process is saved until the algorithm performs well, the model learning cost is reduced, and resources are saved; a large number of grasping attempts, interacting in the simulation environment, reduce the wear of the robot, solve the problem of judging the grasping results, and provide a grasping and tracking model and experience for real environment robot operation.

[0024] The present invention can interact with a real robot in real time on a virtual robot platform. During the interaction, the strategy is continuously updated according to the user's behavior feedback, so that the mobile grasping converges to the optimal strategy of the user's dynamic preference. Visual model training will help reduce offline exploration costs and provide a good training test and grasping tracking model for actual robot grasping operations.

[0025] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0027] Figure 1 A schematic diagram of a program of a robot grasping and tracking a moving target method based on enhanced deep learning in Embodiment 1 of the present invention; Figure 2 Schematic diagram of the robot and mobile target module in the first embodiment of the present invention; Figure 3 A schematic diagram of interaction and robot control in a method for robot grasping and tracking a moving target based on enhanced deep learning in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the composition of a robot grasping and tracking mobile target system based on enhanced deep learning in Example 2 of the present invention. DETAILED DESCRIPTION

[0028] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0029] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0030] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0031] The overall idea proposed by the present invention is: Taking a real three-degree-of-freedom robot and a mobile target as an example, the visual simulation and the actual system use the same robot kinematic model. The reinforcement deep learning reward function strategy is improved, and a reward function based on the distance from the robot fingertips and the end points of each connecting rod to the mobile target is designed. The discrete reward signal is designed to be continuous, which significantly improves the convergence and rapidity. In terms of state observation values, the robot joint angle is removed, and the moving target point vector, the coordinate vector of each arm end point, and the vector coordinates to each moving target point are added, and the grasping result is a vector sequence of observation values. The reinforcement deep learning platform can be used for real robot control. By continuously updating the strategy according to the reinforcement learning environment and method feedback during the interaction process, the intelligent agent converges to the optimal strategy of the user's dynamic preference. Based on the trained neural network model, multiple real robots can be operated to achieve the digital twin effect, thereby saving a lot of online resource consumption.

[0032] Embodiment 1 This embodiment discloses a robot moving target grasping and tracking method based on enhanced deep learning, including: Step 1: Build a robot simulation environment and initialize it; Step 2: Use the deep deterministic policy gradient algorithm to build a control decision network model based on deep reinforcement learning: Step 3, obtain the robot state observation value and construct a state observation value sequence; wherein the state observation value sequence is a vector sequence composed of the coordinates of each link arm end point, the vector coordinates of each link arm end point to the moving target point B, and the grasping result flag; Step 4: According to the state observation value sequence, the corresponding action of the robot is obtained through the control decision network model of deep reinforcement learning; Step 5: After executing the corresponding action of the robot, a new state observation value sequence is obtained; and based on the improved reward function rule, its reward value is calculated, the accumulated reward value is added, and it is determined whether the target is successfully grasped; Step 6: Repeat the training of the next robot action until the grasping requirements are met and the state observation value in the simulation environment is obtained; Step 7: After the grasping requirements are met, the state observation value joint angle information in the simulation environment is sent to the robot controller in real time; the robot controller controls the coordinated movement of the robot.

[0033] In order to explain this embodiment more clearly, Figure 1 As shown, a robot moving target grasping and tracking method based on reinforcement deep learning is carried out through the established robot reinforcement learning environment platform, as follows: The robot reinforcement learning environment platform mainly includes the robot reinforcement deep learning main program programming module III, the robot and mobile target programming module I, the deep deterministic policy gradient algorithm DDPG programming module II, the experience replay data storage device IV, the random sampler VI, the DDPG network model storage module V, etc. All modules are called by the reinforcement deep learning main program programming module III.

[0034] The robot and mobile target programming module I is mainly used to establish the planar visualization model of the three-degree-of-freedom planar robot and mobile target, build the robot simulation environment, and initialize it.

[0035] The main function of the deep deterministic policy gradient algorithm DDPG programming module II is to use the deep deterministic policy gradient algorithm to build a control decision network model based on deep reinforcement learning: every time a robot state observation value is given, an action will be output to the robot and mobile target module. The action includes the three joint angles of the robot. Through the robot kinematics, it can reach the specified joint angle position to achieve grasping. At the same time, the robot grasping tracking target state observation value, rewards and penalties, and whether the grasping is successful are given ( , , ) and other data.

[0036] The deep deterministic policy gradient algorithm DDPG programming module II uses the neural network algorithm DDPG received in step 2 ( , , ) observation data, and train the next optimal robot action through the deep neural network DDPG After executing the action, the robot enters the reinforcement deep learning main program programming module III and switches to the next state This cycle repeats itself, and the robot reinforcement learning environment platform gives rewards based on the distance of the grasping moving target and whether the grasping is successful or not. It constantly learns and adapts to the environment through trial and error until the user's grasping success rate requirements are met.

[0037] Strengthen the deep learning main programming module III and store the set capacity at the same time The data are stored in the experience playback data storage device IV. When the storage capacity is exceeded, the earliest data is automatically overwritten.

[0038] Random Sampler VI randomly samples from the data storage module The data is used as a mini-batch training data for the neural network.

[0039] Based on the above robot reinforcement learning environment platform, a robot moving target grasping and tracking method with reinforcement deep learning can be specifically described as follows: Step 1: Build a robot simulation environment and initialize it.

[0040] OpenGL, VTK, etc. are used to build a three-degree-of-freedom planar robot and a planar visualization model of a mobile target. The mobile target can be dragged with the mouse to any point in the robot's reachable workspace.

[0041] Initialize the robot simulation environment, specifically: randomize the joint angles of the three-degree-of-freedom robot, but in accordance with the robot's kinematic model; initially randomize the position of the moving target, but it should be within the robot's reachable workspace, that is, the robot can grasp and track the target at that position. The main purpose is to be able to traverse all possible spatial positions and find all feasible grasping solutions.

[0042] Step 2: Use the deep deterministic policy gradient algorithm to build a control decision network model based on deep reinforcement learning.

[0043] Reinforcement learning is a general term for a class of algorithms. The present invention adopts the DDPG (Deep Deterministic Policy Gradient) algorithm, which is a deterministic policy gradient algorithm proposed by Lillicrap. The algorithm can run on a continuous action space and can robustly solve simulated physical tasks. The DDPG algorithm is a reinforcement learning algorithm based on deep learning. DDPG combines the ideas of deterministic strategy and experience replay on the basis of the Actor-Critic (AC) structure to solve the problem of continuous action space. It is an off-line deep learning algorithm. It is an Actor-Critic algorithm, in which the Actor network is used to learn the strategy and the Critic network is used to learn the value function. The main feature of the DDPG algorithm is that it can handle the problem of continuous action space and can learn in high-dimensional space. The core idea of ​​the DDPG algorithm is to use an Actor network to output the action and a Critic network to evaluate the value of the action. Both the Actor network and the Actor network are based on deep neural networks and can be trained using the back-propagation algorithm. The advantage of the DDPG algorithm is that it can handle the problems of high-dimensional state space and continuous action space, and can achieve good results in practical applications.

[0044] Step 3, obtain the robot state observation value and construct a state observation value sequence; wherein the state observation value sequence is a vector sequence composed of the coordinates of each link arm endpoint, the vector coordinates of each link arm endpoint to the moving target point B, and the grasping result flag.

[0045] In this embodiment, the specific contents are as follows: Through the DDPG algorithm, all possible states are traversed to obtain the state observation value s.

[0046] Each time the state observation value s sequence is obtained, it is a row of data, and a total of 15 data are arranged in sequence. These coordinates are arranged in a row as a training data for the DDPG algorithm.

[0047] The data sequence is as follows: ; In the formula, B coordinate, G coordinate, F coordinate, and E coordinate represent the coordinates of the target point, Dist1 represents the vector distance from the base link to the target, Dist2 represents the vector distance from the middle link to the target, and Dist3 represents the vector distance from the end to the target. For details, please refer to the parameters given in Table 1.

[0048] In order to accelerate the convergence of the deep neural network learning DDPG algorithm, the above observation sequence can be normalized and processed as follows: ; in, and Indicates the maximum value of the x-axis and y-axis working space, and The component representing the position of the target point, represents the coordinate component of the end of the base link, represents the coordinate component of the end of the intermediate link, Represents the coordinate component of the end fingertip. The coordinates of other points divided by this value will be less than 1, in order to accelerate the convergence and stability of the neural network.

[0049] In the state feature extraction learning, the robot's joint angles and the grab result flags are generally used as the state sequence. However, based on the existing situation, the present invention removes the robot's joint angles and adds the coordinates of each connecting rod arm endpoint, the vector coordinates of each connecting rod arm endpoint to the moving target point B, and the vector sequence composed of the grab result flag. Therefore, the reinforcement learning platform can make corresponding adjustments according to environmental changes and feedback, making the deep neural network learning faster, easier to converge, and the learning speed is significantly improved.

[0050] The grasping result flag is mainly determined by whether the robot's fingertips are in the area of ​​the moving target. If they are in the moving target for a long time, the current cycle will be exited, for example, when the reward statistics reach the specified total score requirement.

[0051] The main parameters and descriptions of the robot are shown in Table 1.

[0052] Table 1 Main parameters and descriptions of robots in the enhanced deep learning platform

[0053] Step 4: According to the state observation value sequence, the corresponding action of the robot is obtained through the control decision network model of deep reinforcement learning.

[0054] In this embodiment, according to the current state observation value sequence s, the DDPG algorithm in step 2 can automatically obtain the corresponding action a of the robot, that is, the joint motion angle ( i 1, i 2, i 3).

[0055] The calculation method of joint motion angle is as follows: like Figure 2 As shown in Table 1, the position of each link of the robot can be calculated, that is, the forward kinematics model can be derived according to the length of the robot link and the joint angle: The coordinates of the base connecting rod end point E(x1,y1) are: (1) The coordinates of the point F(x2,y2) at the end of the intermediate link are: (2) The coordinates of the end fingertip point G(x3,y3) are: (3) The inverse kinematics of the robot can be obtained by inversely calculating the position coordinates of equation (1-3) ( i 1, i 2, i 3).

[0056] Step 5: After the robot performs the corresponding action, it obtains a new state observation value sequence; and based on the improved reward function rule, it calculates its reward value, accumulates the reward value, and determines whether the target is successfully grasped.

[0057] In this embodiment, after the robot performs the corresponding action, it moves from the previous angle to this angle, and obtains a new state observation value sequence s_, a reward r and a cumulative reward sum_r, as well as whether the grasping target is successfully completed.

[0058] In this embodiment, the reward r design rule designed by the robot reinforcement deep learning platform is: ; In the formula, R is a positive value, ranging from The purpose is to ensure that each use of r i Less than 1, and The component representing the position of the target point, represents the coordinate component of the end of the base link, represents the coordinate component of the end of the intermediate link, Represents the end fingertip coordinate component.

[0059] That is, the reward function is based on the distance from the robot's fingertips and the end points of each link to the moving target. As long as the fingertips and the end points of the link are close to the moving target, the reward is large, and the farther away, the smaller the reward is. In addition, the importance of the distance from each end point of the link to the target is lower than the distance from the fingertips to the moving target.

[0060] If the tracking and grasping fails, done=0, and the reward is selected according to the distance. In each step of the attempt, a small reward is used as a mechanism to lure the robot to reach the target. The closer the robot is, the larger the reward is, and the farther the robot is, the smaller the reward is. The reward rule is designed as a continuous reward signal, which can significantly improve the convergence of deep neural learning and is much easier to learn than the discrete reward signal. γ<1 means that the distance from the end point of each link to the target is less important than the distance from the fingertip to the moving target. If the tracking and grasping is successful, done=1, then the reward r=1. Every time the robot touches the target during tracking and grasping, it will receive a large reward.

[0061] By improving the reward function, the sparse reward problem can be solved. The reward function can be adaptively adjusted according to the distance. It does not require expert experience to conduct a comprehensive analysis and judgment of the environment. It can consider long-term rewards to make current decisions and has the ability to make long-term plans.

[0062] Step 6: Repeat the training of the next robot action until the grasping requirements are met and the state observation value in the simulation environment is obtained.

[0063] In this embodiment, the next robot action is repeatedly trained based on the process of step 5. At the same time, after collecting information (s, a, r, s_) obtained by interacting with the environment, they are put into a memory bank, and then a memory is randomly selected for learning. If the minimum batch data is met, the algorithm will start deep learning training, update the neural network parameters, and output the state observation value s_ to be executed in the next round. At the same time, the new state observation value sequence s_ replaces the previous state observation value s.

[0064] If the moving target is captured or the maximum number of steps is reached, the loop will automatically exit.

[0065] When all rounds of cycles are completed, the platform automatically saves the network model file and ends the neural network training. The model can also be reloaded on the reinforcement learning platform. When the robot drags the moving target with the mouse in the robot workspace, the robot can grab and track the target in real time, and can display the grab success rate and visualize the dynamic grabbing and tracking process of the target; if the success rate is too low, you can continue training on the existing basis until the user's needs are met.

[0066] Step 7: After the grasping requirements are met, the state observation value joint angle information in the simulation environment is sent to the robot controller in real time; the robot controller controls the coordinated movement of the robot.

[0067] In this embodiment, after the grasping requirements are met, the state observation value joint angle information in the simulation environment is sent to the robot controller in real time; the robot controller solves the pulse control instructions in real time according to the robot kinematic model and sends them to the motor driver, thereby controlling the coordinated movement of the robot.

[0068] like Figure 3 As shown, in this embodiment, it also includes a real robot device. The enhanced deep learning platform interactive interface is a visual human-computer interactive interface. Taking a three-degree-of-freedom robot as an example, a real visual model is established (for example, using OpenGL, VTK, etc.). The robot simulation model used is completely consistent with the real robot scene VIII model. The simulation and actual system use the same robot motion model. The simulation system can output the joint angles of the three-degree-of-freedom robot through multiple serial port communications and data analysis of the computer. These joint angle data are received by the robot controller (for example Figure 3 ⑥ and ⑦ in ), multiple robots can be controlled to grasp and track moving targets, such as robots Ⅰ-A, Ⅰ-B, etc.

[0069] The visual interactive simulation interface Ⅶ is equipped with a three-degree-of-freedom robot Ⅰ. The lower part of the human-computer interface is equipped with function buttons such as DDPG training, simulation test, save model, load model, move target, success rate, switch serial port, real test, stop and exit, and realizes the corresponding functions.

[0070] In this embodiment, establishing a robot reinforcement learning environment platform includes two stages: a strength deepening learning training stage and an evaluation and testing stage.

[0071] like Figure 4As shown, in the training phase: to establish a robot reinforcement learning platform IV, computers and communication equipment (mainly including computers, storage media, display screens, mouse and keyboard, serial communication equipment, etc.) and computer software and systems (mainly including operating systems, programming languages, human-computer interaction interfaces, deep neural network libraries, etc.) are required.

[0072] The three-degree-of-freedom robot I interacts with the reinforcement learning environment. The three-degree-of-freedom robot uses the deep deterministic policy gradient algorithm DDPG. One output of the robot reinforcement learning platform is ④, which is stored in the experience revisit storage ⑤ as a data set for training the online network. Each time, a random sample of data (mini-batch) is taken from ⑤ as a minimum batch of mini-batch training data for the neural network. After training with the DDPG algorithm, the input of the algorithm is s t , the DDPG module finally outputs a robot action. The robot performs the action and obtains new. This cycle repeats until the robot can successfully grasp and track the moving target. At the end of this training phase, the trained neural network model file can be saved.

[0073] Robot grab target test platform: Based on the previous training, the reinforcement learning platform can load the trained DDPG evaluation network model and execute the second evaluation phase (before opening the serial port). If the effect of grabbing the moving target is satisfactory, the serial port communication and data analysis can be enabled to execute the second phase evaluation test phase and the third phase real robot control code. The robot reinforcement learning platform can send control instructions, that is, the specified joint angle data, through serial port communication. Multiple serial port communications and data analysis can be enabled to communicate with robot controllers ⑥ and ⑦ to control multiple robot controls Ⅰ-A, Ⅰ-B, etc.

[0074] The specific operation contents of the two phases are as follows: Phase 1: Intensive and in-depth learning and training phase (1) Run the main program of the reinforcement learning platform, which starts from i=0 and runs until MAX_EPISODES rounds. MAX_EPISODES should be a large number to facilitate the robot to grasp, track and try and error, and traverse all possible states, that is, the state variable value s. These rounds of learning process may be very long, and the length of learning time also depends on the complexity of the environment.

[0075] (2) Initialize the robot simulation environment. The joint angles of the three-degree-of-freedom robot are randomized, but they conform to the robot's kinematic model. The position of the moving target is initially randomized, but it should be within the robot's reachable workspace, that is, the robot can grasp and track the target at that position. The main purpose is to traverse all possible spatial positions and find all feasible grasping solutions.

[0076] (3) Each round starts from j=0 to the MAX_STEPS step or the step where the target is captured and tracked successfully (done=1), and the reward sum_r of each round is accumulated, with an initial value of 0.

[0077] (4) Nested loops, each round starts from j=0 and ends at the MAX_EP_STEPS step, or when the target grabbing action step is completed (done=1).

[0078] (5-13) The DDPG algorithm automatically obtains the robot's corresponding action a, that is, the joint motion angle ( i 1, i 2, i 3). Execute RobotEnv.step(a), the robot will move from the last angle to this angle, and obtain the new state observation sequence s_, reward r and cumulative reward sum_r, as well as whether the grasping target is completed. Collect information (s, a, r, s_) obtained by interacting with the environment, put them into a memory bank, and then randomly select a memory for learning. If the minimum batch data is met, the algorithm DDPG.learn() will start deep learning training, update the neural network parameters, and output the state observation s_ to be executed in the next round. At the same time, the new state observation sequence s_ replaces the previous state observation s. If the moving target is grasped or the set MAX_EP_STEPS number of steps is reached, the loop will automatically exit.

[0079] (14) When all MAX_EPISODES rounds are completed and the training reaches a satisfactory level, the algorithm will save the deep neural network model and its trained parameters, which can be directly called next time, saving training time.

[0080] Phase 2: Evaluation and testing phase: The reinforcement learning platform calls the trained network model for test evaluation (1) DDPG.restore() loads the deep network model (DDPG.save()) file, that is, calls the trained and automatically saved deep neural network model from the computer hard disk or other storage area.

[0081] (2) The robot and the grasping target will be displayed and rendered.

[0082] (3) Reset the robot and moving target.

[0083] (4) During the test evaluation phase, the total number of successful completions will be recorded, that is, the done flag of each time. If the crawling and tracking fails, done=0; if the crawling and tracking succeeds, done=1.

[0084] (5) When the condition is true, the loop mode is entered, starting from n=0, and the maximum number of loops is MAX_NUM.

[0085] (6) After entering the for loop, the robot model is first visualized, that is, the robot, moving target objects, and action visualization can be seen in the interactive interface display window.

[0086] (7) Using the DDPG algorithm, select the moving object with the mouse and drag it to generate an observation value s. The observation value s is used as the input of the network model to automatically obtain the optimal action a of the deep network model.

[0087] (8) After executing action a, the reinforcement learning platform can obtain data information, such as state observation value s, reward r, and whether the grasping tracking flag is completed. Then the program executes (5) in sequence.

[0088] (9) sum_done represents the sum of the number of successful crawling and tracking in all cycles. By comparing it with the maximum MAX_NUM, the total number of successful crawling and tracking can be calculated. If the user is satisfied with the success rate, they can proceed to the next stage.

[0089] (10) Open the serial port of the robot controller and set the baud rate to 115200.

[0090] (11) Wait for the keyboard input Ctrl+Alt+S to be pressed simultaneously, then execute the subsequent program, otherwise wait.

[0091] (12) The robot returns to the specified initial position.

[0092] (13-16) In the reset state s, the robot action a is obtained and executed to obtain the new state observation value s. Through the robot inverse kinematics, the joint angle is obtained i =( i 1, i 2, i 3) and then sent to the robot controller via the serial port.

[0093] (17) The serial port is closed at the end of the tracking and capturing phase.

[0094] The enhanced deep learning platform can use computer operating systems such as WINDOWS, and use Python, C++, etc. as programming languages. The present invention takes pseudocode programming as an example to illustrate the specific implementation description and introduction, which does not depend on programming languages ​​and computer systems, as follows: [1] Robot and mobile target environment class, for other programs to call. The robot class RobotEnv mainly includes the following functions: Class RobotEnv(object) def __init__():1) def step_motion(action):2) def reset():3) def render():4) Define the RobotEnv class. Define (def) the initialization function __init__(). It indicates the initialization of class members, some hyperparameters, 3D robot model, visualization, etc.

[0095] Define the robot motion function step_motion(action). Execute the robot agent motion action, the input is the action parameter (action), and the function returns the state variable observation value, reward and whether the grasping tracking flag is completed, that is, the robot action input is ( i 1, i 2, i 3) The output is the state variable observation value s, the reward r, and the completed grasp tracking flag done.

[0096] Define the robot reset function reset(). Each time, the angle of the robot body is randomized, but the kinematic relationship of the robot is maintained. The position of the moving target is randomized, but does not exceed the robot workspace, that is, the robot can grasp and track this position.

[0097] Define the render() function for the visualization robot. Whether to visualize and render the robot. If this function is called, the robot's dynamic grabbing process will be displayed; otherwise, the visualization will not be displayed. The benefit is to save computer resources and speed up the learning and training process.

[0098] [2] Pseudo code of the training program of the enhanced deep learning platform: for i in range(MAX_EPISODES):1) s = RobotEnv.reset()2) sum_r = 0.3) for j in range(MAX_STEPS):4) a = DDPG.choose_action(s)5) s_, r, done = RobotEnv.step(a)6) DDPG.store_transition(s, a, r, s_)7) sum_r += r8) if DDPG.memory_full:9) DDPG.learn()10) s=s_11) if done or j == MAX_EP_STEPS-1:12) break13) DDPG.save()14) Embodiment 2 A robot moving target grasping and tracking system based on reinforcement deep learning, comprising: The robot and mobile target programming module is configured to construct a robot simulation environment and initialize it; The deep deterministic policy gradient algorithm DDPG programming module is configured to use the deep deterministic policy gradient algorithm to build a control decision network model based on deep reinforcement learning: The main program programming module of the enhanced deep learning is configured to obtain the robot state observation value and construct a state observation value sequence; wherein the state observation value sequence is a vector sequence composed of the coordinates of each link arm end point, the vector coordinates of each link arm end point to the moving target point B, and the grasping result flag; According to the state observation value sequence, the corresponding action of the robot is obtained through the control decision network model of deep reinforcement learning; After executing the corresponding action of the robot, a new state observation value sequence is obtained; and based on the improved reward function rule, its reward value is calculated, the accumulated reward value is added, and it is judged whether the target is successfully grasped; Repeat the training of the next robot action until the grasping requirements are met and the state observation value in the simulation environment is obtained; The real control robot scene module is configured to send the state observation value joint angle information in the simulation environment to the robot controller in real time after meeting the grasping requirements; the robot controller controls the coordinated movement of the robot.

[0099] Embodiment 3 The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0100] Embodiment 4 The purpose of this embodiment is to provide a computer-readable storage medium.

[0101] A computer-readable storage medium stores a computer program, which executes the steps of the above method when executed by a processor.

[0102] Embodiment 5 The purpose of this embodiment is to provide a computer program product containing instructions, which, when running on a computer, enables the computer to execute the methods and functions involved in any of the above embodiments. The steps involved in the apparatus of the above embodiment correspond to the method embodiment 1, and the specific implementation method can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0103] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0104] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A robot moving target grasping and tracking method based on enhanced deep learning, characterized in that: include: Step 1: Build a robot simulation environment and initialize it; Step 2: Use the deep deterministic policy gradient algorithm to build a control decision network model based on deep reinforcement learning; Step 3, obtain the robot state observation value and construct a state observation value sequence; wherein the state observation value sequence is a vector sequence composed of the coordinates of each link arm end point, the vector coordinates of each link arm end point to the moving target point B, and the grasping result flag; Step 4: According to the state observation value sequence, the corresponding action of the robot is obtained through the control decision network model of deep reinforcement learning; Step 5: After executing the corresponding action of the robot, a new state observation value sequence is obtained, and based on the improved reward function rule, its reward value is calculated, the accumulated reward value is added, and whether the target is successfully grasped is determined; Step 6: Repeat the training of the next robot action until the grasping requirements are met and the state observation value in the simulation environment is obtained; Step 7: After the grasping requirements are met, the state observation value joint angle information in the simulation environment is sent to the robot controller in real time; the robot controller controls the coordinated movement of the robot.

2. A robot moving target grasping and tracking method based on enhanced deep learning as claimed in claim 1, characterized in that: The robot simulation environment is constructed and initialized, specifically: a planar visualization model of a three-degree-of-freedom planar robot and a mobile target is established, and the joint angle values ​​of the three-degree-of-freedom robot are randomized; the position of the mobile target is initially randomized and is within the reachable workspace of the robot, that is, the robot can grasp and track the target at this position.

3. A robot moving target grasping and tracking method based on enhanced deep learning as claimed in claim 1, characterized in that: Each state observation value sequence obtained is a row of data, which is arranged in sequence for a total of 15 data. The data order is as follows: Wherein, B coordinate, G coordinate, F coordinate, and E coordinate represent the coordinates of the target point, Dist1 represents the vector distance from the base link to the target, Dist2 represents the vector distance from the middle link to the target, and Dist3 represents the vector distance from the end to the target.

4. A robot moving target grasping and tracking method based on enhanced deep learning as claimed in claim 1, characterized in that: The state observation value sequence is subjected to normalized feature processing, specifically as follows: in, and is the maximum value of the working space of the x-axis and y-axis, and The component representing the position of the target point, represents the coordinate component of the end of the base link, represents the coordinate component of the end of the intermediate link, Represents the end fingertip coordinate component.

5. A robot moving target grasping and tracking method based on enhanced deep learning as claimed in claim 1, characterized in that: The improved reward function rule is specifically: a reward function based on the distance from the robot fingertips and the end points of each connecting rod to the moving target, when the fingertips and the end points of the connecting rod are close to the moving target, the greater the reward, and the farther away from the moving target, the smaller the reward, and the importance of the distance from each connecting rod end point to the target to obtain the reward is lower than the distance from the fingertips to the moving target; The specific formula is as follows: In the formula, R is a positive value, ranging from The purpose is to ensure that each use Less than 1, and The component representing the position of the target point, represents the coordinate component of the end of the base link, represents the coordinate component of the end of the intermediate link, Represents the end fingertip coordinate component.

6. A robot moving target grasping and tracking method based on enhanced deep learning as claimed in claim 1, characterized in that: Calculate the cumulative reward value, specifically, the reward for each round from the beginning to the maximum cycle value, or when the capture and tracking target is successfully completed.

7. A robot mobile target grasping and tracking system based on enhanced deep learning, characterized in that: include: The robot and mobile target programming module is configured to construct a robot simulation environment and initialize it; The deep deterministic policy gradient algorithm DDPG programming module is configured to use the deep deterministic policy gradient algorithm to build a control decision network model based on deep reinforcement learning: The main program programming module of the enhanced deep learning is configured to obtain the robot state observation value and construct a state observation value sequence; wherein the state observation value sequence is a vector sequence composed of the coordinates of each link arm end point, the vector coordinates of each link arm end point to the moving target point B, and the grasping result flag; According to the state observation value sequence, the corresponding action of the robot is obtained through the control decision network model of deep reinforcement learning; After executing the corresponding action of the robot, a new state observation value sequence is obtained; and based on the improved reward function rule, its reward value is calculated, the accumulated reward value is added, and it is judged whether the target is successfully grasped; Repeat the training of the next robot action until the grasping requirements are met and the state observation value in the simulation environment is obtained; The real control robot scene module is configured to send the state observation value joint angle information in the simulation environment to the robot controller in real time after meeting the grasping requirements; the robot controller controls the coordinated movement of the robot.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are performed.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images