Mechanical arm visual servo control method based on deep reinforcement learning

Abstract

The invention relates to the technical field of mechanical arm visual servo control, and particularly provides a mechanical arm visual servo control method based on deep reinforcement learning, which comprises the following specific steps: setting a target position and selecting feature points on a target, randomly selecting a starting point pose of a mechanical arm, and gradually moving to a target pose, the joint angle of the current pose is calculated according to the joint speed and pose of the last-step movement of the mechanical arm, and the joint speed of the next-step movement is selected by an Actor network according to the joint speed of the last-step movement, the joint angle of the current pose and the feature point error; for each step of movement, the evaluation function correspondingly gives an evaluation value to evaluate and improve the ability of the Actor network to select the joint speed, rewards are calculated according to the evaluation values, and when the average value of the rewards is progressively increased and the trend of each reward tends to be stable, the overall training is ended. The mechanical arm servo control method does not depend on training data, large-range servo tasks can be achieved, and servo control precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of robot arm visual servo control, and specifically provides a robot arm visual servo control method based on deep reinforcement learning. Background Art

[0002] Robotic arm visual servo control is a comprehensive technology that integrates machine vision, automatic control, and computer technologies. This technology realizes servo motion control of the robotic arm through visual signals, which can effectively improve the motion accuracy and robustness of the robotic arm and enhance the robot's ability to interact with complex environments.

[0003] According to different control methods, the robot visual servo control method can be divided into two types: model-based control method and deep learning-based control method. The model-based method is to first select image features according to the task, then calculate the interaction matrix of the image features, and finally design a suitable control rate according to the interaction matrix. This method has a very serious dependence on the image features and the corresponding interaction matrix of the image features, and due to the strong nonlinear relationship between the visual features and the camera speed, the convergence domain is small and global convergence cannot be achieved; the deep learning-based method is to convergently design a neural network, and then learn and train the deep neural network through the collected data samples (image features and camera speed), and finally test the trained deep learning algorithm through the new initial position of the robot. This method uses the already acquired training data to train the robot, which is heavily dependent on the training data, and the training data itself is difficult to obtain, and there are many difficulties in implementation. Therefore, how to solve the large-scale robot visual servo motion control and reduce the dependence on models and learning data is of great significance to the robot visual servo control. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a robot arm visual servo control method based on deep reinforcement learning, establishes an agent network based on the SAC algorithm and uses the Actor network to select and execute joint speeds, and evaluates the training process by establishing an evaluation function to improve the ability of the Actor network to select joint speeds. The method of the present invention does not require existing training data as support, can achieve large-scale convergence, and improve the accuracy of robot arm servo control.

[0005] The present invention provides a robot arm visual servo control method based on deep reinforcement learning, comprising: S1: Establish the camera model and the kinematic model of the robotic arm; S2: Establish an agent network through the SAC algorithm, which includes the Actor network; S3: Set the target position and select at least one point on the target as a feature point; S4: Train the agent network. The training process for each episode is as follows: Randomly select the starting position of the robot arm , starting point pose The corresponding joint angle is ; From the starting point At the beginning, the motion process of the robot arm is: Actor network based on joint velocity , pose Corresponding joint angle The joint speed corresponding to the next movement of the robot arm is selected by the feature point error , and control the robot arm according to the joint speed Movement arrival position , according to the joint angle and joint speed Calculate pose Corresponding joint angle , where the robot arm follows the joint speed The movement time is the preset time interval between two movement instructions. Indicates the number of movement steps of the robot arm, , represents the upper limit of the predetermined number of robot arm movement steps. The feature point error is the error between the feature point coordinates in the target image and the feature point coordinates in the target image of the robot arm at the target position. Indicates that the robot arm has a posture Exercise to position The joint velocity; Use the following formula to calculate the joint speed selected by the Actor network Evaluation value : ; ; ; ; ; in, represents the first gain coefficient, represents the second gain coefficient, represents the third gain coefficient, Indicates The error norm between the coordinates of the feature points in the target image and the coordinates of the feature points in the target image at the target pose of the robot arm, Indicates The energy consumed by walking, Indicates the score, It means that the camera model cannot collect all feature points. represents the threshold of the error two norm, Indicates except and Other circumstances other than when When , the servo movement is successful and the training of this scene is finished; When not all feature points appear in the target image captured by the camera model, the servo movement fails and the training of this scene ends; If the number of robot movement steps exceeds the preset upper limit , the servo movement fails, and the training of this scene ends; After each scene of training, the sum of the evaluation values ​​corresponding to each step of the joint velocity and posture in the scene is calculated as the reward for the scene; S5: No. After the training is completed, calculate the previous The average value of the episode rewards; when the average value increases with the number of training episodes and the fluctuation range of each episode reward is within the allowable error range, the agent network training ends.

[0006] Preferably, the DH parameter method is used to establish the kinematic model of the robot arm.

[0007] Preferably, the camera model is established using a perspective projection transformation method.

[0008] Preferably, the starting position of the robot arm The selection method is as follows: randomly select the robot arm posture, collect the target image at the posture through the camera model, and when all feature points appear in the target image, the selected posture can be used as the starting posture .

[0009] Preferably, based on the starting position , the starting point position is obtained by inverse solution of the kinematic model Corresponding joint angle .

[0010] Preferred, score The value is 5.

[0011] Preferably, the agent network also includes a Critic network, which senses the position and posture through the agent network The spatial state and posture The corresponding joint velocity is input into the Critic network, and the Critic network outputs the action value Q.

[0012] Preferably, posture The spatial state includes: feature point error, robot arm joint angle , No. The robot arm joint speed corresponding to the step.

[0013] Preferably, the activation function of the Actor network is the leaky relu function, and the activation function of the Critic network is the relu function.

[0014] Compared with the prior art, the present invention can achieve the following beneficial effects: The present invention establishes an intelligent agent network based on the SAC algorithm and trains the intelligent agent network. Compared with the control method based on deep learning, the servo control method of the present invention selects and executes joint speeds through the Actor network, automatically performs training, and establishes an evaluation function to evaluate the training process. The Actor network improves the ability to select joint speeds according to the evaluation results. Therefore, during the training process, it is not necessary to collect training data to train the intelligent agent network, and there is no dependence on the training data, making training easier.

[0015] Compared with the model-based control method, the servo control method of the present invention can randomly select targets and feature points. It does not need to select image features according to the task, nor does it need to design a suitable control rate based on the interaction matrix of the image features. Therefore, the convergence domain of the servo task can be greatly improved to achieve global convergence. In addition, the method of the present invention will not have local minimum phenomena, and there is no steady-state control error, which can improve the servo control accuracy.

[0016] The method of the present invention also designs a reward function to evaluate the joint speed selected by the Actor network. The reward function fully considers the error, energy, speed and success of each step of the robot arm action, ensuring that the joint speed of each step can be comprehensively and fully evaluated. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic diagram of an agent network training process provided according to an embodiment of the present invention; Figure 2 is a curve chart showing the variation of the average reward value with the number of training episodes provided according to an embodiment of the present invention; Figure 3 is a curve chart showing how the training reward for each scene varies with the number of scenes provided in an embodiment of the present invention; Figure 4 is a schematic diagram of target posture selected in a verification experiment provided according to an embodiment of the present invention; Figure 5 is a schematic diagram of a first starting point posture selected in a verification experiment provided in an embodiment of the present invention; Figure 6 is a schematic diagram of a second starting point posture selected in a verification experiment provided in an embodiment of the present invention; Figure 7is a schematic diagram of a third starting point posture selected in a verification experiment provided in an embodiment of the present invention; Figure 8 is a schematic diagram of a fourth starting point posture selected according to a verification experiment provided in an embodiment of the present invention; Fig. 9 is a schematic diagram of the fifth starting point posture selected in the verification experiment provided in an embodiment of the present invention; Fig.10 is a schematic diagram of the sixth starting point posture selected in the verification experiment provided in an embodiment of the present invention; Fig.11 is a schematic diagram of the seventh starting point posture selected according to the verification experiment provided by an embodiment of the present invention; Fig.12 is a schematic diagram of an eighth starting point posture selected in a verification experiment provided according to an embodiment of the present invention; Fig.13 Schematic diagram of the angle bisector of the camera model coordinate system in the verification experiment provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the same modules are represented by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, the detailed description thereof will not be repeated.

[0019] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0020] like Figure 1 As shown, an embodiment of the present invention provides a robot arm visual servo control method based on deep reinforcement learning, comprising the following steps: S1: A camera is connected to the robot arm, and the camera can move with the movement of the robot arm. Different robot arm positions correspond to different camera field of view angles. A camera model is established. The camera model is a virtual camera, and what it collects is not a picture, but data information of the object being collected.

[0021] A kinematic model of the robotic arm is established, and the corresponding joint angles of the robotic arm can be obtained by inputting the robotic arm posture data into the kinematic model.

[0022] The embodiment of the present invention adopts the DH parameter method to establish the kinematic model of the robot arm, and adopts the perspective projection transformation method to establish the camera model. Both methods are mature existing technologies and are easy to apply. However, the method for establishing the kinematic model and the camera model is not limited to the above-mentioned DH parameter method and perspective projection transformation method.

[0023] S2: The embodiment of the present invention establishes an intelligent agent network of a robot arm based on a SAC algorithm with adaptively adjustable temperature factors. The intelligent agent network includes an Actor network and a Critic network. During the training of the intelligent agent network, the Actor network selects the joint speed of each step of the robot arm. The temperature factor can change in real time with the training process, thereby adjusting the Actor network. In the embodiment of the present invention, the activation function of the Actor network is a leaky relu function, and the activation function of the Critic network is a relu function. The activation function is a necessary function in the SAC algorithm, and both the leaky relu function and the relu function are prior arts.

[0024] S3: After the agent network is established, the agent network is trained. A target is selected, and at least one representative point on the target is selected as a feature point of the target. During the training of the agent network, the motion of each step of the robot arm is measured with reference to the feature point. The selection of the target has nothing to do with the type of task performed by the robot arm. The target can be any complete, characteristic, and well-positioned graphic.

[0025] S4: After determining the target position and feature points, select a robot arm posture near the target, and collect the target image by the camera model at this posture. The target image has exactly the same geometric features as the actual target, and this posture is called the target posture of the robot arm movement. For example, a square is selected as the target, and the points at the four corners of the square are the feature points. When the robot arm is at a certain distance directly above the square, the target (square) image collected by the camera model is still a square, and the target does not deform in the target image. Then the posture of the robot arm at this moment is the target posture. The target posture is a conventional concept in the field of robot arm training, and is the ideal position to be achieved by artificially set robot arm training.

[0026] The purpose of the robot arm servo control training is to control the robot arm to reach the target position. The complete robot arm servo control training process includes multiple training sessions. Each training session is as follows: Randomly select a certain posture of the robot arm, and collect the target image through the camera model at this posture. If the feature points do not all enter the target image, abandon the posture and reselect; if the collected target image contains all the feature points, use this posture as the starting posture of the training scene , according to the starting point posture , the starting position is obtained by inverse solution of the kinematic model of the robot The corresponding joint angle is ,When the robot performs the first step of movement, the reference data comes from the starting position.

[0027] Successfully selected the starting position After that, when the robot moves from the starting position to the target position, the Actor network Exercise to position Joint speed , pose Corresponding joint angle The joint speed of each step of the robot arm is selected one by one according to the feature point error , the feature point error is the error between the feature point coordinates in the target image and the feature point coordinates in the target image at the target position of the manipulator, and the joint velocity is selected The Actor network then controls the robot arm according to the selected joint speed. Movement arrival position , where the robot arm follows the joint speed The movement time is the time interval between two adjacent movement instructions. The time interval of the movement instructions is a preset value before the robot starts to complete the task. For example, if the user wants the robot to move 50 times per second, then a movement instruction needs to be sent to the robot every 20ms. The movement time of each step of the robot is 20ms. Therefore, as long as the joint speed of the robot is obtained , combined with the instruction time interval, the corresponding position and posture of the robot arm after the movement of this step can be obtained Arrival position According to the posture Joint angle , Joint speed The time interval between two adjacent movements can be used to calculate the posture Corresponding joint angle , joint angle It is the joint speed that the Actor network selects for the next movement of the robot arm. The basis of which, Indicates the number of movement steps of the robot arm, , Indicates the upper limit of the predetermined number of robot arm movement steps, Indicates that the robot arm has a posture Exercise to position The joint velocity is When , the Actor network selects the joint speed of the robot's first step movement Based on the starting position Joint angle and starting position The joint speed of the previous robotic arm movement, but this node has no previous robotic arm movement, so the speed value of the joint speed at this time is 0.

[0028] In order to ensure that the posture of the robot arm in each step of movement is closer and closer to the target posture to complete the task, the embodiment of the present invention designs a reward function. The joint speed selected by the Actor network is evaluated once, and the reward value of the joint speed is calculated through the reward function. The SAC algorithm contains a strategy loss parameter, which is essentially an objective function. Combined with the reward value, the SAC algorithm calculates the objective function internally to optimize the Actor network's ability to select joint speeds. Reward value The calculation formula is as follows: ; ; ; ; ; in, represents the first gain coefficient, represents the second gain coefficient, represents the third gain coefficient, Indicates The error norm between the coordinates of the feature points in the target image and the coordinates of the feature points in the target image at the target pose of the robot arm, Indicates The energy consumed by walking, Indicates The speed of the step, Indicates the score, It means that the camera model cannot collect all feature points. represents the threshold of the error two norm, Indicates except and In other cases, The value of should be determined according to the accuracy range of the robot arm's task.

[0029] In the reward value calculation formula, The error of each step of the robot movement is introduced. The movement speed of the robot arm at each step is introduced. The energy consumption of each step of the robot's movement is introduced. The results of each step of the robot's movement are introduced, and the reward value calculation formula fully considers various factors in the robot's movement process, making the evaluation more comprehensive and sufficient, and the evaluation results more meaningful.

[0030] In the reward value calculation formula In, score It is used to reward successful movements and punish failed movements. The score can be calculated by combining the values ​​of other parameters in the calculation formula. The value is taken to prevent the calculated reward value from being too large or too small, which makes the calculation inconvenient. The value is 5.

[0031] Based on the evaluation value, the SAC algorithm will automatically adjust parameters such as the temperature factor, thereby changing the Actor network and improving the Actor network's ability to select joint speeds.

[0032] when When , it indicates the position that the robot arm reaches The error norm relative to the target pose is at the threshold Within the range, the servo movement is successful and the training of this scene is completed.

[0033] When in position The target image captured by the camera model does not contain all feature points. At this time, the target leaves the field of view of the camera model and the current pose cannot be calculated using the kinematic model. The corresponding joint angle, the Actor network cannot select the joint speed of the next robotic arm, the servo movement fails, and the training of this scene ends.

[0034] When the robot arm is completing a task, it cannot be allowed to move indefinitely, so the upper limit of the number of movement steps of the robot arm in each training scene is preset. If the number of robot movement steps exceeds the preset upper limit , the servo movement fails and the training of this scene ends.

[0035] After each episode of training, all evaluation values ​​obtained in the episode of training are summed up to obtain the corresponding reward for the episode of training, and the starting position is reselected. , proceed to the next scene of training.

[0036] S5: After each scene of training is completed, the training situation of the scene is accumulated as an experience data and stored in the experience playback pool. The content of each stored experience data includes ,in, Indicates the current motion status of the robot arm. Indicates the action of the robot arm corresponding to the joint velocity selected by the Actor network. Indicates that the robot arm performs an action After that, the training environment feeds back the state of the new robot arm's movement. A flag indicating whether the current training is finished. , not yet finished In the Actor network optimization process, the number of experience replay pools that need to be extracted is The empirical data of Article 1 is used as a reference. After the training is completed, calculate the previous The average reward of the episode, such as Figure 2 and Figure 3 As shown in Figure 1, when the average value increases with the number of training episodes and the fluctuation range of the reward for each episode is within the allowable error range, it is considered that the agent network has strategic capabilities, and the joint velocity of each step of movement selected by it reaches the position All of them can successfully approach the target pose, and the agent network training is completed. The allowable error range mentioned here is determined according to the accuracy of the task to be completed by the robot arm.

[0037] The agent network of the embodiment of the present invention also includes a critic network. During the movement of the robot arm, it is necessary not only to ensure that the joint speed of each step of the robot arm is accurately selected, but also to ensure that the movement state of the robot arm is smooth and has no lag. Therefore, the critic network is used to make an overall evaluation of the joint speed and corresponding spatial state of the robot arm. The critic network includes two data paths, one path corresponding to the position and posture , that is, the motion path, the pose path requires input pose The corresponding joint velocity data; the other path corresponds to the spatial state of the robot arm. The spatial state data is sensed and acquired by the intelligent agent, including: feature point error, robot arm joint angle and The robot arm joint speed corresponding to the step, where the feature point error is the error between the feature point coordinates in the target image and the feature point coordinates in the target image at the target position of the robot arm. The Critic network combines the data of the two paths and outputs the action value Q. The benefit value Q can reflect the evaluation ability of the Critic network. In particular, the Critic network inside the SAC algorithm includes Q network 1 and Q network 2. Both Q network 1 and Q network 2 can output the action value Q. In order to prevent the Critic network from overestimating the action value Q, the minimum function min is used to take the smaller value of the action value output by Q network 1 and Q network 2 as the final action value Q of the Critic network. Temporal difference is another function that comes with the SAC algorithm. Its essence is a value function, which is used to update Q network 1 and Q network 2. Q network 1, Q network 2 and temporal difference parameters are inherent characteristics of the SAC algorithm and belong to the prior art.

[0038] In order to verify the feasibility of the servo control method of the present invention, the following verification experiments were also conducted: by Figure 4 The robot arm posture shown is the target posture. The coordinate systems in the figure are the inertial coordinate system, the camera model coordinate system and the robot arm end coordinate system. The threshold of the error norm is set The value is 0.001, and the selection is Figures 5 to 12 The eight groups of robot arm postures shown verify the method of the embodiment of the present invention and the model-based method. Figure 5 The pose shown corresponds to a yaw angle of +30° for the camera model. Figure 6 The yaw angle of the camera model corresponding to the pose shown is -30°. Figure 7 The posture shown corresponds to a camera model with a pitch angle of +30°. Figure 8 The pitch angle of the camera model corresponding to the posture shown is -30°, as shown in Fig.13 The angle bisector a and angle bisector b shown in the figure, the z-axis of the camera model coordinate system rotates +30° and -30° around the angle bisector b between the x-axis and the y-axis of the camera model coordinate system to the following Fig. 9 and Fig.10 The z-axis of the camera model coordinate system rotates +30° and -30° around the angle bisector a between the x-axis and the y-axis of the camera model coordinate system to the position shown in FIG. Fig.11 and Fig.12 The posture shown.

[0039] After the robot arm training is completed, the experimental results shown in Table 1 are obtained. According to the experimental results, it can be seen that compared with the traditional model-based training method, the 8 groups of postures are trained by the method of the embodiment of the present invention, and all 8 groups of postures can converge successfully. The traditional model-based training method only achieves convergence for 3 groups of postures. Compared with the traditional model-based training method, the control accuracy of the method of the embodiment of the present invention is about four times that of the traditional method, and the control accuracy is significantly improved.

[0040] Table 1 Experimental results

[0041] Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. Those skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

[0042] The above specific implementations of the present invention do not constitute a limitation on the protection scope of the present invention. Any other corresponding changes and modifications made based on the technical concept of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A visual servo control method for a robotic arm based on deep reinforcement learning, wherein a camera is connected to the robotic arm, characterized in that: Control methods include: S1: Establish the camera model and the kinematic model of the robotic arm; S2: Establishing an agent network through the SAC algorithm, wherein the agent network includes an Actor network; S3: Set the target position and select at least one point on the target as a feature point; S4: Train the agent network. The training process of each episode is as follows: Randomly select the starting position of the robot arm , the starting point posture The corresponding joint angle is ; From the starting point At the beginning, the motion process of the robot arm is: The Actor network is based on the joint velocity , pose Corresponding joint angle The joint speed corresponding to the next movement of the robot arm is selected by the error of the feature point , and control the robot arm according to the joint speed Movement arrival position , according to the joint angle and joint speed Calculate pose Corresponding joint angle , where the robot arm follows the joint speed The movement time is the preset time interval between two movement instructions. Indicates the number of movement steps of the robot arm, , represents the upper limit of the predetermined number of robot arm movement steps, the feature point error is the error between the feature point coordinates in the target image and the feature point coordinates in the target image of the robot arm at the target posture, Indicates that the robot arm has a posture Exercise to position The joint velocity; Use the following formula to calculate the joint speed selected by the Actor network Evaluation value : ； ； ； ； ； in, represents the first gain coefficient, represents the second gain coefficient, represents the third gain coefficient, Indicates The error norm between the coordinates of the feature points in the target image and the coordinates of the feature points in the target image at the target pose of the robot arm, Indicates The energy consumed by walking, Indicates the score, Indicates that the camera model cannot collect all feature points. represents the threshold of the error two norm, Indicates except and Other circumstances other than when When , the servo movement is successful and the training of this scene is finished; When not all feature points appear in the target image captured by the camera model, the servo movement fails and the training of this scene ends; If the number of robot movement steps exceeds the preset upper limit , the servo movement fails, and the training of this scene ends; After each scene of training, the sum of the evaluation values ​​corresponding to each step of the joint velocity and posture in the scene is calculated as the reward for the scene; S5: No. After the training is completed, calculate the previous The training of the agent network ends when the average value increases with the number of training episodes and the fluctuation range of the reward for each episode is within the allowable error range.

2. The robot arm visual servo control method based on deep reinforcement learning as claimed in claim 1, characterized in that: The DH parameter method is used to establish the kinematic model of the robotic arm.

3. The robot arm visual servo control method based on deep reinforcement learning as claimed in claim 1, characterized in that: The camera model is established by using a perspective projection transformation method.

4. The robot arm visual servo control method based on deep reinforcement learning as claimed in claim 1, characterized in that: The starting position of the robot arm The selection method is as follows: randomly select the robot arm posture, collect the target image at the posture through the camera model, and when all feature points appear in the target image, the selected posture can be used as the starting posture .

5. The robot arm visual servo control method based on deep reinforcement learning as claimed in claim 4, characterized in that: According to the starting point posture , the starting point position is obtained by inverse solution of the kinematic model Corresponding joint angle .

6. The robot arm visual servo control method based on deep reinforcement learning as claimed in claim 1, characterized in that: The score The value is 5.

7. The robot arm visual servo control method based on deep reinforcement learning as claimed in claim 1, characterized in that: The agent network also includes a Critic network, which senses the position through the agent network. The spatial state and the posture The corresponding joint velocity is input into the Critic network, and the Critic network outputs the action value Q.

8. The robot arm visual servo control method based on deep reinforcement learning as claimed in claim 7, characterized in that: The posture The spatial state includes: feature point error, robot arm joint angle , No. The robot arm joint speed corresponding to the step.

9. The robot arm visual servo control method based on deep reinforcement learning as claimed in claim 1, characterized in that: The activation function of the Actor network is the leaky relu function, and the activation function of the Critic network is the relu function.

Images