A robotic arm visual depth estimation method based on deep reinforcement learning

Abstract

The present invention relates to the technical field of robot arm visual depth estimation, and specifically provides a robot arm visual depth estimation method based on deep reinforcement learning. Based on the SAC algorithm, an intelligent agent network is established, targets and feature points are set, a robot arm kinematic model, a camera model and a physical model are established, the starting position of the robot arm is randomly selected, and the robot arm is controlled to move gradually from the starting position to the target position. At each step position, the Actor network estimates the visual depth of the feature point based on the target image collected by the camera model, and the robot arm moves to the next step based on the estimated visual depth. A reward function is set to evaluate the visual depth estimated by the Actor network to improve the ability of the Actor network to estimate visual depth. During the movement of the robot arm, the method of the present invention can realize real-time estimation of the visual depth of the target at the current position, ensure that the interaction matrix of the image can be accurately calculated, achieve global convergence, and improve the success rate of the visual servoing task.

Description

Technical Field

[0001] The present invention relates to the technical field of visual depth estimation of a robotic arm, and specifically provides a method for visual depth estimation of a robotic arm based on deep reinforcement learning. Background Art

[0002] Visual servo control for robotic arms is a comprehensive technology that integrates machine vision, automatic control, and computer technologies. This technology uses visual signals to achieve servo motion control of the robotic arm, effectively improving the robotic arm's motion accuracy and robustness, and enhancing the robotic arm's ability to interact with complex environments. Depending on the control method, robotic arm visual servo methods can be divided into image-based visual servo methods and position-based visual servo methods. Image-based visual servo methods do not rely on precise camera intrinsics and extrinsic parameters and have high robustness, making them widely used in the field of robotic visual servoing. When using image-based visual servo methods, calculating the interaction matrix corresponding to image features is a very important step. The interaction matrix can accurately describe the relationship between image feature changes and camera speed. However, the calculation of the interaction matrix includes visual depth, which cannot be obtained through measurement.

[0003] Existing visual depth estimation methods primarily use the visual depth at an ideal position instead of the global visual depth, which only achieves local convergence. This method cannot guarantee successful results when the initial camera position deviates significantly from the ideal position. To address this issue, a new method for visual depth estimation is proposed. Summary of the Invention

[0004] To address these issues, the present invention provides a method for estimating visual depth for a robotic arm based on deep reinforcement learning. This method establishes an agent network based on the SAC algorithm. After each position of the robotic arm, the Actor network estimates the visual depth of target feature points. An evaluation function is established to evaluate the visual depth estimates of the Actor network, improving the Actor network's ability to estimate visual depth. This method enables real-time visual depth estimation during robotic arm motion, ensuring an accurate interaction matrix, achieving convergence over a wide range, and improving the success rate of visual servoing tasks.

[0005] The present invention provides a method for predicting the visual depth of a robotic arm based on deep reinforcement learning, comprising:

[0006] S1: Establish the camera model and the kinematic model of the robotic arm;

[0007] S2: Establish an agent network through the SAC algorithm, which includes the Actor network;

[0008] S3: Building a physical model of the agent network;

[0009] S4: Set the target position and select at least one point on the target as a feature point;

[0010] S5: The training process of each scene of the robotic arm training is as follows:

[0011] Randomly select the starting position N0 of the robot arm, and the joint angle corresponding to the starting position N0 is q0;

[0012] Starting from the starting position N0, the motion process of the robot arm is:

[0013] The pose N t The coordinates of the feature points in the corresponding target image and the joint angle q t and joint velocity action t-1 Input the Actor network to obtain the visual depth estimation value of the feature points, and calculate the robot arm joint speed action based on the visual depth estimation value of the feature points t ;

[0014] Action according to joint speed t And the preset time interval of the robot arm movement instruction, the Actor network controls the robot arm to move to position N t+1 , in pose N t+1 At the joint angle q t and joint velocity action t Calculate pose N t+1 The joint angle q t+1 , where the robot arm follows the joint speed action t The movement time is the preset time interval between two movement instructions. t Indicates that the robot arm changes from position N t Movement to position N t+1 The joint speed of the robot arm is t, t represents the number of movement steps of the robot arm, T ≥ t > 0, and T represents the upper limit of the predetermined number of movement steps of the robot arm;

[0015] Use the following formula to calculate the evaluation value R corresponding to the visual depth estimated by the Actor network t :

[0016] R t =r1+r2

[0017]

[0018]

[0019] Among them, out of fov means that the camera model cannot collect all feature points, ||e t|| represents the error norm between the coordinates of the feature points in the target image at the tth step and the coordinates of the feature points in the target image at the target pose of the manipulator, δ represents the threshold of the error norm, and other represents the error except out of fov and ||e t In other cases other than |‖<δ, m represents a fraction, ||depth d (t)|| represents the two-norm of the actual value of the visual depth of the feature point at step t obtained by calculating the physical model, ||depth r (t)|| represents the bimodal value of the visual depth estimation value of the target feature point at step t;

[0020] when ||e t When ‖<δ, the servo movement is successful and the training of this scene ends;

[0021] If not all feature points appear in the target image captured by the camera model, the servo movement fails and the training of this scene ends;

[0022] If the number of movement steps of the robot arm exceeds the preset upper limit T, the servo movement fails and the training ends;

[0023] After each scene is trained, the sum of the evaluation values ​​corresponding to each step in the scene is calculated as the reward for the scene;

[0024] S6: After the nth episode of training is completed, the average of the rewards of the first n episodes is calculated; when the average value increases with the number of training episodes and the fluctuation range of the rewards of each episode is within the allowable error range, the agent network training is completed.

[0025] Preferably, the DH parameter method is used to establish the kinematic model.

[0026] Preferably, the camera model is established using a perspective projection transformation method.

[0027] Preferably, the method for selecting the starting position N0 of the robot arm is as follows: randomly select the robot arm position, collect the target image at the position through the camera model, and when all feature points appear in the target image, the selected position can be used as the starting position N0.

[0028] Preferably, based on the feature points in the target image collected at the starting position, the joint angle q0 corresponding to the starting position N0 of the robot arm is obtained through a kinematic model.

[0029] Preferably, the fraction m is 5.

[0030] Preferably, the agent network also includes a Critic network, which senses the pose N through the agent network. t The spatial state and the pose N tThe corresponding joint velocity is input into the Critic network, and the Critic network outputs the benefit value Q.

[0031] Preferably, the pose N t The spatial state includes: feature point error, robot arm joint angle q t , the visual depth estimation value corresponding to the t-1th step, where the feature point error is the error between the feature point coordinates in the target image and the feature point coordinates in the target image of the manipulator at the target pose.

[0032] Preferably, the activation function of the Actor network is the leaky ReLU function, and the activation function of the Critic network is the ReLU function.

[0033] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0034] The present invention establishes an intelligent agent network based on the SAC algorithm with adaptively adjustable temperature factors. Every time the robotic arm reaches a posture, the Actor network is used to estimate the visual depth of the feature points at that posture, and the visual depth estimated by the Actor network is judged by an evaluation function, thereby improving the ability of the Actor network to estimate the visual depth. During the movement of the robotic arm, the method of the present invention estimates the visual depth of the feature points once at each posture step, which can ensure that the visual depth of the feature points is updated in real time. Compared with the traditional method of using the visual depth at the ideal posture instead of the global visual depth, the visual depth of the method of the present invention can calculate a more accurate interaction matrix. The joint speed of the robotic arm movement is obtained by calculation based on the interaction matrix, so that the accuracy of each step of the robotic arm movement can be indirectly improved, large-scale convergence can be achieved, and the success rate of the visual servoing task can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 2 is a schematic diagram of an agent network training process according to an embodiment of the present invention;

[0036] Figure 2 is a graph showing how the average reward value varies with the number of training episodes according to an embodiment of the present invention;

[0037] Figure 3 is a curve chart showing how the training reward for each act varies with the number of acts provided in accordance with an embodiment of the present invention;

[0038] Figure 4 2 is a schematic diagram of target poses selected in a verification experiment according to an embodiment of the present invention;

[0039] Figure 5 This is a schematic diagram of the first starting point posture selected in the verification experiment provided by an embodiment of the present invention;

[0040] Figure 6 2 is a schematic diagram of a second starting point posture selected in a verification experiment provided according to an embodiment of the present invention;

[0041] Figure 7 2 is a schematic diagram of a third starting point posture selected in a verification experiment provided by an embodiment of the present invention;

[0042] Figure 8 1 is a schematic diagram of a fourth starting point posture selected in a verification experiment provided by an embodiment of the present invention;

[0043] Figure 9 Schematic diagram of the fifth starting point posture selected in the verification experiment provided by an embodiment of the present invention;

[0044] Figure 10 is a schematic diagram of the sixth starting point posture selected in the verification experiment provided by an embodiment of the present invention;

[0045] Figure 11 is a schematic diagram of the seventh starting point posture selected according to the verification experiment provided by an embodiment of the present invention;

[0046] Figure 12 1 is a schematic diagram of the eighth starting point posture selected according to the verification experiment provided by an embodiment of the present invention;

[0047] Figure 13 Schematic diagram of the angle bisector of the camera model coordinate system in the verification experiment provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, identical modules are denoted by identical reference numerals. In the case of identical reference numerals, their names and functions are also identical. Therefore, their detailed description will not be repeated.

[0049] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0050] like Figure 1 As shown, an embodiment of the present invention provides a method for predicting the visual depth of a robotic arm based on deep reinforcement learning, and the specific steps are as follows:

[0051] S1: A camera is attached to the robotic arm, and it moves with the arm. Different robotic arm positions correspond to different camera field of view angles. A camera model is created. The camera model is a virtual camera that captures data about the object rather than an image.

[0052] A kinematic model of the robotic arm is established, and the corresponding robotic arm joint angles can be obtained by inputting the robotic arm posture data into the kinematic model.

[0053] In the embodiment of the present invention, the DH parameter method is used to establish the kinematic model of the robotic arm, and the perspective projection transformation method is used to establish the camera model. Both methods are mature existing technologies and are easy to apply. However, the methods for establishing the kinematic model and the camera model are not limited to the above-mentioned DH parameter method and perspective projection transformation method.

[0054] S2: In an embodiment of the present invention, an intelligent agent network for a robotic arm is established based on a SAC algorithm with an adaptively adjustable temperature factor. The intelligent agent network includes an Actor network and a Critic network. During the training of the intelligent agent network, the Actor network selects the position of the robotic arm at each step. The temperature factor can change in real time with the training process, thereby adjusting the Actor network. In this embodiment of the present invention, the activation function of the Actor network is a leaky ReLU function, and the activation function of the Critic network is a ReLU function. Activation functions are essential functions in the SAC algorithm, and both the leaky ReLU function and the ReLU function are prior art.

[0055] S3: Establish a physical model of the intelligent agent network. During the training process of the intelligent agent network, every time the robot arm moves to a new position, the actual target visual depth accuracy value at that position can be obtained through the above physical model. The actual visual depth plays an important role in the subsequent measurement of the estimated visual depth.

[0056] S4: After establishing the agent network, train it. Select a target and choose at least one representative point on the target as its feature point. During the agent network training process, use this feature point as a reference to measure the robot's movement at each step. The target selected is independent of the task the robot performs; it can be any complete, characteristic, and well-defined shape.

[0057] S5: After determining the target position and feature points, a robotic arm pose is selected near the target. The camera model captures a target image at this pose. This target image has identical geometric features to the actual target. This pose is called the target pose for the robotic arm motion. For example, a square is selected as the target, and the four corners of the square are used as feature points. When the robotic arm is at a certain distance directly above the square, the target (square) image captured by the camera model remains a square, and the target does not appear deformed in the target image. In this case, the robotic arm pose at this moment is the target pose.

[0058] The purpose of the robot arm servo control training is to control the robot arm to reach the target position. The complete robot arm servo control training process includes multiple training sessions. Each training session is as follows:

[0059] A certain posture of the robot arm is randomly selected, and the target image is collected through the camera model at this posture. If not all feature points enter the target image, the posture is abandoned and reselected; if the collected target image contains all the feature points, the posture is used as the starting posture N0 of the training scene. According to the coordinates of the feature points in the target image at the starting posture N0, the kinematic model of the robot arm is used to obtain the joint angle q0 corresponding to the starting posture N0.

[0060] Starting from the starting position N0, the robot arm moves gradually towards the target position, using N t Indicates the posture corresponding to any step in the process of the robot arm moving towards the target posture. The robot arm reaches posture N t After that, the Actor network estimates the visual depth of the feature points and converts the pose N t The coordinates of the feature points in the corresponding target image and the joint angle q t and joint velocity, action t-1 Input Actor network, Actor network outputs the robot arm in position N t Visual depth estimation based on feature points.

[0061] The interaction matrix of the feature points can be calculated based on the visual depth estimation of the feature points. The interaction matrix can accurately describe the relationship between the change of the feature points and the camera speed of the camera model. Therefore, the camera speed of the camera model can be calculated based on the interaction matrix of the feature points. The camera is fixedly connected to the robotic arm and the camera follows the movement of the robotic arm. Therefore, the camera speed is the same as the movement speed of the end of the robotic arm, which can indirectly feedback the joint speed of the robotic arm. According to the camera speed, the joint speed action is obtained by inverse calculation of the robotic arm Jacobian matrix. t , Figure 1 J in -1 This means the inverse calculation of the Jacobian matrix.

[0062] The above method can be used to calculate the joint speed action of the next movement of the robot arm t , the joint speed of the robot movement action t The movement time of the robot arm is the key data for the next movement of the robot arm. During the movement of the robot arm, the time interval of the movement instructions issued by the intelligent network is a preset fixed value. For example, if the user requires the robot arm to move 50 times per second, the time interval of the movement instructions is 20ms. This time interval is the movement time of each step of the robot arm. The robot arm moves according to the joint speed action. tBy moving with the above movement time, you can reach the posture N t+1 , and repeat the pose N t+1 Visual depth estimation of feature points at action t Indicates that the robot arm changes from position N t Movement to position N t+1 The joint speed of the robot arm is t, the number of movement steps of the robot arm is T≥t>0, and T represents the upper limit of the predetermined number of movement steps of the robot arm.

[0063] When t=1, the robot moves from the starting position N0 to the position N1. The Actor network estimates the visual depth of the feature point based on the feature point coordinates of the target image at the starting position N0, the joint angle q0 corresponding to the starting position N0, and the joint velocity action 0-1 , here action 0-1 The corresponding speed value is 0.

[0064] In order to ensure that the Actor network can obtain accurate visual depth estimation, the visual depth estimated by the Actor network is evaluated each time. The SAC algorithm contains a policy loss parameter, which is essentially an objective function. According to the evaluation results, the SAC algorithm calculates the objective function internally and adaptively adjusts its internal temperature factor parameters to improve the ability of the Actor network to estimate visual depth. The temperature factor and policy loss parameters are parameters that come with the SAC itself. The adaptive adjustment calculations are all performance of the SAC algorithm itself and belong to the prior art. The embodiment of the present invention sets the reward function for Acto r The network evaluates the visual depth estimated at each step and obtains the corresponding reward value R t , reward value R t The calculation formula is as follows:

[0065] R t =r1+r2;

[0066]

[0067] Among them, out of fov means that the camera model cannot collect all feature points, ||e t || represents the error norm between the coordinates of the feature points in the target image at the t-th step and the coordinates of the feature points in the target image at the target position of the manipulator. δ represents the threshold of the error norm. The value of δ is determined according to the accuracy of the manipulator task. For example, the accuracy of the manipulator in completing the assembly task is higher than that in completing the transportation task. The corresponding δ value in the assembly task is smaller than the δ value in the transportation task. other represents the error except out of fov and ||e t In other cases other than |‖<δ, m represents a fraction, ||depth d(t)|| represents the two-norm of the actual visual depth value of the feature point at step t. The actual visual depth value of the feature point can be obtained by calculating the physical model. r (t)|| represents the binary norm of the visual depth estimation value of the target feature point in the t-th step.

[0068] In r1 of the reward value calculation formula, the fraction m is used to reward successful movements and punish failed movements. The fraction m can be determined based on the values ​​of other parameters in the calculation formula. In this embodiment of the present invention, the fraction m is set to 5.

[0069] In the reward calculation formula, r1 incorporates the result of each robotic arm's movement, and r2 incorporates the accuracy of the visual depth estimated by the Actor network. This formula fully considers all factors in the robotic arm's movement, making the evaluation more comprehensive and meaningful.

[0070] when ||e t When ||<δ, it means the position N reached by the robot arm t If the error binary norm relative to the target pose is within the threshold δ, the servo motion is successful and the training of this episode ends.

[0071] When in position N t Not all feature points appear in the target image captured by the camera model. At this time, the target leaves the field of view of the camera model and the current pose N cannot be calculated by the kinematic model. t The corresponding joint angle, the Actor network cannot estimate the current pose N t The corresponding feature point visual depth, servo motion fails, and the training of this scene ends.

[0072] When the robot arm is completing a task, it cannot be allowed to move indefinitely. Therefore, an upper limit T of the number of movement steps of the robot arm in each training scene is preset. If the number of movement steps of the robot arm exceeds the preset upper limit T, the servo movement fails and the training of this scene ends.

[0073] After each episode of training is completed, all evaluation values ​​obtained in the episode of training are summed up to obtain the reward corresponding to the episode of training, and the starting position N0 is reselected to carry out the next episode of training.

[0074] S6: After each scene of training is completed, the training situation of the scene is accumulated as an experience data and stored in the experience replay pool. The content of each stored experience data includes (s, a, R t,s′,d), where s represents the current state of the robot arm, a represents the visual depth value estimated by the Actor network, s′ represents the new state of the robot arm's movement as fed back by the training environment after the robot arm performs the action according to the visual depth value a, and d represents the flag of whether the current training is finished. If it is finished, d = 1, and if it is not finished, d = 0. During the Actor network optimization process, it is necessary to extract batch_size pieces of experience data from the experience replay pool as a reference. Figure 2 and Figure 3 As shown in the figure, after the nth episode training is completed, the average value of the rewards of the first n episodes is calculated. When the average value increases with the number of training episodes and the fluctuation range of the rewards of each episode is within the allowable error range, it is considered that the agent network has strategic capabilities. t The ability to estimate the visual depth of the feature points meets the task requirements, allowing the robot arm to successfully approach the target pose, and the agent network training is completed. The allowable error range mentioned here is determined by the accuracy of the task to be completed by the robot arm.

[0075] The intelligent agent network of the embodiment of the present invention also includes a critic network, which not only ensures that each step posture N t In order to ensure that the visual depth estimation of the feature points is accurate and the motion of the robot arm is smooth and free of lag, the Critic network is used to make an overall evaluation of the estimated visual depth of the feature points and the corresponding spatial state of the robot arm. The Critic network includes two data paths, one path corresponding to the pose N t , that is, the motion path, the pose path requires input pose N t The corresponding joint velocity data; the other path corresponds to the spatial state of the robot arm. The spatial state data is perceived and acquired by the intelligent agent, including: feature point error, robot arm joint angle q t and pose N t-1 The visual depth estimate of the feature point at the target position is calculated, where the feature point error is the error between the feature point coordinates in the target image and the feature point coordinates in the target image at the target position of the robotic arm. The critic network combines data from the two paths and outputs an action value Q, which reflects the critic network's evaluation capability. Specifically, the critic network within the SAC algorithm includes Q-Network 1 and Q-Network 2, both of which can output an action value Q. To prevent the critic network from overestimating the action value Q, the minimum function min is used to take the smaller of the action values ​​output by Q-Network 1 and Q-Network 2 as the critic network's final action value Q. Temporal difference is another function built into the SAC algorithm. It is essentially a value function used to update Q-Network 1 and Q-Network 2. Q-Network 1, Q-Network 2, and the temporal difference parameters are inherent characteristics of the SAC algorithm and are part of the existing technology.

[0076] In order to verify the feasibility of the servo control method of the present invention, the following verification experiments were conducted:

[0077] by Figure 4 The robot arm posture shown is the target posture. The coordinate systems in the figure are the inertial coordinate system, the camera model coordinate system and the robot arm end coordinate system. In the robot arm visual servo control training environment based on the deep reinforcement learning algorithm, the error norm threshold δ is set to 0.004. Figures 5 to 12 The eight sets of robot arm postures shown verify the method of the embodiment of the present invention and the model-based method. Figure 5 The pose shown corresponds to a yaw angle of +30° for the camera model. Figure 6 The yaw angle of the camera model corresponding to the pose shown is -30°. Figure 7 The camera model's pitch angle corresponding to the pose shown is +30°. Figure 8 The pitch angle of the camera model corresponding to the posture shown is -30°, as shown in Figure 13 The angle bisector a and angle bisector b shown in the figure are rotated by +30° and -30° around the angle bisector b between the x-axis and the y-axis of the camera model coordinate system. Figure 9 and Figure 10 The z-axis of the camera model coordinate system rotates +30° and -30° around the angle bisector a between the x-axis and the y-axis of the camera model coordinate system to the following pose: Figure 11 and Figure 12 The posture shown.

[0078] After the robotic arm training is completed, the experimental results shown in Table 1 are obtained. According to the experimental results, it can be seen that the eight pose groups can be trained successfully using the method of the embodiment of the present invention. The eight pose groups can be successfully converged. The traditional method using fixed visual depth can only achieve convergence for three pose groups, and the convergence error is greater than that of the method of the embodiment of the present invention. The control accuracy of the method of the embodiment of the present invention is significantly improved compared with the traditional method.

[0079] The experimental results show that the target image error still does not converge to 0 using the method of the embodiment of the present invention. This is related to the selected feature points. The experiment selected four image feature points, each with two coordinates. Therefore, the length of the feature vector of the feature point is 8, but the number of degrees of freedom of the robotic arm is 6, resulting in data redundancy. Data redundancy can enhance the robustness of the algorithm, but it will cause the interaction matrix of the feature points to be not full rank. Therefore, it is still possible for the feature point error to fall into the null space of the interaction matrix, resulting in the final feature point error being stable but not 0.

[0080] Table 1 Experimental results

[0081]

[0082] Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

[0083] The above specific embodiments of the present invention do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made based on the technical concept of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for visual depth estimation of a robotic arm based on deep reinforcement learning, wherein a camera is connected to the robotic arm, characterized in that: include: S1: Establish the camera model and the kinematic model of the robotic arm; S2: Establishing an agent network through the SAC algorithm, wherein the agent network includes an Actor network; S3: Building a physical model of the agent network; S4: Set the target position and select at least one point on the target as a feature point; S5: The training process of each scene of the robotic arm training is as follows: Randomly select the starting position of the robot arm , starting point pose The corresponding joint angle is ; From the starting position Initially, the motion process of the robotic arm is: Position The coordinates and joint angles of the feature points in the corresponding target image and joint velocity Input the Actor network to obtain the visual depth estimation of the feature points, and calculate the joint velocity of the robotic arm based on the visual depth estimation of the feature points ; According to joint speed The Actor network controls the movement of the robotic arm to the desired position within the preset robotic arm movement instruction time interval. , in pose At, according to the joint angle and joint velocity Calculate pose Joint angle , where the robot arm follows the joint speed The movement time is the preset time interval between two movement instructions. Indicates the robot arm's posture Exercise to the right posture The joint velocity, Indicates the number of movement steps of the robot arm, , Indicates the upper limit of the predetermined number of robot arm movement steps; Use the following formula to calculate the evaluation value corresponding to the visual depth estimated by the Actor network : ； ； ； in, Indicates that the camera model cannot collect all feature points. Indicates the The error norm between the coordinates of the feature points in the target image and the coordinates of the feature points in the target image at the target pose of the manipulator, represents the threshold of the error two norm, Indicates except and In other cases, Indicates a score, The first The second norm of the actual value of the visual depth of the step feature point, Indicates the The second norm of the visual depth estimation of the target feature point; when When , the servo movement is successful and the training of this scene ends; If not all feature points appear in the target image captured by the camera model, the servo movement fails and the training of this scene ends; If the number of robot movement steps exceeds the preset upper limit , the servo movement fails, and the training of this scene ends; After each scene is trained, the sum of the evaluation values ​​corresponding to each step in the scene is calculated as the reward for the scene; S6: After the training is completed, calculate the The average value of the episode rewards; when the average value increases with the number of training episodes and the fluctuation range of the reward of each episode is within the allowable error range, the agent network training is completed.

2. The method for predicting the visual depth of a robotic arm based on deep reinforcement learning according to claim 1, wherein: The kinematic model is established using the DH parameter method.

3. The method for predicting the visual depth of a robotic arm based on deep reinforcement learning according to claim 1, wherein: The camera model is established by adopting a perspective projection transformation method.

4. The method for predicting the visual depth of a robotic arm based on deep reinforcement learning according to claim 1, wherein: The starting position of the robotic arm The selection method is as follows: randomly select the robot arm posture, collect the target image at the posture through the camera model, and when all feature points appear in the target image, the selected posture is used as the starting posture .

5. The method for predicting the visual depth of a robotic arm based on deep reinforcement learning according to claim 1, wherein: According to the feature points in the target image collected at the starting position, the starting position of the robot arm is obtained through the kinematic model. Corresponding joint angle .

6. The method for predicting the visual depth of a robotic arm based on deep reinforcement learning according to claim 5, wherein: The score The value is 5.

7. The method for predicting the visual depth of a robotic arm based on deep reinforcement learning according to claim 1, wherein: The agent network also includes a Critic network, which perceives the posture through the agent network. The spatial state and the pose The corresponding joint velocity is input into the Critic network, and the Critic network outputs the benefit value Q.

8. The method for predicting robot arm visual depth based on deep reinforcement learning according to claim 1, wherein: The posture The spatial state includes: feature point error, robot arm joint angle , No. The visual depth estimation value corresponding to the step is obtained, wherein the feature point error is the error between the feature point coordinates in the target image and the feature point coordinates in the target image of the robotic arm at the target pose.

9. The method for predicting the visual depth of a robotic arm based on deep reinforcement learning according to claim 7, wherein: The activation function of the Actor network is the leaky ReLU function, and the activation function of the Critic network is the ReLU function.

Images