Mechanical arm visual depth estimation method based on deep reinforcement learning

Abstract

The invention relates to the technical field of mechanical arm visual depth pre-estimation, and particularly provides a mechanical arm visual depth pre-estimation method based on deep reinforcement learning, which comprises the following steps: establishing an intelligent agent network based on an SAC algorithm, setting a target and a feature point, establishing a mechanical arm kinematic model, a camera model and a physical model, randomly selecting a starting point pose of a mechanical arm, and pre-estimating the visual depth of the mechanical arm. And controlling the mechanical arm to gradually move from a starting point pose to a target pose, at each step pose, estimating the visual depth of a feature point by the Actor network according to a target image acquired by the camera model, performing next-step movement by the mechanical arm according to the estimated visual depth, and setting a reward function to evaluate the visual depth estimated by the Actor network. Therefore, the capability of estimating the visual depth of the Actor network is improved. According to the method, in the motion process of the mechanical arm, the visual depth of the target at the current pose can be estimated in real time, it is ensured that an interaction matrix of the image can be accurately calculated, global convergence is achieved, and the success rate of a visual servo task is increased.

Description

Technical Field

[0001] The present invention relates to the technical field of robot arm visual depth estimation, and specifically provides a robot arm visual depth estimation method based on deep reinforcement learning. Background Art

[0002] Robotic arm visual servo control is a comprehensive technology that integrates machine vision, automatic control and computer technologies. This technology realizes the servo motion control of the robot arm through visual signals, which can effectively improve the motion accuracy and robustness of the robot arm and enhance the interaction ability of the robot arm with complex environments. According to different control methods, the robot arm visual servo method can be divided into image-based visual servo method and position-based visual servo method. The image-based visual servo method does not rely on accurate camera intrinsics and extrinsics, and has high robustness, so it is widely used in the field of robot visual servo. Using the image-based visual servo method, it is very important to calculate the interaction matrix corresponding to the image features. The interaction matrix can accurately describe the relationship between the image feature changes and the camera speed, but the calculation of the interaction matrix includes visual depth, which cannot be obtained by measurement.

[0003] Existing visual depth estimation methods mainly use the visual depth at the ideal position instead of the global visual depth, which can only achieve local convergence of errors. When the initial position of the camera is very different from the ideal position, this estimation method cannot guarantee the success of the task. In order to solve the above problems, a new method for estimating visual depth is proposed. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a method for estimating the visual depth of a robot arm based on deep reinforcement learning. An intelligent agent network is established based on the SAC algorithm. After each position of the robot arm, the visual depth of the target feature point is estimated using the Actor network. The visual depth estimated by the Actor network is judged by establishing an evaluation function to improve the ability of the Actor network to estimate the visual depth. During the movement of the robot arm, the method of the present invention can perform real-time visual depth estimation, ensure the accuracy of the interaction matrix, achieve large-scale convergence, and improve the success rate of visual servoing tasks.

[0005] The present invention provides a method for predicting the visual depth of a robotic arm based on deep reinforcement learning, comprising:

[0006] S1: Establish the camera model and the kinematic model of the robotic arm;

[0007] S2: Establish an agent network through the SAC algorithm, which includes the Actor network;

[0008] S3: Building a physical model of the agent network;

[0009] S4: Set the target position and select at least one point on the target as a feature point;

[0010] S5: The training process of each scene of the robot arm training is:

[0011] Randomly select the starting position N of the robot arm 0 , starting position N 0 The corresponding joint angle is q 0 ;

[0012] From the starting position N 0 At the beginning, the motion process of the robot arm is:

[0013] Set the pose N t The coordinates of the feature points and joint angles q in the corresponding target image t and joint velocity action t-1 Input the Actor network to obtain the visual depth estimate of the feature points, and calculate the robot arm joint speed action based on the visual depth estimate of the feature points t ;

[0014] Action according to joint speed t The Actor network controls the robot to move to position N at the preset robot movement command time interval. t+1 , in pose N t+1 At the joint angle q t and joint velocity action t Calculate the pose N t+1 The joint angle q t+1 , where the robot arm acts according to the joint speed t The movement time is the preset time interval between two movement instructions. t Indicates that the robot arm changes from position N t Movement to position N t+1 The joint speed of the robot arm is t, and the number of robot arm movement steps is T≥t>0, where T represents the upper limit of the predetermined number of robot arm movement steps.

[0015] Use the following formula to calculate the evaluation value R corresponding to the visual depth estimated by the Actor network t :

[0016] R t =r 1 +r 2

[0017]

[0018]

[0019] Among them, out of fov means that the camera model cannot collect all feature points, ||e t || represents the error norm between the feature point coordinates in the target image of the tth step and the feature point coordinates in the target image of the robot at the target pose, δ represents the threshold of the error norm, and other represents the error except out of fov and ||e t In other cases except |‖<δ, m represents a fraction, ||depth d (t)|| represents the second norm of the actual value of the visual depth of the feature point at the tth step calculated by the physical model, ||depth r (t)|| represents the bi-norm of the visual depth estimation of the target feature point at the tth step;

[0020] when ||e t When ‖<δ, the servo movement is successful and the training of this scene ends;

[0021] When not all feature points appear in the target image captured by the camera model, the servo movement fails and the training of this scene ends;

[0022] If the number of movement steps of the robot arm exceeds the preset upper limit T, the servo movement fails and the training of this episode ends;

[0023] After each scene of training is completed, the sum of the evaluation values ​​corresponding to each step of the posture in the scene is calculated as the reward for the scene;

[0024] S6: After the nth episode training is completed, the average of the rewards of the first n episodes is calculated; when the average value increases with the number of training episodes and the fluctuation range of the rewards of each episode is within the allowable error range, the agent network training is completed.

[0025] Preferably, the kinematic model is established using the DH parameter method.

[0026] Preferably, the camera model is established using a perspective projection transformation method.

[0027] Preferably, the starting position N of the robot arm 0 The selection method is as follows: randomly select the robot arm posture, collect the target image at the posture through the camera model, and when all feature points appear in the target image, the selected posture can be used as the starting posture N 0 .

[0028] Preferably, the starting position N of the robot arm is obtained through a kinematic model based on the feature points in the target image collected at the starting position. 0 The corresponding joint angle q 0 .

[0029] Preferably, the fraction m is 5.

[0030] Preferably, the agent network also includes a Critic network, which senses the position N through the agent network. t The spatial state and posture N t The corresponding joint velocity is input into the Critic network, and the Critic network outputs the benefit value Q.

[0031] Preferably, the posture N t The spatial state includes: feature point error, robot arm joint angle q t , the visual depth estimate corresponding to the t-1th step, where the feature point error is the error between the feature point coordinates in the target image and the feature point coordinates in the target image at the target pose of the robot arm.

[0032] Preferably, the activation function of the Actor network is the leaky relu function, and the activation function of the Critic network is the relu function.

[0033] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0034] The present invention establishes an intelligent agent network based on a SAC algorithm with adaptively adjustable temperature factors. After each position and posture of the robot arm, the Actor network is used to estimate the visual depth of the feature points at the position and posture, and the visual depth estimated by the Actor network is judged by an evaluation function to improve the ability of the Actor network to estimate the visual depth. During the movement of the robot arm, the method of the present invention estimates the visual depth of the feature points once at each position and posture, which can ensure that the visual depth of the feature points is updated in real time. Compared with the traditional method of using the visual depth at an ideal position instead of the global visual depth, the visual depth of the method of the present invention can calculate a more accurate interaction matrix. The joint speed of the robot arm movement is obtained by calculation based on the interaction matrix, so the accuracy of each step of the robot arm movement can be indirectly improved, a large range of convergence can be achieved, and the success rate of visual servoing tasks can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a schematic diagram of an agent network training process provided according to an embodiment of the present invention;

[0036] Figure 2 is a curve chart showing the variation of the average reward value with the number of training episodes provided according to an embodiment of the present invention;

[0037] Figure 3 is a curve chart showing how the training reward for each scene varies with the number of scenes provided in an embodiment of the present invention;

[0038] Figure 4 is a schematic diagram of target posture selected in a verification experiment provided according to an embodiment of the present invention;

[0039] Figure 5 is a schematic diagram of a first starting point posture selected in a verification experiment provided in an embodiment of the present invention;

[0040] Figure 6 is a schematic diagram of a second starting point posture selected in a verification experiment provided in an embodiment of the present invention;

[0041] Figure 7 is a schematic diagram of a third starting point posture selected in a verification experiment provided in an embodiment of the present invention;

[0042] Figure 8 is a schematic diagram of a fourth starting point posture selected according to a verification experiment provided in an embodiment of the present invention;

[0043] Fig. 9 is a schematic diagram of the fifth starting point posture selected in the verification experiment provided in an embodiment of the present invention;

[0044] Fig.10 is a schematic diagram of the sixth starting point posture selected in the verification experiment provided in an embodiment of the present invention;

[0045] Fig.11 is a schematic diagram of the seventh starting point posture selected according to the verification experiment provided by an embodiment of the present invention;

[0046] Fig.12 is a schematic diagram of an eighth starting point posture selected in a verification experiment provided according to an embodiment of the present invention;

[0047] Fig.13 Schematic diagram of the angle bisector of the camera model coordinate system in the verification experiment provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the same modules are represented by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, the detailed description thereof will not be repeated.

[0049] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0050] like Figure 1 As shown, an embodiment of the present invention provides a method for predicting the visual depth of a robotic arm based on deep reinforcement learning, and the specific steps are as follows:

[0051] S1: A camera is connected to the robot arm, and the camera can move with the movement of the robot arm. Different robot arm positions correspond to different camera field of view angles. A camera model is established. The camera model is a virtual camera, and what it collects is not a picture, but data information of the object being collected.

[0052] A kinematic model of the robotic arm is established, and the corresponding joint angles of the robotic arm can be obtained by inputting the robotic arm posture data into the kinematic model.

[0053] The embodiment of the present invention adopts the DH parameter method to establish the kinematic model of the robot arm, and adopts the perspective projection transformation method to establish the camera model. Both methods are mature existing technologies and are easy to apply. However, the method for establishing the kinematic model and the camera model is not limited to the above-mentioned DH parameter method and perspective projection transformation method.

[0054] S2: The embodiment of the present invention establishes an intelligent agent network of a robot arm based on a SAC algorithm with adaptively adjustable temperature factors. The intelligent agent network includes an Actor network and a Critic network. During the training of the intelligent agent network, the Actor network selects the posture of the robot arm at each step. The temperature factor can change in real time with the training process, thereby adjusting the Actor network. In the embodiment of the present invention, the activation function of the Actor network is a leaky relu function, and the activation function of the Critic network is a relu function. The activation function is a necessary function in the SAC algorithm, and both the leaky relu function and the relu function are prior art.

[0055] S3: Establish a physical model of the intelligent agent network. During the training of the intelligent agent network, every time the robot arm moves to a certain position, the above physical model can be used to obtain the actual target visual depth accuracy at that position. The actual visual depth plays an important role in the subsequent measurement of the estimated visual depth.

[0056] S4: After the agent network is established, the agent network is trained. A target is selected, and at least one representative point on the target is selected as a feature point of the target. During the training of the agent network, the motion of each step of the robot is measured with reference to the feature point. The selection of the target has nothing to do with the type of task performed by the robot. The target can be any complete, characteristic, and well-positioned graphic.

[0057] S5: After determining the target position and feature points, select a robot arm posture near the target, and collect a target image at this posture by the camera model. The target image has exactly the same geometric features as the actual target, and this posture is called the target posture of the robot arm movement. For example, a square is selected as the target, and the points at the four corners of the square are used as feature points. When the robot arm is at a certain distance directly above the square, the target (square) image collected by the camera model is still a square, and the target is not deformed in the target image, etc., then the posture of the robot arm at this moment is the target posture.

[0058] The purpose of the robot arm servo control training is to control the robot arm to reach the target position. The complete robot arm servo control training process includes multiple training sessions. Each training session is as follows:

[0059] Randomly select a certain posture of the robot arm, and collect the target image through the camera model at this posture. If the feature points do not all enter the target image, abandon the posture and reselect; if the collected target image contains all the feature points, use this posture as the starting posture N of the training scene 0 , according to the starting point posture N 0 The coordinates of the feature points in the target image are obtained through the kinematic model of the robot arm to obtain the starting position N 0 The corresponding joint angle is q 0 .

[0060] From the starting position N 0 At the beginning, the robot arm gradually moves towards the target posture, using N t Indicates the posture corresponding to any step in the process of the robot arm moving towards the target posture. The robot arm reaches posture N t After that, the Actor network estimates the visual depth of the feature points and converts the pose N t The coordinates of the feature points and joint angles q in the corresponding target image t and joint velocity, action t-1 Input Actor network, Actor network outputs the robot arm in position N t Visual depth estimation of feature points.

[0061] The interaction matrix of the feature points can be calculated based on the visual depth estimation of the feature points. The interaction matrix can accurately describe the relationship between the change of the feature points and the camera speed of the camera model. Therefore, the camera speed of the camera model can be calculated based on the interaction matrix of the feature points. The camera is fixedly connected to the robot and the camera moves with the robot. Therefore, the camera speed is the same as the movement speed of the end of the robot, which can indirectly feedback the joint speed of the robot. According to the camera speed, the joint speed action is obtained by inverse calculation of the Jacobian matrix of the robot. t , Figure 1 J in-1 That is, it represents the inverse calculation of the Jacobian matrix.

[0062] The above method can be used to calculate the joint speed action of the next movement of the robot arm t , the joint speed of the robot movement action t The movement time of the robot arm is the key data for the next movement of the robot arm. During the movement of the robot arm, the time interval of the movement instructions issued by the agent network is a preset fixed value. For example, if the user requires the robot arm to move 50 times per second, the time interval of the movement instructions is 20ms. This time interval is the movement time of each step of the robot arm. The robot arm moves according to the joint speed action. t By moving with the above movement time, you can reach the position N t+1 , and repeat the pose N t+1 The visual depth estimation of the feature points at action t Indicates that the robot arm changes from position N t Movement to position N t+1 The joint speed of the robot arm is t, the number of movement steps of the robot arm is T≥t>0, and T represents the upper limit of the predetermined number of movement steps of the robot arm.

[0063] When t = 1, the robot arm changes from the starting position N 0 N 1 Movement, the Actor network estimates the visual depth of feature points based on the starting point pose N 0 The feature point coordinates and starting point pose N of the target image 0 The corresponding joint angle q 0 and joint velocity action 0-1 , here action 0-1 The corresponding speed value is 0.

[0064] In order to ensure that the Actor network can obtain accurate visual depth estimates, the visual depth estimated by the Actor network is evaluated each time. The SAC algorithm contains a policy loss parameter, which is essentially an objective function. According to the evaluation results, the SAC algorithm calculates the objective function internally and adaptively adjusts its internal temperature factor parameters to improve the Actor network's ability to estimate visual depth. The temperature factor and policy loss parameters are built-in parameters of the SAC, and their adaptive adjustment calculations are the performance of the SAC algorithm itself, which belongs to the prior art. The embodiment of the present invention sets the reward function for the Actor r The network evaluates the visual depth estimated at each step and obtains the corresponding reward value R t , reward value R t The calculation formula is as follows:

[0065] R t =r1 +r 2 ;

[0066]

[0067] Among them, out of fov means that the camera model cannot collect all feature points, ||e t || represents the error norm between the feature point coordinates in the target image at the tth step and the feature point coordinates in the target image at the target position of the robot arm. δ represents the threshold of the error norm. The value of δ is determined according to the accuracy of the robot arm's task. For example, the accuracy of the robot arm in completing the assembly task is higher than that in completing the transportation task. The corresponding δ value in the assembly task is smaller than the δ value in the transportation task. other represents the error except out of fov and ||e t In other cases except |‖<δ, m represents a fraction, ||depth d (t)|| represents the binary norm of the actual visual depth value of the feature point at step t. The actual visual depth value of the feature point can be obtained by calculating the physical model. r (t)|| represents the binary norm of the visual depth estimate of the target feature point at the tth step.

[0068] In the reward value calculation formula, 1 In the formula, the score m is used to reward successful movements and punish failed movements. The score m can be determined based on the values ​​of other parameters in the calculation formula. In the embodiment of the present invention, the score m is set to 5.

[0069] In the reward value calculation formula, r 1 The result of each step of the robot movement is introduced, r 2 The accuracy of the visual depth estimated by the Actor network is introduced, and the reward value calculation formula fully considers various factors in the movement of the robotic arm, making the evaluation more comprehensive and sufficient, and the evaluation results more meaningful.

[0070] when ||e t When ||<δ, it means the position N reached by the robot t If the error binary norm relative to the target pose is within the threshold δ, the servo motion is successful and the training of this episode ends.

[0071] When in position N t The feature points in the target image collected by the camera model do not all appear. At this time, the target leaves the field of view of the camera model, and the current pose N cannot be calculated by the kinematic model. t The corresponding joint angle, the Actor network cannot estimate the current pose N t The corresponding feature point visual depth, servo motion fails, and the training of this scene ends.

[0072] When the robot arm is completing a task, it cannot be allowed to move indefinitely, so an upper limit T of the number of movement steps of the robot arm in each training scene is preset. If the number of movement steps of the robot arm exceeds the preset upper limit T, the servo movement fails and the training of this scene ends.

[0073] After each episode of training, all evaluation values ​​obtained in the episode of training are summed up to obtain the corresponding reward for the episode of training, and the starting position N is reselected 0 , proceed to the next scene of training.

[0074] S6: After each scene of training is completed, the training situation of the scene is accumulated as an experience data and stored in the experience playback pool. The content of each stored experience data includes (s, a, R t ,s′,d), where s represents the current motion state of the robot, a represents the visual depth value estimated by the Actor network, s′ represents the new motion state of the robot as fed back by the training environment after the robot performs the action according to the visual depth value a, and d represents the flag of whether the current training is finished, d=1 if finished, and d=0 if not finished. In the process of Actor network optimization, it is necessary to extract batch_size pieces of experience data from the experience replay pool as a reference. Figure 2 and Figure 3 As shown in the figure, after the nth episode training is completed, the average of the rewards of the first n episodes is calculated. When the average value increases with the number of training episodes and the fluctuation range of the reward of each episode is within the allowable error range, it is considered that the agent network has strategic ability. t The ability to estimate the visual depth of the feature points can meet the task requirements, allowing the robot to successfully approach the target pose, and the agent network training is completed. The allowable error range mentioned here is determined based on the accuracy of the task to be completed by the robot.

[0075] The agent network of the embodiment of the present invention also includes a Critic network. During the movement of the robot arm, it is necessary not only to ensure that each step posture N t In order to ensure that the visual depth estimation of the feature points is accurate and the motion of the robot arm is smooth and free of lag, the Critic network is used to make an overall evaluation of the estimated visual depth of the feature points and the corresponding spatial state of the robot arm. The Critic network includes two data paths, one path corresponding to the pose N t , that is, the motion path, the pose path requires input pose N t The corresponding joint velocity data; the other path corresponds to the spatial state of the robot arm. The spatial state data is sensed and acquired by the intelligent agent, including: feature point error, robot arm joint angle q t and pose N t-1The visual depth estimation of the feature point at the target position is obtained by calculating the error of the feature point, where the feature point error is the error between the feature point coordinates in the target image and the feature point coordinates in the target image at the target position of the robot arm. The Critic network combines the data of the two paths and outputs the action value Q, which can reflect the evaluation ability of the Critic network. In particular, the Critic network inside the SAC algorithm includes Q network 1 and Q network 2, and both Q network 1 and Q network 2 can output the action value Q. In order to prevent the Critic network from overestimating the action value Q, the minimum function min is used to take the smaller value of the action value output by Q network 1 and Q network 2 as the final action value Q of the Critic network. Temporal difference is another function that comes with the SAC algorithm. Its essence is a value function used to update Q network 1 and Q network 2. Q network 1, Q network 2 and temporal difference parameters are inherent characteristics of the SAC algorithm and belong to the prior art.

[0076] In order to verify the feasibility of the servo control method of the present invention, the following verification experiments were also conducted:

[0077] by Figure 4 The robot arm posture shown is the target posture. The coordinate systems in the figure are the inertial coordinate system, the camera model coordinate system and the robot arm end coordinate system. In the robot arm visual servo control training environment based on the deep reinforcement learning algorithm, the error norm threshold δ is set to 0.004. Figures 5 to 12 The eight groups of robot arm postures shown verify the method of the embodiment of the present invention and the model-based method. Figure 5 The pose shown corresponds to a yaw angle of +30° for the camera model. Figure 6 The yaw angle of the camera model corresponding to the pose shown is -30°. Figure 7 The posture shown corresponds to a camera model with a pitch angle of +30°. Figure 8 The pitch angle of the camera model corresponding to the posture shown is -30°, as shown in Fig.13 The angle bisector a and angle bisector b shown in the figure, the z-axis of the camera model coordinate system rotates +30° and -30° around the angle bisector b between the x-axis and the y-axis of the camera model coordinate system to the following Fig. 9 and Fig.10 The z-axis of the camera model coordinate system rotates +30° and -30° around the angle bisector a between the x-axis and the y-axis of the camera model coordinate system to the position shown in FIG. Fig.11 and Fig.12 The posture shown.

[0078] After the robotic arm training is completed, the experimental results shown in Table 1 are obtained. According to the experimental results, it can be seen that the 8 groups of postures are trained by the method of the embodiment of the present invention, and all 8 groups of postures can converge successfully. The traditional method using fixed visual depth only achieves convergence for 3 groups of postures, and the convergence error is greater than that of the method of the embodiment of the present invention. The control accuracy of the method of the embodiment of the present invention is significantly improved compared with the traditional method.

[0079] It can be seen from the experimental results that the target image error still does not converge to 0 using the method of the embodiment of the present invention, which is related to the selected feature points. The experiment selected 4 image feature points, each with 2 coordinates, so the length of the feature vector of the feature point is 8, but the number of degrees of freedom of the robotic arm is 6, resulting in data redundancy. Data redundancy can enhance the robustness of the algorithm, but it will cause the interaction matrix of the feature points to be not full rank, so it is still possible for the feature point error to fall into the null space of the interaction matrix, resulting in the final feature point error tending to be stable but not 0.

[0080] Table 1 Experimental results

[0081]

[0082] Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. Those skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

[0083] The above specific implementations of the present invention do not constitute a limitation on the protection scope of the present invention. Any other corresponding changes and modifications made based on the technical concept of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for predicting visual depth of a robotic arm based on deep reinforcement learning, wherein a camera is connected to the robotic arm, and the method is characterized in that: include: S1: Establish the camera model and the kinematic model of the robotic arm; S2: Establishing an agent network through the SAC algorithm, wherein the agent network includes an Actor network; S3: Building a physical model of the agent network; S4: Set the target position and select at least one point on the target as a feature point; S5: The training process of each scene of the robot arm training is: Randomly select the starting position N0 of the robot arm, and the joint angle corresponding to the starting position N0 is q0; Starting from the starting position N0, the motion process of the robot arm is: Set the pose N t The coordinates of the feature points and joint angles q in the corresponding target image t and joint velocity action t-1 Input the Actor network to obtain the visual depth estimate of the feature points, and calculate the robot arm joint speed action based on the visual depth estimate of the feature points t ; Action according to joint speed t The Actor network controls the robot to move to position N at the preset robot movement command time interval. t+1 , in position N t+1 At the joint angle q t and joint velocity action t Calculate the pose N t+1 The joint angle q t+1 , where the robot arm acts according to the joint speed t The movement time is the preset time interval between two movement instructions. t Indicates that the robot arm changes from position N t Movement to position N t+1 The joint speed of the robot arm is t, and the number of robot arm movement steps is T≥t>0, where T represents the upper limit of the predetermined number of robot arm movement steps. Use the following formula to calculate the evaluation value R corresponding to the visual depth estimated by the Actor network t : R t ＝r1+r2； Among them, out of fov means that the camera model cannot collect all feature points, ||e t || represents the error norm between the feature point coordinates in the target image of the tth step and the feature point coordinates in the target image of the robot at the target pose, δ represents the threshold of the error norm, and other represents the error except out of fov and ||e t In other cases except ||<δ, m represents a fraction, ||depth d (t)|| represents the second norm of the actual value of the visual depth of the feature point at the tth step calculated by the physical model, ||depth r (t)|| represents the bi-norm of the visual depth estimation of the target feature point at the tth step; This ||e t ||<δWhen the mission was successful, the screening was completed; When not all feature points appear in the target image captured by the camera model, the servo movement fails and the training of this scene ends; If the number of movement steps of the robot arm exceeds the preset upper limit T, the servo movement fails and the training of this episode ends; After each scene of training is completed, the sum of the evaluation values ​​corresponding to each step of the posture in the scene is calculated as the reward for the scene; S6: After the nth episode training is completed, the average of the rewards of the first n episodes is calculated; when the average value increases with the number of training episodes and the fluctuation range of the rewards of each episode is within the allowable error range, the agent network training is completed.

2. The method for predicting the visual depth of a robotic arm based on deep reinforcement learning according to claim 1, characterized in that: The kinematic model is established using the DH parameter method.

3. The method for predicting robot arm visual depth based on deep reinforcement learning according to claim 1, characterized in that: The camera model is established by using a perspective projection transformation method.

4. The method for predicting robot arm visual depth based on deep reinforcement learning according to claim 1, characterized in that: The method for selecting the starting position N0 of the robot arm is as follows: randomly select a robot arm position, collect the target image at the position through the camera model, and when all feature points appear in the target image, the selected position can be used as the starting position N0.

5. The method for predicting robot arm visual depth based on deep reinforcement learning according to claim 1, characterized in that: According to the feature points in the target image collected at the starting position, the joint angle q0 corresponding to the starting position N0 of the robot arm is obtained through the kinematic model.

6. The method for predicting the visual depth of a robotic arm based on deep reinforcement learning according to claim 5, characterized in that: The fraction m is set to 5.

7. The method for predicting robot arm visual depth based on deep reinforcement learning according to claim 1, characterized in that: The agent network also includes a Critic network, which senses the position N through the agent network. t The spatial state and the position N t The corresponding joint velocity is input into the Critic network, and the Critic network outputs the benefit value Q.

8. The method for predicting robot arm visual depth based on deep reinforcement learning according to claim 1, characterized in that: The pose N t The spatial state includes: feature point error, robot arm joint angle q t , the visual depth estimate corresponding to the t-1th step, wherein the feature point error is the error between the feature point coordinates in the target image and the feature point coordinates in the target image of the robot arm at the target pose.

9. The method for predicting robot arm visual depth based on deep reinforcement learning according to claim 1, characterized in that: The activation function of the Actor network is the leaky relu function, and the activation function of the Critic network is the relu function.

Images