A mechanical arm visual servoing control method based on deep reinforcement learning

Abstract

The present application relates to the technical field of mechanical arm visual servo control, and specifically provides a kind of mechanical arm visual servo control method based on deep reinforcement learning, and the specific steps are as follows: setting target position and selecting feature points on target, randomly selecting the starting pose of mechanical arm, and gradually moving to target pose, the joint angle of current pose is calculated according to the joint speed and pose of the last step movement of mechanical arm, and the joint speed of next step movement is selected by Actor network according to the joint speed of last step movement, the joint angle of current pose and feature point error;For each step movement, evaluation function will give evaluation value, to evaluate and improve the ability of Actor network to select joint speed, and the reward is calculated according to the evaluation value, when the average value of reward increases and the trend of each episode reward tends to be stable, the overall training is completed.The mechanical arm servo control method provided by the present application does not depend on training data, can realize wide range of servo tasks, and improves the servo control precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mechanical arm visual servo control, and specifically provides a mechanical arm visual servo control method based on deep reinforcement learning. BACKGROUND

[0002] Mechanical arm visual servo control is a comprehensive technology integrating machine vision, automatic control and computer technology, which realizes the servo motion control of the mechanical arm through visual signals, can effectively improve the motion precision and robustness of the mechanical arm, and enhances the interaction ability of the mechanical arm with the complex environment.

[0003] According to different control methods, the mechanical arm visual servo control method can be divided into two kinds of model-based control method and deep learning-based control method, the model-based method is to select image features according to the task first, then calculate the interaction matrix of the image features, and finally design a suitable control rate according to the interaction matrix, this method has a very serious dependence on the image features and the corresponding interaction matrix of the image features, and due to the strong nonlinear relationship between the visual features and the camera speed, the convergence domain is small, and global convergence cannot be realized; the deep learning-based method is to design a neural network, then learn and train the deep neural network through the collected data samples (image features and camera speed), and finally test the trained deep learning algorithm through the new initial position of the mechanical arm, this method trains the mechanical arm with the training data obtained, and is seriously dependent on the training data, and the training data itself is difficult to obtain, and there are many difficulties in implementation. Therefore, how to solve the wide range of mechanical arm visual servo motion control and reduce the dependence on the model and the learning data has important significance for the mechanical arm visual servo control. SUMMARY

[0004] To solve the above problems, the present application provides a mechanical arm visual servo control method based on deep reinforcement learning, an agent network is established based on the SAC algorithm, and the joint speed is selected and executed by using the Actor network, the training process is evaluated by establishing an evaluation function, and the ability of the Actor network to select the joint speed is improved. The method of the present application does not need existing training data to support, can realize wide range convergence, and improves the servo control precision of the mechanical arm.

[0005] The present application provides a mechanical arm visual servo control method based on deep reinforcement learning, which comprises:

[0006] S1: establishing a camera model and a kinematics model of the mechanical arm;

[0007] S2: establishing an agent network by using the SAC algorithm, the agent network comprising an Actor network;

[0008] S3: setting a target position, selecting at least one point on the target as a feature point;

[0009] S4: training the agent network, the training process of each episode being:

[0010] randomly selecting a start pose of the robot arm , the start pose corresponding to joint angles ;

[0011] starting from the start pose , the movement process of the robot arm being:

[0012] the Actor network selecting a joint velocity corresponding to the next movement of the robot arm according to the joint velocity , the joint angles corresponding to the pose , and a feature point error, and controlling the robot arm to move to the pose according to the joint velocity , calculating the joint angles corresponding to the pose according to the joint angles and the joint velocity , wherein the time for the robot arm to move according to the joint velocity is a preset time interval between two movement instructions, denotes the movement step number of the robot arm, , denotes the upper limit of the preset movement step number of the robot arm, and the feature point error is the error between the feature point coordinates in the target image and the feature point coordinates in the target image of the target at the target pose, denotes the joint velocity for the robot arm to move from the pose to the pose ;

[0013] the evaluation value of the joint velocity selected by the Actor network is calculated by using the following formula:

[0014] ;

[0015] ;

[0016] ;

[0017] ;

[0018] ;

[0019] wherein, This represents the first gain coefficient. This represents the second gain coefficient. This represents the third gain coefficient. Indicates the first The L2 norm of the error between the coordinates of feature points in the target image and the coordinates of feature points in the target image at the target pose of the robotic arm. Indicates the first The energy consumed by walking. Representing fractions, This indicates that the camera model could not capture all feature points. The threshold representing the error norm 2, Indicates except and Other than these;

[0020] when When the servo motion is successful, the training for that scene ends.

[0021] If not all feature points appear in the target image captured by the camera model, the servo motion fails, and the training for that scene ends.

[0022] If the number of steps the robotic arm takes exceeds the predetermined upper limit Servo motion failed, training for this scene ends;

[0023] After each training session, the sum of the evaluation values ​​corresponding to the joint velocity and pose at each step in that session is calculated and used as the reward for that session.

[0024] S5: No. After the training is completed, the calculation is performed before... The average value of the reward for each training session; when the average value increases with the number of training sessions and the fluctuation range of the reward for each session is within the allowable error range, the training of the agent network ends.

[0025] Preferably, the DH parameter method is used to establish the kinematic model of the robotic arm.

[0026] Preferably, the camera model is established using the perspective projection transformation method.

[0027] Preferably, the starting pose of the robotic arm The selection method is as follows: Randomly select a pose for the robotic arm, and acquire a target image at that pose using a camera model. When all feature points appear in the target image, the selected pose can be used as the starting pose. .

[0028] Preferably, based on the starting pose The starting pose is obtained through inverse kinematics model. Corresponding joint angles .

[0029] Preferably, the score is 5.

[0030] Preferably, the intelligent agent network further comprises a Critic network, and the Critic network is used to perceive a spatial state of the pose by the intelligent agent network, and the spatial state and the pose are input into the Critic network, and the Critic network outputs an action value Q of the joint speed.

[0031] Preferably, the pose comprises a feature point error, a joint angle of the robot arm, and a joint speed of the robot arm corresponding to the first step.

[0032] Preferably, an activation function of the Actor network is a leaky relu function, and an activation function of the Critic network is a relu function.

[0033] Compared with the prior art, the present application can achieve the following beneficial effects:

[0034] The present application establishes an intelligent agent network based on the SAC algorithm, and trains the intelligent agent network. Compared with the control method based on deep learning, the servo control method of the present application selects and executes the joint speed through the Actor network, automatically trains, and establishes an evaluation function to evaluate the training process. The Actor network improves the ability of joint speed selection according to the evaluation result, so that the training data does not need to be collected to train the intelligent agent network in the training process, and the training data has no dependency, making the training easier to perform.

[0035] Compared with the model-based control method, the servo control method of the present application can randomly select the target and the feature point, does not need to select the image feature according to the task, and does not need to design a suitable control rate according to the interaction matrix of the image feature, so that the convergence domain of the servo task can be greatly improved, global convergence can be achieved, the local minimum value phenomenon does not occur in the method of the present application, there is no steady-state control error, and the servo control precision can be improved.

[0036] The method of the present application further designs a reward function to evaluate the joint speed selected by the Actor network. The reward function fully considers the error, energy, speed and success of each action of the robot arm, so as to ensure that each joint speed can be fully and comprehensively evaluated. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a schematic diagram of the intelligent agent network training process provided by the embodiment of the present application;

[0038] Figure 2is a curve diagram of the average reward value changing with the number of training scenes according to an embodiment of the present application;

[0039] Figure 3 is a curve diagram of the reward of each scene changing with the number of scenes according to an embodiment of the present application;

[0040] Figure 4 is a schematic diagram of a target pose selected in a verification experiment according to an embodiment of the present application;

[0041] Figure 5 is a schematic diagram of a first start pose selected in a verification experiment according to an embodiment of the present application;

[0042] Figure 6 is a schematic diagram of a second start pose selected in a verification experiment according to an embodiment of the present application;

[0043] Figure 7 is a schematic diagram of a third start pose selected in a verification experiment according to an embodiment of the present application;

[0044] Figure 8 is a schematic diagram of a fourth start pose selected in a verification experiment according to an embodiment of the present application;

[0045] Figure 9 is a schematic diagram of a fifth start pose selected in a verification experiment according to an embodiment of the present application;

[0046] Figure 10 is a schematic diagram of a sixth start pose selected in a verification experiment according to an embodiment of the present application;

[0047] Figure 11 is a schematic diagram of a seventh start pose selected in a verification experiment according to an embodiment of the present application;

[0048] Figure 12 is a schematic diagram of an eighth start pose selected in a verification experiment according to an embodiment of the present application;

[0049] Figure 13 is a schematic diagram of an angle bisector of a camera model coordinate system in a verification experiment according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the following description, the same modules are denoted by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated.

[0051] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not constitute a limitation on the present application.

[0052] As shown in Figure 1 The embodiment of the present application provides a mechanical arm visual servo control method based on deep reinforcement learning, comprising the following steps:

[0053] S1: A camera is connected to the mechanical arm, the camera can move following the action of the mechanical arm, and the field angle of the camera corresponding to different mechanical arm poses is also different. A camera model is established, the camera model is a virtual camera, and data information of the object to be collected is obtained instead of pictures.

[0054] A kinematic model of the mechanical arm is established, and the mechanical arm pose data is input into the kinematic model, so that the corresponding mechanical arm joint angle can be obtained.

[0055] The embodiment of the present application establishes the kinematic model of the mechanical arm by using the D-H parameter method, and establishes the camera model by using the perspective projection transformation method, both of which are mature prior art and easy to apply, but the establishment method of the kinematic model and the camera model is not limited to the D-H parameter method and the perspective projection transformation method.

[0056] S2: The embodiment of the present application establishes an agent network of the mechanical arm based on a SAC algorithm with adaptive temperature factor, the agent network comprises an Actor network and a Critic network, in the process of training the agent network, the joint speed of each step of the mechanical arm is selected by the Actor network, the temperature factor can change in real time following the training process, and then the Actor network is adjusted. In the embodiment of the present application, the activation function of the Actor network is a leaky relu function, and the activation function of the Critic network is a relu function, the activation function is a necessary function in the SAC algorithm, and the leaky relu function and the relu function are both prior art.

[0057] S3: After the agent network is established, the agent network is trained. A target is selected, and at least one representative point on the target is selected as a feature point of the target. In the process of training the agent network, the feature point is taken as a reference to measure the movement of each step of the mechanical arm. The selection of the target is irrelevant to the type of task performed by the mechanical arm, and the target can be any complete, characteristic and position-specific pattern.

[0058] S4: After determining the target position and feature points, select a pose for the robotic arm near the target. At this pose, the camera model acquires an image of the target. This target image has identical geometric features to the actual target; this pose is called the target pose for the robotic arm movement. For example, if a square is selected as the target, and the four corner points of the square are used as feature points, when the robotic arm is at a certain distance directly above the square, the target (square) image acquired by the camera model is still a square, and the target does not undergo any deformation in the target image. Then, the pose of the robotic arm at this moment is the target pose. The target pose is a common concept in the field of robotic arm training; it is the ideal position that the robotic arm training aims to achieve, which is manually set.

[0059] The purpose of robotic arm servo control training is to control the robotic arm to reach the target pose. The complete robotic arm servo control training process includes multiple training sessions, each of which is as follows:

[0060] A random pose of the robotic arm is selected, and a target image is acquired from that pose using a camera model. If not all feature points are captured in the target image, the pose is discarded, and a new pose is selected. If the acquired target image contains all feature points, that pose is used as the starting pose for training that scene. According to the starting pose The starting pose is obtained by inverse kinematics of the robotic arm. The corresponding joint angle is When the robotic arm performs its first movement, the reference data comes from the starting pose.

[0061] Successfully selected starting pose Then, during the movement of the robotic arm from the starting pose to the target pose, the Actor network determines the movement based on the robotic arm's pose. Exercise posture joint speed , pose Corresponding joint angles The joint velocities of the robotic arm for each step of movement are selected one by one based on the feature point error. Feature point error is the error between the coordinates of feature points in the target image and the coordinates of feature points in the target image at the target pose of the robotic arm. The joint velocity is selected as the reference point. The Actor network then controls the robotic arm according to the selected joint speed. Movement reaches position Here, the robotic arm operates according to joint speed. The time of movement is the time interval of two adjacent movement instructions, and the time interval of the movement instruction is a preset value before the robot arm starts to complete the task. For example, if a user wants the robot arm to move 50 times per second, the robot arm needs to be sent a movement instruction every 20 ms, and the movement time of each step of the robot arm is 20 ms. Therefore, as long as the joint speed of the robot arm is obtained , the corresponding pose of the robot arm after completing this step of movement can be obtained in combination with the instruction time interval . When the pose is reached, the joint angle , the joint speed and the time interval of two adjacent movements of the pose are used to calculate the corresponding joint angle of the pose , and the joint angle is the basis for the Actor network to select the joint speed of the next step of movement of the robot arm, wherein represents the number of movement steps of the robot arm, , represents the upper limit of the predetermined number of movement steps of the robot arm, represents the joint speed of the robot arm moving from the pose to the pose , and when , the joint speed of the first step of movement of the robot arm is selected by the Actor network according to the joint angle of the starting pose and the joint speed of the movement of the robot arm two steps before the starting pose , but there is no movement of the robot arm two steps before this node, so the speed value of the joint speed used at this time is 0.

[0062] In order to ensure that the pose of each step of movement of the robot arm is closer and closer to the target pose to complete the task, the present embodiment designs a reward function, and the joint speed selected by the Actor network is evaluated once every time the robot arm reaches a pose . The reward value of the joint speed is calculated by the reward function, and the SAC algorithm includes a policy loss parameter, which is essentially a target function. In combination with the reward value, the target function is calculated inside the SAC algorithm, and the ability of the Actor network to select the joint speed is optimized. The calculation formula of the reward value is as follows:

[0063] ;

[0064] ;

[0065] ;

[0066] ;

[0067] ;

[0068] in, This represents the first gain coefficient. This represents the second gain coefficient. This represents the third gain coefficient. Indicates the first The L2 norm of the error between the coordinates of feature points in the target image and the coordinates of feature points in the target image at the target pose of the robotic arm. Indicates the first The energy consumed by walking. Indicates the first The speed of movement of the step, Representing fractions, This indicates that the camera model could not capture all feature points. The threshold representing the error norm 2, Indicates except and In other cases, here The value should be determined based on the accuracy range of the robotic arm's task.

[0069] In the formula for calculating reward value, The error in each step of the robotic arm's movement was introduced. The motion speed of the robotic arm at each step was introduced. The energy consumption of each step of the robotic arm's movement was introduced. The results of each step of the robotic arm's movement are incorporated, and the reward value calculation formula fully considers various factors in the robotic arm's movement process, making the evaluation more comprehensive and sufficient, and the evaluation results more meaningful.

[0070] In the reward value calculation formula In the middle, fractions Used to reward successful actions and punish unsuccessful actions, the score can be adjusted in conjunction with the values ​​of other parameters in the calculation formula. Values ​​are selected to prevent the calculated reward value from being too large or too small, which could lead to difficulties in calculation. This embodiment of the invention uses scores... The value is 5.

[0071] Based on the evaluation values, the SAC algorithm will automatically adjust parameters such as the temperature factor, thereby changing the Actor network and improving its ability to select joints quickly.

[0072] when At that time, it indicates the position reached by the robotic arm. The error L2 relative to the target pose at the threshold Within the range, the servo motion was successful, and the training for this scene ended.

[0073] When in position If not all feature points appear in the target image captured by the camera model, the target has left the camera model's field of view, making it impossible to calculate the current pose using the kinematic model. With the corresponding joint angle, the Actor network is unable to select the next joint speed of the robotic arm, the servo motion fails, and the training for this scene ends.

[0074] When the robotic arm is completing a task, it cannot move indefinitely. Therefore, a maximum number of steps is preset for each training session. If the number of steps the robotic arm moves exceeds the predetermined upper limit... Servo motion failed, training for this scene ends.

[0075] After each training session, sum all the evaluation values ​​obtained in that session to obtain the corresponding reward, and then select a new starting pose. Then proceed to the next training session.

[0076] S5: After each training session is completed, the training status of that session is accumulated as experience data and stored in the experience replay pool. Each piece of experience data includes... ,in, This indicates the current state of the robotic arm's movement. This represents the robotic arm motion corresponding to the joint velocities selected by the Actor network. Indicates the robotic arm performing an action Subsequently, the training environment provides feedback on the state of the robotic arm's movement in this new action. A marker indicating whether the current training has ended; when it ends... Before it ended During the Actor network optimization process, the number of [items] to be extracted from the experience replay pool is [number]. The empirical data from the first article is used as a reference. After the training is completed, the calculation is performed before... The average value of the reward, such as Figure 2 and Figure 3 As shown, when the average value increases with the number of training sessions and the fluctuation range of the reward per session is within the allowable error range, the agent network is considered to have policy capability, and the pose reached by the joint velocity of each selected movement step is considered to be... If all attempts successfully approach the target pose, the agent network training is complete. The allowable error range mentioned here is determined based on the precision required for the task to be performed by the robotic arm.

[0077] The intelligent agent network in this embodiment of the invention also includes a Critic network. During the movement of the robotic arm, it is necessary not only to ensure the accurate selection of joint velocities for each step of the robotic arm's movement, but also to ensure that the robotic arm's movement is smooth and without any jerks. Therefore, the Critic network is applied to perform an overall evaluation of the joint velocities and corresponding spatial states of the robotic arm. The Critic network includes two data paths, one corresponding to the pose. In other words, the motion path and pose path require the pose input. The corresponding joint velocity data; the other path corresponds to the spatial state of the robotic arm, which is obtained by the agent through perception. Specific data includes: feature point errors, robotic arm joint angles. and the The step corresponds to the joint velocity of the robotic arm, where the feature point error is the error between the coordinates of the feature points in the target image and the coordinates of the feature points in the target image at the target pose of the robotic arm. The Critic network combines the data from the two paths to output the action value Q, and the reward value Q reflects the evaluation ability of the Critic network. Specifically, the Critic network inside the SAC algorithm includes Q network 1 and Q network 2. Both Q network 1 and Q network 2 can output action values ​​Q. To prevent the Critic network from overestimating the action value Q, the smaller of the action values ​​output by Q network 1 and Q network 2 is taken as the final action value Q of the Critic network through the minimization function min. Temporal difference is another function inherent to the SAC algorithm, which is essentially a value function used to update Q network 1 and Q network 2. Q network 1, Q network 2, and temporal difference parameters are inherent characteristics of the SAC algorithm and belong to existing technology.

[0078] To verify the feasibility of the servo control method of the present invention, the following verification experiments were also conducted:

[0079] by Figure 4 The robot arm pose shown is the target pose. The coordinate systems in the figure are the inertial coordinate system, the camera model coordinate system, and the robot arm end effector coordinate system, respectively. A threshold for the error L2 norm is set. The value is 0.001, and the selection is as follows. Figures 5 to 12 The eight sets of robotic arm poses shown are used to verify the method of the embodiments of the present invention and the model-based method. Figure 5 The yaw angle of the camera model corresponding to the pose shown is +30°. Figure 6 The yaw angle of the camera model corresponding to the pose shown is -30°. Figure 7 The pose shown corresponds to a camera model pitch angle of +30°. Figure 8 The pose shown corresponds to a camera model with a pitch angle of -30°, as... Figure 13The angle bisector a and the angle bisector b are shown, and the z axis of the camera model coordinate system is rotated by +30° and -30° around the angle bisector b between the x axis and the y axis of the camera model coordinate system, respectively, to the pose as shown in FIG. 3B. Figure 9 and Figure 10 The angle bisector a and the angle bisector b are shown, and the z axis of the camera model coordinate system is rotated by +30° and -30° around the angle bisector a between the x axis and the y axis of the camera model coordinate system, respectively, to the pose as shown in FIG. 3B. Figure 11 and Figure 12 The angle bisector a and the angle bisector b are shown, and the z axis of the camera model coordinate system is rotated by +30° and -30° around the angle bisector a between the x axis and the y axis of the camera model coordinate system, respectively, to the pose as shown in FIG. 3B.

[0080] After the mechanical arm training is completed, the experimental results shown in Table 1 are obtained. According to the experimental results, compared with the traditional model-based training method, the 8 groups of poses can be successfully converged by using the method of the embodiment of the present application, and only 3 groups of poses can be converged by using the traditional model-based training method. Moreover, compared with the traditional model-based training method, the control accuracy of the embodiment of the present application is about four times that of the traditional method, and the control accuracy is obviously greatly improved.

[0081] Table 1 Experimental results

[0082]

[0083] Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

[0084] The specific embodiments of the present application do not constitute a limitation on the scope of protection of the present application. Any various other corresponding changes and modifications made according to the technical concept of the present application should be included in the scope of protection of the claims of the present application.

Claims

1. A method for visual servoing control of a robot arm based on deep reinforcement learning, wherein a camera is connected to the robot arm, and the method comprises the steps of: The control method comprises: ​ S1: establishing a camera model and a kinematics model of a mechanical arm; S2: establishing an agent network through an SAC algorithm, the agent network comprising an Actor network; S3: setting a target position and selecting at least one point on the target as a feature point; S4: training the agent network, and a training process of each episode being as follows: Randomly select the starting pose of the robot arm , the starting pose corresponding joint angle is ; From the start pose The movement process of the mechanical arm is as follows: The Actor network selects the joint velocity corresponding to the next motion of the robot arm according to the joint velocity , the pose , the corresponding joint angle , and the feature point error , and controls the robot arm to move to the pose according to the joint velocity , calculates the pose corresponding to the joint angle and the joint velocity , wherein the time for the robot arm to move according to the joint velocity is the preset time interval of the two motion instructions, represents the motion step number of the robot arm, , , represents the upper limit of the predetermined motion step number of the robot arm, and the feature point error is the error between the feature point coordinates in the target image and the feature point coordinates in the target image of the robot arm at the target pose, represents the joint velocity of the robot arm moving from the pose to the pose . The joint velocity selected by the actor network is calculated using the following equation :​ ； ； ； ； ； wherein, denotes a first gain coefficient, denotes a second gain coefficient, denotes a third gain coefficient, denotes a fourth gain coefficient, denotes a threshold value for the error two-norm, denotes a threshold value for the error two-norm, denotes the energy consumed by the movement, denotes a score, denotes that the camera model does not capture all feature points, denotes a threshold value for the error two-norm, denotes other cases than and . When the servo motion is successful, the curtain training ends; when feature points in a target image collected by the camera model do not all appear, servo motion fails, and the episode training ends; If the number of motion steps of the robot arm exceeds a predetermined upper limit , the servo motion fails, and the scene training ends. after each episode training ends, a sum of evaluation values corresponding to joint speed poses of each step in the episode is calculated as a reward of the episode; S5: No. After the training is completed, the calculation is performed before... The average value of the training session reward; when the average value increases with the number of training sessions and the fluctuation range of the reward for each session is within the allowable error range, the training of the agent network ends. 2.The deep reinforcement learning based visual servoing control method for a robot arm according to claim 1, wherein, the kinematics model of the mechanical arm is established by using a D-H parameter method. 3.The deep reinforcement learning based visual servoing control method for a robot arm according to claim 1, wherein, the camera model is established by using a perspective projection transformation method. 4.The deep reinforcement learning based visual servoing control method for a robot arm according to claim 1, wherein, The starting pose of the mechanical arm The selection method is as follows: randomly selecting a mechanical arm pose, collecting the target image at the pose through the camera model, and when all feature points appear in the target image, the selected pose is taken as the starting pose .

5. The deep reinforcement learning-based visual servoing control method of claim 4, wherein, According to the start pose , the start pose is obtained by inverse kinematics model Corresponding joint angle .

6. The deep reinforcement learning-based visual servoing control method of claim 1, wherein, The score is 5.

7. The deep reinforcement learning based visual servoing control method for a robot arm of claim 1, wherein, The agent network also includes a Critic network, which perceives the pose through the agent network. The spatial state, and the pose The corresponding joint velocity is input into the Critic network, and the Critic network outputs the motion value Q.

8. The deep reinforcement learning-based visual servoing control method of claim 7, wherein, The pose The spatial state includes: feature point error, mechanical arm joint angle , the first Step corresponding to the mechanical arm joint speed.

9. The deep reinforcement learning-based visual servoing control method of claim 7, wherein, an activation function of the Actor network is a leaky relu function, and an activation function of the Critic network is a relu function.

Images