Robotic arm control method based on deep reinforcement learning

Abstract

The application discloses a mechanical arm control method based on deep reinforcement learning, which comprises the following steps: a servo controller is designed, a mechanical arm visual servo system is built, an adaptive servo gain DDPG model is trained, a DDPG training environment is built, DDPG training parameters are set and DDPG model training is completed, expected images when the mechanical arm grabs a target object are set and acquired, and expected features are extracted, current target images are acquired, current features are extracted, an extended Kalman filtering method is used to estimate a depth value, the depth value is substituted into calculation of an image Jacobian matrix generalized inverse matrix, meanwhile, expected features and current features are operated to obtain image feature error e, according to a current state of the mechanical arm visual servo system, an adaptive servo gain algorithm based on deep deterministic policy gradient DDPG which is trained is used to determine a servo gain value, and the servo controller is used to perform servo control on the mechanical arm. The application has the characteristics of stronger stability, higher robustness and faster convergence speed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, computer vision and artificial intelligence, and particularly relates to a mechanical arm control method based on deep reinforcement learning. BACKGROUND

[0002] Visual servoing is a servo control method that uses visual information to quickly control the pose of a robot or camera to the desired position. When combined with a robot, this method can enable it to perform complex tasks in unstructured environments. Therefore, visual servoing control is attracting more and more attention in various fields. The application of visual servoing in many fields is to pursue the demand for high precision and robustness, such as medical surgery, plant seedling cutting, lunar sample container clamping, automatic positioning of hoisting cranes, rotor flight mechanical arm systems, etc. A camera is usually used as a visual sensor, and a system that uses a visual sensor in feedback is called a visual servoing system. Visual servoing is divided into three different categories according to the composition of the control system: 1) position-based visual servoing (PBVS), which extracts visual features from images to estimate the three-dimensional pose of the camera relative to the target, i.e., reconstructing a three-dimensional model from two-dimensional data; 2) image-based visual servoing (IBVS), which directly determines the control input on the two-dimensional image plane according to the image feature coordinate error; 3) hybrid visual servoing (HVS), which combines three-dimensional PBVS and two-dimensional IBVS.

[0003] In practice, there is inevitably a feature loss problem due to the limited field of view (FOV) of the camera. Both IBVS and PBVS have the problem of target loss, which leads to the failure of visual servoing if the tracked object is lost from the camera's field of view. In the prior art, there are mainly the following three solutions, the first of which is to avoid feature loss through long-FOV trajectory planning or optimization. Researchers use an MPC-based method considering visibility constraints to optimize the feature trajectory in the image frame. The method used by the researchers is not to optimize the feature trajectory in the image, but to plan the path of the camera in three-dimensional Cartesian space to improve the servo efficiency while avoiding feature loss. Although the above method has strong robustness due to long-FOV trajectory optimization, it has a heavy computational burden, which limits its application scenarios, especially real-time tasks. The second solution focuses on ensuring the next state visibility constraint, and researchers design a control law based on potential function to push the feature back to the safe zone when it enters the danger zone. Although this method has no computational burden, it requires careful manual design of the size of the potential field and the shape of the potential function. The third strategy uses a reinforcement learning algorithm to autonomously learn the strategy to complete the servo task and avoid feature loss. With the increasing exploitation of the functions of reinforcement learning, learning-based methods have become increasingly popular in recent years. Researchers use the Dyna-Q learning algorithm to design an adaptive law to adjust the gain of IBVS, which not only guarantees the visibility constraint but also improves the servo efficiency. Although RL algorithms have been tried to avoid target feature loss, it is found that the most popular is the Q-learning algorithm, which has many drawbacks, mainly that the action space considered by the algorithm is discrete. Unlike the DDPG algorithm, which can directly handle continuous action spaces, the use of DRL in robotic arm visual servoing control remains to be explored. In summary, the above visual servoing control systems have problems such as slow servo convergence, poor precision, and easy feature loss. SUMMARY

[0004] The present application aims to overcome the above-mentioned drawbacks and proposes a robotic arm control method based on deep reinforcement learning with stronger stability, higher robustness, and faster convergence speed.

[0005] The robotic arm control method based on deep reinforcement learning of the present application comprises the following steps:

[0006] S1: Build a robotic arm visual servoing system, set and obtain the desired image captured by the camera when the robotic arm grasps the target object, and extract the desired features;

[0007] The robotic arm visual servoing system comprises a seven-degree-of-freedom robotic arm, a controller, and a camera integrated on the end effector of the robotic arm:

[0008] S2: initialize the state of the robot arm and ensure that the target object is in the camera view at this time;

[0009] S3: obtain the current target image and extract the current features;

[0010] S4: estimate the depth value using the extended Kalman filter method, substitute it into the calculation of the image Jacobian matrix generalized inverse matrix, and calculate the image feature error e by operating the expected features and the current features;

[0011] S5: according to the current state of the robot visual servo system, an adaptive servo gain algorithm based on depth deterministic policy gradient DDPG is used to determine the servo gain value, and the process is as follows:

[0012] S51: define the adaptive servo gain DDPG model parameters;

[0013] The definition of adaptive servo gain DDPG model parameters includes:

[0014] (1) Selection of state space: selecting the camera original speed as the state space,

[0015] The selection of the camera original speed as the state space can be represented as:

[0016]

[0017] Where S is the state space, s is the state space sub-element, is the camera original speed, e is the image feature error, is the generalized inverse matrix of the interaction matrix .

[0018] (2) Action space selection: directly selecting the servo gain as the action space;

[0019] (3) Reward function design: design a neural network to simulate the correction term of the artificial reward function on the basis of artificial experience, wherein the input of the neural network is the current action and state, and the output of the neural network is the correction term r of the reward function c ; The reward function r is represented as r=r e +r c .

[0020] Where r e is an artificially designed reward function, represented as:

[0021]

[0022] Where R is the maximum artificial reward value, which is a positive number; κ is the number of image feature points; rn is the height of the image plane, cn is the width of the image plane, p i is the current feature point, pid To the desired feature point;

[0023] The neural network parameters are supervisedly updated according to the feature error change rate and the average radiation distance between the feature point and the center of the field of view, and the correction term mainly plays the role of buffering the empirical error.

[0024] S52: design an adaptive servo gain algorithm based on DDPG, the specific steps are as follows:

[0025] Input: randomly initialize servo gain network parameters θ Q , θ μ , initialize the target network by copying the current network: θ Q’ ← θ Q , θ μ’ ← θ μ , initialize the noise generator as OU t ; repeat the execution number C, the total number of training rounds n;

[0026] Output: optimal servo gain network parameters and optimal strategy;

[0027] Step 1, randomly initialize the pose of the robot arm, and generate the initial state s0 by the state generator;

[0028] Step 2, select the servo gain K t =a t =β(s t )=μ(s t |θ μ )+OU t , where s t is the current state, a t is the current action, β is the state generator, and μ is the action policy function;

[0029] Step 3, calculate the camera speed V t according to the servo controller SMCC-IBVS of the servo gain K c , and execute the camera speed V c according to the robot arm controller, and observe the current reward r et designed by artificial and the next state value s t+1 generated by the state generator;

[0030] The servo controller SMCC-IBVS: introduce a proportional controller into the image-based visual servoing C-IBVS, reduce the servo error in an exponential manner through the proportional controller, in order to improve the stability of the system, the sliding mode control SMC is used to compensate the instability of the system, an enhanced method combining proportional control and sliding mode control SMC is proposed, the image-based visual servoing controller is designed according to the control law formula, which is called servo controller SMCC-IBVS;

[0031] The control law formula is:

[0032]

[0033] Wherein, V c is the camera speed; K p is a fixed non-negative number and the product value of the unit matrix, that is, the proportional servo gain value; K sm is a fixed non-negative number and the product value of the unit matrix, that is, the sliding mode control gain value; ∈R 6×8 represents the generalized inverse matrix of the interaction matrix L p ; sm represents the sliding surface; sat(·) is a saturation function;

[0034] The saturation function formula is:

[0035]

[0036] Wherein, sgn(·) represents the sign function

[0037] Step 4, combine the correction term output by the neural network to obtain the current reward value r t , update the neural network parameters in a supervised manner;

[0038] Step 5, store the four-tuple (s t ,a t ,r t ,s t+1 ) into the replay buffer M;

[0039] Step 6, if the next state s t+1 is a terminal state, terminate this training, and execute step 8;

[0040] Step 7, if the buffer M is full, randomly select N sample four-tuples (s t ,a t ,r t ,s t+1 ) from it, respectively train the critic and actor neural networks, and update the target network parameters;

[0041] Step 8, return to step 2, repeat C times.

[0042] Step 9, detect the number of training rounds, if the total number of training rounds does not reach n, return to step 1; otherwise, exit;

[0043] S53: Train the adaptive servo gain DDPG model: build a DDPG training environment; set DDPG training parameters;

[0044] The DDPG training environment is built: the adaptive servo gain of DDPG uses Peter Corke's visual servo toolbox to design an IBVS controller, uses MATLAB's reinforcement learning toolbox to build a reinforcement learning algorithm, and adds a modified neural network on this basis. The camera in the simulation has the same internal parameters as the camera used in the physical world.

[0045] The DDPG training parameters are set: the training process DDPG contains two hidden layers, and the number of neural network units in the first and second layers of the hidden layer is 60 and 40 respectively; the actor Actor current network and the actor target network receive a six-dimensional state s as input and output an action a; the critic Critic current network and the critic target network receive a state-action pair (s, a) as input and output a one-dimensional action value function Q(s, a). The activation function of the neural network in the hidden layer is set to the Relu function; while the activation function of the output layer of the actor Actor current network and the actor target network adopts a modified Tanh function to ensure that the proportional gain parameter value k p Output range k p ∈(0,2.0], the sliding mode gain parameter value k sm Output range k sm ∈(0,3.0]; in each round of training, randomly select N=64 samples from the retransmission buffer M to train the actor and critic neural networks respectively; the learning rate of the actor Actor and critic Critic network is set to 0.0001, and the update rate of the two target networks is set to 0.001; the artificial maximum reward value R is set to 1.5, the hidden layer neural network unit number of the correction term neural network is h=20, and the activation function of the output layer also adopts the Tanh function to ensure that the output range r c ∈[-1.5,1.5], the learning rate is set to 0.001 during training.

[0046] S6: servo control is carried out on the mechanical arm: the image feature error, the image Jacobian matrix generalized inverse matrix and the servo gain value are input into the servo controller SMCC-IBVS to generate a mechanical arm control signal to control the movement of the mechanical arm, and if the target pose has not been reached, the step S3 is returned to.

[0047] Compared with the prior art, the present application has obvious beneficial effects. As can be seen from the above scheme, the present application introduces a proportional controller into the image-based visual servoing (C-IBVS), reduces the servo error in an exponential manner through the proportional controller, compensates for the instability of the system by using the sliding mode control SMC, and proposes a SMCC-IBVS visual servoing controller design that combines proportional control and sliding mode control. The process of selecting the servo gain is constructed as a Markov decision process (MDP) model, and on this basis, an adaptive servo gain algorithm based on deep deterministic policy gradient (DDPG) is designed to adaptively adjust the servo gain of the controller (SMCC-IBVS), reduce the servo error, and improve the stability. In summary, the present application adaptively adjusts the servo gain of the controller (SMCC-IBVS) through deep reinforcement learning, accelerates the convergence speed, improves the servo precision and stability, avoids feature loss, and provides a control method with stronger stability, higher robustness and faster convergence speed for intelligent control of the mechanical arm.

[0048] The beneficial effects of the present application are further described below through specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 The flow structure diagram of the present application is shown in the figure;

[0050] Figure 2 The neural network diagram of the present application is shown in the figure;

[0051] Figure 3 The flowchart of the adaptive servo gain algorithm based on DDPG of the present application is shown in the figure;

[0052] Figure 4 The training data representation diagram of the present application is shown in the figure;

[0053] Figure 5 The embodiment diagram of the present application is shown in the figure. DETAILED DESCRIPTION

[0054] The specific embodiments, features and effects of a mechanical arm control method based on deep reinforcement learning according to the present application are described in detail as follows in combination with the preferred embodiments and the drawings.

[0055] Referring to Figure 1 A mechanical arm control method based on deep reinforcement learning of the present application includes the following steps:

[0056] S1: Build a mechanical arm visual servo system in eye-in-hand configuration, the visual servo system: composed of a seven degrees of freedom mechanical arm, the end effector integrated with a camera, wherein the trained deep deterministic policy gradient (DDPG) algorithm is responsible for generating the appropriate servo gain required in each state in the whole system, and then the hybrid controller generates the running speed of the mechanical arm end camera, i.e. the end effector at the next time, the robot determines the running state of each joint according to the end speed to control the robot motion, and then the system collects the next feature image, generates the image Jacobian matrix generalized inverse matrix and the current feature error, which are input to the state generator to generate the state at this moment for the next DDPG algorithm to determine the servo gain, and the two are also used in the hybrid controller to generate the final camera speed sent to the joint controller. The system will also optimize the simulation of the trained adaptive servo gain DDPG algorithm. Due to the certain gap between simulation and reality, the simulation trained DDPG is transplanted to the actual object and further trained in the actual scene to make up for the difference between simulation and reality, so as to achieve the optimization goal of DDPG algorithm.

[0057] S2: Obtain the desired image taken when the mechanical arm grasps the target object, and extract the desired features;

[0058] S3: Initialize the state of the mechanical arm and ensure that the target is in the camera field of view at this time,

[0059] S4: Obtain the target image at this time, and extract the current target features;

[0060] S5: Estimate the depth value Z using the extended Kalman filter method, and substitute it into the calculation of the image Jacobian matrix generalized inverse matrix, while the desired features and the current features are operated to obtain the feature error;

[0061] S6: According to the current state, determine the current servo gain value using the adaptive servo gain algorithm based on DDPG;

[0062] S61: Define the adaptive servo gain DDPG model parameters, including:

[0063] S611: Selection of state space: for IBVS, the servo gain is related to the feature error e and the spatial position of the feature in the camera coordinate system. In addition, since even if there are many feature points, the dimension of the camera speed is fixed, and for any type of mechanical arm end camera, the state is relatively independent, other joint states can be derived from it, therefore, the camera speed is selected as the state space, so the state space can be represented as:

[0064]

[0065] S612: Action space selection: directly select the servo gain as the action space;

[0066] S613: Reward function design: The design of the reward function is based on achieving the goal of moving the feature to the desired position, i.e. avoiding feature loss and improving servo efficiency. The reward function mainly depends on three conditions: reaching the desired position, the feature being outside the field of view, and reducing the feature error as quickly as possible. The present application designs a neural network based on artificial experience to simulate the correction term of the artificial reward function, where the input term is the current action and state space, and the output is the correction term r c of the reward function. As shown in Figure 2 . The correction term neural network parameters are updated according to the target distance change rate (feature error change rate) and the average radiation distance between the feature point and the center of the field of view, and the correction term mainly serves to buffer the experience error. Therefore, the reward function is represented as r = r e + r c .

[0067] The artificially designed reward function r e is represented as:

[0068]

[0069] where R represents the maximum artificial reward value, which is a positive number; κ represents the number of feature points; rn represents the height of the image plane, and cn represents the width of the image plane.

[0070] S62: Design an adaptive servo gain algorithm based on DDPG, as shown in Figure 3 the flowchart of the DDPG algorithm;

[0071] The adaptive servo gain algorithm based on DDPG is described as follows:

[0072] Input: Randomly initialize θ Q and θ μ , initialize the target network by copying the original network: θ Q’ ← θ Q , θ μ’ ← θ μ , and initialize the noise generator as OU t .

[0073] Output: Optimal servo gain network parameters θ μ and θ Q , and the optimal policy.

[0074] 1. Randomly initialize the pose of the robot arm, then obtain the relevant data, and generate the initial state s0 by the state generator.

[0075] 2. Select the servo gain K t = a t = β(s t)=μ(s t |θ μ )+OU t .

[0076] 3、According to the SMCC-IBVS controller with servo gain K t , the camera velocity V c is calculated, and the camera velocity observation is performed according to the robot controller to obtain the artificially designed reward r et and the state generator generates the next state value s t+1 .

[0077] 4、The reward value r ct is obtained by combining the output of the correction term neural network r t , and the correction term neural network parameter is updated in a supervised manner.

[0078] 5、The four-tuple (s t , a t , r t , s t+1 ) is stored in the replay buffer M.

[0079] 6、If the next state s t+1 is the terminal state, terminate the current training and return to step 8.

[0080] 7、If the buffer M is full, randomly select N four-tuples (s t , a t , r t , s t+1 ) from it, and then train the critic and actor neural networks according to the formulas

[0081] and respectively, and finally update the target network parameters according to the formula .

[0082] Where the adaptive law of the servo gain will be represented by the parameterized policy θ μ : s→a, which will be calculated using the DDPG algorithm. The parameter vector θ μ is given by the deterministic policy gradient theorem.

[0083] (1)

[0084] In the formula, β represents a random policy for interacting with the environment, which is a parameter generated to find more potential optimal policies. Q(s,a) is the action value function, which is used to evaluate the long-term reward of state-action pairs, and its value is calculated according to the following formula.

[0085] (2)

[0086] DDPG is a policy-based deep reinforcement learning algorithm that can effectively utilize neural networks to solve continuous multiplication tables (MDPs). DDPG involves four neural networks: the Critic target network (Q') and the Critic online network (Q); the Actor target network (μ') and the Actor online network (μ).

[0087] The current online Critic network Q-parameter θ Q The update uses the TD error method in DQN, and the loss function is to minimize the mean squared error:

[0088] (3)

[0089] In the formula, y i,t Let be the target action value at time t, calculated using the target Critic network Q' and the Actor target network μ'. With the loss function L, we can obtain L with respect to θ using the standard backpropagation method. Q gradient▽ θ Q L is optimized and updated to obtain θ. Q .

[0090] (4)

[0091] (5)

[0092] The actor's current online network μ pairs with network parameters θ μ The update follows the deterministic policy gradient theorem and is determined by formula (1). The update formula approximated by the neural network is expressed as follows:

[0093] (6)

[0094] The network parameters of the target network Q' and μ are updated using a moving average method, i.e., soft update:

[0095] (7)

[0096] In the formula, τ is a hyperparameter that determines the update rate.

[0097] 8. Return to step 2 and repeat the process C times.

[0098] 9. Check the number of training rounds. If the total number of training rounds reaches n, exit; otherwise, return to step 1.

[0099] S63: Use Peter Corke's toolbox of visual servoing to design the IBVS controller, and build the reinforcement learning algorithm with MATLAB's Reinforcement Learning Toolbox, and add the modifier neural network on this basis. The camera in the simulation has the same internal parameters as the camera used in the physical world.

[0100] S64: Set the parameters of DDPG simulation training and complete the adaptive servo gain DDPG training, as shown in Figure 4 The initial state position of the camera training is represented by a circle. The direction of the camera is represented by the x-axis, y-axis and z-axis vector frame. The desired pose of the camera is represented by three thick line axes without circles; Figure 4

[0101] The DDPG training parameters are set: the training process DDPG contains two hidden layers, and the number of neural network units in the first and second layers of the hidden layer is 60 and 40 respectively. The actor (Actor) current network and the actor target network receive a six-dimensional state s as input and output an action a. The critic (Critic) current network and the critic target network receive a state-action pair (s, a) as input and output a one-dimensional action value function Q(s, a). The activation function of the neural network in the hidden layer is set to the Relu function. The activation function of the output layer of the actor (Actor) current network and the actor target network adopts the modified Tanh function to ensure that the proportional gain parameter value k p of the servo controller is in the range of k p ∈(0, 2.0], and the sliding mode gain parameter value k sm of the sliding mode controller is in the range of k sm ∈(0, 3.0]. In each episode of training, N=64 samples are randomly selected from the retransmission buffer M to train the actor and critic neural networks respectively. The learning rate of the actor (Actor) and critic (Critic) network is set to 0.0001, and the update rate of the two target networks is set to 0.001. The artificial maximum reward value R is set to 1.5, the number of hidden layer neural network units of the modifier neural network is n=20, and the activation function of the output layer also adopts the Tanh function to ensure that the output range r c ∈[-1.5, 1.5], and the learning rate is set to 0.001 during training.

[0102] ​S65: In the simulation, the model is trained according to the DDPG adaptive servo gain algorithm process and then kept, and then the model parameters are loaded for transfer training on the real robot arm, wherein the process steps of controlling the robot arm to train the network in the real world are the same as the training process of the network in the simulation, until the model training in the real robot arm control re-converges, and the model is saved.

[0103] S7: The feature error, image Jacobian matrix generalized inverse matrix and servo gain value are handed over to the servo controller to generate a robot arm control signal to control the movement of the robot arm, and if the target pose has not been reached, S4 is returned to continue execution;

[0104] The servo controller is designed: the proportional controller is introduced into the image-based visual servoing (C-IBVS), the servo error is reduced in an exponential manner through the proportional controller, and on this basis, in order to improve the stability of the system, the sliding mode control SMC is used to compensate for the instability of the system, an enhanced method combining proportional control and sliding mode control SMC is proposed, and the image-based visual servoing controller is designed according to the control rate formula, which is called SMCC-IBVS controller.

[0105] The control rate formula is:

[0106]

[0107] Wherein, V c is the camera speed; K p is a fixed non-negative number multiplied by the unit matrix value, and its meaning is usually represented as a proportional servo gain value; K sm is a fixed non-negative number multiplied by the unit matrix value, and its meaning is usually represented as a sliding mode control gain value. ∈R 6×8 represents the generalized inverse matrix of the interaction matrix L p ; sm represents a sliding surface; and sat(·) is a saturation function.

[0108] The saturation function formula is:

[0109]

[0110] Performance analysis:

[0111] In order to prove the effectiveness of the IBVS system proposed in the application, experiments are carried out using a seven-degree-of-freedom Baxter robot hand equipped with an integrated camera with an observation plane target, as shown in Figure 5 The circular target object is stationary relative to the robot arm base, and four feature points on the circular target object are taken as Figure 5 the center positions of the small rectangles, and when the image feature error is less than a given pixel threshold, it indicates that the task is completed, wherein the threshold is 2 pixels in size. The initial pose and the desired pose are:

[0112]

[0113]

[0114] Firstly, the DDPG network is trained in the simulation environment, and then the trained adaptive gain network is used to conduct the above scene experiments, and compared with Dyna-Q learning IBVS and fixed gain SMCC-IBVS system, wherein the fixed gain value is set as K p =0.08I, K sm =2I, respectively, 200 groups of experiments are repeated, and the experimental data in Table 1 is obtained. In this experiment, the servo system of the present application can converge in about 4.3s and the image trajectory is smooth, while the fixed gain SMCC-IBVS consumes 18.6s to reach the expected pose, and the fixed gain SMCC-IBVS sometimes cannot be precisely positioned due to the jitter at the expected pose. Since the robot arm itself has noise, it is considered that the operating environment has noise, that is, it indicates that the algorithm of the present application has anti-interference effect on noise. In addition, the IBVS system of the present application does not need camera calibration, which indicates that the system has robustness to calibration error.

[0115] Table 1 Baxter robot visual servoing control experiment results

[0116] Servo strategy Mean convergence time (s) Servo success rate (%) Mean deviation (mm) Feature loss rate (%) Potential-based IBVS 10.2 89 9.1 11 Dyna-Q learning IBVS 8.0 92 7.8 8 Fixed-gain SMCC-IBVS 18.6 71 6.5 26 DDPG learning SMCC-IBVS 4.3 99 0.2 0

[0117] According to the experimental results in Table 1, in the process of precisely positioning the robot arm by the visual servoing system, compared with the fixed gain SMCC-IBVS, the convergence speed of the IBVS system of the present application is improved by 76.9%, the accuracy is improved by 6.3mm, and there is no feature loss phenomenon in the control process. The experimental results show that the feature loss rate of the Dyna-Q learning servo gain system compared with the system of the present application still exists 8%.

[0118] The above is only a preferred embodiment of the present application, and does not limit the present application in any form, any simple modification, equivalent change and modification of the above embodiment without departing from the technical solution content of the present application, and according to the technical essence of the present application, all still belong to the scope of the technical solution of the present application.

Claims

1. A method for controlling a robot arm based on deep reinforcement learning, characterized in that: The method comprises the following steps: S1: build a mechanical arm visual servo system, set and obtain the desired image captured by the camera when the mechanical arm grabs the target object, and extract the desired features; S2: initialize the state of the mechanical arm and ensure that the target object is in the camera field of view at this time; S3: obtain the current target image and extract the current features; S4: estimate the depth value using the extended Kalman filter method, substitute it into the calculation of the image Jacobian matrix generalized inverse matrix, and calculate the image feature error e by operating the desired features and the current features; S5: according to the current state of the mechanical arm visual servo system, an adaptive servo gain algorithm based on depth deterministic policy gradient DDPG is used to determine the servo gain value, and the process is as follows: S51: define the adaptive servo gain DDPG model parameters, including: (1) selection of state space: selecting the original speed of the camera as the state space, (2) action space selection: directly selecting the servo gain as the action space; (3) reward function design: design a neural network to simulate the correction term of the artificial reward function on the basis of artificial experience, wherein the input of the neural network is the current action and state, and the output of the neural network is the correction term r of the reward function c ; the reward function r thereof is represented as r = r e + r c ; wherein r e is an artificially designed reward function, represented as: ； wherein R is a maximum artificial reward value, is a positive number; k is the number of image feature points; rn is the height of the image plane, cn is the width of the image plane, p i is a current feature point, p id is an expected feature point; S52: design an adaptive servo gain algorithm based on DDPG, and the specific steps are as follows: Input: Randomly initialize the servo gain network parameters θ Q , θ μ ; initialize the target network by copying the current network: θ Q’ ← θ Q , θ μ’ ← θ μ ; initialize the noise generator to OU t ; repeat for C number of execution times, total number of training rounds n; Output: optimal servo gain network parameters and optimal policy; Step 1, randomly initialize the pose of the mechanical arm, and generate the initial state s0 from the state generator; Step 2, Selecting the servo gain K t = a t = β(s t )= μ(s t |θ μ )+ OU t where s t is the current state, a t is the current action, β is the state generator, and μ is the action policy function. Step 3, camera velocity V is calculated from the servo gain K t of the servo controller SMCC-IBVS c and the camera velocity V is executed from the robot controller c The current reward r is observed from the human-designed reward function et and the state generator generates the next state value s t+1 ; Step 4, obtain a current reward value r by combining the correction term output by the neural network t updating the neural network parameters in a supervised manner; Step 5, the quadlet (s t ,a t ,r t ,s t+1 ) is stored into the replay buffer M; Step 6, if the next state s t+1 is the termination state, terminate the current training, and perform Step 8. Step 7, if the buffer M is full, randomly select N sample quadruplets (s t , a t , r t , s t+1 ) from it, train the critic and actor neural networks respectively, update the target network parameters; Step 8, return to step 2 and repeat execution C times; Step 9, detect the number of training rounds, if the total number of training rounds has not reached n, return to step 1; otherwise, exit; S53: train the adaptive servo gain DDPG model: build a DDPG training environment; set the DDPG training parameters and complete the training; S6: servo control of the mechanical arm: input the image feature error, image Jacobian matrix generalized inverse matrix and servo gain value into the servo controller SMCC-IBVS to generate a mechanical arm control signal to control the movement of the mechanical arm, and if the target pose has not been reached, return to S3. 2.The deep reinforcement learning-based robot arm control method of claim 1, wherein: In step S1, the mechanical arm visual servo system comprises a seven-degree-of-freedom mechanical arm, a controller, and a camera integrated on the end effector of the mechanical arm. 3.The deep reinforcement learning-based robot arm control method of claim 1, wherein: In step S51(1), the selection of the camera original speed as the state space can be represented as: ； where S is a state space, s is a state space sub-element, is the camera raw velocity, e is the image feature error, is the interaction matrix is the generalized inverse matrix.

4. The robotic arm control method based on deep reinforcement learning as described in claim 1, characterized in that: In step S51(3), the parameters of the neural network are supervisedly updated according to the feature error change rate and the average radiation distance of the feature points from the center of the field of view, and the correction term plays the role of buffering the experience error.

5. The deep reinforcement learning-based robot arm control method of claim 1, wherein: In step 3 of step S52, the servo controller SMCC-IBVS: introduces a proportional controller into the image-based visual servoing C-IBVS, reduces the servo error in an exponential manner through the proportional controller, and in order to improve the stability of the system, adopts a sliding mode control SMC to compensate for the instability of the system, proposes an enhanced method combining proportional control and sliding mode control SMC, and designs an image-based visual servoing controller according to the control law formula, which is called servo controller SMCC-IBVS.

6. The mechanical arm control method based on depth reinforcement learning according to claim 5, wherein: The control law formula is: ； where V c is the camera velocity; K p is a fixed non-negative number multiplied by the identity matrix, i.e., the proportional servo gain value; K sm is a fixed non-negative number multiplied by the identity matrix, i.e., the sliding mode control gain value; ∈R 6×8 represents the generalized inverse matrix of the interaction matrix L p ; sm denotes the sliding surface; sat(·) is a saturation function; The saturation function formula is: ； Wherein, sgn(·) represents the sign function.

7. The deep reinforcement learning-based robot arm control method of claim 1, wherein: In step S53, the DDPG training environment is built: the adaptive servo gain of DDPG uses the toolbox of visual servo of Peter Corke to design the IBVS controller, the reinforcement learning algorithm is built by means of the reinforcement learning toolbox of MATLAB, and the modification term neural network is added on this basis, and the camera in simulation has the same internal parameters as the camera used in the physical world.

8. The deep reinforcement learning-based robot arm control method of claim 1, wherein: In step S53, the setting DDPG training parameters: the training process DDPG contains two hidden layers; the actor Actor current network and the actor target network receive six-dimensional state s as input, and output action a; the critic Critic current network and the critic target network receive state-action pair (s, a) as input, and output one-dimensional action value function Q(s, a); the activation function of the neural network in the hidden layer is set to the Relu function; and the activation function of the output layer of the actor Actor current network and the actor target network adopts the modified Tanh function to ensure the proportional gain parameter value k in the servo controller p Output range k p ∈(0, 2.0], the sliding mode gain parameter value k of the sliding mode controller sm Output range k sm ∈(0, 3.0]; N=64 samples are randomly selected from the retransmission buffer M for training the actor and critic neural networks respectively at each iteration of the training; the learning rate of the actor Actor and critic Critic network is set to 0.0001, and the update rate of the target network is set to 0.001; the artificial maximum reward value R is set to 1.5, the correction term neural network hidden layer neural network unit number is h=20, and the activation function of the output layer also adopts the Tanh function to ensure the output range r c ∈[-1.5, 1.5], and the learning rate during training is set to 0.

001.

9. The deep reinforcement learning-based robot arm control method of claim 8, wherein: The number of neural network units of the first layer and the second layer of the two hidden layers is 60 and 40 respectively.

Images