Mechanical arm visual servo control method and system based on deep reinforcement learning

Abstract

The invention relates to a mechanical arm visual servo control method and system based on deep reinforcement learning. Firstly, real-time state observation and collision detection are conducted on the mechanical arm, and an observation result is obtained; then, a deep reinforcement learning controller is built by means of a maximum entropy deep reinforcement learning SAC algorithm network architecture, and an action space, a state space and a reward function are built according to an observation result and a target position; and then, outputting a first action representation based on the construction content, optimizing a controller according to a calculation result of the first action representation and the reward function, smoothly optimizing the first action representation by using a real-time track interpolation strategy to obtain a second action representation, and controlling the mechanical arm joint according to the second action representation. And the process is continuously repeated until the mechanical arm successfully arrives at the target position. Compared with the prior art, the method has the advantages of high control real-time performance, large application range and the like.

Description

Technical Field

[0001] The present invention relates to the field of robotic arm control, and in particular to a robotic arm visual servo control method and system based on deep reinforcement learning. Background Art

[0002] Traditional robotic arm control methods require the pre-construction of precise environmental maps or obstacle models (e.g., position, shape, and motion patterns). However, in dynamic or unstructured scenarios (e.g., moving obstacles and complex geometric shapes), accurate modeling becomes significantly more challenging. Real-time positional changes of obstacles are difficult to capture, and high-precision modeling of complex geometric obstacles consumes significant computing resources, compromising the real-time performance of obstacle avoidance strategies.

[0003] Obstacle avoidance tasks must simultaneously meet multiple constraints such as end-point posture accuracy, joint speed limits, and energy consumption optimization. Traditional methods require manual design of the weights or priorities of the objective functions, making it difficult to adaptively balance conflicting objectives.

[0004] Traditional methods are sensitive to sensor noise, model mismatch, and external disturbances. For example, false detections from vision sensors or noise from lidar can cause obstacle avoidance trajectories to oscillate or even cause collisions, and controller parameters need to be recalibrated when the load changes.

[0005] For example, the invention patent with publication number CN116533249A discloses a robotic arm control method based on deep reinforcement learning, which uses the extended Kalman filter method to estimate the depth value, substitutes it into the calculated image Jacobian matrix and calculates the generalized inverse matrix, and simultaneously calculates the expected features and the current features to obtain the image feature error e; according to the current state of the robotic arm visual servo system, an adaptive servo gain algorithm based on deep deterministic policy gradient DDPG is used to determine the servo gain value; however, it still requires the pre-establishment of the DDPG training environment and relies on the training of the adaptive servo gain DDPG model, resulting in poor control real-time performance and adaptability. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a robot arm visual servo control method and system based on deep reinforcement learning.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] According to one aspect of the present invention, a method for visual servo control of a robotic arm based on deep reinforcement learning is provided, the method steps comprising:

[0009] S1. Perform real-time state observation and real-time collision detection on the simulated robotic arm to obtain observation results;

[0010] S2. Use the maximum entropy deep reinforcement learning (SAC) algorithm network architecture to build a deep reinforcement learning controller, and build the action space, state space, and reward function based on the observation results and target position;

[0011] S3. Based on the constructed deep reinforcement learning controller, action space, state space, and reward function, output a first action representation, and optimize the deep reinforcement learning controller based on the result calculated by the first action representation and the reward function;

[0012] S4. Smoothly optimize the first action representation using a real-time trajectory interpolation strategy to obtain a second action representation;

[0013] S5. Control the joints of the robotic arm based on the second action representation; return to step S1 and execute until the robotic arm reaches the target position.

[0014] As a preferred technical solution, the deep reinforcement learning controller built in S2 includes an Actor, a Critic, and a Target Critic. The Actor is a policy network responsible for learning the policy. The Critic includes two Q networks for estimating the state-action value function and updating the parameters by minimizing the Bellman error. The Target Critic includes two target Q networks, which correspond to the two Q networks in the Critic, respectively, to stabilize the training process.

[0015] As a preferred technical solution, when adjusting the parameters of the deep reinforcement learning controller built in S2, the number of layers, the number of neurons in each layer, and the activation function of the policy network and Q network are adjusted to ensure that the model has sufficient expressive power but does not overfit; and the learning rate and its scheduling strategy are adjusted to avoid divergence or slow convergence during training.

[0016] As a preferred technical solution, the action space in S2 is composed of a set of joint velocities, Indicates that the range of the action value is set within [-1,1]rad / s, and the maximum change of each control cycle is set to 0.1rad / s.

[0017] As a preferred technical solution, the state space in S2 consists of the manipulator state, the tracking target state, and the obstacle state. The manipulator state includes joint position information and joint velocity information; the tracking target state includes the gap between the target position and the end effector position and the gap between the target velocity and the end effector velocity; the obstacle state includes the gap between the reference joint velocity in the previous time step and the actual strategy output and the safe distance between the manipulator and the obstacle. The specific formula is as follows:

[0018]

[0019]

[0020] Among them, s t For complete observation of the robotic arm; is the status of the robotic arm; To track the target status; Obstacle status; is the joint position; is the joint velocity; Indicates the target location and the end effector position the gap between; Indicates target speed and end effector speed the gap between; Indicates the reference joint velocity of the previous time step and strategy output the gap between; is the safe distance between the robotic arm and the obstacle.

[0021] As a preferred technical solution, the reward function in S2 includes a tracking error reward term, a motion smoothness reward term, and a collision penalty term. The specific formula of the reward function is as follows:

[0022]

[0023] Among them, r t is the reward function; is the tracking error reward, whose size depends on the distance between the end effector of the robot arm and the target point; is the motion smoothness bonus; is the collision penalty term; and is the reward function threshold constraint; and is the tracking error reward function parameter; and is the motion smoothness reward function parameter.

[0024] As a preferred technical solution, the real-time trajectory interpolation strategy in S4 includes a low-pass Butterworth filtering part and a velocity interpolation part.

[0025] As a preferred technical solution, the low-pass Butterworth filter is used to smooth the overall action representation output by reinforcement learning to avoid vibration of the robotic arm or action changes greater than the preset speed caused by irregular changes in the output action representation. The specific formula is:

[0026]

[0027] Among them, ω c is the cutoff frequency; T s is the sampling period; is the output action of the strategy at time t; is the output action of the strategy at time t-1; is the action of time step t after being processed by the low-pass filter; It is the action at time step t-1 after being processed by the low-pass filter.

[0028] As a preferred technical solution, the velocity interpolation part is used to smooth the transition of joint velocity to avoid motion jitter caused by the abrupt or discontinuous motion output by the reinforcement learning strategy. The specific formula is:

[0029]

[0030] α(s)=6s 5 -15s 4 +10s 3 ;

[0031] s = (ι - t1) / (t2 - t1);

[0032] Among them, α(s) is the mixing function, s is the normalized time parameter; It is the action at time step t1 after being processed by the low-pass filter; It represents the joint velocity at time step t1; t1 and t2 are both time steps, and t2 is the next time step after t1.

[0033] According to another aspect of the present invention, a robotic arm visual servo control system based on deep reinforcement learning is provided. The system operates using the robotic arm visual servo control method based on deep reinforcement learning described above. The system includes an environment perception module, a model building module, a motion optimization module, and a robotic arm control module.

[0034] The environmental perception module is used to perform real-time state observation and real-time collision detection on the simulated robotic arm to obtain observation results;

[0035] The model building module is used to build a deep reinforcement learning controller using the maximum entropy deep reinforcement learning (SAC) algorithm network architecture, and to build the action space, state space, and reward function based on the observation results and target position;

[0036] The action optimization module is used to output a first action representation based on the constructed deep reinforcement learning controller, action space, state space, and reward function. The deep reinforcement learning controller is optimized based on the results calculated from the first action representation and the reward function. The first action representation is then smoothly optimized using a real-time trajectory interpolation strategy to obtain a second action representation.

[0037] The robotic arm control module controls the robotic arm joints based on the second action representation and repeats the above reinforcement learning process until the robotic arm reaches the target position.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. In the present invention, first, the state of the robot arm is observed and collision detected in real time to obtain the observation results. Then, a deep reinforcement learning controller is built with the help of the maximum entropy deep reinforcement learning SAC algorithm network architecture, and the action space, state space and reward function are constructed based on the observation results and the target position. Then, based on the above-mentioned construction content, the first action representation is output, and then the controller is optimized according to the calculation result of the reward function, and the first action representation is smoothly optimized using the real-time trajectory interpolation strategy to obtain the second action representation, and the joints of the robot arm are controlled according to the second action representation. The above process is repeated until the robot arm successfully reaches the target position; real-time collision-free visual servo control of the robot arm is realized by deep reinforcement learning technology, without the need to pre-establish a mathematical model for solution. Through the model-free method, the deep reinforcement learning controller can autonomously learn the task objectives, thereby improving the real-time performance of the control, and can be adjusted in real time in complex dynamic environments without relying on the model, thereby increasing the adaptability of the control method.

[0040] 2. In the present invention, the real-time trajectory interpolation strategy includes a low-pass Butterworth filtering part and a speed interpolation part. The low-pass Butterworth filtering part is used to smooth the overall action representation output by the reinforcement learning, and the speed interpolation part is used to smooth the transition of the joint speed. By using the real-time trajectory interpolation strategy to smooth the output of the reinforcement learning strategy, the jitter and instability of the robotic arm during movement caused by the actions generated during the deep reinforcement learning inference process are avoided, the service life of the robotic arm is extended, and the movement continuity of the robotic arm is ensured.

[0041] 3. The action space in the present invention is composed of a set of joint velocities. Joint velocities are used as action primitives to directly map the underlying control instructions of the robot arm, thereby improving motion control accuracy. The state space is composed of the robot arm state, the tracking target state, and the obstacle state. The robot arm body state, the target state, and the obstacle state are integrated to achieve joint representation of multi-source heterogeneous information and enhance the correlation between environmental perception and decision-making. The reward function includes a tracking error reward item, a motion smoothness reward item, and a collision penalty item. This ensures the accuracy, stability, and safety of the optimization strategy and improves the control accuracy of the overall control process.

[0042] 4. When tuning parameters, the deep reinforcement learning controller of the present invention adjusts the number of layers, the number of neurons per layer, and the activation function of the policy network and Q network to ensure sufficient model expressiveness without overfitting. Furthermore, the learning rate and its scheduling strategy are adjusted to avoid divergence or slow convergence during training. This balances the model's expressiveness and generalization, improving the stability and efficiency of the training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of the steps of a visual servo control method for a robotic arm based on deep reinforcement learning in the present invention;

[0044] Figure 2 Schematic diagram of the SAC algorithm flow in the present invention;

[0045] Figure 3 This is a schematic diagram of the deep reinforcement learning process in the present invention;

[0046] Figure 4 This is the overall block diagram of deep reinforcement learning in this invention. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0048] Example 1

[0049] In view of the many problems existing in the existing visual servo control of robotic arms, this embodiment adopts a real-time collision-free visual servo control method for robotic arms based on deep reinforcement learning. The method steps are as follows: Figure 1 As shown, specifically including:

[0050] S1. Perform real-time state observation and real-time collision detection on the simulated robotic arm to obtain observation results;

[0051] S2. Build a deep reinforcement learning controller using the maximum entropy deep reinforcement learning soft actor-critic (SAC) algorithm network architecture, and construct the action space, state space, and reward function based on the observation results and target position;

[0052] S3. Based on the constructed deep reinforcement learning controller, action space, state space, and reward function, output a first action representation, and optimize the deep reinforcement learning controller based on the result calculated by the first action representation and the reward function;

[0053] S4. Smoothly optimize the first action representation using a real-time trajectory interpolation strategy to obtain a second action representation;

[0054] S5. Control the joints of the robotic arm based on the second action representation; return to step S1 and execute until the robotic arm reaches the target position.

[0055] In this embodiment, taking a six-axis robotic arm as an example, first, a simulation training environment is constructed in a physical simulation engine. Secondly, a deep reinforcement learning controller is designed. Combined with the actual working environment and tasks, the state space, action space, reward function, etc. are constructed to train the intelligent agent. Finally, the simulation training results are transferred to the real robotic arm for verification.

[0056] The specific implementation steps are as follows:

[0057] Step 1): Build a deep reinforcement learning and robotic arm simulation environment.

[0058] Step 2): Implement the Maximum Entropy Deep Reinforcement Learning (SAC) algorithm. The SAC algorithm is an advanced algorithm based on the Maximum Entropy Reinforcement Learning framework. Unlike traditional reinforcement learning algorithms, SAC not only focuses on maximizing the cumulative reward, but also enhances the agent's exploration ability by maximizing the entropy of the strategy. The algorithm structure is as follows: Figure 2 This design enables SAC to perform well in continuous control tasks, especially showing strong adaptability in high-dimensional state spaces and complex dynamic environments.

[0059] In traditional reinforcement learning, the agent's goal is to maximize the expected value of the cumulative reward. In maximum entropy reinforcement learning, the objective function is expanded to maximize both the cumulative reward and the entropy of the policy. Entropy is a measure of the randomness of the policy, and a high-entropy policy means that the agent has greater diversity when exploring the environment. The maximum entropy objective function can be expressed as:

[0060]

[0061] in, is the policy π in state s tThe entropy under , α is the temperature parameter used to adjust the relative importance between reward and entropy.

[0062] Defined as:

[0063]

[0064] SAC combines the advantages of the Actor-Critic architecture and entropy regularization, aiming to maximize the entropy of the policy while maximizing the cumulative reward by learning a random policy. This design enables SAC to achieve a better balance between exploration and exploitation, and excels in continuous control tasks. The SAC algorithm is based on the Actor-Critic architecture and includes the following core components:

[0065] (1) Actor: responsible for learning strategy π φ (a t |s t ), where φ is the parameter of the policy network.

[0066] (2) Critic: Contains two Q networks and It is used to estimate the state-action value function and update the parameters by minimizing the Bellman error.

[0067] (3) Target Q network: Use the target network and to stabilize the training process.

[0068] The goal of SAC is to optimize the policy π φ and Q θ To maximize the cumulative reward of entropy regularization. Specifically, the update target of the Q function is:

[0069]

[0070] Among them, γ is the discount factor, and the loss function of the Q function is:

[0071]

[0072] The update target of the policy network is to maximize the entropy regularized Q value:

[0073]

[0074] In addition, SAC also introduces a mechanism to automatically adjust the entropy regularization coefficient α, which is dynamically adjusted by optimizing the following objectives:

[0075]

[0076] in, is the target entropy, usually set to the dimension of the action space.

[0077] By maximizing policy entropy, SAC can achieve a better balance between exploration and exploitation, avoiding premature convergence to suboptimal policies. In addition, using two Q networks and taking the minimum value as the goal reduces the overfitting problem of Q-value estimation.

[0078] Step 3): Action space, state space design and reward function design

[0079] The process of implementing reinforcement learning training is as follows Figure 3 As shown, the specific action space, state space and reward function are designed as follows:

[0080] (1) Action space: The action space is composed of a set of joint velocities. It should be noted that by selecting actions in the joint space as control inputs, singularity problems can be avoided. To achieve more reliable training, the range of action values is set to [-1, 1] rad / s, and the maximum change per control cycle is set to 0.1 rad / s.

[0081] (2) State space: The state space consists of the states of the manipulator, the tracking target, and the obstacles. Its complete observation is defined as Among them, the first part By joint position and joint velocity Composition; Part II middle, Indicates the target location and the end effector position The gap between Indicates target speed and end effector speed The gap between; Part III middle, Indicates the reference joint velocity of the previous time step and strategy output The observations are normalized by subtracting the mean and dividing by the standard deviation.

[0082] (3) Reward function design: To achieve the target tracking and obstacle avoidance tasks of the robot arm in a static environment, this paper designs a multi-objective reward function, which consists of three parts: tracking error, motion smoothness, and collision avoidance. These three rewards work together to guide the reinforcement learning algorithm to optimize the motion trajectory of the robot arm, so that it can accurately track the target point, ensure smooth motion, and avoid collisions with obstacles. The overall form of the reward function can be expressed as:

[0083]

[0084] in, The tracking error reward function is a piecewise function whose magnitude depends on the distance between the robot's end effector and the target point. This function is designed to encourage the robot to approach the target as quickly as possible, while also providing a higher reward when the error is small, thereby improving target tracking accuracy. Specifically, the reward function is defined as follows:

[0085]

[0086] in, is the threshold; i=1,…,4 is the weight coefficient.

[0087] Used to measure the smoothness of the action, the main purpose of this reward term is to constrain the reinforcement learning strategy to generate smoother and more continuous control signals, thereby improving the stability and controllability of the robot arm when performing tasks. Specifically, the reward function is defined as follows:

[0088]

[0089] in, j=1,2 is the weight coefficient.

[0090] is the penalty for collision, defined as follows:

[0091]

[0092] The penalty item is set so that: if the safety distance Less than threshold The strategy will receive a larger penalty term r collision , so that when a collision occurs, Almost overshadows the other two bonus items.

[0093] Step 4): Design a real-time trajectory interpolation strategy to smooth the output of the reinforcement learning strategy

[0094] To avoid jitter and instability in the robot arm's motion caused by the actions generated during reinforcement learning inference, this paper designs a real-time trajectory interpolation strategy to smooth the output of the reinforcement learning strategy and ensure smoother and more stable movement of the robot arm. Specifically, this strategy consists of two main components:

[0095] (1) Low-pass Butterworth filter: Low-pass filters play a vital role in smoothing action values and reducing noise interference. Especially in the process of deep reinforcement learning, the output of the strategy may produce irregular changes due to noise or excessive exploration, causing the robot arm to vibrate or change its movement too quickly. In order to overcome this problem, this paper first uses a low-pass Butterworth filter, which has a strong frequency smoothing ability and can effectively remove high-frequency noise, making the movement of the robot arm smoother. The design of this filter takes into account the real-time requirements of the robot arm motion control, and its specific expression is as follows:

[0096]

[0097] Among them, ω c is the cutoff frequency, T s is the sampling period, is the output action of the strategy, and This is the action after being processed by a low-pass filter.

[0098] (2) Velocity interpolation: In order to further smooth the motion trajectory of the robot arm, this paper designs a velocity interpolation method to generate a smooth joint velocity trajectory to avoid motion jitter caused by the abrupt or discontinuous motion output by the reinforcement learning strategy. Assume that at time step t1, the joint velocity is expressed as The action given by the reinforcement learning policy is expressed as At the next time step t2, the joint velocity The transition can be calculated as:

[0099]

[0100] Where α(s) is the mixing function and s = (ι - t1) / (t2 - t1) is the normalized time parameter. As a boundary condition, the mixing function α(s) is defined as α(s) = 6s in our designed method. 5 -15s 4 +10s 3 Therefore, the direction of the latest velocity trajectory of the reinforcement learning model is corrected in real time at each time step.

[0101] Step 5): When tuning the SAC algorithm, focus on several key hyperparameters. First, select an appropriate network structure (such as the number of layers, number of neurons per layer, and activation function) to ensure the model has sufficient expressive power without overfitting. Second, optimize the learning rate and its scheduling strategy to avoid divergence or slow convergence during training. The specific parameters are shown in Table 1.

[0102] Table 1. Parameters of deep reinforcement learning controller

[0103]

[0104] Step 6): Training environment deployment and training results

[0105] The design of the training environment module is a key step in this research. It organically integrates the robot's kinematic model and obstacle information in the environment into the OpenAI Gym interface, providing a highly flexible and scalable training environment for reinforcement learning algorithms. In this module, precise kinematic modeling is first used to link the robot's joint states with the position and posture information of the end effector, ensuring that each time step in the environment accurately reflects the robot's dynamic behavior and motion. Next, obstacle information is incorporated into the training environment, and the distance between obstacles and the robot arm is calculated in real time to ensure that the algorithm can correctly identify and avoid these obstacles during training.

[0106] In terms of state space design, this study combines the joint state of the robot arm, the target position, and the distance information of obstacles to construct a multi-dimensional state representation. Specifically, the joint state of the robot arm contains the angle and angular velocity information of each joint, which is crucial for describing the movement and control of the robot arm. At the same time, the target position and obstacle distance information as part of the state enable the reinforcement learning model to have a comprehensive understanding of the current environment at every moment, thereby making reasonable decisions. This comprehensive state space design not only effectively reflects the dynamic changes of the environment, but also improves the model's adaptability to target tracking and obstacle avoidance tasks.

[0107] In the design of the reward function, this study adopted a multi-objective weighted approach to collaboratively optimize the objectives of the obstacle avoidance and tracking tasks. Specifically, the reward function includes two main objectives: one is to encourage the robotic arm to avoid collisions with obstacles as much as possible to ensure the safety of movement; the other is to guide the robotic arm to reach the target point as quickly and accurately as possible to complete the target tracking task. On this basis, by assigning different weights to these two objectives, the optimization of the obstacle avoidance and tracking tasks is balanced, ensuring that the robotic arm can make reasonable trade-offs between different objectives when performing tasks. In addition, the reward function also designs a penalty term to avoid inappropriate actions, such as violent joint movements or excessive path adjustments, to ensure that the trained strategy has good robustness and practical usability.

[0108] Step 7): Migrate from simulation to real environment

[0109] To address the physical property differences between simulated and real-world environments, this paper proposes a robustness enhancement method based on environmental noise injection. This method actively introduces state observation noise and action execution perturbations during simulation training to simulate sensor errors, actuator bias, and dynamic uncertainty present in real-world environments, thereby improving the generalization and transfer performance of reinforcement learning strategies. The specific implementation is as follows:

[0110] (1) State space noise injection: In the state observation phase of the simulation environment, Gaussian noise is added to the joint position, speed and obstacle distance information of the robot arm. To simulate the random error in real sensor measurement. The state observation value after noise injection It can be expressed as:

[0111]

[0112] Among them, s t is the original state value, is the noise standard deviation, and its value is set according to the real sensor calibration data. By introducing state noise, the policy network can learn a robust representation of observation deviations.

[0113] (2) Action output disturbance: During the action execution phase, a uniformly distributed disturbance is added to the robot arm joint velocity command. To simulate the dynamic response error of the real actuator. Action command after disturbance for:

[0114]

[0115] Among them, a t is the original action output by the policy network, and b is the perturbation amplitude. Through action perturbation, the policy network can adapt to the non-ideal response characteristics of the actuator.

[0116] (3) To further enhance the generalization ability, the noise parameters are randomly sampled within a preset range during training to avoid overfitting of the strategy to a fixed noise pattern.

[0117] The overall framework of deep reinforcement learning is as follows Figure 4 As shown in the figure, the current state of the robot arm and its target position are fed into the policy network as input data. The policy network also receives real-time collision detection and observation space data to adjust the policy. The policy network outputs a first action representation, which is then used to adjust the policy network inversely. Real-time trajectory interpolation processes the action output of the policy network to generate a second action representation, which smooths the robot arm's motion trajectory. Based on this second action representation, a PID controller is used to control the robot arm.

[0118] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A robotic arm visual servo control method based on deep reinforcement learning, characterized in that: The method steps include: S1. Perform real-time state observation and real-time collision detection on the simulated robotic arm to obtain observation results; S2. Use the maximum entropy deep reinforcement learning (SAC) algorithm network architecture to build a deep reinforcement learning controller, and build the action space, state space, and reward function based on the observation results and target position; S3. Based on the constructed deep reinforcement learning controller, action space, state space, and reward function, output a first action representation, and optimize the deep reinforcement learning controller based on the result calculated by the first action representation and the reward function; S4. Smoothly optimize the first action representation using a real-time trajectory interpolation strategy to obtain a second action representation; S5. Control the joints of the robotic arm based on the second action representation; return to step S1 and execute until the robotic arm reaches the target position.

2. A method for controlling a robotic arm visual servo based on deep reinforcement learning according to claim 1, characterized in that: The deep reinforcement learning controller constructed in S2 includes an Actor, a Critic, and a target Critic; wherein the Actor is a policy network responsible for learning the policy; the Critic includes two Q networks for estimating the state-action value function and updating the parameters by minimizing the Bellman error; the target Critic includes two target Q networks, which correspond to the two Q networks in the Critic respectively and are used to stabilize the training process.

3. A method for controlling a robotic arm visual servo based on deep reinforcement learning according to claim 2, characterized in that: When adjusting the parameters of the deep reinforcement learning controller built in S2, the number of layers of the policy network and the Q network, the number of neurons in each layer, and the activation function are adjusted to ensure that the model has sufficient expressive power but does not overfit; and the learning rate and its scheduling strategy are adjusted to avoid divergence or slow convergence during training.

4. The method for controlling a robotic arm visual servo based on deep reinforcement learning according to claim 1, wherein: The action space in S2 is composed of a set of joint velocities, Indicates that the range of the action value is set within [-1,1]rad / s, and the maximum change of each control cycle is set to 0.1rad / s.

5. A method for controlling a robotic arm visual servo based on deep reinforcement learning according to claim 4, characterized in that: The state space in S2 consists of the manipulator state, the tracking target state, and the obstacle state. The manipulator state includes joint position information and joint velocity information; the tracking target state includes the gap between the target position and the end effector position and the gap between the target velocity and the end effector velocity; the obstacle state includes the gap between the reference joint velocity at the previous time step and the actual strategy output and the safe distance between the manipulator and the obstacle. The specific formula is as follows: Among them, s t For complete observation of the robotic arm; is the state of the robotic arm; To track the target status; Obstacle status; is the joint position; is the joint velocity; Indicates the target location and the end effector position the gap between; Indicates target speed and end effector speed the gap between; Indicates the reference joint velocity of the previous time step and strategy output the gap between; is the safe distance between the robotic arm and the obstacle.

6. A method for controlling a robotic arm visual servo based on deep reinforcement learning according to claim 4, characterized in that: The reward function in S2 includes a tracking error reward term, a motion smoothness reward term, and a collision penalty term. The specific formula of the reward function is as follows: Among them, r t is the reward function; is the tracking error reward, whose size depends on the distance between the end effector of the robot arm and the target point; is the motion smoothness bonus; is the collision penalty term; and is the reward function threshold constraint; and is the tracking error reward function parameter; and is the motion smoothness reward function parameter.

7. The method for controlling a robotic arm visual servo based on deep reinforcement learning according to claim 1, wherein: The real-time trajectory interpolation strategy in S4 includes a low-pass Butterworth filtering part and a velocity interpolation part.

8. A method for controlling a robotic arm visual servo based on deep reinforcement learning according to claim 7, characterized in that: The low-pass Butterworth filter is used to smooth the entire action representation output by reinforcement learning to avoid vibration of the robotic arm or action changes greater than a preset speed caused by irregular changes in the output action representation. The specific formula is: Among them, ω c is the cutoff frequency; T s is the sampling period; is the output action of the strategy at time t; is the output action of the strategy at time t-1; is the action of time step t after being processed by the low-pass filter; It is the action at time step t-1 after being processed by the low-pass filter.

9. The method for controlling a robotic arm visual servo based on deep reinforcement learning according to claim 7, wherein: The velocity interpolation part is used to smooth the transition of joint velocity to avoid motion jitter caused by the abrupt or discontinuous motion output by the reinforcement learning strategy. The specific formula is: α(s)=6s 5 -15s 4 +10s 3 ; s = (ι - t1) / (t2 - t1); Among them, α(s) is the mixing function, s is the normalized time parameter; It is the action at time step t1 after being processed by the low-pass filter; It represents the joint velocity at time step t1; t1 and t2 are both time steps, and t2 is the next time step after t1.

10. A robotic arm visual servo control system based on deep reinforcement learning, characterized in that: The system is applied to a robot arm visual servo control method based on deep reinforcement learning as described in any one of claims 1 to 9, and the system includes an environment perception module, a model building module, a motion optimization module and a robot arm control module; The environment perception module is used to perform real-time state observation and real-time collision detection on the simulation manipulator to obtain observation results; The model building module is used to build a deep reinforcement learning controller using the maximum entropy deep reinforcement learning (SAC) algorithm network architecture, and to build the action space, state space, and reward function based on the observation results and target position; The action optimization module is used to output a first action representation based on the constructed deep reinforcement learning controller, action space, state space and reward function, and optimize the deep reinforcement learning controller according to the result calculated by the first action representation and the reward function; The first action representation is smoothly optimized using a real-time trajectory interpolation strategy to obtain a second action representation; The robotic arm control module controls the robotic arm joints based on the second action representation and repeats the above reinforcement learning process until the robotic arm reaches the target position.

Images