Robotic arm real-time control method based on deep reinforcement learning in dynamic environment

Abstract

This application discloses a real-time control method for a robotic arm based on deep reinforcement learning in a dynamic environment. The method includes acquiring the state information of the robotic arm, obstacles, and a target point; using an artificial potential field method to calculate the attraction velocity of the target point and the repulsion velocity of the obstacle to obtain a first control action of the robotic arm; using a deep reinforcement learning algorithm to obtain a second control action of the robotic arm; and superimposing the first and second control actions to obtain the real-time control action of the robotic arm for path planning. This application combines the advantages of deep reinforcement learning and the artificial potential field method, enabling more precise robotic arm movement in complex control environments. It improves the obstacle avoidance capability and learning efficiency of the robotic arm when performing path planning tasks in unstructured complex environments, achieving accurate and efficient control of robotic arm movement in dynamic environments. This application has wide applications in the field of robotic arm control technology.

Description

Technical Field

[0001] This application relates to the field of robotic arm control technology, and in particular to a real-time control method for robotic arms based on deep reinforcement learning in dynamic environments. Background Technology

[0002] In the field of robotic arm control, traditional control methods often rely on pre-programmed trajectory planning or sensor-feedback-based position control. These methods have some problems and limitations in complex and dynamic environments, including but not limited to the following:

[0003] 1. Lack of real-time perception and adaptation: Traditional control methods often fail to perceive and adapt to changes in dynamic environments in real time, resulting in delayed control response and performance degradation.

[0004] 2. Difficulty in obstacle avoidance: In scenarios with frequently changing obstacles, traditional trajectory planning and position control methods are often unable to flexibly avoid obstacles, resulting in collisions or failure to reach the target position.

[0005] 3. In complex environments, it is prone to getting stuck in local optima and oscillations, leading to decreased control performance and stability issues. Summary of the Invention

[0006] To address at least one of the technical problems existing in the aforementioned related technologies, this application proposes a real-time control method for a robotic arm based on deep reinforcement learning in a dynamic environment, aiming to achieve precise control of the robotic arm's movement and obstacle avoidance.

[0007] This application proposes a real-time control method for a robotic arm based on deep reinforcement learning in a dynamic environment, including:

[0008] Acquire robotic arm status information, obstacle status information, and target point status information; the robotic arm status information includes joint angle information, joint position information, and joint velocity information of each joint;

[0009] Based on the state information of the robotic arm, the state information of the obstacle, and the state information of the target point, the artificial potential field method is used to calculate the attraction velocity of the target point and the repulsion velocity of the obstacle to obtain the first control action of the robotic arm.

[0010] Based on the state information of the robotic arm, a deep reinforcement learning algorithm is used to obtain the second control action of the robotic arm;

[0011] Based on the first control action and the second control action, the actions are superimposed to obtain the real-time control actions of the robotic arm;

[0012] Based on the real-time control actions, the robotic arm is controlled to perform path planning tasks.

[0013] In some embodiments, the method further includes:

[0014] A collision detection method is used to detect whether the robotic arm reaches the target point without collision when performing the real-time control action, and the detection result is determined.

[0015] Based on the detection results, it is determined whether the robotic arm should continue to perform the path planning task.

[0016] In some embodiments, the obstacle state information includes obstacle position information and obstacle velocity information; the target point state information includes target velocity information and target position information; the step of calculating the attraction velocity of the target point and the repulsion velocity of the obstacle using an artificial potential field method based on the robotic arm state information, the obstacle state information, and the target point state information, and obtaining the first control action of the robotic arm, specifically includes:

[0017] Based on the target velocity information, the target position information, and the first joint velocity information and first joint position information of the robotic arm end joint, a gravitational potential field of the target point is established, and the attraction velocity is calculated.

[0018] Based on the obstacle location information and the joint position information of each joint of the robotic arm, the distance between the robotic arm joint and the obstacle location is calculated, and a reference joint point is determined; the reference joint point is the robotic arm joint point that is closest to the obstacle.

[0019] Based on the obstacle position information, obstacle velocity information, and the second joint velocity and second joint position information of the reference joint, a repulsive potential field is established at the obstacle position, and the repulsive velocity is calculated.

[0020] The attraction velocity and the repulsion velocity are mapped to the joint space of the robotic arm, and the Jacobian matrix is ​​used to perform vector superposition of the attraction velocity and the repulsion velocity to obtain the first control action.

[0021] In some embodiments, the step of obtaining the second control action of the robotic arm using a deep reinforcement learning algorithm based on the robotic arm's state information specifically includes:

[0022] Initialize the deep reinforcement learning algorithm parameters; these parameters include the state space S, action space A, Actor network and Critic network architecture, reward function R, experience replay pool capacity L, learning rate α, future reward discount λ, and noise parameter N. t ;

[0023] Based on the robotic arm's state information, the parameters of the deep reinforcement learning algorithm are updated, and the second control action is output through the deep reinforcement learning algorithm.

[0024] In some embodiments, the step of superimposing the first control action and the second control action to obtain the real-time control action of the robotic arm is specifically expressed by the following formula:

[0025] a t =μa 1t +a 2t

[0026] Where μ is the weighting coefficient of the velocity potential field, a 1t For the first control action, a 2t For the second control action, a t This refers to the real-time control action.

[0027] In some embodiments, the step of establishing the gravitational potential field of the target point based on the target velocity information, the target position information, and the first joint velocity and position information of the robotic arm's end effector joint, and calculating the attraction velocity, is specifically expressed by the following formula:

[0028] V att =k1(P t -P g )+k2(V t -V g )

[0029] Among them, P t P is the position vector of the end effector joint of the robotic arm. g V is the position vector of the target point. t V is the velocity vector of the end joint of the robotic arm. g Let k1 be the velocity vector of the target point, k2 be the position weighting coefficient, and k2 be the velocity weighting coefficient.

[0030] In some embodiments, the deep reinforcement learning algorithm specifically includes:

[0031] State space S = (θ, P, D, T)

[0032] Where θ is the set of velocities of each joint of the robotic arm, and is derived from the velocities of each joint θ i Composition, θ i Let P represent the joint velocity of the i-th joint, P represent the pose of the target point, D represent the pose of the obstacle, and T represent whether the task has been completed.

[0033] Action space A = (ω1, ω2, ..., ω i )

[0034] Where, ωi ω represents the angular velocity of each joint of the robotic arm;

[0035] The Actor network architecture is 100*256*256*1, with ReLU function used as the activation function for the hidden layer and tanh function used as the activation function for the output layer.

[0036] The Critic network architecture is 100*256*256*1. The hidden layers use the ReLU function as the activation function, and the output layer uses the tanh function as the activation function.

[0037] Reward function R = R1 + R2 + R3 + R4

[0038]

[0039] Where R1 represents the negative value of the distance from the robotic arm's end effector to the target as a penalty, used to train the robotic arm to approach the target point; R2 is the binary reward given for whether the robotic arm's end effector reaches the target position, guiding the robotic arm to reach the target point; η represents the repulsion factor; and d... i R3 represents the distance from the i-th obstacle to the end of the robotic arm, ρ represents the range of influence of the obstacle, R3 uses a repulsive potential field to construct a reward function, and R4 represents the distance from the end of the robotic arm to the obstacle as a penalty, which is used to guide the robotic arm away from the obstacle. R4 is used to penalize the robotic arm for collisions and guide the robotic arm to avoid collisions.

[0040] In some embodiments, the step of determining whether the robotic arm should continue performing the path planning task based on the detection result specifically includes:

[0041] When the detection result is no collision, the robotic arm is controlled to perform the path planning task according to the real-time control action, collect the action experience set, and store the action experience set in the experience replay pool of the deep reinforcement learning algorithm; the action experience set includes the real-time control action, the current state of executing the real-time control action, the reward value of completing the real-time control action, and the transition state after completing the real-time control action.

[0042] When the detection result indicates a collision, the robotic arm is controlled to stop executing the path planning task, the real-time control action is regenerated, and then the robotic arm is controlled to continue executing the path planning task.

[0043] In some embodiments, the step of establishing a repulsive potential field at the obstacle position based on the obstacle position information, obstacle velocity information, and the second joint velocity and second joint position information of the reference joint, and calculating the repulsive velocity, is specifically expressed by the following formula:

[0044] When an obstacle is present, the repulsion velocity is calculated using the following formula:

[0045] Formula 1:

[0046] Equation 2: v p =||V M -V ob ||

[0047] Formula 3:

[0048] Among them, v p V represents the velocity vector of the reference joint. M and the velocity vector V of the obstacle ob difference, V represents M and V ob The normal velocity, when the robotic arm does not collide with the obstacle, is given in Equation 1. Without changing the direction of the repulsive velocity, when the robotic arm collides with the obstacle, Equation 3 is used to calculate Equation 1. The coefficient K represents the range of the repulsion velocity when there are multiple obstacles, which will vary depending on the distance and speed of the obstacles. The constant ρ represents the range of the repulsion velocity. When the shortest distance d between the obstacle and the joint is less than ρ, the smaller d is, the greater the repulsion velocity. When d is greater than ρ, the repulsion velocity is equal to 0.

[0049] When multiple obstacles are present, the repulsion velocity is calculated using the following formula:

[0050]

[0051] Among them, V reqi V is the repulsion velocity at the i-th obstacle. reqi It is obtained by calculation using Equation 1, Equation 2, and Equation 3.

[0052] In some embodiments, the collision detection method is the AABB-box algorithm.

[0053] This application provides a real-time control method for a robotic arm based on deep reinforcement learning in dynamic environments. It acquires the state information of the robotic arm, obstacles, and the target point. Using an artificial potential field method, it calculates the attraction velocity of the target point and the repulsion velocity of the obstacle to obtain the first control action of the robotic arm. Then, using a deep reinforcement learning algorithm, it obtains the second control action. Based on the first and second control actions, the actions are superimposed to obtain the real-time control action of the robotic arm, enabling path planning. This application combines the advantages of deep reinforcement learning and the artificial potential field method. It utilizes a deep reinforcement learning network to learn the robotic arm's motion strategy and generates appropriate potential forces using the artificial potential field method to guide the robotic arm's movement. This enables more precise robotic arm movement in complex control environments, improves the efficiency of deep reinforcement learning, enhances the obstacle avoidance capability and learning efficiency of the robotic arm when performing path planning tasks in unstructured complex environments, better perceives and responds to dynamically changing obstacles, and can make timely avoidance actions when facing dynamic obstacles, achieving precise and efficient control of the robotic arm's movement in dynamic environments. Attached Figure Description

[0054] Figure 1 This is a flowchart of a real-time control method for a robotic arm based on deep reinforcement learning in a dynamic environment, provided in an embodiment of this application.

[0055] Figure 2 This is a flowchart illustrating the real-time control actions output using an artificial potential field and deep reinforcement learning algorithm in the embodiments of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0059] Reference Figure 1 , Figure 1 This is an optional flowchart of a real-time control method for a robotic arm based on deep reinforcement learning in a dynamic environment, provided by an embodiment of this application. The method may include, but is not limited to, steps S101 to S105:

[0060] Step S101: Obtain the status information of the robotic arm, the status information of the obstacle, and the status information of the target point;

[0061] Step S102: Based on the state information of the robotic arm, the state information of the obstacle, and the state information of the target point, the artificial potential field method is used to calculate the attraction velocity of the target point and the repulsion velocity of the obstacle to obtain the first control action of the robotic arm.

[0062] Step S103: Based on the state information of the robotic arm, a deep reinforcement learning algorithm is used to obtain the second control action of the robotic arm;

[0063] Step S104: Based on the first control action and the second control action, the actions are superimposed to obtain the real-time control action of the robotic arm;

[0064] Step S105: Control the robotic arm to perform path planning tasks based on real-time control actions.

[0065] In step S101 of some embodiments, the path planning task of the robotic arm is first determined, that is, planning the path of the robotic arm from the starting point to avoid obstacles and reach the target. Then, the state of the robotic arm is initialized. At the same time, obstacle state information and target point state information are obtained through sensors. The robotic arm state information includes joint angle information, joint position information, and joint velocity information of each joint; the obstacle state information includes obstacle position information and obstacle velocity information; the target point state information includes target velocity information and target position information.

[0066] In some embodiments, the above method further includes steps S106 to S107:

[0067] Step S106: Using a collision detection method, detect whether the robotic arm can reach the target point without collision when performing real-time control actions, and determine the detection result;

[0068] Step S107: Based on the detection results, determine whether the robotic arm should continue to perform the path planning task.

[0069] In some embodiments, step S102 may include, but is not limited to, steps S201 to S204:

[0070] Step S201: Based on the target velocity information, target position information, and the first joint velocity information and first joint position information of the robotic arm end joint, establish the gravitational potential field of the target point and calculate the attraction velocity.

[0071] Step S202: Based on the obstacle position information and the joint position information of each joint of the robotic arm, calculate the distance between the robotic arm joint and the obstacle position, and determine the reference joint point;

[0072] Step S203: Based on the obstacle position information, obstacle velocity information, and the second joint velocity and second joint position information of the reference joint, establish the repulsive potential field of the obstacle position and calculate the repulsive velocity.

[0073] Step S204: Map the attraction velocity and repulsion velocity to the joint space of the robotic arm, and use the Jacobian matrix to perform vector superposition of the attraction velocity and repulsion velocity to obtain the first control action.

[0074] In step S201 of some embodiments, the gravitational potential field of the target point is established, and the attraction velocity is calculated, specifically expressed by the following formula:

[0075] V att =k1(P t -P g )+k2(V t -V g )

[0076] Among them, P t P is the position vector of the end effector joint of the robotic arm. g V is the position vector of the target point. t V is the velocity vector of the end joint of the robotic arm. g Let k1 be the velocity vector of the target point, k2 be the position weighting coefficient, and k2 be the velocity weighting coefficient.

[0077] In step S202 of some embodiments, the reference joint point is the robotic arm joint point closest to the obstacle. A repulsive potential field is established at the obstacle's location, and the repulsive velocity is calculated, specifically expressed by the following formula:

[0078] When an obstacle is present, the repulsion velocity is calculated using the following formula:

[0079] Formula 1:

[0080] Equation 2: v p =||V M -V ob ||

[0081] Formula 3:

[0082] Among them, v p V represents the velocity vector of the reference joint. M and the velocity vector V of the obstacle ob difference, V representsM and V ob The normal velocity, when the robotic arm does not collide with the obstacle, is given in Equation 1. Without changing the direction of the repulsive velocity, when the robotic arm collides with the obstacle, Equation 3 is used to calculate Equation 1. The coefficient K represents the range of the repulsion velocity when there are multiple obstacles, which will vary depending on the distance and speed of the obstacles. The constant ρ represents the range of the repulsion velocity. When the shortest distance d between the obstacle and the joint is less than ρ, the smaller d is, the greater the repulsion velocity. When d is greater than ρ, the repulsion velocity is equal to 0.

[0083] When multiple obstacles exist, the repulsion velocity is calculated using the following formula:

[0084]

[0085] Among them, V reqi V is the repulsion velocity at the i-th obstacle. reqi The above formulas 1, 2 and 3 are used to calculate the results.

[0086] In step S204 of some embodiments, the attraction velocity and repulsion velocity are mapped to the joint space of the robotic arm, specifically expressed by the following formula:

[0087]

[0088] Among them, a att To increase the attraction speed V att The joint velocity mapped to the joint space of the robotic arm, a req To reduce the repulsion velocity V req Joint velocities mapped to the joint space of the robotic arm, J -1 To attract the pseudo-inverse of the Jacobian matrix of the velocity, J # To exclude the pseudo-inverse of the Jacobian matrix for velocity.

[0089] Since the attraction and repulsion velocities are constructed at different locations, they are mapped to the joint space of the robotic arm and then vector-superimposed to obtain the first control action, specifically expressed by the following formula:

[0090]

[0091] Among them, a 1t For the first control action, a att To increase the attraction speed V att The joint velocity mapped to the joint space of the robotic arm, a reqi To determine the repulsion velocity V of the i-th obstacle reqi Joint velocities mapped to the joint space of the robotic arm.

[0092] In step S103 of some embodiments, the deep reinforcement learning algorithm parameters include the state space S, action space A, Actor network and Critic network architecture, reward function R, experience replay pool capacity L, learning rate α, future reward discount λ, and noise parameter N. t .

[0093] Specifically, the state space is S = (θ, P, D, T).

[0094] Where θ is the set of velocities of each joint of the robotic arm, and is derived from the velocities of each joint θ i Composition, θ i Let P represent the joint velocity of the i-th joint, P represent the pose of the target point, D represent the pose of the obstacle, and T represent whether the task has been completed.

[0095] Action space A = (ω1, ω2, ..., ω i )

[0096] Where, ω i ω represents the angular velocity of each joint of the robotic arm;

[0097] The Actor network architecture is 100*256*256*1, with ReLU function used as the activation function for the hidden layer and tanh function used as the activation function for the output layer.

[0098] The Critic network architecture is 100*256*256*1. The hidden layers use the ReLU function as the activation function, and the output layer uses the tanh function as the activation function.

[0099] Reward function R = R1 + R2 + R3 + R4

[0100]

[0101] Where R1 represents the negative value of the distance from the robotic arm's end effector to the target as a penalty, used to train the robotic arm to approach the target point; R2 is the binary reward given for whether the robotic arm's end effector reaches the target position, guiding the robotic arm to reach the target point; η represents the repulsion factor; and d... i R3 represents the distance from the i-th obstacle to the end of the robotic arm, ρ represents the range of influence of the obstacle, R3 uses a repulsive potential field to construct a reward function, and R4 represents the distance from the end of the robotic arm to the obstacle as a penalty, which is used to guide the robotic arm away from the obstacle. R4 is used to penalize the robotic arm for collisions and guide the robotic arm to avoid collisions.

[0102] In some embodiments, step S103 may include, but is not limited to, steps S301 to S302:

[0103] Step S301: Initialize the parameters of the deep reinforcement learning algorithm;

[0104] Step S302: Update the parameters of the deep reinforcement learning algorithm based on the state information of the robotic arm, and output the second control action through the deep reinforcement learning algorithm.

[0105] In step S302 of some embodiments, robotic arm state information is obtained, and a state value s is determined based on the robotic arm state information. t s t ∈S, the state value s t As input to the Actor network, the Actor network selects actions that maximize the Q-value as output. The Q-value is used in deep reinforcement learning to evaluate the current state value s. t The value of the selected action is given by the Actor network, and the action value output by the Actor network is a. 2t =argmax a Q(s t ,a'),a 2t The second control action is α, which represents the action parameter attribute in the deep reinforcement learning algorithm, and a' is the current state value s. t All available actions.

[0106] In some embodiments, step S104 is specifically represented by the following formula:

[0107] a t =μa 1t +a 2t

[0108] Where μ is the weighting coefficient of the velocity potential field, a 1t For the first control action mentioned above, a 2t For the second control action mentioned above, a t The above refers to the real-time control actions.

[0109] In step S106 of some embodiments, the collision detection method is the AABB-box algorithm, which uses regular geometric shapes to wrap the object, and the collision detection of two objects becomes whether the two regular geometric shapes overlap.

[0110] In some embodiments, step S107 may include, but is not limited to, steps S401 to S402:

[0111] Step S401: When the detection result is no collision, control the robotic arm to perform path planning tasks according to the real-time control action, collect action experience set, and store the action experience set in the experience replay pool in the deep reinforcement learning algorithm.

[0112] In step S402, when the detection result indicates a collision, the robotic arm is controlled to stop executing the path planning task, real-time control actions are regenerated, and then the robotic arm is controlled to continue executing the path planning task.

[0113] In step S401 of some embodiments, the action experience set includes a real-time control action, the current state of executing the real-time control action, the reward value for completing the real-time control action, and the transition state after completing the real-time control action. Optionally, the action experience set is (s t ,a t ,r t ,s t+1 ), where s t To determine the current state for executing real-time control actions, a t For real-time control of actions, r t To achieve the reward value for real-time control actions, s t+1 This is the transition state after completing a real-time control action.

[0114] If the number of action experience sets in the experience replay pool reaches its capacity L, then the oldest action experience set is deleted to store the newest experience set, and action experience sets (s) are collected in batches from the experience replay pool. i ,a i ,r i ,s i+1 The parameters of the deep reinforcement learning algorithm are updated during training.

[0115] In step S402 of some embodiments, when the detection result is that a collision has occurred, steps S101 to S104 are executed again, and a real-time control action is output. According to the real-time control action, the robotic arm is controlled to perform a path planning task, while collision detection is performed. Steps S401 to S402 are executed until the robotic arm can complete the path planning task without collision. When the robotic arm reaches the target point or the path planning time is greater than a preset threshold T, the path planning task is terminated.

[0116] Reference Figure 2 , Figure 2This is an optional flowchart of an embodiment of the present application that uses an artificial potential field method and a deep reinforcement learning algorithm to output real-time control actions. The flowchart describes the process of obtaining a state S, which includes the state information of the robotic arm, obstacles, and target point. Using the artificial potential field method, a first control action A1 is obtained from state S. State S is input into the Actor network, which outputs a second control action A2. Real-time control action A is obtained based on the first and second control actions A1 and A2. The Critic network calculates the value Q obtained by selecting action A under the current state S1. The value Q is returned to the Actor network, causing the Actor network to select and output the second control action A2 that maximizes the value Q. Real-time control action A interacts with the environment, updates state S1, enters the next state S2, and obtains a reward value R. An action experience set (S1, A, R, S2) is collected and stored in an experience replay pool. Action experience sets (S1, A, R, S2) are then collected in batches from the experience replay pool. i A i R i S i+1 Train the Actor network and the Critic network.

[0117] This application provides a real-time control method for a robotic arm based on deep reinforcement learning in a dynamic environment. It acquires the state information of the robotic arm, obstacles, and the target point. Using an artificial potential field method, it calculates the attraction velocity of the target point and the repulsion velocity of the obstacle to obtain the first control action of the robotic arm. Then, using a deep reinforcement learning algorithm, it obtains the second control action. Based on the first and second control actions, the actions are superimposed to obtain the real-time control action of the robotic arm, enabling path planning. This application combines the advantages of deep reinforcement learning and the artificial potential field method. It utilizes a deep reinforcement learning network to learn the robotic arm's motion strategy and generates appropriate potential forces using the artificial potential field method to guide the robotic arm's movement. This enables more precise robotic arm movement in complex control environments, improves the efficiency of deep reinforcement learning, enhances the obstacle avoidance capability and learning efficiency of the robotic arm when performing path planning tasks in unstructured complex environments, better perceives and responds to dynamically changing obstacles, and can make timely avoidance actions when facing dynamic obstacles, achieving precise and efficient control of the robotic arm's movement in dynamic environments.

[0118] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0119] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0120] The above description, with reference to the accompanying drawings, illustrates optional embodiments of this application and does not limit the scope of the claims of this application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of this application shall be within the scope of the claims of this application.

Claims

1. A real-time control method for a robotic arm based on deep reinforcement learning in a dynamic environment, characterized in that, include: Acquire the status information of the robotic arm, obstacles, and target points; The robotic arm status information includes joint angle information, joint position information, and joint speed information for each joint; Based on the state information of the robotic arm, the state information of the obstacle, and the state information of the target point, the attraction velocity of the target point and the repulsion velocity of the obstacle are calculated using the artificial potential field method. The attraction velocity and the repulsion velocity are mapped to the joint space of the robotic arm. The Jacobian matrix is ​​used to perform the vector superposition of the attraction velocity and the repulsion velocity to obtain the first control action of the robotic arm. Based on the robotic arm's state information, a deep reinforcement learning algorithm is used to obtain the robotic arm's second control action, specifically including: initializing the deep reinforcement learning algorithm parameters; the deep reinforcement learning algorithm parameters include the state space S, action space A, Actor network and Critic network architecture, reward function R, experience replay pool capacity L, and learning rate. Future reward discounts and noise parameters Based on the robotic arm's state information, update the parameters of the deep reinforcement learning algorithm, and output the second control action through the deep reinforcement learning algorithm. The deep reinforcement learning algorithm specifically includes: state space in, This is the set of velocities of each joint of the robotic arm, derived from the velocities of each joint. composition, Let P represent the joint velocity of the i-th joint, P represent the pose of the target point, D represent the pose of the obstacle, and T represent whether the task has been completed. Action space in, ω represents the angular velocity of each joint of the robotic arm; The Actor network architecture is 100*256*256*1, with ReLU function used as the activation function for the hidden layer and tanh function used as the activation function for the output layer. The Critic network architecture is 100*256*256*1. The hidden layers use the ReLU function as the activation function, and the output layer uses the tanh function as the activation function. reward function in, Negative values ​​representing the distance from the robotic arm's end effector to the target are used as penalties to train the robotic arm to approach the target point. A binary reward is given for whether the robotic arm's end effector reaches the target position, guiding the robotic arm to the target point. Represents the repulsion factor. This represents the distance from the i-th obstacle to the end effector of the robotic arm. Indicates the range of influence of the obstacle. A repulsive potential field is used to construct the reward function, representing the distance from the robotic arm's end effector to the obstacle as a penalty, which is then used to guide the robotic arm away from the obstacle. Penalize the robotic arm for collisions and guide it to avoid collisions; Based on the first control action and the second control action, the actions are superimposed to obtain the real-time control actions of the robotic arm; Based on the real-time control actions, the robotic arm is controlled to perform path planning tasks.

2. The real-time control method for a robotic arm according to claim 1, characterized in that, Also includes: A collision detection method is used to detect whether the robotic arm reaches the target point without collision when performing the real-time control action, and the detection result is determined. Based on the detection results, it is determined whether the robotic arm should continue to perform the path planning task.

3. The real-time control method for a robotic arm according to claim 1, characterized in that, The obstacle status information includes obstacle position information and obstacle speed information; The target point state information includes target velocity information and target position information; the step of calculating the attraction velocity of the target point and the repulsion velocity of the obstacle using an artificial potential field method based on the robotic arm state information, the obstacle state information, and the target point state information, and obtaining the first control action of the robotic arm, specifically includes: Based on the target velocity information, the target position information, and the first joint velocity information and first joint position information of the robotic arm end joint, a gravitational potential field of the target point is established, and the attraction velocity is calculated. Based on the obstacle location information and the joint position information of each joint of the robotic arm, the distance between the robotic arm joint and the obstacle location is calculated, and a reference joint point is determined; the reference joint point is the robotic arm joint point that is closest to the obstacle. Based on the obstacle position information, obstacle velocity information, and the second joint velocity and second joint position information of the reference joint, a repulsive potential field is established at the obstacle position, and the repulsive velocity is calculated.

4. The real-time control method for a robotic arm according to claim 1, characterized in that, The step of superimposing the first control action and the second control action to obtain the real-time control action of the robotic arm is specifically expressed by the following formula: in, It is the velocity potential field weighting coefficient. This is the first control action. This is the second control action. This refers to the real-time control action.

5. The real-time control method for a robotic arm according to claim 3, characterized in that, The step of establishing the gravitational potential field of the target point and calculating the attraction velocity based on the target velocity information, the target position information, the first joint velocity information and the first joint position information of the robotic arm's end joint is specifically expressed by the following formula: in, This is the position vector of the end joint of the robotic arm. The position vector of the target point. The velocity vector of the end joint of the robotic arm. The velocity vector of the target point. For location weighting coefficients, This is the speed weighting coefficient.

6. The real-time control method for a robotic arm according to claim 2, characterized in that, The step of determining whether the robotic arm should continue to perform the path planning task based on the detection results specifically includes: When the detection result is no collision, the robotic arm is controlled to perform the path planning task according to the real-time control action, collect the action experience set, and store the action experience set in the experience replay pool of the deep reinforcement learning algorithm; the action experience set includes the real-time control action, the current state of executing the real-time control action, the reward value of completing the real-time control action, and the transition state after completing the real-time control action. When the detection result indicates a collision, the robotic arm is controlled to stop executing the path planning task, the real-time control action is regenerated, and then the robotic arm is controlled to continue executing the path planning task.

7. The real-time control method for a robotic arm according to claim 3, characterized in that, The step of establishing a repulsive potential field at the obstacle's position based on the obstacle's position information, obstacle velocity information, and the second joint velocity and second joint position information of the reference joint, and calculating the repulsive velocity, is specifically expressed by the following formula: When an obstacle is present, the repulsion velocity is calculated using the following formula: Formula 1: Formula 2: Formula 3: in, Represents the velocity vector of the reference joint. and the velocity vector of the obstacle difference, express and The normal velocity, when the robotic arm does not collide with the obstacle, is given in Equation 1. =1, without changing the direction of the repulsive velocity, when the robotic arm collides with the obstacle, then Equation 3 is activated to calculate Equation 1. The coefficient K represents the constant K when there are multiple obstacles, as the distance and speed of the obstacles will vary. This indicates the range of the repulsion velocity effect, when the shortest distance d between the obstacle and the joint is less than... When d is smaller, the repulsion velocity is greater; when d is greater than 0, the repulsion velocity is greater. The repulsion velocity is equal to 0. When multiple obstacles are present, the repulsion velocity is calculated using the following formula: in, Let be the repulsion velocity at the i-th obstacle. It is obtained by calculation using Equation 1, Equation 2, and Equation 3.

8. The real-time control method for a robotic arm according to claim 2, characterized in that, The collision detection method is the AABB-box algorithm.

Images