A method and system for dynamic obstacle avoidance of a robot arm based on safety-reinforced learning

Abstract

The application relates to a mechanical arm dynamic obstacle avoidance method and system based on safety reinforcement learning, which comprises the following steps: taking a mechanical arm motion state, a target position and dynamic obstacle information as a state of reinforcement learning, taking angular velocity control amounts of each joint as actions of reinforcement learning, taking a multi-target function as a reward function of reinforcement learning, and performing policy iteration on a reinforcement learning model through proximal policy optimization; obtaining the state of reinforcement learning, obtaining actions by using the reinforcement learning model after policy iteration, and taking the actions as a reinforcement learning policy; based on the state of reinforcement learning and the reinforcement learning policy, taking finding a minimum control correction amount to make the system meet safety constraints again as a target, obtaining an optimal control correction amount as a policy compensation; based on the reinforcement learning policy and the policy compensation, obtaining a final control policy and executing the final control policy; and repeating the execution until the mechanical arm reaches the target position, so that dynamic obstacle avoidance of the mechanical arm is realized.

Description

Technical Field

[0001] This invention relates to the field of robotic arm control technology, and in particular to a dynamic obstacle avoidance method and system for robotic arms based on safety reinforcement learning. Background Technology

[0002] Currently, researchers have proposed various path planning and control methods for obstacle avoidance in robotic arms, such as the artificial potential field method, the Rapidly-exploring Random Tree (RRT) algorithm, and model predictive control. These methods can achieve effective path planning in static environments, but still have certain limitations in dynamic environments. Furthermore, reinforcement learning methods have been increasingly applied to robot control in recent years, enabling robotic arms to learn autonomous decision-making strategies through interaction with the environment. However, existing methods still have certain shortcomings in terms of safety and stability, specifically in the following aspects: (1) Insufficient real-time planning capability in dynamic environments. When traditional path planning methods (such as RRT, RRT*, A*, etc.) search in the high-degree-of-freedom configuration space of a robotic arm, the computational complexity increases rapidly with the increase of degrees of freedom and environmental complexity. In environments with dynamic obstacles, the system often needs to continuously update or replan the path, thereby increasing the computational burden and reducing the system response speed, making it difficult to meet the real-time requirements of dynamic obstacle avoidance tasks.

[0003] (2) Insufficient safety of reinforcement learning methods. Reinforcement learning methods can learn control strategies through interaction with the environment and have strong adaptability in complex environments. However, in the early stages of training or before the strategy has converged, reinforcement learning methods may produce unsafe exploratory behaviors, such as the robotic arm approaching obstacles or even colliding. Such unsafe exploration not only affects the stability of training, but may also pose safety risks to equipment and the environment in practical applications.

[0004] (3) Lack of effective safety constraint mechanisms. Most existing reinforcement learning obstacle avoidance methods rely on reward functions to constrain the safe distance. However, this method is usually a soft constraint. When the environment is complex or the obstacle moves at a high speed, it is difficult to ensure that the system always meets the safety requirements by relying solely on the reward function. Therefore, in complex dynamic environments, there may still be situations where the robotic arm gets too close to the obstacle or even collides with it.

[0005] (4) Insufficient control stability in high-dimensional state space. Dynamic obstacle avoidance tasks of robotic arms usually require processing multi-dimensional data such as the joint state of the robotic arm, the end-effector pose information, and the obstacle motion information at the same time. In high-dimensional state space, traditional control methods are difficult to effectively process complex environmental information, while simple reinforcement learning methods may produce unstable control actions in the absence of safety constraints, affecting the smoothness and stability of the robotic arm's motion.

[0006] Chinese patent application publication number CN120503196A discloses a visual servo control method and system for a robotic arm based on deep reinforcement learning. This method utilizes the maximum entropy deep reinforcement learning SAC algorithm and a real-time trajectory interpolation strategy to address the shortcomings of traditional methods in real-time performance and adaptability in dynamic environments, achieving stable and precise obstacle avoidance and target tracking for the robotic arm. However, this method still has certain limitations: First, while policy learning based on Soft Actor-Critic (SAC) possesses good exploratory capabilities, it is essentially an expectation optimization method, lacking strict constraints on system safety. It struggles to provide provable safety guarantees in complex dynamic environments, especially under conditions of rapid obstacle movement or sudden interference, potentially leading to risky control actions. Second, the visual servoing process is highly dependent on perceptual information. When visual information is noisy, delayed, or occluded, it can easily lead to unstable policy output, affecting control accuracy and system robustness. Furthermore, although the real-time trajectory interpolation strategy improves trajectory smoothness to some extent, it is primarily a post-processing method and cannot suppress abrupt changes in the reinforcement learning strategy at the decision-making level, potentially causing joint velocity oscillations in the robotic arm.

[0007] Furthermore, this method primarily relies on a single reinforcement learning strategy for decision-making, lacking deep integration with model-driven safety mechanisms (such as obstacle control functions), making it difficult to balance task performance with stringent safety constraints. Simultaneously, its handling of dynamic obstacles largely depends on current observation information, lacking effective modeling of future motion trends, leaving room for improvement in its foresight and adaptability in highly dynamic and uncertain environments.

[0008] Therefore, how to introduce effective safety constraint mechanisms on the basis of reinforcement learning control strategies so that the robotic arm can maintain high decision-making efficiency and ensure the safety of system operation in dynamic environments has become a key issue in the current research on dynamic obstacle avoidance of robotic arms. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of the prior art by providing a dynamic obstacle avoidance method and system for robotic arms based on safety reinforcement learning, so as to solve or partially solve the problem that existing methods cannot simultaneously ensure safety and stability.

[0010] The objective of this invention can be achieved through the following technical solutions: One aspect of the present invention provides a dynamic obstacle avoidance method for a robotic arm based on safety reinforcement learning, comprising the following steps: Step S1: The state of the robotic arm's motion, the target position, and the dynamic obstacle information are used as the state of reinforcement learning. The angular velocity control of each joint is used as the action of reinforcement learning. A multi-objective function including collision penalty, tracking error penalty, collision risk penalty, and termination reward is used as the reward function of reinforcement learning. The reinforcement learning model is iterated through proximal policy optimization. Step S2: Obtain the state of reinforcement learning, and use the reinforcement learning model after policy iteration to obtain the action as the reinforcement learning policy; Step S3: Based on the state of the reinforcement learning and the reinforcement learning policy, with the goal of finding the minimum control correction amount that makes the system meet the safety constraints again, the optimal control correction amount is obtained as policy compensation. Step S4: Based on the reinforcement learning strategy and the strategy compensation, perform strategy optimization, integrate CBF information into the reinforcement learning training process, form a guidance control mechanism, obtain the final control strategy and execute it; Step S5: Return to step S3 until the robotic arm reaches the target position, thus achieving dynamic obstacle avoidance by the robotic arm.

[0011] As a preferred technical solution, the strategy compensation is obtained by solving the following problem: in, express Minimum control correction amount at any given time. The hyperparameter represents the range of variation of the probability ratio. The objective is to optimize the objective function or the pruned strategy. , The new strategy and the old strategy are respectively in the state. The following action probability distribution This indicates the transpose operation. Reinforcement learning Moment State The corresponding reinforcement learning strategy, item As the first The second iteration or the... The control obstacle function compensation term corresponding to each safety correction component For the number of iterations of the safety compensation term, The attenuation rate, For the control barrier function, For the system in state The control input mapping matrix below, , The first The lower and upper limits of each component. This represents the total number of components.

[0012] As a preferred technical solution, the tracking error penalty is: in, for Real-time tracking error penalty These are weighting coefficients used to adjust the trend of the reward function across different error intervals. Distance threshold for The Euclidean distance between the end effector of the robotic arm and the target point at any given time. The current position of the end effector. The location of the target point.

[0013] As a preferred technical solution, the collision risk penalty is: in, for Constant collision risk and punishment The Euclidean distance between the end effector position and the obstacle position. For safe distance threshold, The current position of the end effector. This indicates the location of the obstacle.

[0014] As a preferred technical solution, after obtaining the reinforcement learning strategy, the method further includes: Obtain the reinforcement learning policies corresponding to the current time step and the previous time step, and obtain the reinforcement learning policy sequence through linear interpolation; Based on the reinforcement learning policy sequence, the final reinforcement learning policy is obtained through Butterworth low-pass filtering.

[0015] As a preferred technical solution, the linear interpolation is implemented using the following formula: in, for Reinforcement learning strategies at all times , Each is the previous moment Current moment The corresponding reinforcement learning strategy.

[0016] As a preferred technical solution, the Butterworth low-pass filter is implemented using the following formula: in, for The reinforcement learning policy sequence after low-pass filtering at each time step. The cutoff angular frequency, The system sampling period is for Reinforcement learning strategies at all times.

[0017] As a preferred technical solution, the process of iterating the policy of the reinforcement learning model through proximal policy optimization includes the following steps: Calculate the probability ratio between the current policy and the old policy; Based on the probability ratio and the advantage function based on generalized advantage estimation, calculate the shear target loss; Based on the shearing target loss, combined with the loss of the value network and the entropy regularization term, the policy iteration of the reinforcement learning model is realized through policy gradient update; During policy iteration, random noise is introduced during state observation to perturb the environmental state. During action execution, the control commands output by the policy are perturbed, and noise parameters are randomly sampled.

[0018] As a preferred technical solution, the robotic arm motion state includes the joint angles and joint velocities of the robotic arm, the target position includes the difference between the target position and the end effector position, and the difference between the target velocity and the end effector velocity, and the dynamic obstacle information includes the state of the obstacle.

[0019] Another aspect of the present invention provides a dynamic obstacle avoidance system for a robotic arm based on safety reinforcement learning, for implementing the aforementioned dynamic obstacle avoidance method for a robotic arm, the system comprising: The safety reinforcement learning module deploys a reinforcement learning model. It uses the robotic arm's motion state, target position, and dynamic obstacle information as the reinforcement learning state, the angular velocity control of each joint as the reinforcement learning action, and a multi-objective function including collision penalty, tracking error penalty, collision risk penalty, and termination reward as the reinforcement learning reward function. It iterates the reinforcement learning model through proximal policy optimization; it obtains the reinforcement learning state, uses the reinforcement learning model after policy iteration to obtain the action, and uses it as the reinforcement learning policy. The compensation control module is used to find the minimum control correction amount so that the system can meet the safety constraints again, based on the state of the reinforcement learning and the reinforcement learning policy, and obtain the optimal control correction amount as policy compensation. The guidance control module directly integrates CBF information into the reinforcement learning training process, forming a guidance control mechanism. Unlike compensatory control, the core objective of guidance control is to enable the reinforcement learning policy to proactively move towards a safe zone during the learning phase, rather than relying on later corrections. In other words, safety information not only affects the control execution layer but also participates in the policy optimization process. The iterative execution module is used to obtain and execute the final control policy based on the reinforcement learning policy and the policy compensation.

[0020] Compared with the prior art, the present invention has at least one of the following beneficial effects: (1) Balancing Safety and Stability: On the one hand, this invention introduces a multi-objective function, including collision penalty, tracking error penalty, collision risk penalty, and termination reward, as the reward function in the reinforcement learning model. The tracking error penalty improves the tracking accuracy of the robotic arm's end effector towards the target point and avoids the problem of insufficient driving force in the target neighborhood of the traditional linear distance reward function. The collision risk penalty prevents the robotic arm from exhibiting dangerous movement tendencies when approaching obstacles. The termination reward strengthens the policy's sensitivity to key behavioral outcomes. On the other hand, after obtaining the reinforcement learning policy, the invention combines the reinforcement learning state to find the minimum control correction amount that makes the system re-satisfy safety constraints, obtaining the optimal control correction amount as policy compensation. This together ensures the safety and stability of the output control policy.

[0021] (2) Avoiding learning stagnation due to excessively small gradient of the nonlinear function: The tracking error penalty of this invention is defined in a piecewise form. When the end effector of the robotic arm is within the target neighborhood, a nonlinear function consisting of quartic and cubic terms is used to characterize the distance error. This function has a larger gradient change rate when the error is small, thus providing stronger fine-tuning capability during policy learning and improving the positioning accuracy of the robotic arm when it reaches the target point. When the robotic arm is far from the target, a linear penalty is used to ensure that the policy still has a stable convergence direction when it is far from the target area, avoiding the problem of learning stagnation due to excessively small gradient of the nonlinear function.

[0022] (3) Mitigating the impact of sudden control actions on system stability: After obtaining the reinforcement learning strategy, this invention obtains the reinforcement learning strategies corresponding to the current time and the previous time. The reinforcement learning strategy sequence is obtained by linear interpolation. Based on the reinforcement learning strategy sequence, the final reinforcement learning strategy is obtained by Butterworth low-pass filtering. The control input of the robotic arm is processed by continuous processing using velocity interpolation method, and the joint velocity signal is filtered by Butterworth low-pass filter to suppress high-frequency disturbances and achieve smooth optimization of trajectory. Attached Figure Description

[0023] Figure 1This is a flowchart of the dynamic obstacle avoidance method for a robotic arm based on safety reinforcement learning in the embodiment. Figure 2 This is a schematic diagram of the security reinforcement learning framework in the embodiment; Figure 3 This is a schematic diagram of the iterative process of the guidance control strategy in the embodiment; Figure 4 This is a schematic diagram of the CBF boot control architecture in the embodiment; Figure 5 This is a schematic diagram of the iterative process of the compensation control strategy in the embodiment. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] Example 1 To address the problems existing in the aforementioned prior art, this embodiment provides a dynamic obstacle avoidance method for robotic arms based on safety reinforcement learning. Taking a six-axis robotic arm as an example, firstly, a simulation training environment is built in Pybullet; secondly, a deep reinforcement learning controller is designed; and thirdly, the agent is trained by constructing a state space, action space, and reward function in conjunction with the actual working environment and task. Finally, the simulation training results are transferred to a real robotic arm for verification.

[0026] See Figure 1 The method includes the following steps: Step S1: Build a deep reinforcement learning and robotic arm simulation environment.

[0027] In policy gradient-based reinforcement learning methods, maintaining the stability of the policy update process while ensuring learning efficiency is a key issue affecting algorithm performance. In traditional policy gradient methods, improper step size selection during the update process can easily lead to drastic changes in the policy distribution, causing training oscillations and even performance degradation.

[0028] The Proximal Policy Optimization (PPO) algorithm belongs to the on-policy policy optimization method under the Actor-Critic framework, and its core idea comes from Trust Region Policy Optimization (TRPO). Unlike TRPO, which constrains policy updates by limiting KL divergence, this embodiment uses a simpler and more efficient shearing objective function, thereby reducing implementation complexity while maintaining theoretical validity. Let the current policy be... The old strategy was Then construct the probability ratio function between the two: This ratio reflects the degree of change of the new policy relative to the old policy on a given state-action pair.

[0029] In the traditional policy gradient method, the optimization objective is: in This is the advantage function, used to measure the degree to which an action is better than the average. However, when... If the objective function is too large or too small, the policy update may deviate excessively from the old policy, thereby compromising training stability. To address this issue, this embodiment employs a clipped surrogate objective function: in This is a hyperparameter used to limit the range of variation of the probability ratio.

[0030] The core idea of ​​this objective function is to truncate the policy update step size when it exceeds a preset threshold, thereby preventing excessive gradient updates. This proximal update mechanism achieves effective control over the policy update step size without introducing complex constraint optimizations.

[0031] In actual training, the dominance function The calculation is performed using Generalized Advantage Estimation (GAE). Let the time difference error be: The advantage function can be expressed as: in This is an adjustment parameter to balance the bias and variance. As a discount factor, For state The state-value function is used to estimate the expected cumulative reward starting from the current state. By introducing generalized advantage estimation (GAE), the gradient variance is reduced while maintaining estimation accuracy, thereby improving training efficiency.

[0032] In addition to the policy network, a value network (Critic) is included to estimate the state-value function, whose loss function is defined as: in, For the parameter is The value function network in the state The output state value estimation The target value at the corresponding time point is generally obtained from the actual return or calculated based on the time-series difference method, and is used to guide the learning of the value function.

[0033] To enhance strategy exploration capabilities, an entropy regularization term is introduced: in, Entropy is the policy's entropy function, used to measure the uncertainty of the policy distribution. A larger value indicates that the policy has a stronger exploratory ability.

[0034] Based on the above, the overall optimization objective can be expressed as: in , These are the weighting coefficients.

[0035] The main advantages of this step are its simple structure, low implementation difficulty, lack of complex second-order optimization, improved training stability by limiting the policy update amplitude through the shearing mechanism, ability to optimize using the latest sampled data, and applicability to high-dimensional continuous action space control problems, making it one of the benchmark algorithms in the field of reinforcement learning.

[0036] Step S2: Design of safety adjustment mechanism for control barrier function.

[0037] CBF (Capability-Based Filter) compensatory control is a safety filter architecture. Its basic idea is that reinforcement learning is responsible for generating the control policy, while the CBF performs safety compensation on the actions before execution. Within this framework, the reinforcement learning policy still learns independently with the goal of maximizing long-term rewards, while the CBF controller, as an external safety module, only corrects actions when they violate safety constraints. The architecture is as follows: Figure 2 As shown. Due to reinforcement learning strategies It is random, therefore the controller Indicates policy iteration Implementation of the post-randomization strategy. The final execution control input is defined as: in This indicates a safety compensation item.

[0038] Model-free RL controller A control action is proposed to optimize long-term rewards, but it may not be safe. Prior to deploying the RL controller, the CBF controller... It filters suggested control actions and provides the minimum necessary control intervention to ensure overall controller safety. The system state is maintained in a safe centralization. In other words, the CBF controller projects the RL controller into a safe centralization.

[0039] Intuitively, the RL controller provides feedforward control, while the CBF controller compensates by providing minimal control required to achieve forward invariance of the safety set. If such control is unavailable, the CBF controller provides control to bring the state as close as possible to the safety setting. However, while the CBF controller guarantees safety, it does not actively guide the overall policy exploration of the controller because of the surrounding... The updated RL strategy is not deployed on The next learning iteration should focus on a strategy of secure deployment, not insecure deployment.

[0040] To further enhance the safety awareness of the strategy, CBF information can be directly integrated into the reinforcement learning training process to form a guided control mechanism. Unlike compensatory control, the core objective of guided control is to make the reinforcement learning strategy proactively move towards a safe zone during the learning phase, rather than relying on later corrections. In other words, safety information not only affects the control execution layer but also participates in the policy optimization process. The architecture is as follows: Figure 3 As shown. Updates to reinforcement learning policies should not revolve around the potentially insecure control signals output by the original policy, but rather around improvements to the actually deployed secure controller. This achieves a balance between security and learning efficiency. Since the deployed controller is always within the secure set... The control behavior generated by the internal operation is feasible and safe. Therefore, optimizing the strategy around it can effectively avoid the strategy from repeatedly exploring unsafe or invalid state regions, thereby improving sample utilization efficiency and convergence stability.

[0041] Let the first The reinforcement learning policy during the next policy iteration is: It is determined by a random strategy. Sampled; simultaneously, the control barrier function constrains the controller as . This is used to correct control signals that may violate safety constraints. In the initial iteration phase ( Reinforcement learning strategy outputs control signals The safety compensation term is obtained by solving the control obstacle function constraint optimization problem. This forms the actual execution controller: It is important to emphasize that the system executes not the original reinforcement learning actions, but rather control signals that have been security-corrected. To ensure that subsequent policy updates revolve around the deployed security controller, in the... During the next policy iteration, the overall controller is defined as: The key to this architecture lies in accumulating all historical CBF controllers and embedding them into the current reinforcement learning controller. This allows policy updates to no longer revolve around potentially insecure original policies, but rather around improvements to historically deployed secure control behaviors. Figure 4 As shown. Under this mechanism, the item This can be viewed as a "guided reinforcement learning controller," whose proposed control signals may still have a tendency to pose security risks, but ultimately are controlled by the current CBF controller. Constraints are modified to ensure the system meets safety conditions. Through this progressive embedding structure, historical safety modifications are gradually internalized into the policy structure, concentrating policy sampling distribution within the safe and feasible region and avoiding repeated entry into state spaces that clearly violate constraints. This improves sample utilization efficiency and training convergence stability while ensuring the system's forward invariance.

[0042] After gradually embedding the historical CBF controller into the reinforcement learning controller structure, in order to ensure that the overall controller still satisfies the safety constraints in each policy iteration, it is necessary to reconstruct the QP problem containing all historical compensation terms at the current moment: The solution to the current quadratic programming problem of the control barrier function defines the CBF controller. It ensures safety by satisfying forward invariance.

[0043] Step S3: Design of action space, state space, and reward function.

[0044] See Figure 5 To implement the reinforcement learning training process.

[0045] (1) Action Space: The action space defines how the agent controls the environment. Considering the continuous control characteristics of the robotic arm, this embodiment uses a continuous action space, defining the action as the angular velocity control quantity of each joint: in Let be the degrees of freedom of the robotic arm. To ensure the physical rationality of the control signals, the motion range is constrained within [-1, 1] rad / s, and the maximum change in each control cycle is set to 0.1 rad / s.

[0046] (2) State Space: The design of the state space directly affects the policy network's ability to perceive the environment. It mainly includes its own motion state, target position, and dynamic obstacle information. Its complete observation can be represented as follows: .in, This indicates the state of the robotic arm itself, including the joint angles of the robotic arm. and joint velocity The second part contains the target location. With end effector position The gap between them and the target speed and end effector speed The gap between them; the third part represents the state of the obstacle. .

[0047] (3) Reward Function Design: The reward function is the core of the reinforcement learning algorithm's convergence. To ensure the reliability of training, this embodiment designs a multi-objective reward function. The reward function not only guides the robotic arm to the target position but also constrains collision behavior, motion boundary violations, and trajectory optimization to ensure the stability and safety of the policy learning process. The overall form of the reward function can be expressed as: Penalties for collisions: This means that if the robotic arm's end effector collides with the ground or a dynamic obstacle, the strategy will receive a large penalty, so that the penalty received when a collision occurs can overshadow the reward.

[0048] To further improve the tracking accuracy of the robotic arm's end effector towards the target point and avoid the problem of insufficient driving force in the target neighborhood caused by the traditional linear distance reward function, a piecewise tracking error reward function based on the end effector's position error is designed. Let the Euclidean distance between the end effector of the robotic arm and the target point at the current moment be: in The current position of the end effector. The location of the target point.

[0049] The tracking error reward function is defined in the following piecewise form: in This is a distance threshold used to define the target's neighborhood. , which is the weighting coefficient, used to adjust the trend of the reward function in different error intervals.

[0050] When the end effector of the robotic arm is within the target neighborhood (i.e. When the distance error is large, a nonlinear function consisting of quartic and cubic terms is used to characterize the distance error. This function has a larger gradient rate of change when the error is small, thus providing stronger fine-tuning capabilities during policy learning and improving the positioning accuracy of the robotic arm when it reaches the target point. When the robotic arm is far from the target (i.e., When the nonlinear function gradient is too small, a linear penalty is used to ensure that the strategy has a stable convergence direction when it is far from the target region, thus avoiding the problem of learning stagnation caused by the nonlinear function gradient being too small.

[0051] By designing the piecewise function as described above, the reinforcement learning strategy has good global convergence ability in the early stage of training, and obtains higher tracking accuracy when approaching the target area, thereby effectively improving the target arrival performance in the trajectory planning and dynamic obstacle avoidance tasks of the robotic arm.

[0052] To prevent the robotic arm from exhibiting dangerous movement tendencies when approaching obstacles, an obstacle avoidance reward function is designed based on the relative spatial relationship between the obstacle and the robotic arm. The Euclidean distance between the end effector position and the obstacle position is expressed as: To avoid the risk of collision when the robotic arm approaches an obstacle area, this embodiment defines the obstacle avoidance reward function as a continuous penalty function based on distance: in, The system uses a safe distance threshold. When the robotic arm's end effector enters the danger zone of an obstacle, the system applies continuous penalties to the agent based on the change in distance between them. As the distance decreases, the penalty increases, effectively suppressing the robotic arm's tendency to approach the obstacle during policy optimization. Furthermore, when the robotic arm makes actual contact with the obstacle, the environment imposes a significant termination penalty to accelerate policy convergence to a safe and feasible control behavior. Compared to traditional sparse penalty mechanisms that rely solely on collision termination signals, the obstacle avoidance reward design based on continuously changing spatial distance provides effective risk feedback before a collision occurs, enabling the agent to learn forward-looking safe avoidance strategies during training. This improves the stability and generalization ability of the learned control strategy in dynamic obstacle avoidance tasks.

[0053] In order to enhance the sensitivity of the strategy to key behavioral outcomes during reinforcement learning training, this embodiment sets additional termination rewards for different termination scenarios at the end of the round. This design aims to reinforce successful behaviors while significantly penalizing dangerous or failed behaviors, thereby accelerating policy convergence and improving task completion rates. In dynamic obstacle avoidance tasks for robotic arms, round termination mainly includes the following situations: successfully reaching the target point, colliding with an obstacle or environment, exceeding the workspace, self-collision, and exceeding the maximum allowed number of steps. For the above termination situations, this embodiment defines the termination reward function as follows: When the end effector of the robotic arm successfully reaches the target point, a large positive reward is given to reinforce the goal-oriented behavior; when failures such as collision, boundary crossing, self-collision, or timeout occur, negative penalties are applied to guide the agent to avoid dangerous or ineffective behaviors.

[0054] Step S4: Design of a method for smoothing the motion trajectory of the robotic arm.

[0055] To ensure obstacle avoidance safety, the agent may make abrupt control actions at different decision points. These abrupt actions can cause drastic fluctuations in the joint speed of the robotic arm, leading to trajectory oscillations or even system instability. Therefore, it is necessary to smooth the trajectory of the control commands after the strategy output to improve the continuity and safety of the robotic arm's movement. This embodiment uses velocity interpolation to make the robotic arm control input continuous, and combines this with a Butterworth low-pass filter to filter the joint velocity signal to suppress high-frequency disturbances and achieve smooth trajectory optimization.

[0056] Butterworth Low-Pass Filter Design: In a reinforcement learning control framework, the policy network outputs the control actions of each joint of the robotic arm at each time step. Since deep reinforcement learning algorithms need to explore during training, their output action sequences typically contain high-frequency disturbances. Directly using this action signal as the joint control input may lead to oscillations or even control instability during the robotic arm's movement. Therefore, this embodiment introduces a Butterworth low-pass filter after the policy network output layer to smooth the joint control speed signal. The Butterworth low-pass filter features a flat passband response and no ripple, effectively suppressing high-frequency noise while maintaining signal amplitude characteristics, and is therefore widely used in mechanical system control. The reinforcement learning policy outputs the control actions of each joint at each time step. The output of the original joint velocity is The actual joint speed after filtering is Its difference expression is: in The cutoff angular frequency, Given the system sampling period, this difference equation is obtained by discretizing a continuous-time Butterworth low-pass filter through a bilinear transformation, which can effectively suppress high-frequency oscillations while ensuring real-time performance.

[0057] In the motion control of a robotic arm, the actions output by the reinforcement learning policy network are typically given in the form of discrete time-step joint velocities or position increments. Directly inputting this discrete action sequence into the actuator may lead to discontinuities in joint movement. Therefore, an interpolation method is used to process the discrete control commands into a continuous sequence. At time... and The time-policy network outputs the joint velocities as follows: and Then in the interval At any time within The joint velocity can be expressed by linear interpolation as: in The interpolated joint velocity is represented by the above interpolation process, which transforms the originally discrete joint velocity signal into a continuous time function, thereby effectively reducing the abrupt changes in control input and improving the smoothness of the robotic arm's movement.

[0058] Step S5: Training environment deployment and training results.

[0059] In tuning the PPO algorithm, it is important to focus on several key hyperparameters.

[0060] First, select an appropriate network structure, such as the number of layers, the number of neurons per layer, and the activation function, to ensure the model has sufficient expressive power but avoids overfitting. Second, optimize the learning rate and its scheduling strategy to avoid divergence or slow convergence during training. Specific parameters are shown in Table 1.

[0061] Table 1 Parameters of Deep Reinforcement Learning Controller The design of the training environment module is a key aspect of this embodiment. It organically integrates the kinematic model of the robotic arm with obstacle information in the environment into a public platform interface, thereby providing a highly flexible and scalable training environment for reinforcement learning algorithms. In this module, firstly, through precise kinematic modeling, the states of each joint of the robotic arm are correlated with the position and posture information of the end effector, ensuring that each time step in the environment accurately reflects the dynamic behavior and motion process of the robotic arm. Next, obstacle information is also incorporated into the training environment. By calculating the distance between obstacles and the robotic arm in real time, it ensures that the algorithm can correctly identify and avoid these obstacles during training.

[0062] In terms of state space design, this embodiment combines the joint states of the robotic arm, the target position, and the distance information to obstacles to construct a multi-dimensional state representation. Specifically, the joint states of the robotic arm include the angle and angular velocity information of each joint, which is crucial for describing the motion and control of the robotic arm. Simultaneously, the target position and obstacle distance information, as part of the state, enable the reinforcement learning model to have a comprehensive understanding of the current environment at every moment, thereby making reasonable decisions. This comprehensive state space design not only effectively reflects the dynamic changes in the environment but also improves the model's adaptability to target tracking and obstacle avoidance tasks.

[0063] In designing the reward function, this embodiment employs a multi-objective weighted approach, collaboratively optimizing the objectives of obstacle avoidance and tracking tasks. Specifically, the reward function includes two main objectives: first, to encourage the robotic arm to avoid collisions with obstacles as much as possible to ensure safe movement; and second, to guide the robotic arm to reach the target point as quickly and accurately as possible to complete the target tracking task. Based on this, by assigning different weights to these two objectives, the optimization of obstacle avoidance and tracking tasks is balanced, ensuring that the robotic arm can make reasonable trade-offs between different objectives when performing tasks. Furthermore, the reward function also includes a penalty term to avoid inappropriate actions, such as violent joint movements or excessive path adjustments, to ensure that the trained strategy has good robustness and practical usability.

[0064] Step S6: Migration from simulation to real environment.

[0065] To address the differences in physical characteristics between simulated and real environments, this embodiment introduces environmental perturbation and noise modeling methods during the reinforcement learning training phase to enhance the robustness and generalization ability of the policy in real systems. This method actively introduces state observation noise and action execution perturbations during simulation training to simulate factors such as sensor measurement errors, actuator response deviations, and dynamic uncertainties in the real environment, enabling the trained policy to better adapt to complex situations in the real world.

[0066] First, random noise is introduced during the state observation process to perturb the environmental state. Specifically, after acquiring the robotic arm's state information (including joint angles, joint velocities, and obstacle distances) in the simulation environment, Gaussian distributed noise is superimposed on the original state values ​​to simulate random errors that may occur during real sensor measurements. The state observation values ​​after adding noise can be expressed as: in, Represents the original state information. This indicates the observation state after adding noise. For random noise that follows a Gaussian distribution, This represents the noise standard deviation. By introducing state noise during the training phase, the policy network can learn a state representation that is robust to observation errors, thereby improving its adaptability in real-world environments.

[0067] Secondly, the control commands output by the strategy are perturbed during the motion execution phase. Specifically, after the strategy network outputs the joint speed commands of the robotic arm, a certain range of random perturbations is added to them to simulate the errors that may exist in the response process of a real actuator. The motion command after perturbation can be represented as: in, This represents the original control action output by the policy network. This indicates the action to be taken after adding a disturbance. This represents a random perturbation that follows a uniform distribution. This represents the perturbation amplitude. By perturbing the action output during the training phase, the policy can gradually adapt to the actuator response error, thereby improving the policy's stability in real systems.

[0068] Furthermore, to enhance the generalization ability of the strategy, noise parameters are randomly sampled during training, allowing them to vary within a certain range. This prevents the strategy from overfitting to fixed noise patterns. In this way, the reinforcement learning strategy can maintain good adaptability when facing varying degrees of environmental uncertainty.

[0069] Through the above methods, reinforcement learning strategies can be fully exposed to environmental states containing noise and disturbances during the training phase, thereby learning more robust control strategies and providing an effective guarantee for the subsequent transfer from simulation to the real system.

[0070] Example 2 Based on Example 1, this example provides a dynamic obstacle avoidance system for a robotic arm based on safety reinforcement learning, used to implement the dynamic obstacle avoidance method for the robotic arm in Example 1. The system includes: (1) Safety reinforcement learning module, which is equipped with a reinforcement learning model. The robot arm motion state, target position and dynamic obstacle information are used as the reinforcement learning state, the angular velocity control of each joint is used as the reinforcement learning action, and a multi-objective function including collision penalty, tracking error penalty, collision risk penalty and termination reward is used as the reinforcement learning reward function. The reinforcement learning model is iterated through proximal policy optimization; the reinforcement learning state is obtained, and the action is obtained by using the reinforcement learning model after policy iteration as the reinforcement learning policy.

[0071] (2) Compensation control module, used to find the minimum control correction amount so that the system can meet the safety constraints again based on the state of the reinforcement learning and the reinforcement learning policy, and obtain the optimal control correction amount as policy compensation.

[0072] (3) The guidance control module directly integrates CBF information into the reinforcement learning training process to form a guidance control mechanism. Unlike compensation control, the core objective of guidance control is to enable the reinforcement learning strategy to actively move towards the safe zone during the learning phase, rather than relying on later corrections. That is, the safety information not only acts on the control execution layer, but also participates in the strategy optimization process. (4) Iterative execution module, used to obtain the final control policy and execute it based on the reinforcement learning policy and the policy compensation.

[0073] In summary, the present invention has the following beneficial effects: 1. Improve system security.

[0074] This invention introduces a control obstacle function into a reinforcement learning control framework to constrain the safe distance between the robotic arm and obstacles in real time. When the reinforcement learning strategy generates control actions that may lead to collision risks, the control commands are corrected through a CBF (Body-Crossing Filtering) mechanism, thereby ensuring that the system always meets the safety constraints and effectively reducing the collision risk of the robotic arm in dynamic environments.

[0075] 2. Improve obstacle avoidance capabilities in dynamic environments.

[0076] This invention uses a reinforcement learning policy network to learn the control strategy of a robotic arm in complex environments, enabling the robotic arm to autonomously generate obstacle avoidance actions based on environmental conditions. Simultaneously, by incorporating CBF safety constraints, the system maintains both flexible decision-making capabilities and ensures safety during obstacle avoidance when facing dynamic obstacles, thereby enhancing the robotic arm's adaptability in dynamic environments.

[0077] 3. Improve the real-time nature of control decisions.

[0078] Unlike traditional path planning methods that require complex path searches or repetitive planning, the method of this invention only needs to obtain the control action through forward computation of the neural network after the strategy training is completed, and the safety constraint is quickly corrected through the CBF module. Therefore, it can complete the control decision in a short time and meet the real-time control requirements of the robotic arm's dynamic obstacle avoidance task.

[0079] 4. Enhance the stability and robustness of control strategies.

[0080] By introducing a CBF (Constraint-Based Function) safety constraint mechanism on the outer layer of the reinforcement learning control policy, this invention can maintain the safety of system operation even when the reinforcement learning policy has not fully converged or the environment changes. Simultaneously, this safety constraint mechanism can effectively reduce unstable control behaviors generated during the policy exploration phase, improving the overall stability and robustness of the system.

[0081] 5. Improve the smoothness of robotic arm movements.

[0082] This invention incorporates a safety constraint mechanism into the reinforcement learning strategy optimization process, enabling the robotic arm to generate continuous and stable control actions. This reduces the vibration of the robotic arm's movement caused by sudden path changes or discontinuous control, thereby improving the smoothness of the robotic arm's movement when performing tasks.

[0083] 6. It has good generalization ability.

[0084] The method of this invention obtains a general control strategy through reinforcement learning training and combines it with the CBF safety constraint mechanism, so that the system can still maintain good obstacle avoidance performance under different dynamic environmental conditions, thus having strong environmental adaptability and generalization ability.

[0085] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A dynamic obstacle avoidance method for a robotic arm based on safety reinforcement learning, characterized in that, Includes the following steps: Step S1: The state of the robotic arm's motion, the target position, and the dynamic obstacle information are used as the state of reinforcement learning. The angular velocity control of each joint is used as the action of reinforcement learning. A multi-objective function including collision penalty, tracking error penalty, collision risk penalty, and termination reward is used as the reward function of reinforcement learning. The reinforcement learning model is iterated through proximal policy optimization. Step S2: Obtain the state of reinforcement learning, and use the reinforcement learning model after policy iteration to obtain the action as the reinforcement learning policy; Step S3: Based on the state of the reinforcement learning and the reinforcement learning policy, with the goal of finding the minimum control correction amount that makes the system meet the safety constraints again, the optimal control correction amount is obtained as policy compensation. Step S4: Based on the reinforcement learning strategy and the strategy compensation, perform strategy optimization, integrate CBF information into the reinforcement learning training process, form a guidance control mechanism, obtain the final control strategy and execute it; Step S5: Return to step S3 until the robotic arm reaches the target position, thus achieving dynamic obstacle avoidance by the robotic arm.

2. The method for dynamic obstacle avoidance of a robotic arm based on safety reinforcement learning according to claim 1, characterized in that, The strategy compensation is obtained by solving the following problem: in, express Minimum control correction amount at any given time. The hyperparameter represents the range of variation of the probability ratio. The objective is to optimize the objective function or the pruned strategy. , The new strategy and the old strategy are respectively in the state. The following action probability distribution This indicates the transpose operation. Reinforcement learning Moment State The corresponding reinforcement learning strategy, item As the first The second iteration or the first The control obstacle function compensation term corresponding to each safety correction component For the number of iterations of the safety compensation term, The attenuation rate, For the control barrier function, For the system in state The control input mapping matrix below, , The first The lower and upper limits of each component. This represents the total number of components.

3. The dynamic obstacle avoidance method for a robotic arm based on safety reinforcement learning according to claim 1, characterized in that, The tracking error penalty is: in, for Real-time tracking error penalty These are weighting coefficients used to adjust the trend of the reward function across different error intervals. Distance threshold for The Euclidean distance between the end effector of the robotic arm and the target point at any given time. The current position of the end effector. The location of the target point.

4. The dynamic obstacle avoidance method for a robotic arm based on safety reinforcement learning according to claim 1, characterized in that, The collision risk penalty is as follows: in, for Constant collision risk and punishment The Euclidean distance between the end effector position and the obstacle position. For safe distance threshold, The current position of the end effector. This indicates the location of the obstacle.

5. The dynamic obstacle avoidance method for a robotic arm based on safety reinforcement learning according to claim 1, characterized in that, After obtaining the reinforcement learning strategy, the following is also included: Obtain the reinforcement learning policies corresponding to the current time step and the previous time step, and obtain the reinforcement learning policy sequence through linear interpolation; Based on the reinforcement learning policy sequence, the final reinforcement learning policy is obtained through Butterworth low-pass filtering.

6. The dynamic obstacle avoidance method for a robotic arm based on safety reinforcement learning according to claim 5, characterized in that, The linear interpolation is achieved using the following formula: in, for Reinforcement learning strategies at all times , Each is the previous moment Current moment The corresponding reinforcement learning strategy.

7. The dynamic obstacle avoidance method for a robotic arm based on safety reinforcement learning according to claim 5, characterized in that, The Butterworth low-pass filter is implemented using the following formula: in, for The reinforcement learning policy sequence after low-pass filtering at each time step. The cutoff angular frequency, The system sampling period is for A constant reinforcement learning strategy.

8. The dynamic obstacle avoidance method for a robotic arm based on safety reinforcement learning according to claim 1, characterized in that, The process of iterating the policy of the reinforcement learning model through proximal policy optimization includes the following steps: Calculate the probability ratio between the current policy and the old policy; Based on the probability ratio and the advantage function based on generalized advantage estimation, calculate the shear target loss; Based on the shearing target loss, combined with the loss of the value network and the entropy regularization term, the policy iteration of the reinforcement learning model is realized through policy gradient update; During policy iteration, random noise is introduced during state observation to perturb the environmental state. During action execution, the control commands output by the policy are perturbed, and noise parameters are randomly sampled.

9. The dynamic obstacle avoidance method for a robotic arm based on safety reinforcement learning according to claim 1, characterized in that, The robotic arm motion state includes the joint angles and joint velocities of the robotic arm; the target position includes the difference between the target position and the end effector position, and the difference between the target velocity and the end effector velocity; the dynamic obstacle information includes the state of the obstacles.

10. A dynamic obstacle avoidance system for a robotic arm based on safety reinforcement learning, characterized in that, For implementing the dynamic obstacle avoidance method for a robotic arm as described in any one of claims 1-9, the system comprises: The safety reinforcement learning module deploys a reinforcement learning model. It uses the robotic arm's motion state, target position, and dynamic obstacle information as the reinforcement learning state, the angular velocity control of each joint as the reinforcement learning action, and a multi-objective function including collision penalty, tracking error penalty, collision risk penalty, and termination reward as the reinforcement learning reward function. It iterates the reinforcement learning model through proximal policy optimization; it obtains the reinforcement learning state, uses the reinforcement learning model after policy iteration to obtain the action, and uses it as the reinforcement learning policy. The compensation control module is used to find the minimum control correction amount so that the system can meet the safety constraints again, based on the state of the reinforcement learning and the reinforcement learning policy. The guidance and control module is used to integrate CBF information into the reinforcement learning training process to form a guidance and control mechanism, so that the reinforcement learning strategy actively tends to the safe zone during the learning phase. The iterative execution module is used to obtain and execute the final control policy based on the reinforcement learning policy and the policy compensation.

Images