Continuous body mechanical arm motion control method based on deep reinforcement learning

Abstract

The invention provides a continuum mechanical arm motion control method based on deep reinforcement learning. The method relates to the technical field of robots, and comprises the following steps: collecting multi-source information of a mechanical arm, and fusing the multi-source information of the mechanical arm to generate a state vector reflecting current kinematics and dynamics characteristics; and inputting the state vector into a deep reinforcement learning strategy, and outputting a continuous candidate control instruction matched with the state vector. According to the continuum mechanical arm motion control method based on deep reinforcement learning, the high dependence of a traditional control method on a kinetic model and parameter calibration can be effectively overcome, and accurate control over the mechanical arm under the conditions of external disturbance and structural parameter drift is achieved. By fusing multi-source information and dynamically adjusting the control strategy, the control instruction can be corrected in real time and kept stable under different physical conditions, the adaptability and the control effect of the mechanical arm in a complex environment are improved, and safety and execution precision are ensured.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a continuum robotic arm motion control method based on deep reinforcement learning. Background Art

[0002] Continuum robotic arms, due to their structural flexibility and ability to deform along arbitrary curves, offer excellent human-machine safety and environmental adaptability in fields such as medical surgery, pipeline inspection, and marine operations. Currently, motion control relies primarily on classical methods based on precise dynamic models, such as PID, robust control, sliding mode control, and model predictive control. On this basis, online adaptive laws or gain scheduling are introduced to improve resistance to parameter uncertainty. At the same time, some researchers have attempted to apply reinforcement learning algorithms such as Q-Learning and policy gradients in discrete action spaces to simulation environments, learning motion strategies through accumulated experience.

[0003] However, existing technologies still have shortcomings: classical control methods are highly dependent on dynamic models and parameter calibration. When encountering structural parameter drift or external disturbances, the control accuracy and robustness are significantly reduced. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides a continuum robotic arm motion control method based on deep reinforcement learning, which solves the problem of how to use deep reinforcement learning strategy and incremental method to achieve high-precision and high-robustness continuum robotic arm motion control.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: A continuum manipulator motion control method based on deep reinforcement learning, comprising: S1. collecting multi-source information of the manipulator, and fusing the multi-source information of the manipulator to generate a state vector reflecting the current kinematic and dynamic characteristics; S2. Input the state vector into a deep reinforcement learning strategy, and output continuous candidate control instructions that match the state vector, wherein the deep reinforcement learning strategy is pre-trained offline using tracking accuracy, energy consumption, and safety constraint penalty terms; S3. Perform real-time feasibility corrections on the continuous candidate control instructions based on safety requirements constraints to obtain a correction control instruction that satisfies safety constraints; S4. Incrementally fusing the modified control command with the baseline control command generated by the simplified dynamic model to generate a final control output to compensate for control deviations caused by model inaccuracies or external disturbances; S5. Under the influence of the final control output, the deep reinforcement learning strategy and the incremental method are dynamically adjusted according to the execution effect and the degree of satisfaction of the safety constraints to cope with structural parameter drift and environmental changes.

[0006] Preferably, the multi-source information of the robotic arm includes the three-dimensional position of the end effector, the quaternion posture, the curvature of each flexible segment, the tendon tension and the air chamber pressure. The multi-source information of the robotic arm is normalized and fused to generate the state vector.

[0007] Preferably, the offline training randomizes environmental parameters within respective preset amplitude ranges to improve the generalization performance of the strategy under different physical conditions, and the environmental parameters include payload mass, friction coefficient and tendon stiffness.

[0008] Preferably, the security requirement constraints in S3 include: S3.1. The curvature of each flexible segment shall not exceed the preset maximum curvature limit; S3.2. The air chamber pressure shall not exceed the preset maximum pressure limit; S3.3. The tension of the tendons does not exceed the preset maximum tension limit; S3.4. The distance from any point on the robotic arm to any environmental obstacle shall not be less than the preset safety distance.

[0009] Preferably, the deep reinforcement learning strategy model formula is as follows: in, In state and actions The reward value under is the current pose vector of the end effector, is the desired pose vector, For the The violation function of the security constraint, is the total number of security constraints, is the weight coefficient of the tracking error penalty term and the safety constraint penalty term.

[0010] Preferably, the real-time feasibility correction is performed by solving the following quadratic programming problem to obtain the correction control instruction: in, is the corrected control vector to be determined, is the dimension of the control instruction, is a continuous candidate control vector, is the diagonal weight matrix, is the linearization matrix of the safety constraint, For The corresponding constraint threshold vector, is the total number of security constraints.

[0011] Preferably, the dynamic adjustment is triggered only when the tracking error and the safety constraint satisfaction are both within a preset threshold range to avoid damaging the closed-loop stability due to a large-scale policy update. The tracking error does not exceed 3 mm, and the maximum violation of the safety constraint does not exceed 0.05.

[0012] Preferably, the dynamic adjustment is further constrained by a trust region, and the distribution difference between the deep reinforcement learning strategy and the incremental method in the same state before and after the dynamic adjustment does not exceed 0.01 to prevent the strategy from shifting drastically.

[0013] This invention provides a continuum robotic arm motion control method based on deep reinforcement learning. It has the following beneficial effects: The control strategy based on deep reinforcement learning can effectively overcome the traditional control method's high dependence on dynamic models and parameter calibration, realize precise control of the robotic arm under external disturbances and structural parameter drift, and improve control accuracy and system robustness.

[0014] By integrating multi-source information and dynamically adjusting control strategies, this invention can modify control instructions in real time and maintain stability under varying physical conditions, improving the manipulator's adaptability and control effectiveness in complex environments while ensuring safety and execution accuracy. By integrating online corrections with an incremental approach, the proposed method overcomes the shortcomings of simplified dynamic models, effectively responding to environmental changes and ensuring the manipulator's stable operation in these changing environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a schematic diagram of the flow structure of the continuum robot arm motion control method of the present invention; Figure 2 This is a schematic diagram of the safety requirement constraint structure of the present invention; Figure 3 Schematic diagram of the strategy and incremental method for dynamically adjusting the structure of the present invention. DETAILED DESCRIPTION

[0016] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0017] like Figure 1As shown, an embodiment of the present invention provides a continuum robotic arm motion control method based on deep reinforcement learning, including: S1. collecting multi-source information of the robotic arm and fusing the multi-source information to generate a state vector reflecting the current kinematic and dynamic characteristics of the robotic arm. The multi-source information of the robotic arm includes the three-dimensional position of the end effector, the quaternion attitude, the curvature of each flexible segment, the tendon tension, and the air chamber pressure. The multi-source information of the robotic arm is normalized and fused to generate the state vector.

[0018] S2. The state vector is input into a deep reinforcement learning strategy, which outputs a continuous stream of candidate control commands that match the state vector. The deep reinforcement learning strategy is pre-trained offline using tracking accuracy, energy consumption, and safety constraint penalties. This offline training randomizes environmental parameters within a preset range, including payload mass, friction coefficient, and tendon stiffness.

[0019] S3. Perform real-time feasibility correction on the continuous candidate control instructions according to the safety requirement constraints to obtain the corrected control instructions that meet the safety constraints.

[0020] Security requirement constraints include: S3.1. The curvature of each flexible segment shall not exceed the preset maximum curvature limit.

[0021] S3.2. The air chamber pressure shall not exceed the preset maximum pressure limit.

[0022] S3.3. The tension of the tendons shall not exceed the preset maximum tension limit.

[0023] S3.4. The distance from any point on the robotic arm to any environmental obstacle shall not be less than the preset safety distance.

[0024] S4. Incrementally fuse the modified control instructions with the baseline control instructions generated by the simplified dynamics model to generate the final control output to compensate for control deviations caused by model inaccuracies or external disturbances. The real-time feasibility correction obtains the modified control instructions by solving the following quadratic programming problem: in, is the corrected control vector to be determined, is the dimension of the control instruction, is a continuous candidate control vector, is the diagonal weight matrix, is the linearization matrix of the safety constraint, For The corresponding constraint threshold vector, is the total number of security constraints.

[0025] S5. Under the influence of the final control output, the deep reinforcement learning strategy and incremental method are dynamically adjusted according to the execution effect and the degree of satisfaction of safety constraints to cope with structural parameter drift and environmental changes. The deep reinforcement learning strategy model formula is as follows: in, In state and actions The reward value under is the current pose vector of the end effector, is the desired pose vector, For the The violation function of the security constraint, is the total number of security constraints, is the weight coefficient for the tracking error penalty and the safety constraint penalty. Dynamic adjustment is triggered only when both the tracking error and the safety constraint satisfaction are within the preset thresholds, with the tracking error not exceeding 3 mm and the maximum safety constraint violation not exceeding 0.05. Dynamic adjustment is further subject to trust region constraints, and the distribution difference between the deep reinforcement learning strategy before and after dynamic adjustment and the incremental method under the same state is no more than 0.01.

[0026] Example 2 Different from the first embodiment, this embodiment performs real-time feasibility correction on continuous candidate control instructions based on safety requirement constraints to obtain a corrected control instruction that meets the safety constraint.

[0027] 1. Collect control instructions and constraint information During the operation of the robot arm, the system generates continuous candidate control instructions. The control instructions are generated by the deep reinforcement learning strategy model. Assume that the generated control instructions are , define several key security constraints, such as: The curvature of each flexible segment Does not exceed the preset maximum curvature limit .

[0028] The pressure P of the gas chamber does not exceed the preset maximum pressure limit .

[0029] Tendon tension Do not exceed the preset maximum tensile force limit .

[0030] The distance d between any point of the robotic arm and the obstacle is not less than the preset safety distance .

[0031] Constraints are checked in real time during the control process to ensure that the revised control instructions meet the constraints.

[0032] 2. Define the quadratic programming problem Continuous candidate control instructions If the above constraints are met, the system will be corrected through the following quadratic programming problem. Assume that the corrected control instruction is ,The goal of quadratic programming is to minimize the ,adjustment amount of control commands while satisfying the safety ,constraints.

[0033] The quadratic programming problem is formulated as: The following security constraints must be met: , that is, the curvature of each flexible segment cannot exceed the maximum curvature.

[0034] , that is, the air chamber pressure cannot exceed the maximum pressure.

[0035] , that is, the tension of the tendon cord cannot exceed the maximum tension.

[0036] , that is, the distance between the robotic arm and environmental obstacles must be kept within a safe range.

[0037] 3. Solution and correction By using an appropriate quadratic programming algorithm, the system will minimize the control deviation, solve the constraints, and obtain the corrected control instructions that meet the safety requirements. .

[0038] For example, candidate control instructions Leading to the curvature of a flexible segment Exceeding the maximum limit , the system will adjust the control instruction according to the constraint conditions so that the corrected curvature .

[0039] 4. Real-time feedback and updates Corrected control instructions This feedback is fed back to the control system in real time, affecting the motion control of the robotic arm. The system adjusts and corrects the control instructions based on the real-time feedback to ensure that the robotic arm's motion is always within a safe range.

[0040] Example 3 Different from the first embodiment, this embodiment describes how to incrementally fuse the corrected control instructions with the baseline control instructions generated by the simplified dynamics model to compensate for control deviations caused by model inaccuracies or external disturbances.

[0041] 1. Generation of baseline control instructions The simplified dynamic model generates a baseline control command based on the kinematic and dynamic characteristics of the robot arm , which is calculated based on the known model of the robot and the assumed environmental conditions. For example, the target position of the end effector of the robot at a certain moment is , and the simplified dynamics model calculates the corresponding control instructions through inverse dynamics: in, and Represent the current joint position and velocity of the robot arm respectively. Assume that the baseline control instruction is: 2. Correct the acquisition of control instructions Through real-time feasibility correction steps, correction control instructions that meet safety constraints are obtained , ensuring that the control instructions do not violate any safety requirements. Assume that the modified control instructions are: The control of the robotic arm was fine-tuned while correcting the instructions to ensure safety.

[0042] 3. Incremental integration To compensate for the control deviation caused by the inaccurate dynamic model or external disturbance, the modified control command and the baseline control command will be integrated in an incremental manner. Assume that the difference between the baseline control command and the modified control command is the incremental control command , which is calculated as: Based on the previous assumptions, the incremental control instructions are: The incremental control instructions are proportionally integrated with the baseline control instructions to generate the final control instructions. A weight coefficient λ is used to control the influence of the incremental correction. For example, choosing λ=0.7 means that the correction instruction has a weight of 70%, while the baseline instruction has a weight of 30%. The final control instruction is: Substitute the data for calculation: Control instruction application: Final control instructions This approach will be applied to robotic arm motion control systems to compensate for control deviations caused by model inaccuracies or external disturbances. This incremental correction method ensures that the robotic arm maintains the required accuracy during execution, allowing it to perform efficient and safe movements despite environmental changes.

[0043] The incremental fusion process improves control accuracy and reduces the impact of model imperfections or external disturbances on the movement of the robotic arm.

[0044] Example 4 Different from the first embodiment, this embodiment describes how to evaluate the control effect in real time and adjust the strategy and fusion method according to the control deviation and the degree of satisfaction of the safety constraints.

[0045] 1. Calculation of tracking error The system evaluates the tracking error of the end effector of the robot arm in real time. Assuming that the current target position of the robot arm is , the actual position of the end effector of the robot arm is , tracking error Expressed as: For example, assuming the target location is The unit is meters, while the actual position is , then the tracking error is: 2. Checking safety constraints The system checks in real time whether various safety constraints are met. For example, suppose there are the following safety constraints: , that is, the curvature of each flexible segment cannot exceed the maximum curvature.

[0046] , that is, the air chamber pressure cannot exceed the maximum pressure.

[0047] , that is, the tension of the tendon cord cannot exceed the maximum tension.

[0048] Assume that the current security constraint check result is: The curvature of the flexible segment is =0.8rad / m, the maximum curvature limit is =1.0rad / m.

[0049] Air chamber pressure P=2.5, the maximum pressure limit is =3.0bar.

[0050] Tendon tension =20N, the maximum pulling force is limited to =25N.

[0051] The values ​​are all within the predetermined safety range, indicating that the safety constraints are met.

[0052] 3. Dynamic adjustment strategy and incremental integration method By tracking the error and the degree of satisfaction of safety constraints, the system will dynamically adjust the deep reinforcement learning strategy and incremental fusion method. If the tracking error is large, the system will increase the penalty term in the deep reinforcement learning strategy to improve the accuracy. For example, the system increases the penalty factor To punish larger tracking errors. Assuming the current tracking error =0.057 m, if the error is greater than the predetermined threshold =0.05m, then increase the penalty term in the strategy.

[0053] The adjusted reinforcement learning policy is expressed as: in, is the original reward value, is the penalty factor. Assume =10 and select =2, then the adjusted reward value is: 4. Real-time feedback and updates The dynamically adjusted deep reinforcement learning strategy and incremental fusion control instructions are fed back to the control system in real time, influencing the motion control of the robotic arm. As long as the tracking error is small and safety constraints are met, the system maintains a low adjustment frequency to avoid unnecessary fluctuations caused by over-adjustment.

[0054] While embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations may be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A continuum robotic arm motion control method based on deep reinforcement learning, characterized in that: include: S1. Collect multi-source information of the manipulator and fuse the multi-source information of the manipulator to generate a state vector reflecting the current kinematic and dynamic characteristics; S2. Input the state vector into a deep reinforcement learning strategy, and output continuous candidate control instructions that match the state vector, wherein the deep reinforcement learning strategy is pre-trained offline using tracking accuracy, energy consumption, and safety constraint penalty terms; S3. Perform real-time feasibility corrections on the continuous candidate control instructions based on safety requirements constraints to obtain a correction control instruction that satisfies safety constraints; S4. Incrementally fusing the modified control command with the baseline control command generated by the simplified dynamic model to generate a final control output; S5. Under the influence of the final control output, dynamically adjust the deep reinforcement learning strategy and the incremental method according to the execution effect and the degree of satisfaction of the safety constraints.

2. The method for controlling the motion of a continuum manipulator based on deep reinforcement learning according to claim 1, wherein: The multi-source information of the robotic arm includes the three-dimensional position of the end effector, the quaternion posture, the curvature of each flexible segment, the tendon tension and the air chamber pressure. The multi-source information of the robotic arm is normalized and fused to generate the state vector.

3. The method for controlling the motion of a continuum manipulator based on deep reinforcement learning according to claim 1, wherein: The offline training randomizes environmental parameters within respective preset amplitude ranges, wherein the environmental parameters include payload mass, friction coefficient, and tendon stiffness.

4. The method for controlling the motion of a continuum manipulator based on deep reinforcement learning according to claim 1, wherein: The security requirements and constraints described in S3 include: S3.

1. The curvature of each flexible segment shall not exceed the preset maximum curvature limit; S3.

2. The air chamber pressure shall not exceed the preset maximum pressure limit; S3.

3. The tension of the tendons does not exceed the preset maximum tension limit; S3.

4. The distance between any point on the robotic arm and any environmental obstacle shall not be less than the preset safety distance.

5. The method for controlling the motion of a continuum manipulator based on deep reinforcement learning according to claim 1, wherein: The deep reinforcement learning strategy model formula is as follows: in, In state and actions The reward value under is the current pose vector of the end effector, is the desired pose vector, For the The violation function of the security constraint, is the total number of security constraints, is the weight coefficient of the tracking error penalty term and the safety constraint penalty term.

6. The method for controlling the motion of a continuum manipulator based on deep reinforcement learning according to claim 1, wherein: The real-time feasibility correction is performed by solving the following quadratic programming problem to obtain the correction control instruction: in, is the corrected control vector to be determined, is the dimension of the control instruction, is a continuous candidate control vector, is the diagonal weight matrix, is the linearization matrix of the safety constraint, For The corresponding constraint threshold vector, is the total number of security constraints.

7. The method for controlling the motion of a continuum manipulator based on deep reinforcement learning according to claim 1, wherein: The dynamic adjustment is triggered only when the tracking error and the safety constraint satisfaction are both within a preset threshold range, the tracking error does not exceed 3 mm, and the maximum violation of the safety constraint does not exceed 0.

05.

8. The method for controlling the motion of a continuum manipulator based on deep reinforcement learning according to claim 1, wherein: The dynamic adjustment is further subject to a trust region constraint, and the distribution difference between the deep reinforcement learning strategy and the incremental method in the same state before and after the dynamic adjustment does not exceed 0.01.

Images