Robot high-precision motion control algorithm based on reinforcement learning

Abstract

The invention discloses a robot high-precision motion control algorithm based on reinforcement learning. The control algorithm comprises the steps of data acquisition and preprocessing; training a reinforcement learning model; track prediction and deviation correction; fault detection and self-repairing; integrating and verifying the system; and performance optimization and iterative improvement are carried out. The method can better adapt to the dynamic and complex working environment, the motion control strategy can be updated and adjusted in real time, and the robot can quickly find an alternative scheme and a repair strategy through reinforcement learning under the condition that a sensor breaks down or motion control fails.

Description

Technical Field

[0001] The present invention relates to the technical field of motion control algorithms, and in particular to a high-precision motion control algorithm for robots based on reinforcement learning. Background Art

[0002] In recent years, robotics has been widely used in many fields, especially in industrial automation, medical services and public safety. However, in the maintenance and inspection of power transmission lines, due to the complex environment, long lines and the risk of high-altitude operations, the traditional manual inspection method has safety hazards and low efficiency. In order to improve the safety and efficiency of power transmission line inspection, the anti-fall track inspection robot came into being. This robot can conduct autonomous inspections along the transmission line tracks to detect the status and fault conditions of the lines.

[0003] The existing motion control algorithm of the anti-fall track inspection robot mainly relies on preset trajectory planning and control parameters. When the actual working environment does not match the preset trajectory, the robot cannot effectively adjust its motion trajectory, resulting in a decrease in inspection accuracy. In addition, when the sensor fails or the motion control fails, the existing system usually adopts simple fault-tolerant measures, such as data redundancy or error compensation, which cannot fundamentally solve the problem, making it difficult for the robot to maintain high-precision motion control in a complex and changing environment, affecting the reliability and stability of the inspection task. Therefore, it is necessary to design a robot high-precision motion control algorithm based on reinforcement learning to solve the above problems. Summary of the invention

[0004] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0005] In view of the above-mentioned problems existing in the existing high-precision motion control algorithm of the robot based on reinforcement learning, the present invention is proposed.

[0006] Therefore, the purpose of the present invention is to provide a high-precision robot motion control algorithm based on reinforcement learning, which is suitable for solving the problem in the existing operation control method that when the actual working environment does not match the preset trajectory, the robot cannot effectively adjust its motion trajectory, resulting in a decrease in inspection accuracy.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: a high-precision motion control algorithm for robots based on reinforcement learning, the control algorithm comprising:

[0008] Data collection and preprocessing;

[0009] Reinforcement learning model training;

[0010] Trajectory prediction and deviation correction;

[0011] Fault detection and self-repair;

[0012] System integration and verification;

[0013] Performance optimization and iterative improvement.

[0014] As a preferred solution of the robot high-precision motion control algorithm based on reinforcement learning described in the present invention, it is necessary to construct a trajectory deviation prediction formula in trajectory prediction and deviation correction to predict the trajectory deviation of multiple time steps in the future and make corrections in advance. The trajectory deviation prediction formula is as follows:

[0015]

[0016] Among them, λ is the decay constant used to adjust the impact of the action on the future state, α and β are adjustment parameters used to adjust the impact of the sine term, s t represents the state vector at time t, a t represents the action vector at time t, represents the predicted state vector at time t+k;

[0017] Formula output is the predicted state of the next k time steps, and the range is all possible state spaces. If Outside the expected state range, indicating the need for trajectory correction.

[0018] As a preferred solution of the robot high-precision motion control algorithm based on reinforcement learning described in the present invention, wherein: firstly, an error calculation formula is designed, and the error calculation formula is as follows:

[0019] e t =s t -s ref,t

[0020] Among them, e t represents the error vector at the current time t, s t represents the actual state vector at the current time t, representing the current state of the robot, s ref,t represents the reference state vector at the current time t, and represents the desired state.

[0021] As a preferred solution of the robot high-precision motion control algorithm based on reinforcement learning described in the present invention, in the error calculation formula output result:

[0022] If |e t |>e maxIndicates that the error exceeds the expected range, and the self-repair control process is triggered;

[0023] If |e t |≤e max Indicates that the error is within the expected range and no self-repair is required;

[0024] Among them, e max Indicates the maximum allowed error threshold.

[0025] As a preferred solution of the robot high-precision motion control algorithm based on reinforcement learning described in the present invention, the output result of the error calculation formula is brought in, and the error attenuation and adjustment formula is constructed according to the error calculation formula, and the error attenuation and adjustment formula is as follows:

[0026]

[0027] Among them, e adj,t+k represents the error vector adjusted for the next k time steps, μ represents the attenuation constant, which is used to adjust the impact of the error on the future state and determine the decay rate of the error over time, γ represents the regularization parameter, which is used to avoid the denominator of the error term being zero and smooth the impact of the error, x is the integral variable, which represents the continuous change of time, and e t+k Represents the error vector at time t+j, represents the error of future time steps, and m is used to accumulate the error terms of the future m time steps.

[0028] As a preferred solution of the robot high-precision motion control algorithm based on reinforcement learning described in the present invention, in the output result of the error attenuation and adjustment formula:

[0029] If |e adj,t+k |>e adj,max , indicating that the adjusted error still exceeds the allowable range and the control strategy needs to be further adjusted;

[0030] If you want to|e adj,t+k |≤e adj,max , indicating that the adjusted error is within the expected range and no further adjustment of the control strategy is required;

[0031] Among them, |e adj,t+k | represents the modulus of the adjusted error vector, e adj,max Indicates the maximum error threshold allowed after adjustment.

[0032] As a preferred solution of the robot high-precision motion control algorithm based on reinforcement learning described in the present invention, the output result of the error attenuation and adjustment formula is substituted, and a control strategy correction formula is constructed according to the error attenuation and adjustment formula and the error calculation formula, and the control strategy correction formula is as follows:

[0033]

[0034] in, represents the predicted control input vector for the next k time steps, ν represents the decay constant, which is used to adjust the influence of the adjusted error on the control input and determines the decay rate of the control input over time, θ represents the adjustment parameter, which is used to adjust the influence of the cosine term and affects the calculation of the control input, y represents the integral variable, which represents the continuous change of time, is the predicted state vector for the next k time steps, representing the predicted robot state, u t+k is the actual control input vector for the next k time steps, representing the actual control strategy applied.

[0035] As a preferred solution of the robot high-precision motion control algorithm based on reinforcement learning described in the present invention, the predicted control input vector represents the control strategy required in the future time step;

[0036] like Indicates that the predicted control input exceeds the allowable range and needs further adjustment of the control strategy or repair plan;

[0037] like It indicates that the predicted control input is within the allowable range and there is no need to adjust the control strategy or repair plan;

[0038] in, represents the modulus of the predictive control input vector, u max Indicates the maximum allowed control input threshold.

[0039] As a preferred solution of the robot high-precision motion control algorithm based on reinforcement learning described in the present invention, the trajectory deviation prediction formula is The output of is used as part of the input in the control strategy correction formula to adjust the control strategy. The control strategy correction formula generates Can in turn affect the action vector a in the trajectory deviation prediction formula t+k , to achieve closed-loop control.

[0040] As a preferred solution of the reinforcement learning-based robot high-precision motion control algorithm described in the present invention, the trajectory prediction and deviation correction include two aspects: the design of the trajectory prediction module and the application of the correction strategy; the fault detection and self-repair include two aspects: the design of the fault detection module and the generation of the self-repair strategy.

[0041] Beneficial effects of the present invention: Through reinforcement learning training, the robot can predict and optimize motion control in advance when facing different trajectories and environmental changes. This foresight enables the robot to better adapt to dynamic and complex working environments, update and adjust motion control strategies in real time, and significantly improve the robot's response speed to environmental changes;

[0042] In the event of sensor failure or motion control failure, the robot can quickly find alternatives and repair strategies through reinforcement learning. This self-repair capability not only improves the reliability of the robot, but also reduces maintenance costs and downtime. By combining the best correction strategies in different situations, the robot can maintain stable operation under multiple possible failure modes. The reinforcement learning algorithm enables the robot to achieve high-precision motion control after multiple training and optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0044] Figure 1 This is a schematic diagram of the overall structure of the robot high-precision motion control algorithm based on reinforcement learning proposed by the present invention;

[0045] Figure 2 This is a schematic diagram of the trajectory error comparison line of the robot high-precision motion control algorithm based on reinforcement learning proposed in the present invention. DETAILED DESCRIPTION

[0046] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0047] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0048] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0049] Secondly, the present invention is described in detail with reference to the schematic diagram. When describing the embodiments of the present invention in detail, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0050] Embodiment 1

[0051] Reference Figure 1 , is an embodiment of the present invention, and provides a robot high-precision motion control algorithm based on reinforcement learning, the control algorithm includes:

[0052] Data collection and preprocessing;

[0053] Environmental data collection: Collect sensor data of robot motion under various environmental conditions;

[0054] Trajectory data collection: record the robot's motion trajectory and related control parameters under different control strategies;

[0055] Data preprocessing: clean the collected data, remove noise and invalid data, perform standardization, and form a training data set;

[0056] Reinforcement learning model training;

[0057] Define the state space of the robot's motion, including position, velocity, acceleration and other information;

[0058] Train reinforcement learning models in a simulation environment to simulate motion control under various trajectories and fault conditions;

[0059] Use reinforcement learning algorithms to optimize control strategies and obtain the best correction strategies under different trajectory conditions;

[0060] Trajectory prediction and deviation correction;

[0061] Design of trajectory prediction module: The trajectory prediction module based on reinforcement learning can predict trajectory deviations for multiple time steps in the future, use historical trajectory data and current status for prediction, and make corrections in advance;

[0062] Correction strategy application: Apply the trained correction strategy in actual motion and adjust the control parameters in real time according to the predicted trajectory deviation;

[0063] Fault detection and self-repair;

[0064] Design a fault detection module to monitor sensor data and control effects in real time, detect sensor failures or motion control failures, use anomaly detection algorithms to identify faults, and trigger self-repair mechanisms;

[0065] Self-repair strategy generation: The self-repair strategy module based on reinforcement learning quickly generates alternative solutions and repair strategies according to the fault type and current status, and uses the online learning ability of the reinforcement learning model to dynamically adjust the control strategy to maintain high-precision motion control

[0066] System integration and verification;

[0067] Integrate trajectory prediction and deviation correction modules, fault detection and self-repair modules into the robot motion control system;

[0068] Test the system in a real environment to verify its predictability and response speed under different trajectory conditions, simulate sensor failures and motion control failures, and verify the effectiveness of the self-healing mechanism;

[0069] Performance optimization and iterative improvement;

[0070] Perform multiple rounds of iterative training and testing to continuously improve the system.

[0071] In trajectory prediction and deviation correction, it is necessary to construct a trajectory deviation prediction formula to predict the trajectory deviation of multiple time steps in the future and make corrections in advance. The trajectory deviation prediction formula is as follows:

[0072]

[0073] Among them, λ is the decay constant used to adjust the impact of the action on the future state, α and β are adjustment parameters used to adjust the impact of the sine term, s t represents the state vector at time t, a t represents the action vector at time t, represents the predicted state vector at time t+k;

[0074] Formula output is the predicted state of the next k time steps, and the range is all possible state spaces. If Outside the expected state range, indicating the need for trajectory correction.

[0075] First, design the error calculation formula, the error calculation formula is as follows:

[0076] e t =s t -s ref,t

[0077] Among them, e t represents the error vector at the current time t, s t represents the actual state vector at the current time t, representing the current state of the robot, s ref,t represents the reference state vector at the current time t, and represents the desired state.

[0078] In the error calculation formula output results:

[0079] If |e t |>e max Indicates that the error exceeds the expected range, and the self-repair control process is triggered;

[0080] If |e t |≤e max Indicates that the error is within the expected range and no self-repair is required;

[0081] Among them, e max Indicates the maximum allowed error threshold.

[0082] Substitute the output result of the error calculation formula into the error attenuation and adjustment formula, and the error attenuation and adjustment formula is as follows:

[0083]

[0084] Among them, e adj,t+k represents the error vector adjusted for the next k time steps, μ represents the attenuation constant, which is used to adjust the impact of the error on the future state and determine the decay rate of the error over time, γ represents the regularization parameter, which is used to avoid the denominator of the error term being zero and smooth the impact of the error, x is the integral variable, which represents the continuous change of time, and e t+k Represents the error vector at time t+j, represents the error of future time steps, and m is used to accumulate the error terms of the future m time steps.

[0085] In the output of the error attenuation and adjustment formula:

[0086] If |e adj,t+k |>e adj,max , indicating that the adjusted error still exceeds the allowable range and the control strategy needs to be further adjusted;

[0087] If you want to|e adj,t+k |≤e adj,max , indicating that the adjusted error is within the expected range and no further adjustment of the control strategy is required;

[0088] Among them, |e adj,t+k | represents the modulus of the adjusted error vector, e adj,max Indicates the maximum error threshold allowed after adjustment.

[0089] Substitute the output result of the error attenuation and adjustment formula into the control strategy correction formula based on the error attenuation and adjustment formula and the error calculation formula, and the control strategy correction formula is as follows:

[0090]

[0091] in, represents the predicted control input vector for the next k time steps, ν represents the decay constant, which is used to adjust the influence of the adjusted error on the control input and determines the decay rate of the control input over time, θ represents the adjustment parameter, which is used to adjust the influence of the cosine term and affects the calculation of the control input, y represents the integral variable, which represents the continuous change of time, is the predicted state vector for the next k time steps, representing the predicted robot state, u t+k is the actual control input vector for the next k time steps, representing the actual control strategy applied.

[0092] Predictive control input vector represents the control strategy required in the future time step;

[0093] like Indicates that the predicted control input exceeds the allowable range and needs further adjustment of the control strategy or repair plan;

[0094] like It indicates that the predicted control input is within the allowable range and there is no need to adjust the control strategy or repair plan;

[0095] in, represents the modulus of the predictive control input vector, u max Indicates the maximum allowed control input threshold.

[0096] Trajectory deviation prediction formula The output of is used as part of the input in the control strategy correction formula to adjust the control strategy. The control strategy correction formula generates Can in turn affect the action vector a in the trajectory deviation prediction formula t+k , to achieve closed-loop control.

[0097] Embodiment 2

[0098] Reference Figure 2 Table 1 and Table 2 are the second embodiment of the present invention. This embodiment is different from the first embodiment in that, in order to verify its beneficial effects, experimental comparison data of the present invention and the existing motion control algorithm in an actual environment are provided.

[0099] In order to verify the application effect of the robot high-precision motion control algorithm based on reinforcement learning in the transmission line guided anti-fall inspection robot, we carried out the following experimental preparation and implementation process;

[0100] The guided anti-fall track inspection robot of the actual transmission line was selected as the test object, and an environment simulating the transmission line was set up, including flat lines, inclined lines, turning lines, obstacles and other scenarios. By combining the existing motion control algorithm with the deep reinforcement learning algorithm, the actual performance test data of the robot in several routes was obtained;

[0101] Table 1: Inspection robot performance test data comparison table

[0102]

[0103]

[0104] By analyzing the implementation data, the traditional control strategy is compared with the control strategy after reinforcement learning from the following three aspects:

[0105] Trajectory error: In all test scenarios, the average trajectory error of the reinforcement learning control strategy is significantly lower than that of the traditional control strategy. On a flat line, the average trajectory error of the traditional control strategy is 4.8 cm, while that of the reinforcement learning strategy is 1.9 cm, a 60.4% reduction in error.

[0106] In terms of maximum trajectory error: the reinforcement learning control strategy also shows obvious advantages in terms of maximum trajectory error. For example, in the turning line, the maximum trajectory error of the traditional control strategy is 13.6cm, while that of the reinforcement learning strategy is 6.4cm, a reduction of 52.9%;

[0107] In terms of fault recovery time: the reinforcement learning control strategy is significantly better than the traditional strategy in terms of recovery time for sensor failure and control failure. In the recovery time of sensor failure, the traditional strategy takes an average of 7.5 seconds, while the reinforcement learning strategy only takes 2.1 seconds, a reduction of 72%. In the recovery time of control failure, the traditional strategy takes 9.8 seconds, while the reinforcement learning strategy only takes 2.9 seconds, a reduction of 70.4%.

[0108] Data show that the reinforcement learning-based high-precision robot motion control algorithm significantly improves the predictability and response speed of robot motion control in complex power transmission line environments, while greatly improving the system's recovery capabilities in the event of sensor failure and control failure.

[0109] During use, through reinforcement learning training, the robot can predict and optimize motion control in advance when facing different trajectories and environmental changes. This foresight enables the robot to better adapt to dynamic and complex working environments. This algorithm can update and adjust motion control strategies in real time, significantly improving the robot's response speed to environmental changes. Even in rapidly changing working scenes, the robot can quickly adjust its motion trajectory.

[0110] In the event of sensor failure or motion control failure, the robot is able to quickly find alternatives and repair strategies through reinforcement learning. This self-repair capability not only improves the reliability of the robot, but also reduces maintenance costs and downtime. By combining the best correction strategies in different situations, the robot can maintain stable operation under multiple possible failure modes. The reinforcement learning algorithm enables the robot to achieve high-precision motion control after multiple training and optimization. Through continuous learning and adjustment, the robot can achieve higher accuracy and efficiency when performing tasks.

[0111] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A high-precision robot motion control algorithm based on reinforcement learning, characterized in that: The control algorithm includes: Data collection and preprocessing; Reinforcement learning model training; Trajectory prediction and deviation correction; Fault detection and self-repair; System integration and verification; Performance optimization and iterative improvement.

2. The robot high-precision motion control algorithm based on reinforcement learning according to claim 1 is characterized in that: In trajectory prediction and deviation correction, it is necessary to construct a trajectory deviation prediction formula to predict the trajectory deviation of multiple time steps in the future and make corrections in advance. The trajectory deviation prediction formula is as follows: Among them, λ is the decay constant used to adjust the impact of the action on the future state, α and β are adjustment parameters used to adjust the impact of the sine term, s t represents the state vector at time t, a t represents the action vector at time t, represents the predicted state vector at time t+k; Formula output is the predicted state of the next k time steps, and the range is all possible state spaces. If Outside the expected state range, indicating the need for trajectory correction.

3. The robot high-precision motion control algorithm based on reinforcement learning according to claim 2 is characterized in that: First, the error calculation formula is designed, and the error calculation formula is as follows: e t =s t -s ref,t Among them, e t represents the error vector at the current time t, s t represents the actual state vector at the current time t, representing the current state of the robot, s ref,t represents the reference state vector at the current time t, and represents the desired state.

4. The robot high-precision motion control algorithm based on reinforcement learning according to claim 3 is characterized in that: In the error calculation formula output results: If |e t |>e max Indicates that the error exceeds the expected range, and the self-repair control process is triggered; If |e t |≤e max Indicates that the error is within the expected range and no self-repair is required; Among them, e max Indicates the maximum allowed error threshold.

5. The robot high-precision motion control algorithm based on reinforcement learning according to claim 3 is characterized in that: Substitute the output result of the error calculation formula into the error attenuation and adjustment formula, and the error attenuation and adjustment formula is as follows: Among them, e adj,t+k represents the error vector adjusted for the next k time steps, μ represents the attenuation constant, which is used to adjust the impact of the error on the future state and determine the decay rate of the error over time, γ represents the regularization parameter, which is used to avoid the denominator of the error term being zero and smooth the impact of the error, x is the integral variable, which represents the continuous change of time, and e t+k Represents the error vector at time t+j, represents the error of future time steps, and m is used to accumulate the error terms of the future m time steps.

6. The robot high-precision motion control algorithm based on reinforcement learning according to claim 5 is characterized in that: In the output of the error attenuation and adjustment formula: If |e adj,t+k |>e adj,max , indicating that the adjusted error still exceeds the allowable range and the control strategy needs to be further adjusted; If you want to|e adj,t+k |≤e adj,max , indicating that the adjusted error is within the expected range and no further adjustment of the control strategy is required; Among them, |e adj,t+k | represents the modulus of the adjusted error vector, e adj,max Indicates the maximum error threshold allowed after adjustment.

7. The robot high-precision motion control algorithm based on reinforcement learning according to claim 5 is characterized in that: Substitute the output result of the error attenuation and adjustment formula into the control strategy correction formula based on the error attenuation and adjustment formula and the error calculation formula, and the control strategy correction formula is as follows: in, represents the predicted control input vector for the next k time steps, ν represents the decay constant, which is used to adjust the influence of the adjusted error on the control input and determines the decay rate of the control input over time, θ represents the adjustment parameter, which is used to adjust the influence of the cosine term and affects the calculation of the control input, y represents the integral variable, which represents the continuous change of time, is the predicted state vector for the next k time steps, representing the predicted robot state, u t+k is the actual control input vector for the next k time steps, representing the actual control strategy applied.

8. The robot high-precision motion control algorithm based on reinforcement learning according to claim 7 is characterized in that: Predictive control input vector represents the control strategy required in the future time step; like Indicates that the predicted control input exceeds the allowable range and needs further adjustment of the control strategy or repair plan; like It indicates that the predicted control input is within the allowable range and there is no need to adjust the control strategy or repair plan; in, represents the modulus of the predictive control input vector, u max Indicates the maximum allowed control input threshold.

9. The robot high-precision motion control algorithm based on reinforcement learning according to claim 7 is characterized in that: Trajectory deviation prediction formula The output of is used as part of the input in the control strategy correction formula to adjust the control strategy. The control strategy correction formula generates Can in turn affect the action vector a in the trajectory deviation prediction formula t+k , to achieve closed-loop control.

10. The robot high-precision motion control algorithm based on reinforcement learning according to claim 1, characterized in that: The trajectory prediction and deviation correction include two aspects: the design of trajectory prediction module and the application of correction strategy. The fault detection and self-repair include two aspects: the design of fault detection module and the generation of self-repair strategy.

Images