Reinforcement learning based robot adaptive motion control algorithm

Abstract

The present application relates to the technical field of robot control, and particularly relates to a robot adaptive motion control algorithm based on reinforcement learning, which acquires a torque signal of a robot joint, constructs a state vector representing an interaction state of the robot and an environment based on the torque signal, generates an adaptive motion control strategy through a hierarchical reinforcement learning training mechanism, the strategy includes constructing a multi-dimensional reward function, fusing task completion degree, energy efficiency, motion safety and smoothness, and using a Q-learning algorithm to iteratively optimize a mapping relationship between the state vector and an action control sequence, verifying the adaptive motion control strategy under multiple physical parameter environments, and selectively triggering retraining based on a verification result, deploying the adaptive motion control strategy that passes the verification to a robot control system, and significantly improving adaptability, so that the robot control strategy can adapt to a physical parameter change within a range of plus or minus 30%, without the need for manual intervention to readjust parameters.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, in particular to a robot adaptive motion control algorithm based on reinforcement learning, which is suitable for scenarios where industrial robots, service robots, collaborative robots and the like perform tasks in changing environments. BACKGROUND

[0002] Traditional robot control algorithms mainly rely on preset control parameters and deterministic control laws, such as widely used PID controllers or impedance controllers. These controllers perform well when the robot works in an environment with relatively fixed conditions and stable physical parameters. However, when the robot needs to work in different environments with varying physical characteristics, such as changes in friction coefficient, damping coefficient, load mass and other parameters, the traditional controller often has difficulty maintaining stable control performance, requiring manual parameter adjustment, resulting in high debugging costs and poor adaptability.

[0003] In addition, existing control methods based on position and velocity feedback cannot directly perceive the interaction force between the robot and the environment, which limits the performance of the robot in interaction tasks with the environment. For example, in scenarios such as assembly, material handling, surface treatment and other tasks requiring precise force control, traditional control methods are difficult to adapt to material characteristics and environmental changes.

[0004] In recent years, reinforcement learning, as an important branch of artificial intelligence, has achieved remarkable results in the fields of games, autonomous driving and the like. However, applying reinforcement learning to robot motion control still faces many challenges, including how to construct an effective state representation, how to design a reward function suitable for the control scenario, and how to ensure the safety and robustness of the learned strategy in the actual environment.

[0005] Currently, there is a lack of a robot motion control algorithm that can be based on torque perception and adapt to different physical environments in the prior art, and there is an urgent need to develop new control methods to address this challenge. SUMMARY

[0006] The present application aims to solve the problem of the robot control system in the prior art being difficult to adapt to environmental changes and lacking force interaction perception capability, and provides a robot adaptive motion control algorithm based on reinforcement learning. The algorithm realizes adaptive motion control of the robot in different physical environments through a torque perception mechanism, a hierarchical reinforcement learning architecture, a multi-dimensional reward system, a dynamic mapping mechanism and a multi-environment verification system.

[0007] The present application proposes a robot adaptive motion control algorithm based on reinforcement learning, comprising:

[0008] Obtaining the torque signal of the robot joint, and constructing a state vector representing the state of the robot interacting with the environment based on the torque signal;

[0009] generating an adaptive motion control strategy through a hierarchical reinforcement learning training mechanism based on the state vector, comprising:

[0010] constructing a multi-dimensional reward function integrating task completion degree, energy efficiency, motion safety and smoothness;

[0011] iteratively optimizing the mapping relationship between the state vector and the action control sequence using a Q-learning algorithm;

[0012] verifying the adaptive motion control strategy under multiple physical parameter environments and selectively triggering re-training based on the verification results;

[0013] deploying the verified adaptive motion control strategy to a robot control system to achieve adaptive motion control.

[0014] Preferably, the torque signal of the robot joint is acquired, and a state vector representing the interaction state of the robot and the environment is constructed based on the torque signal, comprising:

[0015] real-time acquisition of torque data through torque sensors installed on each joint of the robot;

[0016] low-pass filtering and normalization processing of the torque data;

[0017] constructing a multi-dimensional state vector based on the current torque value, the previous torque value and its rate of change;

[0018] real-time monitoring of the interaction between the robot and virtual or actual obstacles based on the state vector.

[0019] Preferably, the hierarchical reinforcement learning training mechanism comprises:

[0020] a bottom layer joint torque direct control layer responsible for high-frequency and small-granularity torque adjustment;

[0021] a middle layer motion trajectory planning layer responsible for medium-frequency and medium-granularity trajectory optimization;

[0022] a high layer task strategy decision layer responsible for low-frequency and large-granularity task planning;

[0023] a closed-loop control structure formed through top-down command transmission and bottom-up state feedback between the three layers.

[0024] Preferably, the multi-dimensional reward function integrating task completion degree, energy efficiency, motion safety and smoothness comprises:

[0025] defining a task completion degree reward based on target position achievement degree and completion time efficiency evaluation;

[0026] Define energy efficiency reward, calculated based on joint power consumption and total energy consumption;

[0027] Define safety reward, evaluated according to distance to obstacles and degree of dangerous posture;

[0028] Define smoothness reward, calculated based on torque change rate and trajectory continuity;

[0029] Linearly combine the above four-dimensional rewards through dynamic weight coefficients to form a comprehensive reward function.

[0030] As a preferred, the iterative optimization of the mapping relationship between the state vector and the action control sequence by the Q-learning algorithm comprises:

[0031] Discretize the continuous state space using an adaptive grid method, and use fine-grained grids in critical areas;

[0032] Select actions based on the ε-greedy strategy, increase exploration rate at the beginning of training, and increase utilization rate later;

[0033] Optimize the state-action value function through iterative updating controlled by learning rate and discount factor;

[0034] Use the priority experience replay mechanism to improve learning efficiency, and samples with high TD error have higher sampling probability.

[0035] As a preferred, the adaptive motion control strategy is verified under multiple physical parameter environments, and selectively triggers retraining based on the verification results, comprising:

[0036] Construct a parameterized verification environment set with different friction coefficients, damping coefficients and mass distributions;

[0037] Perform multiple independent tests in each verification environment to evaluate functional indicators, efficiency indicators, safety indicators and robustness indicators;

[0038] Calculate the average performance indicators and the worst performance indicators under all verification environments;

[0039] When the worst performance indicator is lower than the threshold or the performance variance exceeds the preset range, trigger the retraining process.

[0040] As a preferred, the adaptive motion control strategy that passes the verification is deployed to the robot control system, comprising:

[0041] Compress and optimize the control strategy to reduce memory usage and improve query efficiency;

[0042] Establish a standardized control interface to achieve compatibility with different robot platforms;

[0043] Design runtime monitoring and fine-tuning mechanism to support online performance optimization;

[0044] Implement policy version management to support rollback to stable version in abnormal situations.

[0045] As preferred, the action control sequence includes joint torque increments, each joint having seven discrete action levels of large increment, medium increment, small increment, hold, small decrement, medium decrement, or large decrement.

[0046] As preferred, the weight coefficients of each dimension in the multi-dimensional reward function are dynamically adjusted according to the task stage, environmental characteristics, and historical performance, ensuring adaptive adjustment of the policy optimization direction in different application scenarios.

[0047] As preferred, each layer in the hierarchical reinforcement learning training mechanism uses different time scales and decision granularities:

[0048] The bottom layer control cycle is 5-10 milliseconds, directly operating joint torque;

[0049] The middle layer control cycle is 50-100 milliseconds, planning short-term trajectory segments;

[0050] The high-level control cycle is 500-1000 milliseconds, responsible for overall task policy adjustment;

[0051] Each layer achieves asynchronous collaborative control through a buffer mechanism.

[0052] The beneficial effects of the present application mainly include:

[0053] 1. Significant improvement in adaptability: Through state representation based on torque signals and reinforcement learning training, the robot control strategy can adapt to ±30% range of physical parameter changes without the need for manual intervention to adjust parameters.

[0054] 2. Energy efficiency is greatly improved: The multi-dimensional reward function includes energy efficiency evaluation indicators, allowing the control strategy to automatically optimize energy consumption during the learning process, reducing energy consumption by 20%-35% compared to traditional control methods.

[0055] 3. Essential improvement in safety performance: Through the safety reward mechanism, the control strategy can actively avoid obstacles, increasing the minimum safety distance from obstacles by 25%, significantly improving operational safety.

[0056] 4. Comprehensive optimization of control quality: Positioning accuracy is improved by 40% in changing environments, and motion trajectory smoothness is improved by 50%, significantly extending the life of mechanical systems.

[0057] 5. Significant improvement in deployment efficiency: The time from training to deployment is reduced by 50%, and the system debugging cycle is reduced by 60%, significantly reducing application costs. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 System architecture diagram of the adaptive motion control algorithm for the robot based on reinforcement learning of the present application;

[0059] Figure 2 Structure diagram of the torque sensing and state construction module of the present application;

[0060] Figure 3 Hierarchical structure diagram of the hierarchical reinforcement learning training mechanism of the present application;

[0061] Figure 4 Flowchart of the multi-dimensional reward function construction of the present application;

[0062] Figure 5 Training flowchart of the Q-learning algorithm of the present application;

[0063] Figure 6 Flowchart of the multi-environment verification and strategy deployment of the present application; DETAILED DESCRIPTION

[0064] Reference should be made to the accompanying Figures 1-6 The present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. These embodiments are only used to illustrate the present application, and are not used to limit the scope of the present application.

[0065] As shown in Figure 1 The system architecture of the adaptive motion control algorithm for the robot based on reinforcement learning provided by the present application mainly includes a torque sensing module 101, a hierarchical reinforcement learning training module 102, an adaptive control strategy module 103, a multi-environment verification module 104 and a strategy deployment module 105.

[0066] The adaptive motion control algorithm for the robot based on reinforcement learning provided by the present application includes acquiring torque signals of joints of the robot, constructing a state vector based on the torque signals, generating an adaptive control strategy through a hierarchical reinforcement learning training mechanism, verifying the strategy in multiple physical environments, and finally deploying the strategy that passes the verification to a robot control system.

[0067] Figure 2 The specific implementation of the torque sensing and state construction module is shown. Taking a six-axis industrial robot as an example, this module acquires torque data in real time through torque sensors 201 installed at each joint of the robot, and the sampling frequency is set to 1000 Hz. Such a high sampling frequency is crucial for capturing rapid torque changes generated by the robot during the execution of assembly tasks or the process of collaboration with humans.

[0068] Preferably, the acquired raw torque signal is low-pass filtered by the signal processing unit 202, and the cutoff frequency of the filter is set to 50Hz. In practical applications, such as when a robot performs precision assembly tasks, environmental vibrations and motor noise often introduce noise signals higher than 50Hz. This cutoff frequency can effectively eliminate these high-frequency noises while retaining useful torque change information generated during the contact between the robot and the workpiece.

[0069] The filtered signal needs to be normalized to map the torque values ​​of different joints to the [-1,1] interval. For example, for an industrial robot with a maximum joint torque of 200 Nm, its original torque value (e.g., 87 Nm) will be normalized to 0.435 to facilitate subsequent reinforcement learning algorithm processing.

[0070] When mapping the torque values ​​of different joints to the interval [-1, 1], the standard normalization formula is used:

[0071] Normalized value ,

[0072] For the joint torque of an industrial robot, assuming the maximum rated torque range is 200 Nm, then:

[0073] The normalized positive torque of 87 Nm is: ,

[0074] The negative torque of -87 Nm is normalized to: -0.435,

[0075] In one embodiment of the present invention, the state construction unit 203 constructs a multidimensional state vector based on the processed torque signal, and the mathematical representation of the state vector is as follows:

[0076] ,

[0077] in, This represents the current torque value of the i-th joint. This represents the torque value of the i-th joint at the previous moment. Indicates the rate of change of torque. , The control cycle is typically 5-10 milliseconds. For a robot with n joints, or a six-axis industrial robot, n=6.

[0078] In a practical example, the torque data for the six joints are [5.2, -3.1, 8.7, 0.5, -1.2, 2.3] Nm. Assuming the maximum rated torque for each joint is 20 Nm, the normalized value should be:

[0079] ,

[0080] ,

[0081] ,

[0082] ,

[0083] ,

[0084] ,

[0085] Therefore, the first part of the normalized state vector should be [0.26, -0.155, 0.435, 0.025, -0.06, 0.115]. Similarly, the torque value and torque rate of change at the previous time can be normalized. This normalization ensures that torque data of different magnitudes have the same weight in the learning algorithm, preventing learning bias caused by different dimensions.

[0086] In addition, the state monitoring unit 204 is responsible for real-time monitoring of the interaction between the robot and virtual or actual obstacles. When the torque value of any joint exceeds the preset threshold (for example, 80% of the rated torque), the system identifies a collision with an obstacle and generates a termination signal. In practical applications, such as when a robot operates in an unknown environment, this torque-based collision detection method can detect potential collision risks in advance and avoid dangerous contact between the robot and the environment or humans. For example, a collaborative robot with a maximum joint torque of 100 Nm will immediately slow down or stop when any joint torque exceeds 80 Nm, ensuring safe operation.

[0087] As shown in FIG. 3, Figure 3 The present application adopts a hierarchical reinforcement learning training mechanism, including a bottom layer joint torque direct control layer 301, a middle layer motion trajectory planning layer 302, and a high layer task strategy decision layer 303.

[0088] The bottom layer joint torque direct control layer 301 is responsible for high-frequency and small-granularity torque adjustment, directly operating the robot joint actuator. In industrial scenarios, for example, when a robot performs precision gear assembly, the bottom controller can sense the small resistance changes in the gear meshing process in real time based on the joint torque signal and adjust the applied torque to ensure smooth insertion without damaging the parts.

[0089] The middle layer motion trajectory planning layer 302 is responsible for medium-frequency and medium-granularity trajectory optimization, converting high-level task instructions into executable trajectory segments. Taking the carrying task as an example, when the robot needs to move heavy objects in a factory environment, the middle planner will consider path smoothness and energy efficiency to generate the optimal trajectory segment, avoiding the shaking of objects or energy waste caused by sudden acceleration or deceleration.

[0090] The high-level task policy decision layer 303 is responsible for low-frequency and large-granularity task planning, and generates policy instructions according to the overall task target. For example, in a complex assembly line, the high-level decision maker plans the optimal work order and station switching strategy for the robot according to the current production demand and workpiece state, thereby improving the overall production efficiency.

[0091] Preferably, a closed-loop control structure is formed among the three layers through top-down command transmission and bottom-up state feedback, realizing multi-scale collaborative control. Specifically, the high-level decision result (such as "grab workpiece A and move to position B") is transmitted to the middle layer through the instruction buffer, the middle layer converts it into a specific trajectory segment (such as moving from the current position to the position of workpiece A, and moving to position B along the planned path after grabbing), and then the specific joint torque control is transmitted to the bottom layer through the control buffer. At the same time, the bottom layer execution result and state information (such as the current torque being too large, which may collide with obstacles) are transmitted upwards through the feedback buffer, helping the upper layer to adjust the decision.

[0092] In a certain automobile assembly line application case, this hierarchical architecture enables the robot to efficiently complete the gearbox assembly task. The high-level decision maker determines the assembly sequence and station switching strategy, the middle layer planner generates a smooth and efficient motion trajectory, and the bottom layer controller accurately controls the insertion depth and alignment force according to the torque feedback during assembly, ultimately improving the assembly success rate from 92% to 99.5% compared to the traditional control method.

[0093] As shown in Figure 4 , the present application constructs a multi-dimensional reward function that integrates task completion degree, energy efficiency, motion safety, and smoothness.

[0094] In the embodiment of the industrial robot performing a precision assembly task, the task completion degree reward is evaluated according to the target position achievement degree and completion time efficiency, and the specific calculation formula is:

[0095] ,

[0096] wherein, is the current position of the robot, represented as a three-dimensional space coordinate [x, y, z] with a unit of meters (m); is the target position, also represented as a three-dimensional space coordinate with a unit of meters (m); represents the Euclidean distance between the current position and the target position, with a unit of meters (m); is a normalized distance reference value, with a unit of meters (m), and is usually set to half of the workspace diameter, for example, for a robot with a work radius of 1.5 m, can be set to 1.5 m; is the current task execution time, with a unit of seconds (s); is the maximum allowed time, in seconds (s), set according to the complexity of the task, for example, 30 s for a precision insertion task; and are the weight coefficients for position accuracy and time efficiency, respectively, dimensionless, and the preferred values are 0.7 and 0.3, respectively.

[0097] For example, in a precision bearing assembly task, the current robot position is [0.82, 0.45, 0.33] m, the target position is [0.80, 0.45, 0.30] m, the current task has been executed for 15 seconds, the maximum allowed time is 30 seconds, is 1.5 m, then the position error is about 0.032 m, and the task completion degree reward is calculated as: . This calculation method takes into account the time efficiency while ensuring the position accuracy.

[0098] Energy efficiency reward Based on joint power consumption and total energy consumption, the calculation formula is:

[0099] ,

[0100] where, is the torque of the i-th joint, in Newton-meters (Nm); is the angular velocity of the i-th joint, in radians per second (rad / s); represents the absolute value of the instantaneous power of the i-th joint, in watts (W); is the control period, in seconds (s); is the energy consumption penalty coefficient, in 1 / J, and the preferred value is 0.05; is the number of robot joints; the negative sign indicates that the higher the energy consumption, the greater the penalty (the smaller the reward).

[0101] For example, in an industrial robot transportation task, assume that the joint torque of a six-axis robot at a certain moment is [15, 25, 30, 5, 8, 3] Nm, and the corresponding angular velocity is [0.2, 0.1, 0.15, 0.3, 0.2, 0.4] rad / s, and the control period is 0.01 seconds, then the energy consumption is calculated as: (15×0.2+25×0.1+30×0.15+5×0.3+8×0.2+3×0.4)×0.01=0.131 Joules, and the corresponding energy efficiency reward is .

[0102] Safety reward According to the distance from the obstacle and the degree of dangerous posture evaluation, the calculation formula is:

[0103] ,

[0104] where, is the distance between the robot and the nearest obstacle, in meters (m); is the distance sensitivity coefficient, in 1 / meter (1 / m), and the preferred value is 5.0, which makes the reward change most sensitive when the obstacle distance is in the range of 0.2-1 m; is the distance decay function, which decreases with increasing distance; is the distance between the robot and the joint limit, expressed as an angle, in radians (rad); is the joint movable range, in radians (rad); and are the weights for obstacle distance and pose safety, respectively, dimensionless, and the preferred values for both are 0.5.

[0105] In human-robot collaboration scenarios, such as when a robot and a human work together to complete an assembly task, assume that the current distance between the robot and the nearest human body part is 0.5 m, and the distance to the joint limit is 0.2 rad (the joint movable range is 2 rad), then the safety reward calculation is:

[0106] ,

[0107] This reward mechanism encourages the robot to maintain a safe distance while avoiding approaching the joint limit.

[0108] Smoothness reward Based on the torque change rate and trajectory continuity, the calculation formula is:

[0109] ,

[0110] where, is the torque change rate of the i-th joint, in Nm / s, and the calculation method is to divide the torque difference of the adjacent two control periods by the control period; is the smoothness penalty coefficient, in s / Nm, and the preferred value is 0.03; n is the number of robot joints; the negative sign indicates that the greater the torque change, the greater the penalty (the smaller the reward).

[0111] In a precision gluing task, the robot needs to apply glue evenly along the edge of the workpiece, at which time the motion smoothness is crucial. Assume that the torque change rates of the six joints at a certain moment are [20, -15, 10, -5, 8, -12] Nm / s, then the smoothness reward calculation is: . This reward mechanism promotes the control system to generate more smooth torque control output, reducing mechanical vibration and process fluctuations.

[0112] Finally, the rewards of the above four dimensions are linearly combined by dynamic weight coefficients to form a comprehensive reward function:

[0113] ,

[0114] wherein, are weight coefficients of each dimension, dimensionless, initial values are set to 0.5, 0.2, 0.2, 0.1 respectively, and can be dynamically adjusted as the training progresses and the nature of the task. The sum of the weight coefficients is equal to 1, that is, .

[0115] In actual implementation, each reward should be normalized to make it in a similar numerical range:

[0116] ,

[0117] wherein, is the original reward value of a certain dimension, and are the minimum and maximum values of the experience of the reward of the dimension respectively, is the normalized reward value.

[0118] Another method is to introduce an adaptive scale coefficient

[0119] ,

[0120] wherein, and are adaptive scale coefficients, which are determined in the following way:

[0121] ,

[0122] For example, if the statistical median amplitudes of the four-dimensional rewards are 0.8, 0.007, 0.9, and 2.0 respectively, the scale coefficient of the energy efficiency reward should be , and the scale coefficient of the smoothness reward should be .

[0123] After applying these scale coefficients, the energy efficiency reward is adjusted to , and the smoothness reward is adjusted to , which are of the same order of magnitude as the task completion reward (0.835) and the safety reward (0.909), ensuring that each dimension reward has appropriate influence in the optimization process.

[0124] As shown in Figure 5 , the present application uses the Q-learning algorithm to iteratively optimize the mapping relationship between the state vector and the action control sequence.

[0125] In practical applications, such as robots performing assembly tasks, adaptive grid method is used to discretize continuous state space. Due to the high dimensionality of torque state space (for a six-axis robot, the dimension of state vector is 18), uniform grid partitioning will lead to state space explosion. Therefore, the present application dynamically adjusts the grid granularity according to the sample distribution density in the state space, and uses fine-grained grid in the key area (such as near the target position or near the obstacle) and coarse-grained grid in the non-key area.

[0126] In specific implementation, for the assembly task of a six-axis robot, the initial uniform grid can be divided into 10 18 units, which is obviously impractical. Through the adaptive grid method, the system focuses on the joints with significant torque value changes and subdivides the state space of these joints, finally reducing the effective state space to about 100,000 discrete states, significantly improving the learning efficiency.

[0127] Based on the ε-greedy strategy, the exploration rate is increased at the beginning of training, and the utilization rate is increased at the later stage. The change of exploration rate ε follows the following formula:

[0128] ,

[0129] wherein, is the minimum exploration rate, dimensionless, set to 0.05, ensuring that the system always maintains a certain exploration ability; is the maximum exploration rate, dimensionless, set to 0.8, so that the system can explore the state space a lot at the beginning of training; is the decay coefficient, unit is 1 / step, set to 0.0001, controlling the decay speed of exploration rate; is the training step number, unit is step; is the exponential decay function.

[0130] In actual assembly task training, for example, at the 1000th step, the exploration rate is calculated as: , At this time, the system has a 72.9% probability of randomly selecting actions for exploration, and a 27.1% probability of selecting the action with the maximum current Q value. As the training progresses, for example, to the 10000th step, the exploration rate drops to 0.207, and the system makes more use of the learned experience.

[0131] Q value update uses the standard Q-learning iteration formula:

[0132] ,

[0133] wherein, is the Q value of action a performed in state s, dimensionless; Learning rate, dimensionless, initial value is set to 0.1, gradually reduced to 0.01 during training; Reward, dimensionless, obtained after performing action a; Discount factor, dimensionless, set to 0.9, balances current reward and future reward; New state after performing action a; New state Maximum Q value, dimensionless; Temporal difference (TD) error, represents the difference between actual reward obtained and expected.

[0134] Taking a robot precision insertion task as an example, assuming that the torque signal corresponding to the current state s indicates that the robot has just touched the edge of the insertion hole, performing action a (fine-tuning the pose and slightly applying a downward force) obtains reward R = 0.5, the new state s' shows better alignment with the insertion hole, and the corresponding maximum Q value is 0.8, the current value is 0.3, and the learning rate a is 0.1, then the Q value is updated to: .

[0135] Preferably, the present application uses a priority experience replay mechanism to improve learning efficiency. The system maintains an experience pool with a size of 30000 samples, which is particularly suitable for learning of complex robot assembly tasks. The priority of each sample is determined according to the absolute value of the TD error:

[0136] ,

[0137] Where, TD error of sample i, dimensionless, calculated by ; Small constant, dimensionless, prevents priority from being zero when error is zero, set to 0.01; Priority index, dimensionless, controls the degree of priority difference, set to 0.6; Priority of sample i, dimensionless.

[0138] Sampling probability is proportional to priority:

[0139] ,

[0140] Where, Probability of sample i being selected, dimensionless; Sum of priorities of all samples, dimensionless.

[0141] In practical applications, for example, when a robot learns a screw tightening task, the experience sample generated when the screw just touches the screw hole but is not aligned usually has a large TD error, because it is a key turning point for successful tightening. Assuming that the absolute value of the TD error of this sample is 0.8, which is much higher than the average value 0.2, the priority calculation is , and the priority of the average sample is . Assuming that there are 10,000 samples in the experience pool, and the total priority sum is 5,000, the probability of this high-value sample being selected is 0.865 / 5000 = 0.000173, which is about 1.73 times the average probability (0.501 / 5000 = 0.0001). This mechanism makes samples with high TD error (i.e., more learnable) have a higher probability of being sampled, thereby accelerating the learning process.

[0142] As shown in Figure 6 , the present application verifies the adaptive motion control strategy under various physical parameter environments, and selectively triggers retraining based on the verification results.

[0143] In practical industrial applications, robots need to adapt to various working conditions, such as assembling parts of different materials, carrying objects of different weights, or performing polishing tasks on different surfaces. To simulate these changes, the present application constructs a set of verification environments with different physical parameters.

[0144] Taking an automobile parts assembly line as an example, the range of environmental parameter changes is set as follows:

[0145] Friction coefficient μ: 0.1-0.5, with a step of 0.1 (corresponding to smooth metal surface to rough plastic surface)

[0146] Damping coefficient ζ: 0.05-0.2, with a step of 0.05 (corresponding to a light damping to a strong damping system)

[0147] Mass variation Δm: -20% to +20%, with a step of 10% (corresponding to weight changes of different models of parts)

[0148] Based on the above parameter combinations, a total of 5x4x5=100 different verification environments can be constructed, covering most of the working conditions that robots may encounter. For example, the verification environment corresponds to a friction coefficient of 0.3, a damping coefficient of 0.15, and a mass variation of +10%, simulating an assembly scenario with medium friction, large damping, and slightly heavy parts.

[0149] In each verification environment, multiple independent tests (usually 10) are performed to evaluate the following indicators:

[0150] Performance indicators: including task completion rate (the proportion of successful tests to the total number of tests), average completion time (the average time required to successfully complete a task), control accuracy (root mean square error of end position)

[0151] Efficiency indicators: including energy consumption (total energy consumption during task execution), computational resource occupation (CPU and memory usage), response delay (time delay from receiving instructions to executing actions)

[0152] Safety indicators: including collision times (number of collisions with the environment or obstacles), minimum safety distance (minimum distance maintained from obstacles), torque peak (maximum joint torque during execution)

[0153] Robustness indicators: including parameter variation sensitivity (degree of fluctuation in performance with parameter changes), fault recovery ability (recovery speed after encountering interference), long-term stability (performance retention ability after long-term operation)

[0154] Calculate the average performance indicators in all validation environments and the worst performance indicators :

[0155] ,

[0156] ,

[0157] where, is the performance indicator in the ith environment, dimensionless, with a value range of 0-1, and the larger the value, the better the performance; is the total number of validation environments, which is 100 in this example; represents the arithmetic mean of the performance indicators in all environments; represents the worst indicator value in all environments.

[0158] Performance indicators are composed of various sub-indicators:

[0159] ,

[0160] where, is the functional indicator, dimensionless, with a value range of 0-1; is the efficiency indicator, dimensionless, with a value range of 0-1; is the safety indicator, dimensionless, with a value range of 0-1; is the robustness indicator, dimensionless, with a value range of 0-1; , , , The weight coefficients of each index are dimensionless, and the typical values are 0.4, 0.2, 0.3, and 0.1, respectively, and the sum of the weights is equal to 1, that is .

[0161] In the application case of an automobile parts assembly line, the indicators of a certain environment (friction coefficient 0.4, damping coefficient 0.1, and mass change -10%) may be: (high completion rate and precision), (good energy efficiency), (high safety), (good robustness), and the comprehensive performance index in this environment is: .

[0162] When the worst performance index is lower than the threshold value (usually set to 0.7) or the performance variance exceeds the preset range (usually 0.2), the retraining process is triggered. For example, if the performance index of 99 environments is above 0.8 among 100 validation environments, but the performance index in a specific environment (such as high friction, low damping, and heavy load) is only 0.65, the system will trigger retraining and increase the training proportion under this environment condition to improve the generalization ability of the control strategy.

[0163] The application deploys the adaptive motion control strategy that passes the verification to a robot control system, including the following steps:

[0164] The control strategy is compressed and optimized to reduce memory occupation and improve query efficiency. In the case of limited resources of an industrial robot controller, this step is particularly important. Taking a certain six-axis industrial robot as an example, the original Q table may contain about 100,000 state-action pairs, occupying 50 MB of memory. Through strategy distillation technology, the strategy represented by the Q table can be converted into a more compact neural network representation (for example, a 3-layer neural network with 50 nodes in each layer), reducing the memory occupation to about 5 MB, and increasing the query speed by 2-3 times. Another method is to delete the state-action pairs with low value (lower than the threshold value 0.1) or rarely accessed (access frequency lower than 0.1% of the total frequency) in the Q table through pruning technology, which can usually reduce 40% to 60% of the table entries with a performance loss of not more than 5%.

[0165] A standardized control interface is established to achieve compatibility with different robot platforms. In practical industrial applications, factories may use robots from multiple brands simultaneously. This invention defines a unified API, including a status input interface (receiving sensor data such as torque and position), a motion output interface (sending torque control commands), and a parameter configuration interface (adjusting control parameters). For example, in a car factory, robots from both ABB and KUKA brands are used on the same assembly line. Through the standard interface layer of this invention, the same adaptive control strategy can be seamlessly applied to both robots, significantly reducing system integration and maintenance costs.

[0166] The system incorporates runtime monitoring and fine-tuning mechanisms to support online performance optimization. Key performance indicators (KPIs) are calculated hourly, such as average positioning error (target value < 0.1 mm), energy consumption (target value < 80% of preset baseline), and motion smoothness (jitter amplitude < 0.5 mm). When a KPI deviates from expectations, a fine-tuning process is triggered. For example, in a precision parts assembly task, if the system detects an increase in average positioning error from 0.08 mm to 0.12 mm, it immediately initiates a fine-tuning process. Through incremental learning, it adjusts the parameters related to precise positioning in the Q-table, reducing the positioning error back to 0.07 mm without affecting other performance metrics.

[0167] Implement strategy version management, supporting rollback to a stable version in case of anomalies. The system maintains the five most recent strategy versions, each containing complete strategy parameters and performance evaluation results. On an electronic component assembly line, after deploying a new strategy version, an 8% assembly failure rate was found on specific part types (compared to only 2% in the previous version). The system immediately rolled back to the previous stable version to ensure the production line continues to operate reliably, while recording the problem data for subsequent strategy improvements.

[0168] The motion control sequence in this invention includes joint torque increments, and each joint can select seven discrete motion levels.

[0169] In scenarios where industrial robots perform precision assembly tasks, the seven discrete motion levels are defined as follows:

[0170] Significant increase: The current torque value increases by 30%, for example, from 20 Nm to 26 Nm;

[0171] Medium increase: The current torque value increases by 15%, for example, from 20 Nm to 23 Nm;

[0172] Slight increase: The current torque value increases by 5%, for example, from 20 Nm to 21 Nm;

[0173] Maintain: Keep the current torque value unchanged at 20 Nm;

[0174] Slight reduction: The current torque value is reduced by 5%, for example, from 20 Nm to 19 Nm;

[0175] Medium reduction: The current torque value is reduced by 15%, for example, from 20 Nm to 17 Nm;

[0176] Significant reduction: The current torque value is reduced by 30%, for example, from 20 Nm to 14 Nm;

[0177] This design is very practical in real-world applications. For example, when a robot is performing a bearing press-fit task, when it detects that the bearing is beginning to enter the shaft hole, the system may choose to slightly increase the motion and gently increase the pressure; when it detects a sudden increase in resistance, it may choose to moderately or significantly reduce the motion to avoid damaging the parts; when the bearing is close to the target position, it may choose to maintain the motion and keep the pressure stable until the press-fit is completed.

[0178] For a six-axis industrial robot, the theoretical motion space is 7. 6 =117649, which is computationally difficult to handle. To reduce the dimensionality of the motion space, dimensionality reduction techniques such as Principal Component Analysis (PCA) can be used in assembly tasks. For example, in a vertical insertion task, only the Z-direction torque of the end effector and the attitude fine-tuning in the XY plane need to be controlled, which can reduce the motion space to 7. 3 =343 effective actions, greatly improving learning efficiency.

[0179] Furthermore, to prevent control signal jitter, this invention also introduces a motion smoothing mechanism. When the new motion selected by the algorithm changes too much from the previous motion, for example, from a large increase (+30%) to a large decrease (-30%), the system automatically inserts an intermediate transition motion, such as a small increase → hold → small decrease → medium decrease → large decrease, making the torque change smoother. After applying this mechanism on a motor assembly line, mechanical vibration was reduced by 65%, and the part damage rate dropped from 3% to below 0.5%.

[0180] In the multi-dimensional reward function of this invention, the weight coefficients of each dimension are dynamically adjusted according to the task stage, environmental characteristics, and historical performance.

[0181] Regarding task phase adaptation, taking the robot performing a pin assembly task as an example, the system divides the task into three phases:

[0182] 1. Initial Phase (Approaching Target Position): This phase prioritizes safety and smoothness of movement, with weights set to... (Task completion rate) (Energy efficiency) (Security) (Smoothness). This ensures that the robot can approach the work area safely and smoothly.

[0183] 2. Execution phase (insertion process): This phase focuses more on task completion and energy efficiency, with weight adjustments as , This encourages the robot to efficiently and accurately complete the insertion action.

[0184] 3. Completion phase (pin securing): This phase mainly focuses on precise positioning, with weight adjustments as . Specifically, the positional accuracy sub-weight in is increased to ensure the pin is fully seated.

[0185] In terms of environmental characteristic adaptation, the system adjusts weights based on the characteristics of the current environment. For example, when a robot needs to operate near a human work area on a production line, sensors detect that a worker is approaching (distance less than 1.5 meters), the system automatically increases the safety weight from the standard value of 0.2 to 0.5, while appropriately reducing other weights, ensuring safe operation. Conversely, in a completely isolated work unit, the safety weight can be appropriately reduced, and the energy efficiency or task completion weight can be increased to achieve more efficient production.

[0186] In terms of historical performance adaptation, the system continuously evaluates performance indicators in each dimension and increases the reward weight for dimensions that perform poorly. For example, on a certain assembly line, the system detects through energy consumption monitoring that the average energy consumption of the robot has exceeded the expected value by 20% for 5 consecutive hours, and automatically increases the energy efficiency reward weight from 0.2 to 0.3, prompting the system to learn more energy-efficient control strategies. After a week, when the energy consumption indicator returns to the normal range, the weight can gradually return to the standard configuration.

[0187] Preferably, the weight adjustment adopts gradual changes to avoid control instability caused by sudden changes. The mathematical representation of weight adjustment is:

[0188] ,

[0189] where is the weight value at time t, dimensionless; is the weight value at the next time, dimensionless; is the weight adjustment amount, dimensionless, satisfying , i.e., the adjustment amplitude does not exceed 20% of the current value. At the same time, the sum of all weights remains 1, i.e., after adjusting one weight, other weights are adjusted proportionally to ensure .

[0190] For example, a collaborative robot performs tasks on an assembly line with initial weights When the safety distance is detected to approach the warning line (0.5 meters) multiple times, it is decided to increase the safety weight. The maximum adjustable amount of the safety weight is , the system selects 0.03, that is . In order to keep the weight and 1, other weights are reduced in proportion: 0.48125, , The adjusted weight is , and 1 is still kept.

[0191] In the hierarchical reinforcement learning training mechanism of the application, different time scales and decision granularities are adopted in each layer, mainly in three aspects of control period, operation granularity and information transmission mechanism.

[0192] In actual industrial applications, the control parameters of each layer are different for different task requirements. Taking the automobile gearbox assembly task as an example:

[0193] The bottom layer control period is set to 5-10 milliseconds, directly operating the joint torque. For example, when the gearbox gear and shaft are connected, the bottom layer controller collects torque sensor data at a period of 10 ms, detects the resistance change in the connection process in real time, and outputs the torque control command at the same frequency to adjust the force and direction. This control period setting is based on the response characteristics of the robot servo system (usually 100-200 Hz), which can meet the real-time requirements of accurate force control. The operation accuracy of the bottom layer controller is usually 1%~2% of the rated torque, for example, for a joint with a maximum torque of 100 Nm, the control accuracy can reach 1-2 Nm.

[0194] The middle layer control period is set to 50-100 milliseconds, planning short-term trajectory segments. In the gearbox assembly process, the middle layer planner processes the assembly instructions from the high layer at a period of 50 ms, decomposes the task of installing the gearbox into a series of trajectory segments such as approaching, aligning, inserting and fixing, each trajectory segment containing 5-10 bottom layer control points. The main responsibility of the middle layer is to ensure smooth transition between stages, continuous motion, and avoid impact and vibration. The time span of the middle layer control is usually 0.5-2 seconds, corresponding to a single action stage in the gearbox assembly.

[0195] The high layer control period is set to 500-1000 milliseconds, responsible for overall task strategy adjustment. In the production line environment, the high layer decision maker updates the task state once per second, adjusts the assembly strategy and priority according to the assembly progress, part supply situation and quality inspection feedback. For example, when it is detected that the gap between the gears of a batch of gearboxes is large, the high layer decision maker will adjust the assembly parameters, increase the alignment time and accuracy requirements. The time span of the high layer control is usually 5-30 seconds, covering the complete gearbox assembly period.

[0196] To ensure the time scale of high-level control cycle (0.5-1 second) matches the time scale of task validation cycle (5-30 seconds), the following methods are adopted:

[0197] 1. Hierarchical reward allocation mechanism, high-level task is decomposed into multiple phased sub-goals, each sub-goal has immediate reward signal. For example, in the gearbox assembly task, the complete task (30 seconds) is decomposed into approaching, aligning, inserting, fixing, etc. stages, each stage has independent evaluation indicators:

[0198] Approaching stage: distance reduction rate to target position;

[0199] Alignment stage: degree of attitude error reduction;

[0200] Insertion stage: appropriateness of contact force and insertion progress;

[0201] Fixing stage: final position accuracy and stability;

[0202] This decomposition allows the high-level controller to obtain meaningful reward signals every 0.5-1 second, without waiting for the final result after 30 seconds.

[0203] 2. Temporal difference learning and n-step backtracking, the algorithm uses temporal difference (TD) learning method, specifically using n-step backtracking mechanism:

[0204] ,

[0205] Where, is the immediate reward of the continuous n time steps, is the discount factor, is the learning rate, is the maximum Q value of the state after n steps. For this system, the typical n value is 10-20, covering a time span of 5-10 seconds, which can capture the reward signal of medium time scale.

[0206] 3. To handle long-time dependence problem, the system introduces eligibility trace (Eligibility Traces) mechanism:

[0207] ,

[0208] ,

[0209] Where, is the eligibility trace of state-action pair , is the eligibility trace decay parameter (typical value 0.9), is the TD error of the current time step, is the indicator function, when the current state-action pair is 1 if same, 0 otherwise.

[0210] The eligibility traces mechanism makes past state-action pairs obtain proper reward updates according to their contribution to the current state, effectively solving the time scale mismatch problem.

[0211] 4. Real-time prediction of task completion indicator, the system trains an auxiliary value network to predict the final task success probability based on the current state:

[0212] ,

[0213] This prediction value serves as an additional reward signal, providing real-time feedback on long-term goals:

[0214] ,

[0215] where, is the predicted reward weight coefficient, is the change in task success probability, a positive value indicates progress towards success, and a negative value indicates deviation from the goal.

[0216] Through the combined application of the above four mechanisms, the invention effectively solves the coordination problem between different time scale control periods and reward signals, enabling the high-level controller to obtain meaningful learning signals in a short period while maintaining the optimization direction of long-term task goals. These mechanisms exhibit excellent learning efficiency and control performance in complex tasks such as gearbox assembly.

[0217] Asynchronous cooperative control between layers is achieved through buffer mechanisms. Taking gearbox assembly as an example, the task instructions passed from the high layer to the middle layer (such as installing an A-type gearbox) are completed through a task buffer with a capacity of 10; the trajectory segments passed from the middle layer to the bottom layer (such as moving downward at a speed of 2 mm / s for 15 mm) are completed through a trajectory buffer with a capacity of 20; the state information feedback from the bottom layer to the upper layer (such as detecting a resistance of 12 N) is completed through a state buffer with a capacity of 50. Each buffer uses a first-in, first-out (FIFO) queue structure to ensure the timing and integrity of information transmission.

[0218] Preferably, the decision granularity of each layer is as follows:

[0219] Bottom layer: torque increment control, for a joint with a maximum torque of 100 Nm, the typical control increment is ±30 Nm, divided into 7 levels;

[0220] Middle layer: trajectory segment planning, typical length is 0.5-2 seconds, containing 10-40 bottom layer control points;

[0221] High layer: task strategy, typical time span is 5-30 seconds, containing 3-10 middle layer trajectory segments;

[0222] In the practical application of a certain automobile assembly plant, this multi-time scale hierarchical control architecture makes the transmission assembly efficiency increase by 35%, the assembly precision increase by 40%, and the energy consumption reduce by 25%. The key lies in that the controllers at different layers focus on different time scale optimization objectives and work together to form a complete control system.

[0223] The above description is merely the preferred embodiment of the present application, and is not used to limit the protection scope of the present application. Those skilled in the art can make various changes or modifications to the present application without departing from the spirit and scope of the present application, and these changes or modifications are within the protection scope of the present application.

Claims

1. A robot adaptive motion control algorithm based on reinforcement learning, characterized in that, The method comprises the following steps: acquiring torque signals of robot joints and constructing a state vector representing the interaction state of the robot and the environment based on the torque signals; generating an adaptive motion control strategy based on the state vector through a hierarchical reinforcement learning training mechanism, including: constructing a multi-dimensional reward function that integrates task completion, energy efficiency, motion safety, and smoothness; using a Q-learning algorithm to iteratively optimize the mapping relationship between the state vector and the action control sequence; verifying the adaptive motion control strategy in multiple physical parameter environments and selectively triggering retraining based on the verification results; deploying the verified adaptive motion control strategy to the robot control system to achieve adaptive motion control; the hierarchical reinforcement learning training mechanism includes: a bottom layer joint torque direct control layer with a control period of 5-10 milliseconds, which directly operates the joint torque; a middle layer motion trajectory planning layer with a control period of 50-100 milliseconds, which plans short-term trajectory segments; a high layer task strategy decision layer with a control period of 500-1000 milliseconds, which is responsible for overall task strategy adjustment; a closed-loop control structure formed by top-down command transmission and bottom-up state feedback between the bottom layer joint torque direct control layer, the middle layer motion trajectory planning layer, and the high layer task strategy decision layer; the construction of the multi-dimensional reward function that integrates task completion, energy efficiency, motion safety, and smoothness includes: defining a task completion reward based on target position achievement and completion time efficiency evaluation; defining an energy efficiency reward based on joint power consumption and total energy consumption, with the calculation formula being: , wherein, is the torque of the i-th joint in Newton-meter; is the angular velocity of the i-th joint in radian / second; denotes the instantaneous power absolute value of the i-th joint in Watt; is the control period in seconds; is the energy consumption penalty coefficient in 1 / Joule; is the number of joints of the robot; the negative sign means that the higher the energy consumption, the greater the penalty; defining a safety reward based on distance from obstacles and degree of dangerous posture, with the calculation formula being: , where, is the distance between the robot and the nearest obstacle, in meters; is the distance sensitivity coefficient, in 1 / m; is the distance decay function, decreasing with increasing distance; is the distance between the robot and the joint limit, expressed as an angle, in radians; is the joint movable range, in radians; and are the weights for obstacle distance and pose safety, respectively, dimensionless; defining a smoothness reward based on torque change rate and trajectory continuity, with the calculation formula being: , wherein, is the rate of change of the torque of the ith joint, in Nm / s; is the smoothness penalty coefficient, in s / Nm; n is the number of joints of the robot; the negative sign indicates that the greater the change in torque, the greater the penalty; linearly combining the energy efficiency reward, task completion reward, safety reward, and smoothness reward through dynamic weight coefficients to form a multi-dimensional reward function; the acquisition of torque signals of robot joints and the construction of a state vector representing the interaction state of the robot and the environment based on the torque signals include: real-time acquisition of torque data through torque sensors installed on each joint of the robot; low-pass filtering and normalization processing of the torque data; construction of a multi-dimensional state vector based on the current torque value, the previous torque value, and their change rates; real-time monitoring of the interaction between the robot and virtual or actual obstacles based on the state vector; the verification of the adaptive motion control strategy in multiple physical parameter environments and the selective triggering of retraining based on the verification results include: constructing a parameterized verification environment set with different friction coefficients, damping coefficients, and mass distributions; performing multiple independent tests in each verification environment to evaluate functional indicators, efficiency indicators, safety indicators, and robustness indicators; Calculate average performance metric across all validation environments and worst performance metric : , , wherein, Pi is the performance indicator under the i-th environment, dimensionless, the value range is 0-1, the larger the value, the better the performance; N is the total number of verification environments; triggering the retraining process when the worst performance indicator is below a threshold value.

2. The reinforcement learning based robot adaptive motion control algorithm according to claim 1, wherein, the iterative optimization of the mapping relationship between the state vector and the action control sequence using the Q-learning algorithm includes: discretizing the continuous state space using an adaptive grid method, with fine-grained grids used in critical areas; Actions are selected based on the epsilon-greedy strategy, with an increasing exploration rate at the beginning and an increasing exploitation rate at the end of the training period. The state-action value function is optimized through iterative updates controlled by learning rate and discount factor. The priority experience replay mechanism is used to improve learning efficiency.

3. The reinforcement learning based robot adaptive motion control algorithm according to claim 1, wherein, The validated adaptive motion control strategy is deployed to the robot control system, including: The control strategy is optimized for compression to reduce memory usage and improve query efficiency. A standardized control interface is established to achieve compatibility with different robot platforms. A runtime monitoring and fine-tuning mechanism is designed to support online performance optimization. Strategy version management is implemented to support rollback to a stable version in abnormal situations.

4. The reinforcement learning based robot adaptive motion control algorithm according to claim 2, wherein, The action control sequence includes joint torque increments.

5. The reinforcement learning based robot adaptive motion control algorithm according to claim 1, wherein, The weight coefficients of each dimension in the multi-dimensional reward function are dynamically adjusted according to the task stage, environmental characteristics, and historical performance.

Images