Robot adaptive motion control algorithm based on reinforcement learning

Abstract

The invention relates to the technical field of robot control, in particular to a robot self-adaptive motion control algorithm based on reinforcement learning, and the algorithm obtains a torque signal of a robot joint, constructs a state vector representing an interaction state of a robot and an environment based on the torque signal, and achieves the self-adaptive motion control of the robot through a layered reinforcement learning training mechanism. The method comprises the following steps: generating a self-adaptive motion control strategy which comprises the steps of constructing a multi-dimensional reward function and fusing task completion degree, energy efficiency, motion safety and smoothness, carrying out iterative optimization on a mapping relation between a state vector and a motion control sequence by utilizing a Q-learning algorithm, verifying the self-adaptive motion control strategy in various physical parameter environments, and carrying out self-adaptive motion control according to the self-adaptive motion control strategy. And the self-adaptive motion control strategy passing the verification is deployed to the robot control system, so that the self-adaptability is remarkably improved, the robot control strategy can adapt to physical parameter changes within the range of + / -30%, and the parameters can be readjusted without manual intervention.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a robot adaptive motion control algorithm based on reinforcement learning, which is suitable for scenarios where industrial robots, service robots, collaborative robots, etc. perform tasks in changing environments. Background Art

[0002] Traditional robot control algorithms primarily rely on preset control parameters and deterministic control laws, such as the widely used PID controller or impedance controller. These controllers perform well when the robot's operating environment is relatively fixed and its physical parameters are stable. However, when the robot is required to operate in an environment with varying physical characteristics, such as when parameters such as the friction coefficient, damping coefficient, and load mass change, traditional controllers often struggle to maintain stable control performance, requiring manual parameter readjustment, resulting in high debugging costs and poor adaptability.

[0003] Furthermore, existing control methods based on position and velocity feedback cannot directly perceive the interaction forces between the robot and its environment, which limits the robot's performance in tasks involving interaction with the environment. For example, in scenarios requiring precise force control, such as assembly, material handling, and surface treatment, traditional control methods struggle to adapt to changes in material properties and the environment.

[0004] In recent years, reinforcement learning, a key branch of artificial intelligence, has achieved remarkable results in areas such as gaming and autonomous driving. However, applying reinforcement learning to robotic motion control still faces many challenges, including how to construct effective state representations, how to design reward functions suitable for control scenarios, and how to ensure the safety and robustness of learned strategies in real-world environments.

[0005] At present, the existing technology lacks a robot motion control algorithm that can be based on torque perception and adapt to different physical environments. There is an urgent need to develop new control methods to meet this challenge. Summary of the Invention

[0006] The purpose of the present invention is to solve the problem in the prior art that robot control systems are difficult to adapt to environmental changes and lack the ability to perceive force interactions, and to provide a robot adaptive motion control algorithm based on reinforcement learning. The algorithm realizes adaptive motion control of robots in different physical environments through a torque perception mechanism, a hierarchical reinforcement learning architecture, a multi-dimensional reward system, a dynamic mapping mechanism and a multi-environment verification system.

[0007] The present invention proposes a robot adaptive motion control algorithm based on reinforcement learning, including:

[0008] Acquire torque signals of the robot joints, and construct a state vector representing the interaction state between the robot and the environment based on the torque signals;

[0009] Based on the state vector, an adaptive motion control strategy is generated through a hierarchical reinforcement learning training mechanism, including:

[0010] Construct a multi-dimensional reward function that integrates task completion, energy efficiency, motion safety, and smoothness;

[0011] Iteratively optimizing the mapping relationship between the state vector and the action control sequence using a Q-learning algorithm;

[0012] Validating the adaptive motion control strategy under various physical parameter environments and selectively triggering retraining based on validation results;

[0013] The verified adaptive motion control strategy is deployed to the robot control system to achieve adaptive motion control.

[0014] Preferably, the step of obtaining torque signals of robot joints and constructing a state vector representing the interaction state between the robot and the environment based on the torque signals includes:

[0015] The torque data is collected in real time by the torque sensors installed at each joint of the robot;

[0016] performing low-pass filtering and normalization processing on the torque data;

[0017] Construct a multidimensional state vector based on the current moment value, the previous moment value and its rate of change;

[0018] The interaction between the robot and the virtual or actual obstacle is monitored in real time based on the state vector.

[0019] Preferably, the hierarchical reinforcement learning training mechanism includes:

[0020] The bottom layer directly controls joint torque, responsible for high-frequency, small-grained torque adjustment;

[0021] The middle-level motion trajectory planning layer is responsible for medium-frequency and medium-granularity trajectory optimization;

[0022] The high-level mission strategy decision-making layer is responsible for low-frequency, large-granularity mission planning;

[0023] The three layers form a closed-loop control structure through top-down command transmission and bottom-up state feedback.

[0024] Preferably, the multi-dimensional reward function integrating task completion, energy efficiency, motion safety and smoothness is constructed, including:

[0025] Define task completion rewards, based on target location achievement and completion time efficiency evaluation;

[0026] Define energy efficiency bonus, calculated based on joint power consumption and total energy consumption;

[0027] Define safety rewards based on distance to obstacles and degree of dangerous posture;

[0028] Define smoothness bonus, calculated based on torque change rate and trajectory continuity;

[0029] The rewards of the above four dimensions are linearly combined through dynamic weight coefficients to form a comprehensive reward function.

[0030] Preferably, the iterative optimization of the mapping relationship between the state vector and the action control sequence using the Q-learning algorithm includes:

[0031] Adaptive grid method is used to discretize the continuous state space, and fine-grained grids are used in key areas;

[0032] Select actions based on the ε-greedy strategy, which improves the exploration rate in the early stage of training and the utilization rate in the later stage;

[0033] Optimize the state-action value function through iterative updates controlled by learning rate and discount factor;

[0034] The priority experience replay mechanism is used to improve learning efficiency, and samples with high TD error have a higher sampling probability.

[0035] Preferably, the verifying the adaptive motion control strategy under multiple physical parameter environments and selectively triggering retraining based on the verification results includes:

[0036] Construct a set of parameterized verification environments with different friction coefficients, damping coefficients, and mass distributions;

[0037] Perform multiple independent tests in each verification environment to evaluate functional indicators, efficiency indicators, safety indicators and robustness indicators;

[0038] Calculate the average performance index and the worst performance index of all verification environments;

[0039] When the worst performance indicator falls below a threshold or the performance variance exceeds a preset range, the retraining process is triggered.

[0040] Preferably, the step of deploying the verified adaptive motion control strategy to the robot control system includes:

[0041] Compression optimization of control strategies to reduce memory usage and improve query efficiency;

[0042] Establish standardized control interfaces to achieve compatibility with different robot platforms;

[0043] Design runtime monitoring and fine-tuning mechanisms to support online performance optimization;

[0044] Implement policy version management and support rollback to stable version in abnormal situations.

[0045] Preferably, the motion control sequence includes joint torque increments, and each joint can select seven discrete motion levels: large increase, medium increase, small increase, maintain, small decrease, medium decrease or large decrease.

[0046] Preferably, the weight coefficients of each dimension in the multi-dimensional reward function are dynamically adjusted according to the task stage, environmental characteristics and historical performance to ensure adaptive adjustment of the strategy optimization direction in different application scenarios.

[0047] Preferably, each layer in the hierarchical reinforcement learning training mechanism adopts different time scales and decision granularities:

[0048] The underlying control cycle is 5-10 milliseconds, directly manipulating joint torque;

[0049] The middle-level control cycle is 50-100 milliseconds, planning short-term trajectory segments;

[0050] The high-level control cycle is 500-1000 milliseconds and is responsible for overall task strategy adjustment;

[0051] Asynchronous collaborative control is achieved between layers through a buffer mechanism.

[0052] The beneficial effects of the present invention are mainly reflected in:

[0053] 1. Significantly improved adaptability: Through state representation based on torque signals and reinforcement learning training, the robot control strategy can adapt to physical parameter changes within the range of ±30% without manual intervention to readjust the parameters.

[0054] 2. Significantly improved energy efficiency: The multidimensional reward function includes energy efficiency evaluation indicators, which enables the control strategy to automatically optimize energy consumption during the learning process, reducing energy consumption by 20% to 35% compared to traditional control methods.

[0055] 3. Essential improvement in safety performance: Through a safety reward mechanism, the control strategy can actively avoid obstacles, increasing the minimum safe distance from obstacles by 25%, significantly improving operational safety.

[0056] 4. Comprehensive optimization of control quality: Positioning accuracy in changing environments is improved by 40%, motion trajectory smoothness is improved by 50%, and the life of the mechanical system is significantly extended.

[0057] 5. Deployment efficiency is significantly improved: the time from training to deployment is shortened by 50%, the system debugging cycle is reduced by 60%, and the application cost is greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Schematic diagram of the system architecture of the robot adaptive motion control algorithm based on reinforcement learning of the present invention;

[0059] Figure 2 This is a schematic diagram of the structure of the torque sensing and state building module of the present invention;

[0060] Figure 3 Schematic diagram of the hierarchical structure of the hierarchical reinforcement learning training mechanism of the present invention;

[0061] Figure 4 A schematic diagram of the process of constructing the multi-dimensional reward function of the present invention;

[0062] Figure 5 This is a schematic diagram of the Q-learning algorithm training process of the present invention;

[0063] Figure 6 This is a schematic diagram of the multi-environment verification and strategy deployment process of the present invention; DETAILED DESCRIPTION

[0064] Please refer to the attached Figure 1-6 The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. These embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0065] like Figure 1 As shown, the system architecture of the robot adaptive motion control algorithm based on reinforcement learning provided by the present invention mainly includes a torque perception module 101, a hierarchical reinforcement learning training module 102, an adaptive control strategy module 103, a multi-environment verification module 104 and a strategy deployment module 105.

[0066] The present invention provides a robot adaptive motion control algorithm based on reinforcement learning, which includes obtaining torque signals of robot joints, constructing state vectors based on the torque signals, generating adaptive control strategies through a hierarchical reinforcement learning training mechanism, verifying the strategies in multiple physical environments, and finally deploying the verified strategies to the robot control system.

[0067] Figure 2 The detailed implementation of the torque sensing and state building module is shown. Taking a six-axis industrial robot as an example, this module collects torque data in real time using torque sensors 201 installed at each joint of the robot, with a sampling frequency set to 1000Hz. This high sampling frequency is crucial for capturing the rapid torque changes generated by the robot during assembly tasks or when collaborating with humans.

[0068] Preferably, the collected raw torque signal is low-pass filtered by the signal processing unit 202, with the filter cutoff frequency set to 50 Hz. In practical applications, such as when a robot performs precision assembly tasks, environmental vibrations and motor noise often introduce noise signals above 50 Hz. This cutoff frequency effectively eliminates these high-frequency noises while retaining useful torque change information generated during the contact between the robot and the workpiece.

[0069] The filtered signal also needs to be normalized to map the torque values ​​of different joints to the range [-1, 1]. For example, for an industrial robot with a maximum joint torque of 200 Nm, the raw torque value (e.g., 87 Nm) collected would be normalized to 0.435 to facilitate subsequent reinforcement learning algorithm processing.

[0070] When mapping the torque values ​​of different joints to the [-1,1] interval, the standard normalization formula is used:

[0071] Normalized value ,

[0072] For the joint torque of the industrial robot, assuming the maximum rated torque range is 200Nm, then:

[0073] The normalized forward torque of 87 Nm is: ,

[0074] The negative torque -87Nm is normalized to: -0.435,

[0075] In one embodiment of the present invention, the state construction unit 203 constructs a multi-dimensional state vector based on the processed torque signal. The mathematical representation of the state vector is:

[0076] ,

[0077] in, represents the current torque value of the i-th joint, represents the torque value of the i-th joint at the previous moment, represents the rate of change of torque, , is the control cycle, typically 5-10 milliseconds. For a robot with n joints, for a six-axis industrial robot, n=6.

[0078] In a practical example, the torque data of the six joints are [5.2, -3.1, 8.7, 0.5, -1.2, 2.3] Nm. Assuming that the maximum rated torque of each joint is 20 Nm, the normalized torque should be:

[0079] ,

[0080] ,

[0081] ,

[0082] ,

[0083] ,

[0084] ,

[0085] Therefore, the first part of the normalized state vector should be [0.26, -0.155, 0.435, 0.025, -0.06, 0.115]. Similarly, the torque value and torque rate of change at the previous moment can be normalized. This normalization ensures that torque data of different magnitudes have equal weight in the learning algorithm, preventing learning bias caused by different dimensions.

[0086] In addition, the state monitoring unit 204 is responsible for real-time monitoring of the robot's interactions with virtual or real obstacles. When the torque value of any joint exceeds a preset threshold (e.g., 80% of the rated torque), the system identifies a collision with the obstacle and generates a termination signal. In practical applications, such as when the robot is operating in an unknown environment, this torque-based collision detection method can detect potential collision risks in advance and avoid dangerous contact between the robot and the environment or humans. For example, a collaborative robot with a maximum joint torque of 100 Nm will immediately slow down or stop when it detects that the torque of any joint exceeds 80 Nm to ensure safe operation.

[0087] like Figure 3 As shown, the present invention adopts a hierarchical reinforcement learning training mechanism, including a bottom joint torque direct control layer 301, a middle motion trajectory planning layer 302 and a high-level task strategy decision layer 303.

[0088] The bottom-level joint torque direct control layer 301 is responsible for high-frequency, fine-grained torque adjustments, directly operating the robot's joint actuators. In industrial scenarios, such as when a robot is performing precision gear assembly, the bottom-level controller, based on joint torque signals, can sense minute changes in resistance during gear meshing in real time and adjust the applied torque to ensure smooth insertion without damaging parts.

[0089] The mid-level motion trajectory planning layer 302 is responsible for medium-frequency, medium-granularity trajectory optimization, converting high-level task instructions into executable trajectory segments. For example, when a robot needs to move heavy objects in a factory environment, the mid-level planner considers path smoothness and energy efficiency to generate optimal trajectory segments, avoiding sudden acceleration or deceleration that could cause object shaking or energy waste.

[0090] The high-level task strategy decision layer 303 is responsible for low-frequency, large-scale task planning and generates strategy instructions based on the overall task objectives. For example, on a complex assembly line, the high-level decision maker will plan the optimal operation sequence and station switching strategy for the robot based on current production requirements and workpiece status, thereby improving overall production efficiency.

[0091] Preferably, a closed-loop control structure is formed between the three layers through top-down command transmission and bottom-up state feedback to achieve multi-scale collaborative control. Specifically, the high-level decision results (such as grasping workpiece A and moving it to position B") are passed to the middle layer through the instruction buffer. The middle layer converts them into specific trajectory segments (such as moving from the current position to the position of workpiece A, and moving to position B" along the planned path after grasping). These are then passed to the bottom layer through the control buffer to execute specific joint torque control. At the same time, the bottom-level execution results and status information (such as the current torque is too large and may hit an obstacle) are passed upward through the feedback buffer to help the upper layer adjust its decision.

[0092] In a specific automotive assembly line application, this layered architecture enables robots to efficiently complete transmission assembly tasks. A high-level decision maker determines the assembly sequence and station switching strategy, a mid-level planner generates smooth and efficient motion trajectories, and a low-level controller precisely controls insertion depth and alignment force based on torque feedback during the assembly process. Ultimately, this increases the assembly success rate from 92% with traditional control methods to 99.5%.

[0093] like Figure 4 As shown, the present invention constructs a multi-dimensional reward function that integrates task completion, energy efficiency, motion safety and smoothness.

[0094] In the embodiment of industrial robots performing precision assembly tasks, task completion rewards According to the target position achievement and completion time efficiency evaluation, the specific calculation formula is:

[0095] ,

[0096] in, is the current position of the robot, expressed as three-dimensional space coordinates [x, y, z], in meters (m); is the target position, also expressed as a three-dimensional space coordinate in meters (m); Indicates the Euclidean distance between the current position and the target position, in meters (m); The normalized distance reference value is in meters (m), which is usually set to half the diameter of the working space. For example, for a robot with a working radius of 1.5m, Can be set to 1.5m; The execution time of the current task, in seconds (s); The maximum allowed time is in seconds (s), which is set according to the complexity of the task. For example, a precision insertion task can be set to 30s. and are the weight coefficients of position accuracy and time efficiency, dimensionless, with the preferred values ​​being 0.7 and 0.3 respectively.

[0097] For example, in the precision bearing assembly task, the current robot position is [0.82, 0.45, 0.33] m, the target position is [0.80, 0.45, 0.30] m, the current task has been executed for 15 seconds, and the maximum allowed time is 30 seconds. If the distance is 1.5m, the position error is about 0.032m, and the task completion reward is calculated as: This calculation method takes into account time efficiency while ensuring position accuracy.

[0098] Energy efficiency incentives Based on joint power consumption and total energy consumption, the calculation formula is:

[0099] ,

[0100] in, is the torque of the i-th joint, in Newton meters (Nm); is the angular velocity of the i-th joint, in radians per second (rad / s); represents the absolute value of the instantaneous power of the i-th joint, in watts (W); is the control period, in seconds (s); is the energy consumption penalty coefficient, the unit is 1 / joule (1 / J), and the preferred value is 0.05; is the number of robot joints; the negative sign indicates that the higher the energy consumption, the greater the penalty (the smaller the reward).

[0101] Taking the industrial robot handling task as an example, assuming that the joint torque of the six-axis robot at a certain moment is [15, 25, 30, 5, 8, 3] Nm, the corresponding angular velocity is [0.2, 0.1, 0.15, 0.3, 0.2, 0.4] rad / s, and the control cycle is 0.01 second, the energy consumption is calculated as: (15×0.2+25×0.1+30×0.15+5×0.3+8×0.2+3×0.4)×0.01=0.131 joules, and the corresponding energy efficiency reward is .

[0102] Security Rewards According to the distance to the obstacle and the degree of dangerous posture, the calculation formula is:

[0103] ,

[0104] in, is the distance between the robot and the nearest obstacle, in meters (m); The distance sensitivity coefficient is expressed in 1 / meter (1 / m). The preferred value is 5.0, which makes the reward change most sensitive when the obstacle distance is within the range of 0.2-1m. is the distance decay function, which decreases as the distance increases; is the distance between the robot and the joint limit, expressed as an angle in radians (rad); is the range of motion of the joint, in radians (rad); and are the weights of obstacle distance and posture safety, dimensionless, and the preferred value is 0.5.

[0105] In a human-robot collaboration scenario, for example, when a robot and a human are working together to complete an assembly task, assuming the robot is 0.5 meters away from the nearest human body part and 0.2 arcs away from the joint limit (the joint range of motion is 2 arcs), the safety bonus is calculated as:

[0106] ,

[0107] This reward mechanism encourages the robot to maintain a safe distance while avoiding approaching joint limits.

[0108] Smoothness Bonus Based on the torque change rate and trajectory continuity calculation, the calculation formula is:

[0109] ,

[0110] in, is the torque change rate of the i-th joint, in Newton-meter-per-second (Nm / s), and is calculated by dividing the torque difference between two adjacent control cycles by the control cycle; is the smoothness penalty coefficient in seconds per Newton meter (s / Nm), with a preferred value of 0.03; n is the number of robot joints; the negative sign indicates that the greater the torque change, the greater the penalty (the smaller the reward).

[0111] In the precision gluing task, the robot needs to apply glue evenly along the edge of the workpiece, and motion smoothness is crucial. Assuming that the torque change rate of the six joints at a certain moment is [20, -15, 10, -5, 8, -12] Nm / s, the smoothness bonus is calculated as: This reward mechanism encourages the control system to generate smoother torque control output, reducing mechanical vibration and process fluctuations.

[0112] Finally, the rewards of the above four dimensions are linearly combined through dynamic weight coefficients to form a comprehensive reward function:

[0113] ,

[0114] in, is the weight coefficient of each dimension, dimensionless, with initial values ​​set to 0.5, 0.2, 0.2, and 0.1 respectively, which can be dynamically adjusted as the training progresses and the nature of the task changes. The sum of the weight coefficients is equal to 1, that is, .

[0115] In actual implementation, the rewards should be normalized so that they are within a similar value range:

[0116] ,

[0117] in, is the original reward value of a certain dimension, and These are the minimum and maximum experience points awarded for this dimension, is the normalized reward value.

[0118] Another approach is to introduce an adaptive scaling factor

[0119] ,

[0120] in, and is the adaptive scaling factor, which is determined as follows:

[0121] ,

[0122] For example, if the statistical median magnitudes of the four dimension rewards are 0.8, 0.007, 0.9, and 2.0, respectively, the scale factor of the energy efficiency reward should be , the scaling factor of the smoothness reward should be .

[0123] After applying these scaling factors, the energy efficiency bonus is adjusted to , the smoothness reward is adjusted to ,They are in the same order of magnitude as the task completion reward (0.835) and the safety reward (0.909), ,ensuring that rewards in each dimension have appropriate influence in the optimization process.

[0124] like Figure 5 As shown, the present invention utilizes the Q-learning algorithm to iteratively optimize the mapping relationship between the state vector and the action control sequence.

[0125] In practical applications, such as when robots perform assembly tasks, adaptive meshing methods are used to discretize the continuous state space. Due to the high dimensionality of the torque state space (for a six-axis robot, the state vector dimension is 18), uniform meshing would cause the state space to explode. Therefore, the present invention dynamically adjusts the mesh size based on the sample distribution density in the state space, using a fine-grained mesh in critical areas (such as near the target location or obstacles) and a coarse-grained mesh in non-critical areas.

[0126] In specific implementation, for the assembly task of the six-axis robot, the initial uniform grid can be divided into 10 18 Using an adaptive meshing method, the system focuses on joints with significant torque changes and subdivides the state space of these joints, ultimately reducing the effective state space to approximately 100,000 discrete states, significantly improving learning efficiency.

[0127] Based on the ε-greedy strategy, actions are selected to increase the exploration rate in the early stage of training and the utilization rate in the later stage. The change of the exploration rate ε follows the following formula:

[0128] ,

[0129] in, is the minimum exploration rate, dimensionless, set to 0.05, to ensure that the system always maintains a certain exploration ability; is the maximum exploration rate, dimensionless, set to 0.8, so that the system can explore the state space extensively in the early stage of training; is the decay coefficient, the unit is 1 / step, set to 0.0001, which controls the decay speed of the exploration rate; is the number of training steps, the unit is step; is an exponential decay function.

[0130] In the actual assembly task training, for example, at the 1000th step, the exploration rate is calculated as: , at this point the system has a 72.9% probability of randomly selecting an action for exploration and a 27.1% probability of selecting the action with the largest current Q value. As training progresses, for example, by the 10,000th step, the exploration rate drops to 0.207, and the system makes more use of learned experience.

[0131] The Q-value update uses the standard Q-learning iterative formula:

[0132] ,

[0133] in, is the Q value of executing action a in state s, dimensionless; is the learning rate, dimensionless, with an initial value of 0.1 and gradually reduced to 0.01 as training progresses; is the reward obtained after performing action a, dimensionless; is the discount factor, dimensionless, set to 0.9, balancing current rewards and future rewards; is the new state after executing action a; New state The maximum Q value under , dimensionless; is the temporal difference (TD) error, which represents the difference between the actual reward and the expected reward.

[0134] Taking the robot precision insertion task as an example, assuming that the torque signal corresponding to the current state s indicates that the robot has just touched the edge of the socket, it performs action a (fine-tuning the posture and slightly applying downward force) and obtains the reward R = 0.5, the new state s' shows better alignment with the jack, and the corresponding maximum Q value is 0.8. The value is 0.3, and the learning rate α is 0.1, then the Q value is updated as: .

[0135] Preferably, the present invention utilizes a priority experience replay mechanism to improve learning efficiency. The system maintains an experience pool of 30,000 samples, which is particularly suitable for learning complex robot assembly tasks. The priority of each sample is determined by the absolute value of the TD error:

[0136] ,

[0137] in, is the TD error of sample i, dimensionless, and the calculation formula is ; It is a small constant, dimensionless, and has a priority of zero when the error is zero, so it is set to 0.01; is the priority index, dimensionless, controls the degree of priority difference, and is set to 0.6; is the priority of sample i, dimensionless.

[0138] The sampling probability is proportional to the priority:

[0139] ,

[0140] in, is the probability that sample i is selected, dimensionless; It is the sum of all sample priorities and is dimensionless.

[0141] In practical applications, such as when a robot learns a screw tightening task, the experience sample generated when the screw just touches the screw hole but is not aligned usually has a large TD error, because this is the key turning point for successful tightening. Assuming that the absolute value of the TD error of this sample is 0.8, which is much higher than the average value of 0.2, its priority is calculated as , and the priority of the average sample is Assuming there are 10,000 samples in the experience pool and a total priority of 5,000, the probability of this high-value sample being selected is 0.865 / 5,000 = 0.000173, which is approximately 1.73 times the average probability (0.501 / 5,000 = 0.0001). This mechanism increases the probability of sampling samples with high TD error (i.e., more learnable samples), thereby accelerating the learning process.

[0142] like Figure 6 As shown, the present invention verifies the adaptive motion control strategy in multiple physical parameter environments and selectively triggers retraining based on the verification results.

[0143] In real-world industrial applications, robots need to adapt to a variety of working conditions, such as assembling parts made of different materials, carrying objects of varying weights, or polishing different surfaces. To simulate these variations, the present invention constructs a set of verification environments with varying physical parameters.

[0144] Taking the automobile parts assembly line as an example, the environmental parameter variation range is set as follows:

[0145] Friction coefficient μ: 0.1-0.5, in steps of 0.1 (corresponding to smooth metal surface to rough plastic surface)

[0146] Damping coefficient ζ: 0.05-0.2, in steps of 0.05 (corresponding to slightly damped to strongly damped systems)

[0147] Mass change Δm: -20% to +20%, in 10% steps (corresponding to weight changes of parts of different models)

[0148] Based on these parameter combinations, a total of 5 × 4 × 5 = 100 different verification environments can be constructed, covering most of the working conditions a robot might encounter. For example, a verification environment corresponds to a friction coefficient of 0.3, a damping coefficient of 0.15, and a mass change of +10%, simulating an assembly scenario with medium friction, high damping, and slightly heavy parts.

[0149] Multiple independent tests (typically 10) are performed within each verification environment, evaluating the following metrics:

[0150] Functional indicators: including task completion rate (the proportion of successfully completed tests to the total number of tests), average completion time (the average time required to successfully complete the task), and control accuracy (the root mean square value of the end point position error)

[0151] Efficiency indicators: including energy consumption (total energy consumption during task execution), computing resource usage (CPU and memory usage), response delay (the time delay from receiving instructions to executing actions)

[0152] Safety indicators: including the number of collisions (the number of collisions with the environment or obstacles), minimum safe distance (the minimum distance to obstacles), and peak torque (the maximum joint torque during execution)

[0153] Robustness indicators: including parameter change sensitivity (the degree of performance fluctuation with parameter changes), fault recovery capability (the speed of recovery after encountering interference), and long-term stability (the ability to maintain performance after long-term operation)

[0154] Calculate the average performance index of all verification environments and worst performance indicators :

[0155] ,

[0156] ,

[0157] in, is the performance index in the i-th environment, dimensionless, ranging from 0 to 1, and the larger the value, the better the performance; The total number of verification environments, in this case 100; Represents the arithmetic mean of performance indicators in all environments; Indicates the worst performance indicator value among all environments.

[0158] Performance indicators It is composed of weighted components of each sub-indicator:

[0159] ,

[0160] in, It is a functional index, dimensionless, and its value range is 0-1; is an efficiency index, dimensionless, ranging from 0 to 1; It is a safety index, dimensionless, and its value range is 0-1; It is a robustness index, dimensionless, and ranges from 0 to 1; , , , is the weight coefficient of each indicator, dimensionless, with typical values ​​of 0.4, 0.2, 0.3, and 0.1, and the sum of the weights is equal to 1, that is, .

[0161] In the case of an automotive parts assembly line, the indicators under a specific environment (friction coefficient 0.4, damping coefficient 0.1, mass change -10%) may be: (High completion rate and accuracy), (good energy efficiency), (High security), (good robustness), the comprehensive performance indicators in this environment are: .

[0162] When the worst performance indicator When the performance index falls below a threshold (usually set to 0.7) or the performance variance exceeds a preset range (usually 0.2), the retraining process is triggered. For example, if the performance index is above 0.8 in 99 out of 100 verification environments, but the performance index is only 0.65 in a specific environment (such as high friction, low damping, and heavy load), the system will trigger retraining and specifically increase the training proportion under these environmental conditions to improve the generalization ability of the control strategy.

[0163] The present invention deploys the verified adaptive motion control strategy into the robot control system, including the following steps:

[0164] Compression and optimization of control strategies reduce memory usage and improve query efficiency. This step is particularly important when industrial robot controllers have limited resources. For a six-axis industrial robot, for example, the original Q-table may contain approximately 100,000 state-action pairs, occupying 50MB of memory. Through policy distillation, the policy represented by the Q-table can be converted into a more compact neural network representation (e.g., a 3-layer neural network with 50 nodes per layer), reducing memory usage to approximately 5MB and increasing query speed by 2-3 times. Another approach is to use pruning techniques to remove low-value (below a threshold of 0.1) or rarely accessed (access frequency less than 0.1% of the total frequency) state-action pairs from the Q-table. This typically reduces table entries by 40% to 60% with no more than a 5% performance loss.

[0165] Establish a standardized control interface to achieve compatibility with different robot platforms. In actual industrial applications, factories may use robots from multiple brands simultaneously. The present invention defines a unified API, including a state input interface (receiving sensor data such as torque and position), an action output interface (sending torque control commands), and a parameter configuration interface (adjusting control parameters). For example, in a certain automobile factory, robots from two brands, ABB and KUKA, are used on the same assembly line. Through the standard interface layer of the present invention, the same set of adaptive control strategies can be seamlessly applied to both robots, significantly reducing system integration and maintenance costs.

[0166] A runtime monitoring and fine-tuning mechanism is designed to support online performance optimization. The system calculates key performance indicators every hour, such as average positioning error (target value <0.1mm), energy consumption (target value <80% of the preset benchmark), and motion smoothness (jitter amplitude <0.5mm). When indicators deviate from expectations, the fine-tuning process is triggered. For example, in a precision parts assembly task, the system detected that the average positioning error increased from 0.08mm to 0.12mm. The fine-tuning process was immediately initiated, and the parameters related to precise positioning in the Q table were adjusted through incremental learning. Without affecting other performance, the positioning error was reduced to 0.07mm.

[0167] Implementing policy version management, the system supports rollback to stable versions in the event of anomalies. The system maintains the five most recent policy versions, each containing complete policy parameters and performance evaluation results. On an electronic component assembly line, after deploying a new policy version, an 8% assembly failure rate was observed for a specific part type (compared to only 2% for the previous version). The system immediately rolled back to the previous stable version, ensuring continued reliable operation of the line and recording the problem data for subsequent policy improvements.

[0168] The motion control sequence in the present invention includes joint torque increments, and seven discrete motion levels can be selected for each joint.

[0169] In the scenario where an industrial robot performs precision assembly tasks, the seven discrete action levels are specifically defined as follows:

[0170] Significant increase: The current torque value increases by 30%, for example, from 20Nm to 26Nm;

[0171] Moderate increase: the current torque value increases by 15%, for example, from 20Nm to 23Nm;

[0172] Slight increase: The current torque value increases by 5%, for example, from 20Nm to 21Nm;

[0173] Hold: Keep the current torque value unchanged at 20Nm;

[0174] Slight decrease: The current torque value decreases by 5%, for example, from 20Nm to 19Nm;

[0175] Moderate reduction: The current torque value is reduced by 15%, for example, from 20Nm to 17Nm;

[0176] Significant reduction: The current torque value is reduced by 30%, for example, from 20Nm to 14Nm;

[0177] This design is very practical in real-world applications. For example, when a robot is performing a bearing press-fitting task, when it detects that the bearing begins to enter the shaft hole, the system may choose to increase the pressure slightly, gently increasing the pressure. If it detects a sudden increase in resistance, it may choose to reduce the pressure moderately or significantly to avoid damaging the part. When the bearing position approaches the target, it may choose to maintain the pressure, maintaining a steady state until the press-fit is complete.

[0178] For a six-axis industrial robot, the theoretical action space size is 7 6 =117649, which is computationally intractable. In order to reduce the dimensionality of the action space, dimensionality reduction techniques such as principal component analysis (PCA) can be used in assembly tasks. For example, in the vertical insertion task, only the Z-direction torque of the end effector and the posture fine-tuning in the XY plane need to be controlled, which can reduce the action space to 7 3 =343 effective actions, greatly improving learning efficiency.

[0179] Furthermore, to prevent control signal jitter, the present invention incorporates a motion smoothing mechanism. When the algorithm's selected motion significantly changes from the previous one, for example, from a large increase (+30%) to a large decrease (-30%), the system automatically inserts an intermediate transition, such as a small increase → hold → a small decrease → a moderate decrease → a large decrease, to smooth out the torque transition. Applying this mechanism on a motor assembly line reduced mechanical vibration by 65% ​​and lowered the component damage rate from 3% to below 0.5%.

[0180] The weight coefficients of each dimension in the multi-dimensional reward function of the present invention are dynamically adjusted according to the task stage, environmental characteristics and historical performance.

[0181] In terms of task stage adaptation, taking the robot performing the latch assembly task as an example, the system divides the task into three stages:

[0182] 1. Initial stage (approaching the target position): This stage focuses more on safety and smoothness of movement, and the weight is set to (task completion), (Energy Efficiency), (security), (Smoothness). This ensures that the robot can approach the work area safely and smoothly.

[0183] 2. Execution phase (insertion process): This phase focuses more on task completion and energy efficiency, and the weight is adjusted to , This enables the robot to perform insertions efficiently and accurately.

[0184] 3. Completion stage (fixing pins): This stage focuses on precise positioning, and the weight is adjusted to . Especially improve The Position Accuracy sub-weight in , ensures that the pin is fully seated.

[0185] In terms of environmental characteristics adaptation, the system adjusts the weight according to the characteristics of the current environment. For example, on a production line, when a robot needs to operate near a human work area, the sensor detects that a worker is approaching (less than 1.5 meters away), and the system automatically adjusts the safety weight to The standard value of 0.2 should be raised to 0.5, while other weights should be appropriately reduced to ensure safe operation. Conversely, in completely isolated work cells, the weight of safety can be appropriately reduced, while the weight of energy efficiency or task completion can be increased to achieve more efficient production.

[0186] In terms of historical performance adaptation, the system continuously evaluates the performance indicators of each dimension and increases the reward weight for dimensions with poor performance. For example, on an assembly line, the system finds through energy consumption monitoring that the average energy consumption of the robot exceeds the expected value by 20% for 5 consecutive hours, and then automatically increases the energy efficiency reward weight. Increasing the weight from 0.2 to 0.3 will force the system to learn a more energy-efficient control strategy. After a week, when the energy consumption index returns to the normal range, the weight can be gradually restored to the standard configuration.

[0187] Preferably, the weight adjustment adopts a gradual change to avoid control instability caused by sudden changes. The mathematical expression of weight adjustment is:

[0188] ,

[0189] in, is the weight value at time t, dimensionless; is the weight value at the next moment, dimensionless; is the weight adjustment quantity, dimensionless, satisfying , that is, the adjustment range each time does not exceed 20% of the current value. At the same time, the sum of all weights remains 1, that is, after adjusting one weight, the other weights are adjusted proportionally to ensure .

[0190] For example, a collaborative robot performs tasks on an assembly line, and the initial weight is When the safety distance is detected to be close to the warning line (0.5 meters) for many times, it is decided to increase the safety weight. The maximum adjustable amount of the safety weight is , the system chooses to adjust 0.03, that is To keep the sum of the weights equal to 1, the other weights are reduced proportionally: 0.48125, , The adjusted weight is , and the sum is still 1.

[0191] Each layer in the hierarchical reinforcement learning training mechanism of the present invention adopts a different time scale and decision granularity, which is mainly reflected in three aspects: control cycle, operation granularity and information transmission mechanism.

[0192] In actual industrial applications, the control parameters of each layer are different according to different task requirements. Take the automobile gearbox assembly task as an example:

[0193] The underlying control cycle is set to 5-10 milliseconds, directly manipulating the joint torque. For example, when the gearbox gear is docked with the shaft, the underlying controller collects torque sensor data at a cycle of 10ms, detects changes in resistance during the docking process in real time, and outputs torque control commands at the same frequency to adjust the applied force and direction. This control cycle setting is based on the response characteristics of the robot servo system (usually 100-200Hz) and can meet the real-time requirements of precise force control. The operating accuracy of the underlying controller is usually 1% to 2% of the rated torque. For example, for a joint with a maximum torque of 100Nm, the control accuracy can reach 1-2Nm.

[0194] The middle-level control cycle is set to 50-100 milliseconds, planning short-term trajectory segments. During the gearbox assembly process, the middle-level planner processes assembly instructions from the upper level with a 50ms cycle, breaking down the gearbox installation task into a series of trajectory segments: approach, align, insert, and secure. Each trajectory segment contains 5-10 low-level control points. The middle-level's primary responsibility is to ensure smooth transitions between stages, continuous motion, and the avoidance of shock and vibration. The middle-level control time span is typically 0.5 to 2 seconds, corresponding to a single action phase in the gearbox assembly.

[0195] High-level control cycles are set at 500-1000 milliseconds and are responsible for overall task strategy adjustments. In a production line environment, the high-level decision maker updates task status every second, adjusting assembly strategies and priorities based on assembly progress, parts availability, and quality inspection feedback. For example, if a batch of transmission gears exhibits significant backlash, the high-level decision maker will adjust assembly parameters, increasing alignment time and accuracy requirements. High-level control typically spans 5 to 30 seconds, covering the entire transmission assembly cycle.

[0196] To ensure that the time scale between the high-level control cycle (0.5-1 second) and the task verification cycle (5-30 seconds) is matched, the following approach is used:

[0197] 1. A hierarchical reward distribution mechanism: high-level tasks are broken down into multiple phased sub-goals, each with an immediate reward signal. For example, in the gearbox assembly task, the complete task (30 seconds) is broken down into stages such as approach, alignment, insertion, and fixation, each with independent evaluation metrics:

[0198] Approach phase: the rate at which the distance to the target location decreases;

[0199] Alignment stage: the degree of reduction of attitude error;

[0200] Insertion phase: appropriateness of contact force and progress of insertion;

[0201] Fixed stage: final position accuracy and stability;

[0202] This decomposition enables the high-level controller to obtain meaningful reward signals every 0.5-1 second, rather than having to wait 30 seconds for the final result.

[0203] 2. Temporal Difference Learning and n-step Backtracking. The algorithm uses the Temporal Difference (TD) learning method, specifically using the n-step backtracking mechanism:

[0204] ,

[0205] in, is the immediate reward for n consecutive time steps, is the discount factor, is the learning rate, is the maximum Q-value of the state after n steps. For this system, a typical value of n is 10-20, covering a time span of 5 to 10 seconds, which can capture reward signals on a medium time scale.

[0206] 3. To handle long-term dependencies, the system introduces the Eligibility Traces mechanism:

[0207] ,

[0208] ,

[0209] in, is a state-action pair Qualifications, is the eligibility trace attenuation parameter (typical value 0.9), is the TD error at the current time step, Is the indicator function, when the current state-action pair is If they are the same, it is 1; otherwise, it is 0.

[0210] The eligibility trace mechanism enables past state-action pairs to receive appropriate reward updates based on their contribution to the current state, effectively solving the time scale mismatch problem.

[0211] 4. Real-time prediction of task completion indicators. The system trains an auxiliary value network to predict the probability of final task success based on the current status:

[0212] ,

[0213] This predicted value acts as an additional reward signal, providing real-time feedback about the long-term goal:

[0214] ,

[0215] in, is the predicted reward weight coefficient, It is the change in the probability of task success. A positive value indicates progress towards success, while a negative value indicates deviation from the target.

[0216] By combining these four mechanisms, the present invention effectively addresses the coordination problem between control cycles of varying timescales and reward signals, enabling high-level controllers to obtain meaningful learning signals within short timeframes while maintaining optimization direction toward long-term task objectives. These mechanisms demonstrate excellent learning efficiency and control performance in complex tasks such as gearbox assembly.

[0217] A buffer mechanism enables asynchronous collaborative control between layers. For example, in gearbox assembly, task instructions (e.g., installing a type A gearbox) transmitted from the upper layer to the middle layer are handled via a task buffer with a capacity of 10. Trajectory segments (e.g., moving 15mm downward at 2mm / s) transmitted from the middle layer to the lower layer are handled via a trajectory buffer with a capacity of 20. Status information (e.g., detecting a resistance of 12N) is fed back from the lower layer to the upper layer via a status buffer with a capacity of 50. Each buffer utilizes a first-in-first-out (FIFO) queue structure to ensure the timing and integrity of information transmission.

[0218] Preferably, the decision granularity of each layer is as follows:

[0219] Bottom layer: torque increment control. For a joint with a maximum torque of 100 Nm, the typical control increment is ±30 Nm, divided into 7 levels.

[0220] Middle layer: trajectory segment planning, with a typical length of 0.5 to 2 seconds, containing 10 to 40 low-level control points;

[0221] High-level: mission strategy, typically spanning 5 to 30 seconds, consisting of 3 to 10 mid-level trajectory segments;

[0222] In actual application at an automobile assembly plant, this multi-timescale hierarchical control architecture has improved transmission assembly efficiency by 35%, assembly accuracy by 40%, and reduced energy consumption by 25%. The key lies in the fact that each layer of controller focuses on optimization objectives at different timescales, working together to form a complete control system.

[0223] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Those skilled in the art may make various changes or modifications to the present invention without departing from the spirit and scope of the present invention, and these changes or modifications are all within the scope of protection of the present invention.

Claims

1. A robot adaptive motion control algorithm based on reinforcement learning, characterized by: include: Acquire torque signals of the robot joints, and construct a state vector representing the interaction state between the robot and the environment based on the torque signals; Based on the state vector, an adaptive motion control strategy is generated through a hierarchical reinforcement learning training mechanism, including: Construct a multi-dimensional reward function that integrates task completion, energy efficiency, motion safety, and smoothness; Iteratively optimizing the mapping relationship between the state vector and the action control sequence using a Q-learning algorithm; Validating the adaptive motion control strategy under various physical parameter environments and selectively triggering retraining based on validation results; The verified adaptive motion control strategy is deployed to the robot control system to achieve adaptive motion control.

2. The robot adaptive motion control algorithm based on reinforcement learning according to claim 1 is characterized in that: The step of obtaining the torque signal of the robot joint and constructing a state vector representing the interaction state between the robot and the environment based on the torque signal includes: The torque data is collected in real time by the torque sensors installed at each joint of the robot; performing low-pass filtering and normalization processing on the torque data; Construct a multidimensional state vector based on the current moment value, the previous moment value and its rate of change; The interaction between the robot and the virtual or actual obstacle is monitored in real time based on the state vector.

3. The robot adaptive motion control algorithm based on reinforcement learning according to claim 1 is characterized in that: The hierarchical reinforcement learning training mechanism includes: The bottom layer directly controls joint torque, responsible for high-frequency, small-grained torque adjustment; The middle-level motion trajectory planning layer is responsible for medium-frequency and medium-granularity trajectory optimization; The high-level mission strategy decision-making layer is responsible for low-frequency, large-granularity mission planning; The three layers form a closed-loop control structure through top-down command transmission and bottom-up state feedback.

4. The robot adaptive motion control algorithm based on reinforcement learning according to claim 1 is characterized in that: The multi-dimensional reward function that integrates task completion, energy efficiency, motion safety, and smoothness is constructed, including: Define task completion rewards, based on target location achievement and completion time efficiency evaluation; Define energy efficiency bonus, calculated based on joint power consumption and total energy consumption; Define safety rewards based on distance to obstacles and degree of dangerous posture; Define smoothness bonus, calculated based on torque change rate and trajectory continuity; The rewards of the above four dimensions are linearly combined through dynamic weight coefficients to form a comprehensive reward function.

5. The robot adaptive motion control algorithm based on reinforcement learning according to claim 1 is characterized in that: The iterative optimization of the mapping relationship between the state vector and the action control sequence using the Q-learning algorithm includes: Adaptive grid method is used to discretize the continuous state space, and fine-grained grids are used in key areas; Select actions based on the ε-greedy strategy, which improves the exploration rate in the early stage of training and the utilization rate in the later stage; Optimize the state-action value function through iterative updates controlled by learning rate and discount factor; The priority experience replay mechanism is used to improve learning efficiency, and samples with high TD error have a higher sampling probability.

6. The robot adaptive motion control algorithm based on reinforcement learning according to claim 1 is characterized in that: The step of validating the adaptive motion control strategy under multiple physical parameter environments and selectively triggering retraining based on the validation results includes: Construct a set of parameterized verification environments with different friction coefficients, damping coefficients, and mass distributions; Perform multiple independent tests in each verification environment to evaluate functional indicators, efficiency indicators, safety indicators and robustness indicators; Calculate the average performance index and the worst performance index of all verification environments; When the worst performance indicator falls below a threshold or the performance variance exceeds a preset range, the retraining process is triggered.

7. The robot adaptive motion control algorithm based on reinforcement learning according to claim 1 is characterized in that: The process of deploying the verified adaptive motion control strategy to the robot control system includes: Compression optimization of control strategies to reduce memory usage and improve query efficiency; Establish standardized control interfaces to achieve compatibility with different robot platforms; Design runtime monitoring and fine-tuning mechanisms to support online performance optimization; Implement policy version management and support rollback to stable version in abnormal situations.

8. The robot adaptive motion control algorithm based on reinforcement learning according to claim 5 is characterized in that: The motion control sequence includes joint torque increments, and each joint can select seven discrete motion levels: large increase, medium increase, small increase, maintain, small decrease, medium decrease or large decrease.

9. The robot adaptive motion control algorithm based on reinforcement learning according to claim 4 is characterized in that: The weight coefficients of each dimension in the multi-dimensional reward function are dynamically adjusted according to the task stage, environmental characteristics and historical performance, ensuring adaptive adjustment of the strategy optimization direction in different application scenarios.

10. The robot adaptive motion control algorithm based on reinforcement learning according to claim 3 is characterized in that: Each layer in the hierarchical reinforcement learning training mechanism uses different time scales and decision granularity: The underlying control cycle is 5-10 milliseconds, directly manipulating joint torque; The middle-level control cycle is 50-100 milliseconds, planning short-term trajectory segments; The high-level control cycle is 500-1000 milliseconds and is responsible for overall task strategy adjustment; Asynchronous collaborative control is achieved between layers through a buffer mechanism.

Images