A robot controller adaptive parameter optimization system based on reinforcement learning
By combining the improved Decision Transformer model with the TD3 algorithm, adaptive optimization of robot controller parameters was achieved, solving the problems of low efficiency and insufficient robustness in traditional methods, and improving the accuracy and safety of the robot control system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- QINGDAO QINGCHENG DIGITAL TECH CO LTD
- Filing Date
- 2026-05-09
- Publication Date
- 2026-06-12
AI Technical Summary
Existing technologies struggle to achieve efficient adaptive adjustment of controller parameters in robot control, especially in multi-objective control tasks where it is difficult to balance control accuracy and safety constraints. Traditional methods are inefficient and lack robustness.
An improved Decision Transformer model is combined with the TD3 algorithm. The control accuracy and safety constraint information are modeled in parallel through a dual-channel structure. A gating mechanism and cross-channel attention are introduced to achieve parameter modulation and feature fusion. The output control target weight parameters and constraint threshold parameters are then used to optimize the controller gain parameters in conjunction with the TD3 algorithm.
It achieves joint adaptive optimization of high precision, stability and safety in robot control system, improves system generalization ability and robustness, and adapts to dynamic changes under complex working conditions.
Smart Images
Figure CN122194628A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the interdisciplinary fields of robot control, artificial intelligence and deep reinforcement learning, and in particular to an adaptive parameter optimization system for robot controllers based on reinforcement learning. Background Technology
[0002] With the widespread application of intelligent robots in manufacturing, healthcare, and service industries, achieving efficient configuration and adaptive adjustment of robot controller parameters has become a key issue in control system design. In actual control processes, robot systems face various complex control objectives, such as precise end-effector trajectory tracking, smooth control response, limited actuator torque output, and joint speed limitations. These objectives often conflict with each other. To achieve a dynamic trade-off between control accuracy, response smoothness, and system safety, traditional control methods typically rely on manual experience to set the gain parameters of the proportional-integral-derivative (PID) controller and manually set the error weighting coefficients and torque / velocity threshold parameters. This approach is not only inefficient but also struggles to adapt to dynamic changes under complex operating conditions, exhibiting significant lag and poor robustness.
[0003] In recent years, deep reinforcement learning has demonstrated powerful performance in control problems with continuous action spaces. Representative algorithms such as TD3 (Twin Delayed Deep Deterministic Policy Gradient) have been widely applied to tasks such as robotic arms and autonomous navigation. However, existing methods often focus on the generation of controller actions, neglecting the joint optimization of weight and constraint parameters, making it difficult to achieve adaptive adjustment at different stages of the task. Furthermore, current reinforcement learning-based control architectures typically use a single information channel to process state features, making it difficult to simultaneously address the modeling requirements of control performance and safety constraints, thus limiting the model's generalization ability and interpretability in multi-objective control tasks.
[0004] Although the Decision Transformer model introduces sequence modeling capabilities, enabling offline decision-making through reinforcement learning in some tasks, its original structure does not model the "performance-safety" coupling characteristics in robot control, and it lacks a mechanism for structured output of control parameters, making it difficult to meet the adaptive requirements of robot control tasks that emphasize both high precision and high safety.
[0005] Therefore, how to provide an adaptive parameter optimization system for robot controllers based on reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0006] One objective of this invention is to propose an adaptive parameter optimization system for robot controllers based on reinforcement learning. This system employs an improved Decision Transformer model combined with the TD3 algorithm. A dual-channel structure is used to model control accuracy and safety constraints in parallel. Gating mechanisms and cross-channel attention are introduced to achieve parameter modulation and feature fusion, outputting control target weight parameters and constraint threshold parameters. Furthermore, the controller gain parameters are trained and optimized using the TD3 algorithm, ultimately achieving joint adaptive adjustment of control law parameters, target weight parameters, and constraint threshold parameters. This system possesses advantages such as strong modeling capability, high control accuracy, good safety, and high parameter optimization efficiency.
[0007] An adaptive parameter optimization system for a robot controller based on reinforcement learning according to an embodiment of the present invention includes:
[0008] The operation status data acquisition module is used to acquire the robot's operation status data in each control cycle;
[0009] The decision input sequence construction module is used to construct the expected return sequence based on the running status data, and to interleave the expected return sequence, running status data and historical parameter actions to construct the decision input sequence.
[0010] The embedding processing module is used to perform expected reward embedding, state embedding, action embedding and time step embedding processing on the decision input sequence to obtain the embedding sequence;
[0011] An improved Decision Transformer model module is used to receive embedded sequences, and sets up performance-oriented and safety-oriented channels inside the Transformer encoder to encode the embedded sequences and output a first encoded feature sequence and a second encoded feature sequence.
[0012] The gated modulation and feature fusion module is used to generate a gated signal based on the expected return embedding, and to conditionally modulate the first coded feature sequence and the second coded feature sequence using the gated signal. The conditionally modulated coded features are fused through the cross-channel attention module to output the control target weight parameters and constraint threshold parameters.
[0013] The control performance reward construction module is used to perform weighted processing, penalty processing and threshold comparison processing on the running status data based on the control target weight parameters and constraint threshold parameters, and generate control performance rewards.
[0014] The TD3 training module is used to train the continuous action policy network by taking control performance feedback and operating status data as input to the TD3 algorithm, and output the controller gain parameters.
[0015] The control execution and parameter update module is used to write the controller gain parameters, control target weight parameters, and constraint threshold parameters into the robot controller and execute the control output to update the historical parameter actions for the next control cycle.
[0016] Optionally, modules can be integrated using the following methods:
[0017] Step 1: Obtain the robot's operating status data in each control cycle, construct the expected reward sequence, and combine the expected reward sequence with the operating status data and historical parameter actions to form the decision input sequence;
[0018] Step 2: Embed the decision input sequence using expected reward embedding, state embedding, action embedding, and time step embedding respectively to obtain the embedding sequence;
[0019] Step 3: Input the embedded sequence into the improved Decision Transformer model, and set up a performance-oriented channel and a safety-oriented channel inside the Transformer encoder. The first encoded feature sequence is output by the performance-oriented channel and the second encoded feature sequence is output by the safety-oriented channel.
[0020] Step 4: Generate a gating signal based on the expected return embedding, and use the gating signal to conditionally modulate the first and second encoded feature sequences. Then, fuse the conditionally modulated encoded features through a cross-channel attention module to output the control target weight parameters and constraint threshold parameters.
[0021] Step 5: Construct the control performance reward based on the control target weight parameters and constraint threshold parameters, and use the control performance reward and running status data as input to the TD3 algorithm;
[0022] Step 6: Train the continuous action policy network using the TD3 algorithm and output the controller gain parameters, which include proportional gain parameters, integral gain parameters, and differential gain parameters;
[0023] Step 7: Write the controller gain parameters, control target weight parameters, and constraint threshold parameters into the robot controller and execute the control output to update the historical parameter actions for the next control cycle.
[0024] Optionally, step one specifically includes:
[0025] In each control cycle, robot operating status data is collected and a state vector is formed. The operating status data includes end-effector pose error, end-effector velocity error, joint velocity, joint torque, and control input.
[0026] The state vector is marked with time steps and cached in order of control cycle to generate a sequence of running state data;
[0027] Generate the expected return sequence based on the running status data sequence, and configure a return identifier corresponding to the time step marker for each time step of the expected return sequence;
[0028] Get the parameters and actions output in the previous control cycle and the N consecutive control cycles before the previous control cycle, cache them in the time order of the control cycles, and generate a historical parameter and action sequence.
[0029] The historical parameter actions include proportional gain parameters, integral gain parameters, differential gain parameters, position error weights, velocity error weights, control input penalty weights, maximum permissible joint torque thresholds, and maximum permissible joint velocity thresholds;
[0030] At each time step, the expected return, operational status data, and historical parameter actions are interleaved to construct a decision input sequence.
[0031] Optionally, step two specifically includes:
[0032] The decision input sequence is analyzed in time step order to obtain the expected return, running status data and historical parameter actions corresponding to each time step;
[0033] The expected reward corresponding to each time step is processed by expected reward embedding to obtain the expected reward embedding sequence, and the expected reward embedding sequence is configured with a time step position code consistent with the time step label;
[0034] The running state data corresponding to each time step is processed by state embedding to obtain a state embedding sequence, and a time step position code consistent with the time step mark is configured for the state embedding sequence.
[0035] Perform action embedding processing on the historical parameter actions corresponding to each time step to obtain an action embedding sequence, and configure the time step position code consistent with the time step label for the action embedding sequence;
[0036] The expected return embedding sequence, state embedding sequence, and action embedding sequence are combined in an alternating order according to the decision input sequence, and the combined sequence is then uniformly aligned to obtain the embedding sequence.
[0037] Optionally, step three specifically includes:
[0038] The Transformer encoder will embed a sequence input improved Decision Transformer model; within the Transformer encoder, a performance-oriented channel and a safety-oriented channel for sequence parallel input will be embedded.
[0039] The performance-oriented channel includes a first encoding network. When encoding the embedded sequence, the first encoding network constructs a query set, a key set, and a value set for attention computation based on the tokens of all time steps in the embedded sequence, and performs self-attention computation on the query set, key set, and value set to output the first encoded feature sequence.
[0040] The secure routing channel includes a second coding network and a risky router network;
[0041] The risk router network is used to perform route calculations on each token in the embedded sequence and output the corresponding route tag;
[0042] Construct an attention-based connection mask for the secure-direction channel based on the routing tags of each token;
[0043] When the second encoding network performs self-attention computation on the embedded sequence, it limits the token pairs that can participate in the attention computation based on the attention connection mask, and performs self-attention computation and feedforward mapping processing between the limited token pairs to output the second encoded feature sequence.
[0044] The first encoded feature sequence output by the performance-oriented channel and the second encoded feature sequence output by the security-oriented channel are obtained respectively.
[0045] Optionally, step four specifically includes:
[0046] Gating signal generation processing is performed on the expected return embedding based on the expected return embedding sequence to obtain the gating signal sequence;
[0047] The gate signal sequence is applied to the first coded feature sequence to obtain the first modulation feature sequence conditionally modulated by the gate signal, and the gate signal sequence is applied to the second coded feature sequence to obtain the second modulation feature sequence conditionally modulated by the gate signal.
[0048] The first modulation feature sequence and the second modulation feature sequence are input into the cross-channel attention module. The cross-channel attention module performs cross-channel attention calculation on the first modulation feature sequence and the second modulation feature sequence to obtain the fused feature sequence.
[0049] Based on the fused feature sequence, parameter output processing is performed to output control target weight parameters and constraint threshold parameters;
[0050] The control target weight parameters include position error weight, velocity error weight, and control input penalty weight, and the constraint threshold parameters include the maximum permissible joint torque threshold and the maximum permissible joint velocity threshold.
[0051] Optionally, the construction of control performance returns based on control target weight parameters and constraint threshold parameters specifically includes:
[0052] The end-effector pose error and end-effector velocity error in the running status data are weighted based on the control target weight parameters.
[0053] Based on the control input penalty weight in the control target weight parameter, penalty processing is performed on the control input in the running status data to obtain the performance error term and the control input term;
[0054] Based on the constraint threshold parameter, perform threshold comparison processing on the joint torque and joint velocity in the running status data to generate constraint satisfaction flags;
[0055] Based on the performance error term, the control input term, and the constraint satisfaction flag, the performance report generation process is performed to obtain the control performance report.
[0056] Optionally, step six specifically includes:
[0057] The input samples for the TD3 algorithm include operational status data and control performance reports;
[0058] An empirical sample set for TD3 algorithm training is constructed based on the input samples of the TD3 algorithm, and the samples in the empirical sample set are written into the empirical playback unit according to the control cycle order.
[0059] Training batch samples are extracted from the experience playback unit and input into the continuous action policy network and the value evaluation network;
[0060] The value assessment network is updated based on training batch samples.
[0061] After completing the parameter update processing of the value assessment network, a delayed update processing is performed on the continuous action strategy network.
[0062] The updated continuous action policy network outputs the controller gain parameters;
[0063] The controller gain parameters include proportional gain parameters, integral gain parameters, and derivative gain parameters.
[0064] Optionally, step seven specifically includes:
[0065] Write the controller gain parameter to the gain parameter register of the robot controller, write the control target weight parameter to the control target weight register of the robot controller, and write the constraint threshold parameter to the constraint threshold register of the robot controller.
[0066] The controller control law parameter configuration is generated based on the written controller gain parameter, the control target weight configuration is generated based on the written control target weight parameter, and the constraint threshold configuration is generated based on the written constraint threshold parameter.
[0067] Within the current control cycle, control calculations are performed on the operating status data using the controller control law parameter configuration, control target weight configuration, and constraint threshold configuration to generate control output;
[0068] The control output is sent to the robot actuator to drive the robot's movement;
[0069] The system acquires the operating status data for the next control cycle and writes the controller gain parameter, control target weight parameter, and constraint threshold parameter corresponding to the current control cycle into the historical parameter action sequence as the output parameter action of the current control cycle, which is then used to update the historical parameter action for the next control cycle.
[0070] Optionally, the control calculation specifically includes:
[0071] Extract the end-effector pose error, end-effector velocity error, joint velocity, and joint torque from the operating status data of the current control cycle;
[0072] A control error signal is constructed based on the end-effector pose error and end-effector velocity error of the current control cycle;
[0073] Based on the proportional gain parameter, integral gain parameter and derivative gain parameter in the controller gain parameter, proportional processing, integral processing and derivative processing are performed on the control error signal respectively to generate the control quantity after gain adjustment;
[0074] Based on the position error weight, velocity error weight and control input penalty weight in the control target weight parameters, weight modulation processing is performed on the control quantity after gain adjustment to generate a weighted control quantity.
[0075] The weighted control quantity is constrained by the maximum allowable joint torque threshold and the maximum allowable joint velocity threshold in the constraint threshold parameters. If the constraint threshold is exceeded, the weighted control quantity is restricted to generate a control input that satisfies the constraint conditions.
[0076] The control input that meets the constraints is used as the control output for the current control cycle.
[0077] The beneficial effects of this invention are:
[0078] This invention introduces an improved Decision Transformer model, which sets up performance-oriented and safety-oriented channels inside the Transformer encoder. This enables parallel modeling of information related to control accuracy and system smoothness, as well as information related to system safety, during robot operation, thereby improving the expressive power of state features and the pertinence of modeling.
[0079] The gating signal generated by the expected return embedding is used to perform conditional modulation on the encoded features of the two channels, and a cross-channel attention mechanism is introduced to fuse the modulated features. This can dynamically adjust the attention to the control target weight parameters and the safety constraint threshold parameters, thereby improving the responsiveness and adaptability of parameter optimization.
[0080] The fused feature sequence is used to output the control target weight parameters and constraint threshold parameters. Combined with the TD3 reinforcement learning algorithm, the proportional gain, integral gain and derivative gain parameters of the controller are trained and optimized, enabling the controller to achieve precise and stable control under constantly changing task objectives and environmental conditions, with good generalization ability and robustness.
[0081] By uniformly writing optimized control parameters and driving the robot to execute control tasks, combined with continuous updates of historical parameter actions, a closed-loop adaptive control optimization system was realized, effectively improving the dynamic performance and safety performance of the control system.
[0082] This invention enables joint adaptive optimization of controller gain parameters, control target weight parameters, and safety constraint threshold parameters, thereby improving the accuracy, stability, and safety of robot control systems. It has good practical value and promising prospects for widespread application. Attached Figure Description
[0083] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0084] Figure 1 This is a schematic diagram of the structure of an adaptive parameter optimization system for a robot controller based on reinforcement learning proposed in this invention.
[0085] Figure 2 This is an overall flowchart of an adaptive parameter optimization method for robot controllers based on reinforcement learning proposed in this invention.
[0086] Figure 3 This is a schematic diagram of the improved Decision Transformer model structure of an adaptive parameter optimization system for robot controllers based on reinforcement learning proposed in this invention. Detailed Implementation
[0087] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0088] refer to Figures 1-3 An adaptive parameter optimization system for robot controllers based on reinforcement learning, comprising:
[0089] The operation status data acquisition module is used to acquire the robot's operation status data in each control cycle;
[0090] The decision input sequence construction module is used to construct the expected return sequence based on the running status data, and to interleave the expected return sequence, running status data and historical parameter actions to construct the decision input sequence.
[0091] The embedding processing module is used to perform expected reward embedding, state embedding, action embedding and time step embedding processing on the decision input sequence to obtain the embedding sequence;
[0092] An improved Decision Transformer model module is used to receive embedded sequences, and sets up performance-oriented and safety-oriented channels inside the Transformer encoder to encode the embedded sequences and output a first encoded feature sequence and a second encoded feature sequence.
[0093] The gated modulation and feature fusion module is used to generate a gated signal based on the expected return embedding, and to conditionally modulate the first coded feature sequence and the second coded feature sequence using the gated signal. The conditionally modulated coded features are fused through the cross-channel attention module to output the control target weight parameters and constraint threshold parameters.
[0094] The control performance reward construction module is used to perform weighted processing, penalty processing and threshold comparison processing on the running status data based on the control target weight parameters and constraint threshold parameters, and generate control performance rewards.
[0095] The TD3 training module is used to train the continuous action policy network by taking control performance feedback and operating status data as input to the TD3 algorithm, and output the controller gain parameters.
[0096] The control execution and parameter update module is used to write the controller gain parameters, control target weight parameters, and constraint threshold parameters into the robot controller and execute the control output to update the historical parameter actions for the next control cycle.
[0097] In this embodiment, the modules are interconnected using the following method:
[0098] Step 1: Obtain the robot's operating status data in each control cycle, construct the expected reward sequence, and combine the expected reward sequence with the operating status data and historical parameter actions to form the decision input sequence;
[0099] Step 2: Embed the decision input sequence using expected reward embedding, state embedding, action embedding, and time step embedding respectively to obtain the embedding sequence;
[0100] Step 3: Input the embedded sequence into the improved Decision Transformer model, and set up a performance-oriented channel and a safety-oriented channel inside the Transformer encoder. The first encoded feature sequence is output by the performance-oriented channel and the second encoded feature sequence is output by the safety-oriented channel.
[0101] Step 4: Generate a gating signal based on the expected return embedding, and use the gating signal to conditionally modulate the first and second encoded feature sequences. Then, fuse the conditionally modulated encoded features through a cross-channel attention module to output the control target weight parameters and constraint threshold parameters.
[0102] Step 5: Construct the control performance reward based on the control target weight parameters and constraint threshold parameters, and use the control performance reward and running status data as input to the TD3 algorithm;
[0103] Step 6: Train the continuous action policy network using the TD3 algorithm and output the controller gain parameters, which include proportional gain parameters, integral gain parameters, and differential gain parameters;
[0104] Step 7: Write the controller gain parameters, control target weight parameters, and constraint threshold parameters into the robot controller and execute the control output to update the historical parameter actions for the next control cycle.
[0105] In this embodiment, step one specifically includes:
[0106] In each control cycle, robot operating status data is collected and a state vector is formed. The operating status data includes end-effector pose error, end-effector velocity error, joint velocity, joint torque, and control input.
[0107] The state vector is marked with time steps and cached in order of control cycle to generate a sequence of running state data;
[0108] Generate the expected return sequence based on the running status data sequence, and configure a return identifier corresponding to the time step marker for each time step of the expected return sequence;
[0109] The expected reward sequence is generated by performing reward generation processing on the running status data of each time step to generate the expected reward sequence corresponding to each time step, and configuring a reward identifier consistent with the corresponding time step label for each time step in the expected reward sequence.
[0110] Get the parameters and actions output in the previous control cycle and the N consecutive control cycles before the previous control cycle, cache them in the time order of the control cycles, and generate a historical parameter and action sequence.
[0111] The historical parameter actions include proportional gain parameters, integral gain parameters, differential gain parameters, position error weights, velocity error weights, control input penalty weights, maximum permissible joint torque thresholds, and maximum permissible joint velocity thresholds;
[0112] At each time step, the expected return, operational status data, and historical parameter actions are interleaved to construct a decision input sequence.
[0113] In this embodiment, step two specifically includes:
[0114] The decision input sequence is analyzed in time step order to obtain the expected return, running status data and historical parameter actions corresponding to each time step;
[0115] The expected reward corresponding to each time step is processed by expected reward embedding to obtain the expected reward embedding sequence, and the expected reward embedding sequence is configured with a time step position code consistent with the time step label;
[0116] The expected return embedding process includes: inputting the expected return corresponding to each time step into the return embedding mapping layer, and converting the expected return into a return embedding vector of a preset dimension through a linear mapping method.
[0117] The running state data corresponding to each time step is processed by state embedding to obtain a state embedding sequence, and a time step position code consistent with the time step mark is configured for the state embedding sequence.
[0118] The state embedding process is performed on the running state data corresponding to each time step, specifically including: inputting the running state data of each time step into the state embedding mapping module, converting the running state data into a state embedding vector of a preset dimension through linear mapping or feedforward neural network; generating a time step position encoding vector according to the corresponding time step label, and combining the time step position encoding vector with the state embedding vector to form a state embedding sequence.
[0119] Perform action embedding processing on the historical parameter actions corresponding to each time step to obtain an action embedding sequence, and configure the time step position code consistent with the time step label for the action embedding sequence;
[0120] The historical parameter actions corresponding to each time step are processed by action embedding, specifically including: inputting the historical parameter actions of each time step into the action embedding mapping module, converting the historical parameter actions into action embedding vectors of a preset dimension through linear mapping or feedforward neural networks; at the same time, generating time step position encoding vectors according to the corresponding time step labels, and combining the time step position encoding vectors with the action embedding vectors to form an action embedding sequence.
[0121] The expected return embedding sequence, state embedding sequence, and action embedding sequence are combined in an alternating order according to the decision input sequence, and the combined sequence is then uniformly aligned to obtain the embedding sequence.
[0122] In this embodiment, step three specifically includes:
[0123] The Transformer encoder will embed a sequence input improved Decision Transformer model; within the Transformer encoder, a performance-oriented channel and a safety-oriented channel for sequence parallel input will be embedded.
[0124] The performance-oriented channel includes a first encoding network. When encoding the embedded sequence, the first encoding network constructs a query set, a key set, and a value set for attention computation based on the tokens of all time steps in the embedded sequence, and performs self-attention computation on the query set, key set, and value set to output the first encoded feature sequence.
[0125] The secure routing channel includes a second coding network and a risky router network;
[0126] The risk router network is used to perform route calculations on each token in the embedded sequence and output the corresponding route tag;
[0127] The risk router network is used to calculate the routing label of each token in the embedded sequence. The calculation process includes: for each time step token in the embedded sequence, obtaining the corresponding token representation and the expected reward embedding corresponding token representation, and concatenating the token representation and the expected reward embedding representation; inputting the concatenated joint representation into the mapping layer of the risk router network, generating an intermediate risk feature representation through nonlinear mapping; further mapping the intermediate risk feature representation into a risk score, and generating a corresponding routing label based on the risk score. The routing label is used to limit the participation relationship of the token in the self-attention calculation of the security-oriented channel.
[0128] Construct an attention-based connection mask for the secure-direction channel based on the routing tags of each token;
[0129] When the second encoding network performs self-attention computation on the embedded sequence, it limits the token pairs that can participate in the attention computation based on the attention connection mask, and performs self-attention computation and feedforward mapping processing between the limited token pairs to output the second encoded feature sequence.
[0130] The first encoded feature sequence output by the performance-oriented channel and the second encoded feature sequence output by the security-oriented channel are obtained respectively.
[0131] In this embodiment, step four specifically includes:
[0132] Gating signal generation processing is performed on the expected return embedding based on the expected return embedding sequence to obtain the gating signal sequence;
[0133] The gate signal sequence is applied to the first coded feature sequence to obtain the first modulation feature sequence conditionally modulated by the gate signal, and the gate signal sequence is applied to the second coded feature sequence to obtain the second modulation feature sequence conditionally modulated by the gate signal.
[0134] The first modulation feature sequence and the second modulation feature sequence are input into the cross-channel attention module. The cross-channel attention module performs cross-channel attention calculation on the first modulation feature sequence and the second modulation feature sequence to obtain the fused feature sequence.
[0135] Based on the fused feature sequence, parameter output processing is performed to output control target weight parameters and constraint threshold parameters;
[0136] The parameter output processing includes: selecting a feature vector corresponding to the current time step from the fused feature sequence; inputting the feature vector into the parameter mapping module, performing mapping processing on the feature vector to generate an intermediate parameter vector; performing parameter grouping processing on the intermediate parameter vector, and outputting control target weight parameters used to balance control accuracy and system smoothness, and constraint threshold parameters used to ensure safe operation of the system.
[0137] The control target weight parameters include position error weight, velocity error weight, and control input penalty weight, and the constraint threshold parameters include the maximum permissible joint torque threshold and the maximum permissible joint velocity threshold.
[0138] In this embodiment, step five specifically includes:
[0139] The end-effector pose error and end-effector velocity error in the running status data are weighted based on the control target weight parameters.
[0140] Based on the control input penalty weight in the control target weight parameter, penalty processing is performed on the control input in the running status data to obtain the performance error term and the control input term;
[0141] The weighted end-effector pose error and the weighted end-effector velocity error are combined to generate a performance error term;
[0142] Control inputs are extracted from the operating status data, and weighted calculations are performed on the control inputs based on the control input penalty weights in the control target weight parameters to generate control input items;
[0143] Based on the constraint threshold parameter, perform threshold comparison processing on the joint torque and joint velocity in the running status data to generate constraint satisfaction flags;
[0144] The threshold comparison process includes: extracting joint torque data and joint velocity data corresponding to the current control cycle from the operating status data; obtaining the maximum permissible joint torque threshold and the maximum permissible joint velocity threshold from the constraint threshold parameters; numerically comparing the joint torque data with the maximum permissible joint torque threshold and the joint velocity data with the maximum permissible joint velocity threshold; and generating a constraint satisfaction flag based on the numerical comparison result to indicate the satisfaction status of joint torque and joint velocity relative to the corresponding constraint threshold within the current control cycle.
[0145] Based on the performance error term, the control input term, and the constraint satisfaction flag, the performance report generation process is performed to obtain the control performance report.
[0146] The reward generation process includes: obtaining a performance error term, a control input term, and a constraint satisfaction flag; performing numerical combination processing on the performance error term and the control input term to generate a basic reward value; performing constraint action processing on the basic reward value based on the constraint satisfaction flag to generate a control performance reward; and outputting the control performance reward as the reward value corresponding to the current control cycle.
[0147] In this embodiment, step six specifically includes:
[0148] The input samples for the TD3 algorithm include operational status data and control performance reports;
[0149] An empirical sample set for TD3 algorithm training is constructed based on the input samples of the TD3 algorithm, and the samples in the empirical sample set are written into the empirical playback unit according to the control cycle order.
[0150] Training batch samples are extracted from the experience playback unit and input into the continuous action policy network and the value evaluation network;
[0151] The value assessment network is updated based on training batch samples.
[0152] After completing the parameter update processing of the value assessment network, a delayed update processing is performed on the continuous action strategy network.
[0153] The updated continuous action policy network outputs the controller gain parameters;
[0154] The controller gain parameters include proportional gain parameters, integral gain parameters, and derivative gain parameters.
[0155] In this embodiment, step seven specifically includes:
[0156] Write the controller gain parameter to the gain parameter register of the robot controller, write the control target weight parameter to the control target weight register of the robot controller, and write the constraint threshold parameter to the constraint threshold register of the robot controller.
[0157] The controller control law parameter configuration is generated based on the written controller gain parameter, the control target weight configuration is generated based on the written control target weight parameter, and the constraint threshold configuration is generated based on the written constraint threshold parameter.
[0158] Within the current control cycle, control calculations are performed on the operating status data using the controller control law parameter configuration, control target weight configuration, and constraint threshold configuration to generate control output;
[0159] The control output is sent to the robot actuator to drive the robot's movement;
[0160] The system acquires the operating status data for the next control cycle and writes the controller gain parameter, control target weight parameter, and constraint threshold parameter corresponding to the current control cycle into the historical parameter action sequence as the output parameter action of the current control cycle, which is then used to update the historical parameter action for the next control cycle.
[0161] In this embodiment, the control calculation specifically includes:
[0162] Extract the end-effector pose error, end-effector velocity error, joint velocity, and joint torque from the operating status data of the current control cycle;
[0163] A control error signal is constructed based on the end-effector pose error and end-effector velocity error of the current control cycle;
[0164] Based on the proportional gain parameter, integral gain parameter and derivative gain parameter in the controller gain parameter, proportional processing, integral processing and derivative processing are performed on the control error signal respectively to generate the control quantity after gain adjustment;
[0165] Based on the position error weight, velocity error weight and control input penalty weight in the control target weight parameters, weight modulation processing is performed on the control quantity after gain adjustment to generate a weighted control quantity.
[0166] The weighted control quantity is constrained by the maximum allowable joint torque threshold and the maximum allowable joint velocity threshold in the constraint threshold parameters. If the constraint threshold is exceeded, the weighted control quantity is restricted to generate a control input that satisfies the constraint conditions.
[0167] The control input that meets the constraints is used as the control output for the current control cycle.
[0168] The original Decision Transformer model typically employs a single Transformer encoder structure, unifying the expected reward, state, and action sequences into the same attention space for modeling. Its core objective is to predict the next action based on historical trajectories, primarily used for policy generation problems. In this structure, different types of state information are mixed and modeled within the same channel, lacking a structural distinction between control performance objectives and safety constraint objectives. Furthermore, the model output usually directly corresponds to action or policy parameters, making it difficult to structurally adjust multiple types of parameters in the controller.
[0169] In this embodiment, the improved Decision Transformer model proposed in this invention introduces a dual-channel structure—a performance-oriented channel and a safety-oriented channel—within the Transformer encoder. This allows for parallel encoding of information related to control accuracy and system smoothness, as well as information related to safety constraints such as joint torque and joint velocity. This structurally achieves decoupled modeling of different control objective information. Simultaneously, a risk router network is introduced into the safety-oriented channel. By performing routing calculations on tokens and constructing attention connection masks, the participation of safety-related information in attention calculations is limited, making the modeling of safety features more targeted, unlike the original Decision Transformer's full attention modeling approach.
[0170] The improved model introduces a gating signal mechanism based on expected reward embedding to perform conditional modulation on the encoded features of the dual-channel output and fuse them through a cross-channel attention module, thereby dynamically adjusting the contribution ratio of performance features and safety features under different expected reward conditions. This structured modulation method is not present in the original Decision Transformer, enabling the model to better adapt to multi-objective control scenarios. Furthermore, the output of the improved model is no longer a direct control action, but rather control objective weight parameters and constraint threshold parameters, providing structured and interpretable parameter inputs for the subsequent TD3 algorithm to optimize the controller gain parameters.
[0171] Example 1:
[0172] To verify the effectiveness of the reinforcement learning-based adaptive parameter optimization system for robot controllers proposed in this invention, the invention was applied to the optimization process of the control system of a six-DOF industrial robot in a precision assembly task. In practical tasks, this robot is responsible for performing highly repetitive operations with extremely high requirements for control accuracy, response speed, and execution safety, such as the insertion and alignment of precision components. In traditional control schemes, controller parameters mainly rely on manual settings or static optimization methods, which are prone to failing to adapt in dynamic environments, leading to insufficient control accuracy, system instability, or operation exceeding safety thresholds.
[0173] In the experimental environment, the robot was required to switch rapidly and accurately between multiple assembly positions. The system initially used a traditional PID controller with a uniform gain value set as the initial controller parameter. In traditional solutions, when the robot performs high-speed insertion tasks, it often encounters problems such as end-effector position deviation exceeding ±1.5mm, large speed fluctuations, and joint torque exceeding limits. The control input continuously jitters over multiple cycles, severely affecting assembly quality and the lifespan of the robot's actuators.
[0174] To address the aforementioned issues, the adaptive optimization system proposed in this invention replaces the original control strategy. At the beginning of the experiment, the system collects data such as end-effector pose error, end-effector velocity error, joint velocity, joint torque, and control input for each control cycle, constructs a state vector, and generates a historical parameter action sequence. Using an improved DecisionTransformer model, the system embeds the state data, historical control parameters, and expected reward information into both the performance-oriented and safety-oriented channels in parallel, extracting representative coding features for accuracy and safety respectively. Then, through a gating mechanism and a cross-channel attention fusion mechanism, the system adaptively outputs control target weight parameters (such as position error weight, velocity error weight, etc.) and constraint threshold parameters (such as maximum permissible joint velocity, joint torque, etc.), further driving the TD3 algorithm to train a continuous action strategy network, dynamically generating optimal PID gain controller parameters.
[0175] After 500 control cycles, comparisons with traditional control strategies across multiple evaluation metrics revealed that the system of this invention significantly improves the stability and accuracy of robot control during execution. Experimental data shows that under the same task conditions, the average end-effector position error of the traditional scheme is 1.42 mm, with a peak value of 3.3 mm, while the scheme of this invention reduces the average error to 0.78 mm and controls the peak error within 1.6 mm; the maximum joint torque value decreases from 78 Nm to within 64 Nm; the overall system energy consumption decreases by approximately 11.3%; the assembly success rate increases from the traditional 91.6% to 97.4%; and the fluctuation amplitude of robot control input is significantly reduced, with the standard deviation of the control input change rate decreasing by 27.1%, demonstrating good system smoothness.
[0176] The system's adaptive adjustment capability of control parameters enables the controller to quickly adjust in the event of sudden load changes or minor adjustments to the end target point position without human intervention, effectively improving the system's robustness and practicality. The entire optimization process is executed automatically by the system, with a short deployment cycle, strong adaptability, and industrial-grade promotion value. Specific experimental data are shown in Table 1:
[0177] Table 1. Performance Comparison Data Between the System of the Invention and Traditional Controllers
[0178] Control Indicators Traditional control scheme Invention Solution Improvement effect Number of test cycles Maximum value reduction rate rate of change of standard deviation Average end position error (mm) 1.42 0.78 ↓45.1% 500 ↓51.5% ↓32.3% Maximum end position error (mm) 3.3 1.6 ↓51.5% 500 ↓51.5% - Maximum joint torque (Nm) 78 64 ↓17.9% 500 ↓17.9% - Average energy consumption (unit / kWh) 2.37 2.10 ↓11.3% 500 - - Standard deviation of control input change rate 0.97 0.71 ↓27.1% 500 - ↓27.1% Assembly task success rate (%) 91.6 97.4 ↑6.3% 500 - -
[0179] As can be seen from Table 1, the present invention not only significantly improves control accuracy and safety, but also reduces the workload of parameter tuning and the errors or losses caused by unreasonable parameter settings, thus verifying the advanced nature and practicality of the present invention in the field of robot control.
Claims
1. An adaptive parameter optimization system for a robot controller based on reinforcement learning, characterized in that, include: The operation status data acquisition module is used to acquire the robot's operation status data in each control cycle; The decision input sequence construction module is used to construct the expected return sequence based on the running status data, and to interleave the expected return sequence, running status data and historical parameter actions to construct the decision input sequence. The embedding processing module is used to perform expected reward embedding, state embedding, action embedding and time step embedding processing on the decision input sequence to obtain the embedding sequence; An improved Decision Transformer model module is used to receive embedded sequences, and sets up performance-oriented and safety-oriented channels inside the Transformer encoder to encode the embedded sequences and output a first encoded feature sequence and a second encoded feature sequence. The gated modulation and feature fusion module is used to generate a gated signal based on the expected return embedding, and to conditionally modulate the first coded feature sequence and the second coded feature sequence using the gated signal. The conditionally modulated coded features are fused through the cross-channel attention module to output the control target weight parameters and constraint threshold parameters. The control performance reward construction module is used to perform weighted processing, penalty processing and threshold comparison processing on the running status data based on the control target weight parameters and constraint threshold parameters, and generate control performance rewards. The TD3 training module is used to train the continuous action policy network by taking control performance feedback and operating status data as input to the TD3 algorithm, and output the controller gain parameters. The control execution and parameter update module is used to write the controller gain parameters, control target weight parameters, and constraint threshold parameters into the robot controller and execute the control output to update the historical parameter actions for the next control cycle.
2. The adaptive parameter optimization system for a robot controller based on reinforcement learning according to claim 1, characterized in that, The modules are connected in the following way: Step 1: Obtain the robot's operating status data in each control cycle, construct the expected reward sequence, and combine the expected reward sequence with the operating status data and historical parameter actions to form the decision input sequence; Step 2: Embed the decision input sequence using expected reward embedding, state embedding, action embedding, and time step embedding respectively to obtain the embedding sequence; Step 3: Input the embedded sequence into the improved Decision Transformer model, and set up a performance-oriented channel and a safety-oriented channel inside the Transformer encoder. The performance-oriented channel outputs the first encoded feature sequence and the safety-oriented channel outputs the second encoded feature sequence. Step 4: Generate a gating signal based on the expected return embedding, and use the gating signal to conditionally modulate the first and second encoded feature sequences. Then, fuse the conditionally modulated encoded features through a cross-channel attention module to output the control target weight parameters and constraint threshold parameters. Step 5: Construct the control performance reward based on the control target weight parameters and constraint threshold parameters, and use the control performance reward and running status data as input to the TD3 algorithm; Step 6: Train the continuous action policy network using the TD3 algorithm and output the controller gain parameters, which include proportional gain parameters, integral gain parameters, and differential gain parameters; Step 7: Write the controller gain parameters, control target weight parameters, and constraint threshold parameters into the robot controller and execute the control output to update the historical parameter actions for the next control cycle.
3. The adaptive parameter optimization system for a robot controller based on reinforcement learning according to claim 2, characterized in that, Step one specifically includes: In each control cycle, robot operating status data is collected and a state vector is formed. The operating status data includes end-effector pose error, end-effector velocity error, joint velocity, joint torque, and control input. The state vector is marked with time steps and cached in order of control cycle to generate a sequence of running state data; Generate the expected return sequence based on the running status data sequence, and configure a return identifier corresponding to the time step marker for each time step of the expected return sequence; Get the parameters and actions output in the previous control cycle and the N consecutive control cycles before the previous control cycle, cache them in the time order of the control cycles, and generate a historical parameter and action sequence. The historical parameter actions include proportional gain parameters, integral gain parameters, differential gain parameters, position error weights, velocity error weights, control input penalty weights, maximum permissible joint torque thresholds, and maximum permissible joint velocity thresholds; At each time step, the expected return, operational status data, and historical parameter actions are interleaved to construct a decision input sequence.
4. The adaptive parameter optimization system for a robot controller based on reinforcement learning according to claim 2, characterized in that, Step two specifically includes: The decision input sequence is analyzed in time step order to obtain the expected return, running status data and historical parameter actions corresponding to each time step; The expected reward corresponding to each time step is processed by expected reward embedding to obtain the expected reward embedding sequence, and the expected reward embedding sequence is configured with a time step position code consistent with the time step label; The running state data corresponding to each time step is processed by state embedding to obtain a state embedding sequence, and a time step position code consistent with the time step mark is configured for the state embedding sequence. Perform action embedding processing on the historical parameter actions corresponding to each time step to obtain an action embedding sequence, and configure the time step position code consistent with the time step label for the action embedding sequence; The expected return embedding sequence, state embedding sequence, and action embedding sequence are combined in an alternating order according to the decision input sequence, and the combined sequence is then uniformly aligned to obtain the embedding sequence.
5. The adaptive parameter optimization system for a robot controller based on reinforcement learning according to claim 2, characterized in that, Step three specifically includes: The Transformer encoder will embed a sequence input improved Decision Transformer model; within the Transformer encoder, a performance-oriented channel and a safety-oriented channel for sequence parallel input will be embedded. The performance-oriented channel includes a first encoding network. When encoding the embedded sequence, the first encoding network constructs a query set, a key set, and a value set for attention computation based on the tokens of all time steps in the embedded sequence, and performs self-attention computation on the query set, key set, and value set to output the first encoded feature sequence. The secure routing channel includes a second coding network and a risky router network; The risk router network is used to perform route calculations on each token in the embedded sequence and output the corresponding route tag; Construct an attention-based connection mask for the secure-direction channel based on the routing tags of each token; When the second encoding network performs self-attention computation on the embedded sequence, it limits the token pairs that can participate in the attention computation based on the attention connection mask, and performs self-attention computation and feedforward mapping processing between the limited token pairs to output the second encoded feature sequence. The first encoded feature sequence output by the performance-oriented channel and the second encoded feature sequence output by the security-oriented channel are obtained respectively.
6. The adaptive parameter optimization system for a robot controller based on reinforcement learning according to claim 2, characterized in that, Step four specifically includes: Gating signal generation processing is performed on the expected return embedding based on the expected return embedding sequence to obtain the gating signal sequence; The gate signal sequence is applied to the first coded feature sequence to obtain the first modulation feature sequence conditionally modulated by the gate signal, and the gate signal sequence is applied to the second coded feature sequence to obtain the second modulation feature sequence conditionally modulated by the gate signal. The first modulation feature sequence and the second modulation feature sequence are input into the cross-channel attention module. The cross-channel attention module performs cross-channel attention calculation on the first modulation feature sequence and the second modulation feature sequence to obtain the fused feature sequence. Based on the fused feature sequence, parameter output processing is performed to output control target weight parameters and constraint threshold parameters; The control target weight parameters include position error weight, velocity error weight, and control input penalty weight, and the constraint threshold parameters include the maximum permissible joint torque threshold and the maximum permissible joint velocity threshold.
7. The adaptive parameter optimization system for a robot controller based on reinforcement learning according to claim 2, characterized in that, The construction of control performance returns based on control target weight parameters and constraint threshold parameters specifically includes: The end-effector pose error and end-effector velocity error in the running status data are weighted based on the control target weight parameters. Based on the control input penalty weight in the control target weight parameter, penalty processing is performed on the control input in the running status data to obtain the performance error term and the control input term; Based on the constraint threshold parameter, perform threshold comparison processing on the joint torque and joint velocity in the running status data to generate constraint satisfaction flags; Based on the performance error term, the control input term, and the constraint satisfaction flag, the performance report generation process is performed to obtain the control performance report.
8. The adaptive parameter optimization system for a robot controller based on reinforcement learning according to claim 2, characterized in that, Step six specifically includes: The input samples for the TD3 algorithm include operational status data and control performance reports; An empirical sample set for TD3 algorithm training is constructed based on the input samples of the TD3 algorithm, and the samples in the empirical sample set are written into the empirical playback unit according to the control cycle order. Training batch samples are extracted from the experience playback unit and input into the continuous action policy network and the value evaluation network; The value assessment network is updated based on training batch samples. After completing the parameter update processing of the value assessment network, a delayed update processing is performed on the continuous action strategy network. The updated continuous action policy network outputs the controller gain parameters; The controller gain parameters include proportional gain parameters, integral gain parameters, and derivative gain parameters.
9. The adaptive parameter optimization system for a robot controller based on reinforcement learning according to claim 2, characterized in that, Step seven specifically includes: Write the controller gain parameter to the gain parameter register of the robot controller, write the control target weight parameter to the control target weight register of the robot controller, and write the constraint threshold parameter to the constraint threshold register of the robot controller. The controller control law parameter configuration is generated based on the written controller gain parameter, the control target weight configuration is generated based on the written control target weight parameter, and the constraint threshold configuration is generated based on the written constraint threshold parameter. Within the current control cycle, control calculations are performed on the operating status data using the controller control law parameter configuration, control target weight configuration, and constraint threshold configuration to generate control output; The control output is sent to the robot actuator to drive the robot's movement; The system acquires the operating status data for the next control cycle and writes the controller gain parameter, control target weight parameter, and constraint threshold parameter corresponding to the current control cycle into the historical parameter action sequence as the output parameter action of the current control cycle, which is then used to update the historical parameter action for the next control cycle.
10. The adaptive parameter optimization system for a robot controller based on reinforcement learning according to claim 9, characterized in that, The control calculation specifically includes: Extract the end-effector pose error, end-effector velocity error, joint velocity, and joint torque from the operating status data of the current control cycle; A control error signal is constructed based on the end-effector pose error and end-effector velocity error of the current control cycle; Based on the proportional gain parameter, integral gain parameter and derivative gain parameter in the controller gain parameter, proportional processing, integral processing and derivative processing are performed on the control error signal respectively to generate the control quantity after gain adjustment; Based on the position error weight, velocity error weight and control input penalty weight in the control target weight parameters, weight modulation processing is performed on the control quantity after gain adjustment to generate a weighted control quantity. The weighted control quantity is constrained by the maximum allowable joint torque threshold and the maximum allowable joint velocity threshold in the constraint threshold parameters. If the constraint threshold is exceeded, the weighted control quantity is restricted to generate a control input that satisfies the constraint conditions. The control input that meets the constraints is used as the control output for the current control cycle.