A robot joint parameter optimization method based on reinforcement learning
By combining reinforcement learning and dynamic response correlation analysis based on working condition stages, along with double-Q value assessment and hardware safety penalties, the stability and safety issues in robot joint parameter optimization are resolved, achieving more efficient adaptive parameter optimization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHANGZHOU MEILE INTELLIGENT TECHNOLOGY CO LTD
- Filing Date
- 2026-05-07
- Publication Date
- 2026-06-05
AI Technical Summary
Existing robot joint parameter optimization methods suffer from poor stability when faced with sensor jitter, motor fluctuations, and mechanical vibration disturbances. They also exhibit parameter migration distortion across operating conditions, causal mismatch between training samples and real results, and insufficient hardware safety margins and physical consistency constraints on parameters, leading to unstable and unsafe parameter optimization.
By employing reinforcement learning, working condition segmentation, dynamic response correlation analysis, and double-Q value assessment techniques, and through parameter-sensitive low-frequency feature extraction, working condition isolation gating coding, and parameter ramp constraint mapping, combined with hardware safety margin penalty units, adaptive optimization of joint parameters is achieved.
It improves the real-time performance and stability of robot joint parameter optimization, reduces the risk of parameter jitter and cross-condition migration distortion, enhances hardware safety constraints, and improves the reliability and consistency of control performance.
Smart Images
Figure CN122143058A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot intelligent control and parameter optimization technology, and in particular to a method for optimizing robot joint parameters based on reinforcement learning. Background Technology
[0002] With the widespread application of industrial robots, autonomous robotic arms, and collaborative robots in precision assembly, flexible material handling, and complex work scenarios, adaptive optimization techniques for robot joint control parameters have received continuous attention. Existing methods for tuning robot joint parameters mainly rely on manual experience, preset rule corrections, or conventional reinforcement learning control strategies for parameter updates. However, these methods commonly suffer from the following problems in practical applications: The collected joint position, velocity, current, and driving torque data are easily affected by sensor jitter, motor commutation fluctuations, and mechanical vibration disturbances, resulting in high noise in the state representation and poor stability of the parameter optimization process. The joint dynamic characteristics vary significantly under different operating conditions, and existing methods are insufficient in suppressing historical state interference during the switching process, which can easily cause distortion in parameter migration across operating conditions. Conventional time window sample construction methods cannot fully represent the dynamic response process after a single parameter adjustment, and they do not pay enough attention to the risk process approaching the safety boundary, resulting in a causal mismatch between training samples and actual parameter tuning effects. At the same time, existing value assessment methods do not adequately consider hardware safety margins and physical consistency constraints of parameters, which can easily lead to overestimation, parameter mutations, and online jitter.
[0003] Therefore, how to provide a method for optimizing robot joint parameters based on reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0004] One objective of this invention is to propose a robot joint parameter optimization method based on reinforcement learning. This invention utilizes reinforcement learning, working condition stage division, dynamic response correlation analysis, and double Q value evaluation technology to describe in detail the intelligent optimization process of robot joint control parameters. It can adaptively generate joint parameter adjustment actions according to the operating state and load changes, and has the advantages of high real-time parameter optimization, strong operational stability, good safety constraint capability, and excellent control effect.
[0005] A robot joint parameter optimization method based on reinforcement learning according to an embodiment of the present invention includes the following steps: Step 1: Collect the running status information of each joint of the target robot, perform parameter-sensitive low-frequency feature extraction and high-frequency electromechanical noise suppression processing, and generate a sequence of robot joint running statuses; Step 2: Divide the current operating condition of the robot into stages and generate operating condition stage information corresponding to the robot joint operating state sequence; Step 3: Introduce a variable-length closed-loop sampling mechanism based on steady-state triggering to perform dynamic response correlation processing on the robot joint running state sequence and working condition stage information, and introduce a divergent pre-risk extraction mechanism to perform risk process screening processing on the dynamic response correlation processing results to generate a training sample set. Step 4: Input the robot joint running state sequence and corresponding working condition stage information from the training sample set into the improved GTrXL policy network. The improved GTrXL policy network includes a working condition isolation gating coding module and a parameter ramp constraint mapping layer, and outputs candidate joint parameter adjustment actions. Step 5: Input the current joint running state vector, the current working condition stage information, and the candidate joint parameter adjustment actions into the dual-Q value network. The dual-Q value network includes two independent Q value evaluation branches and a hardware safety margin penalty unit, and outputs the corresponding target action value. Step 6: Based on the training sample set and candidate joint parameters, adjust the action and target action value, and jointly iterate and update the improved GTrXL policy network and the double Q value network under the constraint of joint dynamic physical consistency; Step 7: Generate joint parameter adjustment actions in real time based on the updated and improved GTrXL policy network, and output the optimal joint parameter combination after processing by fine-tuning dead zone and sliding smoothing unit.
[0006] Optionally, step one specifically includes: Collect joint position data, joint velocity data, joint current data, and joint driving torque data of each joint of the target robot in a continuous control cycle, and aggregate them according to the sampling time corresponding to the unified control clock to form the original joint state sampling sequence, and perform time alignment processing. After time alignment processing, the original joint state sampling sequence is subjected to parameter-sensitive low-frequency feature extraction and high-frequency electromechanical noise suppression processing. The parameter-sensitive low-frequency feature extraction retains low-frequency change components based on the continuous change trend of joint position data, joint velocity data, joint current data, and joint driving torque data between adjacent control cycles. The high-frequency electromechanical noise suppression processing weakens the high-frequency fluctuation components caused by sensor jitter, motor commutation fluctuation, and mechanical vibration disturbance, generating purified joint state data. The purified joint state data is combined with the end-efficiency execution state and load change state at the corresponding sampling time to generate the joint running state vector corresponding to each control time. Arrange the joint running state vectors according to the control time sequence to generate the robot joint running state sequence.
[0007] Optionally, step two specifically includes: Based on the joint running state vector corresponding to each control moment, the joint speed change state, joint driving torque change state, end effector state switching state and load change trend are extracted to form a working condition discrimination data group. The working condition discrimination data group is matched according to the preset stage division rules. The stage division rules are a set of working condition category discrimination rules established for the start-up adjustment stage, acceleration and deceleration transition stage, steady-state operation stage and load disturbance stage. The operating conditions corresponding to each control moment are classified into one of the operating condition categories, and operating condition stage labels corresponding to each control moment are generated. The labels of each working condition stage are arranged in the order of control time to generate working condition stage information corresponding to the robot joint running state sequence.
[0008] Optionally, step three specifically includes: The joint running state vector at each control moment in the robot joint running state sequence is paired with the working condition stage information at the corresponding control moment in the order of control moment to form a joint state-working condition joint time sequence. Read the joint control parameter records corresponding to each control moment, and determine the control moment when the joint control parameters change relative to the previous control moment as the sample preparation start moment; Starting from the initial time of each configuration, the joint running state vector, working condition stage information, joint parameter adjustment actions and trajectory tracking error data corresponding to each subsequent control time are extracted in chronological order to form candidate dynamic response segments. The trajectory tracking error data in the candidate dynamic response segments are processed by first-order difference and moving average. When the absolute value of the moving average is less than or equal to the preset steady-state threshold and the preset number of control cycles is met continuously, the last control time is determined as the sample termination time. The candidate dynamic response segments between the sample start time and the sample termination time are extracted to generate the dynamic response association processing result. First-order difference processing is performed on the trajectory tracking error data in the dynamic response correlation processing results, and the control time interval in which the difference results continuously increase is determined as the error deterioration interval; Read the joint current data, joint driving torque data and load change status data corresponding to each control moment within the error deterioration interval, filter them according to the preset current warning threshold, preset driving torque warning threshold and preset load change warning threshold, and exclude data segments with hardware emergency stop markers. The results of each dynamic response association processing and the corresponding risk process samples are collected in the order of sample construction start time to generate a training sample set.
[0009] Optionally, step four specifically includes: The robot joint running state sequence in the training sample set is aligned with the corresponding working condition stage information according to the control time sequence to form a joint input sequence; The joint running state vector in the joint input sequence is input into the state embedding layer to generate a state embedding sequence; The operating condition stage information in the joint input sequence is input into the operating condition isolation gating coding module. Based on the comparison result of the operating condition stage label of the current control time and the previous control time, the isolation gating coefficient is generated. The isolation gating coefficient is used to adjust the transmission ratio of the historical buffer and the residual update ratio of the current input feature to generate the operating condition gating coding sequence. The state embedding sequence and the operating condition gated coding sequence are input together into the GTrXL backbone coding layer to generate a temporal hidden state sequence. Read the timing hidden state corresponding to the current control moment and input it into the parameter output layer; output the original parameter residual adjustment amount corresponding to each joint. The original parameter residual adjustment amount corresponding to each joint is combined with the joint control parameter corresponding to the current control time to form the original parameter adjustment result; The original parameter adjustment results are input into the parameter slope constraint mapping layer. The joint parameter adjustment actions issued in the previous control moment are read, the parameter difference is calculated, and the parameter difference is compared with the preset maximum parameter change rate joint by joint. For parameter differences that exceed the preset maximum parameter change rate, amplitude limiting mapping is performed, and for parameter differences that do not exceed the preset maximum parameter change rate, the parameter difference remains unchanged. The parameter difference after the amplitude limiting mapping is combined with the joint parameter adjustment actions issued in the previous control moment to generate candidate joint parameter adjustment actions for the current control moment.
[0010] Optionally, step five specifically includes: The current joint running state vector is concatenated with the current working condition stage information to form a state working condition combination vector, and then input into the two Q value evaluation branches in the double Q value network. The network hierarchy structure of the two Q value evaluation branches is the same and the network parameters are set independently. In each Q-value evaluation branch, the state condition feature extraction layer is used to extract features from the state condition combination vector, and the action feature extraction layer is used to extract action features from the candidate joint parameter adjustment action. The state and condition coding features and action coding features are concatenated end to end and multiplied element by element. The combined results are then input into the fusion layer to output the hidden action value features. The value regression layer is used to perform numerical mapping processing on the hidden features of action value, and the original action value of the corresponding Q value evaluation branch is output. Read the joint current data and joint driving torque data from the current joint running state vector, calculate the current saturation and driving torque saturation respectively, and take the larger value between the current saturation and driving torque saturation of the same joint as the joint saturation index. Compare the joint saturation indices corresponding to all joints, take the joint saturation indices with the largest values as the overall saturation indices at the current control moment, and determine the hardware safety margin level by dividing the intervals according to the preset safety margin level. According to the preset penalty coefficient mapping relationship, the value penalty coefficient corresponding to the hardware safety margin level at the current control moment is read, and the value penalty coefficient is used to perform multiplicative attenuation processing on the original action values output by the two Q value evaluation branches to generate the corresponding corrected action values. The correction action values of the outputs of the two Q-value evaluation branches are compared, and the correction action value with the smaller value is taken as the target action value corresponding to the current control moment.
[0011] Optionally, step six specifically includes: Extract training sample fragments from the training sample set, and read the corresponding candidate joint parameter adjustment actions and target action values; The joint running state sequence, working condition stage information and candidate joint parameter adjustment action in the training sample segment are input into the double Q value network to obtain the current action value prediction result; The candidate joint parameter adjustment action is combined with the joint control parameters at the corresponding control time joint by joint to obtain the updated joint control parameter group. Based on the joint stiffness parameters and the current working condition stage information, the corresponding damping allowable range is retrieved from the preset stiffness-damping matching table, and the current joint damping parameters are compared with the damping allowable range to determine the damping matching state; Based on the position loop gain parameters and the current operating condition stage information, the corresponding speed loop gain allowable range is retrieved from the preset position loop-speed loop matching table, and the current speed loop gain parameters are compared with the speed loop gain allowable range to determine the servo matching status. Calculate the differences in joint stiffness parameters, joint damping parameters, position loop gain parameters, and velocity loop gain parameters between adjacent control moments, and compare them with the corresponding preset parameter change thresholds to determine the parameter change state; When the damping matching state is matched, the servo matching state is matched, and the parameter change state is continuously changing, the corresponding training sample segment is determined to meet the preset physical consistency condition; otherwise, the corresponding training sample segment is determined not to meet the preset physical consistency condition. Based on the judgment results of the preset physical consistency conditions corresponding to each training sample segment, the target action value is subjected to consistency correction processing to generate the corrected target action value; Based on the current action value prediction results and the corrected target action value, the parameters of the double-Q value network are updated; The joint running state sequence and working condition stage information are input into the improved GTrXL policy network to generate updated candidate joint parameter adjustment actions. The updated candidate joint parameter adjustment actions are then input into the double Q value network after parameter updates are completed to obtain the updated action value prediction results. Based on the updated action value prediction results and the preset physical consistency condition judgment results of the corresponding training sample segments, the parameters of the improved GTrXL policy network are updated. Repeatedly execute the dual-Q value network update and the improved GTrXL strategy network update until the preset iteration stop condition is met.
[0012] Optionally, step seven specifically includes: Based on the joint running state vector, end effector state, and load change state corresponding to the current control moment, generate the operating condition stage information corresponding to the current control moment; Input the joint running state vector and working condition stage information corresponding to the current control moment into the updated improved GTrXL strategy network, and output the original joint parameter adjustment action corresponding to the current control moment. The original joint parameter adjustment action is input into the fine-tuning dead zone and sliding smoothing unit. The absolute value of the original joint parameter adjustment amount corresponding to each joint is compared with the preset dead zone threshold of the corresponding joint. When the absolute value of the original joint parameter adjustment amount is less than or equal to the preset dead zone threshold of the corresponding joint, the original joint parameter adjustment amount corresponding to the current joint is set to zero. When the absolute value of the original joint parameter adjustment amount is greater than the preset dead zone threshold of the corresponding joint, the original joint parameter adjustment amount corresponding to the current joint remains unchanged, and the dead zone processed joint parameter adjustment amount is generated. The joint parameter adjustment amount after dead zone processing corresponding to the current control time of each joint is combined with the same joint parameter adjustment amount output by multiple consecutive control times in the previous sequence through a sliding window. The moving average processing is then performed according to the time order of the parameter adjustment amount of each control time in the window to generate the smooth joint parameter adjustment amount corresponding to the current control time. The smooth joint parameter adjustment amount corresponding to the current control time of each joint is combined with the current joint control parameter to generate the optimal joint parameter combination corresponding to the current control time.
[0013] The beneficial effects of this invention are: This invention addresses the technical problems of high noise in the original state and insufficient effective parameter tuning information by performing parameter-sensitive low-frequency feature extraction and high-frequency electromechanical noise suppression on joint position, velocity, current, and driving torque data, thus making the robot joint running state sequence more stable. It also solves the problems of working condition switching interference, sample causal mismatch, and insufficient utilization of risk samples by using a working condition stage division, a variable-length closed-loop sampling mechanism based on steady-state triggering, and a divergent pre-risk extraction mechanism. This ensures that training samples can cover the complete dynamic response of a single parameter tuning and highlight the instability front process. Finally, it introduces an improved GTrX model, including a working condition isolation gating coding module and a parameter ramp constraint mapping layer. The L-policy network and the dual-Q value network, which includes a hardware safety margin penalty unit, solve the problems of parameter migration distortion, parameter mutation, and insufficient hardware safety boundary constraints across operating conditions. This enables candidate joint parameter adjustment actions to balance temporal adaptability, execution smoothness, and value assessment reliability. By jointly updating the network under the constraint of joint dynamic physical consistency and introducing fine-tuning dead zones and sliding smoothing units in the online stage, the problems of parameter jitter and unreasonable updates during real machine deployment are solved. This makes the output optimal joint parameter combination more available and stable. Its significance lies in improving the engineering implementation capability of reinforcement learning joint parameter optimization methods in real robot scenarios. Attached Figure Description
[0014] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a robot joint parameter optimization method based on reinforcement learning proposed in this invention; Figure 2 This is a schematic diagram of a robot joint parameter optimization method based on reinforcement learning proposed in this invention; Figure 3 This is a framework diagram of the improved GTrXL policy network in the robot joint parameter optimization method based on reinforcement learning proposed in this invention. Detailed Implementation
[0015] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0016] refer to Figures 1-3 A method for optimizing robot joint parameters based on reinforcement learning includes the following steps: Step 1: Collect the running status information of each joint of the target robot, perform parameter-sensitive low-frequency feature extraction and high-frequency electromechanical noise suppression processing, and generate a sequence of robot joint running statuses; Step 2: Divide the current operating condition of the robot into stages and generate operating condition stage information corresponding to the robot joint operating state sequence; Step 3: Introduce a variable-length closed-loop sampling mechanism based on steady-state triggering to perform dynamic response correlation processing on the robot joint running state sequence and working condition stage information, and introduce a divergent pre-risk extraction mechanism to perform risk process screening processing on the dynamic response correlation processing results to generate a training sample set. Step 4: Input the robot joint running state sequence and corresponding working condition stage information from the training sample set into the improved GTrXL policy network. The improved GTrXL policy network includes a working condition isolation gating encoding module and a parameter ramp constraint mapping layer, and outputs candidate joint parameter adjustment actions. Step 5: Input the current joint running state vector, the current working condition stage information, and the candidate joint parameter adjustment actions into the dual-Q value network. The dual-Q value network includes two independent Q value evaluation branches and a hardware safety margin penalty unit, and outputs the corresponding target action value. Step 6: Based on the training sample set and candidate joint parameters, adjust the action and target action value, and jointly iterate and update the improved GTrXL policy network and the double Q value network under the constraint of joint dynamic physical consistency; Step 7: Generate joint parameter adjustment actions in real time based on the updated and improved GTrXL policy network, and output the optimal joint parameter combination after processing by fine-tuning dead zone and sliding smoothing unit.
[0017] In this embodiment, step one specifically includes: The system collects the running status information of each joint of the target robot within a continuous control cycle. The running status information includes joint position data, joint velocity data, joint current data, and joint driving torque data. The data is then collected according to the sampling time corresponding to a unified control clock to form an original joint status sampling sequence. Time alignment processing is performed on the original joint state sampling sequence. The time alignment processing is to use the sampling time corresponding to the unified control clock as a reference, match the original joint state data with sampling time deviation, and merge the original joint state data corresponding to the same sampling time into the same control cycle. After time alignment processing, the original joint state sampling sequence is subjected to parameter-sensitive low-frequency feature extraction and high-frequency electromechanical noise suppression. The parameter-sensitive low-frequency feature extraction is based on the continuous change trend of joint position data, joint velocity data, joint current data, and joint driving torque data between adjacent control cycles to extract the corresponding low-frequency change components. The high-frequency electromechanical noise suppression process weakens the high-frequency fluctuation components caused by sensor jitter, motor commutation fluctuations, and mechanical vibration disturbances in the original joint state sampling sequence to generate purified joint state data. The purified joint state data is the set of joint state data after low-frequency change component retention and high-frequency fluctuation component weakening. The purified joint state data is combined with the end effector state and load change state at the corresponding sampling time, and arranged in the order of joint number to generate the joint running state vector corresponding to each control time. The end effector state is the action execution state data of the robot end effector at the corresponding sampling time, and the load change state is the load change data of the robot at the corresponding sampling time. Arrange the joint running state vectors corresponding to each control moment in chronological order to generate a sequence of robot joint running states.
[0018] In this embodiment, step two specifically includes: Based on the joint running state vector corresponding to each control moment, the trends of joint speed change, joint driving torque change, end effect state switching and load change are extracted to form a working condition discrimination data set. The stage division rule matching process is performed on the working condition discrimination data group. The stage division rule is a set of working condition classification rules established based on the changing trends of joint speed change state, joint driving torque change state, end execution state switching state and load change state. According to the phase division rules, the operating conditions corresponding to each control moment are divided into one of the following: start-up adjustment phase, acceleration / deceleration transition phase, steady-state operation phase, and load disturbance phase, and a corresponding operating condition phase label is generated for each control moment. Arrange the labels of the working condition stages corresponding to each control moment according to the order of control moments to generate working condition stage information corresponding to the robot joint running state sequence.
[0019] In this embodiment, step three specifically includes: The joint running state vector at each control moment in the robot joint running state sequence is paired with the corresponding working condition stage information according to the control moment order to form a joint state-working condition joint time sequence. Read the joint control parameter records corresponding to each control moment, and determine the control moment when the joint control parameter records change relative to the previous control moment as the sample formation start moment; The steady-state triggered variable-length closed-loop sampling mechanism includes: taking the starting time of each sampling as the starting point, continuously extracting the joint running state vector, working condition stage information, joint parameter adjustment actions, and trajectory tracking error data corresponding to each subsequent control time along the time sequence to form candidate dynamic response segments; performing first-order difference processing on the trajectory tracking error data in the candidate dynamic response segments to obtain the error change amount corresponding to each control time; performing moving average processing on the error change amount with a preset sliding window length to obtain the mean error change amount corresponding to each control time; determining the last control time where the absolute value of the mean error change amount is less than or equal to a preset steady-state threshold and continuously satisfies a preset number of control cycles as the sampling termination time; extracting candidate dynamic response segments between the sampling start time and the sampling termination time to generate dynamic response association processing results; The divergent pre-risk extraction mechanism includes: performing first-order difference processing on the trajectory tracking error data in the dynamic response correlation processing results to obtain the error change amount; continuously increasing the error change amount at multiple consecutive control times, and determining the control time interval with continuously increasing error change amount as the error deterioration interval; reading the joint current data, joint driving torque data and load change state data corresponding to each control time within the error deterioration interval, and comparing the thresholds according to the preset current warning threshold, preset driving torque warning threshold and preset load change warning threshold, and filtering out data segments of joint current data, joint driving torque data or load change state data that fall into the preset safety warning interval and do not have hardware emergency stop markers at the corresponding control time, and determining the data segments as risk process samples; The results of each dynamic response association processing and the corresponding risk process samples are collected in the order of sample construction start time to generate a training sample set; This implementation method uses a variable-length closed-loop sampling mechanism based on steady-state triggering to associate a single joint parameter adjustment action with its subsequent complete dynamic response process, solving the pain points of conventional fixed-length samples being unable to cover the parameter tuning lag effect and the causal mismatch between training samples and real results. At the same time, through a divergent pre-risk extraction mechanism, process samples that are close to the safety boundary but have not yet stopped are screened, solving the problem of insufficient utilization of high-risk key samples, thereby improving the quality of training samples, risk identification ability and the stability of subsequent policy learning.
[0020] In this embodiment, step four specifically includes: Align the robot joint running state sequence in the training sample set with the corresponding working condition stage information according to the control time sequence to form a joint input sequence; The joint running state vectors at each control time point in the joint input sequence are input into the state embedding layer. The state embedding layer includes a fully connected transformation layer and a normalization processing layer. The fully connected transformation layer performs dimension unification processing on the joint running state vectors at each control time point, and the normalization processing layer performs scale unification processing on the fully connected transformation result to generate the state embedding sequence. The operating condition stage information of each control moment in the joint input sequence is input into the operating condition isolation gating coding module. The operating condition isolation gating coding module includes a stage label comparison unit, an isolation gating coefficient generation unit, and a historical cache adjustment unit. The stage label comparison unit reads the operating condition stage label of the current control moment and the operating condition stage label of the previous control moment, and outputs the stage switching determination result. The isolation gating coefficient generation unit generates the isolation gating coefficient corresponding to the current control moment based on the stage switching determination result. When the stage switching determination result indicates that the operating condition has changed, the historical cache adjustment unit reduces the transfer weight of the historical cache of the previous control moment to the current control moment and increases the proportion of the input features of the current control moment in the residual update. When the stage switching determination result indicates that the operating condition has not changed, it maintains the original transfer weight of the historical cache of the previous control moment and generates the operating condition gating coding sequence. The state embedding sequence and the operating condition gated coding sequence are jointly input into the GTrXL backbone coding layer. The GTrXL backbone coding layer consists of multiple GTrXL coding units stacked in chronological order. Each GTrXL coding unit includes a relative position self-attention sub-layer, a gated residual sub-layer, and a feedforward transform sub-layer. The relative position self-attention sub-layer performs correlation calculation on the input features of the current control time and the historical cache features of the previous control time. The gated residual sub-layer performs weighted fusion on the input features of the current layer and the attention output features according to the isolation gate coefficient corresponding to the current control time. The feedforward transform sub-layer performs nonlinear transformation processing on the fusion result to generate a temporal hidden state sequence. Read the timing hidden state corresponding to the current control moment from the timing hidden state sequence and input it into the parameter output layer. The parameter output layer includes a fully connected prediction layer and a joint branch output layer. The fully connected prediction layer performs parameter prediction processing on the timing hidden state corresponding to the current control moment, and the joint branch output layer outputs the original parameter residual adjustment amount corresponding to each joint according to the joint number. The original parameter residual adjustment amount corresponding to each joint is combined with the joint control parameter corresponding to the current control time to form the original parameter adjustment result; The original parameter adjustment results are input into the parameter slope constraint mapping layer, which includes a parameter difference calculation unit, a rate of change comparison unit, and a slope limiting unit. The parameter difference calculation unit reads the joint parameter adjustment actions issued at the previous control time and calculates the parameter difference between the original parameter adjustment results at the current control time and the joint parameter adjustment actions issued at the previous control time. The rate of change comparison unit compares the parameter difference with the preset maximum parameter change rate joint by joint. The slope limiting unit performs limiting mapping on parameter differences that exceed the preset maximum parameter change rate, and keeps parameter differences that do not exceed the preset maximum parameter change rate unchanged. The parameter difference after the amplitude limiting mapping is combined with the joint parameter adjustment actions issued in the previous control moment to generate candidate joint parameter adjustment actions for the current control moment. The improved GTrXL policy network is similar to the original GTrXL model in that it uses a temporally encoded backbone to model the input sequence within a continuous control period, retains the basic structure of relative position self-attention, gated residual update and feedforward transformation, and uses the temporal hidden state as the basic representation for subsequent parameter prediction.
[0021] Compared with the original GTrXL model, this implementation adds a working condition isolation gating coding module on the input side and a parameter ramp constraint mapping layer on the output side: the former introduces working condition stage information into the historical cache update process and adjusts the transfer ratio of the historical cache and the residual update ratio of the current input features when the working condition changes; the latter performs rate of change constraint and amplitude limiting mapping on the parameter difference between the original parameter residual adjustment amount and the action issued at the previous control time.
[0022] Through the above improvements, the problems of easy crosstalk in historical representations and easy sudden changes in output parameters under operating conditions such as no-load, variable load, and disturbance switching of the original model are solved. The adjustment actions of candidate joint parameters generated by the strategy network can maintain the adaptability of operating conditions and meet the requirements of gradual response of the actuator, thereby reducing the risk of cross-operating condition parameter adjustment distortion, oscillation caused by sudden parameter changes, and instability in actual machine execution.
[0023] In this embodiment, step five specifically includes: The current joint running state vector is concatenated with the current working condition stage information to form a state working condition combination vector, and then input into the two Q value evaluation branches in the double Q value network. The network hierarchy structure of the two Q value evaluation branches is the same and the network parameters are set independently. Each Q-value assessment branch includes a state / condition feature extraction layer, an action feature extraction layer, a joint feature fusion layer, and a value regression layer. The state / condition feature extraction layer consists of an input fully connected layer, a residual connected layer, and an output fully connected layer. The action feature extraction layer consists of an action mapping fully connected layer and an action compression fully connected layer. The action mapping fully connected layer maps action features to candidate joint parameter adjustment actions, and the action compression fully connected layer compresses the mapped action features, outputting action encoding features. The joint feature fusion layer first concatenates the state / condition encoding features and the action encoding features, then performs element-wise multiplication, merging the concatenation result and the element-wise multiplication result to form a joint feature vector. This joint feature vector is input to the fusion fully connected layer, which outputs the hidden action value features. The value regression layer is a single-output fully connected layer that performs numerical mapping on the hidden action value features, outputting the original action value of the corresponding Q-value assessment branch. Read the joint current data and joint driving torque data of each joint in the current joint running state vector. Divide the absolute value of the joint current data of each joint by the corresponding preset current warning threshold to obtain the current saturation of each joint. Divide the absolute value of the joint driving torque data of each joint by the corresponding preset driving torque warning threshold to obtain the driving torque saturation of each joint. Take the larger value between the current saturation and the driving torque saturation of the same joint as the joint saturation index of that joint. Compare the joint saturation indices corresponding to all joints, take the joint saturation indices with the largest values as the overall saturation indices at the current control moment, and determine the hardware safety margin level by dividing the intervals according to the preset safety margin level. According to the preset penalty coefficient mapping relationship, the value penalty coefficient corresponding to the hardware safety margin level at the current control moment is read, and the value penalty coefficient is used to perform multiplicative attenuation processing on the original action values output by the two Q value evaluation branches to generate the corresponding corrected action values. The correction action values of the outputs of the two Q-value evaluation branches are compared, and the correction action value with the smaller value is taken as the target action value corresponding to the current control moment.
[0024] In this embodiment, step six specifically includes: Extract training sample fragments from the training sample set according to a preset batch size, and read the corresponding candidate joint parameter adjustment actions and target action values; The joint running state sequence, working condition stage information and candidate joint parameter adjustment action in the training sample segment are input into the double Q value network to obtain the current action value prediction result; The candidate joint parameter adjustment action is combined with the joint control parameters at the corresponding control time joint by joint to obtain the updated joint control parameter group. The updated joint control parameter group includes joint stiffness parameters, joint damping parameters, position loop gain parameters and velocity loop gain parameters. The updated joint control parameter group undergoes joint dynamics physical consistency constraint determination processing, specifically: based on the joint stiffness parameters and current operating condition information, the corresponding damping allowable interval is retrieved from the preset stiffness-damping matching table, and the current joint damping parameter is compared with the damping allowable interval to determine the damping matching state; based on the position loop gain parameters and current operating condition information, the corresponding velocity loop gain allowable interval is retrieved from the preset position loop-velocity loop matching table, and the current velocity loop gain parameter is compared with the velocity loop gain allowable interval to determine the servo matching state; the differences in joint stiffness parameters, joint damping parameters, position loop gain parameters, and velocity loop gain parameters between adjacent control moments are calculated and compared with the corresponding preset parameter change thresholds to determine the parameter change state; when the damping matching state is matched, the servo matching state is matched, and the parameter change state is continuously changing, the corresponding training sample segment is determined to meet the preset physical consistency condition; otherwise, the corresponding training sample segment is determined not to meet the preset physical consistency condition. Based on the judgment results of the preset physical consistency conditions corresponding to each training sample segment, the target action value is subjected to consistency correction processing to generate the corrected target action value; Based on the current action value prediction results and the corrected target action value, the parameters of the double-Q value network are updated; The joint running state sequence and working condition stage information are input into the improved GTrXL policy network to generate updated candidate joint parameter adjustment actions. The updated candidate joint parameter adjustment actions are then input into the double Q value network after parameter updates are completed to obtain the updated action value prediction results. Based on the updated action value prediction results and the preset physical consistency condition judgment results of the corresponding training sample segments, the network parameters of the improved GTrXL policy network are iteratively updated. When the corresponding training sample segment meets the preset physical consistency condition, the update weight of the corresponding updated action value prediction result remains unchanged; when the corresponding training sample segment does not meet the preset physical consistency condition, the update weight of the corresponding updated action value prediction result is reduced. Repeatedly execute the double-Q value network update and the improved GTrXL strategy network update until the preset iteration stop condition is met; In this embodiment, by introducing joint dynamics physical consistency constraint determination processing during the joint iterative update process of the dual-Q value network and the improved GTrXL policy network, the pain point of parameter combination in reinforcement learning training, which can improve the estimated value but does not meet the requirements of joint damping matching, servo matching, and continuous change, is solved. This makes the corrected target action value closer to real-world executable conditions, reduces the risk of training divergence, parameter mutations, and real-machine instability, and improves network convergence stability and online deployment reliability.
[0025] In this embodiment, step seven specifically includes: During the actual operation of the robot, the current operating status information of each joint is collected in real time according to the control cycle, and parameter-sensitive low-frequency feature extraction and high-frequency electromechanical noise suppression processing are performed to generate the joint operating status vector corresponding to the current control moment. Based on the joint running state vector, end effector state, and load change state corresponding to the current control moment, generate the operating condition stage information corresponding to the current control moment; Input the joint running state vector and working condition stage information corresponding to the current control moment into the updated improved GTrXL strategy network, and output the original joint parameter adjustment action corresponding to the current control moment. The original joint parameter adjustment action input parameter adjustment dead zone and sliding smoothing unit compare the absolute value of the original joint parameter adjustment amount corresponding to each joint with the preset dead zone threshold of the corresponding joint. When the absolute value of the original joint parameter adjustment amount is less than or equal to the preset dead zone threshold of the corresponding joint, the original joint parameter adjustment amount corresponding to that joint is set to zero. When the absolute value of the original joint parameter adjustment amount is greater than the preset dead zone threshold of the corresponding joint, the original joint parameter adjustment amount corresponding to that joint remains unchanged, and the dead zone processed joint parameter adjustment amount is generated. The joint parameter adjustment amount after dead zone processing corresponding to the current control time of each joint is combined with the same joint parameter adjustment amount output by multiple consecutive control times in the previous sequence through a sliding window. The moving average processing is then performed according to the time order of the parameter adjustment amount of each control time in the window to generate the smooth joint parameter adjustment amount corresponding to the current control time. The smooth joint parameter adjustment amount corresponding to the current control time of each joint is combined with the current joint control parameter to generate the optimal joint parameter combination corresponding to the current control time.
[0026] Example 1: To verify the feasibility of the invention in practice, it was applied to a six-degree-of-freedom collaborative robotic arm on a new energy vehicle electric drive controller assembly line. This robotic arm performs three types of continuous operations: gripping and transferring, positioning and insertion, and screw fastening. During operation, it simultaneously experiences a start-up adjustment phase, an acceleration / deceleration transition phase, a steady-state operation phase, and a load disturbance phase. During implementation, the control cycle was set to 10ms, and the length of the robot joint running state sequence was set to 32 control cycles. In the steady-state triggered variable-length closed-loop patterning mechanism, the steady-state judgment threshold was set to the absolute value of the sliding mean of the trajectory tracking error not exceeding 0.015rad, and the patterning termination condition was met for 10 consecutive control cycles. In the divergence pre-risk extraction mechanism, the current warning threshold was set to 0.85 of the rated current, the driving torque warning threshold was set to 0.80 of the rated driving torque, and the load change warning threshold was set to 1.15 of the estimated load value of adjacent control cycles. In the parameter ramp constraint mapping layer, the maximum change rate of stiffness and damping parameters was set to 3% per control cycle, and the maximum change rate of position loop gain and velocity loop gain parameters was set to 2.5% per control cycle. In the fine-tuning dead zone and sliding smoothing unit, the dead zone threshold was set to 0.5% of the nominal value of the current joint control parameters, and the sliding average window length was set to 5 control cycles.
[0027] In the comparison, the method of this invention is referred to as Method A; the unmodified model is referred to as Method B, which only retains the basic structure of the original GTrXL and double Q value network, without introducing the working condition isolation gating coding module, parameter ramp constraint mapping layer, variable length closed-loop sampling mechanism based on steady-state triggering, divergence pre-risk extraction mechanism, hardware safety margin penalty unit, joint dynamics physical consistency constraint, and fine-tuning dead zone and sliding smoothing unit; Method C is SAC+MLP parameter optimization; Method D is manual empirical tuning. The results of each method were statistically analyzed after continuous operation for 8 hours on the same production line, the same robotic arm, and the same batch of workpieces.
[0028] Table 1 Comparison of Robot Joint Parameter Optimization Examples
[0029] As shown in Table 1, compared with Method B, Method A reduced the average trajectory error from 0.57 mm to 0.42 mm (a decrease of 26.32%) in the comparison between the improved and unimproved models; the number of safety alert triggers decreased from 4.9 to 1.7 (a decrease of 65.31%); the risk process identification recall rate increased by 13.8 percentage points; and the proportion of invalid parameter updates decreased from 11.8% to 4.6%. This indicates that the working condition isolation gating coding module effectively weakened the crosstalk of historical representations during working condition switching, and the parameter ramp constraint mapping layer, fine-tuning dead zone, and sliding smoothing unit significantly reduced parameter mutations and jitter. Compared with Method C, the steady-state recovery time of this invention was shortened by 39.02%, and the single-cycle energy consumption decreased by 14.07%. This indicates that the variable-length closed-loop sampling mechanism based on steady-state triggering and the divergent pre-risk extraction mechanism improved the causal consistency of the samples, and the hardware safety margin penalty unit in the double-Q value network suppressed the estimation of aggressive actions approaching the hardware boundary. Compared to method D, this invention improves the task completion rate by 5.3 percentage points and reduces the joint temperature rise by 6.1℃, indicating that the joint dynamics physical consistency constraint makes the updated parameter combination more consistent with the actual machine operation.
[0030] As can be seen from this embodiment, the present invention can simultaneously improve tracking accuracy, recovery speed, energy consumption level and hardware security in scenarios involving multiple working conditions, load disturbances and long-term online operation, while reducing the number of invalid parameter updates and safety alarm triggers. This indicates that the method can not only optimize robot joint parameters, but also has strong engineering implementation capabilities and production line application value.
[0031] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A method for optimizing robot joint parameters based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Collect the running status information of each joint of the target robot, perform parameter-sensitive low-frequency feature extraction and high-frequency electromechanical noise suppression processing, and generate a sequence of robot joint running statuses; Step 2: Divide the current operating condition of the robot into stages and generate operating condition stage information corresponding to the robot joint operating state sequence; Step 3: Introduce a variable-length closed-loop sampling mechanism based on steady-state triggering to perform dynamic response correlation processing on the robot joint running state sequence and working condition stage information, and introduce a divergent pre-risk extraction mechanism to perform risk process screening processing on the dynamic response correlation processing results to generate a training sample set. Step 4: Input the robot joint running state sequence and corresponding working condition stage information from the training sample set into the improved GTrXL policy network. The improved GTrXL policy network includes a working condition isolation gating coding module and a parameter ramp constraint mapping layer, and outputs candidate joint parameter adjustment actions. Step 5: Input the current joint running state vector, the current working condition stage information, and the candidate joint parameter adjustment actions into the dual-Q value network. The dual-Q value network includes two independent Q value evaluation branches and a hardware safety margin penalty unit, and outputs the corresponding target action value. Step 6: Based on the training sample set and candidate joint parameters, adjust the action and target action value, and jointly iterate and update the improved GTrXL policy network and the double Q value network under the constraint of joint dynamic physical consistency; Step 7: Generate joint parameter adjustment actions in real time based on the updated and improved GTrXL policy network, and output the optimal joint parameter combination after processing by fine-tuning dead zone and sliding smoothing unit.
2. The method for optimizing robot joint parameters based on reinforcement learning according to claim 1, characterized in that, Step one specifically includes: Collect joint position data, joint velocity data, joint current data, and joint driving torque data of each joint of the target robot in a continuous control cycle, and aggregate them according to the sampling time corresponding to the unified control clock to form the original joint state sampling sequence, and perform time alignment processing. After time alignment processing, the original joint state sampling sequence is subjected to parameter-sensitive low-frequency feature extraction and high-frequency electromechanical noise suppression processing. The parameter-sensitive low-frequency feature extraction retains low-frequency change components based on the continuous change trend of joint position data, joint velocity data, joint current data, and joint driving torque data between adjacent control cycles. The high-frequency electromechanical noise suppression processing weakens the high-frequency fluctuation components caused by sensor jitter, motor commutation fluctuation, and mechanical vibration disturbance, generating purified joint state data. The purified joint state data is combined with the end-efficiency execution state and load change state at the corresponding sampling time to generate the joint running state vector corresponding to each control time. Arrange the joint running state vectors according to the control time sequence to generate the robot joint running state sequence.
3. The method for optimizing robot joint parameters based on reinforcement learning according to claim 1, characterized in that, Step two specifically includes: Based on the joint running state vector corresponding to each control moment, the joint speed change state, joint driving torque change state, end effector state switching state and load change trend are extracted to form a working condition discrimination data group. The working condition discrimination data group is matched according to the preset stage division rules. The stage division rules are a set of working condition category discrimination rules established for the start-up adjustment stage, acceleration and deceleration transition stage, steady-state operation stage and load disturbance stage. The operating conditions corresponding to each control moment are classified into one of the operating condition categories, and operating condition stage labels corresponding to each control moment are generated. The labels of each working condition stage are arranged in the order of control time to generate working condition stage information corresponding to the robot joint running state sequence.
4. The method for optimizing robot joint parameters based on reinforcement learning according to claim 1, characterized in that, Step three specifically includes: The joint running state vector at each control moment in the robot joint running state sequence is paired with the working condition stage information at the corresponding control moment in the order of control moment to form a joint state-working condition joint time sequence. Read the joint control parameter records corresponding to each control moment, and determine the control moment when the joint control parameters change relative to the previous control moment as the sample preparation start moment; Starting from the initial time of each configuration, the joint running state vector, working condition stage information, joint parameter adjustment actions and trajectory tracking error data corresponding to each subsequent control time are extracted in chronological order to form candidate dynamic response segments. The trajectory tracking error data in the candidate dynamic response segments are processed by first-order difference and moving average. When the absolute value of the moving average is less than or equal to the preset steady-state threshold and the preset number of control cycles is met continuously, the last control time is determined as the sample termination time. The candidate dynamic response segments between the sample start time and the sample termination time are extracted to generate the dynamic response association processing result. First-order difference processing is performed on the trajectory tracking error data in the dynamic response correlation processing results, and the control time interval in which the difference results continuously increase is determined as the error deterioration interval; Read the joint current data, joint driving torque data and load change status data corresponding to each control moment within the error deterioration interval, filter them according to the preset current warning threshold, preset driving torque warning threshold and preset load change warning threshold, and exclude data segments with hardware emergency stop markers. The results of each dynamic response association processing and the corresponding risk process samples are collected in the order of the sample construction start time to generate a training sample set.
5. The method for optimizing robot joint parameters based on reinforcement learning according to claim 1, characterized in that, Step four specifically includes: The robot joint running state sequence in the training sample set is aligned with the corresponding working condition stage information according to the control time sequence to form a joint input sequence; The joint running state vector in the joint input sequence is input into the state embedding layer to generate a state embedding sequence; The operating condition stage information in the joint input sequence is input into the operating condition isolation gating coding module. Based on the comparison result of the operating condition stage label of the current control time and the previous control time, the isolation gating coefficient is generated. The isolation gating coefficient is used to adjust the transmission ratio of the historical buffer and the residual update ratio of the current input feature to generate the operating condition gating coding sequence. The state embedding sequence and the operating condition gated coding sequence are input together into the GTrXL backbone coding layer to generate a temporal hidden state sequence. Read the timing hidden state corresponding to the current control moment and input it into the parameter output layer; output the original parameter residual adjustment amount corresponding to each joint. The original parameter residual adjustment amount corresponding to each joint is combined with the joint control parameter corresponding to the current control time to form the original parameter adjustment result; The original parameter adjustment results are input into the parameter slope constraint mapping layer. The joint parameter adjustment actions issued in the previous control moment are read, the parameter difference is calculated, and the parameter difference is compared with the preset maximum parameter change rate joint by joint. For parameter differences that exceed the preset maximum parameter change rate, amplitude limiting mapping is performed, and for parameter differences that do not exceed the preset maximum parameter change rate, the parameter difference remains unchanged. The parameter difference after the amplitude limiting mapping is combined with the joint parameter adjustment actions issued in the previous control moment to generate candidate joint parameter adjustment actions for the current control moment.
6. The method for optimizing robot joint parameters based on reinforcement learning according to claim 1, characterized in that, Step five specifically includes: The current joint running state vector is concatenated with the current working condition stage information to form a state working condition combination vector, and then input into the two Q value evaluation branches in the double Q value network. The network hierarchy structure of the two Q value evaluation branches is the same and the network parameters are set independently. In each Q-value evaluation branch, the state condition feature extraction layer is used to extract features from the state condition combination vector, and the action feature extraction layer is used to extract action features from the candidate joint parameter adjustment action. The state and condition coding features and action coding features are concatenated end to end and multiplied element by element. The combined results are then input into the fusion layer to output the hidden action value features. The value regression layer is used to perform numerical mapping processing on the hidden features of action value, and the original action value of the corresponding Q value evaluation branch is output. Read the joint current data and joint driving torque data from the current joint running state vector, calculate the current saturation and driving torque saturation respectively, and take the larger value between the current saturation and driving torque saturation of the same joint as the joint saturation index. Compare the joint saturation indices corresponding to all joints, take the joint saturation indices with the largest values as the overall saturation indices at the current control moment, and determine the hardware safety margin level by dividing the intervals according to the preset safety margin level. According to the preset penalty coefficient mapping relationship, the value penalty coefficient corresponding to the hardware safety margin level at the current control moment is read, and the value penalty coefficient is used to perform multiplicative attenuation processing on the original action values output by the two Q value evaluation branches to generate the corresponding corrected action values. The correction action values of the outputs of the two Q-value evaluation branches are compared, and the correction action value with the smaller value is taken as the target action value corresponding to the current control moment.
7. The method for optimizing robot joint parameters based on reinforcement learning according to claim 1, characterized in that, Step six specifically includes: Extract training sample fragments from the training sample set, and read the corresponding candidate joint parameter adjustment actions and target action values; The joint running state sequence, working condition stage information and candidate joint parameter adjustment action in the training sample segment are input into the double Q value network to obtain the current action value prediction result; The candidate joint parameter adjustment action is combined with the joint control parameters at the corresponding control time joint by joint to obtain the updated joint control parameter group. Based on the joint stiffness parameters and the current working condition stage information, the corresponding damping allowable range is retrieved from the preset stiffness-damping matching table, and the current joint damping parameters are compared with the damping allowable range to determine the damping matching state; Based on the position loop gain parameters and the current operating condition stage information, the corresponding speed loop gain allowable range is retrieved from the preset position loop-speed loop matching table, and the current speed loop gain parameters are compared with the speed loop gain allowable range to determine the servo matching status. Calculate the differences in joint stiffness parameters, joint damping parameters, position loop gain parameters, and velocity loop gain parameters between adjacent control moments, and compare them with the corresponding preset parameter change thresholds to determine the parameter change state; When the damping matching state is matched, the servo matching state is matched, and the parameter change state is continuously changing, the corresponding training sample segment is determined to meet the preset physical consistency condition; otherwise, the corresponding training sample segment is determined not to meet the preset physical consistency condition. Based on the judgment results of the preset physical consistency conditions corresponding to each training sample segment, the target action value is subjected to consistency correction processing to generate the corrected target action value; Based on the current action value prediction results and the corrected target action value, the parameters of the double-Q value network are updated; The joint running state sequence and working condition stage information are input into the improved GTrXL policy network to generate updated candidate joint parameter adjustment actions. The updated candidate joint parameter adjustment actions are then input into the double Q value network after parameter updates are completed to obtain the updated action value prediction results. Based on the updated action value prediction results and the preset physical consistency condition judgment results of the corresponding training sample segments, the parameters of the improved GTrXL policy network are updated. Repeatedly execute the dual-Q value network update and the improved GTrXL strategy network update until the preset iteration stop condition is met.
8. The method for optimizing robot joint parameters based on reinforcement learning according to claim 1, characterized in that, Step seven specifically includes: Based on the joint running state vector, end effector state, and load change state corresponding to the current control moment, generate the operating condition stage information corresponding to the current control moment; Input the joint running state vector and working condition stage information corresponding to the current control moment into the updated improved GTrXL strategy network, and output the original joint parameter adjustment action corresponding to the current control moment. The original joint parameter adjustment action is input into the fine-tuning dead zone and sliding smoothing unit. The absolute value of the original joint parameter adjustment amount corresponding to each joint is compared with the preset dead zone threshold of the corresponding joint. When the absolute value of the original joint parameter adjustment amount is less than or equal to the preset dead zone threshold of the corresponding joint, the original joint parameter adjustment amount corresponding to the current joint is set to zero. When the absolute value of the original joint parameter adjustment amount is greater than the preset dead zone threshold of the corresponding joint, the original joint parameter adjustment amount corresponding to the current joint remains unchanged, and the dead zone processed joint parameter adjustment amount is generated. The joint parameter adjustment amount after dead zone processing corresponding to the current control time of each joint is combined with the same joint parameter adjustment amount output by multiple consecutive control times in the previous sequence through a sliding window. The moving average processing is then performed according to the time order of the parameter adjustment amount of each control time in the window to generate the smooth joint parameter adjustment amount corresponding to the current control time. The smooth joint parameter adjustment amount corresponding to the current control time of each joint is combined with the current joint control parameter to generate the optimal joint parameter combination corresponding to the current control time.