Multi-degree-of-freedom mechanical arm trajectory tracking control method and system based on decomposition type Q learning

Abstract

The invention discloses a multi-degree-of-freedom mechanical arm trajectory tracking control method and system based on decomposition type Q learning. According to the method, a drifting dynamic matrix and an input dynamic matrix of the mechanical arm system do not need to be known in advance, online learning and optimization can be carried out by directly utilizing input and output data through the Q learning algorithm, and the dependence of a traditional adaptive dynamic programming (ADP) method on an accurate mathematical model is effectively overcome. According to the method, a Q function is innovatively decomposed into three independent components, and three sub neural network channels are used for approximation respectively. Compared with a direct approximation integral Q function, the distributed design has the advantages that the complexity of a single network is reduced, the fitting capability of the neural network to a nonlinear system is fully utilized, and the convergence speed and learning efficiency of the algorithm are improved. In conclusion, high-precision trajectory tracking can be achieved, the error curve convergence is rapid and stable, the speed and position change of each joint is smooth, no obvious jitter exists, and the method has good engineering application value.

Description

Technical Field

[0001] This invention belongs to the field of robot control technology, specifically relating to a multi-degree-of-freedom manipulator trajectory tracking control method and system based on decompositional Q-learning, and particularly to a multi-player Stackelberg hierarchical game optimal control method for nonlinear systems with unknown models. Background Technology

[0002] With the development of industrial intelligence, industrial robots (especially 6-DOF robotic arms) have been widely used in cutting-edge fields such as intelligent manufacturing and human-robot collaboration due to their high efficiency, flexibility, and environmental adaptability. However, in complex actual operations, how to achieve efficient game-theoretic decision-making and accurate trajectory tracking between robots and the environment or other work units is a current research focus in this field.

[0003] Existing Stackelberg game-based optimal control methods for nonlinear systems mostly rely on traditional adaptive dynamic programming (ADP) techniques. These existing techniques typically have the following significant limitations:

[0004] (1) Heavy reliance on accurate system dynamics models: Traditional methods usually assume that the system's drift matrix f(x) and input matrix g(x) are completely known. In actual physical environments, due to factors such as friction, wear, load changes and external disturbances, it is difficult to accurately obtain or fully grasp the system dynamics model data of the robotic arm.

[0005] (2) Control challenges when the model is unknown: When the system dynamics model is completely unknown or there is a large uncertainty, the existing model-based control algorithms often have difficulty converging or the control accuracy drops significantly, failing to meet the high-precision trajectory tracking requirements under complex working conditions.

[0006] (3) High computational complexity: When dealing with high-dimensional nonlinear systems (such as 6-DOF robotic arms), traditional Q-learning or ADP algorithms often face complex dimensional calculations, resulting in huge computational loads and making it difficult to meet the requirements of real-time control.

[0007] Therefore, for continuous-time nonlinear systems with completely unknown models, designing an algorithm that can effectively solve the multi-joint cooperative control problem without relying on system drift and input dynamic matrix is ​​a technical problem that urgently needs to be solved. Summary of the Invention

[0008] This invention aims to address the problem of existing technologies relying on precise mathematical models of systems, and provides an Actor-Critic neural network Q-learning control method for continuous-time nonlinear systems with completely unknown models.

[0009] The first aspect of this invention provides a multi-game Stackelberg control method for a multi-degree-of-freedom robotic arm based on decomposition Q-learning, comprising the following steps:

[0010] Step 1: Establish a nonlinear continuous-time system model of a multi-degree-of-freedom robotic arm, and model the control problem of the robotic arm as a multi-participant Stackelberg game problem; define the value function and Hamiltonian function of the system.

[0011] Step 2: Based on the Hamiltonian function and the optimal value function, define the Q function and decompose it into independent components.

[0012] Step 3: Construct an Actor-Critic parallel neural network approximator, using the Actor-Critic architecture to approximate the independent component functions.

[0013] Step 4: Using the principle of integral reinforcement learning, derive the optimal equation of the Bellman equation.

[0014] Step 5: Calculate the optimal control strategy in the Stackelberg game based on the trained neural network weights.

[0015] A second aspect of the present invention provides a multi-degree-of-freedom robotic arm trajectory tracking control system based on decompositional Q-learning, comprising:

[0016] The game model building module is used to obtain the state information of the robotic arm and build a multi-participant Stackelberg game model.

[0017] The Q-function decomposition and building module is used to construct Q-functions and decompose them into independent components.

[0018] The neural network approximation module includes the Actor neural network and the Critic neural network.

[0019] The weight update module is used to calculate the network error and dynamically update the neural network weights according to the adaptive law.

[0020] The control execution module is used to generate control signals based on the calculated optimal control strategy to drive the movement of each joint of the multi-degree-of-freedom robotic arm.

[0021] A third aspect of the present invention provides an electronic device for multi-degree-of-freedom robotic arm trajectory tracking control based on decompositional Q-learning, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-described multi-degree-of-freedom robotic arm trajectory tracking control method based on decompositional Q-learning when executing the program.

[0022] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program for executing the above-described multi-degree-of-freedom robotic arm trajectory tracking control method based on decompositional Q-learning.

[0023] The present invention produces the following beneficial effects:

[0024] (1) Completely eliminates the dependence on the precise dynamic model of the system, and has strong environmental adaptability:

[0025] Existing technologies typically require knowledge of the drift matrix and input dynamics matrix of the controlled object (such as a robotic arm) to design a controller. However, in real-world industrial scenarios, these parameters are often difficult to measure accurately or vary with the load. This invention introduces an improved Q-learning algorithm based on an Actor-Critic structure, eliminating the need for prior knowledge of the robotic arm system's drift dynamics matrix and input dynamics matrix. Optimal control can be achieved solely using the state data (angle, angular velocity) and control input data generated during online system operation. This model-free characteristic allows the invention to be directly applied to nonlinear systems with unknown parameters or unmodeled dynamics, effectively overcoming the dependence of traditional adaptive dynamic programming (ADP) methods on precise mathematical models and significantly reducing the complexity of system deployment and calibration costs.

[0026] (2) Reduce computational complexity and improve the real-time convergence speed of the control algorithm:

[0027] This invention innovatively decomposes the complex Q-function into three independent components: a state utility term, a linear coupling term, and a quadratic cost term. This structured decomposition strategy allows the optimal control law to be directly derived analytically, avoiding the tedious process of extensive iterative optimization required in the continuous action space of traditional reinforcement learning. Experimental data demonstrate that this method significantly accelerates the convergence speed of neural network weights while maintaining control accuracy (e.g., in the example, the error converges rapidly within the first 200 sampling periods), meeting the requirements for millisecond-level (10ms) real-time control.

[0028] (3) Possesses excellent transient response capability and robustness against interference:

[0029] Tracking experiments using pentagram trajectories (abrupt signals) and hourglass trajectories (complex coupled signals) verified that the present invention maintains extremely high tracking accuracy even under conditions of drastic changes in velocity direction or system dynamics. At trajectory inflection points, the system did not exhibit significant overshoot or divergence, and the error quickly returned to a steady state. This demonstrates that the multi-player hierarchical game mechanism proposed in this invention can effectively compensate for nonlinear interference, possesses strong robustness, and is suitable for complex and variable industrial operation paths.

[0030] (4) Ensure the system's uniformly eventually bounded (UUB) stability and smooth motion:

[0031] This invention, by constructing a stability constraint mechanism based on Lyapunov theory, theoretically guarantees the consistent eventual boundedness of all signals (including state variables and neural network weight estimation errors) within the closed-loop control system. Experimental results show that when executing complex trajectories, the angle and angular velocity change curves of each joint of the robotic arm are continuous and smooth, without high-frequency jitter or abnormal abrupt changes. This smooth control output not only ensures operational accuracy but also effectively avoids mechanical resonance, extends the service life of the actuator, and has significant engineering application value. Attached Figure Description

[0032] Figure 1 This is a hardware diagram of a test example in this application;

[0033] Figure 2 This is a neural network structure diagram of an embodiment of this application;

[0034] Figure 3 This is a flowchart illustrating the algorithm of an embodiment of this application;

[0035] Figure 4 This is a pentagram-shaped trajectory planning and tracking diagram for the test example in this application;

[0036] Figure 5 This is a diagram showing the tracking error of a pentagram-shaped trajectory in the test example of this application.

[0037] Figure 6 This is a diagram showing the angle variation of the pentagonal joint in the test example of this application;

[0038] Figure 7 This is a graph showing the angular velocity variation of the pentagonal joint in the test example of this application;

[0039] Figure 8 This is a test case diagram for hourglass-shaped trajectory planning and tracking in this application.

[0040] Figure 9 This is a graph showing the tracking error of the hourglass-shaped trajectory in the test example of this application.

[0041] Figure 10 This is a diagram showing the angle variation of the hourglass joint in the test example of this application;

[0042] Figure 11 This is a graph showing the change in angular velocity of the hourglass joint in the test example of this application. Detailed Implementation

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] This application discloses a trajectory tracking control method for a multi-degree-of-freedom robotic arm based on decompositional Q-learning. It primarily targets nonlinear systems with completely unknown dynamic models (such as multi-degree-of-freedom robots and other controlled objects). By introducing a Q-function decomposition strategy and constructing an Actor-Critic neural network architecture, it achieves hierarchical game-theoretic optimal control without requiring prior system information. The method includes the following steps:

[0045] Step 1: Constructing the System Dynamics Model and Hierarchical Game Framework: Establish a nonlinear continuous-time system model for the multi-degree-of-freedom robotic arm, modeling the control problem of the robotic arm as a multi-participant Stackelberg game problem. Define the system's value function as an integral cost function, including state error terms and control input terms. Specifically:

[0046] The controlled object is modeled as a nonlinear continuous-time system of the following form:

[0047]

[0048] in, Indicates the system status. For leadership strategies, This is a follower strategy. In this embodiment, the system dynamics matrix... and It is set to be completely unknown.

[0049] Define the value function of this system:

[0050] in, This refers to the coupling between leaders and followers.

[0051] Therefore, the Hamiltonian function of the system is then defined:

[0052]

[0053] Step 2, define the Q function and its structured decomposition: Based on the Hamiltonian function and the optimal value function, define the Q function and decompose it into independent components.

[0054] Traditional Q-learning methods typically use state and control input The data is input as a whole into a single neural network for approximation. However, in continuous-time control systems, this global approximation method has two serious drawbacks:

[0055] Defect 1: Difficulty in finding derivatives: In order to obtain the optimal control law It is necessary to understand the Q function regarding Find the partial derivative (i.e.) If a single complex nonlinear neural network is used, its differentiation process is extremely complex and often cannot be obtained. The analytical expression (explicit solution) necessitates time-consuming iterative optimization in each control cycle, severely impacting real-time performance.

[0056] Defect 2: Model Dependence: Traditional methods, when calculating optimal control, usually implicitly rely on the input dynamic matrix. The requirement is that, in order to achieve completely "model-free" control, a special Q-function structure must be constructed so that the expression of the optimal control law does not contain any system dynamic parameters.

[0057] Therefore, in order to solve for optimal control under unknown models, this application innovatively decomposes the Q function according to the order of the control input:

[0058]

[0059] Specifically, it consists of the following three independent logical components:

[0060] (1) State utility item

[0061] Definition: It is only related to the system state vector (such as the joint angle of the robotic arm) and angular velocity The relevant functions, and the current control input. Irrelevant.

[0062] Physical meaning: It represents the current state of the system. The inherent value or basic cost of the system in this state. Regardless of the action taken by the controller, the system itself incurs certain performance costs (such as the error cost of deviating from the target trajectory).

[0063] Purpose: In the process of differentiation, since this term does not contain... Its derivative is 0. Therefore, it does not directly participate in the calculation of the control law, but it is crucial for accurately assessing the total cost Q and is an indispensable "benchmark surface" in the Bellman equation balance equation.

[0064] (2) Linear coupling terms

[0065] Definition: It refers to the control input. A linear function, generally designed as follows: .in, It is a vector function that is only related to the state.

[0066] Physical meaning: It represents the control input With system status The interaction between them. It reflects the first-order effect (i.e., gradient direction) of applying a unit control input on the total value function in the current state.

[0067] Function: This is a key term in generating the control strategy. When differentiating, this term has a significant impact on... The derivative is It determines the direction and basic magnitude of the control force, and serves as a bridge connecting the state space and the action space.

[0068] (3) Secondary cost term

[0069] Definition: Regarding control input The quadratic form function.

[0070] Physical meaning: It represents the cost of control energy consumption. According to control theory, in order to prevent the controller from outputting infinite torque, control energy must be penalized.

[0071] Function: It ensures the convexity and uniqueness of the solution to the optimal control problem. It acts as a "damper" or "constraint," preventing the control variable from becoming too large.

[0072] This decomposition transforms complex nonlinear optimization problems into parameterized forms that can be learned online using neural networks.

[0073] Step 3: Construct an Actor-Critic parallel neural network approximator: Design the Actor-Critic neural network structure, employing an Actor-Critic (executor-evaluator) architecture to approximate the aforementioned independent component functions. Specifically:

[0074] like Figure 2 As shown. This neural network adopts a distributed design. To address the difficulty of directly differentiating the Q-function, the Critic network is split into three independent sub-network channels, i.e., three independent Critic neural networks are set up to approximate each other. , and ,include:

[0075] Channel 1 (input: State Only) is used to approximate state utility items. It is a multilayer perceptron, with the state as the input. The output is a scalar, representing the basic cost of the current system state.

[0076] Channel 2 (input:State+Action) is used to approximate the gradient vector in the linear coupling term. The input is the state. The output is a vector with the same dimension as the control input. This vector is then compared with the control input. Perform a dot product operation to form a linearly coupled term. .

[0077] Channel 3 (input: Action Only) is used to approximate the weight matrix R in the quadratic cost term (or can be directly set as a constant matrix). This part is related to the control input. Perform quadratic operations to form quadratic terms. .

[0078] During system operation, network weights are updated in real time according to the following logic:

[0079] (1) Error calculation: based on the current time and the previous moment Based on the state and evaluation values, construct the integral time difference (TD) error. Execution error of actor neural network approximator .

[0080] (2) Gradient Descent: The gradient descent method is used to minimize the sum of squared errors. Critic network weights The update rate is:

[0081]

[0082] in, The gain coefficient represents the convergence speed of the control system. , These are the basis functions for the critic neural network.

[0083] Actor Network Weights The update rate is:

[0084]

[0085] in, It is the learning rate of the Actor network. These are the basis functions for the actor neural network.

[0086] (3) Parameter setting: gain coefficient and learning rate The values ​​are set between [0.1, 0.5], and a normalization term is introduced to improve numerical stability.

[0087] Step four, optimal strategy iteration based on hierarchical game theory: Utilizing the principles of Integral Reinforcement Learning, derive the optimal equation for the Bellman Equation. This includes the following sub-steps:

[0088] Step 4-1: Define the Bellman error of the Critic network and the execution error of the Actor network;

[0089] Step 4-2: Design the update law of the Critic neural network weights using the gradient descent method to minimize the Bellman error squared term;

[0090] Step 4-3: Use gradient descent to design the update law of the Actor neural network weights to minimize the execution error and realize policy update.

[0091] Step 5: Calculate the leader's optimal decision based on the followers' optimal response function to the leader's strategy. Update the control law in real-time using an actor network. This causes the overall game system to tend towards Stackelberg equilibrium. It includes the following sub-steps:

[0092] Step 5-1: First, calculate the optimal response function of the follower to the leader's strategy;

[0093] Step 5-2: Calculate the leader's optimal control strategy based on the followers' responses;

[0094] Step 5-3: Apply the calculated control input to each joint of the multi-degree-of-freedom robotic arm to achieve precise tracking of the preset trajectory.

[0095] like Figure 3 As shown, the specific process for calculating the optimal control strategy in a Stackelberg game based on the trained neural network weights is as follows:

[0096] First, configure the parameters: define the state variables. To break the initial symmetry and ensure storage convergence, the sampling period is set to 10ms, and all initial weights of the Actor and Critic neural networks are set to random values ​​within the interval [-0.1, 0.1]. The angles and angular velocities of each joint of the robotic arm are read in real time using sensors.

[0097] After the data enters the system, it is vectorized and assembled to form the state vector at the current moment. The system calls the current Critic network weights and calculates the optimal torque control command in real time.

[0098] The calculated torque command is sent to the servo driver to drive the robotic arm. After one sampling cycle... Then, collect the new system status. .

[0099] Utilize the current state Next moment state In addition to control costs, an integral time-difference (TD) error is constructed. Then, using gradient descent, the weights of all neural networks are updated along the direction of error reduction.

[0100] Finally, the judgment logic is executed: it checks whether the rate of change of weight updates is less than a preset threshold (i.e., reaching a UUB stable state) or whether the preset trajectory has been completed. If not, a new control strategy is generated through loop return, and the next control cycle begins; if satisfied, the current optimal weights are maintained and the task is terminated.

[0101] This application also provides a multi-degree-of-freedom robotic arm trajectory tracking control system based on decompositional Q-learning, including:

[0102] The game model building module is used to obtain the state information of the robotic arm and build a multi-participant Stackelberg game model.

[0103] The Q-function decomposition and building module is used to construct Q-functions and decompose them into independent components.

[0104] The neural network approximation module includes an Actor neural network and a Critic neural network;

[0105] The weight update module is used to calculate the network error and dynamically update the neural network weights according to the adaptive law.

[0106] The control execution module is used to generate control signals based on the calculated optimal control strategy to drive the movement of each joint of the multi-degree-of-freedom robotic arm.

[0107] This application also discloses an electronic device for multi-degree-of-freedom robotic arm trajectory tracking control based on decompositional Q-learning, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a multi-degree-of-freedom robotic arm trajectory tracking control method based on decompositional Q-learning. The memory may include main memory, such as high-speed random access memory, or it may also include non-volatile memory, such as at least one disk storage device. The processor, network interface, and memory are interconnected via an internal bus, which can be an industry-standard architecture bus, a peripheral component interconnection standard bus, an extended industry-standard architecture bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory is used to store the program; specifically, the program may include program code, which includes computer operation instructions. The memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0108] This application also discloses a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps of a multi-degree-of-freedom robotic arm trajectory tracking control method based on decompositional Q-learning. Specifically, the computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. The volatile memory may include random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include read-only memory (ROM), hard disk, flash memory, optical disk, magnetic disk, etc.

[0109] To verify the effectiveness of the above method, this test case constructs a physical experimental system for algorithm verification, such as... Figure 1 The six-degree-of-freedom robotic arm experimental platform shown mainly includes the following core hardware components:

[0110] (1) Master Computer: As the core computing unit of the algorithm, it is equipped with a high-performance processor. The master computer is responsible for running the improved Q-learning algorithm based on the Actor-Critic structure, receiving the state feedback of the robotic arm in real time, solving the HJB equation, and outputting the optimal control strategy. Note: In practical applications, the host computer can be replaced by an industrial control computer, an embedded edge computing module, or a cloud server.

[0111] (2) Main Controller: An embedded real-time controller is used to communicate with the host computer and driver via a high-speed fieldbus (such as EtherCAT or CAN bus). It is responsible for the low-level parsing of instructions, time synchronization and signal distribution.

[0112] (3) Control Cabinet: It integrates multi-axis servo drivers and power management modules, which are responsible for converting logic control signals into high-power current signals to drive motors and provide overcurrent and overvoltage protection.

[0113] (4) Controlled Object (Mechanical Arm): In this embodiment, a six-degree-of-freedom serial industrial robotic arm is selected. Each of its six joints is equipped with a high-precision absolute encoder for real-time acquisition of joint angle positions. and angular velocity .

[0114] Parameter definition: The system state vector is defined as follows: In this embodiment, the sampling period of the control loop is set to 1ms~10ms (preferably 10ms) to meet the requirements of high dynamic real-time control.

[0115] Test Example 1: Pentagram Trajectory Tracking Experiment (Transient Response Performance Verification): By tracking the trajectory of a "pentagram" with abrupt changes in geometric features, the transient response capability of the system under drastically changing operating conditions is verified.

[0116] 1. Experimental Preparation and Setup

[0117] Trajectory planning: Generate a standard pentagram trajectory within the robotic arm's workspace. This trajectory contains five sharp vertices, at which the direction of the desired velocity undergoes discontinuous step changes.

[0118] Initial conditions: Place the end effector of the robotic arm at any position other than the starting point of the trajectory to verify the algorithm's ability to automatically enter the trajectory.

[0119] 2. Detailed Experimental Procedure

[0120] Step 1, Startup Phase: Initialize the neural network weights to random values ​​in the range [-0.1, 0.1]. Start the control loop, and the robotic arm, driven by the Actor network, moves from its initial position toward the target trajectory.

[0121] Step two, tracking phase: The robotic arm moves along the pentagram path. Upon reaching each vertex, the algorithm needs to adjust the torque of each joint within milliseconds to overcome overshoot caused by inertia.

[0122] Step 3, Data Acquisition: Real-time recording of the end-effector's three-dimensional coordinates (x, y, z), joint angles, and angular velocities.

[0123] 3. Analysis of Experimental Results

[0124] Trajectory overlap analysis: Figure 4The experiment demonstrates the tracking performance of a pentagram trajectory by the robotic arm's end effector in three-dimensional space. The red dashed line represents the preset desired trajectory, while the blue solid line represents the actual trajectory driven by the method described in this application. The experimental results clearly show a high degree of spatial overlap between the two. Particularly noteworthy is that at the five geometric vertices of the pentagram trajectory (i.e., the turning points where the velocity direction abruptly changes), the actual trajectory of the robotic arm exhibits no significant overshoot or oscillatory divergence, smoothly completing the path switching. This fully demonstrates that the Q-function decomposition strategy proposed in this application possesses excellent transient response capability and dynamic tracking performance for nonlinear abrupt signals without requiring a known system dynamics model.

[0125] Error convergence analysis: Figure 5 The error curve for tracking the pentagram trajectory is shown, with the horizontal axis representing the number of sampling points and the vertical axis representing the position error (mm). The data indicates that in the initial stage of the experiment (approximately the first 200 sampling periods), there is a certain initial error due to the randomness of the initial position; however, under the online learning adjustment of the Actor-Critic neural network, the error curve decreases exponentially and rapidly converges to a steady-state region close to zero (millimeter-level accuracy). In subsequent runs, even when passing through the trajectory apex, error fluctuations are limited to a very small range, verifying that the method proposed in this application has extremely fast convergence speed and high-precision steady-state maintenance capability.

[0126] Joint motion stability analysis: combined with Figure 6 (Joint angle changes) and Figure 7 (Joint angular velocity changes) It can be seen that during execution, the angle change curves of each joint of the robotic arm are continuous and smooth, and the amplitude of the angular velocity change is always controlled within the rated safety range of the motor. This indicates that the torque command generated by the control strategy of this application is smooth, effectively avoiding control quantity jitter, which is beneficial to protecting the robotic arm hardware and extending its service life.

[0127] Test Example 2: Hourglass-shaped trajectory tracking experiment (robustness and steady-state accuracy verification): By tracking an "hourglass-shaped" trajectory with continuous curvature and intersections, the long-term stability and robustness of the system are verified.

[0128] 1. Experimental Preparation and Setup

[0129] Trajectory planning: Generate a closed, three-dimensional hourglass-shaped curve containing smooth arc segments and a central cross-point. The system dynamics at the cross-point are complex and prone to singularities or oscillations.

[0130] Comparison with benchmark: Calculate the Euclidean distance between the actual trajectory and the expected trajectory as the real-time error.

[0131] 2. Detailed Experimental Procedure

[0132] Step 1, continuous operation: The robotic arm performs continuous cyclical motion on the hourglass trajectory for multiple cycles to test the parameter drift of the algorithm over a long period of time.

[0133] Step 2, Crossing the intersection: Focus on observing the movement of the robotic arm when it passes through the intersection of the hourglass center.

[0134] Step 3, Disturbance Suppression: Relying on the online learning capability of neural networks, it automatically compensates for disturbances caused by joint friction and unmodeled dynamics.

[0135] 3. Analysis of Experimental Results

[0136] Full path tracking analysis: Figure 8 The results demonstrate the tracking of an hourglass-shaped trajectory with continuous curvature variations and a central cross-point. Experiments show that the actual trajectory (solid blue line) can completely and accurately reproduce the desired hourglass shape (dashed red line). The system maintains high tracking accuracy regardless of whether it is in the smooth outer curve segment or at the dynamically complex central cross-point.

[0137] robustness analysis against interference: such as Figure 9 As shown, the error curve remains stable throughout the entire process after initial convergence. Particularly in the intersection region of the hourglass trajectory (corresponding to the small fluctuations in the error curve in the figure), traditional algorithms are prone to oscillations due to strong nonlinear coupling interference at this point; however, using the method of this application, the error only exhibits small pulse-like fluctuations and then quickly returns to a steady state. This further demonstrates that the multi-neural network collaborative approximation strategy of this application can effectively compensate for unmodeled dynamics and external disturbances, exhibiting extremely strong system robustness.

[0138] Dynamic response analysis: Figure 10 and Figure 11 The changes in angle and angular velocity of each joint under the hourglass trajectory were demonstrated. Each joint exhibited a regular sinusoidal periodic motion with smooth, glitch-free waveforms, and good phase coordination between the joints, verifying the decoupling and coordination capabilities of the multi-game hierarchical control mechanism in handling multi-input multi-output (MIMO) coupled systems.

[0139] This invention is not only applicable to trajectory tracking control of multi-degree-of-freedom robotic arms, but can also be extended to optimization control scenarios of various continuous-time nonlinear systems with completely unknown dynamic models.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A trajectory tracking control method for a multi-degree-of-freedom robot arm based on decomposition Q-learning, characterized by, Includes the following steps: Step 1: Establish a nonlinear continuous-time system model for the multi-degree-of-freedom robotic arm, and model the control problem of the robotic arm as a multi-participant Stackelberg game problem; define the value function and Hamiltonian function of the system; Step 2: Based on the Hamiltonian function and the optimal value function, define the Q function and decompose it into independent components; Step 3: Construct an Actor-Critic parallel neural network approximator, using the Actor-Critic architecture to approximate the independent component functions; Step 4: Using the principle of integral reinforcement learning, derive the optimal equation of the Bellman equation; Step 5: Calculate the optimal control strategy in the Stackelberg game based on the trained neural network weights.

2. The method according to claim 1, characterized in that, In step 2, the independent components include: State utility items are only related to the system state and are used to reflect the inherent value or basic cost of the system in its current state. Linear coupling terms, coupled with the system state and control input, are used to characterize the first-order effect of applying a unit control input on the total value function under the current state; The secondary cost term is only related to the control input and is used to reflect the cost of control energy consumption.

3. The method according to claim 2, characterized in that, The Actor-Critic parallel neural network approximator adopts a distributed design, setting up independent Critic neural network sub-network channels to approximate the independent components respectively.

4. The method according to claim 3, characterized in that, The sub-network channels include: Channel 1 is used to approximate the state utility term; its input is the state. The output is a scalar, representing the fundamental cost of the current system state; Channel 2 is used to approximate the gradient vector in the linear coupling term; its input is the state. The output is a vector with the same dimension as the control input, which is then compared with the control input. Perform a dot product operation to form linearly coupled terms; Channel 3 is used to approximate the weight matrix R in the quadratic cost term, thereby relating it to the control input. Perform quadratic operations to form a quadratic cost term.

5. The method according to claim 3 or 4, characterized in that, The Actor-Critic parallel neural network approximator updates the network weights in real time according to the following logic: Based on the current state and evaluation value of the previous time step, an integral temporal difference error and the execution error of the actor neural network are constructed. The gradient descent method is used to minimize the sum of squared errors, using the gain coefficient. The update rate for constructing the Critic network weights is achieved using the learning rate. Construct the update rate of the Actor network weights; right and Values ​​are set, and a normalization term is introduced to improve numerical stability.

6. The method according to claim 1, characterized in that, Step 5 includes the following sub-steps: Step 5-1: Calculate the optimal response function of the followers to the leader's strategy; Step 5-2: Calculate the leader's optimal control strategy based on the followers' responses; Step 5-3: Apply the calculated control input to each joint of the multi-degree-of-freedom robotic arm to achieve tracking of the preset trajectory.

7. A multi-degree-of-freedom robotic arm trajectory tracking control system based on decompositional Q-learning, characterized in that, include: The game model building module is used to obtain the state information of the robotic arm and build a multi-participant Stackelberg game model. The Q-function decomposition and building module is used to construct Q-functions and decompose them into independent components. The neural network approximation module includes an Actor neural network and a Critic neural network; The weight update module is used to calculate the network error and dynamically update the neural network weights according to the adaptive law. The control execution module is used to generate control signals based on the calculated optimal control strategy to drive the movement of each joint of the multi-degree-of-freedom robotic arm.

8. An electronic device for trajectory tracking control of a multi-degree-of-freedom robotic arm based on decompositional Q-learning, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the multi-degree-of-freedom robotic arm trajectory tracking control method based on decompositional Q-learning as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the multi-degree-of-freedom robotic arm trajectory tracking control method based on decomposition Q-learning as described in any one of claims 1-6.

Images