Variable load mobile manipulator trajectory tracking method based on reinforcement learning and model prediction control

Abstract

The invention discloses a variable load mobile manipulator trajectory tracking method based on reinforcement learning and model prediction control, which comprises the following steps: firstly, acquiring a mobile manipulator system model established based on a Lagrange equation, and deducing a kinetic equation through system kinetic energy, potential energy and a generalized force formula; a model predictive control (MPC) module predicts a future state by using a discrete state equation based on the model, and obtains control input by optimizing a cost function. The reinforcement learning (RL) module defines a specific state space, an action space and a reward function, and adopts a Q-learning algorithm to train an intelligent agent. In the cooperative control stage, the weight coefficient is dynamically adjusted according to the system state, and the control input of the MPC and the control input of the RL are fused. According to the method, the trajectory tracking problem of the variable-load mobile mechanical arm is effectively solved, the trajectory tracking precision is remarkably improved, the adaptability and robustness of the system are enhanced, and the method has wide application prospects in the fields of industrial manufacturing, logistics storage and the like.

Description

Technical Field

[0001] The present invention relates to intelligent robot control technology, and in particular to a variable load mobile robot arm trajectory tracking method based on reinforcement learning and model predictive control. Background Art

[0002] As an intelligent robot system that combines the mobility of a mobile platform with the flexibility of a robot arm, the mobile robot arm plays an increasingly important role in modern production and life. In industrial manufacturing scenarios, it can flexibly shuttle through the production line, carry parts of different specifications and weights, and achieve precise assembly; in the field of logistics and warehousing, it can complete the rapid sorting and handling of goods, greatly improving the efficiency of warehousing operations; in rescue and disaster relief situations, it can go deep into dangerous areas to perform key tasks such as material transportation, search and rescue, etc.

[0003] However, in actual operation, mobile robotic arms often face a working environment with variable loads, which places extremely high demands on their control systems. Traditional control methods, such as PID controllers and fuzzy controllers, have many limitations when dealing with such complex situations. PID controllers rely on precise system model parameters. For nonlinear and time-varying mobile robotic arm systems, it is difficult to adjust parameters in real time to adapt to the impact of variable loads, resulting in a significant reduction in control accuracy. Although fuzzy controllers can handle uncertainty to a certain extent, they lack in-depth analysis of the dynamic characteristics of the system. When the load changes rapidly, they cannot make control decisions in a timely and accurate manner, which challenges the stability of the system.

[0004] In recent years, the control strategy combining reinforcement learning with model predictive control (MPC) has become a research hotspot. Many researchers have conducted extensive explorations around this. For example, some studies have effectively improved the motion accuracy of mobile robotic arms under variable load conditions by introducing a new calibration device and utilizing its unique structure and sensor configuration, but this method may increase the complexity and cost of the system. Other studies have combined BP neural networks with other control methods to approximate the complex nonlinear relationships of the system through the powerful learning ability of neural networks, thereby improving control accuracy. However, the training process of neural networks is usually time-consuming and prone to falling into local optimality. The real-time performance and reliability in practical applications need to be further improved. Other studies have designed neural network control methods to identify variable load conditions at the end of space robots, but their application scenarios are relatively specific. For general mobile robotic arm systems, a lot of adjustments and optimizations may be required. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to address the deficiencies in the prior art and to provide a variable load mobile robotic arm trajectory tracking method based on reinforcement learning and model predictive control. By giving full play to the synergistic advantages of reinforcement learning and model predictive control, high-precision trajectory tracking of the mobile robotic arm under variable load working conditions can be achieved, thereby improving the operating efficiency and stability of the system.

[0006] Technical solution: A variable load mobile robot arm trajectory tracking method based on reinforcement learning and model predictive control of the present invention comprises the following steps:

[0007] Step 1: Obtain the established mobile manipulator system model, which is established based on the Lagrange equation, including the calculation of system kinetic energy, potential energy and generalized force, as well as the dynamic equation;

[0008] The system kinetic energy formula is: ;

[0009] In the above formula, is the chassis mass of the mobile robot, is the chassis center of mass velocity of the mobile robot, For the mobile robot The moment of inertia of the joints, For the mobile robot The joint angular velocity, For the mobile robot The quality of the joints, For the mobile robot The velocity of the center of mass of each joint;

[0010] The system potential energy formula is: ;

[0011] In the above formula, is the acceleration due to gravity, For the The height of the center of mass of each joint relative to the reference plane;

[0012] Generalized forces include friction Generalized forces (including but not limited to friction with the ground) and the generalized force of gravity ;

[0013] Then the kinetic equation is obtained: ;

[0014] In the above formula, is the mass matrix, is the Coriolis force and centripetal force matrix, is the gravity vector;

[0015] Step 2: Based on the mobile robot system model in step 1, a prediction model for model predictive control is constructed. Suppose the system discrete state equation at the k+1 sampling time is: ;

[0016] Set the prediction time domain to , at each sampling moment According to the current status Predicting the future The system state at each moment is defined as the cost function , solve the cost function by quadratic programming algorithm Minimum control input sequence And the first control input at the current moment Applied to the system; It refers to the control input, that is, the speed control amount and the driving torque of the robot joint; the function of the prediction model is to predict the state, optimize the control input, and provide a basis for decision-making;

[0017] is the system state and the reference trajectory at time The error vector, , , is the weight matrix;

[0018] Step 3: Determine the state space for reinforcement learning , action space , the reward function , using Q-learning algorithm for training, the Q value function update formula is:

[0019] ;

[0020] Among them, is the current position of the mobile robot, is the robot arm joint angle, is the relative position between the end effector and the target object, is the system load information, System speed information; left and right wheel speed , and They are the left and right wheel speeds, and the control inputs of the robot arm joints (including joint angles, joint angular velocities, and compensation information); , , , , is the weight coefficient adjusted according to the system performance requirements, is the maximum acceptable distance of the target object, is the speed error, is the angle error, is the capsize detection variable;

[0021] Step 4: Input the control input of the model predictive control and reinforcement learning to select actions Fusion is performed using the fusion formula Generate final control input ;

[0022] in is the weight coefficient, and its value range is , and dynamically adjust according to the current state of the system and task requirements.

[0023] Furthermore, the system discrete state equation in step 2 is Current status Contains the position coordinates of the car , heading angle 、The angles of each joint of the robot arm and its speed information, Including the speed control of the car's driving wheels and the driving torque of the robot arm joints, is the system state and the reference trajectory at time The error vector, , , is the corresponding weight matrix.

[0024] Furthermore, the specific training method of step 3 is:

[0025] Step 301, initializing a Q value table: At the beginning of training, a Q value table is initialized, and the Q value table is used to store the Q values ​​of all state-action pairs; initially, all Q values ​​can be set to 0 or a random small value;

[0026] For a discrete state space and action space , the Q value table can be represented as a two-dimensional array ,in , ;

[0027] At the same time, set the hyperparameters: learning rate and discount factor ; Learning rate Controls the step size of each update of the Q value, and the value range is usually between 0 and 1. The learning rate Determines the degree of influence of new experience information on the old Q value; discount factor Used to balance the importance of current rewards and future rewards, the value range is also between 0 and 1. When it is close to 0, the agent pays more attention to the current reward; when When it is close to 1, the agent pays more attention to future rewards;

[0028] Step 302: At the beginning of each training round, the agent is in the initial state of the environment ; In each state, the agent - Greedy strategy chooses an action to execute; - Greedy strategy with probability Choose a random action (for exploring new actions) to Probabilistically select the action with the largest current Q value (used to leverage existing experience); as training progresses, The value of usually decreases gradually, allowing the agent to explore more in the early stages of training and make more use of existing experience in the later stages of training;

[0029] Step 303: The agent selects an action in the state After that, the action Applied to the environment, the environment moves to the next state based on the result of the action , and returns an immediate reward ; This reward is the feedback from the environment to the agent's action, which reflects the quality of the action;

[0030] Step 304: According to the Q-learning update rule, it is necessary to calculate the TD (Temporal Difference) error, that is, the difference between the Q value of the current state-action pair and the target Q value;

[0031] The target Q value is calculated based on the Bellman optimal equation. The formula is:

[0032] ;

[0033] in, Is to take action in the current state The instant rewards you receive, is the discount factor, It is the maximum Q value of all possible actions in the next state;

[0034] Use the TD error to update the Q value, which is updated as: Right now ;

[0035] in, is the learning rate, which controls the update step size of the Q value; by continuously updating the Q value, the agent gradually learns the optimal value of each state-action pair;

[0036] Step 305: Update the current state to the next state, i.e. , and then determine whether the current state is a terminal state; if it is a terminal state, the training round ends and enters the next training round; if it is not a terminal state, return to step 302 and continue to select actions and execute them;

[0037] Repeat the above steps 302 to 305 continuously for multiple training rounds until the Q value converges or reaches a preset number of training times;

[0038] As training progresses, the agent gradually learns the optimal action strategy, so that it can choose the action with the maximum long-term cumulative reward in each state.

[0039] In step 4 It is dynamically adjusted according to the current state of the system and task requirements, and different values ​​will be taken under different working conditions. When the system is in a stable state and the load changes little, increase The value of makes the final control input more dependent on the precise planning of model predictive control; when the system state changes drastically and the load changes greatly, reduce The value of can give full play to the adaptive ability of reinforcement learning; specific judgment conditions and adjustment rules can be given, such as dynamically adjusting according to indicators such as the magnitude of load changes and the size of system errors. The value of .

[0040] Beneficial effects: The present invention achieves high-precision trajectory tracking under variable load working conditions, effectively enhancing the system's operating efficiency and stability. Compared with traditional control methods such as PD and fuzzy controllers, the controller based on RL-MPC of the present invention performs better in terms of control accuracy, system stability and real-time performance. After a large number of simulations and actual tests, it has been verified that in the same variable load task, the trajectory tracking error of the method of the present invention is significantly smaller, the system can adapt to load changes more quickly and maintain stable operation, significantly improving the working ability of the mobile robot arm in complex environments, and providing strong technical support for its wide application in multiple fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of a specific trajectory tracking process of the present invention;

[0042] Figure 2 is the error of the mobile robot arm in the x direction of the embodiment;

[0043] Figure 3 is the error of the mobile robot arm in the y direction of the embodiment;

[0044] Figure 4 The navigation angle of the mobile robot arm of the embodiment is error;

[0045] Figure 5 The mobile robot speed of the embodiment error;

[0046] Figure 6 is the angular velocity of the mobile robot arm of the embodiment error;

[0047] Figure 7 is the position error of the mobile robot arm of the embodiment;

[0048] Figure 8 is the joint angle error of the mobile robot arm of the embodiment. DETAILED DESCRIPTION

[0049] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0050] like Figure 1 As shown, the present invention provides a variable load mobile robot arm trajectory tracking method based on reinforcement learning and model predictive control, comprising the following steps:

[0051] Step 1: Obtain the established mobile manipulator system model, which is established based on the Lagrange equation, including the calculation of system kinetic energy, potential energy and generalized force, as well as the dynamic equation;

[0052] The system kinetic energy formula is: ;

[0053] In the above formula, is the chassis mass of the mobile robot, is the chassis center of mass velocity of the mobile robot, For the mobile robot The moment of inertia of the joints, For the mobile robot The joint angular velocity, For the mobile robot The quality of the joints, For the mobile robot The velocity of the center of mass of each joint;

[0054] The system potential energy formula is: ;

[0055] In the above formula, is the acceleration due to gravity, For the The height of the center of mass of each joint relative to the reference plane;

[0056] Generalized force includes frictional force and the generalized force of gravity ;

[0057] Then the kinetic equation is obtained: ;

[0058] In the above formula, is the mass matrix, is the Coriolis force and centripetal force matrix, is the gravity vector;

[0059] Step 2: Based on the mobile robot system model in step 1, a prediction model for model predictive control is constructed. Suppose the system discrete state equation at the k+1 sampling time is: ;

[0060] Set the prediction time domain to , at each sampling moment According to the current status Predicting the future The system state at each moment is defined as the cost function , solve the cost function by quadratic programming algorithm Minimum control input sequence And the first control input at the current moment Applied to the system;

[0061] is the system state and the reference trajectory at time The error vector, , , is the weight matrix;

[0062] Step 3: Determine the state space for reinforcement learning , action space , the reward function , using Q-learning algorithm for training, the Q value function update formula is:

[0063] ;

[0064] Among them, is the current position of the mobile robot, is the robot arm joint angle, is the relative position between the end effector and the target object, is the system load information, System speed information; left and right wheel speed , and They are the left and right wheel speeds and the control inputs of the robot arm joints;

[0065] , , , , is the corresponding weight coefficient, is the maximum acceptable distance of the target object, is the speed error, is the angle error, is the capsize detection variable;

[0066] Step 4: Input the control input of the model predictive control and reinforcement learning to select actions Fusion is performed using the fusion formula Generate final control input ; is the weight coefficient.

[0067] The discrete state equation of the system in step 2 Current status Contains the position coordinates of the car , heading angle 、The angles of each joint of the robot arm and its speed information, Including the speed control of the car's driving wheels and the driving torque of the robot arm joints, is the system state and the reference trajectory at time The error vector, , , is the corresponding weight matrix.

[0068] The specific training method for step 3 is:

[0069] Step 301, initialize the Q value table: At the beginning of training, initialize a Q value table for the discrete state space and action space , the Q value table is represented as a two-dimensional array ,in , ;

[0070] At the same time, set the hyperparameters: learning rate and discount factor ;

[0071] Step 302: At the beginning of each training round, the agent is in the initial state of the environment ; In each state, the agent - Greedy strategy chooses an action to execute;

[0072] Step 303: The agent selects an action in the state After that, the action Applied to the environment, the environment moves to the next state based on the result of the action , and returns an immediate reward ;

[0073] Step 304: Calculate the TD error, i.e., the difference between the Q value of the current state-action pair and the target Q value, according to the Q-learning update rule;

[0074] The target Q value is calculated based on the Bellman optimal equation. The calculation formula is:

[0075] ;

[0076] in, Is to take action in the current state The instant rewards you receive, is the discount factor, It is the maximum Q value of all possible actions in the next state;

[0077] Use the TD error to update the Q value, which is updated as: Right now ;

[0078] in, is the learning rate, which controls the update step size of the Q value. By continuously updating the Q value, the agent gradually learns the optimal value of each state-action pair.

[0079] Step 305: Update the current state to the next state, i.e. , and then determine whether the current state is a terminal state; if it is a terminal state, the training round ends and enters the next training round; if it is not a terminal state, return to step 302 and continue to select actions and execute them;

[0080] The above steps 302 to 305 are continuously repeated to perform multiple training rounds until the Q value converges or reaches a preset number of training times.

[0081] The trajectory tracking method of the mobile robot arm in this embodiment includes the following steps:

[0082] Step 1: Build a mobile robotic arm system.

[0083] The mobile robot arm system of this embodiment includes a mobile platform and a robot arm. The mobile platform is a non-holonomically constrained type, has two degrees of freedom and is rear-wheel driven. A two-finger gripper is installed at the front end of the robot arm for grabbing heavy objects. The platform weighs 50kg, the wheel radius is 0.5m, and the wheelbase is 1m. The connecting rod masses are 3.7kg, 8.4kg, 2.3kg, 1.2kg, 1.2kg, and 0.2kg, respectively, and the connecting rod lengths are 0.089m, 0.425m, 0.392m, 0.109m, 0.094m, and 0.082m, respectively.

[0084] The mobile robot system of this embodiment is universal, and the proposed analysis and control method is not limited to a specific robot structure. When there is no external force sensor and facing variable load, the system stability is easily challenged. The designed working scenario is that the mobile robot grabs cubes of 200N, 300N, and 400N objects in the predetermined running trajectory.

[0085] Step 2: Establish a mobile robot system model based on the Lagrange equation. The details are as follows:

[0086] Step 201: Calculation of system kinetic energy

[0087]

[0088] in It's the chassis quality. is the chassis center of mass velocity, For the The moment of inertia of the joints, For the The joint angular velocity, For the The quality of the joints, For the The velocity of the center of mass of each joint;

[0089] Step 202: Calculate system potential energy

[0090]

[0091] in is the acceleration due to gravity, For the The height of the center of mass of each joint relative to the reference plane;

[0092] Step 203: Generalized force calculation: Generalized forces include control input force, friction force, gravity and other generalized forces.

[0093] The generalized force of friction is ; Gravity stress is: ;

[0094] Step 204: The dynamic equation of the mobile robot can be obtained by the Lagrange equation:

[0095]

[0096] in is the mass matrix, is the Coriolis force and centripetal force matrix, is the gravity vector.

[0097] Step 3: Construct a prediction model for model predictive control based on the acquired precise dynamics model of the mobile robot. The specific process is as follows:

[0098] Step 301: Assume the discrete state equation of the system is:

[0099]

[0100] in Including car position , heading angle 、The angles of each joint of the robot arm and its velocity information, fully describing the system at time Status; It includes the speed control of the car driving wheels and the driving torque of the robot arm joints, which are the control inputs of the system.

[0101] Step 302: Set the prediction time domain to , at each sampling moment , according to the current state Predicting the future The system state at the next moment is calculated by the current state and the control input at the previous moment. , and repeat this process to predict the subsequent state.

[0102] Step 303: Define the cost function:

[0103]

[0104] in is the system state and the reference trajectory at time The error vector, , , is the weight matrix, which is used to adjust the importance of tracking error and control input in the cost function. Minimum control input sequence , the first control input Applied to the system to optimize the future state of the system.

[0105] Step 4: The reinforcement learning design enables the system to leverage the intelligent decision-making advantages of reinforcement learning to achieve efficient control of the mobile robotic arm under variable load environments.

[0106] Step 401: Determination of state space, action space and reward function:

[0107] State Space: in is the current position of the mobile robot, are all robot joint angles, is the relative position between the end effector and the target object, is the system load information, It is the system speed information.

[0108] Action Space: Used to adjust the speed of the trolley driving wheel and And the robot joint control input .

[0109] Reward function: ,in , , , , is the weight coefficient adjusted according to the system performance requirements, is the maximum acceptable distance of the target object, is the speed error, is the angle error, is the capsize detection variable.

[0110] Step 402: Q-learning algorithm training: Initialize Q value table: At the beginning of training, a Q value table needs to be initialized, which is used to store the Q values ​​of all state-action pairs. Usually, all Q values ​​can be set to 0 or a random small value at the beginning. For a discrete state space and action space , the Q value table can be represented as a two-dimensional array ,in , . You also need to set some hyperparameters, such as the learning rate and discount factor Learning rate Controls the step size of each update of the Q value. The value range is usually between 0 and 1. It determines the influence of new experience information on the old Q value. Used to balance the importance of current rewards and future rewards, the value range is also between 0 and 1. When it is close to 0, the agent pays more attention to the current reward; when When it is close to 1, the agent pays more attention to future rewards.

[0111] Step 403: At the beginning of each training round, the agent is in the initial state of the environment In each state, the agent needs to choose an action To execute. The action selection strategy usually adopts - Greedy strategy, which uses probability Choose a random action (for exploring new actions) to The probability of selecting the action with the largest current Q value (used to take advantage of existing experience). As training progresses, The value of usually decreases gradually, allowing the agent to explore more in the early stages of training and make more use of existing experience in the later stages of training.

[0112] Step 404: The agent selects an action in the state After that, the action is applied to the environment, and the environment will move to the next state according to the execution result of the action. , and returns an immediate reward ; This reward is the feedback from the environment to the agent's action, which reflects the quality of the action.

[0113] Step 405: According to the Q-learning update rule, it is necessary to calculate the TD (Temporal Difference) error, that is, the difference between the Q value of the current state-action pair and the target Q value. The calculation of the target Q value is based on the Bellman optimal equation, and the formula is: ;

[0114] in, Is to take action in the current state The instant rewards you receive, is the discount factor, is the maximum Q value of all possible actions in the next state. Use TD error to update the Q value, which is updated as: Right now in, is the learning rate, which controls the update step size of the Q value. By continuously updating the Q value, the agent gradually learns the optimal value of each state-action pair.

[0115] Step 406: Update the current state to the next state, i.e. . Determine whether the current state is a terminal state. If it is a terminal state, the training round ends and enters the next training round; if it is not a terminal state, return to step 402 and continue to select actions and execute them.

[0116] Step 407: Repeat the above steps 402 to 405 for multiple training rounds until the Q value converges or reaches the preset number of training times. As the training progresses, the agent gradually learns the optimal action strategy, so that it can select the action with the maximum long-term cumulative reward in each state.

[0117] Step 5: Collaborative control strategy

[0118] Step 501: Input the control input of the model predictive control and reinforcement learning to select actions Fusion is performed using the fusion formula Generate final control input ,in is the weight coefficient, and its value range is .

[0119] Step 502: Dynamic weight adjustment: It is dynamically adjusted according to the current state of the system and task requirements, and will have different values ​​under different working conditions. The details are as follows:

[0120] When the system is in a stable state and the load changes slightly, increase The value of makes the final control input more dependent on the precise planning of model predictive control; when the system state changes drastically and the load changes greatly, reduce The value of can give full play to the adaptive ability of reinforcement learning. Specific judgment conditions and adjustment rules can be given, such as dynamically adjusting according to indicators such as the magnitude of load changes and the size of system errors. The value of .

[0121] like Figure 2-Figure 8 As shown in the figure, in order to verify the effectiveness and feasibility of the variable load mobile robot trajectory tracking method based on reinforcement learning and model predictive control, the control method and the traditional Fuzzy and PID methods are compared and simulated on Matlab. All experiments are carried out on a 12th Gen Intel(R) Core(TM) i7-12700H computer with 16G memory.

[0122] from Figure 2 , Figure 3 and Figure 4 It can be seen that the present invention has significant advantages over the other two methods, and its error amplitude is smaller. During the operation of the robot, this method can make adjustments more quickly, compensate for the error, and thus promote the system to reach a stable state.

[0123] analyze Figure 5 and Figure 6It can be seen that the three control methods all caused vibration in the initial stage, but in the subsequent stage, the speed performance of the three methods tended to be stable. It is worth pointing out that the present invention can achieve a stable state at a faster speed and show stronger robustness.

[0124] Figure 7 and Figure 8 The motion error of the entire system, including the mobile platform and the robot arm joints, is accurately portrayed during the entire motion process. Obviously, under the action of the three control methods, the errors fluctuate within a small and controllable range. However, judging by the comprehensive evaluation of various indicators, the present invention presents a better control performance as a whole, and effectively provides effective support for the mobile robot arm system to maintain stability under variable load conditions.

[0125] The present invention organically combines the intelligent decision-making ability of reinforcement learning with the precise prediction and optimization ability of model predictive control. Reinforcement learning continuously explores and optimizes the control strategy based on reward feedback through the interaction between the intelligent agent and the environment, thereby enhancing the system's adaptive ability in complex variable load environments; model predictive control predicts future states based on the system model, and determines the control input sequence by optimizing the cost function to achieve forward-looking control of the system. The combination of the two enables the mobile robot arm to better balance task execution and system stability under variable load conditions, and achieve precise trajectory tracking.

Claims

1. A variable load mobile robot trajectory tracking method based on reinforcement learning and model predictive control, characterized in that: The following steps are involved: Step 1: Obtain the established mobile manipulator system model, which is established based on the Lagrange equation, including the calculation of system kinetic energy, potential energy and generalized force, as well as the dynamic equation; The system kinetic energy formula is: ; In the above formula, is the chassis mass of the mobile robot, is the chassis center of mass velocity of the mobile robot, For the mobile robot The moment of inertia of the joints, For the mobile robot The joint angular velocity, For the mobile robot The quality of the joints, For the mobile robot The velocity of the center of mass of each joint; The system potential energy formula is: ; In the above formula, is the acceleration due to gravity, For the The height of the center of mass of each joint relative to the reference plane; Generalized force includes frictional force and the generalized force of gravity ; Then the kinetic equation is obtained: ; In the above formula, is the mass matrix, is the Coriolis force and centripetal force matrix, is the gravity vector; Step 2: Based on the mobile robot system model in step 1, a prediction model for model predictive control is constructed. Suppose the system discrete state equation at the k+1 sampling time is: ; Set the prediction time domain to , at each sampling moment According to the current status Predicting the future The system state at each moment is defined as the cost function , solve the cost function by quadratic programming algorithm Minimum control input sequence And the first control input at the current moment Applied to the system; refers to the control input; is the system state and the reference trajectory at time The error vector, , , is the weight matrix; Step 3: Determine the state space for reinforcement learning , action space , the reward function , the Q-learning algorithm is used for training, and the Q value function update formula is: ； Among them, is the current position of the mobile robot, is the joint angle of the robot arm, is the relative position between the end effector and the target object, For the The quality of the joints, System speed information; left and right wheel speed , and They are the left and right wheel speeds and the control inputs of the robot arm joints; , , , , is the corresponding weight coefficient, is the maximum acceptable distance of the target object, is the speed error, is the angle error, is the capsize detection variable; Step 4: Input the control input of the model predictive control and reinforcement learning to select actions Fusion is performed using the fusion formula Generate final control input ; is the weight coefficient.

2. The variable load mobile robot arm trajectory tracking method based on reinforcement learning and model predictive control according to claim 1 is characterized in that: The discrete state equation of the system in step 2 Current status Contains the position coordinates of the car , heading angle 、The angles of each joint of the robot arm and its speed information, Including the speed control of the car's driving wheels and the driving torque of the robot arm joints, is the system state and the reference trajectory at time The error vector, , , is the corresponding weight matrix.

3. The variable load mobile robot arm trajectory tracking method based on reinforcement learning and model predictive control according to claim 1 is characterized in that: The specific training method for step 3 is: Step 301, initialize the Q value table: At the beginning of training, initialize a Q value table for the discrete state space and action space , the Q value table is represented as a two-dimensional array ,in , ; At the same time, set the hyperparameters: learning rate and discount factor ; Step 302: At the beginning of each training round, the agent is in the initial state of the environment ; In each state, the agent - Greedy strategy chooses an action to execute; Step 303: The agent selects an action in the state After that, the action Applied to the environment, the environment moves to the next state based on the result of the action , and returns an immediate reward ; Step 304: Calculate the TD error, i.e., the difference between the Q value of the current state-action pair and the target Q value, according to the Q-learning update rule; The target Q value is calculated based on the Bellman optimal equation. The calculation formula is: ； in, Is to take action in the current state The instant rewards you receive, is the discount factor, It is the maximum Q value of all possible actions in the next state; Use the TD error to update the Q value, which is updated as: Right now ; in, is the learning rate, which controls the update step size of the Q value. By continuously updating the Q value, the agent gradually learns the optimal value of each state-action pair. Step 305: Update the current state to the next state, i.e. , and then determine whether the current state is a terminal state; if it is a terminal state, the training round ends and enters the next training round; if it is not a terminal state, return to step 302 and continue to select actions and execute; The above steps 302 to 305 are continuously repeated to perform multiple training rounds until the Q value converges or reaches a preset number of training times.

Images