Adaptive model predictive control method based on incremental double-Q learning

Abstract

The invention belongs to the field of intelligent control, and particularly relates to a self-adaptive model prediction control method based on incremental double-Q learning, which comprises the following steps of: 1) establishing a kinematic model of an incomplete constraint differential mobile robot, and constructing a trajectory tracking error model by combining the kinematic model of the incomplete constraint differential mobile robot; 2) constructing a prediction model of a mobile robot trajectory tracking model prediction controller based on the trajectory tracking error model, and realizing high-performance trajectory tracking control according to the prediction model; 3) performing incremental discretization operation on the action space and the state space of the double-Q learning agent; and 4) continuously optimizing the state and action space of the double-Q learning agent through real-time interaction and online iterative learning between the agent and the environment, and realizing high-precision autonomous trajectory tracking control of the mobile robot by obtaining an online adaptive MPC controller. According to the method, an incremental double Q learning algorithm is utilized to perform online optimization on the weight matrix of the MPC, and the weight matrix of the MPC is adjusted in real time according to the system state so as to optimize the total reward.

Description

Technical Field

[0001] The invention belongs to the field of intelligent control, and in particular to an adaptive model predictive control method based on incremental dual-Q learning. Background Art

[0002] Accurate trajectory tracking of mobile robots is an important fundamental research topic in the field of intelligent control and robotics. Although existing model predictive control (MPC) methods have achieved some success in trajectory tracking control, they rely on accurate mathematical models and are very sensitive to design choices such as cost functions, hyperparameters, and prediction horizons. In addition, existing methods often lack sufficient adaptability and flexibility when dealing with dynamic environments and unknown disturbances. Summary of the invention

[0003] The purpose of the present invention is to provide an incremental doubleQ-learning optimized model predictive control (DQMPC) method, which can improve the trajectory tracking accuracy and adaptability of a mobile robot in a complex and dynamic environment, so as to overcome the defects of the above-mentioned model predictive control method.

[0004] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: an adaptive model predictive control method based on incremental dual-Q learning, comprising the following steps:

[0005] 1) According to the physical characteristics of the differential mobile robot, a kinematic model of the nonholonomic constraint differential mobile robot is established; the desired posture information of the mobile robot is obtained according to the reference trajectory, and the trajectory tracking error model is constructed in combination with its kinematic model;

[0006] 2) Based on the trajectory tracking error model, a prediction model of the mobile robot trajectory tracking model predictive controller is constructed, and high-performance trajectory tracking control is achieved based on the prediction model;

[0007] 3) Incrementally discretize the action space and state space of the dual Q learning agent, only add the new actions and new states of each iteration to the discretized set, and discretize the areas where the states and actions have not changed, thereby avoiding the overall discretization of the action space and state space of the agent;

[0008] 4) Through real-time interaction and online iterative learning between the agent and the environment, the state and action space of the dual Q learning agent are continuously optimized; the optimal action k is selected in each iteration after learning, and the weight matrix of the MPC is adaptively adjusted to improve the control accuracy of the MPC controller, thereby obtaining an online adaptive MPC controller;

[0009] 5) Through the online adaptive MPC controller, high-precision autonomous trajectory tracking control of the mobile robot is achieved.

[0010] According to the physical characteristics of the differential mobile robot, a kinematic model of the non-holonomic constraint differential mobile robot is established, which is specifically:

[0011] The state vector of the kinematic model is (x, y, θ), and the decision vector is (v, ω); then the kinematic model of the nonholonomic constrained differential mobile robot is for:

[0012]

[0013] in, is the speed of the mobile robot along the X-axis and Y-axis, Represents angular velocity.

[0014] The desired posture information of the mobile robot is obtained according to the reference trajectory, and the trajectory tracking error model is constructed in combination with its kinematic model, specifically:

[0015] 1-1) In the global coordinate system, the model of the reference trajectory is set as:

[0016] η r (t) = (x r (t),y r (t),θ r (t)) T

[0017] u r (t)=(v r (t),ω r (t)) T

[0018]

[0019] Among them, η r (t) is the expected pose sequence of the reference trajectory, x r (t),y r (t),θ r (t) represents the expected position in the X-axis direction, the expected position in the Y-axis direction, and the expected posture, u r (t) represents the expected velocity sequence of the reference trajectory, v r (t),ω r (t) represent the expected linear velocity and expected angular velocity, respectively;

[0020] According to the time-constrained position information of the expected trajectory, the expected speed v of the mobile robot is obtained r (t) and the desired angular velocity ω r (t);

[0021] 1-2) The error model between the kinematic model of the nonholonomic constrained differential mobile robot and the reference trajectory in the global coordinate system is obtained as follows:

[0022]

[0023] Among them, x e (t) is the position error along the X-axis, y e (t) is the position error in the Y-axis direction, θ e (t) is the heading angle error, is the velocity change in the X-axis direction, Expressed as the speed change of the Y-axis defense line, It is expressed as the change in angular velocity, v(t) is the desired linear velocity of the robot, and v r (t) is the actual linear velocity of the robot, ω r (t) is the desired angular velocity of the robot, ω(t) is the actual angular velocity of the robot;

[0024] 1-3) The input vector of the control error model is:

[0025]

[0026] Among them, u e (t) is the input vector of trajectory tracking control, which includes the linear velocity component u1(t) and the angular velocity component u2(t). It is expressed as the change in angular velocity, v(t) is the desired linear velocity of the robot, and v r (t) is the actual linear velocity of the robot, ω r (t) is the desired angular velocity of the robot, ω(t) is the actual angular velocity of the robot;

[0027] 1-4) Obtain the trajectory tracking error model, namely:

[0028]

[0029] The step 2) is specifically:

[0030] 2-1) According to the trajectory tracking error model, the state transfer matrix of the error model is obtained Where, T is the sampling period;

[0031] The prediction model of the model predictive controller is constructed as follows:

[0032]

[0033] Among them, the state vector Control vector u=(v,ω) T , C k,t =(I3,03×3 );

[0034] 2-2) The objective function J(k) of the trajectory tracking model predictive controller is designed as:

[0035]

[0036] Among them, Q and R are state weight matrices, Q f is the output weight matrix, N p and N c Represent the prediction time domain and the control time domain respectively;

[0037] 2-3) Therefore, the optimization problem of the prediction model of the model predictive controller is expressed as:

[0038] min J(k)

[0039] s min ≤u(k)≤u max

[0040] Δu min ≤Δu(k)≤Δu max

[0041] Among them, u(k) represents the actual velocity vector value of the robot, u min and u max denote the maximum linear velocity and the minimum linear velocity respectively, Δu(k) denotes the acceleration vector of the robot, Δu min and Δu max They represent the minimum acceleration and maximum acceleration of the robot respectively.

[0042] The step 3) comprises the following steps:

[0043] 3-1) Define the initial mobile robot action space K and state space X;

[0044] 3-2) In the early stage of online training, the state and action space are roughly discretized, that is, the fixed intervals of speed and position, as well as the initial values ​​of state and action are pre-set according to the physical constraints of the system;

[0045] 3-3) Whenever the robot enters a new state x, check whether the state already exists in the discretized state set;

[0046] If it does not exist, that is, the difference between the state and the state in the existing state set exceeds the preset threshold d t+1 When >ρ, the state is added to the discretized state set X, that is:

[0047]

[0048] X={X∪X t+1}←x t+1

[0049] If the state exists in the state set X and appears N times in a row, it is considered that the key area near the state tends to the local optimal solution, and only this part is discretized;

[0050] 3-4) For each selected action k, check whether the state already exists in the discretized action set; if the new action is significantly different from the actions in the existing action set, add it to the action set K; if N consecutive actions do not cause the system state to shift, the area near the action is considered to be locally optimal, and only this part is discretized.

[0051] The step 3-1) is specifically:

[0052] The constituent element of the state space is the state vector, which is x=[x e (t),y e (t),θ e (t)] T , which is composed of the position error, velocity error and heading error of the mobile robot, and the state space X is composed of the state vector;

[0053] The action vector is k = [k r ,k q ,k qf ] T , and the action space K is formed by the action vectors.

[0054] A time-error-based priority experience sampling method is used for past experience to speed up the training of double Q learning.

[0055] The step 4) is specifically:

[0056] 4-1) Set the initialization parameters of the system;

[0057] 4-2) In each iteration, use the ∈-greedy strategy to select action k t , to strike a balance between exploration and exploitation;

[0058] 4-3) Update temporary memory and set the current state x t and the selected action k t Save to temporary memory M, that is:

[0059] M←τ i =(x t ,u t ,r t ,x t+1 )

[0060] 4-4) Execute action k t , get the new state x t+1 and reward r t+1 , then the constructed reward function is:

[0061]

[0062] Among them, β is a constant parameter, x t Represents the current state, x r represents the desired state, r t is the reward value;

[0063] 4-5) Check the most recent N memories (M t , M t-1 ……, M t-N ) are consistent to determine whether the system is in a stable state. If M t =M t-1 =... =M t-N , the system is considered to be in a stable state, and the incremental discretization mechanism is started to further discretize the stable region of the state;

[0064] 4-6) If the new state of the system x t+1 The corresponding new memory M t+1 ≠M t , then the new state x t+1 Provides valuable new information to the system, then add it to the state space set X;

[0065] 4-7) Calculate the time error δ and determine the priority p of the sample according to the absolute value of the TD error t ;

[0066] 4-8) The sample (s t ,a t ,r t ,s t+1 ) is stored in the experience revisit pool and is sorted according to its priority p t , to determine the sampling priority;

[0067] 4-9) Through the above steps to the state of the intelligent body and the update of action k = [k r ,k q ,k qf ] T , and then complete the adaptive tuning of the MPC weight matrix.

[0068] An adaptive model predictive control method based on incremental dual-Q learning includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement an adaptive model predictive control method based on incremental dual-Q learning when executing the computer program.

[0069] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, an adaptive model predictive control method based on incremental dual-Q learning is implemented.

[0070] The present invention has the following beneficial effects and advantages:

[0071] 1. The present invention integrates doubleQ-learning into traditional model predictive control technology, giving full play to the advantages of doubleQ-learning and environmental interactive learning, as well as the advantages of traditional MPC controllers with good stability and high control accuracy. The combination of the two can effectively improve the accuracy of mobile robot trajectory tracking control.

[0072] 2. The present invention establishes a model predictive control (MPC) prediction model based on the kinematic model of the mobile robot; performs non-uniform discretization on the action and state space to improve learning efficiency; uses a priority experience sampling method based on time error to reduce the mutual dependence between past experiences and accelerate the training speed; and dynamically optimizes the MPC weight matrix parameters through real-time interaction with the environment to improve control performance.

[0073] 3. The present invention incrementally discretizes the action and state space, focusing on the action area that has a significant impact on the robot's state change.

[0074] 4. The present invention performs priority sampling on samples according to the time difference (TD) error to improve the training efficiency and model convergence speed.

[0075] 5. The present invention utilizes an incremental dual Q learning algorithm to perform online optimization on the MPC weight matrix, and adjusts the MPC weight matrix in real time according to the system status to optimize the total reward. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 Schematic diagram of the kinematic model of the wheeled differential mobile robot of the present invention;

[0077] Figure 2 The MPC trajectory tracking principle diagram based on incremental dual-Q learning of the present invention;

[0078] Figure 3 Schematic diagram of the incremental discretization of the action space of the present invention;

[0079] Figure 4The state space incremental discretization principle diagram of the present invention;

[0080] Figure 5 A flow chart of a method for obtaining an online adaptive MPC controller according to the present invention. DETAILED DESCRIPTION

[0081] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0082] The present invention realizes online adaptive adjustment of the MPC weight matrix through an incremental doubleQ-learning method, improves the control accuracy of the adaptive MPC trajectory tracking controller, and enhances its adaptability to dynamic working environments. By incrementally discretizing the action space and state space of the reinforcement learning agent, and only discretizing key areas, the overall discretization of the state and action space of the agent is avoided, thereby improving the learning efficiency of reinforcement learning.

[0083] like Figure 2 As shown, it is a schematic diagram of the MPC trajectory tracking principle based on incremental dual-Q learning of the present invention. The adaptive model predictive control method based on incremental dual-Q learning of the present invention is characterized by comprising the following steps:

[0084] 1) According to the physical characteristics of the differential mobile robot, a kinematic model of the nonholonomic constraint differential mobile robot is established; the desired posture information of the mobile robot is obtained according to the reference trajectory, and the trajectory tracking error model is constructed in combination with its kinematic model;

[0085] like Figure 1 As shown in FIG. 1 , it is a schematic diagram of the kinematic model of the wheeled differential mobile robot of the present invention. According to the physical characteristics of the differential mobile robot, a kinematic model of the non-holonomic constrained differential mobile robot is established, specifically:

[0086] The state vector of the kinematic model is (x, y, θ), and the decision vector is (v, ω); then the kinematic model of the nonholonomic constrained differential mobile robot is for:

[0087]

[0088] in, is the speed of the mobile robot along the X-axis and Y-axis, Represents angular velocity.

[0089] In step 1), the desired posture information of the mobile robot is obtained according to the reference trajectory, and the trajectory tracking error model is constructed in combination with its kinematic model, specifically:

[0090] 1-1) In the global coordinate system, the model of the reference trajectory is set as:

[0091] η r (t) = (x r (t),y r (t),θ r (t)) T

[0092] u r (t)=(v r (t),ω r (t)) T

[0093]

[0094] Among them, η r (t) is the expected pose sequence of the reference trajectory, x r (t),y r (t),θ r (t) represents the expected position in the X-axis direction, the expected position in the Y-axis direction, and the expected posture, u r (t) represents the expected velocity sequence of the reference trajectory, v r (t),ω r (t) represent the expected linear velocity and expected angular velocity, respectively;

[0095] According to the time-constrained position information of the expected trajectory, the expected speed v of the mobile robot is obtained r (t) and the desired angular velocity ω r (t);

[0096] 1-2) The error model between the kinematic model of the nonholonomic constrained differential mobile robot and the reference trajectory in the global coordinate system is obtained as follows:

[0097]

[0098] Among them, x e (t) is the position error along the X-axis, y e (t) is the position error in the Y-axis direction, θ e (t) is the heading angle error, is the velocity change in the X-axis direction, Expressed as the speed change of the Y-axis defense line, It is expressed as the change in angular velocity, v(t) is the desired linear velocity of the robot, and v r (t) is the actual linear velocity of the robot, ω r (t) is the desired angular velocity of the robot, ω(t) is the actual angular velocity of the robot;

[0099] 1-3) The input vector of the control error model is:

[0100]

[0101] Among them, u e (t) is the input vector of trajectory tracking control, which includes the linear velocity component u1(t) and the angular velocity component u2(t). It is expressed as the change in angular velocity, v(t) is the desired linear velocity of the robot, and v r (t) is the actual linear velocity of the robot, ω r (t) is the desired angular velocity of the robot, ω(t) is the actual angular velocity of the robot;

[0102] 1-4) Obtain the trajectory tracking error model, namely:

[0103]

[0104] 2) Based on the trajectory tracking error model, a prediction model of the mobile robot trajectory tracking model predictive controller is constructed, and high-performance trajectory tracking control is achieved based on the prediction model;

[0105] 2-1) According to the trajectory tracking error model, the state transfer matrix of the error model is obtained Where, T is the sampling period;

[0106] The prediction model of the model predictive controller is constructed as follows:

[0107]

[0108] Among them, the state vector Control vector u=(v,ω) T , C k,t =(I3,0 3×3 );

[0109] 2-2) The objective function J(k) of the trajectory tracking model predictive controller is designed as:

[0110]

[0111] Among them, Q and R are state weight matrices, Q f is the output weight matrix, N p and N c Represent the prediction time domain and the control time domain respectively;

[0112] 2-3) Therefore, the optimization problem of the prediction model of the model predictive controller is expressed as:

[0113] min J(k)

[0114] s min ≤u(k)≤u max

[0115] Δu min ≤Δu(k)≤Δu max

[0116] Among them, u(k) represents the actual velocity vector value of the robot, u min and u max denote the maximum linear velocity and the minimum linear velocity respectively, Δu(k) denotes the acceleration vector of the robot, Δu min and Δu max They represent the minimum acceleration and maximum acceleration of the robot respectively.

[0117] Among them, the prediction model is the state equation of trajectory tracking control and is also the core model of the entire trajectory tracking control. Trajectory tracking is controlled based on this prediction model (state equation).

[0118] 3) Incrementally discretize the action space and state space of the dual Q learning agent, only add the new actions and new states of each iteration to the discretized set, and discretize the areas where the states and actions have not changed, thereby avoiding the overall discretization of the action space and state space of the agent;

[0119] like Figure 3-4 As shown, it is a schematic diagram of the incremental discretization of the action space and state space of the present invention;

[0120] Specifically, the incremental discretization operation of the action space and state space of the dual Q learning agent includes the following steps:

[0121] 3-1) Define the initial mobile robot action space K and state space X;

[0122] The constituent element of the state space is the state vector, which is x=[x e (t),y e (t),θ e (t)] T , which is composed of the position error, velocity error and heading error of the mobile robot, and the state space X is composed of the state vector;

[0123] The action vector is k = [k r ,k q ,k qf ] T , and the action space K is formed by the action vectors.

[0124] 3-2) In the early stage of online training, the state and action space are roughly discretized, that is, the fixed intervals of speed and position, as well as the initial values ​​of state and action are pre-set according to the physical constraints of the system;

[0125] 3-3) Whenever the robot enters a new state x, check whether the state already exists in the discretized state set;

[0126] If it does not exist, that is, the difference between the state and the state in the existing state set exceeds the preset threshold d t+1 When >ρ, the state is added to the discretized state set X, that is:

[0127]

[0128] X={X∪X t+1}←x t+1

[0129] If the state exists in the state set X and appears N times in a row, it is considered that the key area near the state tends to the local optimal solution, and only this part is discretized;

[0130] 3-4) For each selected action k, check whether the state already exists in the discretized action set; if the new action is significantly different from the actions in the existing action set, add it to the action set K; if N consecutive actions do not cause the system state to shift, the area near the action is considered to be locally optimal, and only this part is discretized.

[0131] 3-5) In steps 3-1) to 3-4), only the key areas are discretized to avoid the overall discretization of the state and action space of the agent, thereby improving the learning efficiency of reinforcement learning.

[0132] During the algorithm operation, the state space and action space are discretized non-uniformly according to the interaction between the robot and the environment. The focus of detailed discretization is mainly on areas that the robot frequently visits or have a significant impact on decision-making.

[0133] 4) Through real-time interaction and online iterative learning between the agent and the environment, the state and action space of the dual Q learning agent are continuously optimized; the optimal action k is selected in each iteration after learning, and the weight matrix of the MPC is adaptively adjusted to improve the control accuracy of the MPC controller, thereby obtaining an online adaptive MPC controller;

[0134] Through active learning in the fourth step, the state space and action space of dual Q learning are continuously expanded. As the amplitude of the state space and action space continues to increase, the probability of selecting the optimal action will increase. Each iteration will be the optimal action at the moment, and the optimal action controls the MPC with the optimal weight, and the trajectory tracking effect is the optimal solution at the moment.

[0135] Specifically, Figure 5As shown, it is a flow chart of the method for obtaining an online adaptive MPC controller according to the present invention, step 4) includes the following steps:

[0136] 4-1) Set the initialization parameters of the system;

[0137] 4-2) In each iteration, use the ∈-greedy strategy to select action k t , to strike a balance between exploration and exploitation;

[0138] 4-3) Update temporary memory and set the current state x t and the selected action k t Save to temporary memory M, that is:

[0139] M←τ i =(x t ,u t ,r t ,x t+1 )

[0140] 4-4) Execute action k t , get the new state x t+1 and reward r t+1 , then the constructed reward function is:

[0141]

[0142] Among them, β is a constant parameter, x t Represents the current state, x r represents the desired state, r t is the reward value;

[0143] 4-5) Check the most recent N memories (M t , M t-1 ……, M t-N ) are consistent to determine whether the system is in a stable state. If M t =M t-1 =... =M t-N , the system is considered to be in a stable state, and the incremental discretization mechanism is started to further discretize the stable region of the state;

[0144] 4-6) If the new state of the system x t+1 The corresponding new memory M t+1 ≠M t , then the new state x t+1 Provides valuable new information to the system, then add it to the state space set X;

[0145] Among them, the valuable new information can be understood as: the system may have only 5 states originally. This new state is different from the previous 5 states, so this state is added as a new state, and the state space becomes 6 states.

[0146] This part describes the implementation process, that is, the incremental active learning process. It is the physical implementation of the following formula, or a method of implementing the theory into practice.

[0147]

[0148] X={X∪X t+1}←x t+1

[0149] 4-7) Calculate the time error δ and determine the priority p of the sample according to the absolute value of the TD error t ;

[0150] a. If you choose to update Q A , by calculating Q A To select the next state x from the entire action space K t+1 The best possible action k′ is:

[0151] k′=argm K QUR A (x t+1 ,K)

[0152] b. Use Q B Estimate the Q value of this action:

[0153] y t =r t +γQ B (x t+1 ,k′)

[0154] y t That is, the target Q value corresponding to time step t, γ is the discount factor, and γ∈(0,1];

[0155] c. Calculate TD error:

[0156] δ t =y t -Q A (x t ,k t )

[0157] d. If you choose to update Q B , then the same process is just using Q B To update, and use Q A To calculate the target value, alternately use Q A and QB The double Q learning method can effectively reduce the overestimation of future reward values;

[0158] e. According to the absolute value of the TD error, the priority is calculated, that is:

[0159] p t =|δ t |.

[0161] 4-8) The sample (s t ,a t ,r t ,s t+1 ) is stored in the experience revisit pool and is sorted according to its priority p t , to determine the sampling priority;

[0162] 4-9) Through the above steps to the state of the intelligent body and the update of action k = [k r ,k q ,k qf ] T , and then complete the adaptive tuning of the MPC weight matrix to improve the dynamic environment adaptability of the MPC trajectory tracking controller.

[0163] 5) Through the online adaptive MPC controller, high-precision autonomous trajectory tracking control of the mobile robot is achieved.

[0164] A system for mobile robot trajectory tracking, comprising:

[0165] 1) A GPS integrated navigation system to obtain the robot's real-time position, speed and posture information;

[0166] 2) A trajectory tracking controller, wherein the controller is configured to execute the above-mentioned incremental doubleQ-learning enhanced model predictive control method to complete the trajectory tracking control of the robot according to the state information obtained by the sensor.

[0167] It also includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement an adaptive model predictive control method based on incremental dual Q learning when executing the computer program.

[0168] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, an adaptive model predictive control method based on incremental dual-Q learning is implemented.

[0169] In summary, the method described in the present invention integrates doubleQ-learning into traditional model predictive control technology, giving full play to the advantages of doubleQ-learning and environmental interactive learning, as well as the advantages of traditional MPC controllers in terms of stability, good control accuracy, and the combination of the two can effectively improve the accuracy of mobile robot trajectory tracking control.

[0170] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units. The above is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the contents of the specification and drawings of this application, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. An adaptive model predictive control method based on incremental dual-Q learning, characterized in that: The following steps are involved: 1) According to the physical characteristics of the differential mobile robot, a kinematic model of the nonholonomic constraint differential mobile robot is established; the desired posture information of the mobile robot is obtained according to the reference trajectory, and the trajectory tracking error model is constructed in combination with its kinematic model; 2) Based on the trajectory tracking error model, a prediction model of the mobile robot trajectory tracking model predictive controller is constructed, and high-performance trajectory tracking control is achieved based on the prediction model; 3) Incrementally discretize the action space and state space of the dual Q learning agent, only add the new actions and new states of each iteration to the discretized set, and discretize the areas where the states and actions have not changed, thereby avoiding the overall discretization of the action space and state space of the agent; 4) Through real-time interaction and online iterative learning between the agent and the environment, the state and action space of the dual Q learning agent are continuously optimized; the optimal action k is selected in each iteration after learning, and the weight matrix of the MPC is adaptively adjusted to improve the control accuracy of the MPC controller, thereby obtaining an online adaptive MPC controller; 5) Through the online adaptive MPC controller, high-precision autonomous trajectory tracking control of the mobile robot is achieved.

2. The adaptive model predictive control method based on incremental dual Q learning according to claim 1, characterized in that: According to the physical characteristics of the differential mobile robot, a kinematic model of the non-holonomic constraint differential mobile robot is established, which is specifically: The state vector of the kinematic model is (x, y, θ), and the decision vector is (v, ω); then the kinematic model of the nonholonomic constrained differential mobile robot is for: in, is the speed of the mobile robot along the X-axis and Y-axis, Represents angular velocity.

3. The adaptive model predictive control method based on incremental dual Q learning according to claim 1, characterized in that: The desired posture information of the mobile robot is obtained according to the reference trajectory, and the trajectory tracking error model is constructed in combination with its kinematic model, specifically: 1-1) In the global coordinate system, the model of the reference trajectory is set as: η r (t)=(x r (t),y r (t),θ r (t)) T u r (t)=(v r (t),ω r (t)) T Among them, η r (t) is the expected pose sequence of the reference trajectory, x r (t),y r (t),θ r (t) represents the expected position in the X-axis direction, the expected position in the Y-axis direction, and the expected posture, u r (t) represents the expected velocity sequence of the reference trajectory, v r (t),ω r (t) represent the expected linear velocity and expected angular velocity, respectively; According to the time-constrained position information of the expected trajectory, the expected speed v of the mobile robot is obtained r (t) and the desired angular velocity ω r (t); 1-2) The error model between the kinematic model of the nonholonomic constrained differential mobile robot and the reference trajectory in the global coordinate system is obtained as follows: Among them, x e (t) is the position error along the X-axis, y e (t) is the position error in the Y-axis direction, θ e (t) is the heading angle error, is the velocity change in the X-axis direction, Expressed as the speed change of the Y-axis defense line, It is expressed as the change in angular velocity, v(t) is the desired linear velocity of the robot, and v r (t) is the actual linear velocity of the robot, ω r (t) is the desired angular velocity of the robot, ω(t) is the actual angular velocity of the robot; 1-3) The input vector of the control error model is: Among them, u e (t) is the input vector of trajectory tracking control, which includes the linear velocity component u1(t) and the angular velocity component u2(t). It is expressed as the change in angular velocity, v(t) is the desired linear velocity of the robot, and v r (t) is the actual linear velocity of the robot, ω r (t) is the desired angular velocity of the robot, ω(t) is the actual angular velocity of the robot; 1-4) Obtain the trajectory tracking error model, namely:

4. The adaptive model predictive control method based on incremental dual Q learning according to claim 1, characterized in that: The step 2) is specifically: 2-1) According to the trajectory tracking error model, the state transfer matrix of the error model is obtained Where, T is the sampling period; The prediction model of the model predictive controller is constructed as follows: Among them, the state vector Control vector u=(v,ω) T , C k,t =(I3,0 3×3 ); 2-2) The objective function J(k) of the trajectory tracking model predictive controller is designed as: Among them, Q and R are state weight matrices, Q f is the output weight matrix, N p and N c Represent the prediction time domain and the control time domain respectively; 2-3) Therefore, the optimization problem of the prediction model of the model predictive controller is expressed as: min J(k) s.t u min ≤u(k)≤u max Δu min ≤Δu(k)≤Δu max Among them, u(k) represents the actual velocity vector value of the robot, u min and u max denote the maximum linear velocity and the minimum linear velocity respectively, Δu(k) denotes the acceleration vector of the robot, Δu min and Δu max They represent the minimum acceleration and maximum acceleration of the robot respectively.

5. The adaptive model predictive control method based on incremental dual Q learning according to claim 1, characterized in that: The step 3) comprises the following steps: 3-1) Define the initial mobile robot action space K and state space X; 3-2) In the early stage of online training, the state and action space are roughly discretized, that is, the fixed intervals of speed and position, as well as the initial values ​​of state and action are pre-set according to the physical constraints of the system; 3-3) Whenever the robot enters a new state x, check whether the state already exists in the discretized state set; If it does not exist, that is, the difference between the state and the state in the existing state set exceeds the preset threshold d t+1 When >ρ, the state is added to the discretized state set X, that is: X={X∪X t+1 }←x t+1 If the state exists in the state set X and appears N times in a row, it is considered that the key area near the state tends to the local optimal solution, and only this part is discretized; 3-4) For each selected action k, check whether the state already exists in the discretized action set; if the new action is significantly different from the actions in the existing action set, add it to the action set K; if N consecutive actions do not cause the system state to shift, the area near the action is considered to be locally optimal, and only this part is discretized.

6. The adaptive model predictive control method based on incremental dual Q learning according to claim 5, characterized in that: The step 3-1) is specifically: The constituent element of the state space is the state vector, which is x=[x e (t),y e (t),θ e (t)] T , which is composed of the position error, velocity error and heading error of the mobile robot, and the state space X is composed of the state vector; The action vector is k = [k r ,k q ,k qf ] T , and the action space K is formed by the action vectors.

7. The adaptive model predictive control method based on incremental dual Q learning according to claim 1, characterized in that: A time-error-based priority experience sampling method is used for past experience to speed up the training of double Q learning.

8. The adaptive model predictive control method based on incremental dual Q learning according to claim 1, characterized in that: The step 4) is specifically: 4-1) Set the initialization parameters of the system; 4-2) In each iteration, use the ∈-greedy strategy to select action k t , to strike a balance between exploration and exploitation; 4-3) Update temporary memory and set the current state x t and the selected action k t Save to temporary memory M, that is: M←τ i =(x t ,u t ,r t ,x t+1 ) 4-4) Execute action k t , get the new state x t+1 and reward r t+1 , then the constructed reward function is: Among them, β is a constant parameter, x t Represents the current state, x r represents the desired state, r t is the reward value; 4-5) Check the most recent N memories (M t , M t-1 ……, M t-N ) are consistent to determine whether the system is in a stable state. If M t =M t-1 =... =M t-N , the system is considered to be in a stable state, and the incremental discretization mechanism is started to further discretize the stable region of the state; 4-6) If the new state of the system x t+1 The corresponding new memory M t+1 ≠M t , then the new state x t+1 Provides valuable new information to the system, then add it to the state space set X; 4-7) Calculate the time error δ and determine the priority p of the sample according to the absolute value of the TD error t ; 4-8) The sample (s t ,a t ,r t ,s t+1 ) is stored in the experience revisit pool and is sorted according to its priority p t , to determine the sampling priority; 4-9) Through the above steps to the state of the intelligent body and the update of action k = [k r ,k q ,k qf ] T , and then complete the adaptive tuning of the MPC weight matrix.

9. The adaptive model predictive control method based on incremental dual Q learning according to claim 8, characterized in that: It includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement the adaptive model predictive control method based on incremental dual Q learning as described in any one of claims 1-8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the adaptive model predictive control method based on incremental dual Q learning as described in any one of claims 1 to 8 is implemented.

Images