Adaptive model predictive control method based on incremental double q-learning
By using an incremental double-Q learning optimization model predictive control method, the problem of dependence on mathematical models and insufficient adaptability in mobile robot trajectory tracking is solved, achieving high-precision trajectory tracking in dynamic environments and improving the controller's adaptive capability and learning efficiency.
Patent Information
- Application Number
- CN202411375800.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-12-09
- Estimated Expiration
- 2044-09-30
AI Technical Summary
Existing model predictive control methods rely heavily on mathematical models for mobile robot trajectory tracking, lacking adaptability and flexibility, and are particularly difficult to achieve high-precision trajectory tracking in dynamic environments.
An incremental double-Q learning optimization model predictive control method is adopted. By establishing a kinematic model with nonholonomic constraints, a trajectory tracking error model is constructed, and the action and state spaces are incrementally discretized. Combined with real-time interaction and online iterative learning of double-Q learning, the MPC weight matrix is optimized to improve control accuracy.
It improves the trajectory tracking accuracy and adaptability of mobile robots in complex and dynamic environments, enhances the adaptive capability of the controller, and improves the accuracy and efficiency of trajectory tracking control.
Smart Images
Figure CN119148531B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the field of intelligent control, specifically an adaptive model predictive control method based on incremental double Q-learning. BACKGROUND
[0002] In the field of intelligent control and robotics, precise trajectory tracking of mobile robots is an important and fundamental research topic. Existing model predictive control (MPC) methods have achieved certain success in trajectory tracking control, but they rely on accurate mathematical models and are very sensitive to design choices such as cost functions, hyperparameters, and prediction ranges. In addition, existing methods often lack sufficient adaptability and flexibility when dealing with dynamic environments and unknown disturbances. SUMMARY
[0003] The purpose of the present application is to provide an incremental double Q-learning optimized model predictive control (DQMPC) method that can improve the trajectory tracking accuracy and adaptability of mobile robots in complex and dynamic environments to overcome the shortcomings of the above-mentioned model predictive control methods.
[0004] The technical solution adopted by the present application to achieve the above-mentioned purpose is: an adaptive model predictive control method based on incremental double Q-learning, comprising the following steps:
[0005] 1) According to the physical characteristics of the differential mobile robot, a kinematic model of the nonholonomic constrained differential mobile robot is established; the desired pose information of the mobile robot is obtained according to the reference trajectory, and a trajectory tracking error model is constructed in combination with the kinematic model;
[0006] 2) Based on the trajectory tracking error model, a prediction model of the mobile robot trajectory tracking model predictive controller is constructed, and high-performance trajectory tracking control is realized according to the prediction model;
[0007] 3) Incremental discretization is performed on the action space and state space of the double Q-learning agent, only the new action and new state of each iteration are added to the discretized set, and the regions where the state and action have not changed are discretized, thereby avoiding full discretization of the action space and state space of the agent;
[0008] 4) Through real-time interaction and online iterative learning between the agent and the environment, the state and action space of the double Q-learning agent are continuously optimized; the optimal action k is selected at each iteration after learning, the weight matrix of the MPC is adaptively adjusted to improve the control accuracy of the MPC controller, and an online adaptive MPC controller is obtained;
[0009] 5) Through the online adaptive MPC controller, high-precision autonomous trajectory tracking control of the mobile robot is realized.
[0010] The kinematic model of the nonholonomic constraint differential mobile robot is established according to the physical characteristics of the differential mobile robot, in particular:
[0011] The state vector of the kinematic model is (x, y, θ), and the decision vector is (v, ω); then the kinematic model of the nonholonomic constraint differential mobile robot is:
[0012]
[0013] wherein, is the velocity of the mobile robot along the X-axis and the Y-axis, represents the angular velocity.
[0014] According to the reference trajectory, the expected pose information of the mobile robot is obtained, and a trajectory tracking error model is constructed in combination with the kinematic model, in particular:
[0015] 1-1) In the global coordinate system, the model of the reference trajectory is set as:
[0016] η r (t)=(x r (t),y r (t),θ r (t)) T
[0017] u r (t)=(v r (t),ω r (t)) T
[0018]
[0019] wherein, η r (t) is the expected pose sequence of the reference trajectory, x r (t), y r (t), θ r (t) respectively represent the expected position in the X-axis direction, the expected position in the Y-axis direction, and the expected attitude, u r (t) represents the expected velocity sequence of the reference trajectory, v r (t), ω r (t) respectively represent the expected linear velocity and the expected angular velocity;
[0020] According to the expected trajectory with time-constrained pose information, the expected velocity v r (t) and the expected angular velocity ω r (t) of the mobile robot are obtained;
[0021] 1-2) Obtain the kinematics model of the nonholonomic constraint differential mobile robot in the global coordinate system and the error model of the reference trajectory, that is:
[0022]
[0023] wherein x e (t) is the position error along the X-axis direction, y e (t) is the position error in the Y-axis direction, θ e (t) is the heading angle error, is the velocity change amount in the X-axis direction, is the velocity change amount in the Y-axis direction, is the angular velocity change amount, v(t) is the desired linear velocity of the robot, v r (t) is the actual linear velocity of the robot, ω r (t) is the desired angular velocity of the robot, and ω(t) is the actual angular velocity of the robot;
[0024] 1-3) The input vector of the error model is:
[0025]
[0026] wherein u e (t) is the input vector of the trajectory tracking control, including the linear velocity component u1(t) and the angular velocity component u2(t), is the angular velocity change amount, v(t) is the desired linear velocity of the robot, v r (t) is the actual linear velocity of the robot, ω r (t) is the desired angular velocity of the robot, and ω(t) is the actual angular velocity of the robot;
[0027] 1-4) Obtain the trajectory tracking error model, that is:
[0028]
[0029] The step 2) is specifically:
[0030] 2-1) According to the trajectory tracking error model, the state transition matrix of the error model is obtained, that is: wherein T is the sampling period;
[0031] Then, the prediction model of the model prediction controller is constructed, that is:
[0032]
[0033] wherein the state vector is the control vector u=(v, ω) T , C k,t =(I3,03×3 );
[0034] 2-2) The objective function J(k) of the trajectory tracking model predictive controller is designed as:
[0035]
[0036] where Q and R are state weight matrices, Q f is the output weight matrix, N p and N c represent the prediction horizon and control horizon, respectively;
[0037] 2-3) Therefore, the optimization problem of the predictive model of the model predictive controller is represented as:
[0038] min J(k)
[0039] s.t u min ≤ u(k) ≤ u max
[0040] Δu min ≤ Δu(k) ≤ Δu max
[0041] where u(k) represents the actual velocity vector value of the robot, u min and u max represent the maximum linear velocity and minimum linear velocity, respectively, Δu(k) represents the acceleration vector of the robot, Δu min and Δu max represent the minimum acceleration and maximum acceleration of the robot, respectively.
[0042] The step 3) comprises the following steps:
[0043] 3-1) Defining the initial moving robot action space K and state space X;
[0044] 3-2) In the initial online training, the state and action spaces are roughly discretized, that is, the fixed interval of the velocity and position according to the system physical constraints, and the initial value of the state and action are pre-set;
[0045] 3-3) Whenever the robot enters a new state x, it is checked whether the state already exists in the discretized state set;
[0046] If it does not exist, that is, the state differs from the existing state set by more than a preset threshold d t+1 > p, the state is added to the discretized state set X, that is:
[0047]
[0048] X = { X U X t+1} <- x t+1
[0049] If the state exists in the state set X and appears continuously for N times, it is considered that the focus area near the state tends to a local optimal solution, and only this part is discretized;
[0050] 3-4) For each selected action k, check whether the state has existed in the discretized action set; if the new action has a significant difference from the action in the existing action set, it is added to the action set K; if the continuous N actions do not cause the transition of the system state, it is considered that the area near the action is locally optimal, and only this part is discretized.
[0051] The step 3-1) is specifically:
[0052] The constituting element of the state space is the state vector, and the state vector is x = [x e (t),y e (t),θ e (t)] T , which is composed of the position error, the speed error and the heading error of the mobile robot, and is composed of the state vector to form the state space X;
[0053] The action vector is k = [k r ,k q ,k qf ] T , and the action space K is composed of the action vector.
[0054] The past experience adopts a priority experience sampling method based on time error to speed up the training speed of the double Q learning.
[0055] The step 4) is specifically:
[0056] 4-1) The initialization parameters of the system are set;
[0057] 4-2) In each iteration, the action k t is selected by using an e-greedy strategy to balance between exploration and utilization;
[0058] 4-3) The temporary memory is updated, and the current state x t and the selected action k t are saved to the temporary memory M, that is:
[0059] M <- tau i = (x t ,u t ,r t ,x t+1 )
[0060] 4-4) Perform action k t , get new state x t+1 and reward r t+1 , then the constructed reward function is:
[0061]
[0062] Where β is a constant parameter, x t represents the current state, x r represents the desired state, and r t is the reward value.
[0063] 4-5) Check whether the last N memories (M t , M t-1 ,..., M t-N ) are consistent to determine whether the system is in a stable state. If M t = M t-1 =... = M t-N , it is considered that the system is in a stable state, and the incremental discretization mechanism is started to further discretize the stable region of the state.
[0064] 4-6) If the new state x t+1 of the system corresponds to a new memory M t+1 ≠ M t , then the new state x t+1 provides valuable new information to the system, and it is added to the state space set X.
[0065] 4-7) Calculate the time error δ and determine the priority p t of the sample according to the absolute value of the TD error.
[0066] 4-8) Store the sample (s t , a t , r t , s t+1 ) in the experience replay pool, and determine the priority of the sample according to its priority p t .
[0067] 4-9) Through the above steps, the state and action of the agent are updated k = [k r , k q , k qf ] T , and the adaptive setting of the MPC weight matrix is completed.
[0068] The adaptive model predictive control method based on incremental double Q learning comprises a memory and a processor; the memory is used for storing a computer program; and the processor is used for realizing the adaptive model predictive control method based on incremental double Q learning when the computer program is executed.
[0069] A computer readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, an adaptive model predictive control method based on incremental double Q learning is realized.
[0070] The present application has the following advantages and benefits:
[0071] 1. The present application integrates double Q-learning into traditional model predictive control technology, takes advantage of the interactive learning of double Q-learning and the environment, and the advantages of traditional MPC controller in stability and high control precision, and the combination of the two effectively improves the precision of mobile robot trajectory tracking control.
[0072] 2. The present application establishes a model predictive control (MPC) prediction model based on the kinematic model of a mobile robot; non-uniformly discretizes the action and state space to improve learning efficiency; uses a priority experience sampling method based on time error to reduce the mutual dependence between past experiences and speed up the training speed; and dynamically optimizes the MPC weight matrix parameters through real-time interaction with the environment to improve control performance.
[0073] 3. The present application incrementally discretizes the action and state space, and focuses on the action area that has a significant impact on the change of the robot state.
[0074] 4. The present application samples the samples according to the time difference (TD) error to improve the training efficiency and model convergence speed.
[0075] 5. The present application uses an incremental double Q-learning algorithm to optimize the weight matrix of MPC online, and adjusts the MPC weight matrix in real time according to the system state to optimize the total reward. BRIEF DESCRIPTION OF DRAWINGS
[0076] Figure 1 Kinematic model principle diagram of the wheeled differential mobile robot of the present application;
[0077] Figure 2 MPC trajectory tracking principle diagram based on incremental double Q learning of the present application;
[0078] Figure 3 Incremental discretization schematic diagram of the action space of the present application;
[0079] Figure 4The state space increment type discretization principle diagram of the application;
[0080] Figure 5 The method flow chart of acquiring the online adaptive MPC controller of the application. DETAILED DESCRIPTION
[0081] The application will be further described in detail below in combination with the drawings and embodiments.
[0082] The application realizes online adaptive adjustment of the MPC weight matrix through the increment type double Q-learning method, improves the control precision of the adaptive MPC trajectory tracking controller, and enhances the adaptability of the adaptive MPC trajectory tracking controller to the dynamic operation environment. Through increment type discretization of the action space and the state space of the reinforcement learning intelligent agent, the method avoids full discretization of the state and action space of the intelligent agent through the way of discretizing only the key area, and improves the learning efficiency of the reinforcement learning.
[0083] As shown in the figure, it is the MPC trajectory tracking principle diagram based on the increment type double Q-learning of the application. Figure 2 The adaptive model predictive control method based on the increment type double Q-learning of the application has the characteristics that it comprises the following steps:
[0084] 1) According to the physical characteristics of the differential mobile robot, a kinematic model of the nonholonomic constraint differential mobile robot is established; according to the reference trajectory, the expected pose information of the mobile robot is acquired, and a trajectory tracking error model is constructed in combination with the kinematic model;
[0085] As shown in the figure, it is the kinematic model principle diagram of the wheeled differential mobile robot of the application. Figure 1 According to the physical characteristics of the differential mobile robot, a kinematic model of the nonholonomic constraint differential mobile robot is established, and specifically:
[0086] The state vector of the kinematic model is (x, y, θ), and the decision vector is (v, ω); then the kinematic model of the nonholonomic constraint differential mobile robot is :
[0087]
[0088] Among them, is the velocity of the mobile robot along the X axis and the Y axis, represents the angular velocity.
[0089] In step 1), according to the reference trajectory, the expected pose information of the mobile robot is acquired, and a trajectory tracking error model is constructed in combination with the kinematic model, and specifically:
[0090] 1-1) In the global coordinate system, the model of the reference trajectory is set as:
[0091] η r (t)=(x r (t),y r (t),θ r (t)) T
[0092] u r (t)=(v r (t),ω r (t)) T
[0093]
[0094] Where, η r (t) represents the desired pose sequence of the open trajectory, x r (t),y r (t),θ r (t) represents the desired position along the X-axis, the desired position along the Y-axis, and the desired attitude, respectively. r (t) represents the desired velocity sequence of the reference trajectory, v r (t),ω r (t) represent the desired linear velocity and the desired angular velocity, respectively;
[0095] Based on the pose information of the desired trajectory with time constraints, the desired velocity v of the mobile robot is calculated. r (t) and desired angular velocity ω r (t);
[0096] 1-2) Obtain the error model between the kinematic model and the reference trajectory of the nonholonomic constrained differential mobile robot in the global coordinate system as follows:
[0097]
[0098] Where, x e (t) represents the position error along the X-axis, y e (t) represents the position error in the Y-axis direction, θ e (t) represents the heading angle error. This represents the change in velocity along the X-axis. This is expressed as the velocity change of the Y-axis defense line. Let v(t) represent the change in angular velocity, and v(t) be the robot's desired linear velocity. r (t) represents the robot's actual linear velocity, ω r ω(t) represents the robot's desired angular velocity, and ω(t) represents the robot's actual angular velocity.
[0099] 1-3) The input vector of the control error model is:
[0100]
[0101] where u e (t) is the input vector of trajectory tracking control, including linear velocity component u1(t) and angular velocity component u2(t), denotes the angular velocity variation, v(t) is the desired linear velocity of the robot, v r (t) is the actual linear velocity of the robot, ω r (t) is the desired angular velocity of the robot, and ω(t) is the actual angular velocity of the robot.
[0102] 1-4) Obtain the trajectory tracking error model, that is:
[0103]
[0104] 2) Based on the trajectory tracking error model, construct the prediction model of the trajectory tracking model predictive controller of the mobile robot, and realize high-performance trajectory tracking control according to the prediction model;
[0105] 2-1) According to the trajectory tracking error model, the state transition matrix of the error model is obtained where T is the sampling period;
[0106] Then the prediction model of the model predictive controller is constructed as:
[0107]
[0108] where the state vector is The control vector u = (v, ω) T , C k,t = (I3,0 3×3 );
[0109] 2-2) The objective function J(k) of the trajectory tracking model predictive controller is designed as:
[0110]
[0111] where Q and R are state weight matrices, Q f is an output weight matrix, N p and N c represent the prediction time domain and the control time domain, respectively;
[0112] 2-3) Therefore, the optimization problem of the prediction model of the model predictive controller is represented as:
[0113] min J(k)
[0114] s.t u min ≤ u(k) ≤ u max
[0115] Δu min ≤Δu(k)≤Δu max
[0116] wherein u(k) represents the actual velocity vector value of the robot, u min and u max represent the maximum linear velocity and the minimum linear velocity respectively, Δu(k) represents the acceleration vector of the robot, Δu min and Δu max represent the minimum acceleration and the maximum acceleration of the robot respectively.
[0117] wherein the prediction model is a state equation of the trajectory tracking control and is also a core model of the whole trajectory tracking control. The trajectory tracking implements the control according to the prediction model (state equation).
[0118] 3) Incremental discretization is performed on the action space and the state space of the double Q learning agent, only the new action and the new state of each iteration are added to the discretized set, and the regions where the state and the action do not change are discretized, so that the action space and the state space of the agent are discretized in whole.
[0119] As shown in Figures 3-4 , it is an incremental discretization schematic diagram of the action space and the state space of the application;
[0120] Specifically, the incremental discretization of the action space and the state space of the double Q learning agent specifically includes the following steps:
[0121] 3-1) defining the initial mobile robot action space K and state space X;
[0122] wherein the constituent elements of the state space are state vectors, and the state vector is x=[x e (t),y e (t),θ e (t)] T , which is composed of the position error, the velocity error and the heading error of the mobile robot, and is composed of the state vector to form the state space X;
[0123] The action vector is k=[k r ,k q ,k qf ] T , and the action space K is composed of the action vector.
[0124] 3-2) In the early stage of online training, the state and action space are roughly discretized, that is, the fixed interval of the velocity and the position according to the physical constraints of the system, and the initial value of the state and the action are pre-set;
[0125] 3-3) Whenever the robot enters a new state x, check whether this state already exists in the discretized state set X;
[0126] If not, i.e. the state differs from the states in the existing state set X by more than a pre-set threshold d t+1 > p, then add this state to the discretized state set X, i.e.
[0127]
[0128] X = {X U X t+1} <- x t+1
[0129] If the state exists in the state set X and appears continuously for N times, then consider that the state is near the key region of local optimal solution, and only this part is discretized;
[0130] 3-4) For each selected action k, check whether the state already exists in the discretized action set K; if the new action differs from the actions in the existing action set K significantly, then add it to the action set K; if the continuous N actions do not cause the transition of the system state, consider that the action is near the local optimal region, and only this part is discretized.
[0131] 3-5) In steps 3-1) to 3-4), only the key region is discretized, which avoids the full discretization of the state and action space of the agent, and improves the learning efficiency of reinforcement learning.
[0132] During the running of the algorithm, the state space and action space are discretized non-uniformly according to the interaction results of the robot and the environment. The places that are discretized in detail are mainly the regions that are frequently visited by the robot or have a significant impact on decision-making.
[0133] 4) Through real-time interaction between the agent and the environment and online iterative learning, the state and action space of the double Q learning agent are continuously optimized; through learning, the optimal action k is selected each time iteration, the weight matrix of the MPC is adaptively adjusted, the control accuracy of the MPC controller is improved, and an online adaptive MPC controller is obtained;
[0134] Through the active learning of the fourth step, the state space and action space of the double Q learning are continuously expanded. When the amplitude of the state space and action space is continuously increased, the probability of selecting the optimal action will be larger. Each iteration will be the optimal action at the moment, and the optimal action that controls the MPC is the optimal weight, and the trajectory tracking effect is the optimal solution at the moment.
[0135] Specifically, as Figure 5As shown, the method flow chart for acquiring an online adaptive MPC controller of the application, step 4) includes the following steps:
[0136] 4-1) Set the initialization parameters of the system;
[0137] 4-2) In each iteration, use the e-greedy strategy to select the action k t to balance between exploration and utilization;
[0138] 4-3) Update the temporary memory, save the current state x t and the selected action k t to the temporary memory M, that is:
[0139] M←τ i =(x t ,u t ,r t ,x t+1 )
[0140] 4-4) Perform the action k t , obtain the new state x t+1 and the reward r t+1 , then the constructed reward function is:
[0141]
[0142] Where β is a constant parameter, x t represents the current state, x r represents the desired state, and r t is the reward value;
[0143] 4-5) Check whether the last N memories (M t , M t-1 , …, M t-N ) are consistent to determine whether the system is in a stable state, if M t =M t-1 =...=M t-N , it is considered that the system is in a stable state, and the incremental discretization mechanism is started, and further discretization is performed on the stable region of the state;
[0144] 4-6) If the new state x t+1 of the system corresponds to the new memory M t+1 ≠M t , the new state x t+1 provides valuable new information for the system, and it is added to the state space set X;
[0145] The valuable new information can be understood as follows: the system originally may have only had 5 states. This new state is different from the previous 5 states, so it is added as a new state, and the state space becomes 6 states.
[0146] This section describes the implementation process, specifically the incremental active learning process. It is the physical implementation of the following formula, or a method for translating theory into practice.
[0147]
[0148] X={X∪X t+1}←x t+1
[0149] 4-7) Calculate the time error δ, and determine the sample priority p based on the absolute value of the TD error. t ;
[0150] a. If you choose to update Q A By calculating Q A From the entire action space K, we select the next state x. t+1 The possible optimal action k′ is:
[0151] k′=argm K axQ A (x t+1 ,K)
[0152] b. Using Q B Estimate the Q value of this action:
[0153] y t =r t +γQ B (x t+1 ,k′)
[0154] y t That is, the target Q value corresponding to time step t, where γ is the discount factor and γ∈(0,1];
[0155] c. Calculate TD error:
[0156] δ t =y t -Q A (x t ,k t )
[0157] d. If you choose to update Q B So the same process is just using Q. B To update, and use Q A To calculate the target value, Q is used alternately. A and QB The double Q learning method can effectively reduce overestimation of future reward values;
[0158] e. Then, the priority is calculated according to the absolute value of the TD error, that is,
[0159] p t =|δ t |
[0160] 4-8) Store the sample (s t ,a t ,r t ,s t+1 ) into the experience replay pool, and determine the priority of the sample according to the priority p t ;
[0161] 4-9) Update the state and action of the agent k=[k r ,k q ,k qf ] T through the above steps, and then complete the adaptive setting of the MPC weight matrix to improve the dynamic environment adaptability of the MPC trajectory tracking controller.
[0162] 5) Realize high-precision autonomous trajectory tracking control of the mobile robot through the online adaptive MPC controller.
[0163] A system for mobile robot trajectory tracking, comprising:
[0164] 1) A GPS integrated navigation system for obtaining real-time position, speed and attitude information of the robot;
[0165] 2) A trajectory tracking controller, the controller being configured to execute the above-mentioned incremental double Q-learning enhanced model predictive control method to complete trajectory tracking control of the robot according to the state information obtained by the sensor.
[0166] Further comprising a memory and a processor; the memory is used to store a computer program; the processor is used to realize the adaptive model predictive control method based on incremental double Q learning when the computer program is executed.
[0167] A computer readable storage medium, the storage medium stores a computer program, when the computer program is executed by a processor, an adaptive model predictive control method based on incremental double Q learning is realized.
[0168] In conclusion, the method fuses doubleQ-learning into the traditional model prediction control technology, plays the advantages of doubleQ-learning and environment interactive learning, and the advantages of the traditional MPC controller, and the combination of the two can effectively improve the precision of the trajectory tracking control of the mobile robot.
[0169] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit. The above is only an embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent flow transformation using the content of the specification and the drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. An adaptive model predictive control method based on incremental double Q-learning, characterized in that, It comprises the following steps: 1) According to the physical characteristics of the differential mobile robot, a kinematic model of the nonholonomic constraint differential mobile robot is established; the desired pose information of the mobile robot is obtained according to the reference trajectory, and a trajectory tracking error model is constructed in combination with the kinematic model; 2) Based on the trajectory tracking error model, a prediction model of the trajectory tracking model predictive controller of the mobile robot is constructed, and high-performance trajectory tracking control is realized according to the prediction model; 3) The action space and state space of the double Q learning agent are incrementally discretized, and only the new action and new state of each iteration are added to the discretized set; the areas where the state and action do not change are discretized; 4) Through real-time interaction and online iterative learning between the agent and the environment, the state and action space of the double Q learning agent are continuously optimized; Through learning, the optimal action k is selected each time, the weight matrix of the MPC is adaptively adjusted, and an online adaptive MPC controller is obtained; 5) Through the online adaptive MPC controller, high-precision autonomous trajectory tracking control of the mobile robot is realized.
2. The incremental double Q-learning based adaptive model predictive control method according to claim 1, wherein, According to the physical characteristics of the differential mobile robot, the kinematic model of the nonholonomic constraint differential mobile robot is established, specifically: The state vector of the kinematics model is (x, y, θ), and the decision vector is (v, ω); then the kinematics model of the nonholonomic constraint differential mobile robot is x = v cos θ wherein, is the velocity of the mobile robot along the X and Y axes, represents the angular velocity.
3. The incremental double Q-learning based adaptive model predictive control method of claim 1, wherein, According to the reference trajectory, the desired pose information of the mobile robot is obtained, and a trajectory tracking error model is constructed in combination with the kinematic model, specifically: 1-1) In the global coordinate system, the model of the reference trajectory is set as: η r (t) = (x r (t), y r (t), θ r (t)) T u r (t) = (v r (t), ω r (t)) T where η r (t) is the desired pose sequence of the reference trajectory, x r (t), y r (t), θ r (t) are the desired position in X-axis direction, the desired position in Y-axis direction, and the desired attitude, respectively, u r (t) represents the desired velocity sequence of the reference trajectory, v r (t), ω r (t) represent the desired linear velocity and the desired angular velocity, respectively; Based on the pose information with time constraint of the desired trajectory, the desired velocity v of the mobile robot is obtained r (t) and the desired angular velocity ω r (t); 1-2) The kinematic model of the nonholonomic constraint differential mobile robot in the global coordinate system and the error model of the reference trajectory are obtained as: where x e (t) is the position error along the X-axis, y e (t) is the position error along the Y-axis, θ e (t) is the heading angle error, is the velocity change along the X-axis, is the velocity change along the Y-axis, is the angular velocity change, v(t) is the actual linear velocity of the robot, v r (t) is the desired linear velocity of the robot, ω r (t) is the desired angular velocity of the robot, and ω(t) is the actual angular velocity of the robot. 1-3) The input vector of the error model is: wherein u e (t) is the input vector of trajectory tracking control, including linear velocity component u1(t) and angular velocity component u2(t), is expressed as angular velocity change amount, v(t) is the actual linear velocity of the robot, v r (t) is the desired linear velocity of the robot, ω r (t) is the desired angular velocity of the robot, ω(t) is the actual angular velocity of the robot; 1-4) The trajectory tracking error model is obtained, that is:
4. The incremental double Q-learning based adaptive model predictive control method of claim 1, wherein, Step 2), specifically: 2-1) is obtained according to a trajectory tracking error model, a state transition matrix of the error model is where T is a sampling period; Then the prediction model of the model predictive controller is constructed as: where the state vector Control vector u = (v, ω) T , C k,t = (I3,0 3×3 ); 2-2) The objective function J(k) of the trajectory tracking model predictive controller is designed as: where Q and R are state weight matrices, Q f is an output weight matrix, N p and N c represent the prediction horizon and control horizon, respectively. 2-3) Therefore, the optimization problem of the prediction model of the model predictive controller is represented as: min J(k) s.t u min ≤u(k)≤u max Δu min ≤ Δu(k) ≤ Δu max where u(k) represents the actual velocity vector value of the robot, u min and u max represent the minimum and maximum linear velocities, respectively, and Δu(k) represents the acceleration vector of the robot, Δu min and Δu max represent the minimum and maximum acceleration of the robot, respectively.
5. The incremental double Q-learning based adaptive model predictive control method of claim 1, wherein, Step 3) comprises the following steps: 3-1) Define the initial action space K and state space X of the mobile robot; 3-2) In the early stage of online training, the state and action space are discretized roughly, that is, the fixed range of velocity and position according to the physical constraints of the system, and the initial values of the state and action are pre-set; 3-3) When the robot enters a new state x, check whether the state exists in the discretized state set; If not, i.e. the state differs from the states in the existing set of states by more than a pre-set threshold d t+1 > p, then the state is added to the set of discretized states X, i.e. X = { X U X t+1} <- x t+1 If the state exists in the state set X and appears continuously for N times, it is considered that the state near the local optimal solution is the key area, and only this part is discretized; 3-4) For each selected action k, check whether the state exists in the discretized action set; if the new action is significantly different from the existing action set, add it to the action set K; if the continuous N actions do not cause the state of the system to change, it is considered that the action near the region is locally optimal, and only this part is discretized.
6. The incremental double Q-learning based adaptive model predictive control method of claim 5, wherein, Step 3-1), specifically: The constitutive elements of the state space are the state vectors, which are x = [x e (t), y e (t), θ e (t)] T , which are composed of the position error, velocity error, and heading error of the mobile robot and are composed of the state vectors to form the state space X; The action vector is k = [k r ,k q ,k qf ] T and the action space K is composed of action vectors.
7. The incremental double Q-learning based adaptive model predictive control method of claim 1, wherein, The past experience is sampled by using a time-error-based priority experience sampling method to accelerate the training speed of the double Q learning.
8. The incremental double Q-learning based adaptive model predictive control method of claim 1, wherein, The step 4) is specifically: 4-1) setting initialization parameters of the system; 4-2) At each iteration, use an ò-greedy policy to select action k t to strike a balance between exploration and exploitation; 4-3) update the short-term memory with the current state x t and the selected action k t into the short-term memory M, That is: M <- τ i = (x t , u t , r t , x t+1 ) 4-4) Perform action k t , get new state x t+1 and reward r t+1 The reward function constructed is: where β is a constant parameter, x t represents the current state, x r represents the desired state, r t is the reward value; 4-5) Check whether the latest N memories (M t , M t-1 ,..., M t-N ) are consistent to determine whether the system is in a stable state. If M t = M t-1 =... = M t-N , it is considered that the system is in a stable state, and an incremental discretization mechanism is started to further discretize the stable state region. 4-6) If the system new state x t+1 The corresponding new memory M t+1 ≠ M t The new state x t+1 Provides valuable new information to the system, it is added to the state space set X; 4-7) Calculate the time error δ and determine the priority p of the sample according to the absolute value of the TD error t ; 4-8) store the samples (s t ,a t ,r t ,s t+1 ) into the experience revisit pool and determine the priority of the samples according to their priority p t ; 4-9) The agent state is updated by the above steps and actions k = [k r ,k q ,k qf ] T , and the adaptive setting of the MPC weight matrix is completed.
9. The incremental double Q-learning based adaptive model predictive control method of claim 8, wherein, The memory is used for storing a computer program; and the processor is used for realizing the adaptive model predictive control method based on the incremental double Q learning when the computer program is executed.
10. A computer-readable storage medium, characterized in that, The storage medium has the computer program stored thereon, and the adaptive model predictive control method based on the incremental double Q learning is realized when the computer program is executed by the processor.