Path tracking method, system and equipment based on RL-MPC four-wheel inspection robot and medium

Abstract

The invention discloses a path tracking method, system and device based on an RL-MPC four-wheel inspection robot and a medium. The method comprises the steps that the current path tracking error, speed and course angle information of the four-wheel inspection robot is acquired; taking the current path tracking error, speed and course angle information of the four-wheel inspection robot as rewards for Q-learning method training, and dynamically calculating a prediction time domain Np and a control time domain Nc of the MPC; the target function is solved according to the prediction time domain Np and the control time domain Nc of the MPC to obtain front wheel steering parameters, the four-wheel inspection robot is controlled to track the path, and the path tracking effect of the method, the system, the equipment and the medium is good.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent optimization control algorithms, and relates to a path tracking method, system, equipment and medium based on an RL-MPC four-wheel inspection robot. Background Art

[0002] As an important component of the smart electric field, the intelligent inspection robot has been widely used in the power field. The essence of achieving its navigation is to make the robot follow a planned route through the steering of the front wheels of the controller, so a suitable control algorithm is needed to solve the path tracking problem.

[0003] Model Predictive Control (MPC) is a predictive control algorithm that avoids temporary adjustments through forward-looking optimization, so that path tracking not only depends on the current state, but also takes into account future behavior, thereby reducing oscillation or unstable steering. The algorithm can handle state constraints (such as vehicle speed) and control constraints (such as turning angle) so that the tracking process does not exceed physical limitations. On the other hand, in traditional path tracking methods, path planning and control are two independent processes, while in MPC, planning and control are solved together through an optimization framework. When optimizing the control input, MPC will consider the target path and the current state at the same time, so that the path planning and control processes are seamlessly connected. Therefore, MPC is a high-precision and stable solution to the wheeled robot following problem.

[0004] The core of the model predictive control algorithm is to solve the objective function. Among them, the two most important parameters that affect the solution of the objective function are the prediction horizon and the control horizon. The prediction horizon is the time range for prediction into the future, which determines the length of time the algorithm considers future control inputs and system states. The control horizon is the length of time for the control inputs selected and optimized by the algorithm in each optimization, which determines the scope of the actual execution of control decisions in each optimization process. Reasonable time domain settings are conducive to the algorithm to achieve a balance between solution accuracy and calculation speed, so it is crucial to select a set of time domain parameters that are suitable for the current system model.

[0005] A common method for selecting the time domain is to first set a set of values ​​based on experience and then adjust them through repeated experiments. However, this method is inefficient and not applicable to different models. Therefore, a solution that can dynamically optimize parameters is needed to improve the performance of the optimization solution and further improve the effect of path following. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a path tracking method, system, device and medium based on an RL-MPC four-wheel inspection robot. By dynamically adjusting algorithm parameters, the method, system, device and medium can improve the path tracking effect.

[0007] To achieve the above object, the present invention discloses a path tracking method based on a RL-MPC four-wheel inspection robot, comprising:

[0008] Establish the basic dynamics model of the four-wheel inspection robot's motion;

[0009] The current path tracking error, speed and heading angle of the four-wheel inspection robot are calculated according to the basic dynamics model, and the Q-learning method of reinforcement learning is introduced as the state for training to obtain the values ​​of the prediction time domain Np and the control time domain Nc of the MPC;

[0010] The objective function of the control parameter process of the four-wheel inspection robot is solved according to the values ​​of the prediction time domain Np and the control time domain Nc of the MPC, and the front wheel turning angle of the robot is obtained. The movement of the four-wheel inspection robot is controlled according to the front wheel turning angle of the robot.

[0011] The further improvement of the path tracking method based on the RL-MPC four-wheel inspection robot of the present invention is:

[0012] Furthermore, the reward function in the Q-learning method training process is:

[0013] R=w1(-e(t))+w2Δu(t)+w3Δ 2 u(t)

[0014] Where e(t) is the current path error, Δu(t) is the change in control input, Δ 2 u(t) is the change of control input change, w i is the weight coefficient.

[0015] Furthermore, the update formula during the Q-learning method training process is:

[0016] Q(S t ,A t )←Q(S t ,A t )+α(R t +γmax (A′) Q(S t +1,A′)-Q(S t ,A t ))

[0017] Among them, Q(S t ,A t ) is the Q value of the action selected in the current state, α is the learning rate, which controls the step size of the Q value update, γ is the discount factor, R t is the immediate reward after taking an action in the current state, max (A′) Q(S t+1, A′) is the maximum Q value in the next state.

[0018] Furthermore, according to the values ​​of MPC in the prediction time domain Np and control time domain Nc, the objective function in solving the control parameters of the four-wheel inspection robot is:

[0019]

[0020] Among them, Q, R is the weight matrix, ρ is the weight coefficient, η is the actual path output in the prediction time domain, η ref is the reference position, and Δu is the control increment in the control time domain.

[0021] The present invention discloses a path tracking system based on a RL-MPC four-wheel inspection robot, comprising:

[0022] Establish a module for establishing the basic dynamics model of the four-wheel inspection robot;

[0023] A parameter optimization module is used to calculate the current path tracking error, speed and heading angle of the four-wheel inspection robot according to the basic dynamics model, and use this as the state to introduce the Q-learning method of reinforcement learning for training, so as to obtain the values ​​of the prediction time domain Np and the control time domain Nc of the MPC;

[0024] The solving control module is used to solve the objective function in the control parameter process of the four-wheel inspection robot according to the values ​​of the prediction time domain Np and the control time domain Nc of the MPC, obtain the front wheel turning angle of the robot, and control the movement of the four-wheel inspection robot according to the front wheel turning angle of the robot.

[0025] The further improvement of the path tracking system based on the RL-MPC four-wheel inspection robot of the present invention is:

[0026] Furthermore, the reward function in the Q-learning method training process is:

[0027] R=w1(-e(t))+w2Δu(t)+w3Δ 2 u(t)

[0028] Where e(t) is the current path error, Δu(t) is the change in control input, Δ 2 u(t) is the change of control input change, w i is the weight coefficient.

[0029] Furthermore, the update formula during the Q-learning method training process is:

[0030] Q(S t ,A t )←Q(S t ,At )+α(R t +γmax (A′) Q(S t +1,A′)-Q(S t ,A t ))

[0031] Among them, Q(S t ,A t ) is the Q value of the action selected in the current state, α is the learning rate, which controls the step size of the Q value update, γ is the discount factor, R t is the immediate reward after taking an action in the current state, max (A′) Q(S t +1, A′) is the maximum Q value in the next state.

[0032] Furthermore, according to the values ​​of MPC in the prediction time domain Np and control time domain Nc, the objective function in solving the control parameters of the four-wheel inspection robot is:

[0033]

[0034] Among them, Q, R is the weight matrix, ρ is the weight coefficient, η is the actual path output in the prediction time domain, η ref is the reference position, and Δu is the control increment in the control time domain.

[0035] The present invention discloses a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the path tracking method based on the RL-MPC four-wheel inspection robot are implemented.

[0036] The present invention discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the path tracking method based on an RL-MPC four-wheel inspection robot are implemented.

[0037] The present invention has the following beneficial effects:

[0038] The path tracking method, system, device and medium based on RL-MPC four-wheel inspection robot of the present invention introduces reinforcement learning into the parameter adjustment of model predictive control during specific operation, directly avoiding the complexity and imprecision of manual parameter adjustment. In addition, the MPC method based on reinforcement learning can automatically learn and adjust the optimal configuration of the prediction time domain and the control time domain according to real-time feedback, so as to better adapt to different dynamic environments and task requirements and improve the tracking ability of the robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0040] Figure 1 The present invention is a flow chart of the method. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0043] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an" and "the" are intended to include plural forms unless the context clearly indicates otherwise.

[0044] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0045] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0046] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0047] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0048] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0049] Embodiment 1

[0050] The main links of MPC in path tracking tasks include system modeling, state estimation, prediction model, optimization solution, constraints and feedback mechanism. For the path tracking problem, the objective function is:

[0051]

[0052] Among them, Q, R is the weight matrix, ρ is the weight coefficient, η is the actual path output in the prediction time domain, that is, the coordinate point, η refis the reference position, and Δu is the control increment in the control time domain. The first part of the above formula reflects the effect of path tracking, ensuring that the error with the reference path is as small as possible, and the second part reflects the ability of smooth movement, ensuring that the front wheel angle does not jump when turning. In addition, during the optimization process, physical constraints are set, including motion constraints (maximum speed, angle, acceleration, etc.); path constraints (delimiting the effective area of ​​the path); dynamic constraints (maximum steering angle, minimum steering radius), etc. MPC needs to meet the above constraints while solving the objective function.

[0053] After the optimization solution, MPC calculates a control sequence and executes the first control input in the control sequence, using it as the front-wheel steering drive for the robot to move, and re-predicts and optimizes based on the new state, thereby continuously updating the control input.

[0054] In order to optimize the prediction time domain Np and control time domain Nc in the objective function, the Q-learning method of reinforcement learning is introduced. The following two aspects are mainly considered when designing the reward function:

[0055] 1) Path tracking error, which reflects the error between the robot’s current position and the target path, expressed as Euclidean distance.

[0056] 2) Control the smoothness of input to reflect the degree of steering change, avoid excessive control of the front wheels, and ensure smooth driving.

[0057] Q-learning is a model-free reinforcement learning method that learns the optimal action strategy through the interaction between the agent and the environment. The goal of the agent is to select the optimal combination of prediction time domain and control time domain to minimize the path error and the smoothness of the control input.

[0058] The specific process is as follows:

[0059] 1) State: In Q-learning, the state represents the state of the system at a certain moment. In the path tracking task, the state can be defined as:

[0060] S={e(t),v(t),θ(t)}

[0061] Where e(t) is the current path tracking error, v(t) is the velocity, and θ(t) is the heading angle.

[0062] 2) Action: An action is a behavior chosen by an agent in each state. In this invention, an action is to adjust Np and Nc of MPC. We can express an action as:

[0063] A=(Np,Nc)

[0064] 3) Reward: Reward is the feedback that the agent gets based on the current state and the selected action, reflecting the quality of the action selected. The reward is defined as:

[0065] R=w1(-e(t))+w2Δu(t)+w3Δ 2 u(t)

[0066] Where e(t) is the current path error, Δu(t) is the change in control input (i.e., the turning angle), Δ 2 u(t) is the change in control input (i.e., acceleration), w i is the weight coefficient used to adjust the impact of path error and control input smoothness on the reward.

[0067] The core of Q-learning is to update the Q-value table and guide the learning process through the reward function. The Q-value represents the long-term reward of choosing an action in a specific state. The update formula of Q-learning is:

[0068] Q(S t ,A t )←Q(S t ,A t )+α(R t +γmax (A′) Q(S t +1,A′)-Q(S t ,A t ))

[0069] Among them, Q(S t ,A t ) is the Q value of the action selected in the current state, is the learning rate, which controls the step size of the Q value update, γ is the discount factor, which controls the impact of future rewards, and R t is the immediate reward after taking an action in the current state, max (A′) Q(S t +1,A′) is the maximum Q value in the next state, that is, the optimal reward in the future. Furthermore, considering that in the path tracking problem, it is necessary to ensure tracking accuracy and driving stability. Therefore, the reward function considers two factors: the smaller the path error, the greater the reward, and the path error is treated as a negative value; on the other hand, in order to avoid sharp turns, the change in control input and acceleration are used to quantify the smoothness of control. Smooth control input will receive higher rewards, specifically:

[0070] 1) Euclidean distance between the current state and the target path:

[0071] e 2 (t) = (x(t) - x g (t)) 2 +(y(t)-yg (t)) 2

[0072] Among them, (x,y) is the coordinate of the current position, (x,y) g are the coordinates of the target path.

[0073] 2) The change in control input is:

[0074] Steering changes:

[0075] Δu(t)=u(t)-u(t-1)

[0076] Steering acceleration:

[0077] u(t)=Δu(t)-Δu(t-1)

[0078] Based on the above, Q-learning training is performed. First, the Q value table is initialized and the initial prediction time domain Np and control time domain Nc values ​​are selected. In each control cycle, the new control input is calculated based on the current state (e(t), v(t), θ(t)) and the selected action (Np, Nc), and path tracking is performed. The immediate reward R is calculated based on the path error and the smoothness of the control input. The Q value table is updated with the Q-learning formula, the optimal prediction time domain and control time domain combination is selected based on the Q value, and the training process is repeated.

[0079] e 2 (t) = (x(t) - x g (t)) 2 +(y(t)-y g (t)) 2

[0080] Among them, (x,y) is the coordinate of the current position, (x,y) g are the coordinates of the target path.

[0081] 3) The change in control input can be expressed as:

[0082] The steering changes are:

[0083] Δu(t)=u(t)-u(t-1)

[0084] The steering acceleration is:

[0085] u(t)=Δu(t)-Δu(t-1)

[0086] Based on the above, Q-learning training is performed. First, the Q value table is initialized and the initial prediction time domain Np and control time domain Nc values ​​are selected. In each control cycle, the new control input is calculated based on the current state (e(t), v(t), θ(t)) and the selected action (Np, Nc), and path tracking is performed. The immediate reward R is calculated based on the path error and the smoothness of the control input. The Q value table is updated with the Q-learning formula, the optimal prediction time domain and control time domain combination is selected according to the Q value, and the training process is repeated.

[0087] Embodiment 2

[0088] The path tracking system based on the RL-MPC four-wheel inspection robot of the present invention comprises:

[0089] Establish a module for establishing the basic dynamics model of the four-wheel inspection robot;

[0090] A parameter optimization module is used to calculate the current path tracking error, speed and heading angle of the four-wheel inspection robot according to the basic dynamics model, and use this as the state to introduce the Q-learning method of reinforcement learning for training, so as to obtain the values ​​of the prediction time domain Np and the control time domain Nc of the MPC;

[0091] The solving control module is used to solve the objective function in the control parameter process of the four-wheel inspection robot according to the values ​​of the prediction time domain Np and the control time domain Nc of the MPC, obtain the front wheel turning angle of the robot, and control the movement of the four-wheel inspection robot according to the front wheel turning angle of the robot.

[0092] As an embodiment of the present invention, the reward function in the Q-learning method training process is:

[0093] R=w1(-e(t))+w2Δu(t)+w3Δ 2 u(t)

[0094] Where e(t) is the current path error, Δu(t) is the change in control input, Δ 2 u(t) is the change of control input change, w i is the weight coefficient.

[0095] As an implementation mode of the present invention, the update formula in the Q-learning method training process is:

[0096] Q(S t ,A t )←Q(S t ,A t )+α(R t +γmax (A′) Q(S t+1,A′)-Q(S t ,A t ))

[0097] Among them, Q(S t ,A t ) is the Q value of the action selected in the current state, is the learning rate, controls the step size of the Q value update, γ is the discount factor, R t is the immediate reward after taking an action in the current state, max (A′) Q(S t +1, A′) is the maximum Q value in the next state.

[0098] As an embodiment of the present invention, the objective function in the process of solving the control parameters of the four-wheel inspection robot according to the values ​​of the prediction time domain Np and the control time domain Nc of the MPC is:

[0099]

[0100] Among them, Q, R is the weight matrix, ρ is the weight coefficient, η is the actual path output in the prediction time domain, η ref is the reference position, and Δu is the control increment in the control time domain.

[0101] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0102] Embodiment 3

[0103] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the path tracking method of the four-wheel inspection robot based on RL-MPC are implemented, for example, including: using the current path tracking error, speed and heading angle information of the four-wheel inspection robot as the basis for reward setting to perform Q-learning training, and obtain the prediction time domain Np and control time domain Nc of the MPC algorithm; then solving the MPC objective function with dynamic time domain parameters to obtain the front wheel steering parameters, and controlling the four-wheel inspection robot to track the target path movement. Among them, the memory may include a memory, such as a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk memory, etc.; the processor, the network interface, and the memory are interconnected through an internal bus, and the internal bus may be an industrial standard architecture bus, a peripheral component interconnection standard bus, an extended industrial standard structure bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory is used to store programs. Specifically, the program may include a program code, and the program code includes computer operation instructions. The memory may include a memory and a non-volatile memory, and provide instructions and data to the processor.

[0104] Embodiment 4

[0105] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the path tracking method of the four-wheel inspection robot based on RL-MPC are implemented, for example, including: using the current path tracking error, speed and heading angle information of the four-wheel inspection robot as the basis for reward setting to perform Q-learning training, and obtain the prediction time domain Np and control time domain Nc of the MPC algorithm; and then solving the above-mentioned MPC objective function with dynamic time domain parameters to obtain the front wheel steering parameters, and control the four-wheel inspection robot to track the target path movement. Specifically, the computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include read-only memory (ROM), hard disk, flash memory, optical disk, disk, etc.

[0106] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0107] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0108] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0110] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and disclosure of the invention. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0111] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

[0112] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A path tracking method for a four-wheel inspection robot based on RL-MPC, characterized in that: include: Establish the basic dynamics model of the four-wheel inspection robot's motion; The current path tracking error, speed and heading angle of the four-wheel inspection robot are calculated according to the basic dynamics model, and the Q-learning method of reinforcement learning is introduced as the state for training to obtain the values ​​of the prediction time domain Np and the control time domain Nc of the MPC; The objective function of the control parameter process of the four-wheel inspection robot is solved according to the values ​​of the prediction time domain Np and the control time domain Nc of the MPC, and the front wheel turning angle of the robot is obtained. The movement of the four-wheel inspection robot is controlled according to the front wheel turning angle of the robot.

2. The path tracking method based on the RL-MPC four-wheel inspection robot according to claim 1 is characterized in that: The reward function of the Q-learning method during training is: R=w1(-e(t))+w2Δu(t)+w3Δ 2 u(t) Where e(t) is the current path error, Δu(t) is the change in control input, Δ 2 u(t) is the change of control input change, w i is the weight coefficient.

3. The path tracking method based on RL-MPC four-wheel inspection robot according to claim 1 is characterized in that: The Q-learning method updates the formula during training as follows: Q(S t ,A t )←Q(S t ,A t )+α(R t +γmax (A′) Q(S t +1,A′)-q(S t ,A t )) Among them, Q(S t ,A t ) is the Q value of the action selected in the current state, is the learning rate, controls the step size of the Q value update, γ is the discount factor, R t is the immediate reward after taking an action in the current state, max (A′) Q(S t +1, A′) is the maximum Q value in the next state.

4. The path tracking method based on RL-MPC four-wheel inspection robot according to claim 1 is characterized in that: According to the values ​​of MPC prediction time domain Np and control time domain Nc, the objective function in solving the control parameters of the four-wheel inspection robot is: Among them, Q, R is the weight matrix, ρ is the weight coefficient, η is the actual path output in the prediction time domain, η ref is the reference position, and Δu is the control increment in the control time domain.

5. A path tracking system based on RL-MPC four-wheel inspection robot, characterized in that: include: Establish a module for establishing the basic dynamics model of the four-wheel inspection robot; A parameter optimization module is used to calculate the current path tracking error, speed and heading angle of the four-wheel inspection robot according to the basic dynamics model, and use this as the state to introduce the Q-learning method of reinforcement learning for training, so as to obtain the values ​​of the prediction time domain Np and the control time domain Nc of the MPC; The solving control module is used to solve the objective function in the control parameter process of the four-wheel inspection robot according to the values ​​of the prediction time domain Np and the control time domain Nc of the MPC, obtain the front wheel turning angle of the robot, and control the movement of the four-wheel inspection robot according to the front wheel turning angle of the robot.

6. The path tracking system based on the RL-MPC four-wheel inspection robot according to claim 5 is characterized in that: The reward function during the Q-learning method training process is: R=w1(-e(t))+w2Δu(t)+w3Δ 2 u(t) Where e(t) is the current path error, Δu(t) is the change in control input, Δ 2 u(t) is the change of control input change, w i is the weight coefficient.

7. The path tracking system based on the RL-MPC four-wheel inspection robot according to claim 5, characterized in that: The update formula during the Q-learning method training process is: Q(S t ,A t )←Q(S t ,A t )+α(R t +γmax (A′) q(S t +1,A′)-Q(S t ,A t )) Among them, Q(S t ,A t ) is the Q value of the action selected in the current state, α is the learning rate, which controls the step size of the Q value update, γ is the discount factor, R t is the immediate reward after taking an action in the current state, max (A′) Q(S t +1, A′) is the maximum Q value in the next state.

8. The path tracking system based on the RL-MPC four-wheel inspection robot according to claim 5, characterized in that: According to the values ​​of MPC prediction time domain Np and control time domain Nc, the objective function in solving the control parameters of the four-wheel inspection robot is: Among them, Q, R is the weight matrix, ρ is the weight coefficient, η is the actual path output in the prediction time domain, η ref is the reference position, and Δu is the control increment in the control time domain.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the path tracking method based on the RL-MPC four-wheel inspection robot as described in any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the path tracking method based on the RL-MPC four-wheel inspection robot are implemented as described in any one of claims 1 to 4.

Images