A tracking control method for mobile robots based on multi-target information fusion
By adopting a reinforcement learning method of multi-target point information fusion in autonomous vehicles, the shortcomings of trajectory tracking control under nonlinear systems are solved, and high-precision and smooth trajectory tracking effect is achieved, which is suitable for autonomous driving of smart cars in complex environments.
Patent Information
- Application Number
- CN202410278022.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-12
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-03-12
AI Technical Summary
Existing autonomous vehicles have problems such as poor response and sensitivity to system parameter changes and external disturbances in trajectory tracking control under nonlinear systems, making it difficult to achieve high-precision path tracking effect.
Using reinforcement learning method based on multi-objective point information fusion, through improved reward function and Q-learning framework, an online reinforcement learning algorithm is developed, and the optimal trajectory tracking control strategy for mobile robots is designed to achieve high-precision and smooth trajectory in complex environments. track.
It improves the accuracy and smoothness of trajectory tracking, reduces the number of iterations and calculation complexity, adapts to complex dynamic environments, and meets the needs of intelligent vehicle autonomous driving.
Smart Images

Figure CN118131628B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of automatic driving of intelligent vehicles, and in particular to a mobile robot tracking control method based on multi-target point information fusion. Background Art
[0002] The emergence of autonomous vehicles provides a solution to problems such as traffic accidents and traffic congestion caused by human errors. With the continuous advancement of autonomous driving technology, the safety issues during vehicle driving have received more and more attention. Therefore, how to accurately perform trajectory tracking control to ensure that autonomous vehicles can drive safely and stably and adapt to various roads and traffic environments has become a research hotspot today.
[0003] Among the current path tracking control methods, PID control, pure tracking control, LQR and MPC methods are widely used, and their main advantages are simple structure and good real-time performance. However, PID control, pure tracking control and LQR control do not respond well to the nonlinear characteristics of the system, and their performance is significantly reduced. At the same time, they are also sensitive to changes in system parameters and external disturbances, and the control strategy needs to be continuously optimized, resulting in high computational costs. In addition, LQR and MPC methods rely on accurate mathematical models. In real life, there is a high degree of uncertainty in the driving environment and vehicle dynamics, and it is difficult to establish an accurate model. Due to the complex and changeable road and traffic conditions and the complex nonlinearity of the vehicle itself, traditional methods are difficult to achieve ideal path tracking effects and can no longer fully meet the requirements of autonomous driving.
[0004] Therefore, in order to improve the path tracking control effect of autonomous driving, an adaptive control method that can handle the complex nonlinearity of the system is urgently needed, so that the system can still achieve high-precision control in a highly dynamic and uncertain environment. In recent years, reinforcement learning (RL) has been widely used. Reinforcement learning is similar to human learning methods, learning how to map from state to behavior to maximize the learning mechanism of the reward. Reinforcement learning presents multiple advantages in comparison with traditional control systems. First, reinforcement learning shows strong adaptability and can adjust strategies in real time to adapt to dynamic environments. Moreover, reinforcement learning has the ability to continuously learn from experience and continuously optimize and improve strategies through feedback (rewards) given by the environment, so there is no need to redesign the control system. Secondly, reinforcement learning is a general method that is applicable to various types of problems, whether it involves discrete decision-making or continuous control. In addition, most importantly, it is model-free and gets rid of the dependence on precise models. For complex environments, nonlinear systems, and situations where precise modeling is not possible, the self-optimization ability of reinforcement learning can still achieve optimal control and is more flexible.
[0005] Taking into account the shortcomings and needs of current autonomous driving technology and control strategies, it is urgent to propose a new tracking control method. Summary of the invention
[0006] The purpose of the present invention is to overcome the shortcomings of the prior art in trajectory tracking control of mobile robots in nonlinear systems. By utilizing an improved reinforcement learning method and improving the reward function, the unknown nonlinear factors of mobile robots in actual complex systems are fully considered. A mobile robot tracking control method based on multi-target point information fusion is proposed, which improves the accuracy of trajectory tracking and provides a theoretical basis and technical guidance for the safety performance of autonomous driving technology.
[0007] The inventive idea of the present invention is: the method provided by the present invention first establishes a kinematic model of the mobile robot and its action space model according to the nonlinear characteristics of the mobile robot and taking into account the complexity, diversity and practicality of the actual working environment of the mobile robot; secondly, based on the kinematic model and space model of the mobile robot, full use is made of real-time tracking trajectory information, and a reward function that integrates multi-target point information is designed; then, using the Q-learning framework, an online reinforcement learning method based on the reward function that integrates multi-target point information is developed, which realizes the optimal solution of the Bellman method and reduces the number of iterations and iteration layers; finally, using the developed online reinforcement learning method, an optimal trajectory tracking control strategy for the mobile robot is designed, which enables the mobile robot to achieve higher precision and smoother trajectory tracking effects in complex trajectory scenes, and achieves the goal of real-time trajectory tracking.
[0008] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is specifically: a mobile robot tracking control method based on multi-target point information fusion, comprising the following steps:
[0009] Step 1: Model the action space of reinforcement learning based on the kinematic model of the mobile robot, and achieve accurate modeling of the steering space of the mobile robot;
[0010] Step 2: Make full use of the predetermined trajectory information and design a reward function that integrates multi-target point information to reduce the number of iterations and iteration layers;
[0011] Step 3, using the online reinforcement learning method under the Q-learning framework to solve the Bellman optimal equation;
[0012] Step 4, the optimal trajectory tracking control strategy of the mobile robot was designed, and algorithm simulation and data comparison were performed in the simulation environment to verify the superiority and feasibility of the proposed algorithm;
[0013] Furthermore, in step 1, based on the bicycle model, the kinematic characteristics of the vehicle are analyzed, and the vehicle motion control is studied, and the kinematic model of the intelligent vehicle is established, and the earth coordinate system is set as the XOY coordinate system, and the coordinate system (x, y) represents the vehicle body coordinate system with the longitudinal direction of the vehicle body as the x-axis and the transverse direction of the vehicle body as the y-axis; define is the current vehicle heading angle, l is the wheelbase of the vehicle, v x and v y are the lateral speed and longitudinal speed of the vehicle respectively, δ is the front wheel turning angle of the vehicle, and the vehicle kinematic equation is as follows:
[0014]
[0015] Select the state of the car as x and the control quantity as u. For any reference point of the reference trajectory, r can be used to represent it. The above formula can be rewritten as the function form of formula (2), where u r =[v r , δ r ] T The function is as follows:
[0016]
[0017] The Taylor series expansion of equation (2) at the reference point r is as follows:
[0018]
[0019] Based on formula (3), and Find the Jacobian matrix, the expression is as follows:
[0020]
[0021] definition is the vehicle state error change, is the change in lateral error, is the change in longitudinal error, is the error change of the heading angle. According to equations (4) and (5), the following expression can be further obtained:
[0022]
[0023] By establishing the concept of error quantity, the state space equation of vehicle kinematics can be derived, which is expressed as follows:
[0024]
[0025] After adding the E matrix, the expression is as follows:
[0026]
[0027] For the above equation, combining equations (4) and (5), the discretized state space equation can be obtained as follows:
[0028]
[0029] Simplify the equation, let The simplified formula is as follows:
[0030]
[0031] The purpose of trajectory tracking control of mobile robots is to obtain the state variable at the next moment through the input quantity u, and to minimize the error between the state and the target position by continuously improving the control quantity u, thereby completing the task of the vehicle following the trajectory point.
[0032] Furthermore, in step 1, the action space of reinforcement learning is modeled based on the kinematic model of the mobile robot, thereby achieving accurate modeling of the turning space of the mobile robot;
[0033] Modeling the reinforcement learning spatial action so that the agent can obtain the maximum cumulative reward in the process of tracking the target trajectory includes the following processes:
[0034] First, the reinforcement learning parameters are initialized. α is the learning rate, which is used to determine the importance of TD (temporal difference) error when updating Q value; γ is the discount factor, which is used to weigh current and future rewards; ∈ is the probability of random operation in the greedy strategy, which is used to control the balance between exploration and utilization in the greedy strategy; num_iterations is the number of iterations, which is used to control the number of iterations of Q value update; Q table is a data table used to record and update the state-action value table.
[0035] Secondly, the state space equation is established. The state space can be described as a four-tuple S is used to represent the value set of environmental observations, which represents the current state in reinforcement learning: x, y are the vehicle position coordinates in the geodetic coordinate system, which are used to record the vehicle position information; is the vehicle heading angle, and the signal comes from the gyroscope; v is the vehicle's longitudinal speed, and the signal comes from the vehicle speed sensor.
[0036] Finally, the action space equation is established. The Ackerman model of the self-driving car can be regarded as a bicycle model, and its action space can be described as A, which is represented as a set of actions, representing the selected actions in reinforcement learning: u is the front wheel angle, and the signal comes from the front wheel angle sensor of the vehicle; due to the limitations of the Ackerman model, its action space constraint is u min ≤u(t)≤umax , because of the requirement of action space modeling in reinforcement learning, A is a finite set of actions, which is composed of several front wheel turning angles, and due to the influence of actual conditions, the front wheel ignores some angles with small changes.
[0037] In order to make full use of the predetermined trajectory information in step 2, a reward function integrating multi-target point information is designed to reduce the number of iterations and iteration layers, including the following processes:
[0038] The reward function formalizes the goal of the reinforcement learning task and provides an important feedback signal for the learning algorithm. The new idea of using multiple sub-goal signals to improve the learning process is designed to design a multi-objective reward function in the following form, which integrates the information of each target point into the information of a field and establishes the following multi-objective reward function as shown in formula (11). The function is as follows:
[0039]
[0040] Three dimensionless scalar coefficients k1, k2 and k3 are introduced into the function, which obey the normalization constraint: k1+k2+k3=1. The purpose of designing this set of adjustment parameters is to enable equation (11) to integrate multiple discrete sub-goals and map them into a unified scalar reward signal;
[0041] Among them, d1, d2 and d3 represent the distance between the current position and the target position and its sub-target, respectively, and Δψ1, Δψ2 and Δψ3 represent the error between the current heading angle and the heading angle of the target and its sub-target. By adding sub-targets, the vehicle can smoothly and stably achieve trajectory tracking control. The multi-objective reward function is shown in formula (11), and the return function expression defined by it is as shown in formula (12):
[0042] r=w d r d +w ψ r ψ (12)
[0043] In formula (12), w d and w ψ Respectively represent the weight coefficients of the position error and heading angle error during vehicle driving, r d and r ψ It represents the reward of the position error and heading angle error of the vehicle during driving, and r represents the sum of the rewards during the vehicle's driving process.
[0044] For step 3, in the Q-learning framework, the Bellman optimal equation is solved using the online reinforcement learning method, including the following process:
[0045] For step 3, under the Q-learning framework, the Bellman optimal equation is solved using the online reinforcement learning method:
[0046] First, the reinforcement learning parameters are initialized, including the learning rate α, discount coefficient γ, ∈ of the ∈-greedy strategy, the maximum number of iterations M, and the maximum number of iteration layers L; and the Q(s, a) values corresponding to all state actions, that is, the Q table, are initialized; the number of iterations m=0, m<M, l<L is set as the initial state.
[0047] Secondly, the action is selected according to the ∈-greedy method, the motion state is updated, and the reward function value is calculated.
[0048] Finally, by analyzing the reward function value, the final reinforcement learning result is obtained.
[0049] For the on-policy version of the Q-learning algorithm in step 3, the temporal difference method is used for learning.
[0050] In the on-policy process of Q learning, an attempt is made to solve the Bellman optimal equation, that is, an attempt is made to directly estimate the optimal value of the action-value function. Specifically, Q learning attempts to estimate the optimal action-value function q(s, a) corresponding to each state s and action a, satisfying the Bellman optimal equation, where q(s, a) represents the final expected reward obtained by executing action a in state s; R t+1 is the immediate reward; γ is the discount factor; Indicates the next state S t+1 The maximum Q value corresponding to the action a that may be taken under the condition is given by formula (13):
[0051]
[0052] The agent first observes the current state s∈S from the environment. Then, based on the ∈-greedy algorithm, it selects the action a∈A with the largest current Q value with probability 1-∈, and randomly selects an action a with probability ε. After executing action a, the environment state is transferred to s t+1 ;
[0053] In order to learn the optimal action value function Q function, a state-action value function q(s, a) is defined to approximate it. Where q(s, a) represents the expected return of performing action a in state s. The Q function follows the Bellman optimal equation for iterative updates, and the expression is as follows:
[0054]
[0055] In formula (14), q t+1 (s t , a t) represents the state s at the current time t, the Q value corresponding to the action a taken, α t (s t , a t ) is the learning rate corresponding to the current state-action, r t+1 represents the immediate reward after executing action a, γ is the discount factor, which is used to measure the importance of future rewards. is the next state s t+1 The maximum Q value corresponding to the action a that may be taken under
[0056] The ∈-greedy algorithm achieves effective balance between exploration and utilization by controlling the degree of random exploration. The algorithm content is as follows:
[0057]
[0058] Furthermore, in step 4, the optimal trajectory tracking control strategy for the mobile robot is designed, and algorithm simulation and data comparison are performed in a simulation environment to verify the superiority and feasibility of the proposed algorithm. The algorithm mainly includes online training, trajectory tracking and algorithm comparison, and the three parts are as follows:
[0059] The online training part is mainly for the on-policy version of Q-learning algorithm. Initialize the Q-table parameters and set the training termination conditions. Interact with the environment, select actions according to the ∈-greedy strategy, and observe the status and reward after executing the actions. Iterate and update the Q-table according to the formula. Repeat this interaction process until the termination condition is met.
[0060] The trajectory tracking part mainly verifies the effect of the learned strategy. The learned Q table is applied to the environment, the greedy action with the largest Q value is selected, and the execution trajectory and cumulative reward are recorded. Compared with the baseline method, the performance of the model in the trajectory tracking task is analyzed, and the effectiveness of the proposed method is evaluated through comparison.
[0061] A comprehensive comparison is made among the model predictive control (MPC), Stanley algorithm, linear quadratic regulator (LQR) algorithm and the proposed improved reinforcement learning algorithm (ORRL), and the trajectory tracking performance is evaluated in a gentle U-shaped curve scenario. By comparing in multiple S-shaped curve scenarios, the trajectory tracking performance of these four algorithms and the improved reinforcement learning algorithm in vehicle turning scenarios is verified.
[0062] Compared with the prior art, the present invention has the following beneficial effects:
[0063] 1. This paper proposes an improved online reinforcement learning algorithm for the trajectory tracking control problem of nonlinear systems. The algorithm is built on the Q-learning framework, can perform iterative optimization based on multiple target point information, and has better environmental adaptability.
[0064] 2. The present invention designs a reward function that integrates multiple sub-goals. Compared with ordinary reinforcement learning methods, this reward mechanism provides richer decision-making information, enhances the foresight of the algorithm, and reduces the number of iterations and computational complexity of the algorithm, thus meeting the needs of real-time control.
[0065] 3. The algorithm proposed in this invention fully combines the ideas of reinforcement learning and multi-objective optimization, and improves the accuracy of trajectory tracking control while ensuring computational efficiency.
[0066] 4. The present invention adopts the method of reinforcement learning, and autonomously learns and gradually optimizes through continuous exploration and trial and error, so that the controlled object can adapt to complex and dynamic environments and make optimal solutions when facing various situations. And by designing a new reward function model, the number of iterations and iteration layers of the Bellman method can be reduced, which can make the reinforcement learning algorithm more efficient, and then more quickly obtain the optimal tracking control method, so as to better meet the needs of the field of autonomous driving of smart cars. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.
[0068] Figure 1 The present invention is a flow chart of the method.
[0069] Figure 2 This is the "bicycle model" of the self-driving car in the present invention.
[0070] Figure 3 This is a simulation scene diagram built on the simulation platform of the present invention.
[0071] Figure 4 This is a diagram of the experimental scene built for the present invention in a real environment.
[0072] Figure 5 This is a simulation result diagram comparing the S path of the present invention with the RL algorithm. DETAILED DESCRIPTION
[0073] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.
[0074] Example 1
[0075] The technical solution provided in this embodiment is a mobile robot tracking control method based on multi-target point information fusion, comprising the following steps:
[0076] Step 1: Model the action space of reinforcement learning based on the kinematic model of the mobile robot, and achieve accurate modeling of the steering space of the mobile robot;
[0077] Additional notes on step 1: Modeling is based on the vehicle kinematics of the bicycle model.
[0078] According to the attached Figure 2 , kinematic modeling of the Ackerman vehicle model.
[0079] The geodetic coordinate system is set to XOY coordinates, and the coordinate system (x, y) represents the body coordinate system with the longitudinal direction of the body as the x-axis and the transverse direction of the body as the y-axis;
[0080] definition is the current vehicle heading angle, l is the wheelbase of the vehicle, v x and v y are the lateral speed and longitudinal speed of the vehicle respectively, δ is the front wheel turning angle of the vehicle, and the vehicle kinematic equation is as follows (1):
[0081]
[0082] Select the state of the car as x and the control quantity as u. For any reference point of the reference trajectory, r can be used to represent it. The above formula can be rewritten as the function form of formula (2), where u r =[v r ,δ r ] T The function is as follows:
[0083]
[0084] For equation (2), Taylor series expansion is adopted at the reference point r, and the expression is as follows:
[0085]
[0086] Based on formula (3), and Find the Jacobian matrix, the expression is as follows:
[0087]
[0088] definition is the vehicle state error change, is the change in lateral error, is the change in longitudinal error, is the error change of the heading angle. According to equations (4) and (5), the following expression can be further obtained:
[0089]
[0090] By establishing the concept of error quantity, the state space equation of vehicle kinematics can be derived, which is expressed as follows:
[0091]
[0092] After adding the E matrix, the expression is as follows:
[0093]
[0094] For the above equation, combining equations (4) and (5), the discretized state space equation can be obtained as follows:
[0095]
[0096] Simplify the equation, let The simplified formula is as follows:
[0097]
[0098] By inputting the control quantity u, the state change of the next step can be obtained By continuously changing the input control quantity u, the state error is reduced, thereby completing the task of the vehicle following the trajectory point.
[0099] Supplementary explanation on step 1: The action space of reinforcement learning is modeled based on the kinematic model of the mobile robot, which realizes the accurate modeling of the turning space of the mobile robot, so that the agent can obtain the maximum cumulative reward in the process of tracking the target trajectory, including the following process:
[0100] First, the reinforcement learning parameters are initialized. α is the learning rate, which is used to determine the importance of TD (temporal difference) error when updating Q value; γ is the discount factor, which is used to weigh current and future rewards; ∈ is the probability of random operation in the greedy strategy, which is used to control the balance between exploration and utilization in the greedy strategy; num_iterations is the number of iterations, which is used to control the number of iterations of Q value update; Q table is a data table used to record and update the state-action value table.
[0101] Secondly, the state space equation is established. The state space can be described as a four-tuple S is used to represent the value set of environmental observations, which represents the current state in reinforcement learning: x, y are the vehicle position coordinates in the geodetic coordinate system, which are used to record the vehicle position information; is the vehicle heading angle, and the signal comes from the gyroscope; v is the vehicle's longitudinal speed, and the signal comes from the vehicle speed sensor.
[0102] Finally, the action space equation is established. The Ackerman model of the self-driving car can be regarded as a bicycle model, and its action space can be described as A, which is represented as a set of actions, representing the selected actions in reinforcement learning: u is the front wheel angle, and the signal comes from the front wheel angle sensor of the vehicle; due to the limitations of the Ackerman model, its action space constraint is u min ≤u(t)≤u max , because of the requirement of action space modeling in reinforcement learning, A is a finite set of actions, which is composed of several front wheel turning angles, and due to the influence of actual conditions, the front wheel ignores some angles with small changes.
[0103] Step 2: Make full use of the predetermined trajectory information and design a reward function that integrates multi-target point information to reduce the number of iterations and iteration layers;
[0104] Supplementary explanation on step 2: A reward function that integrates information of multiple target points is designed;
[0105] Design a reward function based on multi-target point information fusion. The reward function formalizes the goal of the reinforcement learning task and provides important feedback signals for the learning algorithm.
[0106] Using multiple sub-goal signals to improve the new idea of the learning process, a multi-goal reward function of the following form is designed, which integrates the information of each target point into the information of a field and establishes the following multi-goal reward function as shown in formula (11). The function is as follows:
[0107]
[0108] Three dimensionless scalar coefficients k1, k2 and k3 are introduced into the function, which obey the normalization constraint: k1+k2+k3=1. The purpose of designing this set of adjustment parameters is to enable equation (11) to integrate multiple discrete sub-goals and map them into a unified scalar reward signal;
[0109] Among them, d1, d2 and d3 represent the distance between the current position and the target position and its sub-target, respectively, and Δψ1, Δψ2 and Δψ3 represent the error between the current heading angle and the heading angle of the target and its sub-target. By adding sub-targets, the vehicle can smoothly and stably achieve trajectory tracking control. The multi-objective reward function is shown in formula (11), and the return function expression defined by it is as shown in formula (12):
[0110] r=w d r d +w ψ r ψ (12)
[0111] In formula (12), w d and w ψRespectively represent the weight coefficients of the position error and the heading angle error during vehicle driving, r d and r ψ represent the rewards for the position error and the heading angle error during vehicle driving, and r represents the sum of the rewards during vehicle driving.
[0112] In this step 5, through multi-objective information fusion, a reward function is designed, considering the position and heading angle errors, and using coefficients k1, k2, and k3 to integrate multiple sub-goals to establish a unified scalar reward signal. This optimizes the learning process and achieves smooth trajectory tracking control.
[0113] Step 3, under the Q-learning framework, use the online reinforcement learning method to solve the Bellman optimal equation;
[0114] In the said step 3, under the Q-learning framework, use the online reinforcement learning method to solve the Bellman optimal equation. The said step 3 includes the following process:
[0115] First, for the parameter initialization of reinforcement learning, including the learning rate α, the discount factor γ, the parameter ∈ of the ∈-greedy policy, the maximum number of iterations M, and the maximum number of iteration layers L. In the initialization stage, it is also necessary to initialize the Q(s, a) values corresponding to all state actions, that is, the Q-table, so as to be updated during the learning process. Set the iteration number m = 0, m < M, and the moment l < L as the initial state, which is the starting point of the algorithm.
[0116] Secondly, select actions according to the ∈-greedy method, update the environmental state according to the current state and the selected actions, and calculate the reward function value. This step is the key to reinforcement learning. By continuously interacting with the environment and obtaining reward feedback, the algorithm can gradually adjust its behavior to achieve a better policy.
[0117] Finally, through the analysis of the reward function value, use the on-policy version of the Q-learning algorithm combined with the temporal difference method for learning. By solving the Bellman optimal equation, find the optimal policy, so that the designed algorithm has stronger adaptability and good generalization ability to complex environments. Such a design enables the optimal policy to run stably in an uncertain environment and improves the robustness of the system. In the on-policy process of Q-learning, try to solve the Bellman optimal equation, that is, try to directly estimate the optimal value of the action value function. Specifically, Q-learning tries to estimate the optimal action value function q(s, a) corresponding to each state s and action a, which satisfies the Bellman optimal equation, where q(s, a) represents the final expected return obtained by executing action a in state s; R t+1 is the immediate reward; γ is the discount factor; represents the next state St+1 The maximum Q value corresponding to the action a that may be taken under the condition is given by formula (13):
[0118]
[0119] The agent first observes the current state s∈S from the environment. Then, based on the ∈-greedy algorithm, it selects the action a∈A with the largest current Q value with probability 1-∈, and randomly selects an action a with probability ε. After executing action a, the environment state transfers to s t+1 ;
[0120] In order to learn the optimal action value function Q function, a state-action value function q(s,a) is defined to approximate it. Where q(s,a) represents the expected return of performing action a in state s. The Q function follows the Bellman optimal equation for iterative updates, and the expression is as follows:
[0121]
[0122] In formula (14), q t+1 (s t , a t ) represents the state s at the current time t, the Q value corresponding to the action a taken, α t (s t , a t ) is the learning rate corresponding to the current state-action, r t+1 represents the immediate reward after executing action a, γ is the discount factor, which is used to measure the importance of future rewards. is the next state s t+1 The maximum Q value corresponding to the action a that may be taken under
[0123] The ∈-greedy algorithm achieves effective balance between exploration and utilization by controlling the degree of random exploration. The algorithm content is as follows:
[0124]
[0125] Step 4, the optimal trajectory tracking control strategy of the mobile robot was designed, and algorithm simulation and data comparison were performed in the simulation environment to verify the superiority and feasibility of the proposed algorithm;
[0126] Supplementary explanation on step 4: algorithm simulation and data comparison are carried out in the simulation environment to verify the superiority of the proposed algorithm and the operation effect and practicability of the algorithm, which mainly includes three parts: online training, trajectory tracking and algorithm comparison:
[0127] The online training part is mainly for the on-policy version of Q-learning algorithm. Initialize the Q-table parameters and set the training termination conditions. Interact with the environment, select actions according to the ∈-greedy strategy, and observe the status and reward after executing the actions. Iterate and update the Q-table according to the formula. Repeat this interaction process until the termination condition is met.
[0128] The trajectory tracking part mainly verifies the effect of the learned strategy. The learned Q table is applied to the environment and the greedy action with the largest Q value is selected. The execution trajectory and cumulative rewards are recorded. Compared with the baseline method, the performance of the model in the trajectory tracking task is analyzed. The effectiveness of the proposed method is evaluated by comparison. The simulation experiment is shown in the attached figure. Figure 4 It is built in gazebo in , which can provide feedback on the vehicle's following effect.
[0129] A comprehensive comparison is made among the model predictive control (MPC), Stanley algorithm, linear quadratic regulator (LQR) algorithm and the proposed improved reinforcement learning algorithm (ORRL).
[0130] Set the initial conditions, the vehicle speed v is 10m / s, the initial position (x, y) is (-0.5m, 0.5m), the initial heading angle Both are 0.2rad, and the wheelbase l of the vehicle body is 2m.
[0131] In the gentle U-shaped curve scenario, the trajectory tracking performance of the five algorithms is evaluated, as shown in Table 1 below.
[0132] Table 1 Lateral errors of five algorithms under the U-shaped curve
[0133]
[0134] As shown in Table 1, the lateral average error reflects the overall trajectory tracking accuracy. ORRL improves by 76.15%, 452.92%, 646.15% and 481.54% compared with RL, MPC, Stanley and LQR, respectively, indicating that ORRL can significantly improve the overall accuracy.
[0135] As shown in Table 1, the lateral root mean square error focuses on trajectory tracking smoothness and stability. ORRL improves by 123.55%, 480.90%, 637.15% and 507.62% respectively compared with other algorithms. Compared with the Stanley method, it is improved by 69.75%, indicating that the ORRL trajectory is smoother and the dynamic response is more controllable.
[0136] In the gentle S-curve scenario, the trajectory tracking performance of the five algorithms is evaluated and shown in Table 2 below.
[0137] Table 2 Lateral errors of five algorithms under multiple S-curves
[0138]
[0139] As shown in Table 2, the lateral average error reflects the overall trajectory tracking accuracy. ORRL improves by 278.03%, 485.64%, 563.37% and 484.82% compared with RL, MPC, Stanley and LQR, respectively, indicating that ORRL can significantly improve the overall accuracy.
[0140] As can be seen from Table 2, the lateral root mean square error focuses on trajectory tracking smoothness and stability. ORRL has improved by 248.34%, 530.47%, 558.36% and 459.19% respectively compared with other algorithms. Compared with the Stanley method, it has improved by 69.75%, indicating that the ORRL trajectory is smoother and the dynamic response is more controllable. Therefore, in complex scenarios, ORRL has better trajectory tracking accuracy, smoothness and stability than traditional methods, and is suitable for difficult path tracking tasks. By comparing in multiple S-curve scenarios, the trajectory tracking performance of these four algorithms and the improved reinforcement learning algorithm in vehicle turning scenarios is verified.
[0141] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A mobile robot tracking control method based on multi-target point information fusion, characterized in that: The following steps are involved: S1. Model the action space of reinforcement learning based on the kinematic model of the mobile robot and accurately model the turning space of the mobile robot; S2. Make full use of the predetermined trajectory information and design a reward function that integrates multi-target point information; In step S2, a reward function integrating multiple target point information is designed by using the predetermined trajectory information, and a new target information function (1) is established to guide the learning process. The function (1) is shown in the following formula: Three dimensionless scalar coefficients k1, k2 and k3 are introduced, subject to the normalization constraint: k1+k2+k3=1. These coefficients are used to adjust the impact of each sub-benefit. d1, d2 and d3 represent the distance between the current position and the target position and its sub-target, respectively. Δψ1, Δψ2 and Δψ3 represent the error between the current heading angle and the heading angle of the target and its sub-target. r d and r ψ Rewards representing the position error and heading angle error of the vehicle during driving; The improved target information is integrated as a unified scalar reward signal, and the external environment information and trajectory information are used to design a new multi-objective reward function (2) as shown below: r=w d r d +w ψ r ψ (2) In the above formula, w d and w ψ represents the weight coefficient of the position error and heading angle error during the vehicle's driving process, and r represents the sum of the rewards during the vehicle's driving process; S3. Under the Q-learning framework, the Bellman optimal equation is solved using online reinforcement learning methods. S4. The optimal trajectory tracking control strategy for the mobile robot was designed, and algorithm simulation and data comparison were performed in a simulation environment to verify the superiority and feasibility of the proposed algorithm.
2. The mobile robot tracking control method based on multi-target point information fusion according to claim 1 is characterized in that: In step S1, the spatial action of reinforcement learning is modeled so that the agent obtains the maximum cumulative reward in the process of tracking the target trajectory. The process includes: S11. Initialize the reinforcement learning parameters. α is the learning rate, which is used to determine the importance of the TD time difference error when updating the Q value. γ is the discount factor, which is used to weigh the current and future rewards. ∈ is the probability of random operation in the greedy strategy, which is used to control the balance between exploration and exploitation in the greedy strategy; num_iterations is the number of iterations, which is used to control the number of iterations of Q value update; Q table is a data recording table, which is used to record and update the state-action value state-action value table; S12. Establish the state space equation and describe its state space as a four-tuple S represents the value set of environmental observations, which represents the current state in reinforcement learning: T, y is the vehicle position coordinate in the geodetic coordinate system, which is used to record the vehicle position information; is the vehicle heading angle, the signal comes from the gyroscope; v is the vehicle longitudinal speed, the signal comes from the vehicle speed sensor; S13. Establish the action space equation. The Ackerman model of the self-driving car is regarded as a bicycle model. Its action space is described as A, which is expressed as a set of actions, representing the selected actions in reinforcement learning: u is the front wheel angle, and the signal comes from the front wheel angle sensor of the vehicle; due to the limitations of the Ackerman model, its action space constraint is u min ≤u(t)≤u max ,The requirements for action space modeling in reinforcement learning,A is a finite set of actions, which is composed of several front wheel angles, and due to the influence of actual conditions, the front wheel ignores some angles with small changes.
3. The mobile robot tracking control method based on multi-target point information fusion according to claim 1 is characterized in that: The step S3 includes the following steps: S31, parameter initialization for reinforcement learning, including learning rate α, discount factor γ, ∈ - The parameters of the greedy strategy ∈, the maximum number of iterations M and the maximum number of iteration layers L. In the initialization phase, the Q(s, a) values corresponding to all state actions are initialized, that is, the Q table, and the time when the number of iterations m = 0, m < M, l < L is set as the initial state, which is the starting point of the algorithm; S32, select an action according to the ∈-greedy method, update the environment state according to the current state and the selected action, and calculate the reward function value; S33. By analyzing the value of the reward function, the on-policy version of the Q learning algorithm is used in combination with the time difference method to learn, and the optimal strategy is found by solving the Bellman optimal equation.
4. The mobile robot tracking control method based on multi-target point information fusion according to claim 1 is characterized in that: In step S4, the optimal trajectory tracking control strategy of the mobile robot is designed, and algorithm simulation and data comparison are performed in a simulation environment to verify the superiority and feasibility of the proposed algorithm. The algorithm mainly includes online training, trajectory tracking and algorithm comparison parts, and the contents of the three parts are as follows: (1) Online training The online training part is mainly for the on-policy version of Q learning algorithm; initialize Q table parameters, set training termination conditions, interact with the environment, select actions according to the ∈-greedy strategy, observe the state and reward after executing the action, iteratively update the Q table according to the formula, and repeat the interaction process until the termination condition is met; (2) Trajectory tracking Apply the learned Q-table to the environment, select the greedy action with the largest Q value, and record the execution trajectory and cumulative rewards; (3) Algorithm comparison The Stanley algorithm based on model predictive control (MPC), the linear quadratic regulator LQR algorithm and the improved reinforcement learning algorithm are used for comparison, and the trajectory tracking performance is evaluated in a gentle U-shaped curve scenario. By comparing in multiple S-shaped curve scenarios, the trajectory tracking performance of these four algorithms and the improved reinforcement learning algorithm in vehicle turning scenarios is verified.
