A trajectory tracking control method for mobile robots based on fuzzy reasoning and reinforcement learning

Abstract

The present invention discloses a mobile robot trajectory tracking control method based on fuzzy reasoning and reinforcement learning, comprising: establishing a mobile robot kinematic model, obtaining the actual trajectory of the mobile robot based on the mobile robot kinematic model; obtaining the trajectory error and error change rate based on the expected trajectory of the mobile robot and the actual trajectory of the robot; inputting the trajectory error and error change rate into a fuzzy PID controller and a reinforcement learning controller, outputting the front wheel steering angle; and controlling the mobile robot based on the front wheel steering angle. The present invention can cope with various trajectories without changing parameters.

Description

Technical Field

[0001] The present invention belongs to the field of mobile robot control, and in particular relates to a mobile robot trajectory tracking control method based on fuzzy reasoning and reinforcement learning. Background Art

[0002] Mobile robot control is a comprehensive technology involving robotics, control theory, sensor technology, computer science, artificial intelligence, etc. The core task of mobile robot control is to make the mobile robot move along the trajectory generated by the predetermined task in a complex environment, ensure that the mobile robot travels on the predetermined trajectory with minimum error, and complete the task.

[0003] The traditional solution is to design a PID controller or an anti-disturbance control controller based on the kinematic model of the mobile robot to complete the trajectory tracking task. The PID controller has a simple design, a simple structure, and fewer parameters, while the anti-disturbance control controller has strong robustness to interference. However, after the parameters of these controllers are set, they cannot achieve good tracking effects when facing trajectories in different situations, because the same parameters cannot match different trajectory situations. Similarly, when the mobile robot changes due to wear, damage, or load changes, the traditional controller may not be able to automatically adapt to these changes. In addition, in the actual application of mobile robots, the system is often affected by various external interferences and uncertain factors, such as wind interference, terrain changes, etc. It is difficult for traditional controllers to describe all external interferences, which leads to increased trajectory tracking errors.

[0004] In summary, the existing mobile robot control system lacks a motion control method with a simple structure, simple parameter adjustment method, and wide applicability. Therefore, a mobile robot trajectory tracking control method based on fuzzy reasoning and reinforcement learning is urgently needed. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a mobile robot trajectory tracking control method based on fuzzy reasoning and reinforcement learning. The present invention combines the fuzzy PID controller with reinforcement learning as the master controller. Through the powerful autonomous learning ability of reinforcement learning, the output of the fuzzy PID is compensated to enhance the ability of the fuzzy PID to sample different parameters in different situations and enhance the tracking ability under various trajectories. At the same time, the input state of reinforcement learning is fuzzy preprocessed to improve the correlation between reinforcement learning and the fuzzy PID controller. On the basis of using fuzzy rules, the output of the fuzzy PID controller is compensated more flexibly and accurately to achieve a smaller tracking error.

[0006] To achieve the above object, the present invention provides a mobile robot trajectory tracking control method based on fuzzy reasoning and reinforcement learning, comprising:

[0007] Establishing a kinematic model of the mobile robot, and obtaining an actual trajectory of the mobile robot based on the kinematic model of the mobile robot;

[0008] Based on the expected trajectory of the mobile robot and the actual trajectory of the robot, obtaining a trajectory error and an error change rate;

[0009] The trajectory error and the error change rate are input into the fuzzy PID controller and the reinforcement learning controller, and the front wheel steering angle is output;

[0010] The mobile robot is controlled based on the front wheel steering angle.

[0011] Optionally, the kinematic model of the mobile robot is:

[0012]

[0013] in, is the velocity in the x-axis direction, is the velocity in the y-axis direction, is the angular velocity, V is the velocity at the center of mass of the mobile robot, It is expressed as the angle between the mobile robot and the X-axis of the reference coordinate system, l is the distance between the front and rear wheels, and δ f is the front wheel turning angle.

[0014] Optionally, the trajectory error and the error change rate are input into a fuzzy PID controller and a reinforcement learning controller, and the output front wheel steering angle includes:

[0015] Inputting the trajectory error and the error change rate into the fuzzy PID controller to obtain a first front wheel steering angle;

[0016] Inputting the trajectory error and the error change rate into the reinforcement learning controller to obtain a second front wheel steering angle;

[0017] The front wheel steering angle is acquired based on the first front wheel steering angle and the second front wheel steering angle.

[0018] Optionally, inputting the trajectory error and the error change rate into the fuzzy PID controller to obtain the first front wheel steering angle includes:

[0019] Performing differentiation processing on the trajectory error to obtain the error change rate;

[0020] Fuzzifying the trajectory error and the error change rate to obtain a fuzzy input;

[0021] In combination with a fuzzy rule base, the fuzzified input is solved to obtain an anti-fuzzy output;

[0022] Based on the anti-fuzzy output and combined with the controller parameter base value, the first front wheel steering angle is output.

[0023] Optionally, the first front wheel steering angle is:

[0024]

[0025] Among them, u(t) is the overall output of the fuzzy PID controller, K p , K i , K d Represent the basic values ​​of the proportional coefficient, integral coefficient, and differential coefficient respectively, e(t) represents the error at time t, Δk p Represents the change in proportionality coefficient, Δk i is the change in the integral coefficient, Δk d The change value of the differential coefficient.

[0026] Optionally, the trajectory error and the error change rate are input into the reinforcement learning controller, and before obtaining the second front wheel steering angle, it also includes: based on the kinematic model of the mobile robot, combined with the reward function, obtaining the reward value at the previous moment.

[0027] Optionally, inputting the trajectory error and the error change rate into the reinforcement learning controller to obtain the second front wheel steering angle includes:

[0028] The trajectory error and the error change rate are fuzzy processed using a membership function to obtain the current state quantity;

[0029] Inputting the current state quantity and the previous reward value into a DDPG model to obtain the second front wheel steering angle, wherein the DDPG model is trained in real time;

[0030] The DDPG model is trained in real time by extracting states at different times, output actions, obtained feedback reward values, and new states from the experience pool in the DDPG model for training using an experience replacement mechanism.

[0031] Optionally, the DDPG model includes: an actor network and a critic network;

[0032] The actor network is used to estimate the state and action at the next moment;

[0033] The critic network is used to estimate the state quantity at the next moment.

[0034] Compared with the prior art, the present invention has the following advantages and technical effects:

[0035] 1. The present invention provides a mobile robot trajectory tracking control method based on fuzzy reasoning and reinforcement learning, and constructs a solution that can reduce the trajectory tracking error of the mobile robot through fuzzy reasoning and reinforcement learning knowledge.

[0036] 2. The present invention designs a mobile robot trajectory tracking control method based on fuzzy reasoning and reinforcement learning, which can cope with various trajectories without changing parameters.

[0037] 3. The present invention pre-processes the input state quantity of reinforcement learning in the design, and can effectively deal with external interference through reinforcement learning. By dividing the state into fuzzy sets, the reinforcement learning agent can share experience between different states and improve generalization ability. Even in an unseen state, the agent can infer the appropriate action based on the membership degree. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0039] Figure 1 It is a flow chart of a mobile robot trajectory tracking control method based on fuzzy reasoning and reinforcement learning according to an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of the structure of a four-wheel mobile robot according to an embodiment of the present invention;

[0041] Figure 3 A fuzzy PID controller flow chart of an embodiment of the present invention;

[0042] Figure 4 This is a flow chart of a reinforcement learning controller combining fuzzy concepts according to an embodiment of the present invention;

[0043] Figure 5 Schematic diagram of membership function according to an embodiment of the present invention;

[0044] Figure 6 is a longitudinal error graph of the mobile robot obtained after training in an embodiment of the present invention;

[0045] Figure 7 This is a comparison diagram between the actual trajectory and the ideal trajectory of the mobile robot after training according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0047] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0048] The present invention provides a mobile robot trajectory tracking control method based on fuzzy reasoning and reinforcement learning, such as Figure 1 As shown, specifically including:

[0049] Establish a kinematic model of the mobile robot, and obtain the ideal trajectory of the mobile robot based on the kinematic model of the mobile robot;

[0050] Specifically, in order to make the model simple and easy to use, the kinematic modeling of the four-wheeled mobile robot is carried out based on the bicycle model, and the following assumptions are made: only planar motion is considered; the turning angles of the left and right wheels of the mobile robot are consistent; and the body is rigid. Figure 2 This is a schematic diagram of the structure of a four-wheeled mobile robot. In the figure, point O is the origin of the coordinate system, A is the center point of the front wheel, B is the center point of the rear wheel, C is the center of mass of the mobile robot, D is the instantaneous rotation center of the mobile robot, R is the rotation radius of the center of mass of the mobile robot, and l is the distance between the front and rear wheels. Assuming that the mobile robot moves on a plane, there is The mobile robot is represented by x, y, respectively, representing the coordinates of the mobile robot in the reference coordinate system, and the heading angle φ represents the angle between the mobile robot and the X-axis of the reference coordinate system. V is the velocity at the center of mass of the mobile robot, ω represents the angular velocity of the center of mass of the mobile robot vehicle, and the angle between the velocity direction and the direction of the mobile robot body is β. f ,v r Represent the front wheel speed and rear wheel speed respectively, δ f ,δ r They represent the turning angles of the front and rear wheels respectively, which are equivalent to the direction of the mobile robot body.

[0051] The front and rear wheel turning angles are discussed, and the kinematic model is obtained as follows:

[0052]

[0053] in, is the velocity in the x-axis direction, is the velocity in the y-axis direction, is the angular velocity, V is the velocity at the center of mass of the mobile robot, It is expressed as the angle between the mobile robot and the X-axis of the reference coordinate system, β is the angle between the velocity direction and the body direction of the mobile robot, and δ f ,δ r Respectively represent the front and rear wheels equivalent to the turning angle of the mobile robot body, l fis the distance between the center of the front wheel and the center of mass of the vehicle, l r , is the distance between the center of the front wheel and the center of mass of the vehicle, and l is the distance between the front and rear wheels.

[0054] In the actual operation of the mobile robot, it is usually assumed that the rear wheel and the body are in a straight line, so δ r Constant and 0, and ignoring the slip in the vertical direction of the mobile robot body, so β ​​is 0, the simplified kinematic model is:

[0055]

[0056] in, is the velocity in the x-axis direction, is the velocity in the y-axis direction, is the angular velocity, V is the velocity at the center of mass of the mobile robot, It is expressed as the angle between the mobile robot and the X-axis of the reference coordinate system, l is the distance between the front and rear wheels, and δ f is the front wheel turning angle.

[0057] In the present invention, the mobile robot is set to run at a known speed, and the front wheel turning angle δ f as control input.

[0058] Based on the expected trajectory of the mobile person and the ideal trajectory of the robot, the trajectory error and error change rate are obtained;

[0059] The trajectory error and error change rate are input into the fuzzy PID controller and the reinforcement learning controller, and the front wheel steering angle is output;

[0060] The mobile robot is controlled based on the front wheel steering angle.

[0061] Furthermore, the trajectory error and the error change rate are input into the fuzzy PID controller and the reinforcement learning controller, and the output front wheel steering angle includes:

[0062] The trajectory error and the error change rate are input into the fuzzy PID controller to obtain the first front wheel steering angle;

[0063] The trajectory error and the error change rate are input into the reinforcement learning controller to obtain the second front wheel steering angle;

[0064] A front wheel steering angle is acquired based on the first front wheel steering angle and the second front wheel steering angle.

[0065] Furthermore, the trajectory error and the error change rate are input into the fuzzy PID controller, and obtaining the first front wheel steering angle includes:

[0066] Differentiate the trajectory error to obtain the error change rate;

[0067] Fuzzify the trajectory error and the error change rate to obtain the fuzzy input;

[0068] Combined with the fuzzy rule base, the fuzzified input is solved to obtain the anti-fuzzy output;

[0069] Based on the anti-fuzzy output and combined with the basic value of the controller parameters, the first front wheel steering angle is output.

[0070] Specifically, Figure 3 This is a schematic diagram of a fuzzy PID controller. After the kinematic model of the mobile robot is established, a fuzzy PID controller is established based on the model. The fuzzy PID controller uses fuzzy logic and certain fuzzy rules to optimize the parameters of the PID in real time to overcome the disadvantage that the traditional PID cannot adjust the parameters in real time. It generally includes three steps: input fuzzification, fuzzy reasoning based on the rule base, and defuzzification.

[0071] The error and error change rate of trajectory tracking are used as the input of the controller. The domain of the controller input is limited based on experience and scaled by the proportional coefficient. The changes in the proportional, integral, and differential coefficients of the PID controller are used as the controller output. The membership function defines the membership degree of each point in the input space mapped to a value between 0 and 1. Generally, there are triangle membership functions, trapezoid membership functions, Gaussian membership functions, etc. Taking the error e as the input, after fuzzification processing by the membership function, the language variable set T(E) is obtained as shown below, where E represents the fuzzy language variable:

[0072] T(E)={NB,NM,NS,ZO,PS,PM,PB} (3)

[0073] Among them, NB, NM, NS, ZO, PS, PM, and PB represent negative big, negative medium, negative small, zero, positive small, positive medium, and positive big respectively.

[0074] After the input is fuzzified by the membership function, it is processed by fuzzy reasoning of the fuzzy rule base. When setting the fuzzy rule base, the relationship between the PID parameters and the error and its change rate in the previous controller should be referred to. For example: 1. When the error is large, the proportional coefficient should be large to maintain a faster response speed, while the integral coefficient and the differential coefficient should take smaller values ​​to avoid excessive error change rate and overshoot. 2. When the error size is medium, in order to ensure the response speed and control the overshoot, the proportional coefficient should be reduced, the integral coefficient should be increased, and the differential coefficient should be moderate. 3. When the error is small, in order to ensure that the system has good steady-state characteristics, the proportional and integral coefficients should be increased, and the differential coefficient should be taken according to the error change rate.

[0075] The fuzzy rule base is designed based on experience. The input linguistic variables are error e and error change rate Δe, and the output linguistic variable is Δk p ,Δk i ,Δk d , the language variables are selected as NB, NM, NS, ZO, PS, PM, PB, and the rule table is as follows:

[0076] Table 1Δk p Fuzzy rule table

[0077]

[0078] Table 2Δk i Fuzzy rule table

[0079]

[0080] Table 3Δk d Fuzzy rule table

[0081]

[0082] After the input is fuzzy inferenced by the fuzzy rule base, it is defuzzified to obtain the final accurate output. The centroid method is used for defuzzification. The basic idea of ​​the centroid method is to interpolate the membership function of the fuzzy set to obtain a continuous function. By integrating the continuous function, the centroid position is obtained, and then the accurate value is output according to the centroid position to obtain the change of the three parameters. At this moment, the overall output u(t) of the fuzzy PID controller, that is, the first steering angle, is:

[0083]

[0084] Among them, K p ,K i ,K d They represent the basic values ​​of the proportional coefficient, integral coefficient, and differential coefficient respectively, and e(t) represents the error at time t.

[0085] Furthermore, the trajectory error and the error change rate are input into the reinforcement learning controller, and before obtaining the second front wheel steering angle, the method also includes: obtaining the reward value at the previous moment based on the kinematic model of the mobile robot and in combination with the reward function.

[0086] Furthermore, the trajectory error and the error change rate are input into the reinforcement learning controller, and obtaining the second front wheel steering angle includes:

[0087] The current state and the previous reward value are input into the DDPG model to obtain the second front wheel steering angle, wherein the DDPG model is trained in real time;

[0088] The DDPG model is trained in real time by: using the experience replay mechanism to extract the states, output actions, feedback reward values, and new states at different times from the experience pool in the DDPG model for training;

[0089] Specifically, during the training process, the mobile robot performs a certain action in position state A, which causes the mobile robot to move to a new position state B, and at the same time, a feedback reward value is obtained according to the change in position state. The above information is stored in the experience pool as a set of training samples. Multiple sets of training samples are extracted each time, and the ideal state value function of each position state is calculated through the sample information to facilitate model optimization.

[0090] Furthermore, the DDPG model includes: an actor network and a critic network;

[0091] The actor network is used to estimate the state and action at the next moment;

[0092] The critic network is used to estimate the state quantity at the next moment.

[0093] Specifically, the reinforcement learning controller here needs to output continuous values, so the DDPG algorithm is used. The DDPG algorithm uses the actor-critic algorithm as its basic framework, fits the policy network and action value function through a neural network, and uses the stochastic gradient method to train the parameters in the policy network and value network model. The DDPG algorithm contains four neural networks, which are used to approximate the value function and strategy. Among them, the Critic target network is used to approximate the state-action value function at the next moment, while the Actor target network approximates the state value function at the next moment. The Critic training network outputs the state-action value function at the current moment, and the Actor training network outputs the state value function at the current moment.

[0094] Figure 4 This is a flowchart of the reinforcement learning controller combined with fuzzy concepts. The DDPG algorithm framework is built according to the mobile kinematics model. The error and its error change rate are fuzzified. Here, the appropriate membership function is selected according to the range of change of the error and its change rate. The error is fuzzified using the triangular membership function and the Gaussian membership function. The Gaussian membership function is used at the boundary of the membership function. Because the Gaussian function is continuous and smooth, its membership is stable at the boundary without jump points or discontinuities, which helps to reduce the instability of the system. The triangular membership function is used in the middle area because the triangular membership function reaches the maximum membership at the vertex and can quickly drop to 0 on both sides. This clear change can clearly define the membership of the fuzzy set, making the interpretation of the fuzzy rules more intuitive.

[0095] The triangle membership function triangle is described by three parameters {a, b, c}, and a <b<c:

[0096]

[0097] The Gaussian membership function Gauss is described by two parameters {a, b}, a determines the center of the function, and b determines the width of the function:

[0098]

[0099] After the design is completed, the membership function is as follows Figure 5 As shown in the figure, it is divided into seven language variables: NB, NM, NS, ZO, PS, PM, and PB. The error and change rate are fuzzy processed to obtain the state quantity.

[0100] The reward value is returned by the reward function, which consists of r1 and r2, including minimizing the y-axis error at the current x position and guiding to the end of the trajectory. Its reward function r is as follows:

[0101] r=α×r1+β×r2 (7)

[0102] Among them, α and β are proportional coefficients. The formulas for r1 and r2 are as follows:

[0103] r1=-γ1×|y desired -y|

[0104] r2=γ2×(U att (t-1)-U att (t-1))

[0105] U att =d 2 (g goal -g) (8)

[0106] Among them, γ1 and γ2 are positive proportional coefficients. desired represents the ideal y coordinate under the current x coordinate, and y represents the current actual y coordinate. The larger the absolute value of the longitudinal distance error, the larger the feedback negative reward value, and the smaller the error, the smaller the feedback negative reward value. d(g goal-g ) represents the Euclidean distance between the target and the current coordinates. Calculate the Euclidean distance between the agent and the end point, and generate a gravitational field U with the target point as the center att , distance d(g goal-g ) is reduced, the potential energy of the agent is reduced. Feedback positive reward, distance d(g goal-g ) becomes larger, and negative rewards are fed back.

[0107] The state quantity, action value, and reward value are all stored in the experience replay pool. When the number of replay pool storage is greater than the minimum value, a certain amount of samples are extracted for training, and the training parameters will update the actor network and the critic network. The updated network will use the new strategy to determine the current output action value, thereby controlling the mobile robot.

[0108] Figure 1 The block diagram of the overall system after all designs are completed reflects the signal flow of the entire system. The designed fuzzy PID controller is connected in parallel with the reinforcement learning controller. When the trajectory error between the ideal trajectory and the expected trajectory is calculated, it is transferred in two directions. It is sent to the membership function for fuzzification and the fuzzy PID controller respectively. The state variables after fuzzification will be input into the reinforcement learning controller to obtain the reinforcement learning output. The overall controller uses the fuzzy controller as the main controller, and the reinforcement learning controller assists it. The sum of the outputs of the two controllers is limited and used as the actual output of the controller, that is, the front wheel steering angle, to control the mobile robot. After the parallel connection is completed, the overall system is trained. After a period of training, a better control effect than the traditional controller can be obtained.

[0109] like Figure 6 The longitudinal error of the mobile robot after training is shown. Figure 7 The figure is a comparison diagram between the actual trajectory and the ideal trajectory of the mobile robot after training. The above pictures fully prove the feasibility of the present invention.

[0110] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A mobile robot trajectory tracking control method based on fuzzy reasoning and reinforcement learning, characterized in that: include: Establishing a kinematic model of the mobile robot, and obtaining an actual trajectory of the mobile robot based on the kinematic model of the mobile robot; Based on the expected trajectory of the mobile robot and the actual trajectory of the robot, obtaining a trajectory error and an error change rate; The trajectory error and the error change rate are input into the fuzzy PID controller and the reinforcement learning controller, and the front wheel steering angle is output; The trajectory error and error change rate are input into the fuzzy PID controller and the reinforcement learning controller, and the output front wheel steering angle includes: The trajectory error and the error change rate are input into the fuzzy PID controller, and a defuzzified output is obtained using a gravity center method, that is, a first front wheel steering angle is obtained; Inputting the trajectory error and the error change rate into the reinforcement learning controller to obtain a second front wheel steering angle; Inputting the trajectory error and the error change rate into the fuzzy PID controller, and obtaining the first front wheel steering angle comprises: Performing differentiation processing on the trajectory error to obtain the error change rate; Fuzzifying the trajectory error and the error change rate to obtain a fuzzy input; In combination with a fuzzy rule base, the fuzzified input is solved to obtain an anti-fuzzy output; Based on the anti-fuzzy output and in combination with a controller parameter base value, output the first front wheel steering angle; Inputting the trajectory error and the error change rate into the reinforcement learning controller, and obtaining the second front wheel steering angle also includes: obtaining the reward value at the previous moment based on the kinematic model of the mobile robot and in combination with the reward function; Inputting the trajectory error and the error change rate into the reinforcement learning controller, and obtaining the second front wheel steering angle comprises: The trajectory error and the error change rate are fuzzy processed using a membership function to obtain the current state quantity; Inputting the current state quantity and the previous reward value into a DDPG model to obtain the second front wheel steering angle, wherein the DDPG model is trained in real time; The DDPG model is trained in real time by: using an experience replacement mechanism to extract states at different times, output actions, obtained feedback reward values, and new states from the experience pool of the DDPG model for training; Based on the first front wheel steering angle and the second front wheel steering angle, acquiring the front wheel steering angle; The mobile robot is controlled based on the front wheel steering angle.

2. According to claim 1, a mobile robot trajectory tracking control method based on fuzzy reasoning and reinforcement learning is characterized in that: The kinematic model of the mobile robot is: in, is the velocity in the x-axis direction, is the velocity in the y-axis direction, is the angular velocity, V is the velocity at the center of mass of the mobile robot, It is expressed as the angle between the mobile robot and the X-axis of the reference coordinate system, l is the distance between the front and rear wheels, and δ f is the front wheel turning angle.

3. The mobile robot trajectory tracking control method based on fuzzy reasoning and reinforcement learning according to claim 1 is characterized in that: The first front wheel steering angle is: Among them, u(t) is the overall output of the fuzzy PID controller, K p , K i , K d Represent the basic values ​​of the proportional coefficient, integral coefficient, and differential coefficient respectively, e(t) represents the error at time t, Δk p Represents the change in proportionality coefficient, Δk i is the change in the integral coefficient, Δk d The change value of the differential coefficient.

4. The mobile robot trajectory tracking control method based on fuzzy reasoning and reinforcement learning according to claim 1 is characterized in that: The DDPG model includes: an actor network and a critic network; The actor network is used to estimate the state and action at the next moment; The critic network is used to estimate the state quantity at the next moment.

Images