Unmanned mine card path tracking control method based on online self-learning

Abstract

The invention relates to an unmanned mine card path tracking control method based on online self-learning, belongs to the technical field of unmanned vehicle path tracking control, and aims to update network model parameters in real time in a vehicle running process. The defects of a traditional unmanned mine card path tracking control method in the aspects of adapting to path conditions, vehicle state changes and the like are broken through, and the problem of how to ensure safe and accurate path tracking control of an unmanned mine card under the condition that external environments such as the path conditions, the vehicle states and the like change continuously is solved. The purposes of ensuring that the path tracking control effect is not worsened and even the tracking control precision is still stably and well changed under the road condition and the vehicle state change condition, and improving the path tracking control safety and scene adaptability of the unmanned mine truck are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned vehicle path tracking control, and in particular to an unmanned mining truck path tracking control method based on online self-learning, and in particular to an unmanned mining truck online self-learning path tracking control algorithm applied to open-pit mines. Background Art

[0002] Path tracking is a method of executing control in unmanned driving. It mainly calculates the expected front wheel angle or the expected steering wheel angle based on the vehicle's current position, heading and other states and the positional relationship between the vehicle and the expected path. At present, traditional path tracking control methods, whether they are pure tracking, Stanley and other model-free control methods, or LQR, MPC and other model-based control methods, require a set of applicable parameters to be set in advance. However, a single set of control parameters is difficult to meet all control requirements. When the path conditions or vehicle status change, the parameters need to be manually re-adjusted; and the manually adjusted parameters often make it difficult to make the control effect close to or even optimal in all cases.

[0003] On this basis, in recent years, some scholars have proposed to apply reinforcement learning (RL) to the field of unmanned vehicle path tracking control, using the exploration learning ability of reinforcement learning to train the control model so that it can adapt to most path conditions and vehicle states, so that the vehicle path tracking control effect is always close to the optimal state. Among them, some methods abandon the traditional algorithm and only use the reinforcement learning algorithm to directly output the control quantity; some methods combine the traditional algorithm with the reinforcement learning algorithm, or use reinforcement learning to adjust the parameters in the traditional algorithm, or make the reinforcement learning algorithm and the traditional method compensate each other.

[0004] In general, the above two methods still have the following shortcomings: (1) It is difficult for a single parameter of the traditional control algorithm to meet all control requirements, and it is impossible to ensure that the control effect is always close to the optimal level. In addition, there are problems such as difficulty in parameter adjustment and high maintenance cost. (2) The vehicle state will change continuously during driving, and steering failures, median deviation and other problems may occur. Most existing model algorithms are difficult to adjust in response to vehicle state degradation. (3) At present, most of the control methods based on reinforcement learning are fixed after training. For safety and other reasons, the model parameters will not be updated during vehicle operation. However, the fixed model cannot guarantee that it is fully applicable to changing paths and vehicles.

[0005] Chinese patent CN118426294A designs a super-helical sliding mode controller to suppress system chattering, and realizes online self-tuning of sliding mode gain by designing an adaptive law, which can solve the problem of being unable to control the robot in real time and facing large motion errors on complex paths. However, the adaptive law of the modified invention is only designed before the control starts, and it is impossible to make real-time fine-tuning according to the external environment and tracking status during the control process.

[0006] Chinese patent CN118050994A established a tracking error dynamics model that takes into account multi-channel uncertainty disturbances, and designed a feedforward compensator to accurately estimate the multi-channel uncertainty disturbances in the vehicle path tracking process in real time, thereby improving the vehicle's anti-interference ability during the path tracking process and ensuring control accuracy and stability. However, the tracking error dynamics model established by the invention still cannot be updated in real time according to the tracking state during the path tracking process, resulting in the model being unable to perfectly adapt to all states.

[0007] Chinese patent CN118144780A establishes a vehicle operation state parameter model based on a three-degree-of-freedom dynamics model and an MPC algorithm, and uses a filtering algorithm to optimize the vehicle operation state parameters at the next moment according to the vehicle state parameters, thereby completing path tracking control. However, the parameter model established by the invention is difficult to optimize and learn by itself during actual operation, and it is impossible to ensure that the initial model is suitable for all subsequent tracking states. Summary of the invention

[0008] In view of the above problems, the present invention provides a path tracking control method for unmanned mine trucks based on online self-learning, and proposes to update the network model parameters in real time during the operation of the vehicle. It breaks through the deficiencies of the traditional path tracking control method for unmanned mine trucks in adapting to path conditions and vehicle state changes, and solves the problem of how to ensure safe and accurate path tracking control for unmanned mine trucks under the condition of constantly changing external environments such as path conditions and vehicle states. It achieves the purpose of ensuring that the path tracking control effect will not deteriorate and even the tracking control accuracy will still change steadily and improve under the conditions of changing road conditions and vehicle states, thereby improving the safety and scene adaptability of the path tracking control of unmanned mine trucks.

[0009] The present invention provides an unmanned mining truck path tracking control method based on online self-learning, comprising:

[0010] Step S1, building a reinforcement learning framework based on the reinforcement learning Actor-Critic algorithm theory; designing the reinforcement learning state space, action space and reward function and embedding them into the reinforcement learning framework to obtain a reinforcement learning model;

[0011] Step S2, let t = 1, when t = 1, it is the initial time;

[0012] Step S3: Obtain the vehicle path tracking state at time t and the path information at time t, and input them into the reinforcement learning model A t ; Get multiple preview points at time t; Get the preview lateral deviation and preview azimuth deviation of each preview point at time t;

[0013] Step S4, comprehensively calculating the preview lateral deviation and the preview azimuth deviation of each preview point at time t according to the ratio to obtain the equivalent preview lateral deviation at time t and the equivalent preview azimuth deviation at time t;

[0014] Step S5, converting the equivalent lateral deviation at time t and the equivalent azimuth deviation at time t based on PID control to obtain a control amount at time t; representing the control amount at time t as the desired front wheel steering angle of the vehicle at time t;

[0015] Step S6, obtaining a reward value of the expected front wheel turning angle of the vehicle at time t;

[0016] Step S7: Based on the reinforcement learning online self-learning algorithm, compare the reinforcement learning reward value at time t with the reinforcement learning reward value at time t-1 to see if it is reduced. If not, update the network parameters to obtain the reinforcement learning model A. t+1 , let t=t+1, return to step S2, until the reinforcement learning reward value at time t is lower than the reinforcement learning reward value at time t-1; obtain the final reinforcement learning model;

[0017] Step S8: Using the final reinforcement learning model to track the vehicle path.

[0018] Preferably, the ratio in step S4 is the ratio of preview near point: preview mid point: preview far point, and the ratio relationship is 3:5:2.

[0019] Preferably, the specific steps of obtaining the preview lateral deviation and preview azimuth deviation of each preview point at time t in step S3 include:

[0020] Step S31: Preset preview distance l d ;Determine the desired path;

[0021] Step S32, taking the center of the rear axle of the vehicle as the origin and the preview distance as the radius, dividing the range to obtain a corresponding circular range, finding the intersection of the circular range and the expected path, marking it as a preview path point, and obtaining a plurality of preview path points;

[0022] Step S33, selecting the preview near point, preview mid point, and preview far point according to the preview time;

[0023] Step S34, respectively calculating the preview deviation one corresponding to the preview near point;

[0024] Preview deviation 2 corresponding to the preview midpoint;

[0025] Preview deviation three corresponding to the preview far point; the preview deviation includes: preview lateral deviation and preview azimuth deviation,

[0026] Step S35, comprehensively calculating the preview deviation 1, the preview deviation 2 and the preview deviation 3 according to the proportion to obtain an equivalent preview lateral deviation and an equivalent preview azimuth deviation;

[0027] Furthermore, the ratio of the preview near point: the preview midpoint: the preview far point in step S35 is 3:5:2.

[0028] Furthermore, the preview distance l in step S31 d The expression is:

[0029] l d =k v v+l0

[0030] Among them, v is the current vehicle speed, k v is the speed coefficient, and l0 is the basic preview distance.

[0031] Furthermore, the specific steps of obtaining the preview lateral deviation in step S4 include:

[0032] Determine the vehicle rear axle center O coordinate (x O ,y O ), preview path point A coordinates (x A ,y A ), the coordinates of the previous point of the preview path point (x B ,y B ), based on the preview path point A coordinates (x A ,y A ) and the coordinates of the previous point of the preview path point (x B ,y B ) to obtain the preview lateral deviation e of the vehicle at the preview path point A p ;like Figure 2 ;

[0033] Furthermore, the preview lateral deviation e p The expression is:

[0034]

[0035] It can be understood that the preview lateral deviation e of the vehicle at the preview path point A is p The distance from the center of the vehicle's rear axle O to the line connecting the preview path point A and the point before the preview path point:

[0036] Furthermore, the expression of the preview azimuth deviation α is:

[0037]

[0038] Among them, x P is the x-axis coordinate of the preview point, y P is the y-axis coordinate of the preview point, x N is the x-axis coordinate of the current path point, y N is the y-axis coordinate of the current path point, and θ is the angle between the longitudinal axis of the vehicle and the horizontal axis of the ground coordinate system.

[0039] Furthermore, the equivalent preview lateral deviation e equ and the equivalent preview azimuth deviation α equ , the expressions are:

[0040] e equ =b1e p1 +b2e p2 +b3e p3

[0041] α equ =b1α1+b2Δ2+b3α3

[0042] Among them, b1 is the weight coefficient corresponding to the near point of preview aiming, b2 is the weight coefficient corresponding to the midpoint of preview aiming, and b3 is the weight coefficient corresponding to the far point of preview aiming, satisfying the relationship b1+b2+b3=1, e p1 is the lateral deviation of the preview near point, e p2 is the lateral deviation of the preview midpoint, e p3 is the lateral deviation of the preview far point, e equ is the equivalent lateral deviation, α1 is the azimuth deviation of the preview near point, Δ2 is the azimuth deviation of the preview midpoint, Δ3 is the azimuth deviation of the preview far point, α equ is the equivalent azimuth deviation.

[0043] Furthermore, the expression of the desired front wheel turning angle of the vehicle in step S6 is:

[0044] θ t =K p (e t -e t-1 )+K i e t +K d (e t -2e t-1 +e t-2 )

[0045] Among them, θ t is the expected front wheel turning angle at time t; K p is the proportionality coefficient, K i Integral coefficient, K d Differential coefficient; et is the error at time t.

[0046] Preferably, the expression of the state space s in step S1 is:

[0047] s=[ρ1,ρ2,ρ3,e1,e2,e3,α1,α2,α3,v,ω]

[0048] Among them, ρ1 is the preview near point curvature, ρ2 is the preview midpoint curvature, ρ3 is the preview far point curvature, e1 is the preview near point lateral deviation, e2 is the preview midpoint lateral deviation, e3 is the preview far point lateral deviation, α1 is the preview near point azimuth deviation, α2 is the preview midpoint azimuth deviation, e3 is the preview far point azimuth deviation, v is the current vehicle speed, ω is the current yaw angular velocity.

[0049] Preferably, the expression of the action space a in step S1 is:

[0050] a=[l0,b1,b2,b3,K p ,K i ,K d ]

[0051] Among them, l0 is the basic preview distance, b1 is the preview near point weight coefficient, b2 is the preview midpoint weight coefficient, b3 is the preview far point weight coefficient, K p is the proportionality coefficient, K i is the integration coefficient, K d is the differential coefficient.

[0052] Preferably, the step S1 reinforcement learning reward function includes a safety reward and a comfort reward;

[0053] It is understood that the safety bonus takes into account the distance of the vehicle from the left and right boundaries, the lateral deviation and the azimuth deviation.

[0054] The comfort reward takes into account the change in the vehicle's yaw rate, and the weighted sum of the safety reward and the comfort reward is the final reward function.

[0055] Furthermore, the expression of the safety reward is:

[0056] r d =|d x |

[0057]

[0058]

[0059] Among them, r d is the boundary distance reward, r l is the lateral deviation reward, r a is the azimuth deviation reward, dx is the distance from the boundary, e p is the lateral deviation, and α is the azimuth deviation.

[0060] Furthermore, the expression of the smoothness reward is:

[0061] dω=ω t -ω t-1

[0062]

[0063] Where dω is the change in yaw rate, ω t is the yaw angular velocity at the current moment, ω t-1 is the yaw angular velocity at the previous moment; r s Reward for smoothness.

[0064] Compared with the prior art, the present invention has at least the following beneficial effects:

[0065] (1) The present invention can adjust the control parameters in real time according to the path tracking control state and path characteristic information, and can ensure that the path tracking effect is always near the optimal level;

[0066] (2) The method of the present invention can adapt to a variety of path conditions and a variety of vehicle states. The model can be adjusted according to different paths and vehicle states, reducing the impact of changes in path conditions and vehicle states on the path tracking effect;

[0067] (3) The present invention can continuously learn and optimize the network model during the operation of the vehicle, and can continuously improve the path tracking effect on the original basis. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The drawings are only for the purpose of illustrating particular embodiments and are not to be construed as limiting the invention.

[0069] Figure 1 A schematic diagram of multi-point preview in an embodiment of the present invention;

[0070] Figure 2 A schematic diagram of the calculation of the preview lateral deviation in an embodiment of the present invention;

[0071] Figure 3 A schematic diagram of the calculation of the preview azimuth angle deviation in an embodiment of the present invention;

[0072] Figure 4 Schematic diagram of the simulation training and real vehicle testing framework in an embodiment of the present invention. DETAILED DESCRIPTION

[0073] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. In addition, the present invention can also be implemented in other ways different from those described herein, and therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0074] A specific embodiment of the present invention, as Figure 1-4 , discloses an unmanned mining truck path tracking control method based on online self-learning; in order to illustrate the effectiveness of the method proposed in the present invention, the above technical solution of the present invention is described in detail through a specific embodiment below, and the specific implementation steps are as follows:

[0075] The present invention provides an unmanned mining truck path tracking control method based on online self-learning, comprising:

[0076] Step S1, building a reinforcement learning framework based on the reinforcement learning Actor-Critic algorithm theory; designing the reinforcement learning state space, action space and reward function and embedding them into the reinforcement learning framework to obtain a reinforcement learning model;

[0077] Step S2, let t = 1, when t = 1, it is the initial time;

[0078] Step S3: Obtain the vehicle path tracking state at time t and the path information at time t, and input them into the reinforcement learning model A t ; Get multiple preview points at time t; Get the preview lateral deviation and preview azimuth deviation of each preview point at time t;

[0079] Step S4, comprehensively calculating the preview lateral deviation and the preview azimuth deviation of each preview point at time t according to the ratio to obtain the equivalent preview lateral deviation at time t and the equivalent preview azimuth deviation at time t;

[0080] Step S5, converting the equivalent lateral deviation at time t and the equivalent azimuth deviation at time t based on PID control to obtain a control amount at time t; representing the control amount at time t as the desired front wheel steering angle of the vehicle at time t;

[0081] Step S6, obtaining a reward value of the expected front wheel turning angle of the vehicle at time t;

[0082] Step S7: Based on the reinforcement learning online self-learning algorithm, compare the reinforcement learning reward value at time t with the reinforcement learning reward value at time t-1 to see if it is reduced. If not, update the network parameters to obtain the reinforcement learning model A. t+1, let t=t+1, return to step S2, until the reinforcement learning reward value at time t is lower than the reinforcement learning reward value at time t-1; obtain the final reinforcement learning model;

[0083] Step S8: Using the final reinforcement learning model to track the vehicle path.

[0084] Preferably, the ratio in step S4 is the ratio of preview near point: preview mid point: preview far point, and the ratio relationship is 3:5:2.

[0085] Preferably, the specific steps of obtaining the preview lateral deviation and preview azimuth deviation of each preview point at time t in step S3 include:

[0086] Step S31: Preset preview distance l d ;Determine the desired path;

[0087] Step S32, taking the center of the rear axle of the vehicle as the origin and the preview distance as the radius, dividing the range to obtain a corresponding circular range, finding the intersection of the circular range and the expected path, marking it as a preview path point, and obtaining a plurality of preview path points;

[0088] Step S33, selecting the preview near point, preview mid point, and preview far point according to the preview time;

[0089] Step S34, respectively calculating the preview deviation one corresponding to the preview near point;

[0090] Preview deviation 2 corresponding to the preview midpoint;

[0091] Preview deviation three corresponding to the preview far point; the preview deviation includes: preview lateral deviation and preview azimuth deviation,

[0092] Step S35, comprehensively calculating the preview deviation 1, the preview deviation 2 and the preview deviation 3 according to the proportion to obtain an equivalent preview lateral deviation and an equivalent preview azimuth deviation;

[0093] Furthermore, the ratio of the preview near point: the preview midpoint: the preview far point in step S35 is 3:5:2.

[0094] Furthermore, the preview distance l in step S31 d The expression is:

[0095] l d =k v v+l0

[0096] Among them, v is the current vehicle speed, k v is the speed coefficient, and l0 is the basic preview distance.

[0097] Furthermore, the specific steps of obtaining the preview lateral deviation in step S4 include:

[0098] Determine the vehicle rear axle center O coordinate (x o ,y O ), preview path point A coordinates (x A ,y A ), the coordinates of the previous point of the preview path point (x B ,y B ), based on the preview path point A coordinates (x A ,y A ) and the coordinates of the previous point of the preview path point (x B ,y B ) to obtain the preview lateral deviation e of the vehicle at the preview path point A p ;like Figure 2 ;

[0099] Furthermore, the preview lateral deviation e p The expression is:

[0100]

[0101] It can be understood that the preview lateral deviation e of the vehicle at the preview path point A is p The distance from the center of the vehicle's rear axle O to the line connecting the preview path point A and the point before the preview path point:

[0102] Furthermore, the expression of the preview azimuth deviation α is:

[0103]

[0104] Among them, x P is the x-axis coordinate of the preview point, y P is the y-axis coordinate of the preview point, x N is the x-axis coordinate of the current path point, y N is the y-axis coordinate of the current path point, and θ is the angle between the longitudinal axis of the vehicle and the horizontal axis of the ground coordinate system.

[0105] Furthermore, the equivalent preview lateral deviation e equ and the equivalent preview azimuth deviation α equ , the expressions are:

[0106] e equ =b1e p1 +b2e p2 +b3e p3

[0107] α equ =b1α1+b2α2+b3α3

[0108] Among them, b1 is the weight coefficient corresponding to the near point of preview aiming, b2 is the weight coefficient corresponding to the midpoint of preview aiming, and b3 is the weight coefficient corresponding to the far point of preview aiming, satisfying the relationship b1+b2+b3=1, e p1 is the lateral deviation of the preview near point, e p2 is the lateral deviation of the preview midpoint, e p3 is the lateral deviation of the preview far point, e equ is the equivalent lateral deviation, α1 is the azimuth deviation of the preview near point, α2 is the azimuth deviation of the preview midpoint, α3 is the azimuth deviation of the preview far point, and α equ is the equivalent azimuth deviation.

[0109] Furthermore, the expression of the desired front wheel turning angle of the vehicle in step S6 is:

[0110] θ t =K p (e t -e t-1 )+K i e t +K d (e t -2e t-1 +e t-2 )

[0111] Among them, θ t is the expected front wheel turning angle at time t; K p is the proportionality coefficient, K i Integral coefficient, K d Differential coefficient; e t is the error at time t.

[0112] Preferably, the expression of the state space s in step S1 is:

[0113] s=[ρ1,ρ2,ρ3,e1,e2,e3,α1,α2,α3,v,ω]

[0114] Among them, ρ1 is the preview near point curvature, ρ2 is the preview midpoint curvature, ρ3 is the preview far point curvature, e1 is the preview near point lateral deviation, e2 is the preview midpoint lateral deviation, e3 is the preview far point lateral deviation, α1 is the preview near point azimuth deviation, α2 is the preview midpoint azimuth deviation, e3 is the preview far point azimuth deviation, v is the current vehicle speed, ω is the current yaw angular velocity.

[0115] Preferably, the expression of the action space a in step S1 is:

[0116] a=[l0,b1,b2,b3,K p ,K i ,K d ]

[0117] Among them, l0 is the basic preview distance, b1 is the preview near point weight coefficient, b2 is the preview midpoint weight coefficient, b3 is the preview far point weight coefficient, K p is the proportionality coefficient, K i is the integration coefficient, K d is the differential coefficient.

[0118] Preferably, the step S1 reinforcement learning reward function includes a safety reward and a comfort reward;

[0119] It is understood that the safety bonus takes into account the distance of the vehicle from the left and right boundaries, the lateral deviation and the azimuth deviation.

[0120] The comfort reward takes into account the change in the vehicle's yaw rate, and the weighted sum of the safety reward and the comfort reward is the final reward function.

[0121] Furthermore, the expression of the safety reward is:

[0122] r d =|d x |

[0123]

[0124]

[0125] Among them, r d is the boundary distance reward, r l is the lateral deviation reward, r a is the azimuth deviation reward, d x is the distance from the boundary, e p is the lateral deviation, and α is the azimuth deviation.

[0126] Furthermore, the expression of the smoothness reward is:

[0127] dω=ω t -ω t-1

[0128]

[0129] Where dω is the change in yaw rate, ω t is the yaw angular velocity at the current moment, ω t-1 is the yaw angular velocity at the previous moment; r s Reward for smoothness.

[0130] like Figure 4As shown, a model pre-training and online self-learning framework is designed. In the model pre-training stage, in order to avoid safety issues caused by model instability in the early stage of model training, the model training includes two devices, one device is used to carry the Prescan-simulink model and provide simulated vehicle information, and the other device is used to run the RL program. UDP communication is used between the two. The simulation transmits path information and vehicle status to the RL model. The RL model receives the status and calculates various parameters and calculates the expected front wheel angle to pass it to the simulation environment. The online self-learning algorithm designed by the present invention can effectively prevent the problem of reward reduction in the model pre-training stage and accelerate the model convergence speed.

[0131] After a relatively stable model is obtained through simulation training, the model is deployed to a real vehicle for online self-learning testing. The real vehicle test uses an external device to run the RL model program, receives the real vehicle status and outputs various parameters to the real vehicle control unit, and the real vehicle calculation unit calculates the final vehicle front wheel turning angle. Using the online self-learning algorithm designed by the present invention, the real vehicle can continue to perform model training and updating in real time during vehicle operation based on the pre-trained model, and the control effect is continuously improved while ensuring safety, as shown in Table 1.

[0132] Table 1 Online self-learning real vehicle test results

[0133]

[0134]

[0135] From the results in Table 1, we can see that when the maximum speed of the actual vehicle is 15km / h, the lateral control accuracy of the vehicle is continuously improved. The maximum lateral deviation of the first trip is -0.335m, and the maximum lateral deviation has dropped to 0.265m in the fifth trip.

[0136] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A path tracking control method for unmanned mining trucks based on online self-learning, characterized in that: include: Step S1: Building a reinforcement learning framework based on the reinforcement learning Actor-Critic algorithm theory; Design a reinforcement learning state space, action space, and reward function and embed them into the reinforcement learning framework to obtain a reinforcement learning model; Step S2, let t = 1, when t = 1, it is the initial time; Step S3: Obtain the vehicle path tracking state at time t and the path information at time t, and input them into the reinforcement learning model A t ; Get multiple preview points at time t; get the preview lateral deviation and preview azimuth deviation of each preview point at time t; Step S4, comprehensively calculating the preview lateral deviation and the preview azimuth deviation of each preview point at time t according to a certain ratio to obtain an equivalent preview lateral deviation at time t and an equivalent preview azimuth deviation at time t; Step S5, converting the equivalent lateral deviation at time t and the equivalent azimuth deviation at time t based on PID control to obtain a control amount at time t; representing the control amount at time t as the desired front wheel steering angle of the vehicle at time t; Step S6, obtaining a reward value of the expected front wheel turning angle of the vehicle at time t; Step S7: Based on the reinforcement learning online self-learning algorithm, compare the reinforcement learning reward value at time t with the reinforcement learning reward value at time t-1 to see if it is reduced. If not, update the network parameters to obtain the reinforcement learning model A. t+1 , let t=t+1, return to step S2, until the reinforcement learning reward value at time t is lower than the reinforcement learning reward value at time t-1; obtain the final reinforcement learning model; Step S8: Using the final reinforcement learning model to track the vehicle path.

2. The unmanned mining truck path tracking control method based on online self-learning according to claim 1 is characterized in that: The reinforcement learning reward function described in step S1 includes a safety reward and a comfort reward.

3. The unmanned mining truck path tracking control method based on online self-learning according to claim 2 is characterized in that: The safety bonus takes into account the distance of the vehicle from the left and right boundaries, the lateral deviation, and the azimuth deviation; The comfort bonus takes into account vehicle yaw rate changes.

4. The unmanned mining truck path tracking control method based on online self-learning according to claim 1 is characterized in that: The specific steps of obtaining the preview lateral deviation and preview azimuth deviation of each preview point at time t in step S3 include: Step S31: Preset preview distance l d ;Determine the desired path; Step S32, taking the center of the rear axle of the vehicle as the origin and the preview distance as the radius, dividing the range to obtain a corresponding circular range, finding the intersection of the circular range and the expected path, marking it as a preview path point, and obtaining a plurality of preview path points; Step S33, selecting the preview near point, preview mid point, and preview far point according to the preview time; Step S34, respectively calculating the preview deviation one corresponding to the preview near point; Preview deviation 2 corresponding to the preview midpoint; Preview deviation three corresponding to the preview far point; the preview deviation includes: preview lateral deviation and preview azimuth deviation, Step S35, comprehensively calculate the preview deviation 1, the preview deviation 2 and the preview deviation 3 according to the proportion to obtain an equivalent preview lateral deviation and an equivalent preview azimuth deviation.

5. The unmanned mining truck path tracking control method based on online self-learning according to claim 1 is characterized in that: The specific steps of obtaining the preview lateral deviation in step S4 include: Determine the vehicle rear axle center O coordinate (x o ,y o ), preview path point A coordinates (x A ,y A ), the coordinates of the previous point of the preview path point (x B ,y B ), based on the preview path point A coordinates (x A ,y A ) and the coordinates of the previous point of the preview path point (x B ,y b ) to obtain the preview lateral deviation e of the vehicle at the preview path point A p .

6. The unmanned mining truck path tracking control method based on online self-learning according to claim 5 is characterized in that: The preview lateral deviation e p The expression is: Among them, x A is the x-axis coordinate of the preview path point A, x B is the x-axis coordinate of the point before the preview path point, y A The y-axis coordinate of the preview path point A, y B is the y-axis coordinate of the point before the preview path point, x O is the x-axis coordinate of the center O of the rear axle of the vehicle, y O is the y-axis coordinate of the center O of the vehicle's rear axle.

7. The unmanned mining truck path tracking control method based on online self-learning according to claim 4 is characterized in that: The expression of the preview azimuth deviation α is: Among them, x P is the x-axis coordinate of the preview point, y P is the y-axis coordinate of the preview point, x N is the x-axis coordinate of the current path point, y N is the y-axis coordinate of the current path point, and θ is the angle between the longitudinal axis of the vehicle and the horizontal axis of the ground coordinate system.

8. The unmanned mining truck path tracking control method based on online self-learning according to claim 4 is characterized in that: The equivalent preview lateral deviation e equ and the equivalent preview azimuth deviation α equ , the expressions are: yes equ =b1e p1 +b2e p2 +b3e p3 a equ =b1α1+b2α2+b3α3 Among them, b1 is the weight coefficient corresponding to the near point of preview aiming, b2 is the weight coefficient corresponding to the midpoint of preview aiming, b3 is the weight coefficient corresponding to the far point of preview aiming, and e p1 is the lateral deviation of the preview near point, e p2 is the lateral deviation of the preview midpoint, e p3 is the lateral deviation of the preview far point, e equ is the equivalent lateral deviation, α1 is the azimuth deviation of the preview near point, α2 is the azimuth deviation of the preview midpoint, α3 is the azimuth deviation of the preview far point, and α equ is the equivalent azimuth deviation.

9. The unmanned mining truck path tracking control method based on online self-learning according to claim 1 is characterized in that: The expression of the vehicle's expected front wheel turning angle in step S6 is: θ t =K p (e t -e t-1 )+K i e t +K d (e t -2e t-1 +e t-2 ) Among them, θ t is the expected front wheel turning angle at time t; K o is the proportionality coefficient, K i Integration coefficient, K d Differential coefficient; e t is the error at time t.

Images