A mine track vehicle trajectory tracking method and system based on reinforcement learning

Abstract

The application discloses a mine track vehicle trajectory tracking method and system based on reinforcement learning, and relates to the technical field of intelligent control of mine machinery.A dynamic environment state matrix is constructed by collecting vehicle state, obstacles and road characteristics through a plurality of source sensors; a left and right track speed difference is output by a DDPG intelligent agent, and is mapped to the advancing speed and steering angle of the track vehicle; a future trajectory is predicted by using an ESN, a lead deviation is calculated and is corrected by a penalty term; a mutation section is identified based on the predicted trajectory curvature, and is smoothed and optimized by using an adaptive cubic B-spline.The application realizes relatively accurate tracking of the track vehicle trajectory, and improves adaptability and safety in a complex environment.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for mining machinery, specifically to a method and system for tracking the trajectory of tracked mining vehicles using reinforcement learning. Background Technology

[0002] As a key piece of equipment in mining operations, the accuracy of track tracking of mining tracked vehicles directly affects operational efficiency. However, the mining environment is highly complex and dynamic, which places stringent requirements on track tracking technology.

[0003] Existing tracking methods for tracked mining vehicles have several limitations: First, traditional control methods rely on precise vehicle dynamics models and environmental parameters, but parameters such as road surface friction coefficient and slope change dynamically in the mining environment, making it difficult for the model to adapt in real time, which can easily lead to increased tracking deviations. Second, while some reinforcement learning-based methods can adapt to nonlinear environments, they mostly focus on feedback adjustments to the current state and lack the ability to predict and correct future trajectories. Third, insufficient fusion of multi-sensor data makes it difficult to comprehensively integrate information on vehicle status, obstacles, and road surface features, resulting in biased decision-making.

[0004] Therefore, there is an urgent need for a trajectory tracking method that can accurately perceive the dynamic environment and achieve advanced prediction and dynamic smoothing optimization. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a reinforcement learning-based method and system for tracking the trajectory of tracked mining vehicles, which solves the problems of low trajectory tracking accuracy and poor safety in complex environments for tracked mining vehicles.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a reinforcement learning-based method for tracking the trajectory of a tracked mining vehicle, comprising: By using lidar, millimeter-wave radar, inertial navigation system and vision camera, key variables in three aspects, namely vehicle status, obstacles and road surface features, are collected to obtain dynamic environment state matrix; Construct a deep deterministic policy gradient DDPG agent, which takes the dynamic environment state matrix as input and outputs the speed difference between the left and right tracks; Echo State Network (ESN) is used to predict the future trajectory of the tracked vehicle. The three-dimensional lead deviation between the predicted trajectory and the expected trajectory is calculated. If the deviation exceeds the safety threshold at a certain moment, a lead penalty term is added to the reward function of DDPG. The trajectory curvature between two adjacent points is calculated based on the predicted trajectory coordinates. The abrupt change segments of the predicted trajectory are marked, and the abrupt change segments are smoothed and corrected using adaptive cubic B-splines.

[0007] As a further aspect of the present invention, the key variables of the vehicle state include three-dimensional position coordinates, three-dimensional attitude angles, three-dimensional acceleration components, and instantaneous velocity; the key variables of the nearest obstacle include the obstacle's three-dimensional coordinates, size, relative velocity, and reflection intensity; and the key variables of the road surface features include slope vector, texture features, smoothness, and illumination intensity.

[0008] As a further aspect of the present invention, the specific steps for training the DDPG agent are as follows: Construct the current and target networks for Actor and Critic in DDPG, and create an experience replay pool; The agent in the current state Below, basic actions are output through the Actor network. The exploration action is obtained by superimposing OU noise. ; Will This is directly mapped to the tracked vehicle's speed and steering angle, resulting in a new state. and rewards and will Store in the playback pool; When the number of replay samples is greater than or equal to Nth, Nth empirical samples are randomly selected, and the target Q-value is calculated using the Critic target network. And calculate the predicted Q value using the current Critic network: ;in, Nth is the discount factor, and Nth is the sample size threshold; According to the formula Calculate the loss function L of the Critic network and update the current Critic and Actor network parameters using the Adam optimizer; Based on soft update coefficient Update the target network parameters. Evaluate the policy once every 1000 iterations. If the average reward fluctuation of 5 consecutive evaluations is ≤5%, then convergence is determined and training is stopped.

[0009] As a further aspect of the present invention, both Actor and Critic employ fully connected neural networks, with specific parameter settings as follows: The input layer of the Actor is a 24-dimensional state vector, and... Consistent; There are two hidden layers. The first layer has 64 neurons and the second layer has 32 neurons. The activation function for both layers is ReLU, and the weights of the hidden layers are initialized using He normal distribution. The output layer has one neuron, with Tanh as the activation function, and is converted into the difference in basic rotational speed between the left and right tracks through a linear mapping. The specific formula is as follows: ; Where K is the absolute value of the maximum permissible speed difference between the left and right tracks of the tracked vehicle; Select the Adam optimizer, learning rate Set to 0.0001; The input layer of Critic is a 25-dimensional vector, including a 24-dimensional state vector and a 1-dimensional action vector; There are 2 hidden layers. The first layer has 128 neurons, and the second layer has 64 neurons. The activation function for both layers is ReLU. The output layer has one neuron, and the Adam optimizer is selected with a learning rate of [missing information]. Set to 0.001.

[0010] As a further aspect of the present invention, the The specific formula is: ; Where w1, w2, and w3 are weight coefficients and satisfy the following conditions: , , , This represents the three-dimensional deviation between the vehicle's current position and the desired trajectory reference point, where safe(t) represents the safety bonus, and its specific formula is: ,in, , These are the lower and upper limits of the distance, respectively. This represents the minimum three-dimensional distance between the vehicle and the nearest obstacle.

[0011] As a further aspect of the present invention, This is directly mapped to the tracked vehicle's travel speed and steering angle: The travel speed of a tracked vehicle is determined by the average speed of the left and right tracks, according to the formula... Calculate the travel speed v, where k is the proportionality coefficient, specifically... , where i is the transmission ratio of the track drive system. , Indicates the rotational speed of the left and right tracks; The steering of a tracked vehicle is determined by the speed difference. The decision, specifically the formula, is as follows: Where m is the proportionality coefficient and dis is the track spacing, which is the distance between the center lines of the left and right tracks. Sampling time.

[0012] As a further aspect of the present invention, the specific parameters for constructing the ESN network include: The total dimensions of the input layer are set to 18, including the control variables output by DDPG. Current environment state matrix The first three steps of three-dimensional trajectory data, the current environment state matrix Includes the vehicle's current three-dimensional coordinates, heading angle, roll angle, speed, distance to the nearest obstacle, and road surface friction coefficient; The reservoir size is set to 120 neurons, the connection probability is 25%, the spectral radius is 0.9, the leakage rate is 0.3, and the activation function is tanh. The output layer has a dimension of 60, and ridge regression is used to train the output weights.

[0013] As a further aspect of the present invention, the three-dimensional lead deviation between the predicted trajectory and the expected trajectory is calculated using the following formula: , where i∈[1,20] and is an integer, representing the label of the three-dimensional trajectory point; Set the base threshold Based on its construction of actual dynamic thresholds ,in, The current speed of the tracked vehicle. This represents the maximum travel speed of the tracked vehicle. If the deviation at any time Then, add a lead penalty term to the reward function of DDPG. : The weights after adding this term satisfy the following conditions: .

[0014] As a further aspect of the present invention, the specific steps for marking the abrupt change segments of the predicted trajectory are as follows: Calculate the trajectory curvature between two adjacent points based on the predicted trajectory coordinates. It can be expressed as the arithmetic mean of the plane curvature and the vertical curvature: ,in, For plane curvature, Vertical curvature; According to the formula Calculate the rate of change of curvature, if If so, then the trajectory between the two adjacent points is marked as a mutation trajectory; Find the rate of change of curvature from the predicted trajectory. The point at which the current risk level threshold is first exceeded is taken as the starting point, denoted as... ,until The point where the value first falls below the threshold is taken as the endpoint, denoted as... And mark the continuous trajectory segment between these two points as the abrupt change segment.

[0015] A reinforcement learning-based trajectory tracking system for tracked mining vehicles includes: The data acquisition module uses lidar, millimeter-wave radar, inertial navigation system and vision camera to collect key variables in three aspects: vehicle status, obstacles and road features, and obtain dynamic environment status matrix. The DDPG training module constructs a deep deterministic policy gradient DDPG agent, which takes the dynamic environment state matrix as input and outputs the speed difference between the left and right tracks. The deviation correction module uses an echo state network (ESN) to predict the future trajectory of the tracked vehicle and calculates the three-dimensional advance deviation between the predicted trajectory and the expected trajectory. If the deviation exceeds the safety threshold at a certain moment, an advance penalty term is added to the reward function of the DDPG. The trajectory smoothing module calculates the trajectory curvature between two adjacent points based on the predicted trajectory coordinates, marks the abrupt change segments of the predicted trajectory, and uses adaptive cubic B-splines to smooth and correct the abrupt change segments.

[0016] This invention provides a reinforcement learning-based method and system for tracking the trajectory of tracked mining vehicles, which has the following advantages compared with existing technologies: (1) This invention constructs a dynamic environmental state matrix by fusing multiple source sensors and combines it with DDPG intelligent agents to achieve adaptive tracking of complex mining environments, effectively solving the problem of insufficient tracking accuracy caused by dynamic changes in environmental parameters in traditional methods; (2) This invention introduces an ESN network to achieve advance prediction of future trajectories, and through a three-dimensional advance deviation calculation and penalty term mechanism, it corrects potential trajectory deviations in advance, overcoming the lag of existing reinforcement learning methods that only rely on current state feedback. (3) The present invention uses adaptive cubic B-spline to smooth the trajectory change segment and dynamically adjusts the number of control points based on the trajectory length, which effectively reduces the risk of vehicle rollover or collision caused by trajectory change and improves the stability of trajectory tracking. Attached Figure Description

[0017] Figure 1 This is a flowchart of the steps of the present invention; Figure 2 This is the system principle block diagram of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] like Figure 1 This invention provides a reinforcement learning-based method for tracking the trajectory of a tracked mining vehicle, comprising: By utilizing lidar, millimeter-wave radar, inertial navigation systems, and vision cameras, key variables in three aspects—vehicle status, obstacles, and road surface features—are collected to obtain a dynamic environment state matrix. ; Key variables of vehicle status include three-dimensional position coordinates, three-dimensional attitude angles, three-dimensional acceleration components, and instantaneous velocity; The above variables are collected using an inertial navigation system (IMU+GNSS integrated navigation). The specific operation is as follows: When the signal is good, the GNSS module directly outputs the three-dimensional coordinates after latitude and longitude conversion. When the signal is lost, the IMU calculates the position through the integration calculation of the three-axis accelerometer and gyroscope. At this time, it is necessary to match the tunnel map to correct the error. The three-dimensional attitude angles and three-dimensional acceleration components of the current vehicle are collected using the gyroscope and three-axis accelerometer of the IMU, and the instantaneous speed is calculated by combining the speed integral formula. The three-dimensional attitude angles specifically include pitch angle, roll angle and yaw angle. Key variables for obstacles include the obstacle's three-dimensional coordinates, dimensions (length, width, height), relative velocity, and reflection intensity; The above variables are collected using lidar and millimeter-wave radar. The specific operation is as follows: The obstacle around the vehicle is illuminated by a lidar to generate a large amount of three-dimensional point cloud data. The obstacle point set is segmented by a clustering algorithm (such as DBSCAN), and the minimum bounding box of the point set is calculated to obtain the three-dimensional coordinates and size of the obstacle. The three-dimensional coordinates are the center point of the minimum bounding box, and the size is the length, width and height of the minimum bounding box. The radial velocity of an obstacle is detected by the Doppler effect of millimeter-wave radar and converted into relative velocity by combining it with the vehicle's own speed. At the same time, millimeter-wave radar can also output the intensity of the reflected signal of the obstacle to help distinguish the type of obstacle, such as strong reflection of metal obstacles and weak reflection of non-metallic ore piles. Key variables for road surface characteristics include slope vector, texture features, smoothness, and light intensity; The above variables are collected by combining an inertial navigation system, LiDAR, and a visual camera (which can be a dual-camera setup with infrared and RGB sensors). The specific operation is as follows: The pitch and roll angles of the inertial navigation system are used to derive the corresponding longitudinal and lateral slopes. The vertical component of the road slope is usually 0 because the slope vector describes the inclination along the ground. The component perpendicular to the ground does not reflect the slope, thus forming a three-dimensional slope vector. An RGB camera can be selected to capture road surface images. The road surface type is identified based on image texture analysis and digitally encoded. For example, mud is a low-reflectivity dark area, encoded as 0; gravel is a multi-faceted high-contrast area, encoded as 1; and hard road surface is a uniform light-colored high-reflectivity area, encoded as 2. The standard deviation of the z-axis coordinate of the point cloud of the road surface three-dimensional contour acquired by lidar is calculated. The larger the standard deviation, the more rugged the road surface. Calculate the mean grayscale value of the image obtained by the infrared camera. This value reflects the lighting of the surrounding environment more intuitively. For example, the grayscale value is low in the unlit area underground, while the grayscale value is high in the brightly lit open area. This is a structured data matrix at time t, containing all the key variables from the three aspects mentioned above, with a dimension of 1×24.

[0020] Construct a deep deterministic policy gradient (DDPG) agent with a dynamic environment state matrix. As input, output the speed difference between the left and right tracks. ; The specific steps for training the DDPG agent are as follows: Construct the current and target networks of DDPG for Actor (policy network) and Critic (value network), and create an experience replay pool; Both Actor and Critic use fully connected neural networks, with the following parameter settings: The input layer of the Actor is a 24-dimensional state vector, and... Consistent; The hidden layer has 2 layers, and the first layer has 64 neurons. The activation function is ReLU, which is used to handle nonlinear relationships and is suitable for mapping complex states in mines. The second layer has 32 neurons, and the activation function is ReLU, which further compresses features and extracts key decision information; The weights of the hidden layer are initialized using the He normal distribution to adapt to the ReLU activation function and avoid gradient vanishing. By balancing accuracy and efficiency through two hidden layers, overfitting can be avoided while capturing the nonlinear relationship between state and speed difference. The output layer has one neuron, and the activation function chosen is Tanh, which scales the output value to [-1, 1], and then converts it into the basic speed difference between the left and right tracks through a linear mapping. The specific formula is as follows: ; Wherein, K is the absolute value of the maximum permissible speed difference between the left and right tracks of the tracked vehicle, which is determined by the mechanical performance of the tracked vehicle and is the critical value that limits the range of speed difference; Select the Adam optimizer, learning rate Setting the learning rate to 0.0001, a smaller learning rate, can avoid parameter oscillations in sudden mining scenarios; The input layer of Critic is a 25-dimensional vector, including a 24-dimensional state vector and a 1-dimensional action vector; There are 2 hidden layers, with 128 neurons in the first layer, and ReLU is chosen as the activation function. The second layer has 64 neurons, and the activation function chosen is ReLU; The hidden layer has more neurons than actors in order to take into account the interaction between state and action; The output layer has one neuron, the Q-value, which is used to evaluate the long-term reward of the current state-action pair; Select the Adam optimizer, learning rate Set to 0.001 to ensure rapid convergence of the valuation; The capacity of the experience replay pool can be set to 100,000, and the format of a single experience entry can be defined as follows: ,in, This indicates that the agent performs an action at time t. The reward obtained later This represents the state of the agent at time t+1; The specific function formula is as follows: ; Where w1, w2, and w3 are weighting coefficients. , , This represents the three-dimensional deviation between the vehicle's current position and the reference point of the desired trajectory; safe(t) represents the safety bonus, reflecting whether the distance between the vehicle and the obstacle is safe. The above formula The trajectory deviation penalty term is used to incentivize the agent to reduce the three-dimensional deviation by penalizing the vehicle for deviating from the desired trajectory. As a safety incentive, it rewards safe distances and penalizes dangerous distances to prevent vehicles from colliding with obstacles; The value of safe(t) is determined by the minimum three-dimensional distance between the vehicle and the nearest obstacle. The decision, specifically the formula, is as follows: ,in, , These are the lower and upper limits of the distance, respectively. when When this occurs, it indicates a safe state with no risk of collision, and a positive reward should be given. when When this is the case, it indicates that you are in a state of alert and need to pay attention to your surroundings; there will be no reward or punishment. when When this occurs, it indicates a dangerous situation with a high risk of collision, requiring a strong negative reward. As an energy consumption penalty item, it reduces track wear and energy consumption by penalizing violent actions, such as large speed differences; The relationship between the above weights is as follows: The core logic behind this setup is that in high-risk scenarios, avoiding collisions should be prioritized over staying close to the trajectory. Even if the trajectory is temporarily deviated from, as long as a collision does not occur, the task still has the possibility of correction and continuation; however, once a collision occurs, the task will fail directly. The agent in the current state Below, basic actions are output through the Actor network. The exploration action is obtained by superimposing Ornstein-Uhlenbeck (OU) noise. ; OU noise is time-dependent, which is closer to the exploration needs of tracked vehicles in continuous dynamic environments, and is more likely to detect valuable actions than Gaussian noise. according to This is directly mapped to the tracked vehicle's speed and steering angle, resulting in a new state. and rewards and will Store in the playback pool; The travel speed of a tracked vehicle is determined by the average speed of the left and right tracks, according to the formula... Calculate the travel speed v, where k is the proportional coefficient, determined by the track radius and transmission ratio, specifically... 'i' represents the gear ratio of the track drive system, used to convert rotational speed into track linear velocity. , Indicates the rotational speed of the left and right tracks; The steering of a tracked vehicle is determined by the speed difference. The turning angular velocity is determined to be directly proportional to the speed difference and inversely proportional to the track spacing. The specific formula is as follows: Where m is the proportionality coefficient, similar to k above, because the speed difference is ultimately reflected as the difference in linear velocity between the left and right tracks; dis is the track spacing, which is the distance between the center lines of the left and right tracks, and is an inherent parameter of the tracked vehicle. Sampling time; When the number of replay samples is greater than or equal to Nth, Nth empirical samples are randomly selected, and the target Q-value is calculated using the Critic target network. And calculate the predicted Q value using the current Critic network: ;in, This is the discount factor, reflecting the importance of long-term rewards. Nth is the sample size threshold, which needs to be set according to the actual situation. According to the formula Calculate the loss function L of the Critic network and update the current Critic and Actor network parameters using the Adam optimizer; Based on soft update coefficient The target network parameters are updated. Soft updates allow the target network parameters to slowly track the current network, avoiding parameter mutations. 0.001 ensures that the target network is updated by only 1‰ per step, which maintains the stability of the target value and gradually adapts to the current network's optimization direction. The policy is evaluated once every 1000 iterations. If the average reward fluctuation of 5 consecutive evaluations is ≤5%, then convergence is determined and training is stopped. Convergence determination is based on reward fluctuations to ensure that the strategy can consistently output effective actions in different scenarios and avoid overfitting.

[0021] The Echo State Network (ESN) is used to predict the future trajectory of the tracked vehicle. The three-dimensional lead deviation between the predicted trajectory and the expected trajectory is calculated. If the deviation exceeds the safety threshold at a certain moment, a lead penalty term is added to the reward function of the DDPG to prompt the agent to output a correction control quantity. The specific steps for building an ESN network are as follows: The total dimensions of the input layer are set to 18, including the control variables output by DDPG. (1D) Current Environment State Matrix (8-dimensional), the first 3-dimensional trajectory data (9-dimensional), and the current environment state matrix Includes the vehicle's current three-dimensional coordinates, heading angle, roll angle, speed, distance to the nearest obstacle, and road surface friction coefficient; The reserve pool size is set at 120 neurons. Mine trajectory tracking is a medium-complexity task, and the number of neurons needs to be sufficient to capture nonlinearity. Too many neurons will increase redundant calculations. With a connection probability of 25%, sparse connections (non-fully connected) can reduce the redundancy of the reservoir, avoid overfitting, and retain the ability to capture long-term dependencies. The spectral radius is 0.9, which is less than 1 to ensure the stability of the reservoir state and adapt to the continuity of tracked vehicle movement; With a leakage rate of 0.3, the degree of forgetting in the state update of the reserve pool can be controlled, the focus on recent and historical states can be balanced, and prediction bias can be avoided due to outdated historical data. The activation function is tanh; The output layer has a dimension of 60, predicts the sequence of three-dimensional trajectory points within the next 2 seconds, and has a sampling frequency of 10Hz. Therefore, it outputs 20 three-dimensional points at a time, for a total of 60 dimensions. A 2-second prediction window can cover the typical reaction time of a tracked vehicle, while 10Hz sampling ensures that trajectory details are not lost; Ridge regression is used to train the output weights, and the prediction frequency is consistent with the control period. Since ESN has a fixed pool of weights and only trains the output weights, the computational cost of using ridge regression is much less than that of general deep learning models. At the same time, ridge regression imposes penalties on the output weights by introducing L2 regularization, forcing the model to learn mapping relationships with stronger generalization ability. The three-dimensional lead deviation between the predicted trajectory and the expected trajectory is calculated using the following formula: , where i∈[1,20] and is an integer, representing the label of the three-dimensional trajectory point; Set the base threshold A value of 0.6m is generally acceptable, and the actual dynamic threshold can be constructed based on this value. ,in, The current speed of the tracked vehicle. This represents the maximum travel speed of the tracked vehicle. When driving at high speed, the vehicle has a large inertia and the deviation expands faster, so the threshold needs to be lowered to trigger correction in advance; when driving at low speed, the threshold needs to be widened to avoid over-correction that could cause trajectory oscillation. If the deviation at any time Add a lead penalty term to the reward function of DDPG. : The weights after adding this term satisfy the following conditions: This means that safety > current deviation > leading deviation > energy consumption. The penalty is added to the immediate reward r(t) of DDPG in real time, so that the agent can foresee future deviations when outputting the current speed difference and make adjustments in advance.

[0022] Calculate the trajectory curvature between two adjacent points based on the predicted trajectory coordinates. This can be simplified to the arithmetic mean of the plane curvature and the vertical curvature: ,in, For plane curvature, Vertical curvature; Vertical curvature according to the formula Seek; According to the formula Calculate the rate of change of curvature, if If so, then the trajectory between the two adjacent points is marked as a mutation trajectory; Find the rate of change of curvature from the predicted trajectory. The point at which the current risk level threshold is first exceeded is taken as the starting point, denoted as... ,until The point where the value first falls below the threshold is taken as the endpoint, denoted as... The continuous trajectory segment between these two points is the abrupt change segment; The mutation segment starts from the origin Intermediate consecutive points and endpoint Composition, the total length L is the sum of the three-dimensional Euclidean distances between adjacent points: ,in, Let K be the three-dimensional coordinates of the k-th prediction point; For the identified abrupt change segments, adaptive cubic B-splines are used to smooth them out. Based on the mutation segment length L, according to the formula The number of control points is num, and non-uniform interpolation is used to densify the area near the start and end points of mutations, accounting for 40%, while the area in the middle is sparse, accounting for 60%, to ensure a natural transition. The longer the abrupt change segment, the more transition points are needed to achieve uniform smoothness; short abrupt change segments can achieve smoothness with a small number of control points. The ratio of "one control point for every 5 meters" in the formula can ensure that long segments have sufficient transition and that short segments do not excessively increase the amount of calculation.

[0023] like Figure 2 A reinforcement learning-based trajectory tracking system for tracked mining vehicles includes: The data acquisition module uses lidar, millimeter-wave radar, inertial navigation system and vision camera to collect key variables in three aspects: vehicle status, obstacles and road features, and obtain dynamic environment status matrix. The DDPG training module constructs a deep deterministic policy gradient DDPG agent, which takes the dynamic environment state matrix as input and outputs the speed difference between the left and right tracks. The deviation correction module uses an echo state network (ESN) to predict the future trajectory of the tracked vehicle and calculates the three-dimensional advance deviation between the predicted trajectory and the expected trajectory. If the deviation exceeds the safety threshold at a certain moment, an advance penalty term is added to the reward function of the DDPG. The trajectory smoothing module calculates the trajectory curvature between two adjacent points based on the predicted trajectory coordinates, marks the abrupt change segments of the predicted trajectory, and uses adaptive cubic B-splines to smooth and correct the abrupt change segments.

[0024] Some of the data in the above formulas are numerical calculations with dimensions removed, and the contents not described in detail in this specification are all prior art known to those skilled in the art.

[0025] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A reinforcement learning-based method for tracking the trajectory of a tracked mining vehicle, characterized in that, include: By using lidar, millimeter-wave radar, inertial navigation system and vision camera, key variables in three aspects, namely vehicle status, obstacles and road surface features, are collected to obtain dynamic environment state matrix; Construct a deep deterministic policy gradient DDPG agent, which takes the dynamic environment state matrix as input and outputs the speed difference between the left and right tracks; An Echo State Network (ESN) is used to predict the future trajectory of a tracked vehicle. The three-dimensional lead-ahead deviation between the predicted trajectory and the desired trajectory is calculated. If the deviation exceeds a safety threshold at a certain moment, a lead-ahead penalty term is added to the reward function of the DDPG. The specific formula for calculating the three-dimensional lead-ahead deviation between the predicted trajectory and the desired trajectory is as follows: Where i∈[1,20] and is an integer, representing the label of the three-dimensional trajectory point; set the basic threshold. Based on its construction of actual dynamic thresholds ,in, The current speed of the tracked vehicle. This represents the maximum travel speed of the tracked vehicle; if the deviation at any given moment... Then, add a lead penalty term to the reward function of DDPG. : The weights after adding this term satisfy the following conditions: ; The trajectory curvature between two adjacent points is calculated based on the predicted trajectory coordinates. The abrupt change segments of the predicted trajectory are marked, and the abrupt change segments are smoothed and corrected using adaptive cubic B-splines.

2. The reinforcement learning-based method for tracking the trajectory of a tracked mining vehicle according to claim 1, characterized in that, The key variables of the vehicle state include three-dimensional position coordinates, three-dimensional attitude angles, three-dimensional acceleration components, and instantaneous velocity; the key variables of the obstacle include the obstacle's three-dimensional coordinates, size, relative velocity, and reflection intensity; the key variables of the road surface features include slope vector, texture features, smoothness, and illumination intensity.

3. The reinforcement learning-based method for tracking the trajectory of a tracked mining vehicle according to claim 1, characterized in that, The specific steps for training the DDPG agent are as follows: Construct the current and target networks for Actor and Critic in DDPG, and create an experience replay pool; The agent in the current state Below, basic actions are output through the Actor network. The exploration action is obtained by superimposing OU noise. ; Will This is directly mapped to the tracked vehicle's speed and steering angle, resulting in a new state. and rewards and will Store in the playback pool; When the number of replay samples is greater than or equal to Nth, Nth empirical samples are randomly selected, and the target Q-value is calculated using the Critic target network. And calculate the predicted Q value using the current Critic network: ;in, Nth is the discount factor, and Nth is the sample size threshold; According to the formula Calculate the loss function L of the Critic network and update the current Critic and Actor network parameters using the Adam optimizer; Based on soft update coefficient Update the target network parameters. Evaluate the policy once every 1000 iterations. If the average reward fluctuation of 5 consecutive evaluations is ≤5%, then convergence is determined and training is stopped.

4. The reinforcement learning-based method for tracking the trajectory of a tracked mining vehicle according to claim 3, characterized in that, Both Actor and Critic use fully connected neural networks, with the specific parameter settings as follows: The input layer of the Actor is a 24-dimensional state vector, and... Consistent; There are two hidden layers. The first layer has 64 neurons and the second layer has 32 neurons. The activation function for both layers is ReLU, and the weights of the hidden layers are initialized using He normal distribution. The output layer has one neuron, with Tanh as the activation function, and is converted into the difference in basic rotational speed between the left and right tracks through a linear mapping. The specific formula is as follows: ; Where K is the absolute value of the maximum permissible speed difference between the left and right tracks of the tracked vehicle; Select the Adam optimizer, learning rate Set to 0.0001; The input layer of Critic is a 25-dimensional vector, including a 24-dimensional state vector and a 1-dimensional action vector; There are 2 hidden layers. The first layer has 128 neurons, and the second layer has 64 neurons. The activation function for both layers is ReLU. The output layer has one neuron, and the Adam optimizer is selected with a learning rate of [missing information]. Set to 0.

001.

5. The reinforcement learning-based method for tracking the trajectory of a tracked mining vehicle according to claim 3, characterized in that, The The specific formula is: ； Where w1, w2, and w3 are weight coefficients and satisfy the following conditions: This represents the three-dimensional deviation between the vehicle's current position and the desired trajectory reference point, where safe(t) represents the safety bonus, and its specific formula is: ,in, These are the lower and upper limits of the distance, respectively. This represents the minimum three-dimensional distance between the vehicle and the nearest obstacle.

6. The reinforcement learning-based method for tracking the trajectory of a tracked mining vehicle according to claim 3, characterized in that, Will This is directly mapped to the tracked vehicle's travel speed and steering angle: The travel speed of a tracked vehicle is determined by the average speed of the left and right tracks, according to the formula... Calculate the travel speed v, where k is the proportionality coefficient, specifically... , where i is the transmission ratio of the track drive system. Indicates the rotational speed of the left and right tracks; The steering of a tracked vehicle is determined by the speed difference. The decision, specifically the formula, is as follows: Where m is the proportionality coefficient and dis is the track spacing, which is the distance between the center lines of the left and right tracks. Sampling time.

7. The reinforcement learning-based method for tracking the trajectory of a tracked mining vehicle according to claim 1, characterized in that, The specific parameters for building an ESN network include: The total dimensions of the input layer are set to 18, including the control variables output by DDPG. Current environment state matrix The first three steps of three-dimensional trajectory data, the current environment state matrix Includes the vehicle's current three-dimensional coordinates, heading angle, roll angle, speed, distance to the nearest obstacle, and road surface friction coefficient; The reservoir size is set to 120 neurons, the connection probability is 25%, the spectral radius is 0.9, the leakage rate is 0.3, and the activation function is tanh. The output layer has a dimension of 60, and ridge regression is used to train the output weights.

8. The reinforcement learning-based method for tracking the trajectory of a tracked mining vehicle according to claim 1, characterized in that, The specific steps for marking the abrupt change segments of the predicted trajectory are as follows: Calculate the trajectory curvature between two adjacent points based on the predicted trajectory coordinates. It can be expressed as the arithmetic mean of the plane curvature and the vertical curvature: ,in, For plane curvature, Vertical curvature; According to the formula Calculate the rate of change of curvature, if If the trajectory between the two adjacent points is marked as a mutation trajectory, then △kth is the risk level threshold. Find the rate of change of curvature from the predicted trajectory. The point at which the current risk level threshold is first exceeded is taken as the starting point, denoted as... ,until The point where the value first falls below the threshold is taken as the endpoint, denoted as... And mark the continuous trajectory segment between these two points as the abrupt change segment.

9. A reinforcement learning-based track tracking system for mining tracked vehicles, used to execute the reinforcement learning-based track tracking method for mining tracked vehicles according to any one of claims 1-8, characterized in that, include: The data acquisition module uses lidar, millimeter-wave radar, inertial navigation system and vision camera to collect key variables in three aspects: vehicle status, obstacles and road features, and obtain dynamic environment status matrix. The DDPG training module constructs a deep deterministic policy gradient DDPG agent, which takes the dynamic environment state matrix as input and outputs the speed difference between the left and right tracks. The deviation correction module uses an echo state network (ESN) to predict the future trajectory of the tracked vehicle and calculates the three-dimensional advance deviation between the predicted trajectory and the expected trajectory. If the deviation exceeds the safety threshold at a certain moment, an advance penalty term is added to the reward function of the DDPG. The trajectory smoothing module calculates the trajectory curvature between two adjacent points based on the predicted trajectory coordinates, marks the abrupt change segments of the predicted trajectory, and uses adaptive cubic B-splines to smooth and correct the abrupt change segments.

Images