Mine unmanned vehicle steering angle prediction method and related equipment

Abstract

The invention discloses a mine unmanned vehicle steering angle prediction method and related equipment, and aims to solve the technical problem that the prediction precision and stability are insufficient due to the fact that the existing steering angle prediction method is difficult to effectively process time and space information in a continuous scene. Obtaining an RGB image sequence of a to-be-predicted mine unmanned vehicle, and selecting two adjacent frames of images from the RGB image sequence to generate an optical flow image sequence; the RGB image sequence is used for extracting curvature space features of a front road in continuous multi-frame images, and the optical flow image sequence is used for extracting time sequence features of a vehicle driving track. And fusing the road curvature features and the time sequence track features to obtain a predicted steering angle. And adjusting the optimized feature extraction network parameters according to the deviation between the real steering angle and the predicted steering angle to ensure the steering angle prediction effect. According to the method, space-time information is fused and a feedback adjustment mechanism is introduced, so that the defects of an existing method in processing time and space information in a continuous scene are effectively overcome, and the prediction precision and stability are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of unmanned driving technology, and in particular to a steering angle prediction method for an unmanned mine vehicle and related equipment. Background Art

[0002] With the rapid development of intelligent driving technology, the application of unmanned driving systems in specialized environments such as mines is increasing. However, mine scenes are complex and dynamic, and the dynamic driving environment presents challenges such as low light, multiple obstacles, and unstructured roads. Traditional steering angle prediction methods struggle to effectively handle the temporal and spatial information in continuous scenes, resulting in insufficient prediction accuracy and stability.

[0003] Patent publication number CN1188107707A proposes a method for controlling vehicle centering in curves. This method calculates the steering wheel angle using curve curvature and the Ackerman model and adjusts it using a correction coefficient. However, this method relies on a physical model and requires multiple corrections, resulting in poor real-time performance. Patent publication number CN118810780A proposes a vehicle steering control method that directly maps image pixels to steering angles. However, this method fails to consider the temporal continuity of driving behavior and lacks modeling of the dynamic evolution of steering angles, resulting in low prediction accuracy and stability.

[0004] In addition, publication number CN111242044B proposes a dual-channel encoding network based on ConvLSTM for predicting future frames of infrared images for nighttime unmanned vehicles. Its goal is to improve the clarity and continuity of nighttime scene perception and provide a reference for subsequent decision-making. The network consists of two sub-networks: a spatial sub-network (using ResNet multiple residual layers to extract the spatial structural features of the image) and a temporal sub-network (using a convolutional network combined with ConvLSTM to extract the temporal features of the image sequence). The spatiotemporal features are then synthesized through a feature fusion network to generate predicted infrared images. This method focuses on image-level scene generation and prediction, and the output is image frames. It is not directly aimed at steering behavior control. Moreover, it mainly processes infrared images and is suitable for nighttime driving scenarios, but not for environmental perception tasks dominated by RGB images in mines.

[0005] Therefore, in order to solve the steering problem in the existing technology that does not fully consider spatiotemporal feature information such as road information and driving status information, ignores steering continuity and lacks adaptability to complex mine scenes, a steering angle prediction method is needed that can integrate the time series trajectory information of the unmanned vehicle during driving and the curvature information of the mine road in space to improve the control accuracy and stability in complex environments. Summary of the Invention

[0006] The purpose of the present invention is to provide a steering angle prediction method and related equipment for unmanned mine vehicles, so as to overcome the technical problem that the existing steering angle prediction method is difficult to effectively process the time and space information in continuous scenes, resulting in insufficient prediction accuracy and stability.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a method for predicting the steering angle of an unmanned mine vehicle, comprising:

[0009] Obtain the RGB image sequence of the unmanned vehicle in the mine to be predicted;

[0010] Select two adjacent frames in the RGB image sequence, calculate and arrange them using the optical flow estimation algorithm to obtain an optical flow image sequence;

[0011] The RGB image sequence is input into the feature extraction network model to extract the spatial features of the curvature of the road ahead from multiple consecutive frames of RGB images in the predicted mine unmanned vehicle driving scene;

[0012] The optical flow image sequence is input into the feature extraction network model to extract the temporal features of the continuous multi-frame driving trajectory of the unmanned mine vehicle to be predicted;

[0013] By integrating road curvature features and time series trajectory features, the steering angle of the mine unmanned vehicle is predicted to obtain the predicted steering angle;

[0014] According to the deviation between the actual steering angle and the predicted steering angle, the feature extraction network parameters are adjusted to reduce the loss function value so that the output steering angle prediction value reaches the set effect.

[0015] Obtain the RGB image sequence of the unmanned mine vehicle to be predicted, including:

[0016] Continuously shoot the mine unmanned vehicle operation scene at fixed time intervals to obtain RGB image sequences;

[0017] The collected images are stored in chronological order to ensure that the timestamp and order of each frame are consistent to form a complete image sequence;

[0018] Denoise, rectify, and format image sequences.

[0019] Select two adjacent frames in the RGB image sequence, calculate and arrange them using the optical flow estimation algorithm, and obtain an optical flow image sequence, including:

[0020] From the acquired RGB image sequence, two adjacent frames are selected, and the optical flow estimation algorithm is used to calculate the movement direction and speed of the image pixels, generate an optical flow field, and extract the trajectory information of the unmanned vehicle to be predicted between frames;

[0021] Arrange the calculated optical flow data in chronological order to form an optical flow time series, ensuring that it is aligned with the original RGB image sequence;

[0022] The optical flow image time series is stored in a fixed format and normalized.

[0023] The RGB image sequence is input into the feature extraction network model to extract the spatial features of the road curvature in front of the continuous multi-frame RGB image in the predicted mine unmanned vehicle driving scene, including:

[0024] Input a continuous RGB image sequence to the feature extraction network model, normalize the image, and crop out areas unrelated to the road in the mine scene;

[0025] The normalized and cropped image is input into the convolution layer of the two-stream convolutional network to extract the road curvature spatial features and obtain the road curvature feature output after the convolution layer;

[0026] The road curvature feature output after the convolutional layer is input into the gated recurrent unit to obtain the road curvature spatial feature output with historical state information.

[0027] The optical flow image sequence is input into the feature extraction network model to extract the temporal features of the continuous multi-frame driving trajectory of the unmanned mine vehicle to be predicted, including:

[0028] Input continuous optical flow images to the feature extraction network model. The optical flow images contain speed motion information in both horizontal and vertical directions.

[0029] Input the optical flow image to the convolution layer of the two-stream convolutional network to extract the optical flow temporal trajectory features, and obtain the optical flow temporal trajectory feature output of the unmanned vehicle to be predicted after passing through the convolution layer;

[0030] The optical flow temporal trajectory feature output of the unmanned vehicle to be predicted after the convolution layer is input into the gated recurrent unit to obtain the optical flow image temporal trajectory feature output with historical temporal state information.

[0031] By integrating road curvature features and time series trajectory features, the steering angle of the mine unmanned vehicle is predicted. The predicted steering angle is obtained, including:

[0032] Combine the road curvature spatial features and temporal trajectory features to form a fused feature representation;

[0033] The fused features are input into the fully connected layer of the two-stream convolutional network to further map the high-dimensional features;

[0034] The expected steering angle of the mine unmanned vehicle is predicted by the activation function exponential linear unit ReLU.

[0035] According to the deviation between the actual steering angle and the predicted steering angle, the feature extraction network parameters are adjusted to reduce the loss function value so that the output steering angle prediction value reaches the set effect, including:

[0036] Select the root mean square error loss function to calculate the error between the actual steering angle and the predicted steering angle;

[0037] Based on the gradient of the loss function, the backpropagation algorithm is used to calculate the gradient of the parameters of each layer of the two-stream convolutional neural network and the gated recurrent unit, and the optimization algorithm is used to adjust the weights and biases;

[0038] Iterative training reduces the loss function value and ensures the model's ability to fit the data;

[0039] After the loss function converges to the minimum value, the weight parameters of the model are optimized and the steering angle prediction value with the minimum error is output.

[0040] In a second aspect, the present invention provides a steering angle prediction system for an unmanned mine vehicle, comprising:

[0041] A data acquisition module is used to obtain an RGB image sequence of the unmanned vehicle in the mine to be predicted;

[0042] The optical flow processing module is used to select two adjacent frames of images in the RGB image sequence, calculate and arrange them using the optical flow estimation algorithm, and obtain an optical flow image sequence;

[0043] The road feature extraction module is used to input the RGB image sequence into the feature extraction network model to extract the spatial features of the curvature of the road ahead from multiple consecutive frames of RGB images in the predicted mine unmanned vehicle driving scene;

[0044] The trajectory feature extraction module is used to input the optical flow image sequence into the feature extraction network model to extract the temporal features of the continuous multi-frame driving trajectory of the unmanned mine vehicle to be predicted;

[0045] The feature fusion prediction module is used to fuse road curvature features and time series trajectory features to predict the steering angle of the mine unmanned vehicle and obtain the predicted steering angle;

[0046] The error convergence module is used to adjust the feature extraction network parameters according to the deviation between the actual steering angle and the predicted steering angle, reduce the loss function value, and make the output steering angle prediction value reach the set effect.

[0047] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method for predicting the steering angle of an unmanned mine vehicle as described above are implemented.

[0048] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for predicting the steering angle of an unmanned mine vehicle.

[0049] Compared with the prior art, the present invention has the following beneficial technical effects:

[0050] In the first aspect, the present invention provides a method for predicting the steering angle of an unmanned mine vehicle. By fusing temporal and spatial information and combining it with a feedback adjustment mechanism, the present invention effectively solves the shortcomings of existing steering angle prediction methods in continuous scenarios. The spatial features of the road curvature are extracted using an RGB image sequence, and an optical flow image sequence is generated through an optical flow estimation algorithm to capture the temporal features of the vehicle's driving trajectory. These two feature sequences are respectively input into the feature extraction network model to achieve independent extraction of spatial features and temporal features, avoiding mutual interference between features. On this basis, the two features are fused to provide comprehensive spatiotemporal information support for steering angle prediction. In addition, the present invention also introduces a feedback adjustment mechanism to adjust the network parameters according to the deviation between the actual steering angle and the predicted steering angle, reduce the loss function value, and thus gradually optimize the prediction effect and improve the prediction accuracy and stability. This scheme design that integrates spatiotemporal information and feedback optimization enables it to better adapt to the steering angle prediction needs of unmanned mine vehicles in continuous scenarios.

[0051] In a second aspect, the present invention provides a steering angle prediction system for unmanned mine vehicles. Through a systematic design, it effectively integrates multiple functional modules to achieve collaborative processing of temporal and spatial information, thereby overcoming the shortcomings of existing steering angle prediction methods. In this system, the data acquisition module and the optical flow processing module work together to provide RGB image sequences and optical flow image sequences for subsequent processing, respectively, ensuring the synchronous acquisition of temporal and spatial information. The road feature extraction module and the trajectory feature extraction module extract spatial features of road curvature and temporal features of driving trajectories from the RGB images and optical flow images, respectively. Through independent processing, these features avoid mutual interference and ensure accurate feature extraction. The feature fusion prediction module deeply fuses these two features, providing comprehensive spatiotemporal information support for steering angle prediction, resulting in more accurate prediction results. The error convergence module dynamically adjusts the feature extraction network parameters based on the deviation between the actual and predicted steering angles, reducing the loss function value, continuously optimizing the prediction results, and ensuring prediction stability. Overall, the system's modular architecture and collaborative working mechanism enable it to efficiently process temporal and spatial information in continuous scenes, significantly improving the accuracy and stability of steering angle prediction for unmanned mine vehicles.

[0052] In a third aspect, the present invention provides a computer device that can efficiently implement the steps of the method of the present invention by executing a specific computer program through a processor. When performing data processing tasks, the computer device can accurately perform numerical calculations and logical judgments, avoiding errors caused by human factors; at the same time, since the computer program has a high degree of stability and reliability, the accuracy and consistency of the data processing results can be ensured.

[0053] In a fourth aspect, the present invention provides a computer-readable storage medium. By programming the steps of the method of the present invention into a computer program and storing it on a computer-readable storage medium, users can easily load these programs onto any compatible computer device and execute them without rewriting or converting the code, thereby greatly improving the convenience and flexibility of program execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a schematic diagram of a method for predicting the steering angle of an unmanned mine vehicle in a specific embodiment of the present invention;

[0055] Figure 2 This is a flowchart of a method for predicting the steering angle of an unmanned mine vehicle in a specific embodiment of the present invention;

[0056] Figure 3 It is a block diagram of the convolutional layer structure of a dual-stream convolutional network in a specific embodiment of the present invention;

[0057] Figure 4 2 is a diagram of the internal structure of a gated recurrent unit (GRU) in a specific embodiment of the present invention.

[0058] Figure 5 It is a schematic diagram of a steering angle prediction system for an unmanned mine vehicle in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0059] In the field of unmanned driving technology, for applications in special environments such as mines, the mine environment is complex and changeable, with challenges such as low light, multiple obstacles and unstructured roads. Traditional steering angle prediction methods have difficulty effectively processing the temporal and spatial information in continuous scenes, resulting in insufficient prediction accuracy and stability. For example, some existing methods either require multiple corrections to the steering wheel angle, resulting in poor real-time performance, or fail to consider the time series problem of the steering process, resulting in the steering wheel being unable to accurately produce the desired angle.

[0060] With the development of intelligent driving technology, unmanned driving systems are increasingly used in special environments such as mines. Improving the accuracy and stability of steering angle prediction of unmanned vehicles in mines is crucial to ensuring the safe and efficient operation of unmanned vehicles in mines. It can reduce accident risks, improve transportation efficiency, etc., thereby promoting the intelligentization of mines and meeting related operational needs.

[0061] Patent publication number CN111242044B proposes a dual-channel encoding network based on ConvLSTM for predicting future frames of infrared images for nighttime unmanned vehicles. Its goal is to improve the clarity and continuity of nighttime scene perception and provide a reference for subsequent decision-making. The network consists of two subnetworks: a spatial subnetwork (using ResNet multiple residual layers to extract the spatial structural features of the image) and a temporal subnetwork (using a convolutional network combined with ConvLSTM to extract the temporal features of the image sequence). The spatiotemporal features are then synthesized through a feature fusion network to generate a predicted infrared image. This method focuses on image-level scene generation and prediction, outputting image frames and not directly targeting steering behavior control. Furthermore, it primarily processes infrared images, making it suitable for nighttime driving scenarios but not for environmental perception tasks dominated by RGB images in mines.

[0062] The present invention constructs a dual-stream convolutional network (dual-stream CNN) model for predicting the steering angle of unmanned vehicles in mines. Among them, one branch of the dual-stream CNN network and the gated recurrent unit (GRU) are used to extract the spatial features of road curvature from continuous multi-frame RGB images; similarly, the other branch of the dual-stream CNN network and the GRU are used to extract the speed, direction and other temporal dynamic features of the unmanned vehicle's driving trajectory in continuous multi-frame optical flow images. Finally, the two features are fused to predict the continuous steering angle through the fully connected layer and activation function. Compared with the image generation prediction method, this model directly outputs the steering angle regression value. The design is closer to the control decision-making requirements, adapts to the multi-modal information fusion of the complex environment of the mine, and ensures the accuracy and real-time performance of the prediction.

[0063] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0064] Reference Figure 1 As shown, a specific embodiment of the mine unmanned vehicle steering angle prediction method and related equipment provided by the present invention includes:

[0065] S1, obtain the RGB image sequence of the unmanned vehicle in the mine to be predicted;

[0066] S2, select two adjacent frames in the RGB image sequence, calculate and arrange them using the optical flow estimation algorithm to obtain an optical flow image sequence;

[0067] S3, inputting the RGB image sequence into the feature extraction network model to extract the spatial features of the curvature of the road ahead from the continuous multi-frame RGB images in the predicted mine unmanned vehicle driving scene;

[0068] S4, input the optical flow image sequence into the feature extraction network model, and extract the continuous multi-frame driving track time sequence features of the mine unmanned vehicle in the driving process to be predicted;

[0069] S5, fuse the road curvature features and the time sequence track features, and perform steering angle prediction on the mine unmanned vehicle to obtain a predicted steering angle;

[0070] S6, according to the deviation between the real steering angle and the predicted steering angle, adjust the feature extraction network parameters, reduce the loss function value, and make the output steering angle prediction value reach the set effect.

[0071] The application adopts a double-flow convolution network to extract feature information, one branch of which extracts road curvature spatial features in the front mine environment from an RGB image, and the other branch learns time sequence features of the driving track of the unmanned vehicle from an optical flow image. The optical flow image reflects the motion direction and speed of the mine picture pixels, and through the relative motion relationship, the motion time sequence track of the unmanned vehicle to be predicted can be obtained. This method enables the network to pay attention to the static information of the mine environment and the dynamic driving information of the unmanned vehicle at the same time during the learning process, thereby improving the feature expression ability and the accuracy of the steering angle prediction. Meanwhile, a gated recurrent unit (GRU) is introduced to model the features with short-time dependence relationship, so that the spatial road curvature information and the time sequence track information that can be used not only depend on the current frame RGB image and the optical flow image, but also can combine the continuous multi-frame historical RGB image and the optical flow image, so as to better extract and analyze the spatial road curvature features and the time sequence track features, and improve the understanding ability of the mine unmanned vehicle steering.

[0072] Preferably, S1, the RGB image sequence of the mine unmanned vehicle to be predicted is obtained, specifically including:

[0073] S101, continuously photographing the mine unmanned vehicle running scene at a fixed time interval using a camera to obtain an RGB image sequence I with a total frame number N, which is used to extract the curvature and other spatial features of the road in front of the mine;

[0074] S102, storing the collected images in time sequence to ensure that the time stamp and sequence of each frame of image are consistent, and forming a complete image sequence;

[0075] S103, pre-processing the image sequence, such as denoising, correction and format unification.

[0076] The continuous multi-frame RGB image sequence of the mine scene obtained in this specific embodiment mainly includes the road curvature at key locations in the mine environment space, such as turns, obstacles, and complex sections such as narrow passages. In addition, in order to improve the efficiency of the feature extraction model and utilize the road curvature feature information of the mine space environment in the rich image sequence, the image sequence input to the spatial feature extraction network only includes three consecutive RGB images at time t (I t-2 ,I t-1 ,I t ).

[0077] Preferably, S2, selecting two adjacent frames of images in the RGB image sequence, calculating and arranging them using an optical flow estimation algorithm to obtain an optical flow image sequence, specifically includes:

[0078] S201, from the acquired RGB image sequence I with a total number of frames N, select two adjacent frames I t and I t+1 , use the optical flow estimation algorithm to calculate the movement direction and speed of the image pixels, generate the optical flow field, and thus extract the trajectory information of the unmanned vehicle to be predicted between frames;

[0079] S202 , arrange the calculated optical flow data in chronological order to form an optical flow time series F with a total number of frames N-1, ensuring that it is aligned with the original RGB image sequence to preserve the temporal consistency of the unmanned vehicle motion information to be predicted.

[0080] S203: Store the optical flow image time series F in a fixed format and perform normalization processing.

[0081] In this specific embodiment, the method for extracting the continuous multi-frame optical flow time series F in the mine environment uses the Farneback algorithm for calculating dense optical flow in opencv to estimate the optical flow, and the calculated pixel movement direction and speed are used to obtain the driving trajectory information of the unmanned vehicle. The algorithm uses the dual-channel vector matrix The speed and direction of the optical flow are calculated in , and then mapped to a standardized HSV color space to visualize the calculated direction and speed. In addition, in order to improve the efficiency of the temporal feature extraction model and utilize the rich temporal sequence motion information, the optical flow image sequence input to the temporal feature extraction network only includes the two adjacent frames of optical flow images at time t (F t-1 ,F t ).

[0082] Preferably, S3, the RGB image sequence is input into the feature extraction network model, and the front road curvature spatial features of the continuous multiple frames of RGB images in the predicted mine unmanned vehicle driving scene are extracted. One branch in the dual-flow convolutional network and the gated recurrent unit (GRU) network are used as the extraction model of the road curvature spatial features in the mine environment. This step specifically includes:

[0083] S301, input 3 continuous frames of RGB images (I t-2 ,I t-1 ,I t ) with a size of 66x220x3, and perform normalization operation on the images, change the pixel value range of the original RGB image from [0, 255] to [0, 1], and crop the area unrelated to the road in the mine scene;

[0084] S302, the input image obtained in the foregoing step is subjected to road curvature spatial feature extraction through the convolutional layer of the dual-flow convolutional network. The convolutional layer is composed of 5 layers. The channel numbers of the first 3 layers of the convolutional network are 24, 36, and 48 respectively, the convolution kernel size is 5x5, the stride is 2, and the exponential linear unit (ELU) is used as the activation function. The channel numbers of the last 2 layers of the convolutional network are both 64, the convolution kernel size is 3x3, the stride is 1, and the activation function also uses the exponential linear unit (ELU). The road curvature feature output X I1 obtained through the convolutional layer is obtained.

[0085] S303, the output X I1 obtained in the foregoing step is subjected to the gated recurrent unit (GRU). The GRU is mainly composed of an update gate and a reset gate. The update gate determines how much information needs to be inherited from the previous time t h t-1 . That is, the role of the update gate is to set a weight value to measure the proportion of the current output depending on the current input and the past input. The reset gate controls the degree of forgetting of the previous time hidden state h t-1 . That is, a parameter value is set to control whether to retain or forget the historical information. In this way, through the reset gate, the current candidate time state representation is generated by combining the current input and the past information. Through the update gate, the final road curvature spatial feature output X I with historical state information is obtained between the last time hidden state h t-1 and the current candidate time state h t ;

[0086] Among them, in this specific implementation, a dual-stream convolutional network is combined with a gated recurrent unit (GRU) to extract the spatial features of the road curvature in front of the mine unmanned vehicle during its driving process, and a dual-stream convolutional network is used to extract the spatial information of the road curvature of continuous frame images, and combined with GRU for time series modeling. By setting weight parameters to determine the proportion of the current candidate state and the historical state, the model can combine the road curvature characteristics of the current mine spatial environment and the historical road curvature characteristics to improve the feature expression ability.

[0087] Preferably, S4, inputting the optical flow image sequence into the feature extraction network model, extracting the temporal features of the continuous multi-frame driving trajectory of the unmanned mine vehicle to be predicted during driving, and using another branch of the two-stream convolutional network + the gated recurrent unit (GRU) network as the temporal trajectory feature extraction model in the optical flow image of the unmanned mine vehicle. This step specifically includes:

[0088] S401: Input two consecutive optical flow images of the same frame, with a size of 66×220×3. The optical flow images contain speed motion information in both horizontal (u) and vertical (v) directions.

[0089] S402: The input obtained in step S41 is passed through the convolution layer of the two-stream convolutional network to extract the optical flow temporal trajectory features. The convolution layer consists of 5 layers. The number of channels of the first 3 convolutional networks are 24, 36, and 48 respectively, the convolution kernel size is 5×5, and the stride is 2. The number of channels of the last 2 convolutional networks are both 64, the convolution kernel size is 3×3, and the stride is 1. The activation function of the 5-layer network is the exponential linear unit (ELU). The optical flow temporal trajectory feature output X of the unmanned vehicle to be predicted after the convolution layer is obtained. F1 .

[0090] S403, similarly, the output X obtained in step S42 is converted to F1 , after the gated recurrent unit (GRU), the optical flow image temporal trajectory feature output X with historical temporal state information is obtained F .

[0091] In this specific implementation, a dual-stream convolutional network combined with a gated recurrent unit (GRU) is used to extract the optical flow temporal trajectory features of the unmanned vehicle to be predicted in a mine environment. The dual-stream convolutional network is used to extract motion information from the optical flow image and combined with the GRU for temporal modeling, enabling the model to effectively capture the dynamic changes in the optical flow temporal trajectory of the unmanned vehicle. The attention mechanism determines the weight of the motion trajectory information of the current state and the historical state, improving the temporal expression capability of the trajectory motion features.

[0092] Specifically, S5 integrates road curvature features and time series trajectory features to predict the steering angle of the mine unmanned vehicle. The predicted steering angle is obtained, including:

[0093] S501, the road curvature spatial feature X obtained in the above steps is I And the time series trajectory feature X obtained in the previous step I The images are spliced ​​together to form a fusion feature representation. Its spatial features come from the RGB image and include the environmental space road curvature information. The temporal features come from the optical flow image and include the unmanned vehicle trajectory motion change information.

[0094] In step S502, the fused features are input into the fully connected layer (FC layer) of the two-stream convolutional network to further map the high-dimensional features and enhance the feature expression capability. The fully connected layer performs weighted processing on the spatial features of road curvature and the temporal trajectory features, enabling the model to learn and fuse the road curvature information and temporal trajectory information.

[0095] S503, the expected steering angle P of the mine unmanned vehicle is predicted by the activation function exponential linear unit ReLU t ,The activation function ReLU can enhance the nonlinear expression ability of the model, so that the steering angle prediction value can more effectively adapt to different driving scenarios and steering behaviors, ensuring the strong generalization ability of the model.

[0096] Among them, in this specific embodiment, in the feature fusion stage, the spatial features of road curvature extracted from the RGB image and the temporal trajectory features of the unmanned vehicle extracted from the optical flow image are first spliced ​​to construct a complete spatiotemporal fusion feature representation. On this basis, a fully connected layer (FC layer) is used to map the high-dimensional features, and the model's learning ability for key features is improved through weighted processing to ensure the effective fusion of spatial information and temporal information. Subsequently, the ReLU activation function is introduced to enhance the nonlinear expression ability of the model, so that the fused features can more clearly reflect different driving scenarios and steering behaviors, thereby improving the model's ability to distinguish. The processed high-dimensional features are input into the output layer, and regression is used to predict the steering wheel angle of the mine unmanned vehicle.

[0097] Preferably, S6, adjusting the feature extraction network parameters according to the deviation between the actual steering angle and the predicted steering angle, reducing the loss function value, and making the output steering angle prediction value achieve the set effect, includes:

[0098] S601, the deviation between the predicted value and the true value is an important criterion for measuring the quality of the model, and we need to be vigilant against large deviations. Therefore, the root mean square error loss function L is selected to calculate the true steering angle Y t and the predicted steering angle P t The error between

[0099] S602, based on the gradient of the loss function The backpropagation algorithm is used to calculate the gradients of the parameters of each layer of the two-stream convolutional neural network and the gated recurrent unit, and the optimization algorithm is used to adjust the weight w and bias b;

[0100] S603, continuously iterating the training to gradually reduce the loss function L value, ensuring that the model can fit the training data more accurately;

[0101] S604: When the loss function converges to the minimum value, the weight parameters of the model are optimized, and the optimal steering angle prediction value can be output eventually. Minimize the error.

[0102] Among them, in this specific embodiment, during the model training stage, a loss function is used to calculate the error between the predicted value and the true value, and the model parameters are continuously adjusted through the optimization algorithm. The root mean square error (RMSE) is selected as the loss function to measure the deviation between the steering angle output by the model and the true steering angle. The back propagation algorithm is used to calculate the gradient of the loss function to the parameters of each layer of the neural network, and the gradient descent optimization algorithm is used to update the weights and biases to minimize the loss function value. During the training process, the model parameters are continuously adjusted so that the loss function value gradually decreases, thereby improving the fitting ability of the training data. When the loss function converges to the minimum value, the weight parameters of the model reach the optimal state, minimizing the prediction error and improving the steering angle accuracy of the mine unmanned vehicle.

[0103] In order to make the mine unmanned vehicle steering angle prediction method provided by the present invention easier to understand, a more detailed specific implementation method is provided below for illustration. Figure 2 Shown is a flowchart of this specific implementation method.

[0104] Obtaining the RGB image sequence I of the area ahead of the unmanned mine vehicle to be predicted includes the following steps:

[0105] Collect a mine scene RGB image sequence I with a total frame number N, and use it to extract road curvature information in the mine space environment;

[0106] The collected image sequences are stored in chronological order to form a complete image sequence;

[0107] Perform preprocessing on image sequences, such as denoising, rectification, and format unification.

[0108] Next, we obtain the optical flow image sequence I of the mine's unmanned vehicle during its driving process. The optical flow image reflects the movement direction and speed of the image pixels. Through the relative motion relationship, we can extract the time series trajectory information of the unmanned vehicle during its driving process.

[0109] From the acquired RGB image sequence I, select two adjacent frames of image I t and I t+1, using the Farneback algorithm for calculating dense optical flow in OpenCV to estimate the optical flow, calculate the movement direction and speed of the pixels in the optical flow image, generate the optical flow field, and then obtain the motion information of the unmanned vehicle's trajectory to be predicted between frames based on the relative motion relationship. The optical flow calculation process is as follows:

[0110] For each pixel (x, y), let its grayscale value be P(x, y), assuming that its change can be approximated by a second-order polynomial in a small neighborhood:

[0111]

[0112] Among them, u, v are optical flow vectors, that is, the displacement of pixels; P x , P y is the gradient; P xx , p xy , p yy is the second-order derivative.

[0113] The Farneback method uses a neighborhood window to estimate pixel motion instead of a single pixel. For the pixels within the window, a structure tensor is constructed to represent the second-order model:

[0114]

[0115] Where w(x,y) is the weight within the window (usually Gaussian distribution);

[0116] Solve the optical flow v=(u,v) by the least squares method t :

[0117]

[0118] Among them, P t is the temporal gradient, i.e. the pixel change between frames P(x,y,t+1)-P(x,y,t);

[0119] The motion vector is obtained by solving the linear equation group Mv=b.

[0120] The optical flow field is a two-channel matrix that stores the displacement vector (u, v) of each pixel and can calculate the direction and speed of pixel movement:

[0121] Pixel motion speed:

[0122] Pixel motion direction:

[0123] This information can be used to visualize optical flow, using HSV colors to encode motion in different directions and speeds.

[0124] The calculated optical flow image data is arranged in time sequence to ensure that it is aligned with the original RGB image sequence to retain the temporal consistency of the motion information. The optical flow image sequence F is then stored in a fixed format.

[0125] The image sequence is input into a network model composed of a branch of a two-stream convolutional network and a gated recurrent unit (GRU) to obtain the road curvature spatial feature X of the RGB image in the predicted mine unmanned vehicle driving scene. I ;

[0126] Then the original RGB image is normalized, that is, the pixel value range is changed from [0, 255] to [0, 1], and the areas not related to the road in the mine environment are cropped. The normalization calculation formula is as follows:

[0127]

[0128] Where P is the pixel value of the original RGB image (0, 255); P ′ After normalization, the pixel value range becomes (0, 1).

[0129] The image (I t-2 ,I t-1 ,I t ) The road curvature features in the mine environment are extracted through the convolution layer of the two-stream convolutional network. The convolution layer consists of 5 layers, and its network structure is shown in the figure below. Figure 3 The convolution calculation process is:

[0130] For the first three layers of convolution, for layer i (i=1, 2, 3), the convolution is calculated as follows:

[0131] X (i) =ELU(W (i) *X (i-1) +b (i) )

[0132] in,

[0133] X (0) =I is the input image;

[0134] W (i) is the convolution kernel of layer i, size 5×5;

[0135] b (i) is the bias term;

[0136] * represents convolution operation, stride s = 2;

[0137] ELU (Exponential Linear Unit) activation function:

[0138]

[0139] After the last two layers of convolution, for the i-th layer (i=5,5), the convolution is calculated as follows:

[0140] X (i) =ELU(W (i) *X (i-1) +b (i) )

[0141] Among them, W (i) The convolution kernel size is 5×5, stride s=1; the output result is X (5) =X I1 .

[0142] Then the output X I1 After the gated recurrent unit (GRU), the reset gate of GRU controls the degree of forgetting of historical information, enabling GRU to flexibly handle the dependency of memory time and capture the motion information between frames. and the current candidate moment state h t The trade-off is made between the two, and finally the road curvature feature output X in the mine space environment with historical state information is obtained. I The internal structure of GRU is as follows: Figure 4 As shown, the operation process is as follows:

[0143] r t =σ(W xr x t +b xr +W hr h t-1 +b hr )

[0144] z t =σ(W xr x t +b xz +W hz h t-1 +b hz )

[0145]

[0146] Among them, x t is the current input; h t-1 is the hidden state at the previous moment; h t is the current state; r t ∈[0,1] represents the reset gate; z t ∈[0,1] represents the update gate; is a candidate state; w and b are both learnable network parameters;

[0147] ⊙ represents the Hadamard product, which is the multiplication of corresponding elements in the matrix;

[0148] σ and tanh represent activation functions,

[0149] Then, the optical flow image sequence is input into the network model composed of another branch of the two-stream convolutional network and the gated recurrent unit (GRU) to obtain the temporal trajectory feature X of the mine unmanned vehicle driving process to be predicted in the mine environment. F .

[0150] Input two consecutive frames of optical flow images containing motion information in both horizontal (u) and vertical (v) directions, with a size of 66×220×3.

[0151] The optical flow image (F t-1 ,F t ) The time series trajectory feature is extracted through the convolution layer of the two-stream convolutional network. The convolution layer consists of 5 layers, and its network structure is shown in the figure below. Figure 3 The convolution calculation process is the same as the process of extracting road curvature spatial features as described above. The output result of the convolution layer extracting temporal trajectory features is X F1 .

[0152] Afterwards, the result X will be output F1 After the gated recurrent unit (GRU), the time feature output X with the historical optical flow temporal trajectory state information is obtained F The internal structure of GRU is as follows: Figure 4 As shown, the operation process is the same as the process of extracting road curvature features with historical spatial information as described above.

[0153] The obtained road curvature feature X I and the time series trajectory characteristics of the unmanned mine vehicle to be predicted X F Perform vector splicing to obtain fusion feature output, and then predict the steering angle of the mine unmanned vehicle through the fully connected layer of the convolutional neural network to obtain the predicted steering angle P t .

[0154] In this process, the road curvature feature X in the mine space environment obtained by the two-stream convolutional network + gated recurrent unit is firstly I and the unmanned vehicle's time series trajectory feature X F Perform vector splicing, and the calculation formula is as follows:

[0155]

[0156] Among them, X fusion is the fused feature vector.

[0157] The fused features are input into the fully connected layer (FC layer), and the road curvature feature information and the time series trajectory feature information are weighted so that the model can learn and fuse the road curvature and time series trajectory features. The calculation formula is as follows:

[0158] Linear transformation: Z = WX fusion +b

[0159] Among them, X fusion is the fused feature vector; W is the weight matrix of the fully connected layer; b is the bias term; and Z is the output of the fully connected layer. W and b are learnable parameters in the network.

[0160] The output Z of the fully connected layer is passed through the activation function ReLU to predict the expected steering angle P, which is calculated as follows:

[0161] Nonlinear activation function: P = φ(Z) = φ(WX fusion +b)

[0162] Where P is the predicted steering angle; φ(·) is the activation function, and ELU (exponential linear unit) is selected as the activation function. The calculation formula is:

[0163] Then according to the actual steering angle Y t and the predicted steering angle P t The deviation of the network model is continuously adjusted to minimize the loss function, and finally the output steering angle prediction value P is t to achieve the best results.

[0164] In this process, the root mean square error (RMSE) needs to be selected as the loss function L of the network model to calculate the error between the actual steering angle Yt and the predicted steering angle Pt. The calculation formula is:

[0165]

[0166] Where N represents the total number of sample frames; P t Steering angle predicted by the model; Y t True steering angle; L total loss function.

[0167] Based on the gradient of the loss function The back propagation algorithm is used to calculate the gradient of the parameters of each layer of the neural network, and the optimization algorithm is used to adjust the weight W and bias b; the calculation process is as follows:

[0168] First calculate the gradient of RMSE loss with respect to the prediction:

[0169]

[0170] Eventually Substituting into the above formula, it can be simplified to

[0171] Next, backpropagation is used to calculate the gradient of the fully connected layer:

[0172] The forward propagation fully connected layer + activation function is: Z = WX fusion +b,P t =φ(Z)

[0173] Use the chain rule to calculate the gradient of the fully connected layer weight W and bias b:

[0174]

[0175] Derivative with respect to weight W:

[0176]

[0177] Derivative of bias b:

[0178]

[0179] Finally, the gradient descent optimization algorithm is used to update the parameters:

[0180]

[0181] Wherein, η is the learning rate of the optimization algorithm, and in the present invention, η=0.0001.

[0182] Using the gradient of the loss function to the predicted value Based on directional propagation, the chain rule can be used to calculate the parameters of the fully connected layer, and then the parameters can be updated using gradient descent. Similarly, this method can be used to calculate the parameters of the gated recurrent unit and the convolutional layer, and then gradient descent can be used to update the parameters of these layers.

[0183] By continuously updating the parameters W and b of each layer of the network and iteratively training the network model, the loss function L value is gradually reduced, ensuring that the model can fit the training data more accurately.

[0184] After the loss function L converges to the minimum value, the weight parameters of the model are optimized, and the optimal steering angle prediction value can be output eventually. Minimize the error.

[0185] It should be noted that the data set of the network model provided in this specific embodiment can be obtained by the mine vehicle driver manually driving the vehicle in the mine, and the camera takes pictures of the road in front of the vehicle in the mine at a frequency of 20Hz. At the same time, the steering wheel angle sensor records the steering wheel angle at the same frequency, and the steering wheel angle is calculated in radians (rad). The mine sections photographed are preferably straight sections under the mine, turning sections under the mine, obstacles under the mine, and narrow sections under the mine, to ensure that the data set covers most scenes in the mine driving environment. The unclear pictures and pictures that do not contain mine roads are removed from the captured images, and finally a data set of 20,000 pictures is obtained. The data set is randomly rearranged, and the ratio of the training set to the test set is 8:2, that is, 16,000 training sets and 4,000 test sets. The 16,000 training images consist of RGB images containing spatial information. Using the Farneback algorithm in OpenCV for computing dense optical flow, an optical flow image is generated from two adjacent RGB frames. This results in 15,999 optical flow images containing temporal trajectory information. These 16,000 RGB images contain information about road curvature in the mine's spatial environment, as well as 15,999 optical flow images containing predicted unmanned vehicle temporal trajectory information, along with their corresponding true steering wheel angles. The RGB images and optical flow images are fed into a two-stream convolutional network + gated recurrent unit model, and trained to obtain predicted steering wheel angles. The root mean square error between the true and predicted steering wheel angles is used as the model's loss function, and the network model parameters are iteratively trained through backpropagation. Finally, the loss function between the steering wheel angle predicted by the continuously optimized network model and the true steering wheel angle is minimized. This optimal model is then optimized, and the effectiveness of this steering angle prediction method for unmanned mine vehicles can be verified using a test set.

[0186] The steering angle prediction method for unmanned mine vehicles provided by the present invention combines road curvature with vehicle time series trajectory information, and extracts spatial features and temporal features respectively through a dual-stream convolutional network to avoid feature interference and improve the pertinence and efficiency of feature extraction. At the same time, a gated recurrent unit (GRU) is introduced to model time series features, and combined with historical information, the model can simultaneously focus on the static information of the mine environment and the dynamic driving information of the unmanned vehicle, thereby enhancing the feature expression capability. Furthermore, the fused high-dimensional features are nonlinearly mapped through feature splicing and fully connected layers to construct a complete spatiotemporal feature representation, thereby more accurately predicting the steering angle. With the help of a data-driven optimization mechanism, real data is used for training, and the network parameters are continuously adjusted so that the model gradually fits the real steering angle data, thereby improving the prediction accuracy and stability. At the same time, a customized design is carried out for the complexity and dynamics of the mine environment, so that it can better adapt to special scenarios and ensure prediction performance under complex conditions such as low light, multiple obstacles and unstructured roads. In summary, the steering angle prediction method for unmanned mine vehicles provided by the present invention effectively solves the problems of insufficient prediction accuracy and stability of existing steering angle prediction methods in continuous scenarios by integrating spatiotemporal information, adopting advanced network architecture and customized design.

[0187] Reference Figure 5 As shown, a specific embodiment of the present invention further provides a mine unmanned vehicle steering angle prediction system, comprising:

[0188] A data acquisition module is used to obtain the RGB image sequence of the unmanned vehicle in the mine to be predicted;

[0189] The optical flow processing module is used to select two adjacent frames of images in the RGB image sequence, calculate and arrange them using the optical flow estimation algorithm, and obtain an optical flow image sequence;

[0190] The road feature extraction module is used to input the RGB image sequence into the feature extraction network model to extract the spatial features of the curvature of the road ahead from multiple consecutive frames of RGB images in the predicted mine unmanned vehicle driving scene;

[0191] The trajectory feature extraction module is used to input the optical flow image sequence into the feature extraction network model to extract the temporal features of the continuous multi-frame driving trajectory of the unmanned mine vehicle to be predicted;

[0192] The feature fusion prediction module is used to fuse road curvature features and time series trajectory features to predict the steering angle of the mine unmanned vehicle and obtain the predicted steering angle;

[0193] The error convergence module is used to adjust the feature extraction network parameters according to the deviation between the actual steering angle and the predicted steering angle, reduce the loss function value, and make the output steering angle prediction value reach the set effect.

[0194] A computer device is also provided in a specific embodiment of the present invention. Specifically, the computer device includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGAs), or other processors. GateArray, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, and are suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used to obtain an RGB image sequence of the unmanned mine vehicle to be predicted; select two adjacent frames of images in the RGB image sequence, calculate and arrange them using an optical flow estimation algorithm to obtain an optical flow image sequence; input the RGB image sequence into a feature extraction network model to extract the spatial features of the curvature of the road ahead of the continuous multiple frames of RGB images in the driving scene of the unmanned mine vehicle to be predicted; input the optical flow image sequence into the feature extraction network model to extract the temporal features of the continuous multiple frames of driving trajectory during the driving process of the unmanned mine vehicle to be predicted; integrate the road curvature features and the temporal trajectory features to predict the steering angle of the unmanned mine vehicle to obtain a predicted steering angle; adjust the feature extraction network parameters according to the deviation between the actual steering angle and the predicted steering angle, reduce the loss function value, and make the output steering angle prediction value achieve the set effect.

[0195] A storage medium is also provided in a specific embodiment of the present invention, specifically, a computer-readable storage medium, which is a memory device in a terminal device for storing programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and, of course, the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the relevant methods in the above embodiments; one or more instructions in the computer-readable storage medium are loaded by the processor and execute the following steps: obtaining an RGB image sequence of the unmanned mine vehicle to be predicted; selecting two adjacent frames of images in the RGB image sequence, calculating and arranging them using an optical flow estimation algorithm to obtain an optical flow image sequence; inputting the RGB image sequence into a feature extraction network model to extract the spatial features of the curvature of the road ahead of multiple consecutive frames of RGB images in the driving scene of the unmanned mine vehicle to be predicted; inputting the optical flow image sequence into a feature extraction network model to extract the temporal features of the driving trajectories of multiple consecutive frames during the driving process of the unmanned mine vehicle to be predicted; fusing the road curvature features and the temporal trajectory features to predict the steering angle of the unmanned mine vehicle and obtain the predicted steering angle; adjusting the feature extraction network parameters according to the deviation between the actual steering angle and the predicted steering angle, reducing the loss function value, so that the output steering angle prediction value achieves the set effect.

[0196] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0197] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0198] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0199] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0200] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, from all points of view, the embodiments should be regarded as illustrative and non-restrictive. The scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and range of equivalents of the claims are included in the present invention. Any reference signs in the claims should not be construed as limiting the claim to which they relate.

[0201] In addition, it should be understood that although this specification describes the embodiments, not every embodiment contains only one independent technical solution. This description is for clarity only. Those skilled in the art should consider the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art. The above content is only for the purpose of illustrating the technical concept of the present invention and cannot be used to limit the scope of protection of the present invention. Any changes made based on the technical solution in accordance with the technical concept proposed by the present invention fall within the scope of protection of the claims of the present invention.

Claims

1. A method for predicting the steering angle of an unmanned mine vehicle, characterized in that: include: Obtain the RGB image sequence of the unmanned vehicle in the mine to be predicted; Select two adjacent frames in the RGB image sequence, calculate and arrange them using the optical flow estimation algorithm to obtain an optical flow image sequence; The RGB image sequence is input into the feature extraction network model to extract the spatial features of the curvature of the road ahead from multiple consecutive frames of RGB images in the predicted mine unmanned vehicle driving scene; The optical flow image sequence is input into the feature extraction network model to extract the temporal features of the continuous multi-frame driving trajectory of the unmanned vehicle to be predicted in the mine; By integrating road curvature features and time series trajectory features, the steering angle of the mine unmanned vehicle is predicted to obtain the predicted steering angle; According to the deviation between the actual steering angle and the predicted steering angle, the feature extraction network parameters are adjusted to reduce the loss function value so that the output steering angle prediction value reaches the set effect.

2. A method for predicting the steering angle of an unmanned mine vehicle according to claim 1, characterized in that: The step of obtaining an RGB image sequence of the unmanned mine vehicle to be predicted includes: Continuously shoot the mine unmanned vehicle operation scene at fixed time intervals to obtain RGB image sequences; The collected images are stored in chronological order to ensure that the timestamp and order of each frame are consistent to form a complete image sequence; Denoise, rectify, and format image sequences.

3. The method for predicting the steering angle of an unmanned mine vehicle according to claim 1, characterized in that: The step of selecting two adjacent frames of images from the RGB image sequence, calculating and arranging them using an optical flow estimation algorithm, and obtaining an optical flow image sequence includes: From the acquired RGB image sequence, two adjacent frames are selected, and the optical flow estimation algorithm is used to calculate the movement direction and speed of the image pixels, generate an optical flow field, and extract the trajectory information of the unmanned vehicle to be predicted between frames; Arrange the calculated optical flow data in chronological order to form an optical flow time series, ensuring that it is aligned with the original RGB image sequence; The optical flow image time series is stored in a fixed format and normalized.

4. The method for predicting the steering angle of an unmanned mine vehicle according to claim 1, characterized in that: The RGB image sequence is input into the feature extraction network model to extract the spatial features of the curvature of the road ahead from multiple consecutive frames of RGB images in the predicted mine unmanned vehicle driving scene, including: Input a continuous RGB image sequence to the feature extraction network model, normalize the image, and crop out areas unrelated to the road in the mine scene; The normalized and cropped image is input into the convolution layer of the two-stream convolutional network to extract the road curvature spatial features and obtain the road curvature feature output after the convolution layer; The road curvature feature output after the convolutional layer is input into the gated recurrent unit to obtain the road curvature spatial feature output with historical state information.

5. The method for predicting the steering angle of an unmanned mine vehicle according to claim 1, wherein: The optical flow image sequence is input into the feature extraction network model to extract the temporal features of the continuous multi-frame driving trajectory of the unmanned mine vehicle to be predicted, including: Input continuous optical flow images to the feature extraction network model. The optical flow images contain speed motion information in both horizontal and vertical directions. Input the optical flow image to the convolution layer of the two-stream convolutional network to extract the optical flow temporal trajectory features, and obtain the optical flow temporal trajectory feature output of the unmanned vehicle to be predicted after passing through the convolution layer; The optical flow temporal trajectory feature output of the unmanned vehicle to be predicted after the convolution layer is input into the gated recurrent unit to obtain the optical flow image temporal trajectory feature output with historical temporal state information.

6. A method for predicting the steering angle of an unmanned mine vehicle according to claim 1, characterized in that: The method of fusing road curvature features and time series trajectory features to predict the steering angle of the mine unmanned vehicle to obtain the predicted steering angle includes: Combine the road curvature spatial features and temporal trajectory features to form a fused feature representation; The fused features are input into the fully connected layer of the two-stream convolutional network to further map the high-dimensional features; The expected steering angle of the mine unmanned vehicle is predicted by the activation function exponential linear unit ReLU.

7. The method for predicting the steering angle of an unmanned mine vehicle according to claim 1, characterized in that: The method of adjusting the feature extraction network parameters according to the deviation between the actual steering angle and the predicted steering angle, reducing the loss function value, and making the output steering angle prediction value achieve the set effect includes: Select the root mean square error loss function to calculate the error between the actual steering angle and the predicted steering angle; Based on the gradient of the loss function, the backpropagation algorithm is used to calculate the gradient of the parameters of each layer of the two-stream convolutional neural network and the gated recurrent unit, and the optimization algorithm is used to adjust the weights and biases; Iterative training reduces the loss function value and ensures the model's ability to fit the data; After the loss function converges to the minimum value, the weight parameters of the model are optimized and the steering angle prediction value with the minimum error is output.

8. A mine unmanned vehicle steering angle prediction system, characterized in that: include: A data acquisition module is used to obtain the RGB image sequence of the unmanned vehicle in the mine to be predicted; The optical flow processing module is used to select two adjacent frames of images in the RGB image sequence, calculate and arrange them using the optical flow estimation algorithm, and obtain an optical flow image sequence; The road feature extraction module is used to input the RGB image sequence into the feature extraction network model to extract the spatial features of the curvature of the road ahead from multiple consecutive frames of RGB images in the predicted mine unmanned vehicle driving scene; The trajectory feature extraction module is used to input the optical flow image sequence into the feature extraction network model to extract the temporal features of the continuous multi-frame driving trajectory of the unmanned mine vehicle to be predicted; The feature fusion prediction module is used to fuse road curvature features and time series trajectory features to predict the steering angle of the mine unmanned vehicle and obtain the predicted steering angle; The error convergence module is used to adjust the feature extraction network parameters according to the deviation between the actual steering angle and the predicted steering angle, reduce the loss function value, and make the output steering angle prediction value reach the set effect.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method for predicting the steering angle of an unmanned mine vehicle according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for predicting the steering angle of an unmanned mine vehicle as claimed in any one of claims 1 to 7 are implemented.

Images