Vehicle yaw detection method, system and device based on ResNet18 and LSTM

Abstract

The invention belongs to the field of intelligent transportation, and relates to a vehicle yaw detection method, system and device based on ResNet18 and LSTM, and the method comprises the steps: inputting a vehicle surrounding environment image into a trained vehicle yaw detection model, and obtaining a yaw detection result; the vehicle yaw detection model is composed of a ResNet18 network and an LSTM (Long Short Term Memory) module; the processing of the vehicle surrounding environment image comprises the following steps: inputting the vehicle surrounding environment image into a ResNet18 network to extract spatial features of a lane line where a vehicle is located in the image; recognizing two-dimensional coordinates of the lane line according to the spatial features; calculating the driving angle of the current vehicle according to the two-dimensional coordinates of the lane line; inputting the vehicle driving angle into the LSTM module group to obtain a yaw detection result; through the strong feature extraction capability of the ResNet18 network, rich deep features can be extracted from the image, and an accurate basis is provided for vehicle yaw detection.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation, and specifically relates to a vehicle deviation detection method, system and device based on ResNet18 and LSTM. Background Art

[0002] With the rapid development of the automobile industry and people's increasing attention to traffic safety, the vehicle yaw detection system has become an important technical means to improve driving safety. In the modern traffic environment, the safety of vehicle driving is of vital importance, and vehicle yaw is an important factor leading to traffic accidents. Therefore, the development of an accurate and reliable vehicle yaw detection system has great practical significance. Traditional vehicle yaw detection methods mainly rely on mechanical sensors such as gyroscopes and accelerometers. Although these sensors can detect the yaw of the vehicle to a certain extent, they have problems such as low accuracy and susceptibility to interference. In addition, traditional methods can often only provide limited information, which is difficult to meet the needs of modern intelligent driving systems for comprehensive perception of vehicle status. Summary of the invention

[0003] In order to solve the problems existing in the above prior art, the present invention proposes a vehicle yaw detection method based on ResNet18 and LSTM, the method comprising: acquiring a vehicle surrounding environment image; inputting the vehicle surrounding environment image into a trained vehicle yaw detection model to obtain a yaw detection result; the vehicle yaw detection model is composed of a ResNet18 network and an LSTM module; the vehicle yaw detection model processes the vehicle surrounding environment image, comprising: inputting the vehicle surrounding environment image into the ResNet18 network to extract the spatial features of the lane line where the vehicle is located in the image; identifying the two-dimensional coordinates of the lane line according to the spatial features; calculating the driving angle of the current vehicle according to the two-dimensional coordinates of the lane line; and inputting the vehicle driving angle into the LSTM module group to obtain the yaw detection result.

[0004] A vehicle yaw detection system based on ResNet18 and LSTM, the system comprising: a data acquisition module, a spatial feature extraction module, a two-dimensional coordinate recognition module, a vehicle driving angle detection module and a yaw result detection module;

[0005] The data acquisition module is used to acquire an image of the vehicle's surrounding environment;

[0006] The spatial feature extraction module uses a ResNet18 network to extract the spatial features of the lane line where the vehicle is located in the vehicle surrounding environment image;

[0007] The two-dimensional coordinate recognition module is used to perform two-dimensional coordinate recognition of lane lines on spatial features to obtain the two-dimensional coordinates of lane lines;

[0008] The vehicle driving angle detection module is used to calculate the current vehicle driving angle according to the two-dimensional coordinates of the lane line;

[0009] The yaw result detection module is an LSTM module group, which is used to detect the current vehicle's driving angle and obtain the vehicle's yaw detection result.

[0010] To achieve the above objectives, the present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, any of the above-mentioned vehicle deviation detection methods based on ResNet18 and LSTM is implemented.

[0011] To achieve the above-mentioned purpose, the present invention also provides a vehicle yaw detection device based on ResNet18 and LSTM, comprising a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory, and is used to execute the computer program stored in the memory, so that the vehicle yaw detection device based on ResNet18 and LSTM executes any of the above-mentioned vehicle yaw detection methods based on ResNet18 and LSTM.

[0012] Beneficial effects of the present invention:

[0013] The present invention uses the powerful feature extraction capability of the ResNet18 network to extract rich deep features from images, including lane line information, road signs, etc., to provide an accurate basis for vehicle yaw detection; the LSTM network of the present invention can accurately analyze the dependency between consecutive frames, determine the vehicle's motion trend and yaw situation, and improve the accuracy and reliability of detection; the present invention uses batch normalization, Dropout and data enhancement to improve the generalization ability of the model, so that it can adapt to different driving environments and vehicle states; the present invention uses a camera for image acquisition, which is low in cost and easy to install, and can be widely used in various vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is the overall flow chart of the present invention;

[0015] Figure 2 This is a network diagram for determining whether the course is off course in the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] A vehicle yaw detection method based on ResNet18 and LSTM, the method comprising: acquiring a vehicle surrounding environment image; inputting the vehicle surrounding environment image into a trained vehicle yaw detection model to obtain a yaw detection result; the vehicle yaw detection model is composed of a ResNet18 network and an LSTM module; the vehicle yaw detection model processes the vehicle surrounding environment image, comprising: inputting the vehicle surrounding environment image into the ResNet18 network to extract the spatial features of a lane line where the vehicle is located in the image; identifying the two-dimensional coordinates of the lane line according to the spatial features; calculating the driving angle of the current vehicle according to the two-dimensional coordinates of the lane line; and inputting the vehicle driving angle into an LSTM module group to obtain a yaw detection result.

[0018] like Figure 1 As shown in the figure, a specific implementation method of a vehicle deviation detection method based on ResNet18 and LSTM is provided. The method includes: first, image acquisition, obtaining a video frame image in a vehicle driving environment through an image input module, and ensuring that the image contains complete and accurate lane line information. Then feature extraction is performed, and the deep features of the image are accurately extracted by using the convolution layer, batch normalization layer, activation function layer, pooling layer and ResNet18 network to work together. Then feature processing is performed, and the extracted features are converted and optimized through the flattening layer, fully connected layer and Dropout layer. After that, time series analysis is performed, and the LSTM network is used to analyze the dependency between consecutive frames to determine the vehicle deviation. Finally, the result is output. The output module composed of the Dense layer and the Softmax activation function generates lane line coordinates and deviation judgment results, providing an accurate decision-making basis for vehicle driving.

[0019] Aiming at the detection efficiency problem existing in the existing work, a vehicle yaw detection scheme based on ResNet18 and LSTM is proposed. Traditional vehicle yaw detection methods mainly rely on mechanical sensors, which are not accurate and easily interfered with, and it is difficult to meet the needs of modern intelligent driving systems for comprehensive perception of vehicle status. The present invention collects vehicle driving environment images through high-resolution cameras, and realizes accurate detection of vehicle yaw through four steps of image acquisition, feature extraction, feature processing and time series analysis. In each step, different layers and networks work together to give full play to their respective advantages. Finally, efficient processing procedures and advanced technical means enable real-time detection.

[0020] This detection technology has low overhead in the reasoning process and is suitable for the Internet of Things environment. The PC-side simulation results show that the proposed scheme can effectively and reliably complete the detection. Traditional technology has shortcomings in detection efficiency and accuracy, and this technology combines the advantages of convolutional neural networks and long short-term memory networks, and can simultaneously process spatial features and time series information, laying the foundation for improving the safety and reliability of vehicle driving. It not only has high-precision feature extraction and effective time series analysis capabilities, but also has good generalization capabilities and low cost and easy installation characteristics. It is widely applicable to various vehicles and has important practical significance and broad application prospects. The specific steps of the present invention are:

[0021] S1: Use the ResNet18 neural network to detect lane lines and obtain the two-dimensional coordinate information of lane lines.

[0022] Considering that the on-board camera is fixed, the use of a high-resolution camera can capture clearer and more detailed images of the vehicle's driving environment. This can ensure the integrity and accuracy of the lane line information and provide a high-quality data source for subsequent feature extraction and analysis. At the same time, a stable data transmission protocol can ensure real-time transmission of image data and avoid data loss or delay. For example, a high-speed USB interface or wireless transmission technology can be used to ensure that image data can be transmitted to the processing unit quickly and stably. In addition, in order to improve the efficiency and accuracy of image acquisition, appropriate frame rate and resolution parameters can be set and adjusted according to different driving scenarios and needs.

[0023] The input image is subjected to feature extraction. In the feature extraction step, a 7×7 convolutional layer with strides = 2 is used. Batch normalization is performed using a BN layer to speed up the model training process and improve model stability and generalization ability.

[0024] The activation layer uses the Sigmoid activation function, which can introduce nonlinear characteristics into the network and enhance the expressiveness of the model. The pooling layer uses 4×4 average pooling to reduce the size of the feature map, reduce the amount of calculation, and extract the main features at the same time.

[0025] The ResNet18 network module is used to extract deep features, so that the network can learn richer feature representations. The formula for the convolution operation is:

[0026]

[0027] Among them, x is the input image, w is the filter, and y is the output feature map.

[0028] The residual block structure of the ResNet18 network uses a combination of two layers of 3×3 convolution kernels. After the first layer of convolution, batch normalization and ReLU activation are performed. After the second layer of convolution, batch normalization is performed again. Finally, the input is added to the output of this layer to obtain the output of the residual block. When training the ResNet18 network, the adaptive learning rate algorithm Adam is used for optimization. In addition, the data set is re-labeled, and the lane lines blocked by the moving vehicles are extended using data enhancement technology to improve the generalization ability of the network. The loss function is expressed as follows:

[0029]

[0030] Where N is the number of samples, y is the true value, is the predicted value.

[0031] The collaborative work of each layer in the feature extraction process must ensure efficient and accurate extraction of deep features of the image. That is, from the convolution layer to the batch normalization layer, to the pooling, activation layer, and resnet18 module, the order must be guaranteed and the calculations must be layer by layer. The batch normalization formula can be expressed as follows:

[0032]

[0033] Where x is the input, μ is the mean, and σ is the standard deviation. ε is a constant, and γ and β are learnable parameters.

[0034] The feature processing step should be able to effectively transform and optimize the extracted features and improve the generalization ability of the model. Under the synergistic effect of the flattening layer, the fully connected layer and the Dropout layer, the feature processing step can effectively transform and optimize the extracted features. That is, the flattening operation of the flattening layer can convert the multi-dimensional feature map into a one-dimensional vector, which is convenient for the fully connected layer to process. The fully connected layer can combine and transform the features by adjusting the weights and biases of the neurons. The random discarding operation of the Dropout layer can increase the randomness of the model, prevent overfitting, and improve the generalization ability of the model. In the feature processing process, the Sigmod activation function is used to introduce nonlinear characteristics to the model.

[0035] S2: Perform relevant calculations on the two-dimensional coordinate information of the detected lane line to obtain the angle information of the vehicle at this moment as the input of the LSTM module.

[0036] Specifically, first determine whether the output of the ResNet18 module is the default coordinate information. If it is the default coordinate information, set the angle to the default angle. Otherwise, find the inflection point of the horizontal coordinate of each set of coordinates as the upper endpoint. If there is no inflection point, set the maximum horizontal coordinate of the first set of coordinates as the upper left endpoint, and set the minimum horizontal coordinate of the second set of coordinates as the upper right endpoint. The lower left endpoint and the lower right endpoint are the first coordinates of each set of coordinates. The line connecting the midpoints of the two upper endpoints and the lower endpoint is the current driving direction. Finally, the angle with the horizontal axis is calculated to obtain the current driving angle. The calculation formula is as follows:

[0037]

[0038] Among them, (mx1, my1) and (mx2, my2) are the midpoints of the upper and lower ends respectively. This calculation method of obtaining the driving angle through the turning point enables the model to update the angle sequence in time on a curved road.

[0039] S3: Use the LSTM module to process the input angle and analyze the angle dependency between consecutive frames.

[0040] like Figure 2 As shown in the figure, the LSTM network module in the time series analysis step adopts the following structure and parameters: the LSTM module contains two LSTM layers, the first LSTM layer contains 128 hidden units, and the second LSTM layer contains 64 hidden units. The activation function uses a combination of tanh and sigmoid, and the formula is as follows:

[0041]

[0042] Among them, x is the input, tanh is the value of the hyperbolic tangent function, and f(x) is the output value of the combined activation function

[0043] By linearly combining tanh and sigmod, the characteristics of both can be comprehensively utilized. Tanh has the characteristic of central symmetry, and can better handle positive and negative inputs in some cases. Sigmod has the characteristic of smoothness, which can compress the input value to between 0 and 1, and is suitable for situations such as probability output. By combining tanh and sigmod into an activation function, the features of vehicle images can be better extracted in the neural network. For example, by adjusting the weight coefficient, the network can pay more attention to the edges, shapes and other features of the vehicle.

[0044] The input of the LSTM module is the feature vector extracted from each frame of the image after preliminary processing, as well as the corresponding vehicle driving angle information.

[0045] The LSTM network should be able to accurately analyze the dependencies between consecutive frames. When training the LSTM network, the cross entropy loss function is used and optimized with the adaptive learning rate algorithm Adam. The cross entropy loss function can be expressed as follows:

[0046]

[0047] Among them, y is the true value, is the predicted value.

[0048] The adaptive learning rate update algorithm is as follows:

[0049] η t =η0*e -kt

[0050] Where η0 is the initial learning rate, t is the number of training rounds, and k is a tuning parameter.

[0051] As the number of training rounds increases, the value of the exponential term gradually decreases, causing the learning rate to gradually decrease. In the early stages of training, a larger learning rate can help the model move quickly toward the optimal solution. However, as training progresses, the model gradually approaches the optimal solution. At this time, a larger learning rate may cause the model to oscillate near the optimal solution and fail to converge to the optimal solution. By gradually reducing the learning rate, the model can converge more stably when approaching the optimal solution, improving the generalization ability of the model.

[0052] Compared with the traditional fixed learning rate, the exponential decay of the learning rate with the number of training rounds has the following advantages: 1) Adaptive adjustment: The learning rate can be automatically adjusted according to the progress of training, rather than using a fixed learning rate. This allows the model to be updated with an appropriate learning rate at different training stages, improving training efficiency and model performance. 2) Better convergence: In the later stages of training, a smaller learning rate can help the model converge to the optimal solution more stably, avoiding the oscillation and non-convergence problems that may be caused by a fixed learning rate. 3) Improved generalization ability: By gradually reducing the learning rate, the overfitting of the model to the training data can be reduced, and the generalization ability of the model can be improved, so that it performs better on the test data.

[0053] A vehicle yaw detection system based on ResNet18 and LSTM, the system comprising: a data acquisition module, a spatial feature extraction module, a two-dimensional coordinate recognition module, a vehicle driving angle detection module and a yaw result detection module;

[0054] The data acquisition module is used to acquire an image of the vehicle's surrounding environment;

[0055] The spatial feature extraction module uses a ResNet18 network to extract the spatial features of the lane line where the vehicle is located in the vehicle surrounding environment image;

[0056] The two-dimensional coordinate recognition module is used to perform two-dimensional coordinate recognition of lane lines on spatial features to obtain the two-dimensional coordinates of lane lines;

[0057] The vehicle driving angle detection module is used to calculate the current vehicle driving angle according to the two-dimensional coordinates of the lane line;

[0058] The yaw result detection module is an LSTM module group, which is used to detect the current vehicle's driving angle and obtain the vehicle's yaw detection result. The specific implementation of the system is the same as the specific implementation of the method.

[0059] In one embodiment of the present invention, the present invention also includes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned vehicle deviation detection methods based on ResNet18 and LSTM.

[0060] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0061] A vehicle yaw detection device based on ResNet18 and LSTM, comprising a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory and is used to execute the computer program stored in the memory, so that the vehicle yaw detection device based on ResNet18 and LSTM performs any of the above-mentioned vehicle yaw detection methods based on ResNet18 and LSTM.

[0062] Specifically, the memory includes: ROM, RAM, magnetic disk, USB flash drive, memory card or optical disk, etc., which can store program codes.

[0063] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0064] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A vehicle deviation detection method based on ResNet18 and LSTM, characterized in that: include: Acquire an image of the vehicle's surrounding environment; input the image of the vehicle's surrounding environment into the trained vehicle yaw detection model to obtain a yaw detection result; The vehicle yaw detection model consists of a ResNet18 network and an LSTM module; The vehicle yaw detection model processes the vehicle surrounding environment image by: inputting the vehicle surrounding environment image into the ResNet18 network to extract the spatial features of the lane line where the vehicle is located in the image; The two-dimensional coordinates of the lane line are identified according to the spatial features; the driving angle of the current vehicle is calculated according to the two-dimensional coordinates of the lane line; the vehicle driving angle is input into the LSTM module group to obtain the yaw detection result.

2. According to claim 1, a vehicle deviation detection method based on ResNet18 and LSTM is characterized in that: The structure of the ResNet18 network includes: multiple convolutional layers with different convolution kernel sizes, a dynamic convolution layer, a variable convolution, multiple pooling layers, and a residual block consisting of multiple convolutional layers and batch normalization layers.

3. The vehicle deviation detection method based on ResNet18 and LSTM according to claim 2 is characterized in that: The ResNet18 network extracts the spatial features of the lane line of the vehicle in the vehicle's surrounding environment image, including: inputting the environment image into multiple convolution layers with different convolution kernel sizes for feature extraction to obtain feature maps of different scales; using dynamic convolution layers to perform deep features on feature maps of different scales; using deformable convolutions to introduce a position transformation mechanism for deep features; inputting the feature map with the position transformation mechanism into multiple pooling layers for pooling; inputting the pooled feature map into the residual block to obtain two sets of unscaled two-dimensional coordinate points; the two-dimensional coordinate points reflect the relative position of the vehicle in the image, and respectively represent the two-dimensional coordinate information of the left and right lane lines of the lane.

4. The vehicle deviation detection method based on ResNet18 and LSTM according to claim 1, characterized in that: Calculating the current vehicle's driving angle includes: determining whether the two-dimensional coordinates of the lane line are default coordinate information. If they are default coordinates, the angle is the default angle; otherwise, finding the inflection point of the horizontal coordinate of each set of coordinates as the upper endpoint. If there is no inflection point, the maximum horizontal coordinate of the first set of coordinates is set as the upper left endpoint, and the minimum horizontal coordinate of the second set of coordinates is set as the upper right endpoint; the lower left endpoint and the lower right endpoint are the first coordinates of each set of coordinates, and the line connecting the midpoints of the two upper endpoints and the lower endpoint is the current driving direction. Finally, the angle with the horizontal axis is calculated to obtain the current driving angle.

5. The vehicle deviation detection method based on ResNet18 and LSTM according to claim 1, characterized in that: The LSM module group consists of three layers of LSTM modules. The output of each layer is used as the input of the next layer. A Dropout layer is added to the last LSTM layer, and an attention layer is added after the third LSTM layer.

6. The vehicle deviation detection method based on ResNet18 and LSTM according to claim 5 is characterized in that: Vehicle deviation detection in the LSTM module group includes: obtaining the angle sequence data during vehicle driving as the input of the LSTM module group; the angle sequence data passes through three layers of LSTM modules, where the output of each layer is used as the input of the next layer, and the high-level features of the angle change in the time series are gradually extracted, and a Dropout layer is added to the last LSTM layer, and a Dropout module is used on the recursive connection of the LSTM; finally, the change between adjacent angles is calculated, the change range of the vehicle driving angle is evaluated, and the final detection result is obtained.

7. A vehicle deviation detection system based on ResNet18 and LSTM, the system is used to execute the vehicle deviation detection method based on ResNet18 and LSTM according to any one of claims 1 to 6, characterized in that: include: Data acquisition module, spatial feature extraction module, two-dimensional coordinate recognition module, vehicle driving angle detection module and yaw result detection module; The data acquisition module is used to acquire an image of the vehicle's surrounding environment; The spatial feature extraction module uses a ResNet18 network to extract the spatial features of the lane line where the vehicle is located in the vehicle surrounding environment image; The two-dimensional coordinate recognition module is used to perform two-dimensional coordinate recognition of lane lines on spatial features to obtain the two-dimensional coordinates of lane lines; The vehicle driving angle detection module is used to calculate the current vehicle driving angle according to the two-dimensional coordinates of the lane line; The yaw result detection module is an LSTM module group, which is used to detect the driving angle of the current vehicle and obtain the vehicle's yaw detection result.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the vehicle deviation detection method based on ResNet18 and LSTM according to any one of claims 1 to 6.

9. A vehicle deviation detection device based on ResNet18 and LSTM, characterized in that: It comprises a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory and is used to execute the computer program stored in the memory, so that the vehicle yaw detection device based on ResNet18 and LSTM executes any one of the vehicle yaw detection methods based on ResNet18 and LSTM in claims 1 to 6.

Images