Lane departure warning method, device, equipment and storage medium

Abstract

The present invention belongs to the field of driving assistance technology, and discloses a lane departure warning method, device, equipment and storage medium. The present invention processes lane line data and vehicle driving data according to a preset communication protocol to obtain a time series signal data set; trains an initial network prediction model according to the time series signal data set to obtain a target network prediction model, and the target network prediction model is composed of an encoder module, a decoder module and a discriminator module; inputs the data to be predicted into the target network prediction model to obtain a prediction result; when the prediction result is an abnormal lane deviation, a warning signal is issued to make the driver pay attention to driving safety. By analyzing lane parameters and driving parameters, and then using the time series signal data generated by CAN communication, a deep learning method is run to perform modeling, and a multi-task learning method is used to predict whether the vehicle deviates from the lane, thereby improving the accuracy of the prediction and the robustness of the prediction model.

Description

Technical Field

[0001] The present invention relates to the field of driving assistance technology, and in particular to a lane departure warning method, device, equipment and storage medium. Background Art

[0002] The intelligent driving system is a complex intelligent operation system. It obtains signal data through intelligent cameras and various sensors, uses CAN communication technology for data transmission and data interaction, and realizes automatic control and adaptive adjustment of the driving vehicle through intelligent analysis and decision-making of time series data. However, during the intelligent driving process, the vehicle may deviate from the established lane due to various vehicle conditions, road conditions, etc., resulting in dangerous driving behavior. It is of great significance to use computer-related technology to analyze the CAN communication data generated by various signal sources, determine whether the signal source data at the current moment is normal, predict whether the vehicle deviates from the lane, formulate and quickly initiate corresponding safety precautions and reminder measures, ensure the safe operation of the vehicle, and avoid sudden accidents. The traditional lane departure detection method usually detects the position of the lane line, then integrates prior knowledge, and uses other sensor data for intelligent analysis to determine whether it deviates from the driving lane. This method can only be applied to scenes with clear lane lines and good road conditions, and this method relies heavily on the accurate detection and recognition of lane lines. When the lane lines are unclear at night, or there are no lane lines, or the vehicle is driving too fast, the lane departure detection method cannot be used. Fast, resulting in blurred lane line shooting. Since the lane line cannot be accurately detected, it is difficult to determine whether the vehicle has deviated. There are few methods that use time series data modeling and prediction methods for different signal source data generated by lane line parameters and vehicle driving parameters through CAN communication to determine whether the vehicle has deviated. The current processing methods based on time series data are usually based on seq2seq deep learning methods. These methods simply connect the encoder and decoder in series, and cannot extract richer feature information. At the same time, they fail to consider the working conditions of vehicles at different speeds and different states (such as left turn, right turn, U-turn, etc.). The network model is not robust enough. Similar patents either use lane line parameters, vehicle driving parameters and warning function status signals to form a model feature data sample set, and train the model by building a decision tree to predict whether the vehicle has lane deviation during driving. This method uses a decision tree model for training, and needs to judge each parameter one by one. The model is easily affected by biased parameters, resulting in biased models. At the same time, it fails to comprehensively consider the impact of the fusion of various parameters on model prediction, so it is not accurate enough and not robust enough. Other similar patents use relatively simple modeling methods to analyze the impact of parameters on vehicle lane departure, or use expert experience knowledge to make judgments. They fail to fully utilize the advantages of data-driven and fail to explore the intrinsic relationship between related data, making the model judgment not robust enough. Therefore, how to solve the low accuracy of the prediction model has become an urgent problem to be solved.

[0003] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the present invention is to provide a lane departure warning method, device, equipment and storage medium, aiming to solve the technical problem of low accuracy of existing lane departure warning methods.

[0005] To achieve the above object, the present invention provides a lane departure warning method, which comprises the following steps:

[0006] The lane line data and vehicle driving data are processed according to a preset communication protocol to obtain a time series signal data set;

[0007] Training the initial network prediction model according to the time series signal data set to obtain a target network prediction model, wherein the target network prediction model is composed of an encoder module, a decoder module, and a discriminator module;

[0008] Input the data to be predicted into the target network prediction model to obtain the prediction result;

[0009] When the prediction result is that the lane has deviated abnormally, a warning signal is issued to make the driver pay attention to driving safety.

[0010] Optionally, the initial network prediction model is trained according to the time series signal data set to obtain a target network prediction model, wherein the target network prediction model is composed of an encoder module, a decoder module and a discriminator module, including:

[0011] Inputting the time series signal data set into the encoder module of the initial network prediction model to obtain encoding features;

[0012] Inputting the encoded features into a decoder module of an initial network prediction model to obtain prediction features;

[0013] Training a discriminator module of an initial network prediction model according to the prediction features to obtain a target training result;

[0014] The initial network prediction model is trained according to the target training result and the total loss function to obtain the target network prediction model.

[0015] Optionally, the inputting the time series signal data set into an encoder module of an initial network prediction model to obtain encoding features includes:

[0016] Extracting features from the time series signal data set based on a preset neural network to obtain multiple moment features;

[0017] Based on the preset recurrent network, multiple moment features are interacted to obtain a feature vector;

[0018] Input multiple moment features and the feature vector into the feature interaction module of the encoder module to obtain the encoding feature.

[0019] Optionally, the inputting the multiple moment features and the feature vector into a feature interaction module of an encoder module to obtain the encoding feature comprises:

[0020] Calculate the features at multiple moments according to a preset calculation method to obtain the feature mean and feature variance;

[0021] Normalizing multiple time features according to a preset regularization method to obtain normalized features;

[0022] Normalizing the feature vector according to a preset regularization method to obtain a normalized vector;

[0023] Performing weight calculation according to the normalized features and the normalized vector to obtain multiple moment weights;

[0024] Based on the preset attention mechanism and multiple moment weights, the feature mean and feature variance are weighted and calculated to obtain the target mean and target variance;

[0025] The target mean and the target variance are input into a preset alignment network module to obtain encoding features.

[0026] Optionally, the training of a discriminator module of an initial network prediction model according to the prediction feature to obtain a target training result includes:

[0027] Training a true and false data discriminator module in the discriminator module according to the prediction feature and the first loss function to obtain a first training result;

[0028] Training the operating condition classification module in the discriminator module according to the prediction feature and the second loss function to obtain a second training result;

[0029] Training the lane abnormal deviation discrimination module in the discriminator module according to the prediction feature and the third loss function to obtain a third training result;

[0030] A target training result is obtained according to the first training result, the second training result and the third training result.

[0031] Optionally, training a true and false data discriminator module in the discriminator module according to the prediction feature and the first loss function to obtain a first training result includes:

[0032] Perform vector calculation based on the time series signal data set to obtain the actual vector;

[0033] Input the feature vector and the actual vector to the true and false data discriminator module for true and false judgment to obtain a true and false result;

[0034] The true and false data discriminator module is trained according to the true and false results and the first loss function to obtain a first training result.

[0035] Optionally, the training of the lane abnormal departure discrimination module in the discriminator module according to the prediction feature and the third loss function to obtain a third training result includes:

[0036] The actual vector is constructed according to a preset construction method to obtain an abnormal vector;

[0037] Performing data processing on the feature vector, the actual vector and the abnormal vector according to a preset processing method to obtain a processed data set;

[0038] Inputting the processed data set into a lane abnormal deviation discrimination module to obtain a lane result;

[0039] The lane abnormal deviation discrimination module is trained according to the lane result and the third loss function to obtain a third training result.

[0040] In addition, to achieve the above-mentioned purpose, the present invention further provides a lane departure warning device, the lane departure warning device comprising:

[0041] A processing module, used to process the lane line data and the vehicle driving data according to a preset communication protocol to obtain a time series signal data set;

[0042] A training module, used to train the initial network prediction model according to the time series signal data set to obtain a target network prediction model, wherein the target network prediction model is composed of an encoder module, a decoder module and a discriminator module;

[0043] A prediction module, used to input the data to be predicted into the target network prediction model to obtain a prediction result;

[0044] The warning module is used to send out a warning signal when the prediction result is that the lane has deviated abnormally, so as to make the driver pay attention to driving safety.

[0045] In addition, to achieve the above-mentioned purpose, the present invention also proposes a lane departure warning device, which includes: a memory, a processor, and a lane departure warning program stored in the memory and executable on the processor, and the lane departure warning program is configured to implement the steps of the lane departure warning method described above.

[0046] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a lane departure warning program is stored, and when the lane departure warning program is executed by a processor, the steps of the lane departure warning method described above are implemented.

[0047] The present invention processes lane line data and vehicle driving data according to a preset communication protocol to obtain a time series signal data set; trains the initial network prediction model according to the time series signal data set to obtain a target network prediction model, the target network prediction model is composed of an encoder module, a decoder module and a discriminator module; inputs the data to be predicted into the target network prediction model to obtain a prediction result; when the prediction result is an abnormal lane deviation, an early warning signal is issued to make the driver pay attention to driving safety. By analyzing lane parameters and driving parameters, and then using the time series signal data generated by CAN communication, a deep learning method is run to model, and a multi-task learning method is used to predict whether the vehicle deviates from the lane, thereby improving the accuracy of the prediction and the robustness of the prediction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a structural diagram of a lane departure warning device in a hardware operating environment involved in an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of a process flow of a first embodiment of a lane departure warning method according to the present invention;

[0050] Figure 3 A schematic diagram of a model structure of an embodiment of a lane departure warning method of the present invention;

[0051] Figure 4 A schematic diagram of a flow chart of a second embodiment of a lane departure warning method according to the present invention;

[0052] Figure 5 This is a structural block diagram of the first embodiment of the lane departure warning device of the present invention.

[0053] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0054] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0055] Reference Figure 1 , Figure 1 A schematic diagram of the structure of a lane departure warning device in a hardware operating environment according to an embodiment of the present invention.

[0056] like Figure 1As shown, the lane departure warning device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0057] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the lane departure warning device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0058] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a lane departure warning program.

[0059] exist Figure 1 In the lane departure warning device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the lane departure warning device of the present invention can be set in the lane departure warning device, and the lane departure warning device calls the lane departure warning program stored in the memory 1005 through the processor 1001, and executes the lane departure warning method provided by the embodiment of the present invention.

[0060] The embodiment of the present invention provides a lane departure warning method, referring to Figure 2 , Figure 2 The figure is a flow chart of a first embodiment of a lane departure warning method according to the present invention.

[0061] In this embodiment, the lane departure warning method includes the following steps:

[0062] Step S10: Processing the lane line data and vehicle driving data according to a preset communication protocol to obtain a time series signal data set.

[0063] It should be noted that the executor of this embodiment is a lane departure warning device, wherein the lane departure warning device has functions such as data processing, data communication and program running. The lane departure warning device can be an integrated controller, a control computer and other devices, and of course it can also be other devices with similar functions, and this embodiment does not limit this.

[0064] It can be understood that the preset communication protocol refers to the pre-set CAN communication protocol, the lane line data refers to the relevant parameters of the lane line, the vehicle driving data refers to the relevant parameters of vehicle driving, and the timing signal data set refers to multiple timing signals obtained through processing of the CAN communication protocol.

[0065] In a specific implementation, the lane line data and vehicle driving data are processed according to a preset CAN communication protocol to obtain a data set consisting of multiple time series signals.

[0066] Step S20: training the initial network prediction model according to the time series signal data set to obtain a target network prediction model, wherein the target network prediction model is composed of an encoder module, a decoder module and a discriminator module.

[0067] It can be understood that the initial network prediction model refers to the encoder composed of transformer, LSTM and feature interaction modules, the prediction model composed of LSTM-based decoder and multiple task discriminators, the network prediction model trained by the target network prediction model, the encoder module structure consists of transformer, LSTM two parallel branches and a feature interaction module, the decoder module consists of a general LSTM module, the discriminator module consists of a true and false data discriminator module Discriminator, a working condition classification module Mode Discriminator, and a lane abnormal deviation discrimination module Discriminator2.

[0068] In a specific implementation, an initial network prediction model is trained based on a time series signal data set, and a target network prediction model is obtained according to the training results.

[0069] Step S30: input the data to be predicted into the target network prediction model to obtain the prediction result.

[0070] It can be understood that the data to be predicted refers to the data used for predictive analysis, and the prediction result refers to the result of whether an abnormality occurs in the lane. The data to be predicted is predicted by the target network prediction model to obtain the prediction result.

[0071] Step S40: When the prediction result is that the lane has deviated abnormally, a warning signal is issued to alert the driver to driving safety.

[0072] It is understandable that the warning signal refers to a signal that alerts people to possible danger or emergency situations. When the prediction result is determined to be an abnormal lane deviation, a warning signal is issued to the driver.

[0073] It should be noted that if Figure 3 As shown in the figure, by analyzing the time series signal data generated by lane parameters and driving parameters through CAN communication, deep learning methods are used for modeling, and multi-task learning is used to predict whether the vehicle deviates from the lane. In this method, one branch uses the transformer method to learn the mutual influence of feature vectors at different time points, learns new feature vectors, and obtains the mean and variance of each new time series feature vector. The other branch obtains the comprehensive features of the time series T through the LSTM encoder, obtains the feature vector S1 after normalization, and performs point multiplication with each time series feature in the previous branch to obtain the corresponding attention weight, and then uses the weight to point multiply the mean and variance of each time series feature to obtain the final feature mean and variance, and then merges the final feature mean and variance into the feature vector S1 to obtain more robust features, and inputs the obtained features into the LSTM decoder to predict the signal at the next moment. At the same time, the predicted signal and the actual signal are input into the discriminator network, and the GAN training mode is used to judge the true and false data. Considering the driving speed state of the vehicle, a vehicle working condition classification model is constructed to predict the working condition, and abnormal data is constructed at the same time, and the loss relationship between normal data, abnormal data and predicted data is explored to judge whether the next moment is in an abnormal state. This embodiment fully considers the intrinsic relationship between the predicted signal characteristics and the previous T time series characteristics, explores the interactive characteristics of the mean and variance through the attention mechanism, and reconstructs more robust features. At the same time, multiple tasks such as prediction, true and false data discrimination, working condition classification, and abnormal data classification are constructed for model training, making the model more robust and efficient. The above method can be used to analyze and process the CAN communication signal in intelligent driving, determine whether the current signal is normal, and determine whether the lane is deviated, and the abnormal judgment is more accurate.

[0074] This embodiment processes lane line data and vehicle driving data according to a preset communication protocol to obtain a time series signal data set; trains the initial network prediction model according to the time series signal data set to obtain a target network prediction model, which is composed of an encoder module, a decoder module, and a discriminator module; inputs the data to be predicted into the target network prediction model to obtain a prediction result; when the prediction result is an abnormal lane deviation, an early warning signal is issued to make the driver pay attention to driving safety. By analyzing lane parameters and driving parameters, and then running the deep learning method for modeling through the time series signal data generated by CAN communication, a multi-task learning method is used to predict whether the vehicle deviates from the lane, thereby improving the accuracy of the prediction and the robustness of the prediction model.

[0075] refer to Figure 4 , Figure 4 The figure is a flow chart of a second embodiment of a lane departure warning method according to the present invention.

[0076] Based on the first embodiment described above, the lane departure warning method of this embodiment includes, in step S20:

[0077] Step S21: input the time series signal data set into the encoder module of the initial network prediction model to obtain encoding features.

[0078] It can be understood that the coding feature refers to the coding feature extracted and constructed by the system signal feature at each time point, and the coding feature is obtained by encoding the time series signal data set through the encoder module of the initial network prediction model.

[0079] It should be noted that in order to accurately obtain the coding features, further, the time series signal data set is input into the encoder module of the initial network prediction model to obtain the coding features, including: extracting features of the time series signal data set based on a preset neural network to obtain multiple moment features; interacting multiple moment features based on a preset recurrent network to obtain a feature vector; inputting multiple moment features and the feature vector into the feature interaction module of the encoder module to obtain the coding features.

[0080] It can be understood that the preset neural network refers to a neural network architecture (transformer) based on the attention mechanism, multiple moment features refer to the system signal features at each time point, the preset recurrent network refers to a recurrent neural network (LSTM, Long Short-Term Memory), and the feature vector refers to a vector representing the features of a system signal in the vector space.

[0081] In the specific implementation, the features of the time series signal data set are extracted through a preset neural network to obtain the system signal features at each time point, and then the system signal features at each time point are interacted through a preset recurrent network to obtain the feature vector of the system signal features, and finally the system signal features at each time point and the feature vector of the system signal features are input into the feature interaction module to obtain the coding features.

[0082] It should be noted that in order to accurately obtain the coding features, further, the input of multiple moment features and the feature vector into the feature interaction module of the encoder module to obtain the coding features, including: calculating the multiple moment features according to a preset calculation method to obtain the feature mean and the feature variance; normalizing the multiple moment features according to a preset regularization method to obtain the normalized features; normalizing the feature vector according to the preset regularization method to obtain the normalized vector; performing weight calculation according to the normalized features and the normalized vector to obtain multiple moment weights; performing weighted calculation on the feature mean and the feature variance based on the preset attention mechanism and the multiple moment weights to obtain the target mean and the target variance; inputting the target mean and the target variance into the preset alignment network module to obtain the coding features.

[0083] It can be understood that the preset calculation method refers to the preset mean calculation method, the feature mean refers to the average value of all feature values, the feature variance refers to the variance of all feature values, the preset regularization method refers to the preset instance regularization method, the normalized feature refers to the normalized feature, the normalized vector refers to the normalized vector, the multiple moment weights refer to the feature weights at each time point, the preset attention mechanism refers to the preset attention mechanism, the target mean refers to the final feature mean, the target variance refers to the final feature variance, and the preset alignment network module refers to the preset Align alignment network module.

[0084] In the specific implementation, the features at multiple moments are calculated based on a preset mean calculation method to obtain the average value of all feature values ​​and the variance of all feature values, and then the features and feature vectors at multiple moments are normalized according to the instance regularization method to obtain normalized features and normalized vectors, and then the normalized features and normalized vectors are weighted to obtain the feature weights of each time point, and the feature means and feature variances are weighted according to the attention mechanism and the weights of multiple moments to obtain the final feature mean and final feature variance, and finally the final feature mean and final feature variance are input into the Align alignment network module to obtain the encoded features.

[0085] It should be noted that the encoder model consists of transformer, lstm and feature interaction module, where transformer and lstm are two parallel branches. The transformer branch is used to extract the system signal features at each time point and form the feature map C1. The features at different times interact to explore the role of the signal features at subsequent times in affecting the features at the current time. The lstm branch outputs the final feature vector C2; the feature interaction module calculates the corresponding mean U for each feature at each time in the transformer output feature C1. i (i=1,2,…,T) and variance σ i (i=1,2,…,T), and obtain the normalized features through the instance regularization method Instance Norm. The specific operation is: S1 i =(C1 i -U i ) / sqrt((σ i )^2+ε)where (i=1,2,…,T), C1 i is the input feature at time i, S1 i is the corresponding output feature, ε is set to 0.00001; the feature interaction module uses the LSTM output feature C2 to obtain the normalized vector S2 through the Instance Norm method. S2 = (C2-U2) / sqrt((σ2)^2+ε), where C2 is the input feature vector, U2 is the mean of vector C2, σ2 is the variance of vector C2, S2 is the corresponding output feature, and ε is set to 0.00001; the S1 calculated above is used as the key value k, and the S2 feature vector is used as the query vector Q, and the dot multiplication operation is performed, and then normalized through softmax to obtain the weight at each moment. The specific calculation method is w = softmax(K*Q T / sqrt(D)) where K∈R T*D ,Q∈R 1*D ,w∈R T*1 , T is the time period, D is the characteristic dimension of each time series; the mean U calculated above is i (i=1,2,…,T) and variance σi(i=1,2,…,T),w∈R T*1 , the final mean u,σ is obtained by weighting through the attention mechanism. Specifically:

[0086]

[0087] ; Input S2, u, σ into the Align alignment network module to obtain the final encoding feature C3. Specifically, the Align calculation method is: C3=S2*σ+u.

[0088] Step S22: input the encoded features into the decoder module of the initial network prediction model to obtain prediction features.

[0089] It can be understood that the predicted feature refers to the final feature of the prediction. The encoded feature is input into the decoder module of the initial network prediction model to obtain the predicted feature. For example, the encoder finally outputs feature C3 and inputs it into the decoder module to obtain the final predicted feature x't+1.

[0090] Step S23: training the discriminator module of the initial network prediction model according to the prediction features to obtain a target training result.

[0091] It can be understood that the target training result refers to the final model training result, and the discriminator module of the initial network prediction model is trained based on the prediction features to obtain the final model training result.

[0092] It should be noted that in order to accurately obtain the target training result, further, the discriminator module of the initial network prediction model is trained according to the predicted features to obtain the target training result, including: training the true and false data discriminator module in the discriminator module according to the predicted features and the first loss function to obtain the first training result; training the working condition classification module in the discriminator module according to the predicted features and the second loss function to obtain the second training result; training the lane abnormal deviation discrimination module in the discriminator module according to the predicted features and the third loss function to obtain the third training result; and obtaining the target training result based on the first training result, the second training result and the third training result.

[0093] It can be understood that the first loss function refers to the training loss function of the true and false data discriminator module, the first training result refers to the training result of the true and false data discriminator module, the second loss function refers to the training loss function of the training discriminator module, the second training result refers to the training result of the training discriminator module, the third loss function refers to the training loss function of the lane abnormal deviation discrimination module, and the third training result refers to the training result of the lane abnormal deviation discrimination module.

[0094] In a specific implementation, a true and false data discriminator module is trained based on the prediction feature and the training loss function of the true and false data discriminator module to obtain the training result of the true and false data discriminator module, the discriminator module is trained based on the prediction feature and the training loss function of the training discriminator module to obtain the training result of the training discriminator module, the lane abnormal deviation discriminator module is trained based on the prediction feature and the training loss function of the lane abnormal deviation discriminator module to obtain the training result of the lane abnormal deviation discriminator module, and finally the target training result is obtained according to the first training result, the second training result and the third training result.

[0095] It should be noted that in order to accurately obtain the first training result, further, the true and false data discriminator module in the discriminator module is trained according to the predicted features and the first loss function to obtain the first training result, including: performing vector calculation according to the time series signal data set to obtain the actual vector; inputting the feature vector and the actual vector to the true and false data discriminator module for true and false judgment to obtain a true and false result; training the true and false data discriminator module according to the true and false result and the first loss function to obtain the first training result.

[0096] It can be understood that the actual vector refers to the actual data feature vector, and the true or false result refers to whether the data is predicted or the result of the actual real data.

[0097] In a specific implementation, an actual data feature vector is obtained based on a time series signal data set, and the feature vector and the actual data feature vector are input into a true or false data discriminator module for true or false judgment to obtain a result of whether the data is predicted or actual real data. Finally, the true or false data discriminator module is trained based on the true or false result and a first loss function to obtain a first training result.

[0098] It should be noted that the true and false data discriminator module, Discriminator, adopts the generative adversarial network GAN training mode to train the feature vector x' generated by the encoder and decoder at time t+1 t+1 With the actual data feature vector x t+1 To judge whether the data is true or false, the model consists of three fully connected layers (fc1, fc2, fc3), with feature sizes of 256, 512, and 2, respectively. Through the training of the discriminator, the model can reach Nash equilibrium and fool the model into not being able to distinguish whether the data is predicted data or real data. The generation and discrimination loss functions of this branch during the training process are:

[0099]

[0100] Where: G represents the generative network composed of the encoder and decoder in the model, D represents the discriminator, P data (x) represents the real data distribution, P z (z) represents the distribution of the generator. Specifically, z represents the original data input to the model z = {x t-T+1 ,x t-T+2 ,…,x t}, the data is a data set within the time period T, the dimension of the data is the number of signal source signals, and x represents the real data x predicted at the next moment t+1 .

[0101] It should be noted that in order to accurately obtain the third training result, further, the lane abnormal deviation discrimination module in the discriminator module is trained according to the predicted features and the third loss function to obtain the third training result, including: constructing the actual vector according to a preset construction method to obtain an abnormal vector; performing data processing on the feature vector, the actual vector and the abnormal vector according to a preset processing method to obtain a processed data set; inputting the processed data set to the lane abnormal deviation discrimination module to obtain a lane result; training the lane abnormal deviation discrimination module according to the lane result and the third loss function to obtain the third training result.

[0102] It can be understood that the preset construction method refers to a preset abnormal data construction method, the abnormal vector refers to the constructed abnormal vector data, the preset processing method refers to the preset data difference processing method, the processed data set refers to the data set after difference processing, and the lane result refers to the lane normal or abnormal judgment result.

[0103] In a specific implementation, the actual vector is constructed based on a preset abnormal data construction method to obtain constructed abnormal vector data, and the feature vector, the actual vector and the abnormal vector are difference processed based on a preset data difference processing method to obtain a data set after difference processing, and the data set after difference processing is input into a lane abnormal deviation discrimination module to obtain a lane normal or abnormal discrimination result, and finally, the lane abnormal deviation discrimination module is trained according to the lane result and a third loss function to obtain a third training result.

[0104] It should be noted that the lane deviation discrimination module Discriminator2 uses the residual between the predicted data and the real data to determine whether the real data generated at the next moment is in an abnormal state compared with the predicted data at the next moment. Specifically, the model consists of three fully connected layers (fc1, fc2, fc3), and their feature sizes are 256, 512, and 2 respectively. When training the Discriminator2 branch of the discriminator module, it is necessary to first construct a batch of abnormal data x″ t+1 The specific construction method is to swap the column position of the real data, change the value of the corresponding measurement point data to make it exceed the normal data range, change the data of multiple column position points to make it fluctuate greatly, etc., and construct the abnormal data x″ t+1 , actual real data x t+1 , predict data X′ t+1 , and then input into the Discriminator2 discriminator network after making the difference between the two. Only (x′ t+1 -x t+1) is used as the true value label of the input feature as normal, and other combinations are abnormal. The loss function LDiscriminator2 for training this branch is the binary cross entropy loss;

[0105] It can be explained that the working condition classification module mode Discriminator is to classify the working conditions of the data. Its model consists of three fully connected layers (fc1, fc2, fc3), and its feature sizes are 256, 512, and k respectively. Among them, k is a specific working condition category. Specifically, this category can be divided into a specific number of categories according to the vehicle speed and driving status (straight, left turn, right turn, lane change, U-turn, etc.). The loss function when training this branch is L mode_Discriminator is the cross entropy loss.

[0106] Step S24: training the initial network prediction model according to the target training result and the total loss function to obtain the target network prediction model.

[0107] It can be understood that the total loss function refers to the final loss function. The initial network prediction model is trained according to the target training result and the total loss function to obtain the target network prediction model.

[0108] It should be noted that the model training loss function is constructed. The system loss function is divided into the prediction loss L forecast_loss , the generation and discrimination loss L of real data and predicted generated data gan , working condition classification loss L mode_Discriminator , whether the data is abnormal or not Discriminator2 , where L forecast_loss is the mean square error loss between the predicted data and the real data; construct the total loss function, the total loss is:

[0109] L total =r1*L forecast_loss +r2*L gan +r3*L mode_Discrirninator +r4*L Discriminator2 , where r1, r2, r3, r4 are hyperparameters.

[0110] It should be noted that the training network model and inference output results, each batch training data is [{x t-T+1 , x t-T+2 , …, x t}, x t+1 ,x″ t+1 ], where {x t-T+1 , x t-T+2 , …, x t} is the original input data of the model, which forms a data matrix with a width of time period T and a length of the number of signal source signals at a single moment. t+1 is the corresponding true value data at the next moment, x” t+1 For the corresponding abnormal data, in the inference stage, the forecast loss prediction branch outputs the predicted signal data for the next moment, the mode Discriminator condition classification data predicts the condition category of the signal at the next moment, and the Discriminator2 abnormal discrimination network outputs whether the lane has abnormal deviation; the entire process judgment method inputs the lane line parameters and vehicle driving parameters into the trained model according to the specified time series relationship using the signal values ​​generated by CAN communication, and the Discriminator2 discriminator outputs the judgment result to determine whether the lane has abnormal deviation, and through the output judgment result, the system prompts and alarms to remind the driver to pay attention to driving safety.

[0111] This embodiment obtains encoding features by inputting the time series signal data set to the encoder module of the initial network prediction model; inputting the encoding features to the decoder module of the initial network prediction model to obtain prediction features; training the discriminator module of the initial network prediction model according to the prediction features to obtain the target training results; training the initial network prediction model according to the target training results and the total loss function to obtain the target network prediction model. The time series signal data set is encoded by the encoder module to obtain encoding features, and then the encoding features are predicted by the decoder module to obtain prediction features. The discriminator module is trained according to the prediction features to obtain the target network prediction model, thereby improving the robustness of the prediction model.

[0112] In addition, an embodiment of the present invention further proposes a storage medium, on which a lane departure warning program is stored. When the lane departure warning program is executed by a processor, the steps of the lane departure warning method described above are implemented.

[0113] Reference Figure 5 , Figure 5 This is a structural block diagram of the first embodiment of the lane departure warning device of the present invention.

[0114] like Figure 5 As shown, the lane departure warning device proposed in the embodiment of the present invention includes:

[0115] The processing module 10 is used to process the lane line data and the vehicle driving data according to a preset communication protocol to obtain a time series signal data set.

[0116] The training module 20 is used to train the initial network prediction model according to the time series signal data set to obtain a target network prediction model, wherein the target network prediction model is composed of an encoder module, a decoder module and a discriminator module.

[0117] The prediction module 30 is used to input the data to be predicted into the target network prediction model to obtain the prediction result.

[0118] The warning module 40 is used to send out a warning signal when the prediction result is that the lane has deviated abnormally, so as to make the driver pay attention to driving safety.

[0119] This embodiment processes lane line data and vehicle driving data according to a preset communication protocol to obtain a time series signal data set; trains the initial network prediction model according to the time series signal data set to obtain a target network prediction model, which is composed of an encoder module, a decoder module, and a discriminator module; inputs the data to be predicted into the target network prediction model to obtain a prediction result; when the prediction result is an abnormal lane deviation, an early warning signal is issued to make the driver pay attention to driving safety. By analyzing lane parameters and driving parameters, and then running the deep learning method for modeling through the time series signal data generated by CAN communication, a multi-task learning method is used to predict whether the vehicle deviates from the lane, thereby improving the accuracy of the prediction and the robustness of the prediction model.

[0120] In one embodiment, the training module 20 is further used to input the time series signal data set into the encoder module of the initial network prediction model to obtain the encoding features;

[0121] Inputting the encoded features into a decoder module of an initial network prediction model to obtain prediction features;

[0122] Training a discriminator module of an initial network prediction model according to the prediction features to obtain a target training result;

[0123] The initial network prediction model is trained according to the target training result and the total loss function to obtain the target network prediction model.

[0124] In one embodiment, the training module 20 is further used to extract features from the time series signal data set based on a preset neural network to obtain multiple moment features;

[0125] Based on the preset recurrent network, multiple moment features are interacted to obtain a feature vector;

[0126] Input multiple moment features and the feature vector into the feature interaction module of the encoder module to obtain the encoding feature.

[0127] In one embodiment, the training module 20 is further used to calculate the features at multiple moments according to a preset calculation method to obtain a feature mean and a feature variance;

[0128] Normalizing multiple time features according to a preset regularization method to obtain normalized features;

[0129] Normalizing the feature vector according to a preset regularization method to obtain a normalized vector;

[0130] Performing weight calculation according to the normalized features and the normalized vector to obtain multiple moment weights;

[0131] Based on the preset attention mechanism and multiple moment weights, the feature mean and feature variance are weighted and calculated to obtain the target mean and target variance;

[0132] The target mean and the target variance are input into a preset alignment network module to obtain encoding features.

[0133] In one embodiment, the training module 20 is further used to train a true and false data discriminator module in the discriminator module according to the prediction feature and the first loss function to obtain a first training result;

[0134] Training the operating condition classification module in the discriminator module according to the prediction feature and the second loss function to obtain a second training result;

[0135] Training the lane abnormal deviation discrimination module in the discriminator module according to the prediction feature and the third loss function to obtain a third training result;

[0136] A target training result is obtained according to the first training result, the second training result and the third training result.

[0137] In one embodiment, the training module 20 is further used to perform vector calculation according to the time series signal data set to obtain an actual vector;

[0138] Input the feature vector and the actual vector to the true and false data discriminator module for true and false judgment to obtain a true and false result;

[0139] The true and false data discriminator module is trained according to the true and false results and the first loss function to obtain a first training result.

[0140] In one embodiment, the training module 20 is further used to construct the actual vector according to a preset construction method to obtain an abnormal vector;

[0141] Performing data processing on the feature vector, the actual vector and the abnormal vector according to a preset processing method to obtain a processed data set;

[0142] Inputting the processed data set into a lane abnormal deviation discrimination module to obtain a lane result;

[0143] The lane abnormal deviation discrimination module is trained according to the lane result and the third loss function to obtain a third training result.

[0144] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present invention. In specific applications, technicians in this field can make settings as needed, and the present invention does not limit this.

[0145] It should be understood that, although the various steps in the flowchart in the embodiment of the present application are displayed in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and it can be performed in other orders. Moreover, at least a portion of the steps in the figure may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and their execution order is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0146] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0147] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0148] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0149] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0150] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A lane departure warning method, characterized in that: The lane departure warning method comprises: The lane line data and vehicle driving data are processed according to a preset communication protocol to obtain a time series signal data set; Training the initial network prediction model according to the time series signal data set to obtain a target network prediction model, wherein the target network prediction model is composed of an encoder module, a decoder module, and a discriminator module; Input the data to be predicted into the target network prediction model to obtain the prediction result; When the prediction result is that the lane has deviated abnormally, a warning signal is issued to make the driver pay attention to driving safety; The initial network prediction model is trained according to the time series signal data set to obtain a target network prediction model, and the target network prediction model is composed of an encoder module, a decoder module and a discriminator module, including: Inputting the time series signal data set into the encoder module of the initial network prediction model to obtain encoding features; Inputting the encoded features into a decoder module of an initial network prediction model to obtain prediction features; Training a discriminator module of an initial network prediction model according to the prediction features to obtain a target training result; Train the initial network prediction model according to the target training result and the total loss function to obtain a target network prediction model; The step of training the discriminator module of the initial network prediction model according to the prediction features to obtain the target training result includes: Training a true and false data discriminator module in the discriminator module according to the prediction feature and the first loss function to obtain a first training result; Training the operating condition classification module in the discriminator module according to the prediction feature and the second loss function to obtain a second training result; Training the lane abnormal deviation discrimination module in the discriminator module according to the prediction feature and the third loss function to obtain a third training result; A target training result is obtained according to the first training result, the second training result and the third training result.

2. The method according to claim 1, characterized in that The inputting of the time series signal data set into the encoder module of the initial network prediction model to obtain the encoding features includes: Extracting features from the time series signal data set based on a preset neural network to obtain multiple moment features; Based on the preset recurrent network, multiple moment features are interacted to obtain a feature vector; Input multiple moment features and the feature vector into the feature interaction module of the encoder module to obtain the encoding feature.

3. The method according to claim 2, characterized in that The inputting of the multiple moment features and the feature vector into the feature interaction module of the encoder module to obtain the encoding feature includes: Calculate the features at multiple moments according to a preset calculation method to obtain the feature mean and feature variance; Normalizing multiple time features according to a preset regularization method to obtain normalized features; Normalizing the feature vector according to a preset regularization method to obtain a normalized vector; Performing weight calculation according to the normalized features and the normalized vector to obtain multiple moment weights; Based on the preset attention mechanism and multiple moment weights, the feature mean and feature variance are weighted and calculated to obtain the target mean and target variance; The target mean and the target variance are input into a preset alignment network module to obtain encoding features.

4. The method according to claim 1, characterized in that The step of training a true and false data discriminator module in the discriminator module according to the prediction feature and the first loss function to obtain a first training result includes: Perform vector calculation based on the time series signal data set to obtain the actual vector; Input the feature vector and the actual vector to the true and false data discriminator module for true and false judgment to obtain a true and false result; The true and false data discriminator module is trained according to the true and false results and the first loss function to obtain a first training result.

5. The method according to claim 1, characterized in that The training of the lane abnormal deviation discrimination module in the discriminator module according to the prediction feature and the third loss function to obtain a third training result includes: The actual vector is constructed according to a preset construction method to obtain an abnormal vector; Performing data processing on the feature vector, the actual vector and the abnormal vector according to a preset processing method to obtain a processed data set; Inputting the processed data set into a lane abnormal deviation discrimination module to obtain a lane result; The lane abnormal deviation discrimination module is trained according to the lane result and the third loss function to obtain a third training result.

6. A lane departure warning device, characterized in that: The lane departure warning device comprises: A processing module, used to process the lane line data and the vehicle driving data according to a preset communication protocol to obtain a time series signal data set; A training module, used to train the initial network prediction model according to the time series signal data set to obtain a target network prediction model, wherein the target network prediction model is composed of an encoder module, a decoder module and a discriminator module; A prediction module, used to input the data to be predicted into the target network prediction model to obtain a prediction result; A warning module, used to send out a warning signal when the prediction result is that the lane has deviated abnormally, so as to make the driver pay attention to driving safety; The training module is further used to input the time series signal data set into the encoder module of the initial network prediction model to obtain encoding features; Inputting the encoded features into a decoder module of an initial network prediction model to obtain prediction features; Training a discriminator module of an initial network prediction model according to the prediction features to obtain a target training result; Train the initial network prediction model according to the target training result and the total loss function to obtain a target network prediction model; The training module is further used to train a true and false data discriminator module in the discriminator module according to the prediction feature and the first loss function to obtain a first training result; Training the operating condition classification module in the discriminator module according to the prediction feature and the second loss function to obtain a second training result; Training the lane abnormal deviation discrimination module in the discriminator module according to the prediction feature and the third loss function to obtain a third training result; A target training result is obtained according to the first training result, the second training result and the third training result.

7. A lane departure warning device, characterized in that: The device comprises: a memory, a processor, and a lane departure warning program stored in the memory and executable on the processor, wherein the lane departure warning program is configured to implement the steps of the lane departure warning method according to any one of claims 1 to 5.

8. A storage medium, characterized in that: The storage medium stores a lane departure warning program, which, when executed by the processor, implements the steps of the lane departure warning method according to any one of claims 1 to 5.

Images