A driving hazard scene recognition method based on lightweight multimodal neural network

Abstract

The present invention relates to a method for identifying dangerous driving scenarios based on a lightweight multimodal neural network. The method comprises the following steps: obtaining driving video and vehicle data within a current time period; dividing the driving video into three vertically distributed driving zones; vertically averaging the image within each driving zone in each frame of the video to convert it into a row of pixels; then chronologically concatenating the corresponding row of pixels for each frame to form a motion profile for each driving zone; and inputting the motion profile for each driving zone and the vehicle data into a driving risk assessment model to obtain an identification result. The driving risk assessment model is a multimodal neural network comprising a visual data processing layer, a kinematic data processing layer, a data fusion layer, and a prediction layer. Compared with existing technologies, the present invention has the advantages of reducing the amount of running data, simplifying the model calculation process, reducing time consumption, and achieving high accuracy.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving algorithms, and in particular to a method for identifying driving hazard scenarios based on a lightweight multimodal neural network. Background Art

[0002] Autonomous vehicles are currently undergoing large-scale testing worldwide, with safety testing being a primary concern. Unlike traditional vehicle testing, which typically utilizes distance-based methods, autonomous vehicles primarily utilize scenario-based testing. Therefore, the construction of virtual driving scenarios is a key research issue. Among these, hazardous driving scenarios are generally considered more important than normal driving scenarios, as they can be used to more quickly identify potential safety issues, thereby improving testing efficiency.

[0003] To identify dangerous driving scenarios, traditional methods primarily rely on structured data, including kinematic data such as velocity and acceleration collected by radar. For example, the paper "Crash and near-crash prediction from vehicle kinematics data: A SHRP2 naturalistic driving study" uses classic machine learning classifiers based on kinematic data, including kNN, random forest, SVM, decision tree, Gaussian neighborhood, and AdaBoost. However, a major issue with these traditional methods is the high false positive rate due to the poor quality of the extracted structured data and incomplete perception of the driving environment. Summary of the Invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a driving hazard scene identification method based on a lightweight multimodal neural network for use in autonomous driving algorithm testing to improve the accuracy of autonomous driving testing.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A method for identifying dangerous driving scenarios based on a lightweight multimodal neural network, characterized by comprising the following steps:

[0007] S1. Obtain driving video and vehicle data within the current time period;

[0008] S2. Divide the driving video into three driving areas distributed vertically, perform vertical averaging on the image within each driving area of ​​each frame of the video, convert the image into a row of pixels, and then splice the rows of pixels corresponding to each frame together in chronological order to form a motion profile of each driving area;

[0009] S3, inputting the motion profile of each driving area and the vehicle data into the driving risk assessment model to obtain an identification result;

[0010] The driving risk assessment model is a multimodal neural network including a visual data processing layer, a kinematic data processing layer, a data fusion layer and a prediction layer, wherein: the visual data processing layer is a lightweight CNN network, which adopts the AlexNet network structure and introduces an attention mechanism for improvement, and is used to input the motion contour map into the lightweight CNN network and output visual features; the kinematic data processing layer is an LSTM network, which is used to input the vehicle data into the LSTM network and output kinematic features; the data fusion layer is a fully connected layer, which is used to input visual features and kinematic features and output identification results.

[0011] Furthermore, step S2 specifically includes:

[0012] S21. Divide the driving video into three driving areas based on the camera position and the distance from the vehicle, each area being divided by an upper boundary and a lower boundary.

[0013] S22, based on the current time period [t a ,t b ] in the driving video clip, sample each driving area obtained in step S21, and obtain the longitudinal [y l ,y u ], RGB pixel values ​​within the horizontal [0,w] rectangular range, where w is the video width, y l is the sampling lower boundary, y u is the upper sampling boundary;

[0014] S23, perform the following operations on the R, G, and B channels of the image within the rectangular range: take the pixel mean in the vertical direction, and convert (w×(y u -y l )) matrix is ​​compressed into a (w×1) matrix, and then the results of the three channels are superimposed to obtain a row of (w×3) pixel matrix corresponding to each frame;

[0015] S24, stitching together a row of pixel matrices obtained from each frame in time sequence to form (fps×(t b -t a ),w,3 matrix, generates a colored motion contour map according to the pixel matrix, where fps is the number of frames per second of the video.

[0016] Furthermore, in step S3, the lightweight CNN network introduces an attention mechanism module after each convolutional layer, performs channel attention and spatial attention transformation on the feature map, and reconstructs a new feature map. The calculation formulas for channel attention and spatial attention are as follows:

[0017] Attention c =σ(MLP(AvgPool(F))+MLP(MaxPool(F))

[0018] Attention s =σ(Conv([AvgPool(F),MaxPool(F)]))

[0019] Among them, Attention c , Attention s They represent the results of channel attention and spatial attention respectively, F represents the feature map output by a certain convolutional layer, σ represents the Sigmoid function, MLP represents a multi-layer perceptron network, and Conv represents a convolutional layer with an output channel of 1.

[0020] Furthermore, the output training set of the driving risk assessment model includes a normal event set and a high-risk event set, which are obtained as follows:

[0021] A1. Collect historical vehicle data;

[0022] A2. Use the 3σ principle of normal distribution to detect and filter out outliers in historical vehicle data and treat outliers as missing values.

[0023] A3. Use linear interpolation to fill in missing values ​​in historical vehicle data to obtain complete vehicle data;

[0024] A4. Take the vehicle acceleration data a from the complete vehicle data, draw and observe the distribution curve, and determine the acceleration threshold of obvious deceleration behavior, recorded as TH d ;

[0025] A5. Traverse all vehicle acceleration data in time sequence, according to the acceleration condition a≤TH d Collect emergency braking time t d , for each moment t d , take the time segment from d1 to d2 seconds to form the potential high-risk event segment e c ,Combined with video verification, we can eliminate false alarms caused by data collection errors, and group multiple high-risk event clips into a high-risk event set;

[0026] A6. Randomly sample multiple normal non-conflict events as a normal event set from the remaining vehicle acceleration data in step A5 using |d1+d2| as a time window.

[0027] Furthermore, in step A2, each non-empty kinematic feature variable of a piece of historical vehicle-mounted data is subjected to conditional judgment, and the variable that meets the condition is considered an abnormal value. The expression for the conditional judgment is:

[0028] |x-μ|>3σ

[0029] Where x is the non-empty kinematic characteristic variable, μ is the mean value of x, and σ is the standard deviation of x.

[0030] Furthermore, the calculation expression of the linear interpolation method in step A3 is:

[0031]

[0032] in, is a missing value, d i-1 is the previous non-empty nearest neighbor value of the missing value, d i+1 is the next non-empty nearest neighbor value of the missing value, n is the total number of records, t i-1 ,t i ,t i+1 d i-1 , d i+1 The corresponding moment.

[0033] Furthermore, the output training set of the driving risk assessment model includes a CNN network training set, which is obtained as follows:

[0034] Get historical driving videos;

[0035] The historical driving video is divided into three driving areas distributed vertically. The images in each driving area of ​​each frame of the video are averaged in the vertical direction and converted into a row of pixels. Then, the row of pixels corresponding to each frame is spliced ​​together in chronological order to form a motion contour map. All motion contours become the CNN network training set.

[0036] Furthermore, the CNN network training set is expanded by data augmentation processing, wherein the data augmentation processing includes randomly transforming brightness, contrast, saturation and hue, and flipping the motion contour map horizontally with a certain probability.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The present invention first divides the driving video image into regions and generates motion contour maps respectively, and performs data compression while retaining image features; secondly, the present invention designs a multimodal neural network with a visual data processing layer, a kinematic data processing layer, a data fusion layer and a prediction layer as a driving risk assessment model. In this model, a lightweight CNN network is used to simplify the amount of calculation, and an attention mechanism is introduced to improve the classification performance of the model; at the same time, the driving risk assessment model adds kinematic feature extraction in the recognition process through the LSTM network, effectively improving the accuracy of the model prediction. In summary, the present invention can effectively extract video data, reduce the amount of running data, simplify the model calculation process, and has low time consumption, high accuracy, and good practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a structural schematic diagram of the present invention.

[0040] Figure 2 It is a schematic diagram of a driving video of the present invention.

[0041] Figure 3 This is a schematic diagram of generating a motion profile of a driving area according to the present invention.

[0042] Figure 4 This is a motion contour map data enhancement effect diagram of the present invention.

[0043] Figure 5 Schematic diagram of ROC curve comparison of the model of the present invention. DETAILED DESCRIPTION

[0044] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0045] like Figure 1 As shown, this embodiment provides a method for identifying dangerous driving scenes based on a lightweight multimodal neural network, comprising the following steps:

[0046] Step S1: Acquire driving video and vehicle data within the current time period.

[0047] Step S2: Divide the driving video into three driving areas distributed vertically, perform vertical averaging on the image in each driving area of ​​each frame of the video, convert it into a row of pixels, and then splice the row of pixels corresponding to each frame together in chronological order to form a motion contour map of each driving area.

[0048] Step S3: Input the motion profile of each driving area and the vehicle data into the driving risk assessment model to obtain the identification results. The driving risk assessment model is a multimodal neural network including a visual data processing layer, a kinematic data processing layer, a data fusion layer, and a prediction layer:

[0049] The visual data processing layer is a lightweight CNN network. Based on AlexNet, the network structure is lightweight and improved by introducing the attention mechanism. The motion contour map is input into the lightweight CNN network and then the visual features are output.

[0050] The kinematic data processing layer is an LSTM network, which is used to input vehicle data into the LSTM network and then output kinematic features;

[0051] The data fusion layer is a fully connected layer that is used to input visual features and kinematic features and output the recognition results.

[0052] The above steps can be specifically explained in the following six parts:

[0053] 1. Motion contour map generation algorithm:

[0054] 1): For each forward driving video, such as Figure 2 As shown in , according to the camera position, three driving areas are divided from the original video according to the distance from the vehicle. Each area is divided by an upper boundary and a lower boundary, as shown in Figure 3 Shown in the upper part.

[0055] 2): Based on the time period [t a ,t b ], sample each region obtained in step 1.1; let fps be the number of frames per second of the video, w be the video width, y l is the sampling lower boundary, y u is the upper boundary of the sampling. Each driving area sample is processed and finally a length of fps×(t b -t a ), a motion profile with a width of w. The details are as follows:

[0056] I) Get the vertical [y l ,y u ], RGB pixel values ​​within the horizontal [0,w] rectangular range, that is, (y u -y l ,w,3) three-dimensional integer matrix (where '3' represents RGB three channels);

[0057] II) For each channel of RGB in the range, take the mean of the vertical pixels as the pixel value of a point, that is, take (y u -y l,w,3) the average value of the first dimension of the three-dimensional integer matrix, arranged into a row of pixels of 1×w, that is, the (1,w,3) matrix;

[0058] III) Splice a row of pixel matrices obtained from each frame in time sequence to form (fps×(t b -t a ),w,3) matrix, generate a color motion contour map according to the pixel matrix, such as Figure 3 The lower part shows the conversion of the mid-range driving area into a sports contour map.

[0059] 2. Building a lightweight CNN network with attention mechanism

[0060] The lightweight CNN network is based on the AlexNet network structure, and builds a lightweight convolutional neural network that includes a convolution layer for extracting local visual data features and a fully connected layer for global feature processing. At the same time, it also introduces an attention mechanism to improve the model, making the model more focused on the key position information of the motion contour map. When the motion contour map is input into the lightweight CNN network, the three motion contour maps of the three driving areas are first converted into matrices m near 、m mid and m far , merge the three matrices in the image channel domain to obtain a nine-channel matrix m1 for input.

[0061] The detailed construction process of the lightweight CNN network is as follows:

[0062] 1) First, the input layer is constructed. For example, a 224-pixel × 224-pixel image is converted into a (224, 224, 9) matrix.

[0063] 2) Pass m1 through the Conv1 layer and set the convolution layer parameters, including the number of filters, size and stride, activation function, etc. For example, use 16 11×11 filters with a stride of 4 for convolution and pass it through the ReLU activation function to obtain the matrix m2;

[0064] 3) Pass m2 through the Pool1 layer and set the pooling layer parameters, mainly including filter size and type, step size, etc. For example, use a 3×3 filter with a step size of 2 for maximum pooling to obtain matrix m3;

[0065] 4) Similarly, pass m3 through the Conv2 layer, set the convolution layer parameters, for example, use 32 5×5 filters with a stride of 1 and a padding of 2 for convolution, and pass it through the ReLU activation function to obtain the matrix m4;

[0066] 5) Similarly, pass m4 through the Pool2 layer and set the pooling layer parameters. For example, use a 3×3 filter with a stride of 2 to perform maximum pooling to obtain matrix m5.

[0067] 6) Similarly, pass m5 through the Conv3 layer, set the convolution layer parameters, for example, use 32 3×3 filters with a stride of 3 and a padding of 1 for convolution, and pass it through the ReLU activation function to obtain the matrix m6;

[0068] 7) Similarly, pass m6 through the Pool3 layer and set the pooling layer parameters. For example, use a 3×3 filter with a stride of 2 to perform maximum pooling to obtain matrix m7.

[0069] 8) Pass m6 through the AdaptiveAvgPool layer and set the parameters, for example, setting the output size to 3x3 to obtain the matrix m8;

[0070] 9) Pass m8 through the fully connected layer FC4 to smooth the three-dimensional matrix into a one-dimensional matrix m9, and output a one-dimensional matrix m of r×1 (for example, 128×1) 10 ;

[0071] 10) m 10 After the Drop4 layer, some neural nodes are discarded with a certain proportion of Dropout probability (such as 50%) to prevent overfitting, and the matrix m is obtained. 11 ;

[0072] 11) m 11 After the FC5 fully connected layer, the output is a one-dimensional matrix m of r×1 (for example, 32×1) 12 ;

[0073] 12) m 12 After the Drop5 layer, some neural nodes are discarded with a certain proportion of Dropout probability (such as 50%) to prevent overfitting, and the matrix m is obtained. 13 ;

[0074] 13) m 11 After passing through the FC6 fully connected layer, the output is a 2×1 matrix. The two values ​​in the matrix correspond to the predicted values ​​of the probability of belonging to the risk class and the risk-free class. Then, the predicted values ​​are processed using Softmax so that the sum of the probabilities of the two classes is 1. The overall network structure is shown in the following table:

[0075] Table 1. Multimodal network structure table

[0076] layer enter Output Conv1 224×224×9 55×55×16 Pool1 55×55×16 27×27×16 Conv2 27×27×16 27×27×32 Pool2 27×27×32 13×13×32 Conv3 13×13×32 13×13×32 Pool3 13×13×32 6×6×32 AdaptiveAvgPool 6×6×32 3×3×32 FC4 3×3×32 128×1 Drop4 128×1 128×1 FC5 128×1 32×1 Drop5 32×1 32×1 FC6 32x1 2×1

[0077] Finally, an attention mechanism module is introduced after each convolutional layer to perform channel attention and spatial attention transformation on the feature map, and reconstruct a new feature map. The calculation formulas for channel attention and spatial attention are as follows:

[0078] Attentionc =σ(MLP(AvgPool(F))+MLP(MaxPool(F))

[0079] Attention s =σ(Conv([AvgPool(F),MaxPool(F)]))

[0080] Among them, Attention c , Attention s They represent the results of channel attention and spatial attention respectively, F represents the feature map output by a certain convolutional layer, σ represents the Sigmoid function, MLP represents a multi-layer perceptron network, and Conv represents a convolutional layer with an output channel of 1.

[0081] 3. Data enhancement for motion contour maps

[0082] The training set of the driving risk assessment model includes the CNN network training set used for the input side. The acquisition method is basically the same as the motion contour map generation algorithm in the first part, including:

[0083] Obtain historical driving videos; divide the video into three driving areas distributed vertically, average the images in each driving area of ​​each frame in the vertical direction, and convert them into a row of pixels. Then, splice the rows of pixels corresponding to each frame together in chronological order to form a motion contour map. All motion contours become the CNN network training set.

[0084] In order to improve the generalization ability of the model, the CNN network training set is also expanded through data augmentation. The enhancement process includes random transformation of brightness, contrast, saturation and hue, and horizontal flipping of the motion contour map with a certain probability. The effect of data augmentation is shown in the figure below. Figure 4 shown.

[0085] Brightness transformation: Randomly change the brightness of the motion contour image. Let the original image be im1 and the brightness transformation factor be factor b , then the transformed image im2 is:

[0086] im2=factor b ×im1.

[0087] Saturation transformation: randomly change the saturation of the motion contour map, the saturation transformation factor is factor s , let the grayscale image corresponding to im2 be gray2, and the pixel mean of the grayscale image is calculated as mean, then the transformed image im3 is:

[0088] im3=factorc ×im2+(1-factor s )×gray2.

[0089] Contrast transformation: Randomly change the contrast of the motion contour map, the contrast transformation factor is factor c , convert im3 into the corresponding grayscale image, calculate the pixel mean of the grayscale image as mean, then the transformed image im4 is:

[0090] im4=factor c ×im3+(1-factor c )×mean.

[0091] Hue transformation: randomly change the hue of the motion contour map, the hue transformation factor is factor h , convert im4 into HSV format to get hue H, perform random transformation on the hue, and then convert the new HSV image back to the original format to get im5:

[0092] H new =H origin +factor h ×255.

[0093] Flip transformation: With a certain probability, im5 is flipped horizontally to obtain im6.

[0094] 4. Calibration of dangerous scenarios based on acceleration thresholds and auxiliary video verification

[0095] The training set for the driving risk assessment model includes a normal event set and a high-risk event set for the output side. The acquisition method includes using the 3σ principle of normal distribution to detect and filter outliers on historical vehicle data collected by radar, and using linear interpolation to fill missing values; based on the filled historical vehicle data, the acceleration distribution is obtained and the acceleration threshold for dangerous driving events is determined to judge obvious vehicle avoidance behavior; based on the threshold judgment, potential dangerous driving events are extracted; and based on video verification, the normal event set and high-risk event set are calibrated based on potential dangerous driving events. The detailed process is as follows:

[0096] 1) Collect historical vehicle data;

[0097] 2) Most of the vehicle kinematic characteristic variables in the on-board data conform to the normal distribution. The 3σ principle is used to filter outliers. That is, the following conditions are applied to each non-empty kinematic characteristic variable in a driving record. Those that meet the conditions are considered outliers and are treated as missing values:

[0098] |x-μ|>3σ

[0099] Where x is the kinematic parameter, μ is the mean value of x, and σ is the standard deviation of x.

[0100] 3) Due to the complex driving environment and numerous interference sources, the signal strength of the detection equipment will be affected, resulting in missing values ​​in the driving data. Therefore, these missing values ​​need to be filled. Linear interpolation is used to fill the missing values. The calculation formula is as follows:

[0101]

[0102] in, is a missing value, d i-1 is the previous non-empty nearest neighbor value of the missing value, d i+1 is the next non-empty nearest neighbor value of the missing value, n is the total number of records, t i-1 ,t i ,t i+1 d i-1 , d i+1 The corresponding moment.

[0103] 4) Extract vehicle acceleration data a from natural driving data, draw and observe the distribution curve Determine the acceleration threshold for obvious deceleration behavior, denoted as TH d .

[0104] 5) Traverse all vehicle acceleration data in time sequence, according to the acceleration condition a≤TH d Collect emergency braking time t d , for each moment t d , take the time segment from d1 to d2 seconds to form the potential high-risk event segment e c , combined with video verification, to eliminate false alarms caused by data collection errors, conflict_candidate High-risk event segments constitute a high-risk event set To avoid event overlap, ensure that the adjacent emergency braking moments meet the following conditions: d [i+1]-t d [i]≥|d1+d2|.

[0105] 6) Randomly sample n vehicles from the remaining vehicle acceleration data using |d1+d2| as the time window. normal_candidate Normal non-conflicting events as the normal event set

[0106] 5. Overall Construction of Driving Risk Assessment Model

[0107] The driving risk assessment model also uses conventional LSTM to extract kinematic features and fuses them with visual features extracted by a lightweight CNN network to further improve the recognition accuracy of the model.

[0108] 1) Extract vehicle data from the radar at certain time intervals and use the LSTM network to extract kinematic features. The output structure of the LSTM network is denoted as f kinematics .

[0109] 2) Process the potential high-risk event according to the motion contour map generation algorithm in the first part to obtain the corresponding motion contour map, and input the contour map into the lightweight CNN network to extract visual features. The output structure of the network is recorded as f vision .

[0110] 3) f vision With f kinematic Merge, that is [f vision f kinematic ] is used as the input of the fully connected layer, and the output is a 2×1 matrix. The two values ​​in the matrix correspond to the predicted values ​​of the probability of belonging to the risky category and the risk-free category. Then, Softmax is used to process the predicted values ​​so that the sum of the probabilities of the two categories is 1.

[0111] 4) Divide the normal event set from the fourth part and high-risk event sets The training set Θ is divided into 3:1 train Test set Θ test .

[0112] 5) Training the model, during training, the motion contour map is enhanced according to the third part, after n epoch epochs, the loss value of the model converges to a smaller value, stop training, and save the final model M VK .

[0113] 6) For the test set Θ test (including e c Normal events and e n Each event in the high-risk events) calls the trained M VK Model, obtain its predicted classification value, and statistically obtain the normal events predicted by the model and conflict incidents According to the prediction results of the test set, the confusion matrix is ​​generated as follows:

[0114] Table 2. Confusion matrix

[0115]

[0116] Calculate the sensitivity of the model I sensitivity , specificity Ispecificity , the formula is as follows:

[0117] I sensitivity =TP / (TP+FN)

[0118] I specificity =TN / (FP+TN)

[0119] And according to I sensitivity and I specificity Generate ROC curve to evaluate the prediction effect of the model.

[0120] like Figure 5 As shown in FIG, this is the effect diagram of the driving risk assessment model of the present invention. The more accurate the AUC value is, the better the effect is. The driving risk assessment model (VK-Net) of the present invention reaches 0.95, which has very good accuracy and precision.

[0121] 6. Identify dangerous scenarios based on driving risk assessment models.

[0122] The continuous current driving video is used to generate a motion profile through a motion profile generation algorithm. The motion profile and the kinematic feature variables extracted from the vehicle data are used as the input of the driving risk assessment model. After calculation, a prediction value of whether the driving is risky is obtained, and an alarm is issued if there is a risk.

[0123] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, embodiments of the present invention may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware. Furthermore, embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0124] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the process in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0125] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0127] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0128] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A method for identifying dangerous driving scenes based on a lightweight multimodal neural network, characterized in that: The following steps are involved: S1. Obtain driving video and vehicle data within the current time period; S2. Divide the driving video into three driving areas distributed vertically, perform vertical averaging on the image within each driving area of ​​each frame of the video, convert the image into a row of pixels, and then splice the rows of pixels corresponding to each frame together in chronological order to form a motion profile of each driving area; S3, inputting the motion profile of each driving area and the vehicle data into the driving risk assessment model to obtain an identification result; The driving risk assessment model is a multimodal neural network including a visual data processing layer, a kinematic data processing layer, a data fusion layer, and a prediction layer, wherein: the visual data processing layer is a lightweight CNN network, which is based on AlexNet and has a lightweight network structure and an attention mechanism for improvement. It is used to input the motion contour map into the lightweight CNN network and output visual features; the kinematic data processing layer is an LSTM network, which is used to input the vehicle data into the LSTM network and output kinematic features; the data fusion layer is a fully connected layer, which is used to input visual features and kinematic features and output identification results; The lightweight CNN network in S3 introduces an attention mechanism module after each convolutional layer, performs channel attention and spatial attention transformation on the feature map, and reconstructs a new feature map. The calculation formulas for channel attention and spatial attention are as follows: Attention c =σ(MLP(AvgPool(F))+MLP(MaxPool(F)) Attention s =σ(Conv([AvgPool(F),MaxPool(F)])) Among them, Attention c , Attention s They represent the results of channel attention and spatial attention respectively, F represents the feature map output by a certain convolutional layer, σ represents the Sigmoid function, MLP represents a multi-layer perceptron network, and Conv represents a convolutional layer with an output channel of 1.

2. The method for identifying dangerous driving scenes based on a lightweight multimodal neural network according to claim 1 is characterized in that: Step S2 specifically includes: S21. Divide the driving video into three driving areas based on the camera position and the distance from the vehicle, each area being divided by an upper boundary and a lower boundary. S22, based on the current time period [t a ,t b ] in the driving video clip, sample each driving area obtained in step S21, and obtain the longitudinal [y l ,y u ], RGB pixel values ​​within the horizontal [0,w] rectangular range, where w is the video width, y l is the sampling lower boundary, y u is the upper sampling boundary; S23, perform the following operations on the R, G, and B channels of the image within the rectangular range: take the pixel mean in the vertical direction, and convert (w×(y u -y l )) matrix is ​​compressed into a (w×1) matrix, and then the results of the three channels are superimposed to obtain a (w×3) pixel matrix per row for each frame; S24, stitching together a row of pixel matrices obtained from each frame in time sequence to form (fps×(t b -t a ),w,3) matrix, generates a colored motion contour map according to the pixel matrix, where fps is the number of frames per second of the video.

3. The method for identifying dangerous driving scenes based on a lightweight multimodal neural network according to claim 1, characterized in that: The output training set of the driving risk assessment model includes a normal event set and a high-risk event set, which are obtained as follows: A1. Collect historical vehicle data; A2. Use the 3σ principle of normal distribution to detect and filter out outliers in historical vehicle data and treat outliers as missing values. A3. Use linear interpolation to fill in missing values ​​in historical vehicle data to obtain complete vehicle data; A4. Take the vehicle acceleration data a from the complete vehicle data, draw and observe the distribution curve, and determine the acceleration threshold of obvious deceleration behavior, recorded as TH d ; A5. Traverse all vehicle acceleration data in time sequence, according to the acceleration condition a≤TH d Collect emergency braking time t d , for each moment t d , take the time segment from d1 to d2 seconds to form the potential high-risk event segment e c ,Combined with video verification, we can eliminate false alarms caused by data collection errors, and group multiple high-risk event clips into a high-risk event set; A6. Randomly sample multiple normal non-conflict events as a normal event set from the remaining vehicle acceleration data in step A5 using |d1+d2| as a time window.

4. The method for identifying dangerous driving scenes based on a lightweight multimodal neural network according to claim 3 is characterized in that: In step A2, each non-empty kinematic feature variable of a piece of historical vehicle data is subjected to conditional judgment, and the one that meets the condition is an abnormal value. The expression of the conditional judgment is: |x-μ|>3σ Where x is the non-empty kinematic characteristic variable, μ is the mean value of x, and σ is the standard deviation of x.

5. The method for identifying dangerous driving scenes based on a lightweight multimodal neural network according to claim 3 is characterized in that: The calculation expression of the linear interpolation method in step A3 is: in, is a missing value, d i-1 is the previous non-empty nearest neighbor value of the missing value, d i+1 is the next non-empty nearest neighbor value of the missing value, n is the total number of records, t i-1 ,t i ,t i+1 d i-1 , d i+1 The corresponding moment.

6. The method for identifying dangerous driving scenes based on a lightweight multimodal neural network according to claim 1, characterized in that: The output training set of the driving risk assessment model includes the CNN network training set, which is obtained as follows: Get historical driving videos; The historical driving video is divided into three driving areas distributed vertically. The images in each driving area of ​​each frame of the video are averaged in the vertical direction and converted into a row of pixels. Then, the row of pixels corresponding to each frame is spliced ​​together in chronological order to form a motion contour map. All motion contours become the CNN network training set.

7. The method for identifying dangerous driving scenes based on a lightweight multimodal neural network according to claim 6, characterized in that: The CNN network training set is expanded through data augmentation processing, which includes randomly transforming brightness, contrast, saturation and hue, and flipping the motion contour map horizontally with a certain probability.

Images