Dangerous driving behavior identification method and device

Abstract

The invention relates to the technical field of driving monitoring, in particular to a dangerous driving behavior recognition method and device, and aims to solve the problems that personal privacy is leaked, the detection precision is poor and a wearable device is difficult to ensure that a driver is equipped in the aspect of dangerous driving monitoring in the prior art. The identification method comprises the following steps: acquiring original data; preprocessing to obtain a three-domain spectrogram; dividing the three-domain spectrogram into a training set and a verification set; building an auto-encoder, and inputting the training set into the auto-encoder to obtain local features; fusing the time or space dimensions of the local features to obtain a time-distance-Doppler feature Ft and a time-distance-rhythm feature Fs; constructing a branch network by utilizing Transform and a CNN (Convolutional Neural Network), and respectively inputting Ft and Fs into the branch network to obtain a high-dimensional feature vector; adding two trainable parameters alpha and beta to the high-dimensional feature vector to obtain a final feature vector of the dynamic behavior features of the driver; and outputting the driver behavior label to complete dangerous driving behavior identification.

Description

Technical Field

[0001] The present invention relates to the technical field of driving monitoring, and in particular to a method and device for identifying dangerous driving behavior. Background Art

[0002] According to the World Health Organization (WHO) report, the number of people who die in road traffic accidents each year reaches a staggering 1.2 million; dangerous driving is still one of the main causes of these accidents (accounting for about 45%). The main causes of dangerous driving include distracted driving and fatigue driving.

[0003] It is worth noting that significant progress has been made in the technology for real-time monitoring of dangerous driving. This field has been well studied with commercial attempts such as Advanced Driver Assistance Systems (ADAS) and a large number of research works. For example, good applications of computer vision and driving behavior monitoring methods based on body wearable devices have been observed in recent years. However, vision-based methods are challenged by personal privacy issues, especially for public and shared vehicles. In addition, the detection accuracy is also limited by multiple factors such as lighting conditions, camera orientation, etc. Similarly, wearable device-based methods are difficult to generalize because it is not easy to ensure that drivers are always equipped with the required wearable devices. Summary of the invention

[0004] In view of this, the purpose of the present invention is to propose a method and device for identifying dangerous driving behavior to solve the problems of personal privacy leakage, poor detection accuracy and difficulty in ensuring that drivers are equipped with wearable devices in the existing monitoring of dangerous driving.

[0005] Based on the above purpose, the present invention provides a dangerous driving behavior identification method, including obtaining raw data, wherein the raw data is echo data of different driving behaviors of a driver in a vehicle; preprocessing the raw data to obtain a three-domain spectrogram, wherein the three-domain spectrogram includes a time-distance graph, a time-Doppler graph, and a speed rhythm graph of the driver's behavior; dividing the three-domain spectrogram into a training set and a verification set; building an autoencoder composed of a convolutional neural network, inputting the training set into the autoencoder, and obtaining local features related to the driver's dynamic behavior; fusing the time or space dimensions of the local features to obtain: a time-distance-Doppler feature F t and time-distance-rhythm characteristics F s ; Use Transformer and CNN to build a branch network to transform the time-distance-Doppler feature F t and time-distance-rhythm characteristics F sThe high-dimensional feature vector is obtained by respectively inputting the two trainable parameters α and β into the branch network; the high-dimensional feature vector is added with two trainable parameters α and β, α and β respectively represent the contribution of the classification performance of the two branch networks; a trainable weighted average strategy is adopted to fuse the two deep-level driver dynamic behavior features output by the branch network according to α and β, so as to obtain the final feature vector of the driver dynamic behavior feature; the final feature vector is mapped from 64 dimensions to 9 dimensions through a fully connected layer (FC), the driver behavior label is output, and the dangerous driving behavior recognition is completed.

[0006] As a further improvement of the present application, the obtaining of raw data includes using a millimeter wave radar and a data acquisition card to collect raw echo data of the driver's behavior.

[0007] As a further improvement of the present application, the raw data is preprocessed to obtain a three-domain spectrum, including using a mean elimination method to filter out interference from static clutter, and using a two-dimensional fast Fourier transform and incoherent accumulation method to obtain the three-domain spectrum.

[0008] As a further improvement of the present application, the mean elimination method is used to filter out the interference of static clutter, including that the echo of the static target remains basically unchanged in multiple frames of data, while the echo of the dynamic target changes with time. By taking the average of the data of each distance unit on all frames, the echo signal of each frame is subtracted from this mean, the influence of the static target is removed, and only the dynamic target signal related to the detected target is retained.

[0009] As a further improvement of the present application, the method of using incoherent accumulation includes accumulating the amplitude or power of multiple pulse echoes when the target speed changes greatly and the phase changes rapidly. Incoherent accumulation can effectively enhance the strength of the target signal.

[0010] As a further improvement of the present application, the three-domain spectrogram is obtained, including performing one-dimensional FFT on the distance dimension and Doppler dimension of each frame of preprocessed data respectively to obtain a range-Doppler spectrum, stacking the distance information and Doppler information on the time dimension respectively to obtain a time-distance graph and a time-Doppler graph; performing one-dimensional FFT along the time axis on each frequency bin in the data matrix of the time-Doppler spectrum to obtain a speed rhythm graph.

[0011] As a further improvement of the present application, the autoencoder composed of the convolutional neural network is constructed, and the training set is input into the autoencoder to obtain local features related to the driver's dynamic behavior, including

[0012] The autoencoder is composed of two convolutional encoders: encoder I and encoder II; encoder I and encoder II both include an input layer, a convolutional layer, a batch normalization layer and an activation function layer;

[0013] The shape of the input layer is determined according to the structure of the three-domain spectrogram; each convolutional layer is created using the Conv2D function, specifying the number of filters and filter size of the convolutional layer, specifying the specific activation function of the activation function layer and the filling method of the convolutional layer; each convolutional layer is followed by a batch normalization layer to normalize the features of the output channel, and the output after the batch normalization layer is processed is imported into the activation function layer, and the above steps are repeated five times to construct a five-layer convolutional network.

[0014] As a further improvement of the present application, the fusion of the time or space dimension of the local features includes:

[0015] Encoder I of the autoencoder is used to extract distance and Doppler semantic features from the time-distance graph (TR) and time-Doppler graph (TD) of the training set respectively; at the same time, encoder II of the autoencoder is used to extract rhythm features from the speed rhythm graph of the training set;

[0016] After feature extraction, the encoding features of the time-distance map (TR) and the time-Doppler map (TD) are concatenated according to the time dimension to obtain the time-distance-Doppler feature F t , the encoding features of the time-distance map (TR) and the time-Doppler map (TD) are concatenated according to the spatial dimension to obtain the time-distance-rhythm feature F s .

[0017] As a further improvement of the present application, the branch network constructed by using Transformer and CNN includes constructing an abstract representation extracted by the Transformer module using layer normalization, multi-head attention mechanism and multi-layer perceptron to process the time-distance-Doppler feature F t Get deeper feature Fcnn; use flattening layer, convolution layer, activation function, batch normalization, maximum pooling and output layer to build the abstract representation extracted by 1D-CNN module to process time-distance-rhythm feature F s Get deeper features Ftran; use a trainable weighted average strategy to fuse the final feature vector F of the input fully connected layer fuse .

[0018] A dangerous driving behavior identification device, comprising

[0019] An acquisition module is used to acquire raw data, wherein the raw data is echo data of different driving behaviors of the driver in the vehicle;

[0020] A preprocessing module, used for preprocessing the raw data to obtain a three-domain spectrogram, wherein the three-domain spectrogram includes a time-distance graph, a time-Doppler graph, and a speed rhythm graph of the driver's behavior;

[0021] A partitioning module, used to divide the three-domain spectra into a training set and a validation set;

[0022] The first stage processing module is used to build an autoencoder composed of a convolutional neural network, input the training set into the autoencoder, and obtain local features related to the driver's dynamic behavior;

[0023] The fusion module is used to fuse the time or space dimensions of local features to obtain: time-distance-Doppler feature F t and time-distance-rhythm characteristics F s ;

[0024] The second stage processing module is used to build a branch network using Transformer and CNN to transform the time-distance-Doppler feature F t and time-distance-rhythm characteristics F s Input them into the branch networks respectively to obtain high-dimensional feature vectors;

[0025] The feature processing output module is used to add two trainable parameters α and β to the high-dimensional feature vector, where α and β represent the contribution of the classification performance of the two branch networks respectively, and adopt a trainable weighted average strategy to fuse the two deep-level driver dynamic behavior features output by the branch network according to α and β to obtain the final feature vector of the driver dynamic behavior feature;

[0026] The output module is used to map the final feature vector of 64 dimensions to 9 dimensions through a fully connected layer (FC), output the driver behavior label, and complete the identification of dangerous driving behavior.

[0027] The beneficial effects of the present invention are as follows: by obtaining the original echo data to judge the different driving behaviors of the driver in the car, the problems of personal privacy leakage, poor detection accuracy and the difficulty in ensuring that the driver is equipped with wearable devices are simultaneously solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 A schematic diagram of a process of identifying dangerous driving behavior according to an embodiment of the present invention;

[0030] Figure 2 A schematic diagram of a dangerous driving behavior identification method according to an embodiment of the present invention;

[0031] Figure 3 A schematic diagram of a process of obtaining a three-domain spectrum in a method for identifying dangerous driving behaviors according to an embodiment of the present invention;

[0032] Figure 4 A schematic diagram of the structure of an autoencoder composed of a convolutional neural network in a method for identifying dangerous driving behavior according to an embodiment of the present invention;

[0033] Figure 5 A schematic diagram of the structure of a 1D-CNN model of a dangerous driving behavior identification method according to an embodiment of the present invention;

[0034] Figure 6 A schematic diagram of the structure of a Transformer model of a dangerous driving behavior identification method according to an embodiment of the present invention;

[0035] Figure 7 A schematic diagram of the structure of a network model of a dangerous driving behavior identification method according to an embodiment of the present invention;

[0036] Figure 8 Schematic diagrams of nine dangerous driving behaviors and corresponding collection patterns of a dangerous driving behavior identification method according to an embodiment of the present invention;

[0037] Fig. 9 2Dt-SNE, schematic diagram of the classification results of the validation set of a dangerous driving behavior identification method according to an embodiment of the present invention;

[0038] Fig.10 3Dt-SNE,schematic diagram of the classification results of the validation set of a dangerous driving behavior identification method according to an embodiment of the present invention;

[0039] Fig.11 A schematic diagram of a confusion matrix of a validation set classification result of a dangerous driving behavior identification method according to an embodiment of the present invention;

[0040] Fig.12 The figure shows the accuracy curves of the training set and the validation set of a dangerous driving behavior identification method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0042] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0043] like Figure 1-3 As shown, a dangerous driving behavior identification method includes:

[0044] The original data is obtained, wherein the original data is echo data of different driving behaviors of the driver in the vehicle; the original echo data of the driver's behavior is collected using a millimeter wave radar and a data acquisition card.

[0045] like Figure 8 As shown in the figure, the nine behaviors collected in this experiment are (a) holding the steering wheel with both hands, (b) lowering the head, (c) dozing off, (d) being hit, (e) bending over, (f) leaning left and right, (g) taking both hands off the steering wheel, (h) leaning back, and (i) shaking the head left and right. In these experiments, there are two scenes: stationary and driving. Each action contains 400 frames of data, and each frame lasts for 100ms.

[0046] like Figure 3 As shown, the raw data is preprocessed to obtain a three-domain spectrogram, which includes a time-distance graph, a time-Doppler graph, and a speed rhythm graph of the driver's behavior.

[0047] The parameters of the preprocessing module are: the total number of range bins N, the total number of Doppler bins M, the Hamming window length L, and the sliding window length S are set to 256, 128, 64, and 32, respectively. The ROI is set to the 20th to 120th range bins to eliminate the near-ground effect.

[0048] The mean elimination method is used to filter out the interference of static clutter, and the three-domain spectrum is obtained by using two-dimensional fast Fourier transform and incoherent accumulation methods.

[0049] The method of using mean elimination to filter out the interference of static clutter includes:

[0050] The echo of static targets remains basically unchanged in multiple frames of data, while the echo of dynamic targets changes over time. By taking the average of the data of each distance unit in all frames and subtracting this average from the echo signal of each frame, the influence of static targets is removed, and only the dynamic target signals related to the detected target are retained.

[0051] The method using incoherent accumulation includes:

[0052] When the target speed changes greatly and the phase changes rapidly, incoherent accumulation can effectively enhance the strength of the target signal by accumulating the amplitude or power of multiple pulse echoes.

[0053] One-dimensional FFT is performed on the distance dimension and Doppler dimension of each frame of the preprocessed data to obtain a range-Doppler spectrum. The distance information and Doppler information are stacked on the time dimension to obtain a time-distance graph and a time-Doppler graph. One-dimensional FFT is performed on each frequency bin in the data matrix of the time-Doppler spectrum along the time axis to obtain a speed rhythm graph.

[0054] In order to obtain the three-domain spectrum of dangerous driving behavior, FFT is used on the original echo data to obtain the distance-Doppler spectrum of dangerous driving behavior; it can be obtained by the following formula:

[0055]

[0056] Where R[r,t] represents the time distance matrix, r and t represent the distance index and time index; N: represents the total number of samples; s[n,t] represents the original signal data of the nth sampling point at time index t; n represents the index of the sampling point, ranging from 0 to N-1; exp(-j2πrn / N) represents the complex exponential function used for Fourier transform, where j is the imaginary unit and 2π is a constant; RD[r,k,t] represents the matrix of distance, Doppler frequency and time; k: Doppler frequency index; L represents the total number of samples; M represents the sampling point index of the window function, ranging from 0 to L-1; R[r,tm]: the value of the time distance matrix, taking into account the time offset m; ω[m] represents the time window function, which weights different time points; exp(-j2πkm / L): a complex exponential function used to perform Doppler transform.

[0057] The time-distance matrix is ​​obtained by incoherent accumulation along the Doppler axis, and then a low-pass filter is used to filter out the redundant spectrum that does not belong to dangerous driving behavior; it can be obtained by the following formula:

[0058]

[0059] Where, TR[r,t] represents the time distance matrix; RD[r,i,t] represents the matrix of distance, Doppler frequency and time; r, i and t represent the distance, Doppler bin and time index respectively; M represents the total number of Doppler bins.

[0060] The time-Doppler matrix is ​​obtained by integrating the three-dimensional range-Doppler-time data within the region of interest (ROI); it can be obtained by the following formula:

[0061]

[0062] Where TD[k,t] is the time-Doppler matrix; RD[j,k,t] represents the matrix of range, Doppler frequency, and time; r, i, and t represent the range, Doppler bin, and time index, respectively; j∈ROI represents the index within the region of interest (ROI) when summing.

[0063] The rhythm speed graph is obtained by performing FFT on each frequency bin along the time axis, which can be obtained by the following formula:

[0064]

[0065] Among them, CVD[k,u] represents the rhythm velocity matrix, k and u represent the Doppler frequency and rhythm; F: represents the total number of frames; h represents the frame index, ranging from 0 to F-1; exp(-j2πhu / F) represents the complex exponential function used for Fourier transform, where j is the imaginary unit and 2π is a constant.

[0066] The three-domain spectra were divided into training sets and validation sets. In order to verify the recognition accuracy of the constructed neural network, 80% of the data were selected for model training, and the remaining 20% ​​were used as validation sets to test the model's recognition performance for each driver behavior.

[0067] like Figure 4 As shown, an autoencoder consisting of a convolutional neural network is built, and the training set is input into the autoencoder to obtain local features related to the driver's dynamic behavior;

[0068] The autoencoder is composed of two convolutional encoders: encoder I and encoder II; encoder I and encoder II both include an input layer, a convolutional layer, a batch normalization layer and an activation function layer;

[0069] The shape of the input layer is determined according to the structure of the three-domain spectrogram; each convolutional layer is created using the Conv2D function, specifying the number of filters and filter size of the convolutional layer, specifying the specific activation function of the activation function layer and the filling method of the convolutional layer; each convolutional layer is followed by a batch normalization layer to normalize the features of the output channel, and the output after the batch normalization layer is processed is imported into the activation function layer, and the above steps are repeated five times to construct a five-layer convolutional network.

[0070] The time or space dimension of local features is fused to obtain: time-distance-Doppler feature F t and time-distance-rhythm characteristics F s ;

[0071] Encoder I of the autoencoder is used to extract distance and Doppler semantic features from the time-distance graph (TR) and time-Doppler graph (TD) of the training set respectively; at the same time, encoder II of the autoencoder is used to extract rhythm features from the speed rhythm graph of the training set;

[0072] After feature extraction, the encoding features of the time-distance map (TR) and the time-Doppler map (TD) are concatenated according to the time dimension to obtain the time-distance-Doppler feature F t , the encoding features of the time-distance map (TR) and the time-Doppler map (TD) are concatenated according to the spatial dimension to obtain the time-distance-rhythm feature F s .

[0073] The first unit of encoder I is MobileNetBlock1, which includes 16 filters, the convolution kernel size is 8x4, the stride is 4x2, the padding is 2x1, the activation function (ReLU) is used, and the batch normalization layer contains 8 channels.

[0074] The second unit is MobileNetBlock2, which includes 32 filters, a convolution kernel size of 8x4, a stride of 4x2, a padding of 2x1, an activation function (ReLU), and a batch normalization layer containing 16 channels.

[0075] The third unit is MobileNetBlock3, which includes 48 filters, a convolution kernel size of 4x3, a stride of 2x1, a padding of 1x1, an activation function (ReLU, and a batch normalization layer containing 32 channels.

[0076] The fourth unit is MobileNetBlock4, which includes 64 filters, a convolution kernel size of 4x3, a stride of 1x1, a padding of 0x1, an activation function (ReLU), and a batch normalization layer containing 64 channels.

[0077] The fifth unit is MobileNetBlock5, which includes 64 filters, a convolution kernel size of 4x3, a stride of 1x1, a padding of 0x1, an activation function (ReLU), and a batch normalization layer containing 64 channels.

[0078] The first unit of encoder II is MobileNetBlock1, which includes 8 filters, the convolution kernel size is 8x8, the stride is 4x4, the padding is 2x2, the activation function (ReLU) is used, and the batch normalization layer contains 8 channels.

[0079] The second unit is MobileNetBlock2, which includes 16 filters, a convolution kernel size of 8x8, a stride of 4x4, a padding of 2x2, an activation function (ReLU), and a batch normalization layer containing 16 channels.

[0080] The third unit is MobileNetBlock3, which includes 32 filters, a convolution kernel size of 4x4, a stride of 2x2, a padding of 1x1, an activation function (ReLU), and a batch normalization layer containing 32 channels.

[0081] The fourth unit is MobileNetBlock44, which includes 64 filters, a convolution kernel size of 4x4, a stride of 1x1, a padding of 0x1, an activation function (ReLU), and a batch normalization layer containing 64 channels.

[0082] The fifth unit is MobileNetBlock5, which includes 64 filters, a convolution kernel size of 4x4, a stride of 1x1, a padding of 0x1, an activation function (ReLU), and a batch normalization layer containing 64 channels.

[0083] The coding features of TR / TD are spliced ​​according to the time dimension. Both TR and TD features have changes in the time dimension. Through splicing in the time dimension, the consistency of time information can be retained and the modeling ability of time series data can be improved. Secondly, the coding features of TR / CVD are spliced ​​according to the spatial dimension, which can enrich the expression of spatial information. Through splicing in the spatial dimension, the spatial interaction of cross-domain features can be enhanced to improve the modeling ability of geographic spatial data. The feature fusion of the first stage can be expressed as:

[0084]

[0085] Among them, H TR , H TD , H CVD Represent the encoding features of the three domains respectively; F t 、F s Represent the temporal fusion features and spatial fusion features respectively; and Represents a join operation along the time dimension and along the space dimension.

[0086] like Figure 6 As shown in the figure, the Transformer module is constructed using position encoding, layer normalization, multi-head attention mechanism and multi-layer perceptron, and the Transformer network is used to extract the global features of the driver's dynamic behavior. Combining local features with global features can fully extract the characteristic information of the driver's dynamic behavior. In addition, the multi-head attention (MHA) mechanism of the transformer network can assign different weights to different feature vectors, thereby obtaining effective driver dynamic behavior features.

[0087] Transformer and CNN are used to build a branch network to transform the time-distance-Doppler feature F t and time-distance-rhythm characteristics F s Input them into the branch networks respectively to obtain high-dimensional feature vectors;

[0088] The abstract representation extracted by the Transformer module is constructed using layer normalization, multi-head attention mechanism and multi-layer perceptron to process the time-distance-Doppler feature F. t Get deeper feature Fcnn; use flattening layer, convolution layer, activation function, batch normalization, maximum pooling and output layer to build the abstract representation extracted by 1D-CNN module to process time-distance-rhythm feature F s Get deeper features Ftran; use a trainable weighted average strategy to fuse the final feature vector F of the input fully connected layer fuse .

[0089] Two trainable parameters α and β are added to the high-dimensional feature vector, where α and β represent the contribution of the classification performance of the two branch networks respectively. A trainable weighted average strategy is used to fuse the two deep-level driver dynamic behavior features output by the branch networks according to α and β to obtain the final feature vector of the driver dynamic behavior features.

[0090] The final feature vector of 64 dimensions is mapped to 9 dimensions through a fully connected layer (FC), and the driver behavior label is output to complete the identification of dangerous driving behavior.

[0091] The position encoding in the Transformer module can be expressed as:

[0092]

[0093] Where pos represents the index of the position in the input sequence; i represents the even or odd position index in the feature dimension, corresponding to the sine and cosine functions respectively; d modelInput feature dimension (e.g., 128), which determines the period of sine and cosine functions, ensuring that the encoded values ​​at different positions are sufficiently distinguishable in high-dimensional space.

[0094] Layer normalization can be expressed as:

[0095]

[0096] Where x represents the input tensor to be normalized; μ represents the mean of the input samples. σ: standard deviation of the input samples. γ and β represent learnable scaling and offset parameters;

[0097] The multi-head attention (MHA) mechanism can be expressed as:

[0098]

[0099] Among them, Q, K, and V are query matrix, key matrix, and value matrix respectively; is the learnable weight matrix of the i-th head; W O The final linear transformation output matrix obtained by calculating attention; d k is a scaling factor used to prevent the gradient from vanishing or exploding due to the dot product being too large; i represents the number of attention heads, i∈(1,2,...,n).

[0100] like Figure 5 As shown, the 1D-CNN module is constructed using convolutional layers, activation functions, maximum pooling layers, and flattening layers;

[0101] The first unit is MobileNetBlock1, which includes 16 filters, a convolution kernel size of 4, a stride of 2, a padding of 1, and an activation function (ReLU); the pooling layer has a convolution kernel size of 2 and a stride of 2.

[0102] The second unit is MobileNetBlock2, which includes 32 filters, a convolution kernel size of 4, a stride of 2, a padding of 1, and an activation function (ReLU); the pooling layer has a convolution kernel size of 4, a stride of 4, and a padding of 1.

[0103] The third unit is MobileNetBlock3, which includes 64 filters, a convolution kernel size of 4, a stride of 1, a padding of 0, and an activation function (ReLU).

[0104] like Figure 7As shown in the figure, in the first stage of the two-stage network model, two autoencoders composed of convolutional neural networks are built, and a five-layer autoencoder I is used to extract distance and Doppler semantic information from TR and TD of the training set respectively. At the same time, another autoencoder II is used to extract Doppler velocity rhythm related information from CVD of the training set. After the feature extraction is completed, the encoding features of TR / TD are spliced ​​according to the time dimension, and the encoding features of TR / CVD are spliced ​​according to the spatial dimension to obtain time fusion features and spatial fusion features. In the second stage, the 1D-CNN module is used to process the time-distance-rhythm feature (F s ); Transformer module is used to process the input time-range-Doppler features (F t );

[0105] The fusion process using the trainable weighted average strategy is as follows: for features F extracted from two different branches t 、F s , the branch network adjusts the values ​​of α and β according to the loss function during the training process, so that these two parameters can effectively allocate the feature weights of the two branches; the second stage fusion strategy can be expressed as:

[0106] F fuse =α×F t +β×F s

[0107] where F t 、F s and F fuse Represent the high representation outputs of the two branches and the second stage fusion features, respectively.

[0108] Finally, the high representation output of the two-stage fusion features is passed through a fully connected (FC) layer to finally obtain the driver behavior probability vector and make a recognition decision.

[0109] Based on the above technical solution, in order to verify the effect of the method of the present invention, this embodiment is experimentally verified, as follows:

[0110] AWR843B00ST millimeter-wave radar is used in conjunction with a DCA1000 high-speed data acquisition card to collect radar echoes of driver behavior and construct an echo data set. The radar echoes are transmitted to the PC through the data acquisition card and the data is processed using MATLAB2024 software. The CNN-Transformer network is based on the PyTorch deep learning framework and is trained on an Inteli5-13500HX processor and an NVIDAGTX4050 graphics card.

[0111] A dangerous driving behavior identification device, comprising

[0112] An acquisition module is used to acquire raw data, wherein the raw data is echo data of different driving behaviors of the driver in the vehicle;

[0113] A preprocessing module, used for preprocessing the raw data to obtain a three-domain spectrogram, wherein the three-domain spectrogram includes a time-distance graph, a time-Doppler graph, and a speed rhythm graph of the driver's behavior;

[0114] A partitioning module, used to divide the three-domain spectra into a training set and a validation set;

[0115] The first stage processing module is used to build an autoencoder composed of a convolutional neural network, input the training set into the autoencoder, and obtain local features related to the driver's dynamic behavior;

[0116] The fusion module is used to fuse the time or space dimensions of local features to obtain: time-distance-Doppler feature F t and time-distance-rhythm characteristics F s ;

[0117] The second stage processing module is used to build a branch network using Transformer and CNN to transform the time-distance-Doppler feature F t and time-distance-rhythm characteristics F s Input them into the branch networks respectively to obtain high-dimensional feature vectors;

[0118] The feature processing output module is used to add two trainable parameters α and β to the high-dimensional feature vector, where α and β represent the contribution of the classification performance of the two branch networks respectively, and adopt a trainable weighted average strategy to fuse the two deep-level driver dynamic behavior features output by the branch network according to α and β to obtain the final feature vector of the driver dynamic behavior feature;

[0119] The output module is used to map the final feature vector of 64 dimensions to 9 dimensions through a fully connected layer (FC), output the driver behavior label, and complete the identification of dangerous driving behavior.

[0120] like Fig. 9 and Fig.10As shown in the figure, the t-SNE (t-distributed Stochastic Neighbor Embedding) algorithm is used to demonstrate the classification effect of the model on the validation set. t-SNE is a nonlinear dimensionality reduction technique used to map high-dimensional data to a low-dimensional space for visualization. The mapping from high dimension to low dimension is achieved through a two-step process: first, the conditional probability between data points is calculated using a Gaussian distribution in a high-dimensional space, then the similarity is calculated using a t-distribution in a low-dimensional space, and the position of the data points is optimized by minimizing the Kullback-Leibler divergence (KL divergence) between the two distributions. It can be seen that the dynamic behaviors of the nine types of drivers have clear dividing lines, and different actions can be identified.

[0121] like Fig.11 As shown in the figure, the training process of the model is demonstrated. According to the accuracy of each round in the dynamic training process of the model, the accuracy change curves of the training set and the validation set are drawn. The blue one is the accuracy change curve of the training set, and the red one is the accuracy change curve of the validation set.

[0122] like Fig.12 As shown, the confusion matrix of the test results of the CNN-Transformer network on the test set is given. The confusion matrix is ​​calculated by comparing the position and classification of each measured pixel with the corresponding position and classification in the classified image. In the confusion matrix, each row represents the actual category and each column represents the predicted category. In this way, evaluation indicators such as overall accuracy, mapping accuracy and user accuracy are calculated, thereby reflecting the accuracy of image classification from different aspects. In a mixed data set of moving and stationary environments, the method of the present invention can achieve a recognition accuracy of more than 98% for each driver behavior, especially for being impacted, dozing off and bending over, all of which have a recognition accuracy of 100%.

[0123] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present invention (including the claims) is limited to these examples. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity.

[0124] The present invention is intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for identifying dangerous driving behavior, characterized in that: include Acquiring original data, wherein the original data is echo data of different driving behaviors of the driver in the vehicle; Preprocessing the raw data to obtain a three-domain spectrogram, wherein the three-domain spectrogram includes a time-distance graph, a time-Doppler graph, and a speed rhythm graph of the driver's behavior; Divide the three-domain spectra into training sets and validation sets; Build an autoencoder consisting of a convolutional neural network, input the training set into the autoencoder, and obtain local features related to the driver's dynamic behavior; The time or space dimension of local features is fused to obtain: time-distance-Doppler feature F t and time-distance-rhythm characteristics F s ; Transformer and CNN are used to build a branch network to transform the time-distance-Doppler feature F t and time-distance-rhythm characteristics F s Input them into the branch networks respectively to obtain high-dimensional feature vectors; Two trainable parameters α and β are added to the high-dimensional feature vector, where α and β represent the contribution of the classification performance of the two branch networks respectively. A trainable weighted average strategy is used to fuse the two deep-level driver dynamic behavior features output by the branch networks according to α and β to obtain the final feature vector of the driver dynamic behavior features. The final feature vector is mapped to 9 dimensions through a fully connected layer (FC), and the driver behavior label is output to complete the identification of dangerous driving behaviors and obtain the initial model; Input the validation set data into the initial model for validation; If the initial model converges, then the initial model is the required network model; if it does not converge, modify the values ​​of the trainable parameters α and β; re-input the validation set data into the initial model for verification; and repeat this process until the initial model converges to obtain the required network model.

2. The dangerous driving behavior identification method according to claim 1, characterized in that: The obtaining of raw data includes: Use millimeter wave radar and data acquisition card to collect raw echo data of driver behavior.

3. The dangerous driving behavior identification method according to claim 1, characterized in that: Said The raw data were preprocessed to obtain three-domain spectra, including The mean elimination method is used to filter out the interference of static clutter, and the three-domain spectrum is obtained by using two-dimensional fast Fourier transform and incoherent accumulation methods.

4. The dangerous driving behavior identification method according to claim 3, characterized in that: Said The mean elimination method is used to filter out the interference of static clutter, including The echo of static targets remains basically unchanged in multiple frames of data, while the echo of dynamic targets changes over time. By taking the average of the data of each distance unit in all frames and subtracting this average from the echo signal of each frame, the influence of static targets is removed, and only the dynamic target signals related to the detected target are retained.

5. The dangerous driving behavior identification method according to claim 1, characterized in that: The method using incoherent accumulation includes: When the target speed changes greatly and the phase changes rapidly, the amplitude or power of multiple pulse echoes is accumulated. The dangerous driving behavior identification method according to claim 1, characterized in that obtaining the three-domain spectrogram comprises: One-dimensional FFT is performed on the distance dimension and Doppler dimension of each frame of the preprocessed data to obtain a range-Doppler spectrum. The distance information and Doppler information are stacked on the time dimension to obtain a time-distance graph and a time-Doppler graph. One-dimensional FFT is performed on each frequency bin in the data matrix of the time-Doppler spectrum along the time axis to obtain a speed rhythm graph.

6. The dangerous driving behavior identification method according to claim 1, characterized in that: The autoencoder composed of the convolutional neural network is constructed, and the training set is input into the autoencoder to obtain local features related to the driver's dynamic behavior, including The autoencoder is composed of two convolutional encoders: encoder I and encoder II; encoder I and encoder II both include an input layer, a convolutional layer, a batch normalization layer and an activation function layer; The shape of the input layer is determined according to the structure of the three-domain spectrogram; each convolutional layer is created using the Conv2D function, specifying the number of filters and filter size of the convolutional layer, specifying the specific activation function of the activation function layer and the filling method of the convolutional layer; each convolutional layer is followed by a batch normalization layer to normalize the features of the output channel, and the output after the batch normalization layer is processed is imported into the activation function layer, and the above steps are repeated five times to construct a five-layer convolutional network.

7. The dangerous driving behavior identification method according to claim 1, characterized in that: The fusion of the temporal or spatial dimensions of local features includes Encoder I of the autoencoder is used to extract distance and Doppler semantic features from the time-distance graph (TR) and time-Doppler graph (TD) of the training set respectively; at the same time, encoder II of the autoencoder is used to extract rhythm features from the speed rhythm graph of the training set; After feature extraction, the encoding features of the time-distance map (TR) and the time-Doppler map (TD) are concatenated according to the time dimension to obtain the time-distance-Doppler feature F t , the encoding features of the time-distance map (TR) and the time-Doppler map (TD) are concatenated according to the spatial dimension to obtain the time-distance-rhythm feature F s .

8. The dangerous driving behavior identification method according to claim 1, characterized in that: The branch network is constructed by using Transformer and CNN, including The abstract representation extracted by the Transformer module is constructed using layer normalization, multi-head attention mechanism and multi-layer perceptron to process the time-distance-Doppler feature F. t Get deeper feature Fcnn; use flattening layer, convolution layer, activation function, batch normalization, maximum pooling and output layer to build the abstract representation extracted by 1D-CNN module to process time-distance-rhythm feature F s Get deeper features Ftran; The final feature vector F of the input fully connected layer is obtained by using a trainable weighted average strategy. fuse .

9. A dangerous driving behavior identification device, characterized in that: include An acquisition module is used to acquire raw data, wherein the raw data is echo data of different driving behaviors of the driver in the vehicle; A preprocessing module, used for preprocessing the raw data to obtain a three-domain spectrogram, wherein the three-domain spectrogram includes a time-distance graph, a time-Doppler graph, and a speed rhythm graph of the driver's behavior; A partitioning module, used to divide the three-domain spectra into a training set and a validation set; The first stage processing module is used to build an autoencoder composed of a convolutional neural network, input the training set into the autoencoder, and obtain local features related to the driver's dynamic behavior; The fusion module is used to fuse the time or space dimensions of local features to obtain: time-distance-Doppler feature F t and time-distance-rhythm characteristics F s ; The second stage processing module is used to build a branch network using Transformer and CNN to transform the time-distance-Doppler feature F t and time-distance-rhythm characteristics F s Input them into the branch networks respectively to obtain high-dimensional feature vectors; The feature processing output module is used to add two trainable parameters α and β to the high-dimensional feature vector, where α and β represent the contribution of the classification performance of the two branch networks respectively, and adopt a trainable weighted average strategy to fuse the two deep-level driver dynamic behavior features output by the branch network according to α and β to obtain the final feature vector of the driver dynamic behavior feature; The output module is used to map the final feature vector of 64 dimensions to 9 dimensions through a fully connected layer (FC), output the driver behavior label, and complete the identification of dangerous driving behavior.

Images