A method, device and storage medium for identifying distracted driving behavior

By acquiring and fusing temporal and spatial features to identify distracted driving behavior, and combining datasets of different drivers with a lightweight network structure, the method solves the problems of model complexity and adaptability in existing technologies, achieving high-precision distracted driving recognition and improved safety.

CN114529889BActive Publication Date: 2024-11-15YANSHAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210109527.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-28
Publication Date
2024-11-15
Estimated Expiration
2042-01-28

AI Technical Summary

Technical Problem

Existing distracted driving behavior recognition algorithms are complex, difficult to embed, and lack time-dimensional and spatiotemporal information fusion, resulting in low recognition accuracy and inability to adapt to different drivers, increasing the risk of misoperation.

Method used

A distracted driving behavior recognition method based on time-series image data is adopted. By fusing temporal and spatial features, the model is trained by combining multiple driving behavior datasets of different drivers. Then, knowledge distillation technology is used to transfer lightweight network structures, such as MobileNetv3-GRU and GhostNet-GRU, for feature fusion and early warning.

Benefits of technology

It achieves accurate identification of distracted driving behavior, reduces the risk of accidental takeover, improves driving safety, enhances model generalization ability and hardware resource utilization efficiency, and reduces computing costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114529889B_ABST
    Figure CN114529889B_ABST
Patent Text Reader

Abstract

The present invention relates to a method for identifying distracted driving behavior, the method comprising the following steps: obtaining time-series image data of a driver's distracted driving in a human-machine co-driving takeover state; first obtaining spatial features of the time-series image data using a trained first model or a trained second model, and obtaining spatiotemporal features based on the spatial features; fusing and strengthening the spatiotemporal features with the spatial features, and then obtaining distracted driving behavior features for identifying the category of distracted driving behavior. The method can accurately identify the driver's distracted driving behavior, reduce the risk of mis-takeover, and thus improve driving safety.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation, and in particular to a distracted driving behavior recognition method, device and storage medium. Background Art

[0002] The automation of cars is an inevitable trend, and some car manufacturers have already provided some autonomous driving functions. However, due to the current technical bottlenecks and the complexity of real-life scenarios and driving tasks, autonomous cars will be at the L3-L4 stage for a period of time, requiring the driver to participate in the driving task. At this stage, the switching of control between people and cars will become the norm.

[0003] When an autonomous vehicle issues a takeover request, the driver may have various distracted driving behaviors, which will occupy the driver's visual resources, auditory resources, cognitive resources and action resources to varying degrees, affecting the driver's perception, judgment, decision-making and operation process, thus leading to takeover out of control and increasing the risk of traffic accidents. Therefore, identifying the driver's distracted driving behavior can reduce the risk of erroneous takeover and thus improve driving safety.

[0004] Existing distraction recognition algorithms have complex models that cannot be well implemented in embedded development, and some algorithms lack information in the time dimension or fail to fully integrate time and space information, resulting in confusion in the recognition of distracted driving behaviors with similar characteristics. At the same time, some models do not consider the impact of different drivers during training, resulting in a sharp drop in recognition accuracy when changing drivers. Summary of the invention

[0005] In view of this, the main purpose of this application is to use analysis features including both time features and spatial features when identifying distracted driving behavior, so as to accurately identify the driver's analytical driving behavior, reduce the risk of mistaking, and thus improve driving safety.

[0006] Based on the above purpose, the present invention provides a method for identifying distracted driving behavior, the method comprising the following steps:

[0007] S100, obtaining time-series image data of a driver's distracted driving in a human-machine co-driving takeover state;

[0008] S200, using the trained first model or the trained second model to obtain spatial features based on the time series image data, and obtaining spatiotemporal features based on the spatial features;

[0009] S300: Fusing and strengthening the spatiotemporal features with the spatial features, and thereby obtaining distracted driving behavior features for identifying distracted driving behavior categories.

[0010] Preferably, in the method, the data set used in training the first model or the second model is obtained by the following steps:

[0011] S101, obtaining a distraction dataset containing various driving behaviors of different drivers;

[0012] S102, for the same distracted driving action image data of the same driver, obtaining a plurality of images with continuous context as a distracted driving action sequence;

[0013] S103, obtaining the type of each analyzed driving action sequence, and performing label processing on the type, thereby obtaining an evaluation data set of the first model or the second model, and dividing the evaluation data set into a training set and a test set;

[0014] The training set is used to train the first model or the second model, and the test set is used to verify the first model or the second model.

[0015] Preferably, in the method, the second model obtains knowledge of the first model through knowledge distillation.

[0016] Preferably, in the method, the second model obtains the knowledge of the first model through knowledge distillation, comprising the following steps:

[0017] S110, inputting the training set into the trained first model, and performing classification processing on the output after heating it to T to obtain a first output result; wherein T is greater than 0;

[0018] S111, inputting the same training set into the second model, and outputting the training set as the second output result;

[0019] S112, raising the temperature of the second model output result by T and performing classification processing to obtain a third output result;

[0020] S113, performing cross entropy calculation on the second output result and the actual label to obtain a first loss loss1, and performing KL divergence calculation on the first output result and the third output result to obtain a second loss loss2;

[0021] S114. Calculate the total loss function as:

[0022] loss 总 =λloss1+(1-λ)loss2, 0<λ<1

[0023] S115. When the total loss function value meets a threshold condition, use the second model for distracted driving behavior category identification.

[0024] Preferably, in the method, the first model is a network structure consisting of a cascade of MobileNetv3 and GRU;

[0025] The second model is a network structure consisting of a cascade of GhostNet and GRU.

[0026] Preferably, in the method, the distracted driving action sequence includes at least 6 images.

[0027] Preferably, in the method, the evaluation data set further includes the following processing: processing the evaluation data set by Gaussian noise and performing an amplification process on the training set therein.

[0028] Preferably, in the method, the method further comprises the following steps:

[0029] Based on the type of distracted driving identified, an early warning will be issued when the driver needs to take over;

[0030] The warning includes a warning with sound prompts and a dual warning with sound and steering wheel vibration.

[0031] Based on the above method, the present invention also proposes a distracted driving behavior identification device, including a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and execute any of the above methods.

[0032] Furthermore, the present invention also proposes a computer-readable storage medium, characterized in that: a computer program capable of being loaded by a processor and executing any of the above methods is stored.

[0033] Compared with the prior art, the method of the present invention has the following beneficial effects:

[0034] (1) By acquiring time features and spatial features from time-series image data and fusing the two features, distracted driving behavior features are obtained; the distracted driving behavior features can be used to accurately identify the driver's analytical driving behavior, reduce the risk of mis-takeover, and thus improve driving safety.

[0035] (2) By using distraction datasets of various driving behaviors of different drivers to train and verify the model, the recognition accuracy of the model can be improved, and further Gaussian processing and amplification processing can enhance the generalization ability of the model.

[0036] (3) By adopting knowledge distillation technology, the knowledge of the first model is transferred to the second model, which can make the second model have the characteristics of simplified parameters and higher accuracy, convenient for use on resource-constrained devices, and helpful for the implementation of embedded development.

[0037] (4) The network structures used in the first model and the second model are both lightweight. When implementing the method of the present invention, while maintaining recognition accuracy, the demand for hardware resources can be further reduced and the computing cost can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0039] Figure 1 , a schematic diagram of the process of the present invention;

[0040] Figure 2 ,Schematic diagram of training using knowledge distillation;

[0041] Figure 3 , Schematic diagram of distraction recognition model of MobileNetV3-GRU or GhostNet-GRU cascade network. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] In one embodiment, a method for identifying the driver's distracted driving behavior is used to reduce the risk of mis-taking over, thereby improving driving safety. The method includes the following steps: Figure 1 As shown:

[0044] S100, obtaining time-series image data of a driver's distracted driving in a human-machine co-driving takeover state;

[0045] S200, using the trained first model or the trained second model to obtain spatial features based on the time series image data, and obtaining spatiotemporal features based on the spatial features;

[0046] S300: Fusing and strengthening the spatiotemporal features with the spatial features, and thereby obtaining distracted driving behavior features for identifying distracted driving behavior categories.

[0047] Furthermore, based on the identified type of distracted driving, an early warning is issued when the driver needs to take over; the early warning includes a warning with a sound prompt, and a dual warning with a sound and steering wheel vibration.

[0048] Before applying the above method, the first model or the second model needs to be trained. The image data source used for model training can be the Kaggle distraction dataset released by State Farm in 2016. The distraction dataset contains 10 types of driving behaviors, the training set contains 22,424 labeled distraction behavior images, the test set contains 79,728 unlabeled distraction behavior images, and the training set and the test set contain different drivers.

[0049] Next, the State Farm dataset is transformed into a time series. Specifically, the dataset is classified according to the driver, and then the driving action of each driver is classified. When classifying the driving action of each driver, several images with continuous context are intercepted as a distracted driving action sequence, and the category of the analyzed driving behavior corresponding to the distracted driving action sequence is labeled. When the last action sequence is not enough to completely constitute a distracted driving action sequence, the remaining distracted images that are not enough to constitute a complete sequence are discarded. The number of the several images is at least 6, for example, it can be 6, 8, 10, etc., and the specific number is determined according to actual needs. Through this processing, a total of 3617 distracted driving action sequences of 10 categories are obtained, and then the dataset is divided into a training set and a test set according to 8:2. The training set is used to train the first model or the second model, and the test set is used to verify the first model or the second model.

[0050] Considering the changes of drivers in real life, which may affect the accuracy of the model, in order to evaluate the generalization ability of the model for different drivers, the same driver is only placed in the training set or test set, among which 749 distraction action sequences of 6 drivers are used in the test set, and 2868 distraction action sequences of 20 drivers are used in the training set.

[0051] In order to further improve the generalization ability of the model, the images in the training set are processed by Gaussian noise, and the training set is expanded by 1 times, 2 times, etc. When expanded by 1 times, 5736 distracted driving action sequences are obtained in the training set and 749 in the test set.

[0052] The trained first model and the second model can both be used for distracted driving behavior category recognition. They obtain distracted driving behavior characteristics by acquiring time features and spatial features from time-series image data and fusing the two features. The distracted driving behavior characteristics can accurately identify the driver's analytical driving behavior, reduce the risk of mis-takeover, and thus improve driving safety.

[0053] Furthermore, by using knowledge distillation technology, the knowledge of the first model is transferred to the second model, and the second model is used for distracted driving behavior category recognition. This not only makes the second model more parameter-simplified than the first model, but also has higher accuracy, making it easier to use on resource-constrained devices and helping to implement embedded development. The network structure diagram when using knowledge distillation technology for training is shown in the figure below: Figure 2 As shown, the specific training process is as follows:

[0054] S110, input the training set into the trained first model, and use the softmax function to process its output after heating it by T1 to obtain the first output result Soft labels;

[0055] S111, input the same training set into the second model, and directly process its output using the softmax function to obtain the second output result Hard prediction;

[0056] S112, the output result of the second model is heated up by T1 and then processed using the softmax function to obtain a third output result Soft predictions;

[0057] S113, performing cross entropy calculation on the second output result Hard prediction and the actual label Hard labels to obtain the first loss loss1;

[0058] Perform KL divergence calculation on the first output result and the third output result to obtain the second loss loss2;

[0059] S114. Calculate the total loss function as:

[0060] loss 总 =λloss1+(1-λ)loss2

[0061] Where: λ∈(0,1);

[0062] S115. When the total loss function meets a threshold condition, the second model is a model for identifying types of distracted driving.

[0063] In the above process, the first loss loss1 means that the second model also needs to learn to generate correct labels. That is to say, the second model must learn the generalization ability of the first model and make error correction adjustments to the first model for hard targets.

[0064] The first model provides the dark knowledge guidance required for classification, so that the second model can not only learn the hard knowledge of the true label but also learn the soft knowledge of the relationship between positive and negative examples. The learning of hard knowledge and soft knowledge can, to a certain extent, enable the second model to break through the bottleneck of training and achieve good accuracy, providing algorithm support for embedded distracted driving recognition.

[0065] The classification process with temperature T is implemented by the softmax function, as shown in the following formula:

[0066]

[0067] Where: z k is the output value of the kth neuron in the fully connected layer, k = 0, 1, ..., n-1, n is the number of categories, and T1 is the distillation temperature.

[0068] The calculation formula for the first loss loss1 is as follows:

[0069]

[0070] The second loss loss2 is calculated as follows:

[0071]

[0072] In the above two loss calculation formulas, N is the number of classification types. for The value of the ith position of for, is the result after y′ is processed by the softmax function with temperature T1, and y′ is the output result of the first model. for The value of the ith position, y i is the value of the i-th position after y is processed by the softmax function, for The value of the i-th position after being processed by the softmax function. is the output of the second model. y is the one-hot encoding of the true label.

[0073] Depend on Figure 2 It can be seen that when knowledge distillation technology is applied in the model training stage, the model training is guided not only by the real labels, but also by the labels softened by the softmax function with temperature T1 provided by the trained first model. This enables the second model to exceed the accuracy of the real labels alone. The soft labels provided by the first model, to a certain extent, contain information about the similarity between positive and negative examples. In this way, the accuracy of the second model can be effectively improved.

[0074] If the network structures used in the first model and the second model are both lightweight, the demand for hardware resources can be reduced during calculation, thereby reducing hardware costs. However, in order to achieve high recognition accuracy, it is preferred that the first model adopts a network structure consisting of MobileNetv3 and GRU cascade; the second model adopts a network structure consisting of GhostNet and GRU cascade, such as Figure 3 shown.

[0075] Combine the following Figure 3 The network structure composed of MobileNetv3 and GRU cascade is shown to further describe this method.

[0076] MobileNetv3 uses deep separable convolution to extract spatial features from distracted driving time series image data. Deep separable convolution consists of deep convolution and point convolution. From the following computational analysis, it can be seen that MobileNetv3 has less computational complexity than traditional convolutional neural networks and is a lightweight network.

[0077] Given a D F ×D F ×M feature map, generating a D F ×D F ×N feature map, where D F is the width and height of the feature map, M and N are the number of channels of the feature map.

[0078] (i) The normal convolution kernel size is D K ×D K ×M×N,D K is the width and height of the convolution kernel, then the amount of calculation is:

[0079] 2(D K ×D K ×M)×N×D F ×D F

[0080] (ii) The depthwise convolution size is D K ×D K ×M, the amount of calculation is:

[0081] 2(M×D K ×D K )×D F ×D F

[0082] (iii) The size of the point convolution is M×1×1×N, and the computational effort is:

[0083] 2×M×N×D F ×D F

[0084] Therefore, it can be concluded that the ratio of the computational amount of depth-separable convolution to the computational amount of traditional convolution is as follows:

[0085]

[0086] Table 1 shows the network structure of MobileNetV3. In the table, RE represents the ReLU activation function, HS is the Hard-switch activation function, SEBneck is the Bottleneck module in SE-Net, the step size represents the step size of the depthwise separable convolution, and NBN means no batch normalization.

[0087] Table 1

[0088]

[0089]

[0090] As can be seen from Table 1, different activation functions are used at both ends of the Bottleneck structure, which enables the first model to improve the nonlinear fitting ability of the driving behavior recognition model.

[0091] MobileNetv3 outputs the extracted spatial features to GRU. GRU is a type of recurrent neural network that can mine temporal and semantic information in data and process data information with sequence characteristics. GRU mines temporal features based on spatial features and outputs spatiotemporal features. GRU can maintain long-term memory, but cannot fully integrate the periodic information of all sequence frames, especially early image frames. Therefore, it is necessary to fuse the spatial features extracted by MobileNetV3 with the spatiotemporal features extracted by GRU to make up for the information loss of GRU.

[0092] like Figure 3 As shown in the figure, the number of MobileNetV3 modules in the first model is equal to the number of distracted driving action sequences. Each MobileNetV3 module extracts the spatial features of an image in the distracted driving action sequence. Therefore, before fusing the spatial features with the spatiotemporal features extracted by GRU, the following linear operation can be used to fuse the spatial features:

[0093] x s =f(x)

[0094] where x=(x 0 , ..., x N-1 ) is the image feature of N frames before linear fusion, x t is the image spatial feature at time t, t = 0, 1, ..., N-1, f is the linear operation, x s is the fused image feature.

[0095] The spatial features x that will be fused into a frame s And the spatiotemporal features x extracted by GRU G Fusion is performed to obtain distracted driving behavior characteristics x F :

[0096] x F =x S +x G

[0097] In summary, the first model composed of MobileNetV3 and GRU cascade can combine the spatial feature extraction advantages of MobileNetV3 with the temporal feature information mining capabilities of GRU. Further, by adopting the feature linear fusion strategy, the front-end deep spatial features and the back-end temporal features with spatial features are fused and enhanced to obtain more effective distracted driving behavior features and improve the model's accuracy in identifying distracted driving behavior categories.

[0098] It has been verified that the first model composed of MobileNetV3 and GRU cascade can deeply analyze similar distracted driving behavior characteristics and achieve good recognition of distracted behaviors with high recognition similarity. MobileNetv3, MobileNet-RNN, and MobileNet-LSTM are used to identify the same distracted driving behavior category, and the recognition results are shown in Table 2:

[0099] Table 2

[0100] Model Accuracy MobileNetV3 83.24% MobileNetV3-RNN 88.25% MobileNetV3-LSTM 90.39% MobileNetV3-GRU 90.92%

[0101] It can be seen from Table 2 that the accuracy of the first model composed of MobileNetV3 and GRU cascade is relatively high.

[0102] In the second model, GhostNet was used to replace MobileNetv3. After the replacement, the model parameters dropped by 0.2M, further reducing the calculation parameters, and the FLOPs dropped by 78M times, further improving the calculation speed. The Ghost module in GhostNet can generate another feature map from one feature map through linear transformation, using fewer parameters to generate the same number of feature maps as ordinary convolutional layers, and the required computing resources are lower than ordinary convolutional layers. It can be integrated into the existing designed neural network structure to reduce the computing cost.

[0103] Through the description of the above implementation modes, those skilled in the art can clearly understand that the method of the present invention can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits, etc. However, for the present disclosure, software program implementation is a better implementation mode in most cases.

[0104] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields, and the above specific embodiments are only illustrative and instructive, rather than restrictive. A person of ordinary skill in the art can also make many forms under the guidance of this specification and without departing from the scope of protection of the claims of the present invention, all of which belong to the protection of the present invention.

Claims

1. A method for identifying distracted driving behavior, characterized in that: The method comprises the following steps: S100, obtaining time-series image data of a driver's distracted driving in a human-machine co-driving takeover state; S200, using the trained second model to obtain spatial features based on the time series image data, and obtaining spatiotemporal features based on the spatial features, as follows: By adopting knowledge distillation technology, the knowledge of the first model is transferred to the second model, the second model is a network structure composed of GhostNet and GRU cascade, the first model is a network structure composed of MobileNetv3 and GRU cascade, the number of MobileNetv3 or GhostNet modules is the same as the number of distracted driving time series images, and is used to extract spatial features of each frame of the image. Based on the spatial features, GRU extracts spatiotemporal features; S300, fusing and strengthening the spatiotemporal features with the spatial features to obtain distracted driving behavior features for distracted driving behavior category identification; for N frames of distracted driving time series images, the fusion enhancement is to fuse the N spatial features using linear operations, and then fuse the linearly fused spatial features with the spatiotemporal features to obtain distracted driving behavior features.

2. The method according to claim 1, characterized in that The data set used in training the first model or the second model is obtained by the following steps: S101, obtaining a distraction dataset containing various driving behaviors of different drivers; S102, for the same distracted driving action image data of the same driver, obtaining a plurality of images with continuous context as a distracted driving action sequence; S103, obtaining the type of each analyzed driving action sequence, and performing label processing on the type, thereby obtaining an evaluation data set of the first model or the second model, and dividing the evaluation data set into a training set and a test set; The training set is used to train the first model or the second model, and the test set is used to verify the first model or the second model.

3. The method according to claim 1, characterized in that The second model obtains the knowledge of the first model through knowledge distillation, including the following steps: S110, inputting the training set into the trained first model, and performing classification processing on the output after heating it to T to obtain a first output result; wherein T is greater than 0; S111, inputting the same training set into the second model, and outputting the training set as the second output result; S112, raising the temperature of the second model output result by T and performing classification processing to obtain a third output result; S113, performing cross entropy calculation on the second output result and the actual label to obtain a first loss loss1, and performing KL divergence calculation on the first output result and the third output result to obtain a second loss loss2; S114. Calculate the total loss function as: loss 总 =λloss1+(1-λ)loss2,0<λ<1 S115. When the total loss function value meets a threshold condition, use the second model for distracted driving behavior category identification.

4. The method according to claim 2, characterized in that: The distracted driving action sequence includes at least 6 images.

5. The method according to claim 2, characterized in that: The evaluation data set also includes the following processing: processing the evaluation data set through Gaussian noise and performing an amplification process on the training set therein.

6. The method according to claim 1, characterized in that The method further comprises the steps of: Based on the type of distracted driving identified, an early warning will be issued when the driver needs to take over; The warning includes a warning with sound prompts and a dual warning with sound and steering wheel vibration.

7. A distracted driving behavior recognition device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program which can be loaded by the processor and executes the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 6.