Driver distraction detection method and device

Abstract

The present invention discloses a driver distraction detection method and device. The present application provides a driver distraction detection method and a detection device deployed on a vehicle-mounted terminal. The driver's facial image dataset is preprocessed; a trained distraction detection network model is input, and the driver's line of sight estimation result, that is, the discrete label probability of the pitch angle and azimuth angle of the line of sight, is output, and the pitch angle and azimuth angle are obtained by calculation; the pitch angle and azimuth angle are mapped to the pre-divided driver interest area, and it is determined whether the line of sight deviates from the normal area for a long time, and an early warning is issued for the driver's distracted behavior. The present invention significantly reduces the computational complexity of the deep learning distraction detection network model and maintains a high estimation accuracy, so it can reduce the processor performance requirements of the terminal device. At the same time, the line of sight estimation result can pave the way for further analysis of the driver's driving behavior and status.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and device for detecting driver distraction. Background Art

[0002] As the number of cars continues to increase, driving safety issues have become a widespread concern. Furthermore, with the widespread use of in-vehicle devices and mobile phones, irrelevant information is increasingly distracting drivers while driving. Studies have shown that the increasing number of distractions is making drivers more susceptible to distraction, posing a serious safety hazard. Therefore, driving safety is most threatened when the driver's eyes are off the road. Inattention, drowsiness, looking around, and fatigue are the main manifestations of visual distraction, and they are also common behaviors that drivers often engage in while driving. Therefore, identifying and warning distracting behaviors based on the driver's visual characteristics can effectively prevent traffic accidents.

[0003] In recent years, deep learning-based gaze estimation and distraction detection methods have become a hot topic. They exhibit numerous advantages over traditional methods. 1) They can extract high-level, abstract gaze features from high-dimensional images. 2) They learn a highly nonlinear mapping function from eye appearance to gaze. 3) Compared to traditional appearance-based methods, deep learning-based methods are more robust to interference such as lighting changes, eyeglass occlusion, and head movement.

[0004] However, although deep learning neural networks are more sensitive to feature information, the model has many parameters and large computational complexity, resulting in high cost for distraction detection. Summary of the Invention

[0005] The present invention aims to address the deficiencies of the prior art and to provide a method and device for detecting driver distraction.

[0006] The object of the present invention is achieved through the following technical solution: A method for detecting driver distraction, comprising the following steps:

[0007] Step 1: Dataset preprocessing: The driver face image dataset is divided into a training set and a test set. Each image in the dataset contains the expected result label, which is divided into continuous labels and discrete classification labels. The image is then scaled and normalized.

[0008] Step 2: Build a distraction detection network based on gaze estimation. The network includes a feature extraction part and two fully connected layers. The feature extraction part combines the residual structure, depth-wise separable convolution and channel-wise hierarchical convolution to achieve lightweight multi-scale feature extraction.

[0009] Step 3: Input a training sample from the preprocessed training set into the feature extraction part of the distraction detection network to obtain a feature map. Perform global average pooling on the feature map to obtain a total feature vector. The total feature vector is input into the two fully connected layers of the gaze estimation distraction detection network to obtain the discrete label probabilities of the pitch angle and azimuth angle.

[0010] Step 4: Calculate the loss function by fusing the mean square error and cross entropy of the discrete label probabilities of the pitch angle and azimuth angle, and update the network parameters according to the loss function;

[0011] Step 5: Select other training set training samples and adjust the network parameters using steps 3-4 in sequence to obtain the final detection model with the line of sight estimation error within the preset threshold;

[0012] Step 6: Select an image to be tested from any test set and input it into the final detection model obtained in Step 5. Obtain the discrete label probabilities for the pitch and azimuth angles. Calculate the azimuth and pitch angles of the driver's line of sight and estimate them. Pre-divide the cockpit into regions of interest (ROIs), map the estimated results to the ROIs, and make a distraction detection judgment based on the distraction judgment criteria for the pre-divide ROIs.

[0013] Furthermore, the expected result labels in step 1 are divided into continuous labels and discrete classification labels. The continuous labels are the line of sight pitch angle and azimuth angle, and the discrete classification labels are obtained by mapping the continuous labels. Specifically, the 0-180° angle range is first divided equally, and the continuous labels are mapped to the nearest range. The range label is the discrete classification label.

[0014] Furthermore, the specific network structure of the feature extraction part of the gaze estimation distraction detection network in step 2 includes:

[0015] The first layer uses a 7×7 convolution kernel and keeps the convolution kernel stride as 1. The activation layer uses the ReLU nonlinear activation function;

[0016] The subsequent network adopts an 8-layer, 10-layer, or 12-layer network structure that alternates between strides 2 and 1, and includes a lightweight multi-scale feature extraction backbone.

[0017] Furthermore, the lightweight multi-scale feature extraction Backbone uses deep separation convolution to replace ordinary convolution to reduce the amount of calculation and achieve lightweight: adopts residual structure to enhance network performance and accuracy; uses channel hierarchical convolution to achieve multi-scale feature extraction.

[0018] Furthermore, the channel hierarchical convolution is as follows: the feature map channel 4 is equally divided into C1, C2, C3, and C4, channel C1 is not processed, channel C2 is convolved once, channel C3 and channel C2 outputs are accumulated and convolved once, channel C4 and channel C3 outputs are accumulated and convolved once, the feature maps of each channel are re-spliced, and multi-scale features are obtained through feature fusion.

[0019] Furthermore, the loss function calculation in step 4 includes: the output discrete label probability passes through Softmax, and the weighted sum of the mean square error and the cross entropy is calculated;

[0020] Softmax is calculated as follows:

[0021]

[0022] The cross entropy is calculated as follows:

[0023]

[0024] Calculate the predicted angle value by discrete label probability:

[0025]

[0026] Calculate the mean squared error:

[0027]

[0028] Final loss function calculation:

[0029] Loss=H(y,p)+αMSE(angle,p)

[0030] Where p is the set of all expected discrete classification labels, p i is the expected discrete classification label specific value, N is the number of discrete labels, y is the vector of discrete label probabilities output by the network, and y i is the value corresponding to 1-N in vector y; α adjusts the weight of the mean square error.

[0031] Furthermore, in step 5, if the neural network model fails to meet the expected line of sight estimation requirements, the data set is replaced and training is continued.

[0032] Furthermore, comparing the estimation result in step 6 with the pre-divided driver interest area includes the following sub-steps:

[0033] Step 6-1: Divide the driver's area of ​​interest by selecting the left / right front window, left and right rearview mirrors, center rearview mirror, instrument panel, center console, and gear lever areas;

[0034] Step 6-2: Collect the corresponding sight line pitch angle and azimuth angle of each area and construct a distraction judgment data set;

[0035] Step 6-3: Use the SVM classifier to train and classify gaze angles, determine the area where the gaze stays, and record and remind any distracting behavior that deviates from the normal area for more than 2 seconds. The number of images per second is determined by the sampling frequency.

[0036] Another aspect of this specification provides a driver distraction detection device, which includes: a camera, a processor, and a memory;

[0037] The camera is preferably placed in front of the driver to perform the driver's area of ​​interest division;

[0038] The memory is used to store the network model structure, network model parameters, and the corresponding mobile terminal deployment architecture NCNN so that it can be run in the processor;

[0039] The processor is configured to read the computer program stored in the memory and execute the distraction detection and determination in step 6.

[0040] Furthermore, the device also includes an optional display screen or speaker according to usage requirements, and the optional display screen or speaker is used to provide timely warning after detecting distraction.

[0041] Beneficial effects of the present invention:

[0042] Through channel multi-scale feature extraction, deep separable convolution, residual network structure, and combined with continuous and discrete label loss function calculation methods, a multi-scale lightweight line of sight estimation network is realized. While reducing the computational complexity of the deep learning distraction detection network model, it can also maintain a high estimation accuracy, thereby reducing the processor performance requirements of the terminal device. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 An overall flow chart of a driver distraction detection method provided by an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of a distraction detection network structure provided by an embodiment of the present invention;

[0045] Figure 3 A diagram showing a distraction detection network feature extraction structure provided by an embodiment of the present invention;

[0046] Figure 4 A diagram of the lightweight multi-scale feature extraction Backbone structure provided by an embodiment of the present invention;

[0047] Figure 5 A schematic diagram of depth separation convolution provided by an embodiment of the present invention;

[0048] Figure 6Schematic diagram of feature fusion and loss function calculation provided by an embodiment of the present invention;

[0049] Figure 7 This is a visual line estimation effect diagram provided by an embodiment of the present invention;

[0050] Figure 8 A driver's interest area division diagram provided by an embodiment of the present invention;

[0051] Figure 9 A schematic diagram of a vehicle-mounted terminal device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] like Figure 1 As shown in the figure, the overall flow chart of the driver distraction detection method of the present invention is given, and the specific steps of the method are as follows

[0054] S101 is to first preprocess the data set to obtain the model training set and test set.

[0055] In this step, the dataset images are first divided into a training set and a test set. Each image in the dataset contains an expected result label to guide model training, and then the image is scaled and normalized. The expected result labels are divided into continuous labels and discrete classification labels. The continuous labels are the line of sight pitch angle and azimuth angle, and the discrete classification labels are obtained by mapping the continuous labels. The specific method is to first divide the angle range equally, map the continuous label to the nearest range, and the range label is the discrete classification label.

[0056] In this embodiment, the dataset images are scaled to 144*144, and the number of discrete classification labels is 90, evenly distributed between 0 and 180 degrees.

[0057] After constructing the distraction detection network, S102 inputs the pre-processed face image into the feature extraction part of the distraction detection network to implement a lightweight multi-scale feature extraction function;

[0058] like Figure 2 Figure 2 shows a schematic diagram of the distraction detection network structure. The network outputs discrete label probabilities for azimuth and elevation angles. During training, the loss function is calculated by integrating mean square error and cross entropy. During verification, the predicted azimuth and elevation angles are obtained through weighted summation.

[0059] like Figure 3The figure shows the structure of the lightweight multi-scale feature extraction part of the distraction detection network in this embodiment. The first layer uses a 7×7 convolution kernel with a convolution kernel sampling interval of 1, and the activation layer samples the ReLU nonlinear activation function. The subsequent network adopts an 8-layer network structure that alternates between strides 2 and 1 and includes a lightweight multi-scale feature extraction backbone.

[0060] like Figure 4 The figure shows the lightweight multi-scale feature extraction backbone selected in this embodiment. The feature map is first expanded to the expected number of output channels, and the number of channels is divided into 4 equal parts: C1, C2, C3, and C4. Channel C1 is not processed, channel C2 is convolved once, channel C3 is convolved with channel C2 output after accumulation, and channel C4 is convolved with channel C3 output after accumulation. The feature maps of each channel are rewritten and spliced, and multi-scale features are obtained through feature fusion. The bypass adopts a residual structure to enhance network performance and accuracy.

[0061] In the lightweight multi-scale feature extraction Backbone used in this embodiment, a 64*144*144 feature map is input, the stride is selected as 2, and the number of output channels is 128, then a 128*72*72 feature map can be obtained.

[0062] Specifically, Figure 5 This is a diagram of depthwise separable convolution, which is used to replace conventional convolution. Depthwise separable convolution decomposes conventional convolution into 1×1 depthwise convolution and n×n point convolution, which is used to reduce the amount of computation and achieve lightweight.

[0063] S103 Figure 6 As shown in the figure, the feature map is globally averaged pooled to compress the dimension, and the total feature vector is input into two fully connected layers respectively to obtain the discrete label probabilities of the line of sight estimated azimuth and pitch angles;

[0064] S104 calculates the loss function by fusing the mean square error and cross entropy of the discrete label probabilities of the azimuth and elevation angles, and updates the network parameters based on the loss function. The specific calculation process of the loss function is as follows:

[0065] The discrete label probabilities of azimuth and elevation angles are calculated by Softmax and then the weighted sum of mean square error (l2 loss) and cross entropy is calculated.

[0066] Softmax is calculated as follows:

[0067]

[0068] The cross entropy is calculated as follows:

[0069]

[0070] Calculate the predicted angle value by discrete label probability:

[0071]

[0072] Calculate the mean squared error:

[0073]

[0074] Final loss function calculation:

[0075] Loss=H(y,p)+αMSE(angle,p)

[0076] Where p is the set of all expected discrete classification labels, p i is the expected discrete classification label specific value, N is the number of discrete labels, y is the vector of discrete label probabilities output by the network, and y i is the value corresponding to 1-N in vector y; α adjusts the weight of the mean square error.

[0077] S105 selects other data set training samples and adjusts the model parameters using the above steps in sequence until the line of sight estimation error of the network model reaches within the preset error threshold range, thereby obtaining the final detection model.

[0078] In this example, the initial learning rate was set to 0.001, the image batch size was set to 16, and after 20 epochs of training, the learning rate was modified to 0.0001. To improve accuracy, the Adma optimizer was used for parameter optimization. The error threshold can be set according to actual application requirements, such as a line of sight estimation error of less than 12°.

[0079] S106 arbitrarily inputs an image to be tested, inputs the model, obtains the driver's sight estimation result, maps it to the pre-divided driver interest area, and makes a distraction detection judgment result.

[0080] like Figure 7 The figure shows the effect diagram of the line of sight estimation of this embodiment, which visualizes the line of sight direction based on the azimuth angle Pitch and pitch angle Yaw predicted by the network.

[0081] like Figure 8 The figure shows the driver's area of ​​interest (ROI). The left and right front windows, left and right rearview mirrors, center rearview mirror, instrument panel, center console, and gear lever are selected. The corresponding sight line pitch and azimuth angles for each area are collected to construct a judgment dataset. Using the sight line angles, an SVM classifier is trained and classified. Distractions that deviate from the normal area for more than 2 seconds are recorded and alerted. In this example, 60 images are collected within this 2-second period.

[0082] Figure 9A schematic diagram of the vehicle-mounted terminal device of the present invention is provided, comprising a camera, a processor, a memory, and an optional display screen or speaker. The camera is preferably placed in front of the driver to perform the driver's area of ​​interest segmentation and facial image data acquisition.

[0083] The memory is used to store the network model structure, network model parameters, and the corresponding mobile terminal deployment architecture NCNN so that it can run in the processor; the processor is used to read the computer program stored in the memory and perform distraction detection and judgment; an optional display screen or speaker is used to detect distraction and issue a timely warning.

[0084] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modifications and changes made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A method for detecting driver distraction, characterized in that: The following steps are involved: Step 1: Dataset preprocessing: The driver face image dataset is divided into a training set and a test set. Each image in the dataset contains the expected result label, which is divided into continuous labels and discrete classification labels. The image is then scaled and normalized. Step 2: Build a distraction detection network based on gaze estimation. The network includes a feature extraction part and two fully connected layers. The feature extraction part combines the residual structure, depth-wise separable convolution and channel-wise hierarchical convolution to achieve lightweight multi-scale feature extraction. Step 3: Input a training sample from the preprocessed training set into the feature extraction part of the distraction detection network to obtain a feature map. Perform global average pooling on the feature map to obtain a total feature vector. The total feature vector is input into the two fully connected layers of the gaze estimation distraction detection network to obtain the discrete label probabilities of the pitch angle and azimuth angle. Step 4: Calculate the loss function by fusing the mean square error and cross entropy of the discrete label probabilities of the pitch angle and azimuth angle, and update the network parameters according to the loss function; Step 5: Select other training set training samples and adjust the network parameters using steps 3-4 in sequence to obtain the final detection model with the line of sight estimation error within the preset threshold; Step 6: Select an image to be tested from any test set and input it into the final detection model obtained in Step 5. Obtain the discrete label probabilities for the pitch and azimuth angles. Calculate the azimuth and pitch angles of the driver's line of sight and estimate them. Pre-divide the cockpit into regions of interest (ROIs), map the estimated results to the ROIs, and make a distraction detection judgment based on the distraction judgment criteria for the pre-divide ROIs.

2. The driver distraction detection method according to claim 1, characterized in that: The expected result labels in step 1 are divided into continuous labels and discrete classification labels. The continuous labels are the line of sight pitch angle and azimuth angle. The discrete classification labels are obtained by mapping the continuous labels. Specifically, the 0-180° angle range is first divided equally, and the continuous labels are mapped to the nearest range. The range label is the discrete classification label.

3. The driver distraction detection method according to claim 1, characterized in that: The specific network structure of the line of sight estimation and distraction detection network feature extraction part in step 2 includes: The first layer uses a 7×7 convolution kernel and keeps the convolution kernel stride as 1. The activation layer uses the ReLU nonlinear activation function; The subsequent network adopts an 8-layer, 10-layer, or 12-layer network structure that alternates between strides 2 and 1, and includes a lightweight multi-scale feature extraction backbone.

4. The driver distraction detection method according to claim 3, characterized in that: The lightweight multi-scale feature extraction Backbone uses deep separation convolution to replace ordinary convolution to reduce the amount of calculation and achieve lightweight: adopts residual structure to enhance network performance and accuracy; uses channel hierarchical convolution to achieve multi-scale feature extraction.

5. The driver distraction detection method according to claim 4, characterized in that: The channel hierarchical convolution is as follows: the feature map channel 4 is equally divided into C1, C2, C3, and C4. Channel C1 is not processed, channel C2 is convolved once, channel C3 and channel C2 outputs are accumulated and convolved once, channel C4 and channel C3 outputs are accumulated and convolved once, and the feature maps of each channel are re-spliced ​​to obtain multi-scale features through feature fusion.

6. The driver distraction detection method according to claim 1, characterized in that: The loss function calculation in step 4 includes: the output discrete label probability passes through Softmax, and the weighted sum of the mean square error and the cross entropy is calculated; Softmax is calculated as follows: The cross entropy is calculated as follows: Calculate the predicted angle value by discrete label probability: Calculate the mean squared error: Final loss function calculation: in is the set of all expected discrete classification labels, is the expected discrete classification label value, N is the number of discrete labels, The network outputs a vector of discrete label probabilities, is a vector The corresponding value of 1-N; Adjust the weight of the mean squared error.

7. The driver distraction detection method according to claim 1, characterized in that: In step 5, if the neural network model does not meet the expected line of sight estimation requirements, the data set is replaced and training is continued.

8. The driver distraction detection method according to claim 1, characterized in that: Comparing the estimated results with the pre-divided driver interest area in step 6 includes the following sub-steps: Step 6-1: Divide the driver's area of ​​interest by selecting the left / right front window, left and right rearview mirrors, center rearview mirror, instrument panel, center console, and gear lever areas; Step 6-2: Collect the corresponding sight line pitch angle and azimuth angle of each area and construct a distraction judgment data set; Step 6-3: Use the SVM classifier to train and classify gaze angles, determine the area where the gaze stays, and record and remind any distracting behavior that deviates from the normal area for more than 2 seconds. The number of images per second is determined by the sampling frequency.

9. A driver distraction detection device for implementing the method according to any one of claims 1 to 8, characterized in that: include: a camera, processor, and memory, and an optional display or speakers; The camera is preferably placed in front of the driver to perform driver interest area division and driver facial image acquisition; The memory is used to store the network model structure, network model parameters, and the corresponding mobile terminal deployment architecture NCNN so that it can be run in the processor; The processor is configured to read the computer program stored in the memory and execute the distraction detection and determination in step 6.

10. The driver distraction detection device according to claim 9, characterized in that: The device also includes an optional display screen or speaker according to usage requirements, and the optional display screen or speaker is used to provide timely warning after detecting distraction.

Images