Training method of distraction detection model, distraction detection method and device

By using two lightweight convolutional neural network models in distraction detection in parallel to extract features and perform fusion processing, the problem of many parameters and large calculations of deep convolutional neural network models is solved, and more efficient and accurate division is achieved Heart test.

CN114255454BActive Publication Date: 2024-11-29HANGZHOU DIANZI UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202111561183.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-16
Publication Date
2024-11-29
Estimated Expiration
2041-12-16

AI Technical Summary

Technical Problem

The existing deep convolutional neural network model has many parameters and high calculation amounts in distraction detection, resulting in high cost and insufficient detection accuracy.

Method used

Two lightweight convolutional neural network models are used to extract features in parallel, and the features of image samples are extracted through the first feature extraction model and the second feature extraction model, and then fuse them and input them into the classification model for multi-scale feature pooling. To obtain global scale features, update model parameters to improve detection accuracy.

Benefits of technology

The cost of distraction detection is reduced, the detection accuracy is improved, and the comprehensiveness of features and computational efficiency is ensured.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114255454B_ABST
    Figure CN114255454B_ABST
Patent Text Reader

Abstract

The present application provides a training method, a distraction detection method and a device for a distraction detection model, which obtains a plurality of image samples and labels corresponding to each of the plurality of image samples, and respectively inputs the plurality of image samples into a first feature extraction model and a second feature extraction model in an initial distraction detection model, obtains a first feature and a second feature corresponding to each image sample, and then inputs the first feature and the second feature corresponding to each image sample into a classification model in the initial distraction detection model, obtains a driving state prediction result corresponding to each image sample, and then updates the network parameters of the initial distraction detection model according to the driving state prediction result corresponding to each image sample and the labels corresponding to each of the plurality of image samples, thereby obtaining a distraction detection model. The embodiments of the present application can reduce the cost of distraction detection, and can ensure that the extracted features are more comprehensive, thereby improving the accuracy of the distraction detection model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a distraction detection model training method, a distraction detection method and a distraction detection device. Background Art

[0002] In recent years, the number of deaths caused by traffic accidents has continued to increase, and distracted driving is one of the main causes of traffic accidents. "People" are the determining factor in driving, and distracted driving behavior is often a subconscious behavior of the driver, which occurs inadvertently. Because of this, distracted driving is easily ignored by the driver and causes serious traffic accidents. Therefore, fast and accurate detection of distracted driving is of great practical significance for reducing road traffic accidents caused by distracted driving and protecting the lives and property of drivers.

[0003] At present, in the process of distracted driving detection, convolutional neural network models are usually used for detection. In order to ensure the accuracy of the model and the comprehensiveness of the image features extracted from the convolutional neural network model, deep convolutional neural networks are usually used for feature extraction.

[0004] However, although deep convolutional neural networks are more sensitive to feature information, the model has many parameters and large computational complexity, resulting in a high cost for distraction detection. Summary of the invention

[0005] The present application provides a distraction detection model training method, a distraction detection method and a device, which are used to solve the problem in the prior art that when extracting features from a deep convolutional neural network model, there are many parameters and a large amount of calculation, resulting in a high distraction detection cost.

[0006] In a first aspect, the present application provides a method for training a distraction detection model, comprising:

[0007] A plurality of image samples and labels corresponding to the plurality of image samples are obtained, where the labels are used to represent the driving state of the driver in the image samples, and the driving state includes a normal driving state and a distracted driving state.

[0008] Multiple image samples are respectively input into the first feature extraction model and the second feature extraction model in the initial distraction detection model to obtain the first feature and the second feature corresponding to each image sample. The first feature extraction model and the second feature extraction model are both lightweight network models.

[0009] The first feature and the second feature corresponding to each image sample are input into the classification model in the initial distraction detection model to obtain the driving state prediction result corresponding to each image sample.

[0010] According to the driving state prediction results corresponding to each image sample and the labels corresponding to the multiple image samples, the network parameters of the initial distraction detection model are updated.

[0011] Optionally, the first feature and the second feature corresponding to each image sample are input into a classification model in the initial distraction detection model to obtain a driving state prediction result corresponding to each image sample, including:

[0012] For each image sample, the first feature and the second feature corresponding to the image sample are fused to obtain a fused feature.

[0013] The fused features are input into multiple dilated convolutional layers in the initial distraction detection model to obtain multi-scale features, and the driving state prediction results corresponding to the image samples are obtained based on the multi-scale features and the classification model.

[0014] Optionally, based on the multi-scale features and the classification model, a driving state prediction result corresponding to the image sample is obtained, including:

[0015] The multi-scale features are pooled by the maximum pooling method to obtain the processed multi-scale features.

[0016] The processed multi-scale features are fused to obtain the global scale features.

[0017] The global scale features are input into the classification model to obtain the driving state prediction results corresponding to the image samples.

[0018] Optionally, the dilation rates corresponding to the multiple dilated convolutional layers are different.

[0019] Optionally, according to the driving state prediction results and labels corresponding to each image sample, the network parameters of the initial distraction detection model are updated, including:

[0020] According to the driving state prediction results and labels corresponding to each image sample, a loss function corresponding to each image sample is constructed.

[0021] According to the loss function corresponding to each image sample, the network parameters of the initial distraction detection model are updated.

[0022] In a second aspect, the present application provides a distraction detection method, comprising:

[0023] Get the image to be detected.

[0024] The image to be detected is input into a pre-trained distraction detection model to obtain the driving state of the driver, and the driving state includes a normal driving state and a distracted driving state; wherein the distraction detection model is to input multiple image samples into the first feature extraction model and the second feature extraction model in the initial distraction detection model respectively to obtain the first feature and the second feature, and input the first feature and the second feature into the classification model in the initial distraction detection model, after obtaining the driving state prediction result corresponding to each image sample, according to the driving state prediction result corresponding to each image sample and the labels corresponding to each of the multiple image samples, the first feature extraction model and the second feature extraction model are both lightweight network models, and the labels are used to represent the driving state of the driver in the image samples.

[0025] Optionally, the image to be detected is input into a pre-trained distraction detection model to obtain the driver's driving state, including:

[0026] The image to be detected is input into the first feature extraction model and the second feature extraction model of the distraction detection model to obtain the first feature and the second feature corresponding to the image to be detected.

[0027] The first feature and the second feature are fused to obtain a fused feature.

[0028] The fused features are input into multiple dilated convolutional layers in the distraction detection model to obtain multi-scale features, and the driver's driving state is obtained based on the multi-scale features and the classification model.

[0029] Optionally, the driving state of the driver is obtained based on the multi-scale features and the classification model, including:

[0030] The multi-scale features are pooled by the maximum pooling method to obtain the processed multi-scale features.

[0031] The processed multi-scale features are fused to obtain the global scale features.

[0032] The global scale features are input into the classification model to obtain the driver's driving state.

[0033] In a third aspect, the present application provides a training device for a distraction detection model, comprising:

[0034] The acquisition module is used to acquire multiple image samples and labels corresponding to each of the multiple image samples, where the labels are used to represent the driving state of the driver in the image samples, and the driving state includes a normal driving state and a distracted driving state.

[0035] The extraction module is used to input multiple image samples into the first feature extraction model and the second feature extraction model in the initial distraction detection model respectively to obtain the first feature and the second feature corresponding to each image sample. The first feature extraction model and the second feature extraction model are both lightweight network models.

[0036] The prediction module is used to input the first feature and the second feature corresponding to each image sample into the classification model in the initial distraction detection model to obtain the driving state prediction result corresponding to each image sample.

[0037] The updating module is used to update the network parameters of the initial distraction detection model according to the driving state prediction results corresponding to each image sample and the labels corresponding to each of the multiple image samples.

[0038] Optional, prediction module, specifically used for:

[0039] For each image sample, the first feature and the second feature corresponding to the image sample are fused to obtain a fused feature.

[0040] The fused features are input into multiple dilated convolutional layers in the distraction detection model to obtain multi-scale features, and the driving state prediction results corresponding to the image samples are obtained based on the multi-scale features and the classification model.

[0041] Optional, prediction module, specifically used for:

[0042] The multi-scale features are pooled by the maximum pooling method to obtain processed multi-scale features; the processed multi-scale features are fused to obtain global scale features.

[0043] The global scale features are input into the classification model to obtain the driving state prediction results corresponding to the image samples.

[0044] Optionally, the dilation rates corresponding to the multiple dilated convolutional layers are different.

[0045] Optional, update module, specifically for:

[0046] According to the driving state prediction results and labels corresponding to each image sample, a loss function corresponding to each image sample is constructed.

[0047] According to the loss function corresponding to each image sample, the network parameters of the initial distraction detection model are updated.

[0048] In a fourth aspect, the present application provides a distraction detection device, comprising:

[0049] The acquisition module is used to acquire the image to be detected.

[0050] The detection module is used to input the image to be detected into a pre-trained distraction detection model to obtain the driver's driving state, which includes a normal driving state and a distracted driving state; wherein the distraction detection model is to input multiple image samples into the first feature extraction model and the second feature extraction model in the initial distraction detection model to obtain the first feature and the second feature, and input the first feature and the second feature into the classification model in the initial distraction detection model to obtain the driving state prediction result corresponding to each image sample, and then obtain the first feature extraction model and the second feature extraction model according to the driving state prediction result corresponding to each image sample and the labels corresponding to each of the multiple image samples, wherein both the first feature extraction model and the second feature extraction model are lightweight network models, and the labels are used to represent the driving state of the driver in the image samples.

[0051] Optional, detection module, specifically used for:

[0052] The image to be detected is input into the first feature extraction model and the second feature extraction model of the distraction detection model to obtain the first feature and the second feature corresponding to the image to be detected.

[0053] The first feature and the second feature are fused to obtain a fused feature.

[0054] The fused features are input into multiple dilated convolutional layers in the distraction detection model to obtain multi-scale features, and the driver's driving state is obtained based on the multi-scale features and the classification model.

[0055] Optional, detection module, specifically used for:

[0056] The multi-scale features are pooled by the maximum pooling method to obtain processed multi-scale features; the processed multi-scale features are fused to obtain global scale features.

[0057] The global scale features are input into the classification model to obtain the driver's driving state.

[0058] In a fifth aspect, the present application provides an electronic device, including: a memory and a processor;

[0059] Memory, used to store computer programs.

[0060] The processor is used to read the computer program stored in the memory and execute the training method of the distraction detection model of the first aspect according to the computer program in the memory, or to execute the distraction detection method of the second aspect.

[0061] In a sixth aspect, the present application provides a readable storage medium having a computer program stored thereon, wherein the computer program stores computer execution instructions, which, when executed by a processor, are used to implement a training method for a distraction detection model as in the first aspect above, or to implement a distraction detection method as in the second aspect above.

[0062] In a seventh aspect, an embodiment of the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the training method of the distraction detection model of the first aspect above, or is used to implement the distraction detection method of the second aspect above.

[0063] The training method, distraction detection method and device of the distraction detection model provided in the present application obtain multiple image samples and labels corresponding to each of the multiple image samples, and input the multiple image samples into the first feature extraction model and the second feature extraction model in the initial distraction detection model respectively, so as to obtain the first feature and the second feature corresponding to each image sample, and then input the first feature and the second feature corresponding to each image sample into the classification model in the initial distraction detection model to obtain the driving state prediction result corresponding to each image sample, and then update the network parameters of the initial distraction detection model according to the driving state prediction result corresponding to each image sample and the labels corresponding to each of the multiple image samples, so as to obtain the distraction detection model. Since the first feature extraction model and the second feature extraction model are both lightweight network models, the features corresponding to each image sample are extracted by two lightweight network models. Compared with the deep network model, the model parameters are fewer and the amount of calculation is small, so that the cost of distraction detection can be reduced. In addition, by extracting the features corresponding to each image sample by two feature extraction models, it can be ensured that the extracted features are more comprehensive, so that the accuracy of the distraction detection model can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A flowchart of a method for training a distraction detection model provided in an embodiment of the present application;

[0065] Figure 2 A schematic diagram of the structure of an initial distraction detection model provided in an embodiment of the present application;

[0066] Figure 3 A schematic diagram of a processing process of multiple dilated convolutions and maximum pooling provided in an embodiment of the present application;

[0067] Figure 4 A flowchart of a method for verifying a distraction detection model provided in an embodiment of the present application;

[0068] Figure 5 A schematic diagram of a training accuracy and a verification accuracy provided in an embodiment of the present application;

[0069] Figure 6 A schematic diagram of a training loss and a validation loss provided in an embodiment of the present application;

[0070] Figure 7 A flowchart of a distraction detection method provided in an embodiment of the present application;

[0071] Figure 8 A schematic diagram of the structure of a distraction detection model training device provided in an embodiment of the present application;

[0072] Fig. 9 A schematic diagram of the structure of a distraction detection device provided in an embodiment of the present application;

[0073] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0074] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0075] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0076] The technical solution provided in the embodiment of the present application can be applied to the scenario of detecting whether the driver is distracted driving. Drivers' distracted actions such as answering the phone, drinking water, and picking up things while driving are prone to traffic accidents. By extracting the characteristics of the driver's driving behavior and detecting whether it is distracted driving behavior, it has important practical significance for reducing road traffic accidents caused by distracted driving.

[0077] As one of the common methods for feature extraction, convolutional neural networks also have a wide range of application scenarios in distracted driving detection. In the prior art, a single convolutional neural network model is usually used for distracted driving detection. In the process of distracted driving detection, different convolutional neural networks have different feature extraction capabilities due to the different depths of the architecture. Using a single deep convolutional neural network is more sensitive to general features, but the model parameters are large and the computational cost is high; while the features extracted using a single shallow convolutional neural network are too general to distinguish subtle differences, that is, the image features extracted from a single shallow convolutional neural network model are not comprehensive enough, which greatly limits the application of distracted driving behavior detection methods based on convolutional neural networks in actual application scenarios.

[0078] In order to solve the above problems, the present application proposes a training method for a distraction detection model. Since the lightweight convolutional neural network model has fewer model parameters and a small amount of calculation, the features extracted by the lightweight convolutional neural network model are not comprehensive enough. Based on this, in the present application, when performing feature extraction, two lightweight convolutional neural network models can be used to extract features in each image sample in parallel. Since the features extracted by the two lightweight convolutional neural network models each have different semantics, the features extracted by the two convolutional neural network models in parallel are more comprehensive than a single convolutional neural network model. At the same time, the lightweight convolutional neural network model has the characteristics of being lightweight and high-precision, and is faster and less computationally intensive than the traditional convolutional neural network model, which makes the cost of distraction detection lower.

[0079] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0080] Figure 1 A flowchart of a distraction detection model training method provided in an embodiment of the present application is provided. The distraction detection model training method can be executed by software and / or hardware devices. For example, the hardware device can be an electronic device such as a terminal or a server. For example, see Figure 1 As shown, the training method of the distraction detection model may include:

[0081] S101. Acquire multiple image samples and labels corresponding to each of the multiple image samples, where the labels are used to represent the driving state of the driver in the image samples, and the driving state includes a normal driving state and a distracted driving state.

[0082] In this step, multiple image samples can be obtained by shooting images or capturing multiple frames of images in a video, and the image samples all include the driver. The label indicates whether the driver in the image sample is in a normal driving state or a distracted driving state. For example, if the driver in the image sample is making a phone call or drinking water, the label information of the image sample is a distracted driving state; if the driver in the image sample is concentrating on driving, the label information of the image sample is a normal driving state.

[0083] Exemplarily, the label corresponding to each image sample may be manually annotated.

[0084] S102. Input a plurality of image samples into a first feature extraction model and a second feature extraction model in an initial distraction detection model respectively to obtain first features and second features corresponding to each image sample. Both the first feature extraction model and the second feature extraction model are lightweight network models.

[0085] Specifically, Figure 2 It is a structural diagram of the initial distraction detection model. Figure 2 As shown, after acquiring the image sample, the image sample can be input into the first feature extraction model and the second feature extraction model to perform feature extraction operations, thereby obtaining the first feature corresponding to the first feature extraction model and the second feature corresponding to the second feature extraction model.

[0086] Exemplarily, before inputting each image sample into the first feature extraction model and the second feature extraction model in the initial distraction detection model, each image sample needs to be preprocessed first. For example, the size of each image sample can be scaled to meet the requirements of the first feature extraction model and the second feature extraction model. For example, if the image input size of the first feature extraction model and the second feature extraction model is 224*224, the size of each image sample needs to be set to 224*224.

[0087] In this step, the first feature extraction model and the second feature extraction model can be models that have not undergone any training. Of course, in order to increase the speed of training and avoid training from scratch, the first feature extraction model and the second feature extraction model can also be convolutional neural network models whose model parameters have been pre-trained on a small scale. These two network models are pre-trained using transfer learning, and some parameters are fine-tuned.

[0088] Optionally, to improve the accuracy of feature extraction model training, you can use multiple image samples to fine-tune and modify the last few layers of the model. For example, for pre-trained MobileNetV2, freeze the weights from layer 0 to layer 50, and only train the weights from layer 50 to the top layer; for pre-trained DenseNet, freeze the weights from layer 0 to layer 70, and only train the weights from layer 70 to the top layer.

[0089] Exemplarily, before training the feature extraction model, some parameters are set for the model, for example, the initial learning rate is set to 0.001, the image batch size is set to 64, and each feature extraction model is trained for 40 cycles in the Graphics Processing Unit (GPU). When training the feature extraction model, categorical cross entropy is used as the loss function and Softmax is used as the activation function. At the same time, in order to improve the accuracy of the model, the Adma optimizer is used to optimize the parameters.

[0090] Exemplarily, the first feature may be understood as an image feature extracted from a first feature extraction model, and the second feature may be understood as an image feature extracted from a second feature extraction model.

[0091] Exemplarily, since the features extracted by different feature extraction models have different semantics, in order to ensure the comprehensiveness of the extracted features, the first feature extraction model and the second feature extraction model are two different models. In addition, due to the rapidity and timeliness of the driver distraction detection task, the first feature extraction model and the second feature extraction model should select lightweight and high-precision lightweight network models, such as MobileNetV2 and DenseNet.

[0092] Among them, DenseNet is a convolutional neural network with dense connections. In this network, all layers are directly connected to each other. In order to retain the feedforward characteristics, a separate short circuit is added to all the previous layers of each layer to connect to this layer, so that any two layers of the network can communicate directly. Each layer can get its input from all previous layers and pass its own feature map to all subsequent layers. Compared with traditional convolutional networks, it does not need to relearn redundant feature maps, so it requires fewer parameters.

[0093] The MobileNetV2 architecture is based on an inverted residual structure. The original residual structure has three main branches, two point-by-point convolutions with more channels, while the inverted residual structure is just the opposite, with more channels in the middle and fewer channels on the sides. The network model first expands the input low-dimensional compressed representation to a high dimension, uses lightweight deep convolution for filtering, and then uses a linear bottleneck to project the features back to a low-dimensional compressed representation. Using the structure of the inverted residual block, higher accuracy can be achieved with less computation. MobileNetV2 is lighter and faster than other algorithms in image recognition.

[0094] S103: Input the first feature and the second feature corresponding to each image sample into the classification model in the initial distraction detection model to obtain the driving state prediction result corresponding to each image sample.

[0095] In this step, the driving state prediction result can be understood as the prediction result of whether the driver is distracted driving in each image sample. Among them, the classification model mainly includes a fully connected layer and a Softmax classification layer. The fully connected layer maps the input first feature and second feature representation to the sample label space, and finally obtains the final driving state prediction result through the Softmax function.

[0096] For example, continue to refer to Figure 2 As shown, when determining the driving state prediction result corresponding to each image sample, for each image sample, the first feature and the second feature corresponding to the image sample can be fused to obtain a fused feature, and then the fused feature is input into multiple dilated convolutional layers in the initial distraction detection model to obtain a multi-scale feature, and based on the multi-scale feature and the classification model, the driving state prediction result corresponding to the image sample is obtained.

[0097] Specifically, the feature matrices of the first feature and the second feature are both 7*7*2048, and the fused feature after the two feature matrices are fused is a feature matrix containing 14*7*2048 vectors.

[0098] Exemplarily, the fusion method may be a convolutional fusion strategy, which is a simple and efficient method that can effectively solve the problem that a single convolutional neural network model is not comprehensive enough in extracting image features, and can effectively enrich feature content.

[0099] After inputting the obtained fused features into multiple dilated convolutional layers in the initial distraction detection model, multi-scale features can be obtained. The multi-scale features are then pooled through the maximum pooling method to obtain processed multi-scale features. The processed multi-scale features are fused to obtain global scale features. The global scale features can be input into the classification model to obtain the driving state prediction results corresponding to the image samples.

[0100] Specifically, Figure 3 This is a schematic diagram of the processing of multiple dilated convolutions and maximum pooling. Figure 3 As shown in the figure, after the fusion features are input into multiple dilated convolutional layers, the multi-scale features corresponding to each dilated convolutional layer are obtained, and then each multi-scale feature is pooled by the maximum pooling method, so that the processed multi-scale features can be obtained. Among them, multi-dilated pooling can fully fuse the multi-scale features of the image and perform effective processing, reducing the problem of context information loss between different sub-regions during feature extraction.

[0101] In the field of image classification, convolutional neural networks will gradually reduce the image resolution, resulting in the loss of the internal data structure and spatial hierarchical information of the image, and the inability to reconstruct small object information. However, the detailed information of the image is crucial to understanding the image. In order to solve this problem, this application introduces dilated convolution, which is mainly born to solve the problem of information loss caused by first reducing and then expanding the image size, the inability to reconstruct object information, and the spatial structure in the image is no longer clear. The main function of introducing dilated convolution is to increase the receptive field without changing the image size. The principle is to add 0-value pixels between each pixel of the kernel, and increase the size of the kernel in a directional manner to increase the receptive field.

[0102] Exemplarily, a plurality of dilated convolutional layers may be set to have different corresponding dilation rates, so that feature information of objects of different sizes in the image may be retained to avoid the problem of information loss.

[0103] In this embodiment, performing multiple dilated convolution and maximum pooling operations on the fused features can reduce the loss of image information, retain more image detail features, make the spatial structure in the image clearer, and thus improve the accuracy of the driving state prediction results.

[0104] S104: Update the network parameters of the initial distraction detection model according to the driving state prediction results corresponding to each image sample and the labels corresponding to each of the multiple image samples.

[0105] Exemplarily, when updating the network parameters of the initial distraction detection model, a loss function corresponding to each image sample can be constructed based on the driving state prediction results corresponding to each image sample and the labels corresponding to each of the multiple image samples, and the network parameters of the initial distraction detection model can be updated based on the loss function corresponding to each image sample.

[0106] Specifically, according to the driving state prediction results and labels corresponding to each image sample, a loss function corresponding to each image sample can be constructed, and an average function of the loss function corresponding to each image sample can be determined, so as to update the network parameters of the initial distraction detection model according to the average function. If the initial distraction detection model after updating the network parameters converges, the converged initial distraction detection model is determined as the distraction detection model. If the initial distraction detection model after updating the network parameters does not converge, the network parameters of the initial distraction detection model after updating the network parameters are continuously adjusted according to the method in the above embodiment until convergence is achieved.

[0107] In this embodiment, by constructing a driving state prediction result corresponding to each image sample and a loss function corresponding to each label of multiple image samples, the network parameters of the distraction detection model are continuously updated, thereby improving the accuracy of distraction detection model training.

[0108] The training method of the distraction detection model provided in the embodiment of the present application obtains multiple image samples and labels corresponding to each of the multiple image samples, the labels are used to represent the driving state of the driver in the image samples, and the driving state includes a normal driving state and a distracted driving state, and the multiple image samples are respectively input into the first feature extraction model and the second feature extraction model in the initial distraction detection model to obtain the first feature and the second feature corresponding to each image sample, and the first feature extraction model and the second feature extraction model are both lightweight network models. For each image sample, the first feature and the second feature corresponding to the image sample are fused to obtain a fused feature, and the fused feature is input into multiple dilated convolution layers in the initial distraction detection model to obtain a multi-scale feature, and the multi-scale feature is pooled by the maximum pooling method to obtain a processed multi-scale feature, and the processed multi-scale feature is fused to obtain a global scale feature, and the global scale feature is input into the classification model to obtain a driving state prediction result corresponding to the image sample, so as to update the network parameters of the initial distraction detection model according to the driving state prediction result corresponding to each image sample and the labels corresponding to each of the multiple image samples. When extracting features, two lightweight convolutional neural network models can be used to extract features in each image sample in parallel. Since the features extracted by the two lightweight convolutional neural network models have different semantics, the features extracted by the two network models in parallel are more comprehensive than those of a single network model. At the same time, the lightweight convolutional neural network model has the characteristics of lightness and high precision, and is faster and less computationally intensive than the traditional convolutional neural network model, making detection faster and reducing the cost of distraction detection. In addition, by performing multiple dilated convolutions on image features, the problem of image information loss is reduced, the detailed features of the image are retained, and the accuracy of the distraction detection model is improved.

[0109] Furthermore, in order to verify the detection accuracy of the trained distraction detection model, in practical applications, the trained distraction detection model can also be tested using the verification images in the verification set.

[0110] Figure 4 A flowchart of a method for verifying a distraction detection model provided in an embodiment of the present application is provided. The method for verifying a distraction detection model may be executed by software and / or hardware devices. For example, the hardware device may be an electronic device. For example, the method for verifying a distraction detection model may include:

[0111] S401, obtaining multiple verification images.

[0112] Exemplarily, all image samples used in the training and verification process can be regarded as a data set, and the data set can be divided into a training set and a verification set. The training set can be understood as the image samples used in the training process of the distraction detection model, and the verification set can be understood as the verification images used in the verification process of the distraction detection model.

[0113] S402: Input multiple verification images into a pre-trained distraction detection model to obtain a driving state verification result of the driver.

[0114] The distraction detection model is trained according to any of the aforementioned embodiments.

[0115] Exemplarily, before inputting the multiple verification images into the distraction detection model, the size of the verification images is preprocessed to meet the input requirements of the first feature extraction model and the second feature extraction model. For example, if the image input size of the first feature extraction model and the second feature extraction model is 224*224, the size of each verification image needs to be set to 224*224.

[0116] Exemplarily, when multiple verification images are input into a pre-trained distraction detection model to determine the driver's driving state, the multiple verification images can be input into the first feature extraction model and the second feature extraction model of the distraction detection model to obtain the first features and the second features corresponding to the multiple verification images, and the first features and the second features are fused to obtain fused features, and the fused features are then input into multiple dilated convolutional layers in the distraction detection model to obtain multi-scale features, and based on the multi-scale features and the classification model, the driver's driving state verification result is obtained.

[0117] Specifically, the multi-scale features are pooled through the maximum pooling method to obtain processed multi-scale features, the processed multi-scale features are fused to obtain global scale features, and the global scale features are input into the classification model to obtain the driver's driving state verification result.

[0118] For example, Figure 5 It is a schematic diagram of the training accuracy and verification accuracy obtained through the training process and the verification process. Figure 6 This is a diagram of training loss and validation loss, as shown in Figure 5 and Figure 6 As shown in the figure, a set of image samples including 26 driver images is set as a data set, and the data set is divided into 10 categories according to the differences in driver behavior, and the ratio of the number of image samples in the training set and the verification set is set at about 8:2. As the number of training and verification times increases, the accuracy of training and verification also increases until they remain basically unchanged.

[0119] The verification method of the distraction detection model provided in the embodiment of the present application can not only reduce the amount of calculation in the feature extraction process and the cost of distraction detection, making the detection faster and the extracted image features more comprehensive, but also reduce the problem of image information loss, retain the detailed features of the image, and make the verification result more accurate. In addition, the accuracy of the trained distraction detection model can also be verified.

[0120] Figure 7 This is a flow chart of a distraction detection method provided in an embodiment of the present application. The distraction detection method can be executed by software and / or hardware devices. For example, the hardware device can be an electronic device. For example, see Figure 7 As shown, the distraction detection method may include:

[0121] S701: Acquire an image to be detected.

[0122] In this step, the image to be detected may be an image representing the current driving state of the driver, and may be acquired through a real-time image acquisition device such as a monitoring device or a camera device.

[0123] S702: Input the image to be detected into a pre-trained distraction detection model to obtain the driving state of the driver.

[0124] Exemplarily, before the image to be detected is input into the distraction detection model, the size of the image to be detected is preprocessed to meet the input requirements of the first feature extraction model and the second feature extraction model. For example, if the image input size of the first feature extraction model and the second feature extraction model is 224*224, the size of the image to be detected is also set to 224*224.

[0125] In this step, the distraction detection model can be based on Figure 1 The model obtained after training the embodiment shown. The driving state includes a normal driving state and a distracted driving state. The driving state can be understood as representing whether the current driving action of the detected driver is normal driving or distracted driving.

[0126] Exemplarily, when the image to be detected is input into a pre-trained distraction detection model to determine the driver's driving state, the image to be detected can be input into the first feature extraction model and the second feature extraction model of the distraction detection model to obtain the first feature and the second feature corresponding to the image to be detected, and the first feature and the second feature are fused to obtain a fused feature, and the fused feature is then input into multiple dilated convolutional layers in the distraction detection model to obtain multi-scale features, and the driver's driving state is obtained based on the multi-scale features and the classification model.

[0127] Specifically, the multi-scale features are pooled by the maximum pooling method to obtain processed multi-scale features, the processed multi-scale features are fused to obtain global scale features, and the global scale features are input into the classification model to obtain the driving state of the driver.

[0128] For example, when the driver is detected to be in a distracted driving state, the user can be reminded by vibration, sound, or flashing reminders to urge the driver to return to a normal driving state. If the driver continues to be distracted driving, the driver can be continuously reminded until he or she returns to a normal driving state.

[0129] The distraction detection method provided in the embodiment of the present application obtains the image to be detected, inputs the image to be detected into the first feature extraction model and the second feature extraction model of the distraction detection model, obtains the first feature and the second feature corresponding to the image to be detected, and fuses the first feature and the second feature to obtain the fused feature, and then inputs the fused feature into multiple dilated convolution layers in the distraction detection model to obtain multi-scale features, and performs pooling processing on the multi-scale features through the maximum pooling method to obtain the processed multi-scale features, fuses the processed multi-scale features to obtain the global scale features, and inputs the global scale features into the classification model to obtain the driving state of the driver. When performing feature extraction, the features in each image sample can be extracted in parallel by two lightweight convolutional neural network models. Since the features extracted by the two lightweight convolutional neural network models have different semantics, the features extracted in parallel by the two network models are more comprehensive compared to a single network model. At the same time, the lightweight convolutional neural network model has the characteristics of lightness and high precision, and is faster and less computationally intensive than the traditional convolutional neural network model, making the detection faster and the cost of distraction detection lower. In addition, by performing multiple dilated convolution processing on image features, the loss of image information is reduced, the detailed features of the image are retained, and the accuracy of the distraction detection model is improved.

[0130] Figure 8 A structural diagram of a distraction detection model training device 80 provided in an embodiment of the present application, for example, see Figure 8As shown, the training device 80 of the distraction detection model includes:

[0131] The acquisition module 801 is used to acquire multiple image samples and labels corresponding to each of the multiple image samples, where the labels are used to represent the driving state of the driver in the image samples, and the driving state includes a normal driving state and a distracted driving state.

[0132] The extraction module 802 is used to input multiple image samples into the first feature extraction model and the second feature extraction model in the initial distraction detection model respectively to obtain the first feature and the second feature corresponding to each image sample. The first feature extraction model and the second feature extraction model are both lightweight network models.

[0133] The prediction module 803 is used to input the first feature and the second feature corresponding to each image sample into the classification model in the initial distraction detection model to obtain the driving state prediction result corresponding to each image sample.

[0134] The updating module 804 is used to update the network parameters of the initial distraction detection model according to the driving state prediction results corresponding to each image sample and the labels corresponding to each of the multiple image samples.

[0135] Optionally, the prediction module 803 is specifically used for:

[0136] For each image sample, the first feature and the second feature corresponding to the image sample are fused to obtain a fused feature.

[0137] The fused features are input into multiple dilated convolutional layers in the distraction detection model to obtain multi-scale features, and the driving state prediction results corresponding to the image samples are obtained based on the multi-scale features and the classification model.

[0138] Optionally, the prediction module 803 is specifically used for:

[0139] The multi-scale features are pooled by the maximum pooling method to obtain processed multi-scale features; the processed multi-scale features are fused to obtain global scale features.

[0140] The global scale features are input into the classification model to obtain the driving state prediction results corresponding to the image samples.

[0141] Optionally, the dilation rates corresponding to the multiple dilated convolutional layers are different.

[0142] Optionally, the update module 804 is specifically used to:

[0143] According to the driving state prediction results and labels corresponding to each image sample, a loss function corresponding to each image sample is constructed.

[0144] According to the loss function corresponding to each image sample, the network parameters of the initial distraction detection model are updated.

[0145] The training device 80 of the distraction detection model shown in the embodiment of the present application can execute the technical solution of the training method of the distraction detection model in the above-mentioned embodiment. Its implementation principle and beneficial effects are similar to the implementation principle and beneficial effects of the training method of the distraction detection model. Please refer to the implementation principle and beneficial effects of the training method of the distraction detection model, which will not be repeated here.

[0146] Fig. 9 This is a structural diagram of a distraction detection device 90 provided in an embodiment of the present application. For example, see Fig. 9 As shown, the distraction detection device 90 includes:

[0147] The acquisition module 901 is used to acquire the image to be detected.

[0148] The detection module 902 is used to input the image to be detected into a pre-trained distraction detection model to obtain the driving state of the driver, and the driving state includes a normal driving state and a distracted driving state; wherein the distraction detection model is to input multiple image samples into the first feature extraction model and the second feature extraction model in the initial distraction detection model respectively to obtain the first feature and the second feature, and input the first feature and the second feature into the classification model in the initial distraction detection model, and after obtaining the driving state prediction result corresponding to each image sample, the first feature extraction model and the second feature extraction model are obtained according to the driving state prediction result corresponding to each image sample and the labels corresponding to each of the multiple image samples, and the labels are used to represent the driving state of the driver in the image sample.

[0149] Optionally, the detection module 902 is specifically configured to:

[0150] The image to be detected is input into the first feature extraction model and the second feature extraction model of the distraction detection model to obtain the first feature and the second feature corresponding to the image to be detected.

[0151] The first feature and the second feature are fused to obtain a fused feature.

[0152] The fused features are input into multiple dilated convolutional layers in a distraction detection model to obtain multi-scale features, and the driving state of the driver is obtained based on the multi-scale features and the classification model.

[0153] Optionally, the detection module 902 is specifically configured to:

[0154] The multi-scale features are pooled by the maximum pooling method to obtain processed multi-scale features; the processed multi-scale features are fused to obtain global scale features.

[0155] The global scale features are input into the classification model to obtain the driver's driving state.

[0156] The distraction detection device 90 shown in the embodiment of the present application can execute the technical solution of the distraction detection method in the above embodiment. Its implementation principle and beneficial effects are similar to the implementation principle and beneficial effects of the distraction detection method. Please refer to the implementation principle and beneficial effects of the distraction detection method, and no further details will be given here.

[0157] Fig.10 This is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application, for example, see Fig.10 As shown, the electronic device 100 may include a processor 1001 and a memory 1002; wherein,

[0158] The memory 1002 is used to store computer programs.

[0159] The processor 1001 is used to read the computer program stored in the memory 1002, and execute the distraction detection model training method in the above embodiment according to the computer program in the memory 1002, or execute the distraction detection method in the above embodiment.

[0160] Optionally, the memory 1002 may be independent or integrated with the processor 1001. When the memory 1002 is a device independent of the processor 1001, the electronic device 100 may further include: a bus for connecting the memory 1002 and the processor 1001.

[0161] Optionally, this embodiment further includes: a communication interface, which can be connected to the processor 1001 through a bus. The processor 1001 can control the communication interface to implement the acquisition and transmission functions of the electronic device 100.

[0162] For example, in the embodiment of the present application, the electronic device 100 may be a terminal or a server, and may be specifically configured according to actual needs.

[0163] The electronic device 100 shown in the embodiment of the present application can execute the technical solution of the distraction detection model training method in the above embodiment, and its implementation principle and beneficial effects are similar to the implementation principle and beneficial effects of the distraction detection model training method, and reference can be made to the implementation principle and beneficial effects of the distraction detection model training method, or execute the technical solution of the distraction detection method in the above embodiment, and its implementation principle and beneficial effects are similar to the implementation principle and beneficial effects of the distraction detection method, and reference can be made to the implementation principle and beneficial effects of the distraction detection method, which will not be repeated here.

[0164] An embodiment of the present application further provides a computer-readable storage medium, which stores computer-executable instructions. When a processor executes the computer-executable instructions, the technical solution of the distraction detection model training method in the above embodiment is implemented, and its implementation principle and beneficial effects are similar to the implementation principle and beneficial effects of the distraction detection model training method. Please refer to the implementation principle and beneficial effects of the distraction detection model training method. Alternatively, the technical solution of the distraction detection method in the above embodiment is implemented, and its implementation principle and beneficial effects are similar to the implementation principle and beneficial effects of the distraction detection method. Please refer to the implementation principle and beneficial effects of the distraction detection method, and no further details will be given here.

[0165] An embodiment of the present application further provides a computer program product, including a computer program. When the computer program is executed by a processor, the computer program implements the technical solution of the distraction detection model training method in the above embodiment, and its implementation principle and beneficial effects are similar to the implementation principle and beneficial effects of the distraction detection model training method, and reference may be made to the implementation principle and beneficial effects of the distraction detection model training method, or implements the technical solution of the distraction detection method in the above embodiment, and its implementation principle and beneficial effects are similar to the implementation principle and beneficial effects of the distraction detection method, and reference may be made to the implementation principle and beneficial effects of the distraction detection method, which will not be repeated here.

[0166] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0167] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0168] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0169] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0170] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0171] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0172] The computer-readable storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium may be any available medium that can be accessed by a general or special purpose computer.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training a distraction detection model, characterized in that: include: Acquire a plurality of image samples and labels corresponding to the plurality of image samples, wherein the labels are used to represent a driving state of a driver in the image samples, wherein the driving state includes a normal driving state and a distracted driving state; Inputting the plurality of image samples into a first feature extraction model and a second feature extraction model in an initial distraction detection model respectively to obtain a first feature and a second feature corresponding to each image sample, wherein the first feature extraction model and the second feature extraction model are both lightweight network models; Inputting the first feature and the second feature corresponding to each image sample into the classification model in the initial distraction detection model to obtain a driving state prediction result corresponding to each image sample; updating the network parameters of the initial distraction detection model according to the driving state prediction results corresponding to the image samples and the labels corresponding to the plurality of image samples; The step of inputting the first feature and the second feature corresponding to each image sample into the classification model in the initial distraction detection model to obtain the driving state prediction result corresponding to each image sample includes: For each of the image samples, fusing the first feature and the second feature corresponding to the image sample to obtain a fused feature; The fused features are input into a plurality of dilated convolutional layers in the initial distraction detection model to obtain multi-scale features, and the driving state prediction result corresponding to the image sample is obtained based on the multi-scale features and the classification model.

2. The method according to claim 1, characterized in that The obtaining, according to the multi-scale features and the classification model, the driving state prediction result corresponding to the image sample comprises: Performing pooling processing on the multi-scale features by a maximum pooling method to obtain processed multi-scale features; Fusing the processed multi-scale features to obtain global scale features; The global scale feature is input into the classification model to obtain the driving state prediction result corresponding to the image sample.

3. The method according to claim 1 or 2, characterized in that: The dilation rates corresponding to the multiple dilated convolutional layers are different.

4. The method according to claim 1 or 2, characterized in that: The updating of the network parameters of the initial distraction detection model according to the driving state prediction results corresponding to each image sample and the label includes: Constructing a loss function corresponding to each image sample according to the driving state prediction result corresponding to each image sample and the label; According to the loss function corresponding to each image sample, the network parameters of the initial distraction detection model are updated.

5. A distraction detection method, characterized in that: include: Acquire the image to be detected; Input the image to be detected into a pre-trained distraction detection model to obtain the driving state of the driver, wherein the driving state includes a normal driving state and a distracted driving state; wherein the distraction detection model is obtained by inputting a plurality of image samples into a first feature extraction model and a second feature extraction model in an initial distraction detection model to obtain a first feature and a second feature, and inputting the first feature and the second feature into a classification model in the initial distraction detection model to obtain a driving state prediction result corresponding to each image sample, and then obtaining the driving state prediction result corresponding to each image sample and a label corresponding to each of the plurality of image samples, wherein the first feature extraction model and the second feature extraction model are both lightweight network models, and the label is used to represent the driving state of the driver in the image sample; The step of inputting the image to be detected into a pre-trained distraction detection model to obtain the driving state of the driver includes: Inputting the image to be detected into the first feature extraction model and the second feature extraction model of the distraction detection model to obtain the first feature and the second feature corresponding to the image to be detected; Fusing the first feature and the second feature to obtain a fused feature; The fused features are input into a plurality of dilated convolutional layers in the distraction detection model to obtain multi-scale features, and the driving state of the driver is obtained based on the multi-scale features and the classification model.

6. The method according to claim 5, characterized in that The obtaining the driving state of the driver according to the multi-scale features and the classification model includes: Performing pooling processing on the multi-scale features by a maximum pooling method to obtain processed multi-scale features; Fusing the processed multi-scale features to obtain global scale features; The global scale feature is input into the classification model to obtain the driving state of the driver.

7. An electronic device, characterized in that: include: Memory and processor; The memory is used to store computer programs; The processor is used to read the computer program stored in the memory and execute the distraction detection model training method described in any one of claims 1 to 4 according to the computer program in the memory, or to execute the distraction detection method described in claim 5 or 6.

8. A readable storage medium having a computer program stored thereon, characterized in that: The computer program stores computer executable instructions, and when the computer executable instructions are executed by a processor, they are used to implement the training method of the distraction detection model as described in any one of claims 1 to 4 above, or to implement the distraction detection method as described in claim 5 or 6 above.

Citation Information

Patent Citations

  • Method and device for detecting distraction of driver

    CN110575163A

  • Driver distraction detection method

    CN111626186A