Driver state detection method and system based on deep learning

Abstract

The invention discloses a driver state detection method and system based on deep learning. The method comprises the following steps: constructing an improved YOLOv11s target detection model; setting parameters of the YOLOv11s model, and inputting the training set into the YOLOv11s model for training, so as to obtain an optimized YOLOv11s model; preprocessing the driving state data; and inputting the preprocessed driving state data into the optimized YOLOv11s model, and obtaining the state of the current driver. According to the method, multi-scale convolution and an improved C2PSAMLCA module are integrated through a YOLOv11s model, multi-scale feature extraction and fusion are enhanced, and the feature capture capability is enhanced; the fusion layer and the detection head layer by layer detect different targets, and the detection precision is improved by combining comprehensive loss function optimization; the method can efficiently obtain the state of the driver, is suitable for different driving scenes, and improves the accuracy and reliability of driving safety monitoring.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of driver state detection, and relates to a driver state detection method and system based on deep learning. BACKGROUND

[0002] With the rapid development of global transportation industry, the number of motor vehicles continues to grow, and traffic safety problems are increasingly prominent. Although traditional traffic safety measures can reduce the accident rate to a certain extent, they still cannot effectively deal with sudden accidents caused by abnormal driver state. Especially in high-risk scenarios such as long-time driving, night driving, and single driving, the mental state of the driver is prone to fluctuation, which may cause traffic accidents.

[0003] Currently, driver state detection mainly focuses on fatigue detection, attention detection, and emotion recognition. Traditional fatigue detection methods are mostly based on questionnaire surveys and somatosensory feedback, which are highly subjective and have poor real-time performance, and are difficult to apply to dynamic driving environments. With the rapid development of computer science, researchers have found that the fusion of image features and physiological features can effectively improve the accuracy of fatigue judgment, but the equipment is complex, requiring a camera and multiple physiological signal acquisition devices, which is costly and inconvenient to wear. At the same time, the data fusion algorithm is complex, and it is difficult to determine the weight of different feature data, which may affect the detection accuracy. The fusion of image features and driver behavior features is easily affected by the environment, and the image is easily disturbed by light. Driving behavior is affected by road conditions and driving habits, and individual differences limit the generalization ability of the model. In addition, a large amount of data is required to train the model, which requires high computing resources. The fusion of driver behavior features and physiological features has the disadvantage of inconvenient wearing of physiological signal acquisition devices, which affects the driving experience, and the behavior features are easily disturbed by the environment, making data fusion difficult, and the adaptability of the model to complex situations needs to be improved. Therefore, there is an urgent need for a method that can improve the accuracy of driver state detection. SUMMARY

[0004] The purpose of the present application is to solve the problems of complex data fusion algorithm, low detection accuracy, and easy environmental disturbance caused by the mutual fusion of image features, physiological features, and driver behavior features in the prior art, and to provide a driver state detection method and system based on deep learning.

[0005] To achieve the above purpose, the following technical solutions are adopted:

[0006] The driver state detection method based on deep learning comprises:

[0007] constructing a driver state detection model based on an improved YOLOv11s;

[0008] obtaining a public data set and dividing it to obtain a training set and a test set;

[0009] Set the driver state detection model parameter based on the improved YOLOv11s, and input the training set into the improved YOLOv11s model for training;

[0010] Test the trained YOLOv11s model based on the test set, if the preset threshold condition is passed, the optimized improved YOLOv11s model is obtained;

[0011] Collect the driving state data of the driver, and pretreat the driving state data;

[0012] The pretreated driving state data is input into the optimized improved YOLOv11s model, and the state of the current driver is obtained.

[0013] Further improvement of the application is:

[0014] Further, the driver state detection model based on the improved YOLOv11s is constructed, specifically:

[0015] The improved YOLOv11s includes an input end, a backbone network, a fusion layer and a detection head;

[0016] The input end is used for receiving and pretreating original image data;

[0017] The backbone network includes a convolution layer, a C3k2 module, a SPPF module and an improved C2PSA_MLCA module;

[0018] The convolution layer performs sliding convolution through different size convolution kernels, extracts local features of the image, and realizes down sampling through convolution with a step of 2, reduces the size of the feature map and increases the number of channels;

[0019] The C3k2 module introduces 3*3 and 5*5 multi-scale convolution kernels, divides the input features into two parts: one part is directly transmitted through CBS convolution, and the other part is deep feature extraction through multiple Bottleneck structures, finally the two parts of features are spliced and fused through a single convolution kernel;

[0020] The SPPF module fuses multi-scale features through different scale maximum pooling operations, and captures the features of targets of different sizes;

[0021] The C2PSA_MLCA module integrates a multi-scale lightweight channel attention mechanism MLCA; MLCA extracts multi-scale features through 1*1 and 3*3 parallel convolution, and fuses local and global features, channel and spatial features;

[0022] The fusion layer splices the feature maps from different stages of the backbone network in the channel dimension through a Concat operation; meanwhile, the fusion layer performs up-sampling processing on the low-resolution feature maps to restore their size to the same level as other feature maps; after the fusion of the feature maps, the C3k2 module is used to perform convolution processing on the fused features to remove noise and redundant information.

[0023] The detection head adopts a feature pyramid network (FPN) architecture to detect the feature maps from different stages output by the fusion layer; after several rounds of up-sampling, splicing, and feature processing operations, the Detect layer receives feature maps at different levels and outputs detection results for different types of targets to accurately detect different types of targets in the image.

[0024] Further, the hierarchical structure and parameter configuration of the backbone network are as follows:

[0025] The image is input to the convolution layer for processing, with an output channel of 64, a convolution kernel of 3x3, and a step of 2 for down-sampling to obtain a P1 feature map with a size of 1 / 2 of the input.

[0026] The P1 feature map is input to the convolution layer for processing, with an output channel of 128 and a step of 2 for down-sampling to obtain a P2 feature map with a size of 1 / 4 of the input.

[0027] The P2 feature map is processed by two C3k2 modules to output a channel of 256; the P2 feature map processed by the C3k2 module is down-sampled by the convolution layer to output a channel of 256 to obtain a P3 feature map with a size of 1 / 8 of the input.

[0028] The P3 feature map is processed by two C3k2 modules to output a channel of 512; the P3 feature map processed by the C3k2 module is down-sampled by the convolution layer to output a channel of 512 to obtain a P4 feature map with a size of 1 / 16 of the input.

[0029] The P4 feature map is processed by two C3k2 modules, and the processed P4 feature map is down-sampled by the convolution layer to obtain a P5 feature map.

[0030] The P5 feature map is processed by two C3k2 modules, and the processed P5 feature map is sequentially processed by the SPPF module and two improved C2PSA_MLCA modules to improve the detection capability of the model.

[0031] Further, the fusion layer splices the feature maps from different stages of the backbone network in the channel dimension through a Concat operation, specifically as follows:

[0032] The P5 feature map output by the backbone network is first enlarged in size by upsampling, and is spliced with the P4 level feature in the backbone network through Concat; the spliced feature is processed by the C3k2 module to obtain the optimized P4 level fusion feature;

[0033] The optimized P4 level fusion feature is again upsampled, and is spliced with the P3 level feature in the backbone network through Concat, and the P3 level fusion feature is obtained after processing by the C3k2 module;

[0034] Meanwhile, the P3 level fusion feature is downsampled by a convolution layer with a step of 2, and is spliced with the part of the P4 level fusion feature that is not upsampled through Concat, and the optimized P4 level output feature is obtained after processing by the C3k2 module;

[0035] The optimized P4 level output feature is further downsampled by a convolution layer, and is spliced with the P5 feature map of the deepest feature in the backbone network through Concat, and the P5 level output feature is obtained after processing by the C3k2 module;

[0036] Through several rounds of upsampling, splicing and optimization, the fusion layer finally outputs P3, P4 and P5 feature maps of three scales, which adapt to the detection needs of different types of targets.

[0037] Further, the loss function of the YOLOv11s model includes a classification loss, a boundary loss and a confidence loss; wherein the boundary loss is used to predict the difference between the boundary box and the true boundary box, and the formula is:

[0038]

[0039] Wherein S represents the size of the grid; B is the number of boundary boxes predicted by each grid unit; represents whether the jth boundary box in the ith grid unit meets the responsible prediction target; x, y are the coordinates of the center point of the boundary box; w, h are the width and height of the boundary box; a coord is a weight coefficient for balancing the loss of different parts; is the loss of the center coordinates, which measures the square of the Euclidean distance between the predicted boundary box center point and the true boundary box center point, and the purpose is to make the predicted boundary box center as close as possible to the center of the true boundary box; is the difference between the width and height of the predicted boundary box and the width and height of the true boundary box, and taking the square root of the width and height reduces the influence of the size of the boundary box;

[0040] The classification loss is used to optimize the part of the model that accurately predicts the target class; the classification loss ensures that the model correctly identifies the object class in the image, and the formula is:

[0041]

[0042] where p i (c) is the probability of the target belonging to class c in the i-th grid cell predicted by the model; is the true label, indicating whether the i-th grid cell belongs to class c; the classification loss ensures that the model can identify the class to which the target in the image belongs;

[0043] The confidence loss is to solve the class imbalance problem in target detection and improve the performance of the model in processing small targets and difficult samples; the formula is:

[0044]

[0045] where N is the number of samples; C is the number of classes; y ic is the true label of the i-th sample; p ic is the predicted probability of the i-th sample belonging to class c; α is the balance factor, used to adjust the weight between positive and negative samples; γ is the focus parameter, used to control the attention degree to difficult samples;

[0046] The loss function of the YOLOv11s model is the sum of the classification loss, the boundary loss and the confidence loss; the smaller the loss function of the YOLOv11s model, the closer the prediction result of the YOLOv11s model to the true label.

[0047] Further, the disclosed dataset is the beh Computer Vision Project dataset in the Roboflow platform;

[0048] The image is labeled, and the label format is: class number x_center, y_center, width and height; where the class number is from 0 to 3, 0 represents a face, 1 represents drinking water, 2 represents using a mobile phone, and 3 represents smoking; x_center, y_center, width, height are the center coordinates of the target frame and the width and height, and these values are all relative to the width and height of the image;

[0049] The training set is input into the improved YOLOv11s model for training, specifically:

[0050] By comparing the prediction result with the labeled class and boundary box, the boundary box loss, the classification loss and the confidence loss are calculated, and the network parameters are continuously optimized through back propagation; until the optimal improved YOLOv11s model is obtained through the preset threshold condition.

[0051] Further, the target detection model parameters of the YOLOv11s include: picture size, iteration number, image and data set processed per batch input during training;

[0052] If the preset threshold condition is met, the optimized YOLOv11s model is obtained, specifically: the threshold condition of the YOLOv11s model includes: precision, recall and average precision; when the result output by the YOLOv11s model is greater than the precision, recall or average precision, the training is stopped, and the optimized YOLOv11s model is obtained; if not, continue to train the YOLOv11s model until the preset condition is met; or when the iteration number of training exceeds the preset number, the training is stopped, and the optimized YOLOv11s model is obtained.

[0053] Further, the driving state data is preprocessed, specifically: the collected images are cleaned, the images with blurred / invalid frames are removed, and the images are subjected to brightness adjustment, rotation and flipping operations to complete image enhancement; the enhanced images are uniformly adjusted to a fixed size and then subjected to normalization operation, and the pixel value of the image is scaled to a specific range [0, 1].

[0054] Further, the driving state data after preprocessing is input into the improved YOLOv11s model optimized to obtain the current driver state, specifically: the driving state data after preprocessing is input into the optimized YOLOv11s model, and the bounding box coordinates [x, y, w, h] containing the face position are output to crop the face area; after positioning the cropped face area, a plurality of feature points including eyes and mouth are extracted; wherein 6 key points are extracted for each eye; 8 to 12 key points are extracted for the mouth area; all key point coordinates are in 2D format; if the target is an eye or a mouth, the eye aspect ratio EAR and the mouth aspect ratio MAR are calculated; wherein when the eye aspect ratio EAR is less than the threshold value 0.2, it indicates that the eyes are closed, which may be a sign of drowsiness or fatigue; when the mouth aspect ratio MAR is greater than the threshold value 0.5, it indicates that the mouth is open, which may be a sign of yawning or fatigue.

[0055] The driver state detection system based on deep learning comprises:

[0056] The construction module constructs the driver state detection model based on the improved YOLOv11s;

[0057] The division module obtains a public data set and divides it to obtain a training set and a test set;

[0058] A training module sets driver state detection model parameters based on the improved YOLOv11s, and inputs a training set into the improved YOLOv11s model for training.

[0059] A testing module tests the trained YOLOv11s model based on a test set, and obtains an optimized improved YOLOv11s model if a preset threshold condition is passed.

[0060] A preprocessing module collects driver driving state data and pre-processes the driving state data.

[0061] An obtaining module inputs the pre-processed driving state data into the optimized improved YOLOv11s model, and obtains the state of a current driver.

[0062] Compared with the prior art, the present application has the following beneficial effects:

[0063] The present application can efficiently obtain the state of a driver, adapt to different driving scenes, and improve the accuracy and reliability of driving safety monitoring. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0065] Figure 1 The figure is a flowchart of the driver state detection method based on deep learning of the present application.

[0066] Figure 2 The figure is a structural diagram of the driver state detection system based on deep learning of the present application.

[0067] Figure 3 The figure is a training diagram of the improved YOLOv11s model. DETAILED DESCRIPTION

[0068] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some but not all of the embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0069] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0070] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0071] In the description of the embodiments of the present application, it should be noted that, if the orientation or position relationship indicated by the terms "upper", "lower", "horizontal", "inner" and the like is based on the orientation or position relationship shown in the drawings, or is the orientation or position relationship when the product of the present application is usually placed, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, therefore, it cannot be understood as a limitation on the present application. In addition, the terms "first", "second" and the like are only used for differentiation in description, and cannot be understood as indicating or implying relative importance.

[0072] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly inclined. For example, "horizontal" only means that its direction is relatively more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined.

[0073] In the description of the embodiments of the present application, it should also be noted that, unless otherwise explicitly specified and limited, if the terms "arrange", "mount", "connect", "connect" appear, they should be understood in a broad sense, for example, can be fixedly connected, can be detachably connected, or integrally connected; can be mechanically connected, can be electrically connected; can be directly connected, can be indirectly connected through an intermediate medium, can be the communication inside two elements. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0074] The present application will be described in further detail below with reference to the drawings:

[0075] The application discloses a driver state detection method based on deep learning, comprising:

[0076] S101, collecting driving state data of a driver, and preprocessing the driving state data;

[0077] The collected images are cleaned to remove images of blur / invalid frames, and brightness adjustment, rotation and flipping operations are performed on the images to complete image enhancement; after the enhanced images are uniformly adjusted to a fixed size, normalization operation is performed, and the pixel values of the images are scaled to a specific range [0, 1].

[0078] S101, constructing a driver state detection model based on an improved YOLOv11s;

[0079] The improved YOLOv11s comprises an input end, a backbone network, a fusion layer and a detection head.

[0080] The input end is used for receiving and preprocessing original image data.

[0081] The backbone network comprises a convolution layer, a C3k2 module, an SPPF module and an improved C2PSA_MLCA module.

[0082] The convolution layer performs sliding convolution through convolution kernels of different sizes to extract local features of the images, and realizes down-sampling through convolution with a step of 2 to reduce the size of the feature maps and increase the number of channels.

[0083] In the C3k2 module, 3x3 and 5x5 multi-scale convolution kernels are introduced to divide the input features into two parts: one part is directly transmitted through CBS convolution, and the other part is subjected to deep feature extraction through multiple Bottleneck structures; finally, the two parts of features are spliced and fused through a single convolution kernel.

[0084] The SPPF module fuses multi-scale features through different scale maximum pooling operations to capture features of targets of different sizes.

[0085] The C2PSA_MLCA module integrates a multi-scale lightweight channel attention mechanism MLCA; the MLCA extracts multi-scale features through 1x1 and 3x3 parallel convolution to fuse local and global features and channel and spatial features.

[0086] The fusion layer splices feature maps from different stages of the backbone network in the channel dimension through a Concat operation; meanwhile, the fusion layer implements up-sampling processing on the low-resolution feature maps through an Upsample operation to restore the size of the low-resolution feature maps to the same level as that of other feature maps; after the fusion of the feature maps, the C3k2 module is used to perform convolution processing on the fused features to remove noise and redundant information.

[0087] The detection head adopts a feature pyramid network (FPN) architecture to detect feature maps at different stages output by the fusion layer; after several rounds of upsampling, splicing, and feature processing operations, the Detect layer receives feature maps at different levels and outputs detection results for different types of targets to accurately detect different types of targets in the image.

[0088] The hierarchical structure and parameter configuration of the backbone network are as follows:

[0089] The image is input to a convolutional layer for processing, with an output channel of 64, a convolutional kernel of 3x3, and a step size of 2 for downsampling to obtain a P1 feature map with a size of 1 / 2 of the input;

[0090] The P1 feature map is input to a convolutional layer for processing, with an output channel of 128 and a step size of 2 for downsampling to obtain a P2 feature map with a size of 1 / 4 of the input;

[0091] The P2 feature map is processed through two C3k2 modules with an output channel of 256; the P2 feature map processed through the C3k2 modules is downsampled through a convolutional layer with an output channel of 256 to obtain a P3 feature map with a size of 1 / 8 of the input;

[0092] The P3 feature map is processed through two C3k2 modules with an output channel of 512; the P3 feature map processed through the C3k2 modules is downsampled through a convolutional layer with an output channel of 512 to obtain a P4 feature map with a size of 1 / 16 of the input;

[0093] The P4 feature map is processed through two C3k2 modules, and the processed P4 feature map is downsampled through a convolutional layer to obtain a P5 feature map;

[0094] The P5 feature map is processed through two C3k2 modules, and the processed P5 feature map is sequentially processed through an SPPF module and two improved C2PSA_MLCA modules to improve the detection capability of the model.

[0095] The fusion layer splices the feature maps from different stages of the backbone network in the channel dimension through a Concat operation, specifically as follows:

[0096] The P5 feature map output by the backbone network is first upscaled in size and spliced with the P4-level feature in the backbone network through Concat; the spliced feature is processed through a C3k2 module to obtain an optimized P4-level fusion feature;

[0097] The optimized P4-level fusion feature is upscaled again and spliced with the P3-level feature in the backbone network through Concat, and the P3-level fusion feature is obtained after processing through a C3k2 module;

[0098] At the same time, the P3 level fusion feature is down-sampled by a convolution layer with a step of 2, spliced with the part not up-sampled in the P4 level fusion feature through Concat, and processed by the C3k2 module to obtain the optimized P4 level output feature;

[0099] The optimized P4 level output feature is down-sampled by a convolution layer, spliced with the P5 feature map which is the deepest layer feature of the backbone network through Concat, and processed by the C3k2 module to obtain the P5 level output feature;

[0100] Through several rounds of up-sampling, splicing and optimization, the fusion layer finally outputs the feature maps of P3, P4 and P5 three scales, which adapt to the detection needs of different types of targets.

[0101] The loss function of the YOLOv11s model includes classification loss, boundary loss and confidence loss; wherein,

[0102] The boundary loss is used to predict the difference between the predicted boundary box and the real boundary box, and the formula is:

[0103]

[0104] Wherein, S represents the size of the grid; B is the number of boundary boxes predicted by each grid unit; indicates whether the jth boundary box in the ith grid unit meets the responsible prediction target; x, y are the coordinates of the center point of the boundary box; w, h are the width and height of the boundary box; α coord is a weight coefficient for balancing the loss of different parts; is the loss of the center coordinates, which measures the square of the Euclidean distance between the predicted boundary box center point and the real boundary box center point, and the purpose is to make the predicted boundary box center as close as possible to the center of the real boundary box; is the difference between the width and height of the predicted boundary box and the width and height of the real boundary box, and taking the square root of the width and height is to reduce the influence of the size of the boundary box;

[0105] The classification loss is used to optimize the part of the model that predicts the target class accurately; the classification loss ensures that the model correctly identifies the object class in the image, and the formula is:

[0106]

[0107] Wherein, p i (c) is the probability of the target belonging to class c in the ith grid unit predicted by the model; is the real label, indicating whether it belongs to class c in the ith grid unit; the classification loss ensures that the model can identify the class to which the target in the image belongs;

[0108] The confidence loss is used to solve the class imbalance problem in target detection and improve the performance of the model in processing small targets and difficult samples.

[0109]

[0110] wherein N is the number of samples; C is the number of classes; y ic is the true label of the i-th sample; p ic is the predicted probability of the i-th sample belonging to class c; alpha is a balance factor used to adjust the weight between positive and negative samples; gamma is a focus parameter used to control the degree of attention to difficult samples.

[0111] The loss function of the YOLOv11s model is the sum of the classification loss, the boundary loss and the confidence loss; the smaller the loss function of the YOLOv11s model, the closer the prediction result of the YOLOv11s model to the true label.

[0112] S102, obtaining a public dataset and dividing it to obtain a training set and a test set;

[0113] The public dataset is the beh ComputerVision Project dataset in the Roboflow platform.

[0114] The image is labeled, and the label format is: class number x_center, y_center, width and height; wherein the class number is from 0 to 3, 0 represents a face, 1 represents drinking water, 2 represents using a mobile phone, and 3 represents smoking; x_center, y_center, width and height are the center coordinates of the target frame and the width and height, and these values are all relative to the width and height of the image.

[0115] S103, setting the driver state detection model parameter based on the improved YOLOv11s, and inputting the training set into the improved YOLOv11s model for training;

[0116] The target detection model parameter of the YOLOv11s includes: the picture size input during training, the iteration number, the image and dataset processed per batch.

[0117] The training set is input into the improved YOLOv11s model for training, specifically:

[0118] By comparing the prediction result with the labeled class and boundary box, the boundary box loss, the classification loss and the confidence loss are calculated, and the network parameters are continuously optimized through back propagation; until the optimal improved YOLOv11s model is obtained through the preset threshold condition.

[0119] S104, test the trained YOLOv11s model based on the test set, and if a preset threshold condition is passed, an optimized improved YOLOv11s model is obtained;

[0120] If the preset threshold condition is passed, the optimized improved YOLOv11s model is obtained, specifically: the threshold condition of the YOLOv11s model includes: precision, recall and average precision mean; when the result output by the YOLOv11s model is greater than the precision, recall or average precision mean, the training is stopped, and the optimized YOLOv11s model is obtained; if not satisfied, continue to train the YOLOv11s model until the preset condition is met.

[0121] S105, collecting driving state data of the driver and preprocessing the driving state data;

[0122] The collected images are cleaned to remove blurred / invalid frame images, and the images are subjected to brightness adjustment, rotation, and flipping operations to complete image enhancement; the enhanced images are uniformly adjusted to a fixed size and then normalized to scale the pixel values of the images to a specific range [0, 1].

[0123] S106, input the preprocessed driving state data into the optimized improved YOLOv11s model to obtain the current state of the driver.

[0124] The preprocessed driving state data is input into the optimized improved YOLOv11s model, and the bounding box coordinates [x, y, w, h] containing the face position are output to crop the face region; after positioning the cropped face region, a plurality of feature points including eyes and mouth are extracted; wherein 6 key points are extracted for each eye; 8 to 12 key points are extracted for the mouth region; all key point coordinates are in 2D format; if the target is an eye or a mouth, the eye aspect ratio EAR and the mouth aspect ratio MAR are calculated; wherein when the eye aspect ratio EAR is less than a threshold value 0.2, the eye is closed, which may be a sign of drowsiness or fatigue; when the mouth aspect ratio MAR is greater than a threshold value of 0.5, the mouth is open, which may be a sign of yawning or fatigue.

[0125] Referring to Figure 2 The application discloses a driver state detection system based on deep learning, comprising:

[0126] A construction module constructs a driver state detection model based on an improved YOLOv11s;

[0127] A division module obtains a public data set and divides it to obtain a training set and a test set;

[0128] The training module sets the driver state detection model parameter based on the improved YOLOv11s, and inputs the training set into the improved YOLOv11s model for training.

[0129] The test module tests the trained YOLOv11s model based on the test set, and if the preset threshold condition is passed, the optimized improved YOLOv11s model is obtained.

[0130] The preprocessing module collects the driving state data of the driver and pre-processes the driving state data.

[0131] The acquisition module inputs the pre-processed driving state data into the optimized improved YOLOv11s model to acquire the state of the current driver.

[0132] Embodiment:

[0133] The application discloses a driver state detection method based on deep learning, which comprises the following steps: a Roboflow platform is used as a professional data resource platform, image data with behaviors of drinking, smoking and playing mobile phones is labeled in a beh Computer Vision Project data set in the Roboflow platform, and the data set comprises 5052 images of a training set and 1445 images of a test set.

[0134] The image is scaled to a fixed size of 640*640 pixels. When scaling, the aspect ratio of the image needs to be maintained to prevent the image from being deformed. If the image is deformed, the shape and proportion of the object may change, thereby affecting the learning and identification of the object features by the model. After the image scaling is completed, normalization processing is required, and the pixel value is normalized to the range of [0, 1] by applying random brightness adjustment (±20%), contrast adjustment (±15%), horizontal flip (probability 0.5) and other enhancement operations. Normalization processing is helpful for the training and convergence of the model, because the normalized data can make the gradient update process of the model more stable.

[0135] Before carrying out the model training work, it is necessary to convert the data labeling into the YOLO format. The labeling format is: class number x_center y_center width height. Among them, the class number is an integer starting from 0, and in the system, there are four categories, the class number is from 0 to 3, 0 represents a face, 1 represents drinking, 2 represents using a mobile phone, and 3 represents smoking; x_center, y_center, width and height are the center point coordinates and width and height of the target box, and these values are all proportions relative to the width and height of the image,

[0136] The training parameters are set: the input image size is 640x640, the number of iterations is 300, the batch size is 16, the number of working threads is 4, and ImageNet standardization (mean = [0.485, 0.456, 0.406], standard deviation = [0.229, 0.224, 0.225]) is applied during model training. The training set input converted to BCHW format is input into the improved YOLOv11s model for training. The model continuously optimizes network parameters by comparing the predicted results with the labeled class and bounding box loss, classification loss, and confidence loss, and by backpropagation. Until the preset threshold condition is met, the optimized improved YOLOv11s model is obtained.

[0137] The beh Computer Vision Project dataset is used to divide the training set and test set, and the parameters are continuously adjusted. The improved YOLOv11s model is trained on the training set to obtain the best.pt optimal solution weight, as shown in Figure 3 . Figure 3 The training results of the model show the key information in the training of the target detection model. The three loss indicators of the training set and the test set: bounding box loss, classification loss, and confidence loss, all decrease with the training round, meaning that the model's prediction of bounding box position, identification of target class, and related performance optimization effect gradually improve. In the evaluation indicators, the precision, recall, and average precision (mAP50, mAP50-95) continue to rise, reflecting the steady improvement of the model's accuracy in target recognition, the ability to capture real targets, and the comprehensive detection performance under different intersection-over-union thresholds, showing that the model training tends to converge well.

[0138] After obtaining the optimized improved YOLOv11s model, the preprocessed driving state data is input into the optimized improved YOLOv11s model, and the bounding box coordinates [x, y, w, h] containing the face position are output to crop the face region. After positioning the cropped face region, several feature points including eyes and mouth are extracted. Among them, 6 key points are extracted for each eye, and 8 to 12 key points are extracted for the mouth region. All key point coordinates are in 2D format. If the target is an eye or a mouth, calculate the eye aspect ratio EAR and the mouth aspect ratio MAR. When the eye aspect ratio EAR is less than the threshold value 0.2, it indicates that the eyes are closed, which may be a sign of drowsiness or fatigue. When the mouth aspect ratio MAR is greater than the threshold value 0.5, it indicates that the mouth is open, which may be a sign of yawning or fatigue.

[0139] The above examples are only used to illustrate the technical solutions of the present application but not to limit the present application. Although the present application is described in detail with reference to the above examples, it should be understood by those skilled in the art that the specific embodiments of the present application can be modified or equivalently replaced without departing from the spirit and scope of the present application, and any modification or equivalent replacement should be covered within the protection scope of the claims of the present application.

Claims

1. A driver state detection method based on deep learning, characterized in that, include: Construct a driver state detection model based on the improved YOLOv11s; Obtain a publicly available dataset and split it into training and test sets; Set the parameters of the driver state detection model based on the improved YOLOv11s and input the training set into the improved YOLOv11s model for training; The trained YOLOv11s model is tested on the test set. If it passes the preset threshold condition, the optimized improved YOLOv11s model is obtained. Collect driver's driving status data and preprocess the driving status data; The preprocessed driving state data is input into the optimized and improved YOLOv11s model to obtain the current driver's state.

2. The driver state detection method based on deep learning according to claim 1, characterized in that, The construction of the driver state detection model based on the improved YOLOv11s is specifically as follows: The improved YOLOv11s includes an input terminal, a backbone network, a fusion layer, and a detection head; The input terminal is used to receive and preprocess raw image data; The backbone network includes convolutional layers, C3k2 modules, SPPF modules, and an improved C2PSA_MLCA ​​module; The convolutional layer performs sliding convolution using convolutional kernels of different sizes to extract local features of the image, while downsampling is achieved through convolution with a stride of 2 to reduce the size of the feature map and increase the number of channels; The C3k2 module introduces 3×3 and 5×5 multi-scale convolution kernels to divide the input features into two parts: one part is directly passed through CBS convolution, and the other part is extracted for deep features through multiple Bottleneck structures. Finally, the two parts of features are concatenated and fused through a single convolution kernel. The SPPF module fuses multi-scale features through max pooling operations at different scales to capture features of targets of different sizes. The C2PSA_MLCA ​​module integrates the multi-scale lightweight channel attention mechanism MLCA; MLCA extracts multi-scale features through 1×1 and 3×3 parallel convolutions, and fuses local and global features, channel and spatial features. The fusion layer uses the Concat operation to stitch feature maps from different stages of the backbone network along the channel dimension; at the same time, it uses the Upsample operation to upsample the low-resolution feature maps to restore their size to the same level as other feature maps; after fusing the feature maps, the C3k2 module performs convolution processing on the fused features to remove noise and redundant information. The detection head adopts the Feature Pyramid Network (FPN) architecture to detect feature maps at different stages output by the fusion layer. After several rounds of upsampling, stitching, and feature processing operations, the Detect layer receives feature maps at different levels and outputs detection results for different types of targets, thus accurately detecting different types of targets in the image.

3. The driver state detection method based on deep learning according to claim 2, characterized in that, The hierarchical structure and parameter configuration of the backbone network are as follows: The image is input into a convolutional layer for processing. The output channel is 64, the convolutional kernel is 3×3, and the stride is 2 for downsampling to obtain the P1 feature map, which is half the size of the input. The P1 feature map is input into a convolutional layer for processing. The output channel is 128, the stride is 2, and downsampling is performed to obtain the P2 feature map, which is 1 / 4 the size of the input. The P2 feature map is processed through two C3k2 modules, with an output of 256 channels; the P2 feature map after processing by the C3k2 modules is downsampled through a convolutional layer, with an output of 256 channels, to obtain the P3 feature map, which is 1 / 8 the size of the input. The P3 feature map is processed through two C3k2 modules, with an output channel of 512; the convolutional layer of the P3 feature map after processing by the C3k2 module is downsampled, with an output channel of 512, to obtain the P4 feature map, which is 1 / 16 of the input. The P4 feature map is processed through two C3k2 modules, and the processed P4 feature map is downsampled by a convolutional layer to obtain the P5 feature map. The P5 feature map is processed through two C3k2 modules, and the processed P5 feature map is then processed sequentially through the SPPF module and two improved C2PSA_MLCA ​​modules to improve the model's detection capability.

4. The driver state detection method based on deep learning according to claim 3, characterized in that, The fusion layer uses a Concat operation to concatenate feature maps from different stages of the backbone network along the channel dimension, specifically: The P5 feature map output by the backbone network is first upsampled to increase its size, and then concatenated with the features of the P4 layer in the backbone network using Concat. The concatenated features are then processed by the C3k2 module to obtain the optimized P4 layer fused features. The optimized P4-level fusion features are upsampled again and concatenated with the P3-level features in the backbone network using Concat. After processing by the C3k2 module, the P3-level fusion features are obtained. Meanwhile, the P3 level fusion features are downsampled through a convolutional layer with a stride of 2, and the unsampled parts of the P4 level fusion features are concatenated with the P4 level fusion features through Concat. The optimized P4 level output features are then processed by the C3k2 module. The optimized P4-level output features are then downsampled through a convolutional layer and concatenated with the deepest feature map of the backbone network, P5, using Concat. The P5-level output features are then processed by the C3k2 module. Through several rounds of upsampling, stitching, and optimization, the fusion layer finally outputs feature maps at three scales: P3, P4, and P5, which are adapted to the detection needs of different types of targets.

5. The driver state detection method based on deep learning according to claim 4, characterized in that, The loss function of the YOLOv11s model includes classification loss, boundary loss, and confidence loss; among them, the boundary loss is used to predict the difference between the bounding box and the ground truth bounding box, and the formula is: Where S represents the size of the grid; B is the number of bounding boxes predicted for each grid cell; This indicates whether the j-th bounding box in the i-th grid cell conforms to the assigned prediction target; x and y are the coordinates of the bounding box center point; w and h are the width and height of the bounding box; α coord These are weighting coefficients used to balance the losses of different parts; The loss is the center coordinate, which measures the square of the Euclidean distance between the center point of the predicted bounding box and the center point of the actual bounding box. The goal is to make the center of the predicted bounding box as close as possible to the center of the actual bounding box. It measures the difference between the width and height of the predicted bounding box and the width and height of the actual bounding box. Taking the square root of the width and height reduces the influence of the size bounding box. The classification loss is used to optimize the part of the model that accurately predicts the target category; the classification loss ensures that the model correctly identifies the object category in the image, and the formula is: Where, p i (c) is the probability that the target in the i-th grid cell belongs to category c, as predicted by the model; The label is the true label, indicating whether the i-th grid cell belongs to category c; the classification loss ensures that the model can identify the category to which the target in the image belongs; Confidence loss is used to address class imbalance in object detection and improve model performance when handling small targets and difficult samples; the formula is: Where N is the number of samples; C is the number of categories; y ic p is the true label of the i-th sample; ic α is the predicted probability that the i-th sample belongs to class c; α is the balance factor used to adjust the weights between positive and negative samples; γ is the focus parameter used to control the degree of attention given to difficult samples. The loss function of the YOLOv11s model is the sum of classification loss, boundary loss, and confidence loss; the smaller the loss function of the YOLOv11s model, the closer the prediction result of the YOLOv11s model is to the true label.

6. The driver state detection method based on deep learning according to claim 5, characterized in that, The publicly available dataset is the beh Computer Vision Project dataset from the Roboflow platform; The image is labeled with the following format: category number x_center, y_center, width, and height; where the category number ranges from 0 to 3, 0 represents a face, 1 represents drinking water, 2 represents using a mobile phone, and 3 represents smoking; x_center, y_center, width, and height are the center coordinates, width, and height of the target bounding box, respectively, and these values ​​are all relative to the width and height of the image. The process of inputting the training set into the improved YOLOv11s model for training specifically involves: By comparing the predicted results with the labeled categories and bounding boxes, the bounding box loss, classification loss, and confidence loss are calculated, and the network parameters are continuously optimized through backpropagation until the preset threshold condition is met, resulting in the optimal and improved YOLOv11s model.

7. The driver state detection method based on deep learning according to claim 6, characterized in that, The parameters of the YOLOv11s object detection model include: the size of the input image during training, the number of iterations, the images processed in each batch, and the dataset. The process of obtaining the optimized YOLOv11s model if the preset threshold conditions are met is as follows: The threshold conditions for the YOLOv11s model include precision, recall, and mean precision. When the output of the YOLOv11s model is greater than the precision, recall, or mean precision, training is stopped, and the optimized YOLOv11s model is obtained. If the conditions are not met, training of the YOLOv11s model continues until the preset conditions are met. Alternatively, if the number of training iterations exceeds a preset number, training is stopped, and the optimized YOLOv11s model is obtained.

8. The driver state detection method based on deep learning according to claim 1, characterized in that, The preprocessing of driving status data specifically involves: cleaning the acquired images, removing blurry / invalid frames, and performing brightness adjustment, rotation, and flipping operations on the images to enhance them; then, after uniformly adjusting the enhanced images to a fixed size, performing normalization operations to scale the pixel values ​​of the images to a specific range [0, 1].

9. The driver state detection method based on deep learning according to claim 8, characterized in that, The process of inputting preprocessed driving state data into an optimized and improved YOLOv11s model to obtain the current driver's state involves the following steps: The preprocessed driving state data is input into the optimized YOLOv11s model, which outputs bounding box coordinates [x, y, w, h] containing the face location. The face region is then cropped. After locating the cropped face region, several feature points, including the eyes and mouth, are extracted. Six key points are extracted for each eye, and 8 to 12 key points are extracted for the mouth region. All key point coordinates are in 2D format. If the target is the eyes or mouth, the eye aspect ratio (EAR) and mouth aspect ratio (MAR) are calculated. When the eye aspect ratio (EAR) is less than a threshold of 0.2, it indicates closed eyes, which may indicate drowsiness or fatigue. When the mouth aspect ratio (MAR) is greater than a threshold of 0.5, it indicates an open mouth, which may indicate yawning or fatigue.

10. A driver state detection system based on deep learning, characterized in that, include: A building module that constructs a driver state detection model based on an improved YOLOv11s; The partitioning module obtains the publicly available dataset and partitions it to obtain a training set and a test set; The training module sets the parameters of the driver state detection model based on the improved YOLOv11s and inputs the training set into the improved YOLOv11s model for training. The testing module tests the trained YOLOv11s model based on the test set. If the model passes the preset threshold condition, the optimized improved YOLOv11s model is obtained. A preprocessing module collects the driver's driving status data and preprocesses the driving status data. The acquisition module inputs the preprocessed driving state data into the optimized and improved YOLOv11s model to obtain the current driver's state.

Images