Fatigue driving detection method and system, readable storage medium and computer
Through video frame acquisition and face detection models, the driver's eye, mouth and head features are extracted, and the fatigue feature classification model is constructed, which solves the problems of invasiveness and high cost of equipment in the existing technology, and achieves non-invasiveness , high-precision fatigue driving detection.
Patent Information
- Application Number
- CN202411112976.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-11-15
AI Technical Summary
The existing fatigue driving detection methods have problems such as strong invasiveness, high cost and difficult to popularize equipment, and it is difficult to find universal judgment standards based on the driving status of the vehicle.
Using video frame acquisition and face detection models, the eyes, mouth and head features are extracted through the face key point detection model, and a fatigue feature classification model is constructed to predict the driver's fatigue state.
It realizes non-invasive, high-precision and cost-effective fatigue driving detection, and can be widely used in practical applications.
Smart Images

Figure CN118968477A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of driving detection, and in particular to a fatigue driving detection method, system, readable storage medium and computer. Background Art
[0002] In recent years, the automobile industry has developed rapidly and has made great progress in manufacturing technology and quality. Taking new energy vehicles as an example, the continuous improvement in the research and development and popularization of electric and hybrid vehicles and the breakthrough in intelligent driving system technology have provided the general public with a safer, more convenient and environmentally friendly way of travel, and provided a guarantee for the public's safe travel.
[0003] Despite the continuous progress of the automobile industry and the continuous breakthroughs in intelligent driving technology, there are still many traffic accidents caused by improper driving behavior of drivers. How to regulate the driving behavior of drivers is still a challenge that cannot be ignored. Improper driving behavior of drivers includes: drunk driving, reckless driving, distracted driving and fatigue driving. Taking drunk driving and reckless driving as examples, the country has introduced relevant policies and regulations, set strict testing standards and punishment regulations, and effectively regulated the improper driving behavior of drivers to a large extent.
[0004] There are two main existing fatigue detection methods: one is based on the driver's physiological signals. However, during fatigue detection in actual driving, the driver needs to wear invasive detection equipment, which will affect the driver's driving habits and the reliability of the data. At the same time, when detecting the driver's physiological signals, multiple physiological signals are usually captured, requiring a large number of sensors and other instruments. These detection instruments are generally expensive and are difficult to be widely used in driver fatigue driving detection in practice.
[0005] Another method of fatigue detection based on the vehicle's driving status can detect the driver's fatigue status. However, there are differences in driving habits between different drivers, which will cause the vehicle's driving status to show different dynamic characteristics. In addition, in the actual driving environment, the road traffic environment will also affect the driver's driving style, causing the vehicle to show different dynamic characteristics. Therefore, it is difficult to find a universal method to judge the driver's driving status. The fatigue detection method based on the driver's physiological signals is similar. In order to collect the vehicle's dynamic characteristics, it is necessary to install more signal acquisition equipment on the motor vehicle, etc., and these devices are expensive and difficult to be widely used in driver fatigue driving detection in practice. Summary of the invention
[0006] Based on this, the purpose of the present invention is to provide a fatigue driving detection method, system, readable storage medium and computer to at least solve the deficiencies in the above-mentioned technology.
[0007] The present invention provides a fatigue driving detection method, comprising: The video frames of the driver driving the vehicle are collected by a collection device, and a face detection model is constructed, and the face detection model is used to identify and locate the video frames to obtain the coordinate information of the driver's face position; Performing facial region cropping according to the facial position coordinate information to obtain a facial region corresponding to the driver; Constructing a facial key point detection model, and using the facial key point detection model to extract key point information from the facial area to obtain facial key point coordinate information; A fatigue feature classification model is constructed, and the current fatigue feature of the driver is calculated according to the coordinate information of the key points of the face. The state of the current fatigue feature is predicted according to the fatigue feature classification model to predict the driving fatigue state of the driver.
[0008] Furthermore, the steps of building a face detection model include: Perform layered convolution on the feature extraction layer of the preset single-stage face detection model, and use the preset convolution kernel to perform point-by-point convolution to adjust the number of channels to obtain a preliminary detection model; Constructing a lightweight network, and optimizing the lightweight network using structural weight parameters during a training phase to obtain a lightweight optimized network; The lightweight optimization network is imported into the preliminary detection model and trained on a public face detection dataset to obtain the face detection model.
[0009] Furthermore, the step of using the face detection model to identify and locate the video frame to obtain the driver's face position coordinate information includes: Inputting the video frame into the face detection model to extract features to obtain feature data; Candidate regions are extracted from the feature data, and the extracted candidate regions are classified to obtain the facial position coordinate information of the driver.
[0010] Furthermore, the step of cropping the face region according to the face position coordinate information to obtain the face region corresponding to the driver includes: Performing non-maximum suppression on the facial position coordinate information of the driver, and excluding overlapping areas based on a confidence algorithm to obtain a preliminary facial area; The position and size of the region frame of the preliminary face region are adjusted by using a regression algorithm to obtain an adjusted preliminary face region, and the adjusted preliminary face region is post-processed to obtain a face region corresponding to the driver.
[0011] Furthermore, the steps of constructing a facial key point detection model include: Apply the preset facial key point detection algorithm to the end-to-end single-stage network architecture to build a preliminary facial key point detection model, wherein the feature extraction network in the main network of the preliminary facial key point detection model uses the MobileNetv2 network, and the auxiliary network uses the posture angle auxiliary network; During the training phase, the preset lightweight network is optimized using structural weight parameters, the feature extraction network of the preliminary detection model of facial key points is optimized using the optimized lightweight network, and the loss function of the preliminary detection model of facial key points is optimized using the Wing loss function to obtain the facial key point detection model.
[0012] Furthermore, the step of calculating the current fatigue characteristics of the driver according to the facial key point coordinate information includes: Calculating the eye aspect ratio, mouth aspect ratio and head posture information according to the facial key point coordinate information; The eye aspect ratio is used to calculate the eye fatigue characteristics of the driver, the mouth aspect ratio is used to calculate the mouth fatigue characteristics of the driver, and the head fatigue characteristics of the driver are calculated according to the head posture information.
[0013] The present invention also provides a fatigue driving detection system, comprising: The recognition and positioning module is used to collect video frames of the driver driving the vehicle through the acquisition device, and build a face detection model, and use the face detection model to recognize and locate the video frames to obtain the coordinate information of the driver's face position; A region cropping module, used for cropping the face region according to the face position coordinate information to obtain the face region corresponding to the driver; A key point extraction module is used to construct a face key point detection model and use the face key point detection model to extract key point information from the face area to obtain face key point coordinate information; The state prediction module is used to construct a fatigue feature classification model, calculate the current fatigue feature of the driver according to the coordinate information of the key points of the face, and perform state prediction on the current fatigue feature according to the fatigue feature classification model to predict the driving fatigue state of the driver.
[0014] Furthermore, the identification and positioning module includes: A hierarchical convolution unit is used to perform hierarchical convolution on the feature extraction layer of the preset single-stage face detection model, and to adjust the number of channels by performing point-by-point convolution using a preset convolution kernel to obtain a preliminary detection model; A network optimization unit, used to construct a lightweight network and optimize the lightweight network using structural weight parameters during a training phase to obtain a lightweight optimized network; A model building unit is used to import the lightweight optimization network into the preliminary detection model and train it on a public face detection dataset to obtain the face detection model.
[0015] Furthermore, the identification and positioning module includes: A feature extraction unit, used for inputting the video frame into the face detection model to extract features to obtain feature data; The region classification unit is used to extract candidate regions from the feature data and classify the extracted candidate regions to obtain the facial position coordinate information of the driver.
[0016] Furthermore, the key point extraction module includes: A detection model construction unit, used for applying a preset facial key point detection algorithm to an end-to-end single-stage network architecture to construct a preliminary facial key point detection model, wherein the feature extraction network in the main network of the preliminary facial key point detection model uses a MobileNetv2 network, and the auxiliary network uses a posture angle auxiliary network; The detection model optimization unit is used to optimize the preset lightweight network using structural weight parameters during the training phase, optimize the feature extraction network of the preliminary detection model of facial key points using the optimized lightweight network, and optimize the loss function of the preliminary detection model of facial key points using the Wing loss function to obtain the facial key point detection model.
[0017] Furthermore, the state prediction module includes: A data calculation unit, used to calculate the eye aspect ratio, mouth aspect ratio and head posture information according to the facial key point coordinate information; A fatigue feature calculation unit is used to calculate the eye fatigue feature of the driver using the eye aspect ratio, calculate the mouth fatigue feature of the driver using the mouth aspect ratio, and calculate the head fatigue feature of the driver according to the head posture information.
[0018] The present invention also provides a readable storage medium on which a computer program is stored, and when the program is executed by a processor, the above-mentioned fatigue driving detection method is implemented.
[0019] The present invention also proposes a computer, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned fatigue driving detection method when executing the computer program.
[0020] The fatigue driving detection method, system, readable storage medium and computer in the present invention divide the fatigue driving detection task into four subtasks: face detection, face key point detection, facial fatigue feature calculation and fatigue detection classification, and use the constructed face detection model to identify and locate the video frame to obtain the driver's face position coordinate information, which can achieve a better balance between accuracy and reasoning speed; use the constructed face key point detection model to extract key point information of the face area to obtain face key point coordinate information, use the face detection model and the face key point detection model in the driver sleepiness detection data set to extract the key point coordinates of the driver's face and calculate the eye, mouth and head fatigue feature information, and use this to predict the driver's driving fatigue state. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 is a flow chart of a fatigue driving detection method in a first embodiment of the present invention; Figure 2 for Figure 1 Detailed flow chart of step S101; Figure 3 It is a structural diagram of RepVGG Block in the first embodiment of the present invention; Figure 4 for Figure 1 A detailed flowchart of another implementation of step S101; Figure 5 for Figure 1 Detailed flow chart of step S102; Figure 6 for Figure 1 Detailed flow chart of step S103; Figure 7 for Figure 1 Detailed flow chart of step S104; Figure 8 is a distribution diagram of eye contours in the first embodiment of the present invention; Fig. 9 is a distribution diagram of the mouth contour in the first embodiment of the present invention; Fig.10 Schematic diagram of three-dimensional position information of the head in the first embodiment of the present invention; Fig.11 is a schematic diagram of a world coordinate system in the first embodiment of the present invention; Fig.12 A schematic diagram of the relationship between the image coordinate system and the world coordinate system in the first embodiment of the present invention; Fig.13 is a structural block diagram of a fatigue driving detection system in a second embodiment of the present invention; Fig.14FIG. 4 is a structural block diagram of a computer in a third embodiment of the present invention.
[0022] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION
[0023] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.
[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.
[0025] Embodiment 1 See also Figure 1 , which shows a fatigue driving detection method in a first embodiment of the present invention, and the method specifically includes steps S101 to S105: S101, collecting video frames of a driver driving a vehicle through a collection device, building a face detection model, and using the face detection model to identify and locate the video frames to obtain the coordinate information of the driver's face position; For further information, see Figure 2 , the step S101 specifically includes steps S1011 to S1013: S1011, performing layered convolution on the feature extraction layer of the preset single-stage face detection model, and performing point-by-point convolution using a preset convolution kernel to adjust the number of channels, so as to obtain a preliminary detection model; S1012, constructing a lightweight network, and optimizing the lightweight network using structural weight parameters during a training phase to obtain a lightweight optimized network; S1013, importing the lightweight optimization network into the preliminary detection model, and training it on a public face detection dataset to obtain the face detection model.
[0026] In this embodiment, the present invention is based on the RentinaFace face detection model, uses the re-parameterized improved RepGhost network to optimize its feature extraction network, trains the model on a public face detection dataset, compares the optimization effects of multiple lightweight feature extraction networks on the RentinaFace face detection model, and analyzes the detection effects of the models before and after the improvement.
[0027] Among them, RetinaFace is a single-stage face detector that can directly predict the location and key points of a face in a unified network structure without relying on multiple stages of detection processes. RetinaFace trains neural networks by optimizing the loss functions of multiple tasks simultaneously through multi-task learning (MTL). RetinaFace jointly supervises the three tasks of face detection, key point location, and face attribute recognition. The correlation between learning tasks helps the model learn richer features and improve the generalization ability and accuracy of the network.
[0028] In the specific implementation, the feature extraction layer (MobileNetv1) of the RentinaFace face detection model is layered convolution using a 3x3 convolution kernel, and then the number of channels is adjusted by point-by-point convolution using a 1x1 convolution kernel. Secondly, the RetinaFace face detection model compresses the number of channels of the MobileNetv1 used to 1 / 4 of the original, thereby increasing the speed of feature extraction and significantly reducing the amount of calculation and parameters.
[0029] It should be noted that in the RentinaFace face detection model, the number of channels of the feature maps of the last three effective feature layers of the feature extraction network is adjusted through convolution operations to make the number of channels of the three feature maps consistent. After the number of channels is adjusted, the feature maps of each layer are scaled to the same size by upsampling and then fused.
[0030] In this embodiment, in order to enhance the feature extraction capability, the SSH module is used in the RentinaFace face detection model to enhance the receptive field. For the parallel structure, a stack of 3x3 convolutions is used to replace the effects of 5x5 and 7x7 convolutions: the first layer is 3x3 convolution, two 3x3 convolutions are used in the middle to replace 5x5 convolutions, and the last layer uses three 3x3 convolutions instead of 7x7 convolutions.
[0031] Further, a lightweight network is constructed. In this embodiment, the lightweight network uses the GhostNet network, and is improved by using the structure reparameterization technology in the training stage to obtain a corresponding lightweight optimization network, namely the RepGhost network. Specifically, the above lightweight optimization network is introduced into the preliminary detection model and trained on the public face detection data set to obtain the face detection model. In the training stage, a parallel 1x1 convolution branch and an identity mapping branch are added to each 3x3 convolution layer to construct a RepVGG Block. After the training is completed, the model is equivalently converted to obtain the face detection model; please refer to Figure 3 According to the linear additivity of convolution, the three 3x3 convolution kernels are , and , then we have the following formula: ; The parallel 1x1 convolution branch of RepVGG Block can be regarded as a special 3x3 convolution, that is, the convolution kernel is filled with a large number of 0s. The identity block can be regarded as a special 3x3 convolution after a 1x1 unit convolution is padded.
[0032] The Batch Norm layer in each RepVGG Block can be equivalently converted into a convolutional layer with a bias. Indicates that the number of input channels is , the number of output channels is The 3x3 convolution kernel, Indicates the number of input channels , the number of output channels is The 1x1 convolution kernel. , , and They represent the mean, standard deviation, learned scaling factor and bias parameter of the BN layer after the 3x3 convolution, and the output features are obtained. : ; The BN operation can be equivalent to: ; The process of merging BN parameters into convolution is: ; The output of each RepVGG Block before and after conversion is exactly the same, so the trained model can be equivalently converted into a single-channel model with only 3x3 convolution.
[0033] It can be understood that structural reparameterization realizes that in the training phase, the model structure corresponds to a set of learned parameters. In the actual reasoning phase, in order to speed up the operation of the model, reduce the computing cost, and maintain or even improve the performance of the model, the model structure is simplified to correspond to another set of parameters. Moreover, this other set of parameters can be obtained from the parameters in the training phase through equivalent conversion, realizing an effective transition from a complex model to a simplified model, which not only improves efficiency but also retains model performance.
[0034] See also Figure 4 In some optional embodiments, the step S101 specifically includes steps S1111 to S1112: S1111, inputting the video frame into the face detection model to extract features to obtain feature data; S1112, extracting candidate regions from the feature data, and classifying the extracted candidate regions to obtain facial position coordinate information of the driver.
[0035] In the specific implementation, the preset face detection technology is used to identify the face area in the video frame. In order to more accurately calculate the driver's fatigue characteristics and improve the accuracy of the fatigue feature classification prediction model, it is necessary to locate the driver's facial area, and input the video frame into the face detection model for feature extraction to obtain feature data. In the image, multiple candidate areas that may contain faces are generated through sliding windows or other methods, and each candidate area is input into a pre-trained classifier or neural network for classification to determine whether the area contains a face, for multiple overlapping candidate areas.
[0036] S102, performing face region cropping according to the face position coordinate information to obtain a face region corresponding to the driver; For further information, see Figure 5 , the step S102 specifically includes steps S1021~S1022: S1021, performing non-maximum suppression on the facial position coordinate information of the driver, and excluding overlapping areas based on a confidence algorithm to obtain a preliminary facial area; S1022, using a regression algorithm to adjust the position and size of the region frame of the preliminary face region to obtain an adjusted preliminary face region, and post-processing the adjusted preliminary face region to obtain a face region corresponding to the driver.
[0037] In the specific implementation, the non-maximum suppression method is used to select the most confident area and exclude other overlapping areas; for the detected face area, the position and size of the frame are adjusted through the regression algorithm to make it cover the face area more accurately; the detected face area is post-processed, including face alignment, posture correction, etc., to improve the accuracy of the detection results.
[0038] S103, constructing a facial key point detection model, and using the facial key point detection model to extract key point information from the facial region to obtain facial key point coordinate information; For further information, see Figure 6 , the step S103 specifically includes steps S1031~S1032: S1031, applying a preset facial key point detection algorithm to an end-to-end single-stage network architecture to construct a preliminary facial key point detection model, wherein a feature extraction network in a main network of the preliminary facial key point detection model uses a MobileNetv2 network, and an auxiliary network uses a posture angle auxiliary network; S1032, using structural weight parameters to optimize the preset lightweight network during the training phase, using the lightweight network after network optimization to optimize the feature extraction network of the preliminary detection model of facial key points, and using the Wingloss function to optimize the loss function of the preliminary detection model of facial key points to obtain a facial key point detection model.
[0039] In this embodiment, the feature extraction network of PFLD is optimized by RepGhost network to improve the key point detection accuracy and inference speed, the loss function of the PFLD face key point detection model is optimized, and the Wing loss function, which has better effect in the face key point detection and positioning task, is used to enable the model to learn more accurate key point location information.
[0040] Among them, PFLD (Practical Facial Landmarks Detector) is an efficient facial landmark detection algorithm. PFLD uses an end-to-end single-stage network architecture. The feature extraction network is built by MobileNetv2. PFLD designs a multi-scale fully connected layer to increase its receptive field, thereby improving its ability to capture facial structures.
[0041] In specific implementation, PFLD consists of two main parts: one is the main network responsible for facial key point detection, and the other is the auxiliary network for posture angle prediction. MobileNetv2 is used as the feature extraction network in the main network. The auxiliary network only supervises the network during the training phase and does not participate in the reasoning process. The purpose of this sub-network is to estimate the three-dimensional Euler angle of each input face sample. The ground truth is calculated from the key point information in the training data. The overall structure of the PFLD network is shown in Table 1, and the structure of the posture angle auxiliary network is shown in Table 2.
[0042] Table 1 Overall structure of PFLD network
[0043] Table 2 Attitude angle auxiliary network structure
[0044] Furthermore, PFLD uses MobileNetv2 as the feature extraction network. In addition to continuing to use depthwise separable convolution, MobileNetv2 introduces linear bottlenecks and inverse residual structures. The activation features obtained by the activation function in the convolutional neural network can be embedded in low dimensions. However, since the layers of the deep convolutional neural network have nonlinear activation functions, taking ReLU as an example, this activation function will cause negative input features to become zero, resulting in feature loss. Therefore, MobileNetv2 does not use the ReLU activation function and calls this new module a linear module.
[0045] Different from the traditional residual structure, the inverted residual structure first performs an extended convolution on the input features to expand their dimensions, then extracts richer features through depthwise separable convolution, and finally uses linear projection to compress the data, thereby reducing the network size. The reason for expanding the channel is to extract more information, because the depthwise separable convolution cannot change the number of channels, so the number of channels is expanded first, and performing depthwise separable convolution in high-dimensional space can extract more information.
[0046] In order to effectively utilize the feature information of different dimensions in the main network, PFLD designs a three-layer feature pyramid structure for face key point detection and outputs the position of the key points through a fully connected layer.
[0047] Specifically, since the human face has strong structural information, such as symmetry and the layout of facial features, the auxiliary network can help the main network better understand and utilize the structural information of the face by predicting the posture deflection angle, thereby improving the accuracy of key point positioning. Therefore, in the posture angle prediction auxiliary network, PFLD sets five convolutional layers and two fully connected layers to predict the three-dimensional Euler angles of the face. In the early stages of training, the key points predicted by the main network may not be accurate enough. If these key points are used directly to calculate the posture deflection angle, inaccurate estimates may result. By introducing the auxiliary network, the two problems of key point positioning and posture estimation can be decoupled, thereby reducing the difficulty of network training and avoiding problems such as excessive punishment or failure to converge.
[0048] The auxiliary network is a branch network derived from the middle layer of the main network. It does not directly participate in the calculation of the inference phase and only works in the training phase. This design allows the auxiliary network to effectively assist the training of the main network without increasing the computational burden of the inference phase.
[0049] Furthermore, improvements were made based on the PFLD face key point detection model, as shown in Table 3. Indicates the expansion multiple of the BottleNeck middle channel. Represents the number of output channels of BottleNeck, Represents the number of BottleNecks connected in series. Represents the stride, and the number of branches in all models is 4.
[0050] Table 3 Overall structure of improved PFLD network
[0051] In this embodiment, the Wing loss function is used to optimize the loss function of the preliminary detection model of face key points, and its formula is as follows: ; in, is a non-negative threshold, limiting Nonlinear range of Responsible for controlling the curvature of the logarithmic function curve; constant It can achieve smooth transition between linear and nonlinear regions of piecewise functions.
[0052] Understandably, Wing loss provides larger gradients in the range of small and medium errors, which helps the network to adjust the prediction values more finely during training, thereby improving the accuracy of key point positioning. By limiting the range of the nonlinear part, the problem of excessive gradients in large errors is avoided, which helps the network remain stable in the face of outliers.
[0053] Furthermore, the facial key point detection model is used to extract key point information from the facial area to obtain the coordinate information of the facial key points. By detecting the facial key points, the various parts of the face can be described and located more accurately, providing important information for subsequent facial feature analysis and processing, which is the application basis for research on face recognition, face replacement, and face special effects. According to different task requirements, the precision of the key point positioning obtained by the facial key point detection model regression is also different. In this embodiment, in order to extract more accurate facial features for fatigue detection, 98 key points are selected, such as: eye contour, nose contour, face contour, and mouth contour.
[0054] S104, constructing a fatigue feature classification model, and calculating the current fatigue feature of the driver according to the facial key point coordinate information, and performing state prediction on the current fatigue feature according to the fatigue feature classification model to predict the driving fatigue state of the driver.
[0055] For further information, see Figure 7 , the step S104 specifically includes steps S1041-S1042: S1041, calculating the eye aspect ratio, the mouth aspect ratio and the head posture information according to the facial key point coordinate information; S1042, calculating the eye fatigue characteristics of the driver using the eye aspect ratio, calculating the mouth fatigue characteristics of the driver using the mouth aspect ratio, and calculating the head fatigue characteristics of the driver according to the head posture information.
[0056] In specific implementation, the coordinate position information of key facial points is used to calculate facial feature information that can reflect the fatigue state, such as the aspect ratio of the eyes, the aspect ratio of the mouth, and the head posture.
[0057] (1) Characteristics of eye fatigue The state of the eyes is related to the driver's fatigue level. The eyes change significantly when they are tired and awake. This embodiment uses the eye aspect ratio (EAR) to quantify the state of eye fatigue to characterize the fatigue characteristic information provided by the eyes. The EAR calculation formula is as follows: ; In the formula, The key points of the upper eye contour; The key points for the lower eye contour; is the eye corner key point, such as Figure 8 shown.
[0058] In addition, PERCLOS is also an indicator used to assess fatigue. The PERCLOS criterion assesses fatigue by calculating the proportion of time the eyes are closed over a period of time. In a fixed time interval, it is calculated by calculating whether the proportion of the time the eyes are closed to the total time exceeds a certain threshold. The formula is as follows: ; In the formula, is the number of frames with eyes closed, is the total number of frames in a fixed time.
[0059] (2) Mouth fatigue characteristic indicators Mouth features are also one of the fatigue feature indicators in this embodiment. A common method is to use the mouth aspect ratio (MAR) to evaluate the mouth state. MAR is used to quantify the degree of mouth opening. The concept of mouth aspect ratio is similar to the concept of eye aspect ratio. Fig. 9 As shown, the MAR calculation formula is as follows: ; In the formula, The key points of the upper lip contour; The key points of the lower lip contour; Key points of mouth corners.
[0060] The principle of PYAWN is consistent with PERCLOS. PYAWN represents the ratio of mouth yawning frames to total frames per unit time. The calculation formula is as follows: ; In the formula, is the number of frames of yawning, is the total number of frames in a fixed time.
[0061] (3) Head fatigue characteristic indicators The head fatigue characteristics can be expressed by the Euler angles of the head posture, namely the pitch angle (Pitch), yaw angle (Yaw) and roll angle (Roll), such as Fig.10 As shown. Since the collected video is a facial video frame image, this embodiment uses a head posture estimation method to predict the three-dimensional position information of the head.
[0062] The driver's fatigue state will affect his head posture, such as nodding. In this embodiment, the facial key point coordinate information extracted by the facial key point detector and the internal and external parameter information of the camera of the video acquisition device are used to map the two-dimensional image coordinates to the three-dimensional space through the mutual conversion of different coordinate systems, so as to estimate the Euler angle of the head as the head posture fatigue feature information. This calculation process involves the conversion of three coordinate systems: image coordinate system , video acquisition device coordinate system and the world coordinate system .
[0063] The image coordinate system is a commonly used coordinate system in image processing and computer vision to describe the position of points in an image. In the pixel coordinate system, an image is viewed as a grid of pixels, each pixel has a unique coordinate, and the coordinates of a point can be expressed as ,in is the horizontal offset, is the vertical offset.
[0064] The video acquisition device coordinate system simulates the process of light being projected from the video acquisition device lens to the image sensor through a perspective projection model, mapping points in three-dimensional space to a two-dimensional image plane. This projection model can be expressed in the form of a homogeneous coordinate matrix as follows: ; In the formula, is the focal length of the video capture device, usually measured in pixels. are the principal point coordinates of the image, that is, the position of the optical center on the image plane. It is only related to the focal length and principal point coordinates of the video acquisition device and is an internal parameter of the video acquisition device lens. It is called the internal parameter matrix.
[0065] Specifically, the video acquisition device has its own position information and posture information in the world coordinate system. Fig.11 As shown in the figure, the position information of the video acquisition device describes the exact position of the coordinate system of the device relative to the entire world coordinate system, which is expressed as the displacement of the origin of the original coordinate system relative to the origin of the world coordinate system. It is expressed as a translation vector At the same time, the posture information of the video acquisition device describes the direction of the device's coordinate system relative to the entire world coordinate system. This orientation is called the rotation of the device coordinate system relative to the world coordinate system, which is represented by the rotation matrix To express.
[0066] Combined translation vector , rotation matrix , we can construct a 4×4 homogeneous transformation matrix, namely the projection matrix (or video capture device matrix), which completely describes the transformation relationship between the video capture device coordinate system and the world coordinate system: ; Using this homogeneous transformation matrix, the point in the video acquisition device coordinate system can be Convert to a point in world coordinates : ; In the formula, It is related to the location information and posture information of the video acquisition device. Specifically, Contains the information of rotation (Rotation) and translation (Translation) of the video acquisition device from one position to another, which is called the external parameter matrix. Substituting this formula into the conversion formula between the image coordinate system and the video acquisition device coordinate system is: ; The final calculation formula of the image coordinate system and the world coordinate system is: ; The relationship between the image coordinate system and the world coordinate system is as follows Fig.12 shown.
[0067] In this embodiment, 14 key points in the face key point detection model are selected, as shown in Table 4. The solvePnP function provided by OpenCV is used to minimize the error between the predicted two-dimensional image points and the actually observed two-dimensional image points through an iterative method, and the rotation vector and translation vector that minimize the error are found, which are used as the translation vector and rotation vector of the video acquisition device relative to the face.
[0068] Table 4 Correspondence between 2D and 3D coordinates of key facial points
[0069] The calculated rotation matrix It is composed of three rotations: Yaw angle of axis , around Axis pitch angle and around Roll angle of the axis The Rodriguez formula converts a rotation vector into a rotation matrix. Rotation Angle The rotation matrix , the Rodriguez formula is as follows: ; in, is the identity matrix, is a vector The rotation matrix can be decomposed into the product of three rotation matrices: ; Among them, the rotation matrices around each axis are: ; ; ; Finally, the Euler angles can be calculated: ; Euler angles provide a method to quantify head posture, which can accurately describe the direction and tilt of the head. The Euler angle method can be used to extract features of driver head fatigue information and to judge the driver's fatigue driving.
[0070] Furthermore, the fatigue feature classification model in this embodiment adopts three machine learning models, namely, Adaboost, Random Forest (RF) and XGboost, in the machine learning model, and also adopts the following deep learning models: multi-layer perceptron, multi-scale convolutional neural network (Multi-ScaleCNN) and long short-term recurrent neural network. These models are used for training and verification on the fatigue feature data set constructed in this embodiment to evaluate their accuracy in processing such data.
[0071] In this embodiment, the five facial features of the eye aspect ratio, mouth aspect ratio, head pitch angle, head yaw angle and head roll angle are used as inputs of the fatigue feature classification model to predict the fatigue state at the current moment, i.e., fatigue or non-fatigue.
[0072] Among them, this embodiment uses the driving videos disclosed in the preset driver fatigue detection dataset as the data source of fatigue feature extraction samples, and predicts the driving fatigue state based on the data source. The dataset includes 36 drivers with different facial features and five different scenes: bare face, glasses, sunglasses, night-bare face and night-glasses. Among them, the frame rate of the night scene is 15 frames / second, and the frame rate of the day scene is 30 frames / second. All videos in this dataset are collected using active infrared (IR) lighting with a resolution of 640 pixels × 480 pixels.
[0073] In this embodiment, facial images of four drivers at different fatigue levels are randomly selected from the video data set. Each scene contains video samples of four driving behaviors, namely, yawning, slow blinking and nodding, sleepy combination, and non-fatigue combination.
[0074] In summary, the fatigue driving detection method in the above-mentioned embodiment of the present invention divides the fatigue driving detection task into four subtasks: face detection, face key point detection, facial fatigue feature calculation and fatigue detection classification, and uses the constructed face detection model to identify and locate the video frame to obtain the driver's face position coordinate information, which can achieve a better balance between accuracy and reasoning speed; uses the constructed face key point detection model to extract key point information of the face area to obtain face key point coordinate information, and uses the face detection model and the face key point detection model in the driver sleepiness detection data set to extract the key point coordinates of the driver's face and calculate the eye, mouth and head fatigue feature information, so as to predict the driver's driving fatigue state.
[0075] Embodiment 2 Another aspect of the present invention is to provide a fatigue driving detection system. Fig.13 , which shows a fatigue driving detection system in a second embodiment of the present invention, the system comprises: The recognition and positioning module 11 is used to collect video frames of the driver driving the vehicle through a collection device, and to construct a face detection model, and to use the face detection model to recognize and locate the video frames to obtain the coordinate information of the driver's face position; Furthermore, the identification and positioning module 11 includes: A hierarchical convolution unit is used to perform hierarchical convolution on the feature extraction layer of the preset single-stage face detection model, and to adjust the number of channels by performing point-by-point convolution using a preset convolution kernel to obtain a preliminary detection model; A network optimization unit, used to construct a lightweight network and optimize the lightweight network using structural weight parameters during a training phase to obtain a lightweight optimized network; A model building unit is used to import the lightweight optimization network into the preliminary detection model and train it on a public face detection dataset to obtain the face detection model.
[0076] Furthermore, the identification and positioning module 11 also includes: A feature extraction unit, used for inputting the video frame into the face detection model to extract features to obtain feature data; The region classification unit is used to extract candidate regions from the feature data and classify the extracted candidate regions to obtain the facial position coordinate information of the driver.
[0077] A region cropping module 12, configured to crop a face region according to the face position coordinate information to obtain a face region corresponding to the driver; A key point extraction module 13 is used to construct a face key point detection model, and use the face key point detection model to extract key point information from the face area to obtain face key point coordinate information; Furthermore, the key point extraction module 13 includes: A detection model construction unit, used for applying a preset facial key point detection algorithm to an end-to-end single-stage network architecture to construct a preliminary facial key point detection model, wherein the feature extraction network in the main network of the preliminary facial key point detection model uses a MobileNetv2 network, and the auxiliary network uses a posture angle auxiliary network; The detection model optimization unit is used to optimize the preset lightweight network using structural weight parameters during the training phase, optimize the feature extraction network of the preliminary detection model of facial key points using the optimized lightweight network, and optimize the loss function of the preliminary detection model of facial key points using the Wing loss function to obtain the facial key point detection model.
[0078] The state prediction module 14 is used to construct a fatigue feature classification model, and calculate the current fatigue feature of the driver according to the facial key point coordinate information, and perform state prediction on the current fatigue feature according to the fatigue feature classification model to predict the driving fatigue state of the driver.
[0079] Furthermore, the state prediction module 14 includes: A data calculation unit, used to calculate the eye aspect ratio, mouth aspect ratio and head posture information according to the facial key point coordinate information; A fatigue feature calculation unit is used to calculate the eye fatigue feature of the driver using the eye aspect ratio, calculate the mouth fatigue feature of the driver using the mouth aspect ratio, and calculate the head fatigue feature of the driver according to the head posture information.
[0080] The functions or operation steps implemented when the above modules and units are executed are generally the same as those in the above method embodiments, and will not be repeated here.
[0081] The fatigue driving detection system provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the system embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.
[0082] Embodiment 3 The present invention also provides a computer, see Fig.14 , shown is a computer in the third embodiment of the present invention, including a memory 10, a processor 20, and a computer program 30 stored in the memory 10 and executable on the processor 20. When the processor 20 executes the computer program 30, the above-mentioned fatigue driving detection method is implemented.
[0083] The memory 10 includes at least one type of readable storage medium, which includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 10 may be an internal storage unit of a computer, such as a hard disk of the computer. In other embodiments, the memory 10 may also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Further, the memory 10 may also include both an internal storage unit of the computer and an external storage device. The memory 10 may be used not only to store application software and various types of data installed in the computer, but also to temporarily store data that has been output or is to be output.
[0084] Among them, in some embodiments, the processor 20 can be an electronic control unit (Electronic Control Unit, abbreviated as ECU, also known as a vehicle computer), a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor or other data processing chip, used to run the program code stored in the memory 10 or process data, such as executing access restriction programs, etc.
[0085] It should be pointed out that Fig.14 The structure shown does not constitute a limitation on the computer. In other embodiments, the computer may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.
[0086] The embodiment of the present invention further provides a readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the fatigue driving detection method as described above is implemented.
[0087] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.
[0088] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.
[0089] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0090] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0091] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A method for detecting fatigue driving, characterized in that: include: The video frames of the driver driving the vehicle are collected by a collection device, and a face detection model is constructed, and the face detection model is used to identify and locate the video frames to obtain the coordinate information of the driver's face position; Performing facial region cropping according to the facial position coordinate information to obtain a facial region corresponding to the driver; Constructing a facial key point detection model, and using the facial key point detection model to extract key point information from the facial area to obtain facial key point coordinate information; A fatigue feature classification model is constructed, and the current fatigue feature of the driver is calculated according to the coordinate information of the key points of the face. The state of the current fatigue feature is predicted according to the fatigue feature classification model to predict the driving fatigue state of the driver.
2. The fatigue driving detection method according to claim 1, characterized in that: The steps to build a face detection model include: Perform layered convolution on the feature extraction layer of the preset single-stage face detection model, and use the preset convolution kernel to perform point-by-point convolution to adjust the number of channels to obtain a preliminary detection model; Constructing a lightweight network, and optimizing the lightweight network using structural weight parameters during a training phase to obtain a lightweight optimized network; The lightweight optimization network is imported into the preliminary detection model and trained on a public face detection dataset to obtain the face detection model.
3. The fatigue driving detection method according to claim 1, characterized in that: The step of using the face detection model to identify and locate the video frame to obtain the driver's face position coordinate information includes: Inputting the video frame into the face detection model to extract features to obtain feature data; Candidate regions are extracted from the feature data, and the extracted candidate regions are classified to obtain the facial position coordinate information of the driver.
4. The fatigue driving detection method according to claim 1, characterized in that: The step of cutting out the face region according to the face position coordinate information to obtain the face region corresponding to the driver includes: Performing non-maximum suppression on the facial position coordinate information of the driver, and excluding overlapping areas based on a confidence algorithm to obtain a preliminary facial area; The position and size of the region frame of the preliminary face region are adjusted by using a regression algorithm to obtain an adjusted preliminary face region, and the adjusted preliminary face region is post-processed to obtain a face region corresponding to the driver.
5. The fatigue driving detection method according to claim 1, characterized in that: The steps to build a face key point detection model include: Apply the preset facial key point detection algorithm to the end-to-end single-stage network architecture to build a preliminary facial key point detection model, wherein the feature extraction network in the main network of the preliminary facial key point detection model uses the MobileNetv2 network, and the auxiliary network uses the posture angle auxiliary network; During the training phase, the preset lightweight network is optimized using structural weight parameters, the feature extraction network of the preliminary detection model of facial key points is optimized using the optimized lightweight network, and the loss function of the preliminary detection model of facial key points is optimized using the Wing loss function to obtain the facial key point detection model.
6. The fatigue driving detection method according to claim 1, characterized in that: The step of calculating the current fatigue characteristics of the driver according to the facial key point coordinate information comprises: Calculating the eye aspect ratio, mouth aspect ratio and head posture information according to the facial key point coordinate information; The eye aspect ratio is used to calculate the eye fatigue characteristics of the driver, the mouth aspect ratio is used to calculate the mouth fatigue characteristics of the driver, and the head fatigue characteristics of the driver are calculated according to the head posture information.
7. A method for detecting fatigue driving, characterized in that: include: The recognition and positioning module is used to collect video frames of the driver driving the vehicle through the acquisition device, and build a face detection model, and use the face detection model to recognize and locate the video frames to obtain the coordinate information of the driver's face position; A region cropping module, used for cropping the face region according to the face position coordinate information to obtain the face region corresponding to the driver; A key point extraction module is used to construct a face key point detection model and use the face key point detection model to extract key point information from the face area to obtain face key point coordinate information; The state prediction module is used to construct a fatigue feature classification model, calculate the current fatigue feature of the driver according to the coordinate information of the facial key points, and perform state prediction on the current fatigue feature according to the fatigue feature classification model to predict the driving fatigue state of the driver.
8. The fatigue driving detection system according to claim 7, characterized in that: The identification and positioning module includes: A hierarchical convolution unit, used to perform hierarchical convolution on the feature extraction layer of the preset single-stage face detection model, and to adjust the number of channels by performing point-by-point convolution using a preset convolution kernel, so as to obtain a preliminary detection model; A network optimization unit, used to construct a lightweight network and optimize the lightweight network using structural weight parameters during a training phase to obtain a lightweight optimized network; A model building unit is used to import the lightweight optimization network into the preliminary detection model and train it on a public face detection dataset to obtain the face detection model.
9. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the fatigue driving detection method as described in any one of claims 1 to 6 is implemented.
10. A computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the fatigue driving detection method as described in any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Face candidate box extraction method
CN110490170A
Intelligent business hall face recognition method based on lightweight deep learning
CN110781829A
Unsafe driving behavior recognition method based on face feature points
CN114973214A
Fatigue driving detection method based on deep learning and facial features
CN115393830A
Unmanned aerial vehicle image target detection method based on separable convolution and re-parameterization
CN118135432A
