A fatigue detection method based on driver face recognition

Abstract

The application discloses a kind of fatigue detection methods based on driver face recognition, including the driving image is input to the face extraction neural network model of pre-set and is identified, obtains global face image and local eye area image;Global face image is input into lightweight neural network, and global feature map is obtained, and local eye area image is input into lightweight neural network, and double local feature map is obtained;Global feature map is input into global feature extraction model and is extracted, and global feature information is obtained, and the first eye feature map and the second eye feature map in double local feature map are fused and analyzed, and local feature information is obtained;Global feature information and local feature information are subjected to multiple fusion processing, and the eye state of driver is classified according to processing result, to determine the fatigue degree of driver and issue warning, guarantee traffic safety.

Description

Technical Field

[0001] This invention relates to the field of fatigue driving detection technology, and in particular to a fatigue detection method based on driver facial recognition. Background Technology

[0002] Driver fatigue is one of the core contributing factors to major traffic accidents in commercial vehicles. The proportion of traffic accidents caused by fatigue driving is relatively high in long-distance freight and passenger transport scenarios. To address this safety hazard, intelligent fatigue detection technology has become a core module of active safety systems for commercial vehicles.

[0003] Existing fatigue detection technologies include vehicle behavior analysis methods (such as analyzing the frequency of steering wheel rotation, braking, and vehicle speed). These methods are susceptible to interference from external environmental factors and are difficult to achieve accurate fatigue detection, resulting in poor detection accuracy and an inability to effectively warn of dangerous driver fatigue states and prevent traffic accidents.

[0004] Therefore, how to effectively detect fatigued driving and maintain traffic safety has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] This invention provides a fatigue detection method based on driver facial recognition, which solves the problem of how to extract and enhance features of the driver's face from both global and local perspectives, so as to improve the accuracy of the model output results and achieve timely early warning of traffic accidents.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide a fatigue detection method based on driver facial recognition, comprising:

[0007] The obtained driving image of the driver inside the target vehicle is input into a preset facial extraction neural network model for recognition, thereby obtaining the driver's global facial image and local eye area image;

[0008] The global facial image is input into a pre-trained first lightweight neural network, and a global feature map is output. The local eye region image is input into a pre-trained second lightweight neural network, and a dual local feature map is output.

[0009] The global feature map is input into a global feature extraction model for feature extraction to obtain global feature information corresponding to the global facial image. The first eye feature map and the second eye feature map in the dual local feature map are fused and analyzed to obtain local feature information corresponding to the local eye region image.

[0010] The global feature information and the local feature information are subjected to multiple fusion processing, and the driver's eye state is classified according to the processing results;

[0011] The fatigue detection results corresponding to the driver's fatigue level are obtained based on the classification results.

[0012] Furthermore, the process of acquiring the driver's global facial image and local eye region image includes:

[0013] The driving image is input into the face extraction neural network model, which outputs the boundary vertex coordinates and key point coordinates of the driver's overall face; the face extraction neural network model is obtained by pre-training a multi-task cascaded convolutional network.

[0014] The driving image is cropped based on the boundary vertex coordinates and the key point coordinates to obtain the global face image and the local eye region image.

[0015] Further, the step of inputting the global feature map into a global feature extraction model for feature extraction to obtain global feature information corresponding to the global facial image includes:

[0016] In the global feature extraction model, the global feature map is subjected to dimensionality reduction processing by global average pooling to obtain the first feature tensor;

[0017] The first feature tensor is input into the fully connected layer configured in the global feature extraction model for channel adjustment to obtain the second feature tensor;

[0018] The second feature tensor is nonlinearly optimized using an activation function to obtain the third feature tensor;

[0019] The global feature information is determined by integrating the third feature tensor with the global feature map.

[0020] Further, the fusion analysis of the first eye feature map and the second eye feature map in the dual local feature maps to obtain the local feature information corresponding to the local eye region image includes:

[0021] After inputting the dual local feature maps into the selected local fusion model, the first eye feature map and the second eye feature map are fused element by element to generate a local fusion feature map.

[0022] The local fusion feature map is sequentially subjected to convolution and nonlinear activation processing, and an attention-weighted modulation mechanism is introduced during the processing to generate a local enhanced feature map;

[0023] The local enhanced feature map is integrated with the pre-selected second eye feature map and then the feature is split to determine the local feature information.

[0024] Furthermore, the step of performing multiple fusion processing on the global feature information and the local feature information, and classifying the driver's eye state based on the processing results, includes:

[0025] The global feature information and the local feature information are sequentially subjected to weight analysis, channel enhancement and weighted fusion processing to obtain the target facial feature data;

[0026] The target facial feature data is input into a preset classifier model to classify the driver's eye closing and opening states.

[0027] Further, the step of sequentially performing weight analysis, channel enhancement, and weighted fusion processing on the global feature information and the local feature information to obtain the target facial features includes:

[0028] The global feature information and the local feature information are concatenated into vectors, and weights are assigned to the concatenated global feature information and local feature information respectively;

[0029] The weighted global feature information is sequentially subjected to first-channel enhancement and spatial alignment processing to obtain global optimized features, and the weighted local feature information is subjected to second-channel enhancement to determine local optimized features;

[0030] The target facial features are obtained by weighted fusion of the global optimization features and the local optimization features.

[0031] Further, obtaining the fatigue detection result corresponding to the driver's fatigue level based on the classification result includes:

[0032] Based on the classification results, determine the probability distribution of the driver's eye state;

[0033] Based on the probability distribution of eye state, calculate the target number of frames in which the driver's eyes are closed within a preset time period;

[0034] The quantification result corresponding to the fatigue level is determined by calculating the percentage of the target frame count to the total frame count within a preset time period.

[0035] When the quantification result exceeds the preset fatigue threshold, a warning is issued, and the fatigue detection result is generated based on the warning content and the quantification result.

[0036] Another embodiment of the present invention provides a fatigue detection system based on driver facial recognition, comprising:

[0037] The facial feature detection module is used to input the obtained driving image of the driver inside the target vehicle into a preset facial extraction neural network model for recognition, and to obtain the driver's global facial image and local eye area image;

[0038] The feature extraction module is used to input the global facial image into a pre-trained first lightweight neural network and output a global feature map, and input the local eye region image into a pre-trained second lightweight neural network and output a dual local feature map.

[0039] The feature optimization module is used to input the global feature map into the global feature extraction model for feature extraction to obtain the global feature information corresponding to the global facial image, and to perform fusion analysis on the first eye feature map and the second eye feature map in the dual local feature map to obtain the local feature information corresponding to the local eye region image.

[0040] The classification module is used to perform multiple fusion processing on the global feature information and the local feature information, and classify the driver's eye state according to the processing results;

[0041] The fatigue detection module is used to obtain the fatigue detection result corresponding to the driver's fatigue level based on the classification result.

[0042] Another embodiment of the present invention provides a computer device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the fatigue detection method based on driver facial recognition as described above.

[0043] In another embodiment of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the fatigue detection method based on driver facial recognition as described above.

[0044] Compared with the prior art, the beneficial effects of the embodiments of the present invention are at least one of the following:

[0045] This invention employs a task-cascaded convolutional network for face detection, simultaneously extracting global face features and local eye features. This allows for effective judgment even if some facial information is lost, improving the model's robustness. Local facial features, such as the left and right eye regions, and global facial features are analyzed by lightweight neural networks, significantly enhancing detection efficiency. By combining a stream fusion module and a global feature extraction module to fuse and analyze local and global features respectively, facial information at different scales can be preserved, achieving high-precision fatigue driving detection. Attached Figure Description

[0046] Figure 1 This is a schematic flowchart of a fatigue detection method based on driver facial recognition in one embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram of the overall process of driver facial recognition in one embodiment of the present invention;

[0048] Figure 3 This is a global feature analysis architecture diagram in one embodiment of the present invention;

[0049] Figure 4 This is a diagram of a local feature fusion architecture in one embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of a fatigue detection system based on driver facial recognition in one embodiment of the present invention;

[0051] Figure 6 A structural block diagram of a preferred embodiment of a computer device provided by the present invention;

[0052] Figure labels: M1, facial feature detection module; M2, feature extraction module; M3, feature optimization module; M4, classification module; M5, fatigue detection module. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0054] In the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0055] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. The terms "vertical," "horizontal," "left," "right," "upper," "lower," and similar expressions used herein are for illustrative purposes only and do not indicate or imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as limiting the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0056] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing specific embodiments only and is not intended to limit the invention. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0057] To address the issue of inaccurate driver status detection in existing methods, one embodiment of the present invention provides a fatigue detection method based on driver facial recognition. For details, please refer to [link to relevant documentation]. Figure 1 , Figure 1 The diagram shown is a flowchart of a fatigue detection method based on driver facial recognition according to one embodiment of the present invention, including the following steps:

[0058] S1. Input the obtained driving image of the driver inside the target vehicle into a preset facial extraction neural network model for recognition, and obtain the driver's global facial image and local eye area image.

[0059] In this embodiment of the invention, a single frame image from a video stream captured by an RGB camera inside the target vehicle can be selected as the input to the face extraction neural network model. It is understood that Multi-Task Convolutional Networks (MTCNNs), through mechanisms such as cascaded structure, multi-task learning, and keypoint localization, significantly improve the robustness, accuracy, and real-time performance of driver fatigue detection in practical image recognition applications, especially performing excellently under occlusion and complex lighting conditions, providing an efficient solution for fatigue detection tasks. Therefore, preferably, this embodiment selects a multi-task cascaded convolutional network to pre-train the face extraction neural network model.

[0060] After training, the driving image is input into the facial extraction neural network model, which outputs the coordinates of the boundary vertices and key points of the driver's overall face. Understandably, in this embodiment, the MTCNN model outputs a rectangular bounding box (including the coordinates of the four vertices of the bounding box) and five key point coordinates. These five coordinates include the coordinates of the center point of the left eye, the center point of the right eye, the nose, the left corner of the mouth, and the right corner of the mouth. For details, please refer to... Figure 2 As shown on the far left, Figure 2 This diagram illustrates the overall process of driver facial recognition in one embodiment of the present invention. In this embodiment, the driving image is cropped based on the coordinates of boundary vertices and key points, thereby obtaining a global facial image and a local eye region image. Figure 2 As can be seen, the local eye region image includes both the driver's left and right eyes.

[0061] For example, in some embodiments of the present invention, the vertex distribution of the facial rectangle output by MTCNN is set as follows: top left corner P1 = (x1 + y1), top right corner P2 = (x2 + y2), bottom right corner P3 = (x3 + y3), and bottom left corner P4 = (x4 + y4). The width of the bounding box is then represented as: W face Height is represented as: H face Regarding the eye area, in some embodiments of the present invention, E is used. L =(x L ,y L () represents the center point of the left eye, using E R =(x R ,y R The coordinates of the four vertices of the facial rectangle are used to represent the center point of the right eye. Each region follows a set of proportional relationships derived from the size of the eyelid. Based on the coordinates of the four vertices of the facial rectangle, a global facial image reflecting the overall facial features of the driver is cropped out, with a size of 64*64*3. Correspondingly, the left eye region and the right eye region are cropped out according to the coordinates of the center points of the left and right eyes, respectively, each with a size of 32*32*3.

[0062] Specifically, the overall face is represented as:

[0063] Face = (x1, y1, W) face H face )

[0064] The left eye region can be represented as:

[0065]

[0066] The right eye region can be represented as:

[0067]

[0068] Where α and β are the ratios of the width and height of the eyes to the facial dimensions. In this example, α = 0.25 and β = 0.17.

[0069] S2. Input the global facial image into a pre-trained first lightweight neural network to obtain a global feature map, and input the local eye region image into a pre-trained second lightweight neural network to obtain a dual local feature map.

[0070] Combination Figure 2 As can be seen, this embodiment employs a dual-stream architecture based on a lightweight neural network (MobileNetV3) for the feature extraction stages of global facial images and local eye region images. The lightweight network, serving as the backbone, is suitable for real-time applications such as ADAS (Advanced Driver Assistance Systems). The trained lightweight model can be directly deployed on the vehicle for offline recognition, better protecting user privacy. In this embodiment, MobileNetV3-Large and MobileNetV3-Small are two different configurations of the MobileNetV3 model, optimized for different application scenarios.

[0071] The first lightweight neural network applied to the global facial image is MobileNetV3-Large, which has a large receptive field and can effectively capture semantic feature information widely distributed in the facial region, such as the spatial relationship between areas like the eyes and corners of the mouth, and the overall facial contour, thus providing semantic priors for subsequent local fine-grained feature extraction. In this embodiment, the global facial image is used as a global stream input to the pre-trained MobileNetV3-Large for feature analysis, and the final output is a 4*4*704 global feature map.

[0072] The second lightweight neural network applied to local eye region images is MobileNetV3-Small, which is more capable of capturing fine-grained appearance and texture details from local eye images, including the contour changes of the eyelid edges, the light and dark distribution of the pupil and iris regions, and the degree of eye opening. In this embodiment, the left and right eye regions of the local eye image are input as local stream 1 and local stream 2, respectively, into the pre-trained MobileNetV3-Small for feature extraction, and the corresponding output is a dual local feature map reflecting the features of the driver's left and right eye regions. For example, local feature maps of size 4*4*704 can be output respectively. It should be understood that this embodiment adopts a multi-stream parallel processing strategy, which can significantly improve the detection performance.

[0073] S3. Input the global feature map into the global feature extraction model for feature extraction to obtain the global feature information corresponding to the global facial image. Perform fusion analysis on the first eye feature map and the second eye feature map in the dual local feature map to obtain the local feature information corresponding to the local eye area image.

[0074] This step is a further feature analysis process. Faced with the three feature maps output by the lightweight network, this embodiment inputs them into different neural network modules for feature processing.

[0075] For the global feature map corresponding to the global facial image, please refer to... Figure 3 , Figure 3 The diagram illustrates a global feature analysis architecture in one embodiment of the present invention. It can be seen that, preferably, in this embodiment, the global feature map is input into a global feature extraction model, i.e., a global feature extraction module (GFEM). Integrating this module into the neural network architecture can significantly suppress redundant noise and enhance the weights of key features. In this embodiment, the global feature extraction module can be divided into two distinct stages: a "squeezing" stage, which uses global average pooling to aggregate global spatial information and reduce sensitivity to local noise; and an "activation" stage, which adjusts the importance of each channel through fully connected layers and uses the Sigmoid activation function for optimization.

[0076] Specifically, in the "squeezing" stage, the global feature map is subjected to dimensionality reduction processing using global average pooling to obtain the first feature tensor. Understandably, global average pooling can compress the spatial dimension of each channel, thereby accurately capturing the global statistical features of the channels. Then, the first feature tensor is input into two fully connected layers configured in the global feature extraction model for channel adjustment, resulting in the second feature tensor.

[0077] In the "activation" phase, the activation function is used to perform nonlinear optimization on the second feature tensor to obtain the third feature tensor.

[0078] For example, for the input global feature map X∈R H×W×C H, W, and C represent the height, width, and number of channels of the feature map, respectively. After global average pooling, X is reduced to a 1×1×C tensor, which is the first feature tensor X. pool .

[0079] Next, the pooled feature map X pool The output tensor is generated through two consecutive fully connected layers. That is, the second characteristic tensor Z is expressed as:

[0080] Z=((X pool ·W1+b1)·W2+b2)

[0081] In the formula, W1 is the weight of the first fully connected layer, b1 is the bias term of the first fully connected layer; W2 is the weight of the second fully connected layer, and b2 is the bias term of the second fully connected layer.

[0082] Subsequently, a sigmoid activation function σ is applied to Z, which normalizes the weights of each channel to between 0 and 1, resulting in a tensor S∈R. 1×1×C :

[0083] S = σ(Z)

[0084] After further reshaping the tensor S into a 1×1×C shape and then broadcasting it for dimensionality upscaling, we obtain the third feature tensor S. broadcast It should be understood that this dimensionality-upgrading process replicates and expands S along the spatial dimensions (height H and width W) to generate a weight matrix of the same dimension as the input feature map X, thereby achieving channel calibration.

[0085] After the above two stages, the third feature tensor is integrated with the global feature map to determine the global feature information. For example, the original feature map X and S are... broadcast Element-wise penalty is applied to generate the output feature map, i.e., the global feature information Output∈R H×W×C :

[0086] Output = X·S broadcast

[0087] For the dual local feature maps corresponding to local eye region images, please refer to... Figure 4 , Figure 4 The diagram illustrates a local feature fusion architecture in one embodiment of the present invention. It can be seen that, preferably in this embodiment, the first eye feature map and the second eye feature map from the dual local feature maps are respectively input into a selected local fusion model, namely the local fusion module (SFM). This module can effectively merge the outputs from the eye feature stream. In this embodiment, the local fusion module integrates skip connections, which can alleviate the gradient vanishing problem in deep networks. It performs element-wise fusion of the first eye feature map and the second eye feature map to generate a local fused feature map.

[0088] Specifically, this embodiment performs convolution and nonlinear activation processing on the local fused feature map sequentially, introducing an attention-weighted modulation mechanism during the processing to generate a locally enhanced feature map. In this embodiment, channel-group convolution is performed on the input dual-stream feature map, such as... Figure 4 As shown, the eye area is divided into two groups and subjected to 1×1 convolution (Conv) on each group. Batch normalization and GuLU activation are then performed.

[0089] For example, the first eye feature map is used as input stream 1 and the second eye feature map is used as input stream 2. 1×1 convolution is performed on each of them to generate two independent feature streams X1 and X2, each with a size of 4×4×30.

[0090] Then, the feature map X is generated by summing X1 and X2 element by element. sum , is represented as:

[0091] X sum =X1+X2

[0092] The fused feature map X sum After activation by the GeLU function, a feature map X is generated. GELU The GeLU function can enhance the nonlinear relationships between features by calculating the normal distribution and capture more complex feature relationships in the feature map.

[0093] Then X GeLU Apply a 1×1 convolution again to generate feature map X. conv The feature map X is calculated using the Sigmoid function. conv Attention weights guide the network to focus on relevant regions and suppress irrelevant information while maintaining computational efficiency. This example will focus on feature map X. conv Local enhanced feature map X generated after Sigmoid processing sigmoid It is expressed as follows:

[0094] X sigmoid =σ(X) conv ),X sigmoid ∈R 4×4×2C

[0095] Finally, the local enhanced feature map is integrated with the pre-selected second eye feature map and then subjected to feature decomposition to determine local feature information. This involves performing a 1×1 convolution on the original feature map of input stream 2 to obtain the feature map, and then combining it with the local enhanced feature map X. sigmoid Element-wise multiplication yields a 4×4×2C feature map. Then, the feature map is split at the middle channel as the breakpoint to generate two 4×4×C local feature maps, which include the local feature information of the left-eye and right-eye feature maps.

[0096] S4. Perform multiple fusion processing on the global feature information and the local feature information, and classify the driver's eye state according to the processing results.

[0097] In this embodiment, the designed multi-stream fusion processing is a three-stream fusion architecture, which fuses two local information streams and one global feature information stream. This three-stream fusion architecture sequentially performs weight analysis, channel enhancement, and weighted fusion processing on the global and local feature information. Specifically:

[0098] First, the global and local feature information are concatenated as vectors. In this embodiment, after global average pooling of each input stream, vector concatenation is performed, and the resulting joint feature X is represented as:

[0099] X = [x1, x2, x3]

[0100] In the formula, x1, x2, and x3 are the global feature information flow, local information flow 1, and local information flow 2 after pooling, respectively.

[0101] A linear layer assigns weights to the concatenated global and local feature information, and a softmax layer outputs the probability distribution of the classification.

[0102] Secondly, channel enhancement and spatial alignment are performed on the weighted global and local feature information. Specifically, the weighted global feature information undergoes first-channel enhancement and spatial alignment processing sequentially to promote semantic alignment between different streams (such as the association between global face features and local iris features), resulting in globally optimized features. Simultaneously, the weighted local feature information undergoes second-channel enhancement processing to determine locally optimized features. Preferably, lightweight convolution can be used for channel enhancement.

[0103] Finally, the global and local optimized features are weighted and fused to obtain the target facial features, which are then input into a preset classifier model to classify the driver's eye opening and closing states. In some embodiments of this invention, the classifier model is a binary classification probability distribution algorithm such as logistic regression, likelihood estimation, or Bernoulli distribution model. The final output is a binary classification result: the probability distribution of eye opening / eye closing in the current single-frame image.

[0104] In some embodiments of the present invention, the classification results are output through a linear layer during the classification process.

[0105] S5. Obtain the fatigue detection results corresponding to the driver's fatigue level based on the classification results.

[0106] Based on the driver's eye state distribution probability output by the softmax layer, the driver's eye state (open or closed) is determined. For each frame in the captured video stream, the eye state distribution probability is calculated, and the target number of frames with the driver's eyes closed is counted within a preset time period. Then, by calculating the percentage of the target number of frames out of the total number of frames within the preset time period, the quantitative result corresponding to the degree of fatigue is determined. For example, taking 300 seconds, the percentage of frames with closed eyes out of the total number of frames in 300 seconds is calculated, which is the PERCLOS value. PERCLOS is a biometric algorithm used to assess human visual attention and fatigue levels, and can accurately assess the driver's fatigue level. A higher percentage of closed eyes indicates that the driver's eyes have been closed for a long time, indicating that the driver may be fatigued or drowsy. The calculation formula is as follows:

[0107]

[0108] Where, n close N represents the number of frames in which the eyes are closed. total This represents the total number of frames captured during the observation period. The relationship between the PERCLOS metric and fatigue status is shown in the table below:

[0109] PERCLOS value Fatigue 0-0.05 In a normal waking state, the eyes close normally, and there are no signs of fatigue. 0.05-0.2 Mild fatigue, possibly accompanied by a slight feeling of drowsiness. 0.2-0.4 Moderate fatigue, significant drowsiness, and decreased concentration. 0.4-0.7 Severe fatigue, noticeably tired, easily distracted, and slowed reaction time. 0.7 and above Severe fatigue and extreme drowsiness may lead to falling asleep at any time, posing a significant safety risk.

[0110] Relationship between PERCLOS value and fatigue state

[0111] When the calculated PERCLOS value, which quantifies the degree of fatigue, exceeds a preset fatigue threshold (usually 0.7 or 70%), the driver is considered to be in a state of fatigue, and a warning will be issued. Based on the warning content and the quantification result, fatigue detection results will be generated, such as a detection log or a report including the entire detection process.

[0112] For example, if the driver is determined to be moderately fatigued, a voice prompt could be triggered, such as reminding the driver to pay attention. If the driver is determined to be severely fatigued, a seat vibration and emergency braking request could be triggered.

[0113] In summary, this invention provides a driver fatigue detection method based on multi-stream facial feature analysis. By combining a multi-task cascaded convolutional network and a lightweight neural network to extract global and local features from the acquired driver's driving images, the method improves anti-occlusion capabilities. By combining a global feature extraction module and a local stream fusion module for feature fusion, the method can more fully capture facial features at different scales, ensuring the complementarity of local and global information. Furthermore, by fusing local and global features, the method improves recognition accuracy.

[0114] One embodiment of the present invention provides a fatigue detection system based on driver facial recognition. For details, please refer to [link to documentation]. Figure 5 , Figure 5 The diagram shown illustrates the structure of a driver facial recognition-based fatigue detection system according to one embodiment of the present invention, comprising:

[0115] The facial feature detection module M1 is used to input the obtained driving image of the driver inside the target vehicle into a preset facial extraction neural network model for recognition, and obtain the driver's global facial image and local eye area image.

[0116] The feature extraction module M2 is used to input the global facial image into a pre-trained first lightweight neural network and output a global feature map, and input the local eye region image into a pre-trained second lightweight neural network and output a dual local feature map.

[0117] The feature optimization module M3 is used to input the global feature map into the global feature extraction model for feature extraction to obtain the global feature information corresponding to the global facial image, and to perform fusion analysis on the first eye feature map and the second eye feature map in the dual local feature map to obtain the local feature information corresponding to the local eye region image.

[0118] The classification module M4 is used to perform multiple fusion processing on the global feature information and the local feature information, and classify the driver's eye state according to the processing results;

[0119] The fatigue detection module M5 is used to obtain the fatigue detection result corresponding to the driver's fatigue level based on the classification result.

[0120] like Figure 6 As shown, this embodiment of the invention also provides a computer device. Figure 6 This is a structural block diagram of a preferred embodiment of a computer device provided by the present invention. The computer device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the method described above.

[0121] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2, ...), and the one or more modules / units are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device.

[0122] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor can be any conventional processor. The processor is the control center of the terminal device, connecting various parts of the terminal device through various interfaces and lines.

[0123] The memory mainly includes a program storage area and a data storage area. The program storage area can store the operating system, applications required for at least one function, etc., while the data storage area can store related data, etc. Furthermore, the memory can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard drive, a SmartMedia Card (SMC), a Secure Digital (SD) card, and a Flash Card, or other volatile solid-state storage devices.

[0124] It should be noted that the aforementioned terminal devices may include, but are not limited to, processors and memory, as will be understood by those skilled in the art. Figure 6 The structural block diagram is merely an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown, or combine certain components, or use different components. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0125] Accordingly, embodiments of the present invention provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the steps in the method of the above embodiments, for example... Figure 1 Steps S1 to S5 as described above.

[0126] The technical features and effects of the fatigue detection system based on driver facial recognition proposed in this embodiment of the invention are the same as those of the fatigue detection method based on driver facial recognition proposed in this embodiment of the invention, and will not be repeated here.

[0127] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A fatigue detection method based on driver facial recognition, characterized in that, include: The obtained driving image of the driver inside the target vehicle is input into a preset facial extraction neural network model for recognition, thereby obtaining the driver's global facial image and local eye area image; The global facial image is input into a pre-trained first lightweight neural network, and a global feature map is output. The local eye region image is input into a pre-trained second lightweight neural network, and a dual local feature map is output. The global feature map is input into a global feature extraction model for feature extraction to obtain global feature information corresponding to the global facial image. The first and second eye feature maps in the dual local feature maps are then fused and analyzed to obtain local feature information corresponding to the local eye region image. Specifically, after inputting the dual local feature maps into a selected local fusion model, the first and second eye feature maps are fused element-wise to generate a local fusion feature map. The local fusion feature map is then subjected to convolution and nonlinear activation processing sequentially, with an attention-weighted modulation mechanism introduced during the processing to generate a local enhancement feature map. The local enhancement feature map is then integrated with a pre-selected second eye feature map and subjected to feature decomposition to determine the local feature information. The global feature information and the local feature information are subjected to multiple fusion processing, and the driver's eye state is classified according to the processing results; The fatigue detection results corresponding to the driver's fatigue level are obtained based on the classification results.

2. The fatigue detection method based on driver facial recognition as described in claim 1, characterized in that, The process of acquiring the driver's global facial image and local eye region image includes: The driving image is input into the face extraction neural network model, which outputs the boundary vertex coordinates and key point coordinates of the driver's overall face; the face extraction neural network model is obtained by pre-training a multi-task cascaded convolutional network. The driving image is cropped based on the boundary vertex coordinates and the key point coordinates to obtain the global face image and the local eye region image.

3. The fatigue detection method based on driver facial recognition as described in claim 1, characterized in that, The step of inputting the global feature map into a global feature extraction model for feature extraction to obtain global feature information corresponding to the global facial image includes: In the global feature extraction model, the global feature map is subjected to dimensionality reduction processing by global average pooling to obtain the first feature tensor; The first feature tensor is input into the fully connected layer configured in the global feature extraction model for channel adjustment to obtain the second feature tensor; The second feature tensor is nonlinearly optimized using an activation function to obtain the third feature tensor; The third feature tensor is integrated with the global feature map to determine the global feature information.

4. The fatigue detection method based on driver facial recognition as described in claim 1, characterized in that, The process of performing multiple fusion processing on the global feature information and the local feature information, and classifying the driver's eye state based on the processing results, includes: The global feature information and the local feature information are sequentially subjected to weight analysis, channel enhancement and weighted fusion processing to obtain the target facial feature data; The target facial feature data is input into a preset classifier model to classify the driver's eye closing and opening states.

5. The fatigue detection method based on driver facial recognition as described in claim 4, characterized in that, The process of sequentially performing weight analysis, channel enhancement, and weighted fusion on the global feature information and the local feature information to obtain the target facial features includes: The global feature information and the local feature information are concatenated into vectors, and weights are assigned to the concatenated global feature information and local feature information respectively; The weighted global feature information is sequentially subjected to first-channel enhancement and spatial alignment processing to obtain global optimized features, and the weighted local feature information is subjected to second-channel enhancement to determine local optimized features; The target facial features are obtained by weighted fusion of the global optimization features and the local optimization features.

6. The fatigue detection method based on driver facial recognition as described in claim 1, characterized in that, The step of obtaining the fatigue detection result corresponding to the driver's fatigue level based on the classification result includes: Based on the classification results, determine the probability distribution of the driver's eye state; Based on the probability distribution of eye state, calculate the target number of frames in which the driver's eyes are closed within a preset time period; The quantification result corresponding to the fatigue level is determined by calculating the percentage of the target frame count to the total frame count within a preset time period. When the quantification result exceeds the preset fatigue threshold, a warning is issued, and the fatigue detection result is generated based on the warning content and the quantification result.

7. A fatigue detection system based on driver facial recognition, characterized in that, include: The facial feature detection module is used to input the obtained driving image of the driver inside the target vehicle into a preset facial extraction neural network model for recognition, and to obtain the driver's global facial image and local eye area image; The feature extraction module is used to input the global facial image into a pre-trained first lightweight neural network and output a global feature map, and input the local eye region image into a pre-trained second lightweight neural network and output a dual local feature map. The feature optimization module is used to input the global feature map into a global feature extraction model for feature extraction to obtain global feature information corresponding to the global facial image. It also performs fusion analysis on the first and second eye feature maps in the dual local feature maps to obtain local feature information corresponding to the local eye region image. Specifically, after inputting the dual local feature maps into a selected local fusion model, the first and second eye feature maps are fused element-wise to generate a local fusion feature map. The local fusion feature map is then subjected to convolution and nonlinear activation processing sequentially, with an attention-weighted modulation mechanism introduced during the processing to generate a local enhancement feature map. Finally, the local enhancement feature map is integrated with a pre-selected second eye feature map and then feature decomposition is performed to determine the local feature information. The classification module is used to perform multiple fusion processing on the global feature information and the local feature information, and classify the driver's eye state according to the processing results; The fatigue detection module is used to obtain the fatigue detection result corresponding to the driver's fatigue level based on the classification result.

8. A computer device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the fatigue detection method based on driver facial recognition as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the fatigue detection method based on driver facial recognition as described in any one of claims 1 to 6.

Images