Lightweight face key point detection model generation method, fatigue driving detection method and system

Abstract

The invention discloses a lightweight face key point detection model generation method and a fatigue driving detection method and system, and the method comprises the following steps: S1, image collection and preprocessing: collecting and preprocessing a training sample image, the training sample including a public data set and / or data collected and manually labeled for an actual driving scene; s2, model optimization: carrying out optimization design on the PFLD-based face key point detection model from a network structure, an output form and a training reasoning process; and S3, a model quantization method: generating a low-bit fixed-point reasoning model by adopting the model quantization method for realizing INT8 quantization deployment of the face key point detection model by adopting a QAT method. Through structural optimization and quantitative acceleration of the face key point detection model and systematic design of fatigue judgment logic, the real-time performance, stability and engineering deployment of the fatigue driving detection system in a computing power limited industrial vehicle scene are remarkably improved on the premise of ensuring the detection precision.

Description

Technical Field

[0001] This invention discloses a fatigue driving detection method and system, particularly a lightweight facial key point detection model generation method and fatigue driving detection method and system, belonging to the technical field of driving safety detection methods. Background Technology

[0002] With the continuous growth in the number of motor vehicles and industrial vehicles, fatigued driving has become one of the major factors causing traffic accidents. Studies have shown that traffic accidents caused by fatigued driving account for a high proportion and have a high fatality rate, especially in industrial vehicle operation scenarios such as forklifts, where drivers work at high intensity for long periods in complex and changing environments, making the risk of fatigue even more prominent. Therefore, accurate and real-time detection and assessment of driver fatigue is of great significance for ensuring driving safety.

[0003] Existing methods for detecting driver fatigue mainly include those based on driving behavior, vehicle status, and driver physiological characteristics. Among these, methods based on physiological signals require the collection of information such as heart rate and electroencephalogram (EEG), which are costly and pose risks of discomfort and privacy breaches. Methods based on vehicle status or driving behavior are easily affected by road conditions, vehicle performance, and environmental interference, resulting in insufficient stability of the detection results.

[0004] With the development of computer vision technology, fatigue detection methods based on facial features have gradually attracted attention. This method achieves non-contact detection by analyzing the dynamic facial features of the driver, such as the eyes and mouth, and has the advantages of flexible deployment and high user acceptance. Although existing fatigue driving detection technologies can identify the driver's fatigue state to a certain extent, they still have many shortcomings in practical applications and cannot meet the requirements of real-time performance, reliability, and low-cost deployment for industrial vehicles such as forklifts.

[0005] First, existing vision-based fatigue detection methods generally rely on complex deep learning models, which have a large number of model parameters, high computational complexity, high requirements for computing power and hardware resources, and insufficient adaptability to changes in lighting and individual differences. As a result, they are difficult to achieve stable and real-time operation on vehicle terminals (especially forklifts and other industrial vehicle terminals with limited computing power) or embedded devices, thus limiting their widespread application in industrial vehicle scenarios.

[0006] Secondly, existing technologies are insufficient in terms of model lightweighting. They mostly use floating-point models for inference without effective structural optimization or quantization acceleration, resulting in high inference latency and low frame rate. This makes it difficult to achieve real-time monitoring of driver fatigue, especially in long-term operation scenarios where detection lag is likely to occur.

[0007] Furthermore, existing fatigue assessment methods are mostly based on discrete behaviors such as closing eyes or yawning within a single or short time window, without fully considering the characteristics of fatigue state accumulating over time. This can easily lead to misjudgments. The judgment logic triggered by a single behavior is easily affected by individual differences, instantaneous actions, and environmental interference, resulting in false alarms or missed alarms and reducing the overall reliability of the system.

[0008] In addition, the existing system is not adaptable to complex operating environments. Its detection accuracy drops significantly under conditions such as changes in lighting, facial occlusion, and wearing glasses. The model's generalization ability is limited, making it difficult to meet the needs of industrial vehicles for long-term stable operation in variable environments.

[0009] Therefore, there is an urgent need for a lightweight and robust fatigue driving detection method. By using an efficient facial key point detection model, combined with model optimization and quantization acceleration technology, the computational burden can be reduced while ensuring detection accuracy. Furthermore, by using reasonable logic for judging fatigue when eyes are closed and fatigue when yawning, the method can achieve continuous and reliable assessment of the driver's fatigue state, thereby meeting the application requirements of high safety and real-time performance in industrial vehicles. Summary of the Invention

[0010] To address the significant shortcomings of existing fatigue driving detection technologies in terms of model optimization, lightweight deployment, and the rationality of fatigue judgment logic, this invention provides a lightweight facial key point detection model generation method, a fatigue driving detection method, and a system. Through structural optimization of the facial key point detection model, quantization acceleration, and systematic design of the fatigue judgment logic, the invention reduces computational burden while ensuring detection accuracy. This significantly improves the real-time performance, stability, and engineering deployability of the fatigue driving detection system in computationally limited industrial vehicle scenarios. Furthermore, it enhances detection reliability through a scientific fatigue accumulation judgment mechanism.

[0011] The technical solution adopted by this invention to solve its technical problem is: a lightweight facial landmark detection model generation method, which includes the following steps: Step S1, Image Acquisition and Preprocessing: Acquire training sample images and perform preprocessing. Training samples include public datasets and / or data collected and manually labeled for actual driving scenarios. Step S2, Model Optimization: The face landmark detection model based on PFLD was optimized in terms of network structure, output format, and training and inference process; Step S3, Model Quantization Method: The QAT method is used to implement the INT8 quantization of the face key point detection model to generate a low-ratio key point inference model.

[0012] A lightweight facial landmark detection model generation system, comprising the following functional units: Image acquisition and preprocessing unit: used to acquire training sample images and perform preprocessing. Training samples include public datasets and / or data collected and manually labeled for actual driving scenarios. Model optimization unit: This unit optimizes the PFLD-based face landmark detection model in terms of network structure, output format, and training and inference process. Model Quantization Method Unit: A model quantization method for deploying a face landmark detection model using QAT method to generate a low-ratio landmark inference model.

[0013] A lightweight method for detecting driver fatigue, comprising the following steps: Step S4-1: Begin; Step S4-2: Acquire video stream frames: Acquire video streams using a camera or other video capture device; Step S4-3, Face Detection and Key Point Extraction: The face key point detection model generated by the lightweight face key point detection model generation method described above is used to detect faces in the video stream and extract key point information. Step S4-4: Calculate the EAR of the left and right eyes and the MAR of the mouth: Calculate the EAR of the left and right eyes and the MAR of the mouth based on the key points; Step S4-5, Monocular dynamic selection: Calculate the horizontal distance H for the left and right eyes respectively. L With H R The eye with the larger horizontal distance is used as the effective eye for EAR calculation in the current frame; Step S4-6, Threshold and continuous time window judgment: When EAR or MAR meets the threshold condition and continues to reach the preset duration, output the closing eyes / yawning event; Step S4-7: Has the fatigue condition been met?: Based on the events of closing eyes / yawning, fatigue is judged. If the fatigue condition is met, proceed to step S4-8. If the fatigue condition is not met, directly jump back to step S4-2 and reacquire the video stream. Step S4-8: Save the alarm image and push a warning, then jump back to step S4-2 to reacquire the video stream.

[0014] A lightweight driver fatigue detection system includes the following functional units: Start Unit: Used to start the system and begin operation; Video Stream Frame Acquisition Unit: Used to acquire video streams via a camera or other video capture devices; Face detection and key point extraction unit: used to detect faces in the video stream and extract key point information using the face key point detection model generated by the lightweight face key point detection model generation system as described above. Calculate the left and right eye EAR and mouth MAR units: used to calculate the left and right eye EAR and mouth MAR information based on key points; Monocular dynamic selection unit: used to calculate the horizontal distance H of the left and right eyes respectively. L With H R The eye with the larger horizontal distance is used as the effective eye for EAR calculation in the current frame; Threshold and continuous time window judgment unit: used to output the closing eyes / yawning event when EAR or MAR meets the threshold condition and continues for a preset duration; Fatigue condition met unit: Used to judge fatigue based on eye closing / yawning events. If the fatigue condition is met, it will jump to save the alarm image and push the warning unit. If the fatigue condition is not met, it will jump back to the video stream frame acquisition unit to reacquire the video stream. Save alarm image and push warning unit: Used to save alarm image and push warning, then jump back to the video stream frame acquisition unit to reacquire video stream.

[0015] A storage medium storing a software program that, when run, can implement the lightweight face landmark detection model generation method described above.

[0016] An electronic product that, when running, can implement the lightweight facial landmark detection model generation method described above.

[0017] The technical solution adopted by the present invention to solve its technical problem further includes: In step S1, the image preprocessing of the training samples includes the following sub-steps: Step S1-1, Image Normalization and Keypoint Mapping: Let the input image size be... The key points of the face are:

[0018] Where W represents the width of the input image and H represents the height of the input image. P i Let N represent the facial keypoints of the i-th person, and N represent the number of keypoints. x i The x-coordinate of the key point y i The ordinate of the key point; Perform image size normalization to resize the image to a fixed size. W0 ×H 0 , W 0 Indicates the width of the input image. H 0 This represents the height of the input image size and performs a linear mapping of keypoints.

[0019] Among them, images W 0 and H 0 112 pixels each The normalized x-axis, The ordinate is the normalized ordinate; Step S1-2, Constraint-based stochastic affine transformation: Apply a random affine transformation to the image; the affine transformation matrix is ​​represented as follows:

[0020] Where M is the affine transformation formula, s is the random scaling factor, and θ is the random rotation angle. t x ，t y This represents the translation amount along the X and Y axes; Coordinates of key points after affine transformation for: Key constraints: The corresponding affine transformation result is used only when all keypoints meet the above conditions; otherwise, the transformation parameters are regenerated. Represents the x-coordinate after affine transformation. Represents the ordinate after affine transformation. This indicates that each facial key point is processed; Steps S1-3, Illumination and Color Perturbation Model: If the brightness, hue, and saturation of an image are randomly perturbed, and the original pixel value is I, and the perturbed pixel value is I′, then:

[0021] Where α is the brightness scaling factor and β is the brightness offset, both of which are randomly selected within the range of [0.5,1]. Steps S1-4: Enhance localized areas by avoiding key points: Random occlusion blocks are generated in non-critical point regions. Let the occlusion block region be a rectangle R, with its boundaries as follows: Then the occluded area must meet the following requirements:

[0022] And the minimum Euclidean distance from the keypoint to the occluded block region satisfies: Where, is d min The preset minimum safe distance, This represents the pixel value of the occluded region of the i-th face keypoint after the affine transformation.

[0023] In step S2, model optimization includes the following sub-steps: Step S2-1, Lightweight backbone network structure pruning and optimization: The face landmark detection model is based on a lightweight convolutional neural network. The backbone network consists of multiple layers of convolutional modules based on the Inverted Residual structure. The Inverted Residual structure is optimized by a combination of channel expansion, depthwise separable convolution and channel compression, reducing the number of residual modules in the first feature stage to 3 and the number of residual modules in the second feature stage to 3. Step S2-2, Simplifying the design of the number of key point outputs: The number of facial key points output is reduced to 42, and the key points cover areas that are representative of the facial geometry, such as the eyes, eyebrows, nose, and mouth. Steps S2-3: Removal of auxiliary networks and simplification of inference process: During model optimization, only the main regression branch of key points is retained.

[0024] In step S3, model quantization includes the following sub-steps: Step S3-1: Model operator fusion and graph structure preparation: The convolutional layers, normalization layers, and activation function layers in the model are fused together, and a computational graph for quantization training is constructed. Step S3-2: Set the quantization configuration and insert pseudo-quantization nodes: Select the quantization backend based on the deployment platform, and configure the activation quantization and weighted quantization strategies: Activation employs a minimum-maximum statistical method based on a moving average and maps it to an 8-bit unsigned integer range; The weights are mapped to an 8-bit signed integer range using channel-based quantization. Under this quantization configuration, pseudo-quantized nodes are inserted into the network computation graph, enabling the forward propagation process during the training phase to simulate INT8 inference error. Step S3-3: QAT training and learning rate scheduling optimization: In the early stages of training, the learning rate is gradually increased to improve convergence stability. In the later stages of training, the learning rate is smoothly decreased according to the cosine law to improve the final accuracy. Step S3-4: Freeze the quantitative statistical parameters and perform fixed-point model transformation: After QAT training is completed, the update of pseudo-quantization statistical parameters is stopped, and the training model containing pseudo-quantization nodes is converted into a true INT8 fixed-point inference model to obtain a deployable quantization model. Step S3-5: Deployment and Export: The converted INT8 model is then exported to a model format suitable for edge deployment.

[0025] The image acquisition and preprocessing unit includes the following sub-units for image preprocessing of training samples: Image normalization and keypoint mapping subunit: Let the input image size be... The key points of the face are:

[0026] Where W represents the width of the input image and H represents the height of the input image. P i Let N represent the facial keypoints of the i-th person, and N represent the number of keypoints. x i The x-coordinate of the key point y i The ordinate of the key point; Perform image size normalization to resize the image to a fixed size. W 0 ×H 0 , W 0 Indicates the width of the input image. H 0 This represents the height of the input image size and performs a linear mapping of keypoints.

[0027] Among them, images W 0 and H 0 112 pixels each The normalized x-axis, The ordinate is the normalized ordinate; Constraint-based stochastic affine transformation sub-unit: Apply a random affine transformation to the image; the affine transformation matrix is ​​represented as follows:

[0028] Where M is the affine transformation formula, s is the random scaling factor, and θ is the random rotation angle. t x ，t y This represents the translation amount along the X and Y axes; Coordinates of key points after affine transformation for: Key constraints: The corresponding affine transformation result is used only when all keypoints meet the above conditions; otherwise, the transformation parameters are regenerated. Represents the x-coordinate after affine transformation. Represents the ordinate after affine transformation. This indicates that key facial features are processed for each individual. Lighting and color perturbation model sub-unit: If the brightness, hue, and saturation of an image are randomly perturbed, and the original pixel value is I, and the perturbed pixel value is I′, then:

[0029] Where α is the brightness scaling factor and β is the brightness offset, both of which are randomly selected within the range of [0.5,1]. Avoiding local occlusion at key points to enhance sub-units: Random occlusion blocks are generated in non-critical point regions. Let the occlusion block region be a rectangle R, with its boundaries as follows: Then the occluded area must meet the following requirements:

[0030] And the minimum Euclidean distance from the keypoint to the occluded block region satisfies: Where, is d min The preset minimum safe distance, This represents the pixel value of the occluded region of the i-th face keypoint after the affine transformation.

[0031] And / or, the model optimization unit includes the following sub-units: Lightweight backbone network structure pruning and optimization sub-units: The face landmark detection model is based on a lightweight convolutional neural network. The backbone network consists of multiple layers of convolutional modules based on the Inverted Residual structure. The Inverted Residual structure is optimized by a combination of channel expansion, depthwise separable convolution and channel compression, reducing the number of residual modules in the first feature stage to 3 and the number of residual modules in the second feature stage to 3. Simplified design sub-unit for key output quantity: The number of facial key points output is reduced to 42, and the key points cover areas that are representative of the facial geometry, such as the eyes, eyebrows, nose, and mouth. Auxiliary network removal and inference process simplification sub-unit: During model optimization, only the main regression branch of key points is retained.

[0032] And / or, the model quantization method unit includes the following sub-steps: Model operator fusion and graph structure preparation sub-unit: The convolutional layers, normalization layers, and activation function layers in the model are fused together, and a computational graph for quantization training is constructed. Configure quantization and insert pseudo-quantization node sub-cells: Select the quantization backend based on the deployment platform, and configure the activation quantization and weighted quantization strategies: Activation employs a minimum-maximum statistical method based on a moving average and maps it to an 8-bit unsigned integer range; The weights are mapped to an 8-bit signed integer range using channel-based quantization. Under this quantization configuration, pseudo-quantized nodes are inserted into the network computation graph, enabling the forward propagation process during the training phase to simulate INT8 inference error. QAT Training and Learning Rate Scheduling Optimization Subunit: In the early stages of training, the learning rate is gradually increased to improve convergence stability. In the later stages of training, the learning rate is smoothly decreased according to the cosine law to improve the final accuracy. Freeze the quantitative statistical parameters and perform fixed-point model transformation sub-unit: After QAT training is completed, the update of pseudo-quantization statistical parameters is stopped, and the training model containing pseudo-quantization nodes is converted into a true INT8 fixed-point inference model to obtain a deployable quantization model. Deploy export subunit: The converted INT8 model is then exported to a model format suitable for edge deployment.

[0033] The beneficial effects of this invention are as follows: Compared with the prior art, this invention significantly improves the real-time performance, stability, and engineering deployability of the fatigue driving detection system in computing-constrained industrial vehicle scenarios by optimizing the structure of the facial key point detection model, accelerating quantization, and systematically designing the fatigue judgment logic, while ensuring detection accuracy. Specific beneficial effects are as follows: (1) While ensuring the accuracy of key point detection, this invention significantly reduces the computational complexity and resource consumption of the model. By structurally pruning and simplifying the PFLD-based face key point detection model, without changing the overall feature extraction process and scale variation law of the network, the number of repeatedly stacked Inverted Residual modules is reduced, and the key point output dimension is simplified. This effectively reduces the model parameter scale and floating-point operation (FLOPs); significantly reduces the model inference latency and memory consumption; and enables the model to run stably on vehicle terminals, embedded devices, and industrial vehicle platforms with limited computing power. Compared with existing visual fatigue detection methods that rely on highly complex network structures, this invention can achieve a higher inference frame rate under the same hardware conditions, providing reliable computing power guarantee for real-time detection of fatigue driving.

[0034] (2) By optimizing the number of key points, the stability and engineering practicality of the model are improved. This invention reduces the number of facial key points output from the traditional 64 points to 42 points, retaining only the core key points that have key representational significance for the eyes, mouth, and facial geometry: reducing the computational burden and training difficulty of the output layer; reducing the interference of redundant key points on the fatigue judgment logic; and improving the predictive stability of the model in actual driving scenarios. This design makes the model more focused on the facial area necessary for fatigue detection, effectively avoiding the problem of fatigue misjudgment caused by the amplification of errors due to irrelevant key points, and enhancing the practical value of the system in actual engineering deployment.

[0035] (3) Removing the auxiliary pose network simplifies the inference process and improves real-time response capability. Compared with existing key point detection models that introduce pose-assisted networks for constraints, this invention removes the pose estimation auxiliary branch while ensuring the accuracy of key point regression, retaining only the key point regression backbone network. This reduces additional computational paths and model parameters, mitigates synchronization and latency issues caused by multi-branch inference, and makes the model inference process more concise and efficient. This optimization significantly improves the model's response speed in real-time video stream processing scenarios, and is particularly suitable for industrial vehicle applications such as forklifts that have high requirements for real-time safety warnings.

[0036] (4) By using quantization-aware training, the keypoint offset problem caused by fixed-point quantization is effectively solved. Addressing the issue of accuracy degradation in keypoint regression tasks caused by traditional PTQ quantization, this invention employs quantization-aware training (QAT), enabling the model to perceive and adapt to fixed-point quantization errors during the training phase. This significantly reduces keypoint offset in side-view, large-pose, and occluded scenarios using the INT8 quantization model; ensures that the quantized model's detection accuracy is essentially consistent with the FP32 floating-point model; and achieves approximately a 3-fold increase in model inference speed while significantly reducing storage and computational costs. This technology directly improves the long-term stable operation capability of the fatigue detection system on embedded devices, solving the problem of high-precision quantization deployment of existing visual fatigue detection models.

[0037] (5) Based on the monocular dynamic selection logic for eye closure determination, the robustness of detection under complex postures is improved. Addressing the issue of EAR jitter in existing fatigue detection methods when the face is turned to the side, head tilted, or one eye is occluded, this invention proposes a monocular dynamic selection strategy based on horizontal eye distance: automatically selecting the effective eye with more complete key points and less occlusion for EAR calculation; effectively avoiding outlier interference caused by binocular fusion; and significantly improving the stability and accuracy of eye-closure fatigue detection in complex posture scenarios. This logic enables the system to adapt to the driver's natural head posture changes in actual industrial vehicle operations, reducing the risk of false alarms and missed alarms.

[0038] (6) Introducing a time accumulation mechanism to accurately reflect the persistence characteristics of fatigue: This invention adopts a fatigue judgment mechanism based on continuous time windows to confirm the duration of eye-closing and yawning behaviors; effectively distinguishing between normal blinking, short-term mouth opening and real fatigue behaviors; avoiding false alarms caused by instantaneous movements or key point shaking; and better conforming to the objective physiological characteristics of fatigue gradually accumulating over time. Compared with existing judgment methods that rely on single behavior triggers, this invention significantly improves the reliability and stability of fatigue detection results.

[0039] (7) The overall system improves the deployability and safety value in industrial vehicle scenarios. Through the collaborative design of model optimization, quantization acceleration and scientific fatigue judgment logic, the present invention achieves the following comprehensive technical effects: high-precision and low-latency fatigue detection on computing-limited devices; stable operation in complex lighting, posture changes and occlusion scenarios; and meets the application needs of industrial vehicles such as forklifts for continuous safety monitoring and real-time alarms.

[0040] Therefore, this invention differs from existing fatigue driving detection technologies in that it not only identifies fatigue behavior but also provides a highly reliable, low-cost, and scalable fatigue driving detection solution for industrial vehicle applications, demonstrating significant engineering application value and promising prospects for wider adoption.

[0041] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of 42 key points in this invention.

[0043] Figure 2 This is a comparison diagram of the quantitative effects in this invention.

[0044] Figure 3 This is a flowchart of the fatigue driving judgment logic in this invention. Detailed Implementation

[0045] This embodiment is a preferred embodiment of the present invention. All other embodiments that are the same as or similar to this embodiment in principle and basic structure are within the protection scope of the present invention.

[0046] This invention primarily protects a lightweight fatigue driving detection method, which includes the following steps: Step S1, Image Acquisition and Preprocessing: In order to generate training sample images for the facial landmark detection model, in this embodiment, image acquisition and preprocessing are used to improve the robustness and generalization ability of the facial landmark detection model in complex driving environments. Training samples include public datasets and data collected and manually labeled for actual driving scenarios.

[0047] Facial landmarks refer to the set of feature points used to characterize the geometric structure of a face, including but not limited to feature points in areas such as the eyes, eyebrows, nose, and mouth. In this invention, facial landmarks are used to calculate the eye aspect ratio (EAR) and mouth aspect ratio (MAR) to support the determination of fatigued driving behavior.

[0048] In this embodiment, the image preprocessing for the training samples includes the following sub-steps: Step S1-1, Image Normalization and Keypoint Mapping: Let the input image size be... The key points of the face are:

[0049] Where W represents the width of the input image and H represents the height of the input image. P i Let N represent the facial keypoints of the i-th person, and N represent the number of keypoints. x i The x-coordinate of the key point y i The vertical coordinates of the key points are shown below.

[0050] First, the image is normalized to a fixed size.W 0 ×H 0 , W 0 Indicates the width of the input image. H 0 This represents the height of the input image size and performs a linear mapping of keypoints.

[0051] This ensures that the image and key points are processed in a unified coordinate system. The image... W 0 and H 0 112 pixels each The normalized x-axis, The ordinate is the normalized ordinate.

[0052] Step S1-2, Constraint-based stochastic affine transformation: To simulate changes in driver head posture, a random affine transformation is applied to the image. The affine transformation matrix is ​​represented as follows:

[0053] Where M is the affine transformation formula, s is the random scaling factor, and θ is the random rotation angle. t x ，t y These represent the translation amounts along the X and Y axes. The coordinates of the keypoints after affine transformation. for: To prevent key points from going out of bounds, key point constraints are introduced in this embodiment:

[0054] The corresponding affine transformation result is only used if all keypoints meet the above conditions; otherwise, the transformation parameters (including scaling factor s, rotation angle θ, and translation amount t) are regenerated. x Translational displacement t y (one or more), where, Represents the x-coordinate after affine transformation. Represents the ordinate after affine transformation. This indicates that key facial features are processed for each individual.

[0055] Steps S1-3, Illumination and Color Perturbation Model: To enhance the model's adaptability to changes in lighting, the image brightness, hue, and saturation are randomly perturbed. Let the original pixel value be I, and the perturbed pixel value be I′, then:

[0056] Here, α is the brightness scaling factor, and β is the brightness offset, both randomly selected within the range of [0.5, 1]. Simultaneously, random grayscale processing is introduced so that some samples retain only brightness information, thereby enhancing the model's stability in low-light environments.

[0057] Steps S1-4: Enhance localized areas by avoiding key points: To simulate facial occlusion and noise interference, random occlusion blocks are generated in non-critical point regions. Let the occlusion region be a rectangle R, with its boundaries as follows: Then the occluded area must meet the following requirements:

[0058] And the minimum Euclidean distance from the keypoint to the occluded block region satisfies: Where, is d min The preset minimum safe distance, The value represents the pixel value of the occluded area of ​​the i-th face keypoint after affine transformation. It is generated by low-brightness filling or random texture overlay to simulate occlusion, noise or background interference.

[0059] Step S2, Model Optimization: To further reduce the computational complexity, parameter size, and inference latency of the model while ensuring the accuracy of facial landmark localization, this invention optimizes the facial landmark detection model based on PFLD in terms of network structure, output format, and training and inference process.

[0060] Among them, the PFLD (Practical Facial Landmark Detector) model is a facial landmark detection model based on a lightweight convolutional neural network structure, suitable for deployment on mobile or embedded devices. This invention is based on this model and optimizes it through network pruning, reduction of the number of landmarks, and simplification of the inference process.

[0061] Model optimization includes the following sub-steps: Step S2-1, Lightweight backbone network structure pruning and optimization: The facial landmark detection model used in this invention is based on a lightweight convolutional neural network. Its backbone network consists of multiple cascaded convolutional modules based on an inverted residual structure. The inverted residual structure effectively reduces computational cost while maintaining feature representation capability through a combination of channel expansion, depthwise separable convolution, and channel compression. Channel expansion expands the input channels by 4 using 1×1 convolution, providing a higher-dimensional feature representation space for depthwise convolution. Depthwise separable convolution extracts spatial features in the expanded high-dimensional space through 3×3 channel-wise convolution, significantly reducing the number of parameters. Channel compression then compresses the number of channels back to the original dimension using 1×1 convolution, forming a bottleneck structure.

[0062] Among them, the Inverted Residual structure is a lightweight convolutional network structure that typically includes channel expansion, depthwise separable convolution, and channel compression steps. It is used to reduce the number of model parameters and computational complexity while ensuring feature representation capabilities.

[0063] Based on this, the present invention optimizes and prunes the mid-to-late layer convolutional modules in the backbone network that are repeatedly stacked and have a relatively low contribution to keypoint localization. In the original model, four consecutive Inverted Residual modules with the same number of channels are set in the first feature stage, and five consecutive Inverted Residual modules with the same number of channels are set in the second feature stage. Although the above structure helps to enhance the deep feature representation capability, it also introduces a large amount of computational redundancy.

[0064] To reduce model complexity, this invention reduces the number of repeatedly stacked residual modules while maintaining the overall hierarchical structure of the network and the way the feature scale changes. The number of residual modules in the first feature stage is reduced from 4 to 3, and the number of residual modules in the second feature stage is reduced from 5 to 3, thereby effectively reducing network depth and computational complexity.

[0065] By using the above-mentioned pruning method, the number of model parameters and floating-point operations are effectively reduced without significantly affecting the key point detection performance, thereby improving the model's running efficiency on resource-constrained devices.

[0066] Step S2-2, Simplifying the design of the number of key point outputs: Traditional facial landmark detection models typically output a large number of landmarks to describe facial structure. However, in real-world applications, some landmarks contribute little to the final task and may even increase the computational burden on the model.

[0067] Therefore, this invention optimizes the number of key points output, reducing the original number of facial key points from 64 to 42. The retained key points mainly cover areas representative of facial geometry, such as the eyes, eyebrows, nose, and mouth. Please refer to the appendix. Figure 1 .

[0068] By reducing the number of keypoint outputs, the parameter size and computational cost of the model's output layer are reduced accordingly, making the model more efficient in the training and inference phases, while also improving the model's stability and practicality in real-world deployment scenarios.

[0069] Steps S2-3: Removal of auxiliary networks and simplification of inference process: In traditional face landmark detection models, auxiliary networks are typically introduced to estimate face pose information to help constrain the landmark regression process. However, these auxiliary networks introduce additional computational overhead and inference latency.

[0070] This invention proposes a neural network architecture for facial landmark detection designed for efficient deployment. This architecture adopts a single-task-focused design philosophy, constructing an end-to-end landmark coordinate regression network. Specifically, the model consists of two main parts: a feature extraction backbone network and a landmark regression output layer. The backbone network is responsible for extracting multi-level feature representations from the input image, while the regression output layer maps the features to the final landmark coordinates.

[0071] In terms of technical implementation, this model avoids the complex branching structures common in multi-task learning. Instead, it ensures keypoint detection performance by optimizing the feature representation capabilities of the backbone network. The direct advantage of this design is the simplification of the inference process: the model only needs to perform a single forward propagation to obtain the coordinates of all keypoints, eliminating the need for additional task coordination or result fusion steps. This inference process not only reduces computational complexity but also decreases the model's overhead in terms of memory usage, power consumption, and latency, making it particularly suitable for deployment on embedded devices, mobile terminals, and edge computing platforms with limited computing power.

[0072] This invention removes the pose-aiding network during model optimization, retaining only the keypoint regression main branch. This eliminates the need for additional pose estimation calculations during the inference phase, further reducing overall computational complexity and model size. This approach simplifies the model inference process, making it easier to deploy in applications with high real-time requirements.

[0073] By implementing multiple optimization measures, such as pruning the backbone network structure, reducing the number of key point outputs, and removing auxiliary networks, the face key point detection model of this invention achieves a significant reduction in the number of model parameters and computational load while maintaining high detection accuracy.

[0074] The above optimization scheme effectively improves the model's real-time performance and adaptability to engineering deployment, and is especially suitable for facial landmark detection applications in mobile terminals, embedded devices, and computing-limited scenarios.

[0075] Step S3, Model Quantization Method: To meet the real-time deployment requirements of computing-constrained devices such as vehicle-mounted or industrial terminals, this invention further employs model quantization technology based on the aforementioned facial landmark detection model. This transforms the floating-point inference model into a low-ratio fixed-point inference model, reducing computational complexity, storage overhead, and inference latency. The effect of QAT quantization is as follows: Figure 2 As shown, the results are basically consistent with the accuracy of FP32 on the computer, ensuring consistent detection results between the embedded device and the computer. Furthermore, the model quantized by QAT is 3 times faster than the model quantized by FP32.

[0076] Existing Post-Training Quantization (PTQ) is a model compression method that directly performs fixed-point quantization on a floating-point model after training. This method eliminates the need for retraining, but it can easily introduce significant quantization errors in high-precision regression tasks. It typically estimates the quantization scale and maps floating-point to fixed-point values ​​directly based on a small amount of calibration data without retraining the model. In practical tests, it was found that for facial landmark regression tasks, especially in scenarios involving side profiles, significant pose changes, or partial occlusion, the PTQ-quantized INT8 model (also known as the INT8 fixed-point model, a neural network model that uses 8-bit integers to represent weights and / or activation values. Compared to the FP32 floating-point model, the INT8 model has lower storage overhead and higher inference efficiency, making it suitable for embedded devices with limited computing power) exhibits significant landmark shifts. This leads to a significant increase in prediction errors for some landmarks, thereby affecting the stability and reliability of fatigue detection logic. The reason is that keypoint regression is a high-precision continuous value prediction task, and the model is sensitive to changes in feature distribution. When PTQ is used for direct quantization, the quantization error cannot be compensated through the training process. In particular, the feature distribution is much different from that of the frontal face in the side face pose, which makes it easier for activation quantization error to accumulate, thus causing the overall keypoint shift.

[0077] To address the aforementioned issues, this invention proposes a model quantization method based on Quantization Aware Training (QAT), which introduces a pseudo-quantization operator during model training to enable the model to perceive and adapt to quantization errors during training, thereby maintaining high inference accuracy after conversion to a fixed-point model. This invention employs the QAT method to implement INT8 quantization deployment for a facial landmark detection model. This method introduces a pseudo-quantization operator during model training to simulate quantization errors, allowing model parameters to adapt to fixed-point representations during training, thus significantly reducing the accuracy loss caused by quantization.

[0078] This invention employs a quantization-aware training process based on graph mode (FX Graph Mode), which includes the following sub-steps: Step S3-1: Model operator fusion and graph structure preparation: The convolutional layers, normalization layers, and activation function layers in the model are fused (e.g., fusion of convolutional layers and batch normalization layers) to reduce the number of operators in the inference graph and reduce the accumulation of quantization error; at the same time, a computation graph for quantization training is constructed so that pseudo-quantization nodes can be inserted at the graph level.

[0079] Step S3-2: Set the quantization configuration and insert pseudo-quantization nodes: Select the quantization backend based on the deployment platform (e.g., QNNPACK for ARM platforms, FBGEMM for x86 platforms), and configure the activation quantization and weighted quantization strategies: Activation employs a moving average-based minimum-maximum statistical method and maps it to an 8-bit unsigned integer range; The weights are mapped to an 8-bit signed integer range using a per-channel quantization method to reduce quantization errors caused by distribution differences between channels.

[0080] Under this quantization configuration, pseudo-quantized nodes are inserted into the network computation graph, enabling the forward propagation process during the training phase to simulate INT8 inference error.

[0081] Step S3-3: QAT training and learning rate scheduling optimization: During the QAT training phase, fine-tuning is performed using the original floating-point model parameters as initial weights. To ensure training stability, this invention employs a learning rate adjustment strategy combining Warmup and Cosine Annealing: the learning rate is gradually increased in the early stages of training to improve convergence stability, and the learning rate is smoothly decreased according to a cosine law in the later stages of training to improve final accuracy.

[0082] Step S3-4: Freeze the quantitative statistical parameters and perform fixed-point model transformation: After QAT training is completed, the update of pseudo-quantization statistical parameters is stopped, so that the quantization scale and zero-point parameters are fixed; and the training model containing pseudo-quantization nodes is converted into a true INT8 fixed-point inference model to obtain a deployable quantization model.

[0083] Step S3-5: Deployment and Export: The converted INT8 model is further exported into a model format suitable for edge deployment (e.g., TorchScript model file) to support direct loading and execution in embedded edge inference frameworks.

[0084] Compared with PTQ quantization, the QAT quantization perceptual training method adopted in this invention can significantly reduce the impact of quantization error on keypoint regression accuracy. In particular, it can effectively suppress keypoint offset problems in side face pose or large pose scenes, so that the quantized INT8 model is basically consistent with the FP32 floating-point model (i.e., 32-bit floating-point model, which refers to a neural network model that uses 32-bit floating-point numbers to represent weights and activation values, usually used as the benchmark form for model training and accuracy evaluation) in terms of prediction accuracy.

[0085] Therefore, this invention achieves accelerated model inference and storage compression while ensuring the accuracy and robustness of key point detection, thereby improving the real-time performance and deployability of the fatigue detection system in scenarios with limited computing power.

[0086] Step S4, Fatigue Driving Judgment Logic: This invention provides a fatigue driving judgment logic based on facial key point extraction and behavior analysis from video streams. The system processes the input video stream frame by frame, acquiring the driver's face region and extracting the coordinates of key points on the eyes and mouth. It calculates the eye aspect ratio (EAR, a geometric ratio index calculated based on eye key points to describe the degree of eye opening; it is calculated as the ratio of the vertical distance to the horizontal distance between the eyes; when the EAR is below a preset threshold for a certain period, it can be judged as closed-eye behavior) and the mouth aspect ratio (MAR, a geometric ratio index calculated based on mouth key points to describe the degree of mouth opening; it is calculated as the ratio of the vertical distance between the upper and lower lips to the horizontal distance between the corners of the mouth; when the MAR is above a preset threshold for a certain period, it can be judged as yawning behavior). The system identifies closed-eye and yawning behaviors by combining thresholds and continuous time windows, and triggers a fatigue alarm and report after a preset duration is met. The specific flowchart is shown below. Figure 3 As shown.

[0087] The process of determining driver fatigue includes the following sub-steps: Step S4-1: Begin; Step S4-2: Acquire video stream frames: Acquire video streams using a camera or other video capture device; Step S4-3, Face Detection and Key Point Extraction: Use a face key point detection model to detect faces in the video stream and extract key point information; Step S4-4: Calculate the EAR of the left and right eyes and the MAR of the mouth: Calculate the EAR of the left and right eyes and the MAR of the mouth based on the key points; Step S4-5, Monocular dynamic selection: Calculate the horizontal distance H for the left and right eyes respectively. L With H R The eye with the larger horizontal distance is used as the effective eye for EAR calculation in the current frame; Step S4-6, Threshold and continuous time window judgment: When EAR or MAR meets the threshold condition and continues to reach the preset duration, output the closing eyes / yawning event; Step S4-7: Has the fatigue condition been met?: Based on the events of closing eyes / yawning, fatigue is judged. If the fatigue condition is met, proceed to step S4-8. If the fatigue condition is not met, directly jump back to step S4-2 and reacquire the video stream. Step S4-8: Save the alarm image and push a warning, then jump back to step S4-2 to reacquire the video stream.

[0088] In this embodiment, the indicator calculation and key point constraints include the following two points: Calculation of the eye aspect ratio (EAR): For a single eye, the horizontal distance and two sets of vertical distances are calculated based on eye key points, yielding:

[0089] Where H represents the horizontal distance between the two key points at the ends of the eye, and V1 and V2 represent the vertical distance between the two sets of key points on the upper and lower eyelids.

[0090] In real-world driving scenarios, drivers may tilt their heads or turn their faces to the side, causing one eye to be obstructed, poorly lit, or have incomplete key point localization. If the two eyes are still fused using average or minimum values, abnormal EAR jitter can easily occur, leading to false alarms or missed alarms.

[0091] To improve stability in side-view and occluded scenarios, this invention employs a monocular dynamic selection strategy: calculating the horizontal distance H between the left and right eyes respectively. L With H R The eye with the larger horizontal distance is used as the effective eye for EAR calculation in the current frame, that is: If H L ≥HR If so, then the left eye is selected to calculate the EAR; If H L <H R If so, then the right eye is selected to calculate the EAR.

[0092] Since a larger horizontal distance usually indicates that the eye is closer to the front and less obstructed, this strategy can effectively alleviate the instability of key points caused by side profiles or partial occlusion, thereby improving the reliability of closed-eye determination.

[0093] Calculation of the mouth's aspect ratio (MAR): Based on the calculation of horizontal distances and three sets of vertical distances from key points at the mouth, we obtain:

[0094] Where K represents the horizontal distance between the corners of the mouth, and T1, T2, and T3 represent the vertical distance between the key points of the upper and lower lips at different locations.

[0095] In this embodiment, the fatigue driving determination mechanism is as follows: To avoid misjudgment caused by blinking, short-term mouth opening, or instantaneous errors at key points, this invention adopts a continuous time window confirmation mechanism: the closed eye / yawning event is only output when EAR or MAR meets the threshold condition and continues for a preset duration.

[0096] In the specific implementation, "lasting 2 seconds" is converted to the corresponding number of frames N = 2 × fps in video frame rate (fps). When the number of consecutive frames that meet the condition reaches N, the behavior is confirmed to be valid, and fatigue judgment and reporting are triggered.

[0097] In this embodiment, the eye aspect ratio (EAR) threshold is set to 0.15. The driver is determined to be in a closed-eye state when the following conditions are met: An effective eye EAR value was obtained by employing a monocular selection strategy. If EAR < 0.15, then the count will be accumulated even if the eyes are closed. If the above conditions are met continuously for 2 seconds, the act of closing the eyes is confirmed and the closing eyes event is output.

[0098] When the EAR recovers to above the threshold, the closed-eye count is reset to zero or decayed according to the set method to ensure timely recovery of the state.

[0099] In this embodiment, the mouth aspect ratio (MAR) threshold is set to 0.5. The driver is determined to be yawning when the following conditions are met: If MAR > 0.5, then the yawn count will be incremented. If the above conditions are met continuously for 2 seconds, the yawning behavior is confirmed and the yawning event is output.

[0100] When the MAR falls below the threshold, the yawn count is reset to zero or decays according to the set method.

[0101] This invention also protects a lightweight fatigue driving detection system, which includes the following functional units: Image acquisition and preprocessing unit: In this embodiment, image acquisition and preprocessing are used to improve the robustness and generalization ability of the facial landmark detection model in complex driving environments in order to generate training sample images for the model. Training samples include publicly available datasets and manually labeled data collected for real-world driving scenarios.

[0102] In this embodiment, the image preprocessing for training samples includes the following sub-units: Image normalization and keypoint mapping subunit: Let the input image size be... The key points of the face are:

[0103] Where W represents the width of the input image and H represents the height of the input image. P i Let N represent the facial keypoints of the i-th person, and N represent the number of keypoints. x i The x-coordinate of the key point y i The vertical coordinates of the key points are shown below.

[0104] First, the image is normalized to a fixed size. W 0 ×H 0 , W 0 Indicates the width of the input image. H 0 This represents the height of the input image size and performs a linear mapping of keypoints.

[0105] This ensures that the image and key points are processed in a unified coordinate system. The image... W 0 and H 0 112 pixels each The normalized x-axis, The ordinate is the normalized ordinate.

[0106] Constraint-based stochastic affine transformation sub-unit: To simulate changes in driver head posture, a random affine transformation is applied to the image. The affine transformation matrix is ​​represented as follows:

[0107] Where M is the affine transformation formula, s is the random scaling factor, and θ is the random rotation angle. t x ，t y These represent the translation amounts along the X and Y axes. The coordinates of the keypoints after affine transformation. for: To prevent key points from going out of bounds, key point constraints are introduced in this embodiment: The corresponding affine transformation result is used only when all keypoints meet the above conditions; otherwise, the transformation parameters are regenerated. Represents the x-coordinate after affine transformation. Represents the ordinate after affine transformation. This indicates that key facial features are processed for each individual.

[0108] Lighting and color perturbation model sub-unit: To enhance the model's adaptability to changes in lighting, the image brightness, hue, and saturation are randomly perturbed. Let the original pixel value be I, and the perturbed pixel value be I′, then:

[0109] Here, α is the brightness scaling factor, and β is the brightness offset, both randomly selected within the range of [0.5, 1]. Simultaneously, random grayscale processing is introduced so that some samples retain only brightness information, thereby enhancing the model's stability in low-light environments.

[0110] Avoiding local occlusion at key points to enhance sub-units: To simulate facial occlusion and noise interference, random occlusion blocks are generated in non-critical point regions. Let the occlusion block region be a rectangle R, with its boundaries as... Then the occluded area must meet the following requirements:

[0111] And the minimum Euclidean distance from the keypoint to the occluded block region satisfies: Where, is d min The preset minimum safe distance, The value represents the pixel value of the occluded area of ​​the i-th face keypoint after the affine transformation. It is generated by low-brightness filling or random texture overlay to simulate occlusion, noise or background interference.

[0112] Model optimization unit: To further reduce the computational complexity, parameter size, and inference latency of the model while ensuring the accuracy of facial landmark localization, this invention optimizes the facial landmark detection model based on PFLD in terms of network structure, output format, and training and inference process.

[0113] Model optimization includes the following sub-units: Lightweight backbone network structure pruning and optimization sub-units: The facial landmark detection model used in this invention is based on a lightweight convolutional neural network. Its backbone network consists of multiple cascaded convolutional modules based on an inverted residual structure. The inverted residual structure effectively reduces computational cost while maintaining feature representation capability through a combination of channel expansion, depthwise separable convolution, and channel compression. Channel expansion expands the input channels by 4 using 1×1 convolution, providing a higher-dimensional feature representation space for depthwise convolution. Depthwise separable convolution extracts spatial features in the expanded high-dimensional space through 3×3 channel-wise convolution, significantly reducing the number of parameters. Channel compression then compresses the number of channels back to the original dimension using 1×1 convolution, forming a bottleneck structure.

[0114] Based on this, the present invention optimizes and prunes the mid-to-late layer convolutional modules in the backbone network that are repeatedly stacked and have a relatively low contribution to keypoint localization. In the original model, four consecutive Inverted Residual modules with the same number of channels are set in the first feature stage, and five consecutive Inverted Residual modules with the same number of channels are set in the second feature stage. Although the above structure helps to enhance the deep feature representation capability, it also introduces a large amount of computational redundancy.

[0115] To reduce model complexity, this invention reduces the number of repeatedly stacked residual modules while maintaining the overall hierarchical structure of the network and the way the feature scale changes. The number of residual modules in the first feature stage is reduced from 4 to 3, and the number of residual modules in the second feature stage is reduced from 5 to 3, thereby effectively reducing network depth and computational complexity.

[0116] By using the above-mentioned pruning method, the number of model parameters and floating-point operations are effectively reduced without significantly affecting the key point detection performance, thereby improving the model's running efficiency on resource-constrained devices.

[0117] Simplified design sub-unit for key output quantity: Traditional facial landmark detection models typically output a large number of landmarks to describe facial structure. However, in real-world applications, some landmarks contribute little to the final task and may even increase the computational burden on the model.

[0118] Therefore, this invention optimizes the number of key points output, reducing the original number of facial key points from 64 to 42. The retained key points mainly cover areas representative of facial geometry, such as the eyes, eyebrows, nose, and mouth. Please refer to the appendix. Figure 1 .

[0119] By reducing the number of keypoint outputs, the parameter size and computational cost of the model's output layer are reduced accordingly, making the model more efficient in the training and inference phases, while also improving the model's stability and practicality in real-world deployment scenarios.

[0120] Auxiliary network removal and inference process simplification sub-unit: In traditional face landmark detection models, auxiliary networks are typically introduced to estimate face pose information to help constrain the landmark regression process. However, these auxiliary networks introduce additional computational overhead and inference latency.

[0121] This invention proposes a neural network architecture for facial landmark detection designed for efficient deployment. This architecture adopts a single-task-focused design philosophy, constructing an end-to-end landmark coordinate regression network. Specifically, the model consists of two main parts: a feature extraction backbone network and a landmark regression output layer. The backbone network is responsible for extracting multi-level feature representations from the input image, while the regression output layer maps the features to the final landmark coordinates.

[0122] In terms of technical implementation, this model avoids the complex branching structures common in multi-task learning. Instead, it ensures keypoint detection performance by optimizing the feature representation capabilities of the backbone network. The direct advantage of this design is the simplification of the inference process: the model only needs to perform a single forward propagation to obtain the coordinates of all keypoints, eliminating the need for additional task coordination or result fusion steps. This inference process not only reduces computational complexity but also decreases the model's overhead in terms of memory usage, power consumption, and latency, making it particularly suitable for deployment on embedded devices, mobile terminals, and edge computing platforms with limited computing power.

[0123] This invention removes the pose-aiding network during model optimization, retaining only the keypoint regression main branch. This eliminates the need for additional pose estimation calculations during the inference phase, further reducing overall computational complexity and model size. This approach simplifies the model inference process, making it easier to deploy in applications with high real-time requirements.

[0124] By implementing multiple optimization measures, such as pruning the backbone network structure, reducing the number of key point outputs, and removing auxiliary networks, the face key point detection model of this invention achieves a significant reduction in the number of model parameters and computational load while maintaining high detection accuracy.

[0125] The above optimization scheme effectively improves the model's real-time performance and adaptability to engineering deployment, and is especially suitable for facial landmark detection applications in mobile terminals, embedded devices, and computing-limited scenarios.

[0126] Model quantization method unit: To meet the real-time deployment requirements of computing-constrained devices such as vehicle-mounted or industrial terminals, this invention further employs model quantization technology based on the aforementioned facial landmark detection model. This transforms the floating-point inference model into a low-ratio fixed-point inference model, reducing computational complexity, storage overhead, and inference latency. The effect of QAT quantization is as follows: Figure 2 As shown, the results are basically consistent with the accuracy of FP32 on the computer, ensuring consistent detection results between the embedded device and the computer. Furthermore, the model quantized by QAT is 3 times faster than the model quantized by FP32.

[0127] Existing Post-Training Quantization (PTQ) typically estimates the quantization scale and maps floating-point to fixed-point values ​​directly based on a small amount of calibration data without retraining the model. In actual testing, the inventors found that for facial landmark regression tasks, especially in scenarios involving side profiles, significant pose changes, or partial occlusion, the PTQ-quantized INT8 model exhibits a significant landmark shift, leading to a substantial increase in prediction errors for some landmarks and consequently affecting the stability and reliability of fatigue detection logic. This is because landmark regression is a high-precision continuous value prediction task, and the model is highly sensitive to changes in feature distribution. When using direct PTQ quantization, the quantization error cannot be compensated for during training. Especially in side profile poses, where the feature distribution differs significantly from that of a frontal face, the accumulation of activation quantization errors is more likely, resulting in an overall landmark shift.

[0128] To address the aforementioned issues, this invention proposes a model quantization method based on Quantization Aware Training (QAT). This method introduces a pseudo-quantization operator during the model training phase to simulate quantization error, enabling the model parameters to adapt to fixed-point representation during training, thereby significantly reducing the accuracy loss caused by quantization.

[0129] This invention employs a quantization-aware training process based on graph mode (FX Graph Mode), which includes the following sub-units: Model operator fusion and graph structure preparation sub-unit: The convolutional layers, normalization layers, and activation function layers in the model are fused (e.g., fusion of convolutional layers and batch normalization layers) to reduce the number of operators in the inference graph and reduce the accumulation of quantization error; at the same time, a computation graph for quantization training is constructed so that pseudo-quantization nodes can be inserted at the graph level.

[0130] Configure quantization and insert pseudo-quantization node sub-cells: Select the quantization backend based on the deployment platform (e.g., QNNPACK for ARM platforms, FBGEMM for x86 platforms), and configure the activation quantization and weighted quantization strategies: Activation employs a moving average-based minimum-maximum statistical method and maps it to an 8-bit unsigned integer range; The weights are mapped to an 8-bit signed integer range using a per-channel quantization method to reduce quantization errors caused by distribution differences between channels.

[0131] Under this quantization configuration, pseudo-quantized nodes are inserted into the network computation graph, enabling the forward propagation process during the training phase to simulate INT8 inference error.

[0132] QAT Training and Learning Rate Scheduling Optimization Subunit: During the QAT training phase, fine-tuning is performed using the original floating-point model parameters as initial weights. To ensure training stability, this invention employs a learning rate adjustment strategy combining Warmup and Cosine Annealing: the learning rate is gradually increased in the early stages of training to improve convergence stability, and the learning rate is smoothly decreased according to a cosine law in the later stages of training to improve final accuracy.

[0133] Freeze the quantitative statistical parameters and perform fixed-point model transformation sub-unit: After QAT training is completed, the update of pseudo-quantization statistical parameters is stopped, so that the quantization scale and zero-point parameters are fixed; and the training model containing pseudo-quantization nodes is converted into a true INT8 fixed-point inference model to obtain a deployable quantization model.

[0134] Deploy export subunit: The converted INT8 model is further exported into a model format suitable for edge deployment (e.g., TorchScript model file) to support direct loading and execution in embedded edge inference frameworks.

[0135] Compared with PTQ quantization, the QAT quantization perceptual training method adopted in this invention can significantly reduce the impact of quantization error on the accuracy of key point regression. In particular, it can effectively suppress key point offset problems in side face pose or large pose scenes, so that the quantized INT8 model is basically consistent with the FP32 floating point model in terms of prediction accuracy.

[0136] Therefore, this invention achieves accelerated model inference and storage compression while ensuring the accuracy and robustness of key point detection, thereby improving the real-time performance and deployability of the fatigue detection system in scenarios with limited computing power.

[0137] Fatigue driving detection logic unit: This invention provides a fatigue driving determination logic based on facial key point extraction and behavior analysis from video streams. The system processes the input video stream frame by frame, acquires the driver's face region, and extracts the coordinates of key points on the eyes and mouth, calculating the eye aspect ratio (EAR) and mouth aspect ratio (MAR). It identifies closed eyes and yawning behaviors by combining thresholds and continuous time windows, triggering a fatigue alarm and reporting after a preset duration. The specific flowchart is shown below. Figure 3 As shown.

[0138] The fatigue driving determination includes the following sub-units: Start subunit: System program startup; Video stream frame acquisition subunit: used to acquire video streams via cameras or other video acquisition devices; Face detection and key point extraction subunit: Used to detect faces in video streams using a face key point detection model and extract key point information; Calculate the left and right eye EAR and mouth MAR sub-units: used to calculate the left and right eye EAR and mouth MAR information based on key points; Monocular dynamic selection subunit: used to calculate the horizontal distance H of the left and right eyes respectively. L With H R The eye with the larger horizontal distance is used as the effective eye for EAR calculation in the current frame; Threshold and continuous time window judgment sub-unit: used to output the closing eyes / yawning event when EAR or MAR meets the threshold condition and continues for a preset duration; The fatigue condition subunit is used to determine fatigue based on the events of closing eyes / yawning. If the fatigue condition is met, it jumps to the unit to save the alarm image and push the warning. If the fatigue condition is not met, it jumps back to the unit to acquire video stream frames and reacquire the video stream. Save alarm image and push warning sub-unit: Used to save alarm image and push warning, then jump back to the video stream frame acquisition sub-unit to reacquire video stream.

[0139] In this embodiment, the indicator calculation and key point constraints include the following two points: Calculation of the eye aspect ratio (EAR): For a single eye, the horizontal distance and two sets of vertical distances are calculated based on eye key points, yielding:

[0140] Where H represents the horizontal distance between the two key points at the ends of the eye, and V1 and V2 represent the vertical distance between the two sets of key points on the upper and lower eyelids.

[0141] In real-world driving scenarios, drivers may tilt their heads or turn their faces to the side, causing one eye to be obstructed, poorly lit, or have incomplete key point localization. If the two eyes are still fused using average or minimum values, abnormal EAR jitter can easily occur, leading to false alarms or missed alarms.

[0142] To improve stability in side-view and occluded scenarios, this invention employs a monocular dynamic selection strategy: calculating the horizontal distance H between the left and right eyes respectively. L With H R The eye with the larger horizontal distance is used as the effective eye for EAR calculation in the current frame, that is: If H L ≥H R If so, then the left eye is selected to calculate the EAR; If H L <H R If so, then the right eye is selected to calculate the EAR.

[0143] Since a larger horizontal distance usually indicates that the eye is closer to the front and less obstructed, this strategy can effectively alleviate the instability of key points caused by side profiles or partial occlusion, thereby improving the reliability of the closed-eye determination.

[0144] Calculation of the mouth's aspect ratio (MAR): Based on the calculation of horizontal distances and three sets of vertical distances from key points at the mouth, we obtain:

[0145] Where K represents the horizontal distance between the corners of the mouth, and T1, T2, and T3 represent the vertical distance between the key points of the upper and lower lips at different locations.

[0146] In this embodiment, the fatigue driving determination mechanism is as follows: To avoid misjudgment caused by blinking, short-term mouth opening, or instantaneous errors at key points, this invention adopts a continuous time window confirmation mechanism: the closed eye / yawning event is only output when EAR or MAR meets the threshold condition and continues for a preset duration.

[0147] In the specific implementation, "lasting 2 seconds" is converted to the corresponding number of frames N = 2 × fps in video frame rate (fps). When the number of consecutive frames that meet the condition reaches N, the behavior is confirmed to be valid, and fatigue judgment and reporting are triggered.

[0148] In this embodiment, the eye aspect ratio (EAR) threshold is set to 0.15. The driver is determined to be in a closed-eye state when the following conditions are met: An effective eye EAR value was obtained by employing a monocular selection strategy. If EAR < 0.15, then the count will be accumulated even if the eyes are closed. If the above conditions are met continuously for 2 seconds, the act of closing the eyes is confirmed and the closing eyes event is output.

[0149] When the EAR recovers to above the threshold, the closed-eye count is reset to zero or decayed according to the set method to ensure timely recovery of the state.

[0150] In this embodiment, the mouth aspect ratio (MAR) threshold is set to 0.5. The driver is determined to be yawning when the following conditions are met: If MAR > 0.5, then the yawn count will be incremented. If the above conditions are met continuously for 2 seconds, the yawning behavior is confirmed and the yawning event is output.

[0151] When the MAR falls below the threshold, the yawn count is reset to zero or decays according to the set method.

[0152] This invention also protects a storage medium that stores a software program capable of implementing the above-described method when it is run.

[0153] This invention also protects an electronic product that can implement the above-described method when in operation.

[0154] Compared with existing technologies, this invention significantly improves the real-time performance, stability, and engineering deployability of the fatigue driving detection system in computing-constrained industrial vehicle scenarios by optimizing the structure of the facial key point detection model, accelerating quantization, and systematically designing the fatigue judgment logic, while ensuring detection accuracy. Specific beneficial effects are as follows: (1) While ensuring the accuracy of key point detection, the computational complexity and resource consumption of the model are significantly reduced.

[0155] This invention reduces the number of repeatedly stacked InvertedResidual residual modules and simplifies the keypoint output dimension by structurally pruning and simplifying the PFLD-based facial landmark detection model, without changing the overall feature extraction process and scale variation rules of the network. ● Effectively reduces the model parameter size and floating-point operation volume (FLOPs); ● Significantly reduced model inference latency and memory usage; ● Enables the model to run stably on in-vehicle terminals, embedded devices, and industrial vehicle platforms with limited computing power.

[0156] Compared with existing visual fatigue detection methods that rely on highly complex network structures, this invention can achieve a higher inference frame rate under the same hardware conditions, providing reliable computing power for real-time detection of fatigued driving.

[0157] (2) Improve the stability and engineering applicability of the model by optimizing the number of key points.

[0158] This invention reduces the number of facial key points output from the traditional 64 points to 42 points, retaining only the core key points that are crucial for representing the eyes, mouth, and facial geometry. ●Reduces the computational burden and training difficulty of the output layer; ●Reduces interference from redundant key points on fatigue judgment logic; ● Improved the model's predictive stability in real-world driving scenarios.

[0159] This design allows the model to focus more on the facial area necessary for fatigue detection, effectively avoiding fatigue misjudgment caused by the amplification of errors from irrelevant key points, and enhancing the practical value of the system in actual engineering deployment.

[0160] (3) Remove the auxiliary pose network to simplify the inference process and improve real-time response capability.

[0161] Compared to existing keypoint detection models that incorporate attitude-assisted networks for constraints, this invention removes the attitude estimation auxiliary branch while maintaining keypoint regression accuracy, retaining only the keypoint regression backbone network. ●Reduces additional computational paths and model parameters; ●Reduces synchronization and latency issues caused by multi-branch reasoning; ● Makes the model inference process simpler and more efficient.

[0162] This optimization significantly improves the model's response speed in real-time video stream processing scenarios, making it particularly suitable for industrial vehicle applications such as forklifts that have high requirements for real-time safety warnings.

[0163] (4) By using quantitative perception training, the problem of key point offset caused by fixed-point quantization can be effectively solved.

[0164] To address the issue of accuracy degradation in traditional PTQ quantization for keypoint regression tasks, this invention employs Quantization-Aware Training (QAT), enabling the model to perceive and adapt to fixed-point quantization errors during the training phase. ● Significantly reduced keypoint offset in INT8 quantization models in side-view, large-pose, and occluded scenarios; ● This ensures that the quantized model has essentially the same detection accuracy as the FP32 floating-point model; ● Achieve approximately 3 times faster model inference speed while significantly reducing storage and computing power consumption.

[0165] This technology directly improves the ability of fatigue detection systems to operate stably on embedded devices for extended periods, and solves the problem of high-precision quantification and deployment of existing visual fatigue detection models.

[0166] (5) Based on the closed-eye determination logic of single-eye dynamic selection, the detection robustness under complex postures is improved.

[0167] To address the issue that existing fatigue detection methods are prone to EAR jitter when the face is turned to the side, the head is tilted, or one eye is obstructed, this invention proposes a dynamic single-eye selection strategy based on the horizontal distance of the eye: ● Automatically selects effective eyes with more complete key points and less occlusion for EAR calculation; ● Effectively avoids interference from abnormal values ​​caused by binocular fusion; ● Significantly improves the stability and accuracy of eye-closed fatigue detection in complex posture scenarios.

[0168] This logic enables the system to adapt to the driver's natural head posture changes during actual industrial vehicle operations, reducing the risk of false alarms and missed alarms.

[0169] (6) Introduce a time accumulation mechanism to accurately reflect the continuous characteristics of fatigue state.

[0170] This invention employs a fatigue determination mechanism based on continuous time windows to confirm the duration of eye-closing and yawning behaviors: ● Effectively distinguishes between normal blinking, brief mouth opening, and genuine fatigue behavior; ● Avoid false alarms caused by momentary movements or vibrations at key points; ● It is more in line with the objective physiological characteristics of fatigue gradually accumulating over time.

[0171] Compared with existing judgment methods that rely on single-action triggers, this invention significantly improves the reliability and stability of fatigue detection results.

[0172] (7) Improve the overall deployment and safety value of the system in industrial vehicle scenarios.

[0173] Through the synergistic design of model optimization, quantization acceleration, and scientific fatigue judgment logic, this invention achieves the following comprehensive technical effects: ● Achieve high-precision, low-latency fatigue detection on devices with limited computing power; ● Maintain stable operation in complex lighting, posture changes, and occlusion scenarios; ● Meets the application needs of industrial vehicles such as forklifts for continuous safety monitoring and real-time alarms.

[0174] Therefore, this invention differs from existing fatigue driving detection technologies in that it not only identifies fatigue behavior but also provides a highly reliable, low-cost, and scalable fatigue driving detection solution for industrial vehicle applications, demonstrating significant engineering application value and promising prospects for promotion.

Claims

1. A lightweight method for generating facial landmark detection models, characterized by: The method includes the following steps: Step S1, Image Acquisition and Preprocessing: Acquire training sample images and perform preprocessing. Training samples include public datasets and / or data collected and manually labeled for actual driving scenarios. Step S2, Model Optimization: The face landmark detection model based on PFLD was optimized in terms of network structure, output format, and training and inference process; Step S3, Model Quantization Method: The QAT method is used to implement the INT8 quantization of the face key point detection model to generate a low-ratio key point inference model.

2. The lightweight facial landmark detection model generation method according to claim 1, characterized in that: In step S1, the image preprocessing of the training samples includes the following sub-steps: Step S1-1, Image Normalization and Keypoint Mapping: Let the input image size be... The key points of the face are: Where W represents the width of the input image and H represents the height of the input image. P i Let N represent the facial keypoints of the i-th person, and N represent the number of keypoints. x i The x-coordinate of the key point y i The ordinate of the key point; Perform image size normalization to resize the image to a fixed size. W 0 ×H 0 , W 0 Indicates the width of the input image. H 0 This represents the height of the input image size and performs a linear mapping of keypoints. , among which, image W 0 and H 0 112 pixels each The normalized x-axis, The ordinate is the normalized ordinate; Step S1-2, Constraint-based stochastic affine transformation: Apply a random affine transformation to the image; the affine transformation matrix is ​​represented as follows: Where M is the affine transformation formula, s is the random scaling factor, and θ is the random rotation angle. t x ，t y This represents the translation amount along the X and Y axes; Coordinates of key points after affine transformation for: Key constraints: The corresponding affine transformation result is used only when all keypoints meet the above conditions; otherwise, the transformation parameters are regenerated. Represents the x-coordinate after affine transformation. Represents the ordinate after affine transformation. This indicates that key facial features are processed for each individual. Steps S1-3, Illumination and Color Perturbation Model: If the brightness, hue, and saturation of an image are randomly perturbed, and the original pixel value is I, and the perturbed pixel value is I′, then: Where α is the brightness scaling factor and β is the brightness offset, both of which are randomly selected within the range of [0.5,1]. Steps S1-4: Enhance localized areas by avoiding key points: Random occlusion blocks are generated in non-critical point regions. Let the occlusion block region be a rectangle R, with its boundaries as follows: Then the occluded area must meet the following requirements: And the minimum Euclidean distance from the keypoint to the occluded block region satisfies: Where, is d min The preset minimum safe distance, This represents the pixel value of the occluded region of the i-th face keypoint after the affine transformation.

3. The lightweight facial landmark detection model generation method according to claim 1, characterized in that: In step S2, model optimization includes the following sub-steps: Step S2-1, Lightweight backbone network structure pruning and optimization: The face landmark detection model is based on a lightweight convolutional neural network. The backbone network consists of multiple layers of convolutional modules based on the InvertedResidual structure. The InvertedResidual structure is optimized by a combination of channel expansion, depthwise separable convolution and channel compression, reducing the number of residual modules in the first feature stage to 3 and the number of residual modules in the second feature stage to 3. Step S2-2, Simplifying the design of the number of key point outputs: The number of facial key points output is reduced to 42, and the key points cover areas that are representative of the facial geometry, such as the eyes, eyebrows, nose, and mouth. Steps S2-3: Removal of auxiliary networks and simplification of inference process: During model optimization, only the main regression branch of key points is retained.

4. The lightweight facial landmark detection model generation method according to claim 1, characterized in that: In step S3, model quantization includes the following sub-steps: Step S3-1: Model operator fusion and graph structure preparation: The convolutional layers, normalization layers, and activation function layers in the model are fused together, and a computational graph for quantization training is constructed. Step S3-2: Set the quantization configuration and insert pseudo-quantization nodes: Select the quantization backend based on the deployment platform, and configure the activation quantization and weighted quantization strategies: Activation employs a minimum-maximum statistical method based on a moving average and maps it to an 8-bit unsigned integer range; The weights are mapped to an 8-bit signed integer range using channel-based quantization. Under this quantization configuration, pseudo-quantized nodes are inserted into the network computation graph, enabling the forward propagation process during the training phase to simulate INT8 inference error. Step S3-3: QAT training and learning rate scheduling optimization: In the early stages of training, the learning rate is gradually increased to improve convergence stability. In the later stages of training, the learning rate is smoothly decreased according to the cosine law to improve the final accuracy. Step S3-4: Freeze the quantitative statistical parameters and perform fixed-point model transformation: After QAT training is completed, the update of pseudo-quantization statistical parameters is stopped, and the training model containing pseudo-quantization nodes is converted into a true INT8 fixed-point inference model to obtain a deployable quantization model. Step S3-5: Deployment and Export: The converted INT8 model is then exported to a model format suitable for edge deployment.

5. A lightweight facial landmark detection model generation system, characterized by: The system includes the following functional units: Image acquisition and preprocessing unit: used to acquire training sample images and perform preprocessing. Training samples include public datasets and / or data collected and manually labeled for actual driving scenarios. Model optimization unit: This unit optimizes the PFLD-based face landmark detection model in terms of network structure, output format, and training and inference process. Model Quantization Method Unit: A model quantization method for deploying a face landmark detection model using QAT method to generate a low-ratio landmark inference model.

6. The lightweight face landmark detection model generation unit according to claim 5, characterized in that: The image acquisition and preprocessing unit includes the following sub-units for image preprocessing of training samples: Image normalization and keypoint mapping subunit: Input image size is The key points of the face are: Where W represents the width of the input image and H represents the height of the input image. P i Let N represent the facial keypoints of the i-th person, and N represent the number of keypoints. x i The x-coordinate of the key point y i The ordinate of the key point; Perform image size normalization to resize the image to a fixed size. W 0 ×H 0 , W 0 Indicates the width of the input image. H 0 This represents the height of the input image size and performs a linear mapping of keypoints. , among which, image W 0 and H 0 112 pixels each The normalized x-axis, The ordinate is the normalized ordinate; Constraint-based stochastic affine transformation sub-unit: Apply a random affine transformation to the image; the affine transformation matrix is ​​represented as follows: Where M is the affine transformation formula, s is the random scaling factor, and θ is the random rotation angle. t x ，t y This represents the translation amount along the X and Y axes; Coordinates of key points after affine transformation for: Key constraints: The corresponding affine transformation result is used only when all keypoints meet the above conditions; otherwise, the transformation parameters are regenerated. Represents the x-coordinate after affine transformation. Represents the ordinate after affine transformation. This indicates that key facial features are processed for each individual. Lighting and color perturbation model sub-unit: If the brightness, hue, and saturation of an image are randomly perturbed, and the original pixel value is I, and the perturbed pixel value is I′, then: Where α is the brightness scaling factor and β is the brightness offset, both of which are randomly selected within the range of [0.5,1]. Avoiding local occlusion at key points to enhance sub-units: Random occlusion blocks are generated in non-critical point regions. Let the occlusion block region be a rectangle R, with its boundaries as follows: Then the occluded area must meet the following requirements: And the minimum Euclidean distance from the keypoint to the occluded block region satisfies: Where, is d min The preset minimum safe distance, This represents the pixel value of the occluded region at the coordinate position of the i-th facial keypoint after the affine transformation. And / or, the model optimization unit includes the following sub-units: Lightweight backbone network structure pruning and optimization sub-units: The face landmark detection model is based on a lightweight convolutional neural network. The backbone network consists of multiple layers of convolutional modules based on the InvertedResidual structure. The InvertedResidual structure is optimized by a combination of channel expansion, depthwise separable convolution and channel compression, reducing the number of residual modules in the first feature stage to 3 and the number of residual modules in the second feature stage to 3. Simplified design sub-unit for key output quantity: The number of facial key points output is reduced to 42, and the key points cover areas that are representative of the facial geometry, such as the eyes, eyebrows, nose, and mouth. Auxiliary network removal and inference process simplification sub-unit: During model optimization, only the main regression branch of key points is retained; And / or, the model quantization method unit includes the following sub-steps: Model operator fusion and graph structure preparation sub-unit: The convolutional layers, normalization layers, and activation function layers in the model are fused together, and a computational graph for quantization training is constructed. Configure quantization and insert pseudo-quantization node sub-cells: Select the quantization backend based on the deployment platform, and configure the activation quantization and weighted quantization strategies: Activation employs a minimum-maximum statistical method based on a moving average and maps it to an 8-bit unsigned integer range; The weights are mapped to an 8-bit signed integer range using channel-based quantization. Under this quantization configuration, pseudo-quantized nodes are inserted into the network computation graph, enabling the forward propagation process during the training phase to simulate INT8 inference error. QAT Training and Learning Rate Scheduling Optimization Subunit: In the early stages of training, the learning rate is gradually increased to improve convergence stability. In the later stages of training, the learning rate is smoothly decreased according to the cosine law to improve the final accuracy. Freeze the quantitative statistical parameters and perform fixed-point model transformation sub-unit: After QAT training is completed, the update of pseudo-quantization statistical parameters is stopped, and the training model containing pseudo-quantization nodes is converted into a true INT8 fixed-point inference model to obtain a deployable quantization model. Deploy export subunit: The converted INT8 model is then exported to a model format suitable for edge deployment.

7. A lightweight method for detecting driver fatigue, characterized in that: The detection method includes the following steps: Step S4-1: Begin; Step S4-2: Acquire video stream frames: Acquire video streams using a camera or other video capture device; Step S4-3, Face Detection and Key Point Extraction: The face key point detection model generated by the lightweight face key point detection model generation method as described in any one of claims 1 to 4 is used to detect faces in the video stream and extract key point information; Step S4-4: Calculate the EAR of the left and right eyes and the MAR of the mouth: Calculate the EAR of the left and right eyes and the MAR of the mouth based on the key points; Step S4-5, Monocular Dynamic Selection: Calculate the horizontal distance for the left and right eyes respectively. and The eye with the larger horizontal distance is used as the effective eye for EAR calculation in the current frame; Step S4-6, Threshold and continuous time window judgment: When EAR or MAR meets the threshold condition and continues to reach the preset duration, output the closing eyes / yawning event; Step S4-7: Has the fatigue condition been met?: Based on the events of closing eyes / yawning, fatigue is judged. If the fatigue condition is met, proceed to step S4-8. If the fatigue condition is not met, directly jump back to step S4-2 and reacquire the video stream. Step S4-8: Save the alarm image and push a warning, then jump back to step S4-2 to reacquire the video stream.

8. A lightweight fatigue driving detection system, characterized in that: The system includes the following functional units: Start Unit: Used to start the system and begin operation; Video Stream Frame Acquisition Unit: Used to acquire video streams via a camera or other video capture devices; Face detection and key point extraction unit: used to detect faces in the video stream and extract key point information using the face key point detection model generated by the lightweight face key point detection model generation system as described in claim 5; Calculate the left and right eye EAR and mouth MAR units: used to calculate the left and right eye EAR and mouth MAR information based on key points; Monocular dynamic selection unit: used to calculate the horizontal distances of the left and right eyes separately. and The eye with the larger horizontal distance is used as the effective eye for EAR calculation in the current frame; Threshold and continuous time window judgment unit: used to output the closing eyes / yawning event when EAR or MAR meets the threshold condition and continues for a preset duration; Fatigue condition met unit: Used to judge fatigue based on eye closing / yawning events. If the fatigue condition is met, it will jump to save the alarm image and push the warning unit. If the fatigue condition is not met, it will jump back to the video stream frame acquisition unit to reacquire the video stream. Save alarm image and push warning unit: Used to save alarm image and push warning, then jump back to the video stream frame acquisition unit to reacquire video stream.

9. A storage medium, characterized in that: The storage medium contains a software program that, when running, can implement the lightweight face landmark detection model generation method as described in any one of claims 1 to 4.

10. An electronic product characterized by: The electronic product described herein can implement the lightweight facial landmark detection model generation method as described in any one of claims 1 to 4 when it is in operation.

Images