Driver fatigue monitoring method based on multi-modal fusion

Abstract

The invention discloses a driver fatigue monitoring method based on multi-modal fusion, and belongs to the field of computer vision and artificial intelligence. The method comprises the following steps: constructing a visual perception assembly line, and extracting visual physiological features of a driver from a video frame; constructing an auditory perception assembly line, and extracting a fatigue-related acoustic event from the audio stream; a multi-modal fusion decision engine is constructed, and a final fatigue driving early warning decision is generated by adopting a double-track parallel mechanism according to the visual physiological features and the acoustic events; and the deployment performance is optimized by adopting a model lightweight technology. According to the invention, visual and auditory information is fused, and dual-channel complementation is utilized, so that the detection robustness in a complex environment with weak light, high noise and face shielding is improved; and meanwhile, a lightweight model is selected and combined with INT8 integer quantization optimization, so that the problem that a high-precision model is difficult to deploy on a low-computing-power vehicle-mounted platform is effectively solved, and the balance of high precision and high efficiency is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and artificial intelligence, in particular to a driver fatigue monitoring method based on multi-modal fusion. BACKGROUND

[0002] With the continuous growth of global car ownership, fatigue driving has become one of the main human factors leading to serious traffic accidents, posing a great threat to road traffic safety. To address this challenge, the importance of driver monitoring system (DMS) that can monitor the driver's state in real time is increasingly prominent, and it is becoming one of the key technologies to improve the active safety performance of cars.

[0003] Currently, the mainstream technology for vehicle fatigue driving detection is a non-contact method based on computer vision. This method mainly realizes through analyzing the driver's face image captured by the in-vehicle camera, and its core technical path includes: (1) eye state analysis: judge the degree of drowsiness by monitoring the closing frequency and duration of the eyes (such as PERCLOS index); (2) mouth shape analysis: identify yawning and other typical fatigue-related behaviors; (3) head posture estimation: track the direction of the driver's head and signs of distraction such as nodding.

[0004] However, the above mainstream visual detection method faces many limitations in practical application. First, in weak light environments such as night and tunnel, its perception robustness will decrease significantly, and it usually needs to rely on high-cost infrared hardware. Second, as a single visual modality, when the driver's face is physically obscured by a mask or other objects, key feature extraction will fail. In addition, high-precision detection algorithms require high computing power, and deployment on cost- and power-constrained vehicle embedded platforms faces performance bottlenecks. Therefore, how to realize a fatigue detection scheme with high robustness, low cost and easy deployment in complex scenarios is still a challenge to be solved. SUMMARY

[0005] In view of the above problems, the present application provides a driver fatigue monitoring method based on multi-modal fusion, aiming to solve the problems of insufficient robustness in complex environments, perception blind area of single modality, and difficulty in efficient deployment on low-power vehicle platforms in the prior art.

[0006] The present application provides a driver fatigue monitoring method based on multi-modal fusion, comprising the following steps:

[0007] Step S1, constructing a visual perception pipeline for real-time extraction of visual physiological features of the driver from video frames collected by the vehicle camera;

[0008] Step S2, constructing an auditory perception pipeline for real-time extraction of acoustic events related to driver fatigue from audio streams collected by the vehicle microphone;

[0009] Step S3, constructing a multi-modal fusion decision engine, generating a final fatigue driving warning decision by using a double-track parallel mechanism according to the visual physiological features and the acoustic events;

[0010] Step S4, using a model quantization technology to optimize the deployment performance of the models used in the visual and auditory perception pipelines, to ensure efficient real-time operation of the method on a low-computing-power vehicle-mounted embedded platform.

[0011] Preferably, the step S1 specifically comprises:

[0012] Step S1-1, performing illumination self-adaptive preprocessing on the input video frame, judging whether it is a low-light environment by calculating the average brightness of the image, and if so, activating a low-light image enhancement model for processing;

[0013] Step S1-2, periodically detecting the face region by using the Google BlazeFace model, recording and cropping the face position coordinates;

[0014] Step S1-3, using the PFLD model to locate the facial key points frame by frame in the detected face region;

[0015] Step S1-4, based on the coordinates of the facial key points, generating visual physiological features including eye height-width ratio and mouth height-width ratio through geometric calculation quantization;

[0016] Preferably, step S1-1 specifically comprises:

[0017] The low-light image enhancement model predicts a pixel-level brightness enhancement high-order curve through a deep neural network, and its enhancement process can be represented as:

[0018] I enhanced (x)=I input (x)+LF(I input (x))·I input (x)

[0019] Where I input (x) is the brightness value of the input image at pixel x, LE() is the enhancement curve prediction network, I enhanced (x) is the enhanced image brightness value.

[0020] Preferably, step S1-3 specifically comprises:

[0021] The eye height-width ratio quantifies the degree of eye opening (EAR) by calculating the ratio of the sum of distances between key point pairs in the vertical direction of the eye to the distance between key point pairs in the horizontal direction, and the calculation formula is:

[0022]

[0023] where p 36 to p 41 are the coordinates of the left eye landmarks defined in the standard 68-point model. i -p j || denotes the Euclidean distance between the landmarks p i and p j .

[0024] The mouth height-width ratio quantifies the degree of mouth opening by calculating the average distance between multiple pairs of landmarks in the vertical direction of the mouth and the distance between the pairs of landmarks in the horizontal direction, and the calculation formula is:

[0025]

[0026] where p 60 to p 67 are the coordinates of the inner mouth contour landmarks defined in the standard 68-point model.

[0027] Preferably, step S2 specifically comprises:

[0028] Step S2-1, performing standardized preprocessing on the collected audio stream, dividing the continuous audio stream into audio frames with fixed time length, and resampling the audio frames to unify their sampling rate;

[0029] Step S2-2, performing first-level signal-level noise suppression on the preprocessed audio frames, using a lightweight RNNNoise model to process the audio to suppress non-stationary noise and improve the signal-to-noise ratio of the target sound;

[0030] Step S2-3, using a large-scale pre-trained acoustic model Google YAMNet to perform acoustic event detection on the noise-reduced audio to identify fatigue-related acoustic events including yawning, snoring, and sighing;

[0031] Step S2-4, performing second-level semantic-level noise suppression, when detecting high-confidence music or speech interference events within the same time window, automatically ignoring or reducing the confidence of fatigue-related acoustic events;

[0032] Preferably, step S2-1 specifically comprises:

[0033] In the standardized preprocessing, the signal is first passed through a low-pass anti-aliasing filter, and then the sampling rate is reduced to the target value through decimation operation. This process can be represented as:

[0034]

[0035] where x(n) is the original sampling rate R nativeaudio signal, h(n) is an anti-aliasing filter, M = R native / R target is an extraction factor, y(m) is a target sampling rate R target new signal.

[0036] Preferably, step S2-2 specifically comprises:

[0037] The RNNNoise model selectively attenuates the noise component while preserving the speech component by calculating a gain mask between 0 and 1 for each frequency band in real time, and its noise reduction process can be represented as:

[0038] S clean [k] = G[k] · S nousy [k]

[0039] where S noisy [k] is the energy of the noisy audio in the kth frequency band, G[k] is the calculated gain mask, and S clean [k] is the energy of the denoised audio.

[0040] Preferably, step S2-4 specifically comprises:

[0041] The confidence adjustment rule of the second-level semantic-level noise suppression logic is implemented through conditional judgment: when the highest confidence of the detected music or speech event exceeds the preset interference threshold, a decay weight less than 1 is applied to the original confidence of the fatigue event, otherwise the original confidence remains unchanged, and its rule is:

[0042]

[0043] where c fatigue is the original fatigue event confidence, c′ fatigue is the adjusted confidence, c music and c speec are the confidences of the music and speech events respectively, and T noise is the preset interference confidence threshold.

[0044] Preferably, step S3 specifically comprises:

[0045] Step S3-1, establishing a unified fatigue event scoring rule to assign corresponding score values to different types of detected fatigue events;

[0046] Step S3-2, using direct trigger rail processing for extreme dangerous behavior, directly triggering the highest level of warning when the preset conditions are met;

[0047] Step S3-3, integral accumulation tracks are used to integrate and accumulate the progressive fatigue behavior in a preset sliding time window, and different levels of fatigue warnings are triggered according to the accumulated fatigue score;

[0048] Preferably, step S3-1 specifically comprises:

[0049] The segmented scoring function divides the eye closing time ratio into multiple preset intervals, and assigns a non-linear, stepwise increasing integral value to the ratio value falling in different intervals, so as to finely quantify the fatigue degree.

[0050] Preferably, step S3-3 specifically comprises:

[0051] There is a decay parameter in the accumulated fatigue score calculation formula of the integral accumulation track to simulate the natural relief process of fatigue over time, and the calculation formula is:

[0052]

[0053] Wherein, S fatigue (t-1) is the fatigue score of the last time, e -λΔt is the time decay factor, lambda is the decay coefficient, Delta t is the time interval, N is the total number of new events at the current time, is the integral value corresponding to the i-th fatigue event.

[0054] Preferably, step S4 specifically comprises:

[0055] The deployment optimization adopts INT8 integer quantization technology, and the quantization process is realized by affine transformation, and the mapping formula is:

[0056]

[0057] Wherein, r float is the original 32-bit floating point value, r quantized is the quantized 8-bit integer value, min and max are the range of floating point value.

[0058] Compared with the prior art, the present application has at least the following beneficial effects:

[0059] (1) The present application fuses the information of two modalities of vision and hearing, uses the complementary of double channel information, effectively solves the problem of insufficient robustness and perception blind area of single perception modality in complex environment. The hearing channel can provide effective yawning and snoring information when the driver's face is blocked by the mask object, and the vision channel can cover the silent fatigue behavior that the hearing cannot perceive. This synergistic effect builds a more comprehensive detection system, significantly improves the overall detection performance and reliability of the system under weak light, high noise and face blocking conditions.

[0060] (2) The application selects a series of verified lightweight artificial intelligence models, and combines a mandatory INT8 integer quantization deployment optimization technology, so that the calculation complexity and resource consumption of the algorithm are significantly reduced under the premise of ensuring detection accuracy. This enables the method to meet the stringent requirements of the vehicle embedded platform for low power consumption and real-time performance, solves the industry pain point that existing high-precision models are difficult to run efficiently on resource-constrained hardware, and has good platform applicability and deployment feasibility. BRIEF DESCRIPTION OF DRAWINGS

[0061] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and serve to explain the principles of the application.

[0062] Figure 1 A flowchart of a driver fatigue monitoring method based on multi-modal fusion provided by the application.

[0063] Figure 2 A system architecture and algorithm step diagram of a driver fatigue monitoring method based on multi-modal fusion provided by the application.

[0064] Figure 3 A fatigue degree integral trend chart during operation provided by the application. DETAILED DESCRIPTION

[0065] In order to more clearly understand the above-mentioned purposes, features and advantages of the application, the application will be further described in detail below in combination with the drawings and specific embodiments. It should be noted that the embodiments of the application and the features in the embodiments can be combined with each other without conflict. In addition, the application can also be implemented in other ways different from those described herein, therefore, the protection scope of the application is not limited by the specific embodiments disclosed below.

[0066] In order to illustrate the effectiveness of the method proposed by the application, the above technical solutions of the application will be described in detail below through a specific embodiment, as shown in Figure 1 A driver fatigue monitoring method based on multi-modal fusion is disclosed, and the specific implementation steps are as follows:

[0067] Step S1, constructing and running a visual perception pipeline.

[0068] In this step, the system receives video frames from a standard RGB camera, and extracts structured visual physiological features after a series of processing, and the processing process is specifically as follows:

[0069] 1) Perform light adaptive preprocessing. For each input image I, the system first converts it to a grayscale image I gray , and calculates its average brightness value L avg :

[0070]

[0071] Where W and H are the width and height of the image. Additionally, a low-light threshold T is set. light If L avg <T light This activates the Zero-DCE low-light image enhancement model. The enhancement process can be represented as follows:

[0072] I enhanced (x)=I input (x)+LE(I input (x))·I input (x)

[0073] Among them, I input (x) represents the brightness value of the input image at pixel x, and LE() is the enhancement curve prediction network.

[0074] In this embodiment, T light Set to 80, and set the enhancement curve iteration number n in the Zero-DCE model to 8.

[0075] 2) The lightweight Google BlazeFace model is used to detect faces in the input video frames to locate the driver's facial region. To balance system overhead, this system adopts a periodic detection strategy. The face detection model's runtime f BlazeFace It can be represented as:

[0076] B set =f BlazeFace (I frame )

[0077] Among them, I frame This is the current video frame to be detected, B set It is the set of all detected face candidate boxes. Each candidate box B i The output can be represented as:

[0078] B i =[x min ,y min ,x max ,y max C score ]

[0079] Among them, (x min ,y min ) represents the coordinates of the top-left corner of the face region, (x max ,y max ) represents the coordinates of the lower right corner of the face region, C score The system will assign confidence that the bounding box is a real human face. set Filter out confidence level Cscore the highest candidate box B best , and its coordinates [x min , y min , x max , y max ] are taken as the region of interest (ROI) for the subsequent step.

[0080] In this embodiment, the face detection period T detect is set to 5 frames, and the face detection confidence threshold T conf is set to 0.85. When the C best of B score is higher than 0.85, the detection result is valid and used to update the ROI.

[0081] 3) Within the detected face region, a high-precision PFLD model is used to locate facial key points frame by frame. The operation of the PFLD model can be formally represented as:

[0082] L pts = f PFLD (I ROI ) = {(x1,y1),(x2,y2),...,(x 68 ,y 68 )}

[0083] where I ROI is the face region image cropped according to the ROI coordinates, L pts is the coordinate set of the facial key points (containing 68 accurate coordinate points), and (x i , y i ) is the local coordinate relative to the I ROI image.

[0084] In this embodiment, the face region image I ROI input to the PFLD model is uniformly adjusted to a specification of 112x112 pixels to match the optimal input size of the model.

[0085] 4) Physiological feature quantification. Based on the 68 accurate facial key point coordinates output by the PFLD model, physiological indicators are quantified in real time through geometric calculations.

[0086] The eye height-width ratio (EAR) is used to measure the degree of eye opening, which is calculated by the following formula:

[0087]

[0088] where p 36 to p 41 are the left eye key point coordinates defined in the standard 68-point model.

[0089] The Mouth Area Ratio (MAR) is used to quantify the degree of mouth opening to detect yawning, which is calculated by the following formula:

[0090]

[0091] where p 60 is the coordinate of the key point of the inner contour of the mouth defined in the standard 68-point model. 67

[0092] In this embodiment, when the MAR value exceeds the threshold T MAR , i.e. 0.6, and the duration exceeds T yawn_duration , i.e. 1.5 seconds, it is determined as a visual yawning event.

[0093] Step S2, construct and run the auditory perception pipeline in parallel.

[0094] In this step, the audio processing pipeline works in parallel with the visual pipeline to extract fatigue-related acoustic events from the audio stream received from the standard MEMS microphone, and the processing process is as follows:

[0095] 1) The continuous audio stream is segmented into audio frames with fixed time length. After applying an anti-aliasing filter to the original signal audio stream, it is decimated to convert it to a single-channel format A resampled , and then segmented into a series of audio frames A frame with fixed time length. The number of sample points contained in each frame is determined by the sampling rate and the frame time length:

[0096] N samples = f s × T frame

[0097] In this embodiment, to meet the input requirements of the YAMNet model, the sampling rate f s of the audio signal is set to 16 kHz, and converted to a single-channel format. The fixed time length T frame of each frame is set to 0.975 seconds.

[0098] 2) Use the RNNNoise model to suppress non-stationary noise on the audio frames A frame output in the previous step. This model first decomposes the audio signal into multiple Bark scale critical frequency bands, and then uses a recurrent neural network to estimate a gain mask, which is used to separate speech and noise. Its noise reduction process can be represented as:

[0099] S clean [k] = G[k] · S noisy [k]

[0100] where S noisy ​[k] is the energy of the noisy audio in the kth frequency band, G[k] is the computed gain mask, S clean [k] is the energy of the denoised audio.

[0101] In this embodiment, the RNNNoise model decomposes the audio stream into 22 Bark-scaled critical frequency bands.

[0102] 3) Perform acoustic event classification on the denoised audio frame A denoised using the YAMNet model. After processing the audio frame, the YAMNet model outputs a score vector c covering 521 acoustic classes defined internally:

[0103] c = [c1, c2,..., c 521 ] = f YAMNet (A denoised )

[0104] where c is a 521-dimensional vector, c i is the confidence score of the model predicting the occurrence of the i-th acoustic event.

[0105] In this embodiment, the system focuses on monitoring yawning, snoring, and sighing, three fatigue-related classes, and extracts the corresponding confidence scores c yawn , c snore , and c sig in real time for subsequent analysis and determination.

[0106] 4) Introduce a rule-based post-processing logic to check whether there are high-confidence common strong interferers in the vehicle, namely music and speech, in the result vector. Set the confidence threshold T interference . If the confidence of any interferer exceeds this threshold, the filtering logic will automatically reduce the confidence of the fatigue-related events detected in the same time window, and its update rule can be expressed as:

[0107]

[0108] where c fatigue is the original fatigue event confidence, and c′ fatigue is the adjusted confidence.

[0109] In this embodiment, T interference is set to 0.8.

[0110] Step S3, build and run a multi-modal fusion decision engine.

[0111] In this step, the decision engine intelligently fuses the structured information from the visual and auditory perception pipelines and adopts a dual-track parallel mechanism to make the final fatigue determination and warning.

[0112] The first track is the "direct trigger track", with the highest priority. This mechanism is used to detect extremely dangerous driving states, and once triggered, will bypass all the integration logic and directly issue the highest level of warning. This mechanism contains two independent trigger conditions: visual microsleep and snoring in driving.

[0113] The judgment of visual microsleep is based on the duration of the EAR value (eye height-width ratio). Its judgment logic can be expressed as:

[0114]

[0115] Where T EAR is the closed-eye threshold, t closed is the duration of the EAR value continuously below the threshold, and t micro_sleep is the safety time limit. When t closed exceeds T micro_sleep , the system determines that a microsleep event has occurred.

[0116] The judgment of snoring in driving is based on the "Snoring" category confidence output by the YAMNet model. Its judgment logic can be expressed as:

[0117]

[0118] Where C' snore is the snoring confidence after interference suppression, T snore_direct is the high confidence threshold. When C' snore exceeds T snore_direct , the system determines that an extremely dangerous fatigue state has occurred.

[0119] In this embodiment, the closed-eye threshold T EAR is set to 0.22, and the visual microsleep safety time limit T micro_sleep is set to 2.5 seconds. At the same time, the snoring confidence threshold T snore_direct is set to 0.9.

[0120] The second track is the "integration accumulation track", which is used to quantitatively evaluate and grade the progressive fatigue behavior. The processing process is as follows:

[0121] 1) Integrate the detected fatigue events of various types within a sliding time window. The system assigns a score to each event according to a unified scoring rule, and calculates the total fatigue score S fatigue :

[0122]

[0123] Where S fatigue(t-1) is the fatigue score of the last time, e -λΔt is the time decay factor, λ is the decay coefficient, Δt is the time interval, N is the total number of new events at the current time, is the integral value corresponding to the ith fatigue event.

[0124] In this embodiment, the decay coefficient λ is set to 0.01, and the unit of Δt is set to seconds. The fatigue event scoring rules used are as follows:

[0125] a) Eye closure degree (PERCLOS) is scored according to the percentage (P) of the driver's eye closure time within the past 60 seconds, and the score S PERCLOS is calculated using a piecewise function with the following rules:

[0126]

[0127] b) Visual yawn (VISUAL_YAWN): When the mouth aspect ratio (MAR) value is detected to exceed the preset threshold, and the state lasts for 1.5 seconds, 10 points are scored.

[0128] c) Audio sigh (AUDIO_SIGH): When a high-confidence sigh audio event is detected, 5 points are scored.

[0129] d) Audio yawn (AUDIO_YAW): When a high-confidence yawn audio event is detected, 10 points are scored.

[0130] e) Fused yawn (FUSED_YAWN): Within a 3-second time window, if the system simultaneously detects a visual yawn and an audio yawn event, 25 points are scored.

[0131] 2) The system triggers a three-level warning mechanism according to the accumulated fatigue score S fatigue , and simultaneously links the vehicle to provide different intensity of warning information.

[0132] In this embodiment, the warning threshold of each level is set as follows: a) when S fatigue ≥ 15, trigger level 1 (mild fatigue) warning, and play a soft prompt sound through the vehicle audio; b) when S fatigue ≥ 30, trigger level 2 (moderate fatigue) warning, the system plays a voice prompt "Please remain vigilant", and simultaneously links the seat to produce a slight vibration; c) when S fatigueWhen the value is greater than or equal to 45, a third-level (dangerous drowsiness) early warning is triggered, the system issues a continuous and loud alarm sound, cooperates with the broadcast of the voice instruction "Danger! Please stop and rest immediately!", and links the seat to produce strong and continuous vibration to maximize the awakening of the driver. If the system does not detect the effective response of the driver within the preset time, the emergency call function will be automatically triggered to contact the preset emergency contact person.

[0133] Step S4, using model lightweight technology to improve performance. To ensure that the whole scheme can be run efficiently and with low power consumption on low-performance vehicle chips, the present application has carried out special optimization in the deployment stage.

[0134] In this step, the system uses the TensorFlow Lite framework for deployment. All AI models used in the pipeline are converted using the TFLite Converter tool chain before deployment. In the conversion process, the mandatory INT8 integer quantization technology is applied. This technology maps the floating-point parameters to integer parameters through an affine transformation, and the relationship is:

[0135] r = S(q-Z)

[0136] Where r is the original floating-point value, q is the quantized 8-bit integer value, S is the scaling factor, and Z is the zero point. The quantization process can be represented as:

[0137]

[0138] Where r float is the original 32-bit floating-point value, r quantized is the quantized 8-bit integer value, min and max are the range of floating-point values.

[0139] In this embodiment, the model weight uses symmetric quantization, i.e. the zero point Z is fixed at 0; the activation value uses asymmetric quantization, allowing the zero point Z to be adjusted adaptively according to the value distribution, to more accurately represent the asymmetric activation function output, thereby minimizing the loss of precision.

[0140] In some embodiments, the above-mentioned method of the present application can be implemented in Python language, using a modular program structure, by combining different functional modules to realize real-time monitoring and early warning of the driver's fatigue state. To ensure the modularity and data interaction of the system, the construction of each functional component of the system is as follows: all program modules interact with data through standard interfaces, and each program module realizes the functions of data acquisition, feature extraction, fusion decision and early warning output. Each program module is:

[0141] 1) Visual perception module: light adaptive preprocessing on input image frames, facial key point positioning, geometric calculation to quantify EAR, MAR and PERCLOS visual physiological features.

[0142] 2) Auditory perception module: noise suppression on input audio using RNNNoise model, acoustic event detection using Google YAMNet model, semantic level noise suppression and auditory feature extraction.

[0143] 3) Decision fusion module: realize double-track parallel decision mechanism, and calculate the final fatigue score according to the unified scoring rule and time decay model.

[0144] 4) Main module: responsible for initializing and scheduling all the above modules, integrating multi-modal data flow, executing fatigue monitoring logic, and finally outputting graded warning signals, and linking vehicles for processing.

[0145] The application provides a pseudo code of a driver fatigue monitoring method based on multi-modal fusion, as shown in Table 1:

[0146] Table 1

[0147]

[0148] The application is a driver fatigue monitoring method based on multi-modal fusion, which adopts a visual and auditory parallel perception pipeline and a double-track fusion decision engine, realizes high robustness and real-time monitoring of the driver fatigue state on a resource-limited vehicle-mounted platform, and effectively solves the balance problem between the perception limitation of a single mode and the deployment efficiency of a high-precision model in a complex environment.

[0149] The specific embodiments of the application, although depicting each action or step in a specific order, should be understood as requiring such actions or steps to be performed in the specific order shown or in a sequential order, or requiring all the illustrated actions or steps to be performed to achieve the desired results. In certain circumstances, multitasking and parallel processing can be advantageous. Similarly, although the above discussion contains a number of specific implementation details, these should not be interpreted as limiting the scope of the disclosure. Certain features described in the context of separate embodiments can also be combined in a single implementation. Conversely, various features described in the context of a single implementation can also be separated and implemented in multiple implementations.

[0150] The above is only the preferred specific embodiment of the application, but the protection scope of the application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the application, which should be covered within the protection scope of the application.

Claims

1. A driver fatigue monitoring method based on multi-modal fusion, characterized in that, The method comprises the following steps: Step S1, constructing a visual perception pipeline for real-time extraction of visual physiological features of a driver from video frames collected by a vehicle-mounted camera; Step S1 specifically comprises: Step S1-1, performing illumination self-adaptive preprocessing on the input video frames, determining whether it is a low-light environment by calculating the average brightness of the image, and if so, activating a low-light image enhancement model for processing; Step S1-2, periodically detecting the face region using the Google BlazeFace model, recording and cropping the face position coordinates; Step S1-3, using the PFLD model to locate facial key points frame by frame within the detected face region; Step S1-4, based on the coordinates of the facial key points, generating visual physiological features including eye height-width ratio and mouth height-width ratio through geometric calculation; Step S2, constructing an auditory perception pipeline for real-time extraction of fatigue-related acoustic events from audio streams collected by a vehicle-mounted microphone; Step S2 specifically comprises: Step S2-1, performing standardization preprocessing on the collected audio stream, dividing the continuous audio stream into audio frames with a fixed time length, and resampling the audio frames to unify their sampling rates; Step S2-2, performing first-level signal-level noise suppression on the preprocessed audio frames, using a lightweight RNNNoise model to process the audio to suppress non-stationary noise and improve the signal-to-noise ratio of the target sound; Step S2-3, using a large-scale pre-trained acoustic model Google YAMNet to detect acoustic events from the noise-reduced audio, identifying fatigue-related acoustic events including yawning, snoring, and sighing; Step S2-4, performing second-level semantic-level noise suppression, automatically ignoring or reducing the confidence of fatigue-related acoustic events when detecting high-confidence music or speech interference events within the same time window; Step S3, constructing a multi-modal fusion decision engine to generate a final fatigue driving warning decision based on the visual physiological features and the acoustic events using a dual-track parallel mechanism; Step S3 specifically comprises: Step S3-1, establishing a unified fatigue event scoring rule to assign corresponding score values to different types of fatigue events detected; Step S3-2, using a direct trigger track to handle extreme dangerous behavior, directly triggering the highest level of warning when the preset conditions are met; Step S3-3, using an integral accumulation track to accumulate scores for gradual fatigue behavior within a preset sliding time window, and triggering fatigue warnings of different levels according to the accumulated fatigue scores; Step S4, using model quantization techniques to optimize the deployment performance of the models used in the visual and auditory perception pipelines to ensure efficient real-time operation of the method on low-power vehicle-mounted embedded platforms.

2. The method of claim 1, wherein, Step S1-1 specifically comprises: The low-light image enhancement model predicts a pixel-level brightness enhancement high-order curve through a deep neural network, and the enhancement process can be represented as: I enhanced (x) = I input (x) + LE(I input (x)) · I input (x) where I input (x) is the luminance value of the input image at pixel x, LE() is the enhancement curve prediction network, I enhanced (x) is the enhanced image luminance value.

3. The method of claim 1, wherein, Step S1-3 specifically comprises: The eye height-width ratio quantifies the eye opening degree (EAR) by calculating the ratio of the sum of distances between key point pairs in the vertical direction of the eye to the distances between key point pairs in the horizontal direction, and the calculation formula is: where p 36 to p 41 are the left eye keypoint coordinates defined in the standard 68-point model, and i -p j represents the Euclidean distance between keypoint p i and p j . The mouth height-width ratio quantifies the mouth opening degree by calculating the ratio of the average distance between multiple pairs of key points in the vertical direction of the mouth and the distance between key point pairs in the horizontal direction, and the calculation formula is: where p 60 to p 67 are the coordinates of the inner mouth contour key points defined in the standard 68-point model.

4. The method of claim 1, wherein, Step S2-1 specifically includes: In the standardization preprocessing, the signal is first passed through a low-pass anti-aliasing filter, and then the sampling rate is reduced to the target value through decimation operation. The process can be represented as: where x(n) is an audio signal with original sampling rate R native , h(n) is an anti-aliasing filter, M = R native / R target is a decimation factor, and y(m) is a new signal with target sampling rate R target .

5. The method of claim 1, wherein, Step S2-2 specifically includes: The RNNNoise model calculates a gain mask between 0 and 1 for each frequency band in real time to selectively attenuate noise components while preserving speech components, and its noise reduction process can be represented as: S clean [k] = G[k] · S noisy [k] where S noisy [k] is the energy of the noisy audio in the kth frequency band, G[k] is the computed gain mask, S clean [k] is the energy of the noise-reduced audio.

6. The method of claim 1, wherein, Step S2-4 specifically includes: The confidence adjustment rule of the second-level semantic-level noise suppression logic is realized through conditional judgment: when the highest confidence of the detected music or speech event exceeds the preset interference threshold, a decay weight less than 1 is applied to the original confidence of the fatigue event, otherwise the original confidence remains unchanged, and the rule is: where c fatigue is the original fatigue event confidence, c′ fatigue is the adjusted confidence, c music and c spe are the music and speech event confidences, respectively, and T noise is a preset interference confidence threshold.

7. The method of claim 1, wherein, Step S3-1 specifically includes: The segmented scoring function divides the eye closure time proportion into multiple preset intervals, and assigns a non-linear, stepwise increasing integral value to the proportion value falling in different intervals, to finely quantify the fatigue degree.

8. The method of claim 1, wherein, Step S3-3 specifically includes: There is a decay parameter in the integral accumulation track's accumulated fatigue score calculation formula to simulate the natural relief process of fatigue over time, and the calculation formula is: where S fatigue (t-1) is the fatigue score of the previous time, e -λΔt is the time decay factor, λ is the decay coefficient, Δt is the time interval, N is the total number of new events at the current time, S eveni is the integral value corresponding to the i-th fatigue event.

9. The method of claim 1, wherein, Step S4 specifically includes: The deployment optimization adopts INT8 integer quantization technology, and the quantization process is realized by affine transformation, and the mapping formula is: where r float is the original 32-bit floating-point value, r quantized is the quantized 8-bit integer value, and min and max are the range of floating-point values.

Images