A driver distraction and fatigue detection method and system based on an infrared camera and a storage medium

Abstract

The application provides a driver distraction and fatigue detection method based on an infrared camera, and relates to the technical field of automobile safety.Compared with the prior art, the application provides a driver distraction and fatigue detection method and system based on an infrared camera, uses multiple key points to jointly judge distraction and fatigue, has high detection precision, first excavates expression features of a person in a fatigue state, then quantifies these qualitative expression features, extracts facial feature points and feature indexes as judgment basis, combines experimental data to summarize a recognition method based on these parameters, and finally inputs acquired state data to perform recognition and judgment; whether the driver is tired and distracted is judged from the head posture of the driver, if the time frequency at which the driver does not look at the front road in a period of time is greater than a predetermined threshold, a warning signal is generated; if the driver looks straight ahead, the percentage of time that the driver closes eyes in a specific time period exceeds a predetermined threshold, a warning signal is also generated.

Description

Technical Field

[0001] This invention relates to the field of automotive safety technology, and in particular to a method, system, and storage medium for detecting driver distraction and fatigue based on an infrared camera. Background Technology

[0002] The development of automobiles has not only had a significant impact on the socio-economic development but has also brought about a major influence on people's lives. With the continuous improvement of living standards, automobiles have become the main means of transportation. As the number of cars continues to increase, traffic congestion is becoming more severe, and traffic accidents are becoming more frequent. A large proportion of these accidents are caused by fatigued driving. Therefore, monitoring the driver's state during driving is crucial to effectively avoid fatigued driving.

[0003] However, defining fatigue and distraction during driving, as well as measuring attention, remain unsolved challenges in the field of automotive safety technology. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a driver distraction fatigue detection method based on an infrared camera, which uses multiple key points to jointly determine distraction fatigue, resulting in high detection accuracy.

[0005] The technical solution adopted in this invention is as follows:

[0006] A method for detecting driver distraction and fatigue based on an infrared camera includes the following steps:

[0007] (1) Image acquisition: An infrared camera is used to acquire images of the driver's face to obtain pictures;

[0008] (2) Image data processing: bilateral filtering, grayscale conversion of color images, and transformation of images in the HSV color space are employed;

[0009] (3) Facial feature extraction: The facial features of the image after data processing in step (2) are extracted by the face detection model retinaface;

[0010] (4) Face detection: The retainface model is used to detect faces on the image from which facial features were extracted in step (3). After obtaining the face region, the face region is cropped and input into the PFLD face 98 key point detection model.

[0011] (5) Replace the model backbone: In the key point regression model part of the PFLD face 98 key point detection model, the backbone is extracted using ResNet_18, and the loss function is replaced with the smoothL1 loss function.

[0012] (6) Remove auxiliary network: Remove the pose estimation part of the PFLD face 98 key point detection model, remove the neural network modules related to the output yaw angle, rotation angle and roll angle in the auxiliary network of the PFLD neural network, and only retain the modules related to face key points;

[0013] (7) Modify the gradient descent learning rate adjustment strategy: Replace the original PFLD face 98 key point detection model with a specific metric by using the cosine annealing learning rate adjustment strategy CosineAnnealingLR;

[0014] (8) Calculate the coordinates of key points output by the model: After the optimized PFLD model is trained, the infrared image outputs 98 facial key points after testing. Among them, there are 9 key points for the nose, 18 key points for the mouth, 9 key points for the left and right eyes, 9 key points for the left and right eyebrows, and the remaining key points are set along the facial contour.

[0015] (9) Determine the driver's state: Determine if the driver is distracted: Calculate the offset relative to the face detection box based on the 9 key points of the nose to determine the head posture; Determine if the driver is fatigued: Calculate the degree of opening and closing of the glasses based on the 18 key points of the eyes and the degree of opening and closing of the mouth based on the 18 key points of the mouth.

[0016] Preferably, in step (9), the offset of the nine key points of the nose relative to the center of the face detection box is calculated to determine the head posture. When the offset exceeds the left preset value and / or the right preset value, a left and right look warning is triggered. When the offset exceeds the downward movement preset value, a head-down warning is triggered. The coordinate distance between the upper and lower key points of each eye is calculated to determine the degree of eye opening and closing, and the eye opening and closing frequency is calculated. When the eye opening and closing frequency exceeds the preset value, a fatigue warning is triggered. The degree of mouth opening and closing is determined based on the coordinate distance between the upper and lower key points of the mouth, and the mouth opening and closing frequency is calculated. When the mouth opening and closing frequency exceeds the preset value, a fatigue warning is triggered.

[0017] More preferably, the following warnings are provided: Head-down warning: If the driver's head is down for more than 1.0 second, an alarm is triggered; Looking left and right warning: If the driver's gaze is deviated for more than 1.0 second, an alarm is triggered; Eye-closed detection is divided into two levels: Eye-closed for 0.8 seconds triggers a Level 1 warning, and Eye-closed for 2 seconds triggers a Level 2 warning; Yawning warning: If the driver's yawning is detected for more than 0.8 seconds, an alarm is triggered.

[0018] More preferably, the inability of the infrared camera to detect a face can trigger an unmanned driver's seat warning.

[0019] Preferably, the formula for SmoothL1Loss in step (5) is: , where x and y are the model's output and label, respectively, and |xy| represents the difference between them; when |xy| is less than 1, the squared error is used; otherwise, the linear error is used.

[0020] A driver distraction and fatigue detection system based on an infrared camera includes:

[0021] The image acquisition module is used to acquire images of the driver;

[0022] The image data processing module is used to process the acquired driver images.

[0023] The facial feature extraction module is used to extract facial features from images after data processing.

[0024] The face detection module is used to perform face detection on the extracted facial features and input them into the PFLD face 98 key point detection model;

[0025] Replace the model backbone module with the ResNet_18 backbone extraction module, and replace the loss function with the smoothL1 loss function.

[0026] Remove auxiliary network modules: This step removes neural network modules in the auxiliary network of the PFLD neural network that are related to the output yaw angle, rotation angle, and roll angle, and only retains modules related to facial key points.

[0027] The learning rate adjustment strategy module is used to adjust the learning rate strategy of Cosine AnnealingLR to replace the original PFLD face 98 key point detection model when a specific metric is used.

[0028] The output module is used to output 98 facial key points in an infrared image after the optimized PFLD model has been trained and tested. Among them, there are 9 key points for the nose, 18 key points for the mouth, 9 key points for the left and right eyes, 9 key points for the left and right eyebrows, and the remaining key points are set along the facial contour.

[0029] Driver status assessment module: It calculates the offset relative to the face detection box based on the 9 key points of the nose to determine whether the driver is distracted; it calculates the degree of eye opening and closing based on the 18 key points of the eyes and the degree of mouth opening and closing based on the 18 key points of the mouth to determine whether the driver is fatigued.

[0030] Preferably, in the driver status judgment module, the following warnings are provided: head-down warning: if the driver's head is down for more than 1.0 second, an alarm is triggered; looking left and right warning: if the driver's gaze is deviated for more than 1.0 second, an alarm is triggered; eye-closing detection is divided into two levels: eye-closing for 0.8 seconds triggers a level 1 warning, and eye-closing for 2 seconds triggers a level 2 warning; yawning warning: if the driver's gaze is yawned for more than 0.8 seconds, an alarm is triggered.

[0031] Preferably, the image acquisition module is further configured to trigger an unmanned driver's seat warning when the infrared camera fails to detect a face.

[0032] Preferably, the formula for SmoothL1Loss is: , where x and y are the model's output and label, respectively, and |xy| represents the difference between them; when |xy| is less than 1, the squared error is used; otherwise, the linear error is used.

[0033] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the driver distraction and fatigue detection method based on an infrared camera.

[0034] Compared with existing technologies, the advantages of this invention are as follows: This invention provides a driver distraction and fatigue detection method and system based on an infrared camera. It uses multiple key points to jointly determine distraction and fatigue, achieving high detection accuracy. Based on driver facial image processing, it studies the real-time state of the driver. First, it mines the facial expression features of a person in a fatigued state. Then, it quantifies these qualitative facial features, extracting facial feature points and feature indicators as judgment criteria. Next, it combines experimental data to summarize a recognition method based on these parameters. Finally, it inputs the acquired state data for recognition and judgment. Fatigue and distraction are determined by the driver's head posture. The driver's head posture indicates whether they are paying sufficient attention to the road ahead. If the driver does not see the road ahead for a period of time at a frequency greater than a predetermined threshold, a warning signal is generated through distraction detection. If the driver is looking forward and the percentage of time with their eyes closed exceeds a predetermined threshold within a specific time period, the fatigue detection system also generates a warning signal. Attached Figure Description

[0035] Figure 1 A flowchart of a driver distraction fatigue detection method based on an infrared camera provided by the present invention;

[0036] Figure 2 A schematic diagram of the ResNet model in the driver distraction fatigue detection method based on an infrared camera provided by the present invention;

[0037] Figure 3The variation diagram of the SmoothL1Loss curve in a driver distraction fatigue detection method based on an infrared camera provided by the present invention;

[0038] Figure 4 A graph illustrating the training process of the cosine annealing learning rate adjustment strategy in a driver distraction and fatigue detection method based on an infrared camera provided by this invention.

[0039] Figure 5 This is a schematic diagram of 98 facial key points in the output infrared image of a driver distraction and fatigue detection method based on an infrared camera provided by the present invention. Detailed Implementation

[0040] The preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0041] This invention provides a driver distraction fatigue detection method based on an infrared camera, comprising the following steps: (1) Image acquisition: using an infrared camera to acquire an image of the driver's face to obtain a picture;

[0042] (2) Image data processing: bilateral filtering, grayscale conversion of color images, and transformation of images in the HSV color space are employed;

[0043] (3) Facial feature extraction: The facial features of the image after data processing in step (2) are extracted by the face detection model retinaface;

[0044] (4) Face detection: The retainface model is used to detect faces on the image from which facial features were extracted in step (3). After obtaining the face region, the face region is cropped and input into the PFLD face 98 key point detection model.

[0045] (5) Replace the model backbone: In the key point regression model part of the PFLD face 98 key point detection model, the backbone is extracted using ResNet_18, and the loss function is replaced with the smoothL1 loss function.

[0046] (6) Remove auxiliary network: Remove the pose estimation part of the PFLD face 98 key point detection model, remove the neural network modules related to the output yaw angle, rotation angle and roll angle in the auxiliary network of the PFLD neural network, and only retain the modules related to face key points;

[0047] (7) Modify the gradient descent learning rate adjustment strategy: Replace the original PFLD face 98 key point detection model with a specific metric by using the cosine annealing learning rate adjustment strategy CosineAnnealingLR;

[0048] (8) Calculate the coordinates of key points output by the model: After the optimized PFLD model is trained, the infrared image outputs 98 facial key points after testing. Among them, there are 9 key points for the nose, 18 key points for the mouth, 9 key points for the left and right eyes, 9 key points for the left and right eyebrows, and the remaining key points are set along the facial contour.

[0049] (9) Determine the driver's state: Determine if the driver is distracted: Calculate the offset relative to the face detection box based on the 9 key points of the nose to determine the head posture; Determine if the driver is fatigued: Calculate the degree of opening and closing of the glasses based on the 18 key points of the eyes and the degree of opening and closing of the mouth based on the 18 key points of the mouth.

[0050] In step (1), compared with ordinary cameras, infrared cameras can work in all working conditions (including daytime, nighttime, front light, backlight, etc.) and can adapt to various light source environments. That is, they can provide high-quality data even at night and in backlight. At the same time, when the driver is wearing a hat, glasses, sunglasses, or mask, the infrared characteristics can penetrate the sunglasses lens, and the information of the blocked eyes can still be imaged normally, which solves the problem of reflection of the driver's glasses or wearing sunglasses.

[0051] Step (2) employs bilateral filtering, color image grayscale conversion, and image transformation in the HSV color space to achieve data augmentation. Bilateral filtering is a commonly used method that preserves edges and smooths noise reduction. To achieve a similar effect to the infrared image after color image grayscale conversion, the image is not converted from three channels to a single channel, but rather the three channels are retained and each channel is made consistent. Bilateral filtering, color image grayscale conversion, and transformation in the HSV color space are three data augmentation methods that greatly help the training results of the model. The data acquisition in step (1) is mainly for application, while the data processing in step (2) is for the training dataset. The dataset includes not only the infrared images acquired in step (1) but also color images. Color image grayscale conversion is mainly aimed at making the color images fit the infrared images.

[0052] In step (3), facial feature extraction also includes facial detection. Facial key point detection is the foundation for other face-related tasks such as face recognition, expression analysis, and 3D face reconstruction; in face recognition technology, it is the next step in face detection. Facial landmark detection refers to locating the key points of a face given a face image, including the points of eyebrows, eyes, nose, mouth, and facial contour areas. The facial feature extraction in step (3) is implemented in the face detection model RetinaFace. When the feature map size is 80*80 / 40*40 / 20*20, features are extracted once each time, with 32 maps each time (32 = 2 * (1 + 1 + 4 + 10), where 2 is 2 anchors, the two 1s are the face confidence (cls) and non-face confidence (not used), 4 is the box offset (box), and 10 is the feature point offset (pts)). In this way, a total of 96 maps are used to detect faces of sizes 16 / 32, 64 / 128, and 256 / 512.

[0053] PFLD stands for A Practical Facial Landmark Detector, a high-precision, fast, and small-model facial landmark detection model. In step (4), the face region is cropped after the face coordinates are output by the face detection model retinaface. In step (4), the face detection algorithm uses the PFLD 98-point face landmark detection model, which first needs to achieve the face detection function. This function uses the retainface model to detect faces in the image. After obtaining the face region, the image of this part is cropped and input into the PFLD 98-point face landmark detection model. The original officially released model contains two parts: the first part is the landmark regression model, and the second part is the pose estimation. The existing model is a regression model based on MobileNet_V2.

[0054] In step (5), compared to the MobileNet_V2 backbone for feature extraction, the ResNet_18 backbone has a relatively stronger ability to extract image features. The core of the ResNet model ( Figure 2 As shown, by establishing "shortcuts" (skip connections) between earlier and later layers for residual learning, it is possible to facilitate backpropagation of gradients during training, thereby enabling the training of deeper CNN networks. The loss function is replaced with the smoothL1 loss function, a commonly used loss function typically employed in regression tasks. Its advantage over the mean squared error (MSE) loss function lies in its smaller penalty for outliers (such as excessively large or small outliers), thus making the model more robust. Figure 3 As shown, Figure 3 The SmoothL1Loss curve variation graph is shown. The formula for SmoothL1Loss in step (5) is: Here, x and y are the model's output and label, respectively, and |xy| represents the difference between them; when |xy| is less than 1, squared error is used; otherwise, linear error is used. This makes SmoothL1Loss more robust than MSE, meaning it responds more smoothly to outliers.

[0055] In step (6), the neural network modules related to the output yaw angle, rotation angle and roll angle in the auxiliary network of the PFLD neural network are removed, and only the modules related to the key points of the face are retained. This is because the yaw angle test results under the camera view are frequently changing and fluctuate greatly. It is not accurate to judge whether the driver is looking ahead by relying solely on the yaw angle.

[0056] In step (7), the Cosine AnnealingLR (Cosine Annealing LR) strategy is used to replace the original model's strategy of changing the learning rate when specific metrics, such as training loss, validation loss, or accuracy, no longer change. The Cosine Annealing LR strategy differs from previous learning rate adjustment strategies. Model parameters are highly unstable during initialization, so a small learning rate is needed at the beginning of training. However, a small learning rate makes the training process very slow. Therefore, a "warm-up" phase for network training is achieved by gradually increasing the learning rate from a lower one to a higher one. If the network training loss is to be minimized, using a consistently high learning rate is inappropriate because it causes the gradient of the weights to oscillate back and forth, making it difficult to reach the global minimum loss value. Therefore, the learning rate still needs to be reduced, which can be achieved using a cosine function. Figure 3 As shown, in the cosine function, the cosine value first decreases slowly as x increases, then decreases more rapidly, and then decreases slowly again. This decreasing pattern can be combined with the learning rate to produce excellent results in a very efficient calculation method.

[0057] After the optimized PFLD model in step (8) is trained, the infrared image outputs the location map of 98 facial key points after testing, as shown below. Figure 5 As shown.

[0058] In step (9), the offset of the nine key points of the nose relative to the center of the face detection box is calculated to determine the head posture. When the offset exceeds the left preset value and / or the right preset value, a left and right look warning is triggered. When the offset exceeds the downward shift preset value, a head-down warning is triggered. The coordinate distance between the upper and lower key points of each eye is calculated to determine the degree of eye opening and closing, and the frequency of eye opening and closing is calculated. When the frequency of eye opening and closing exceeds the preset value, a fatigue warning is triggered. The degree of mouth opening and closing is determined based on the coordinate distance between the upper and lower key points of the mouth, and the frequency of mouth opening and closing is calculated. When the frequency of mouth opening and closing exceeds the preset value, a fatigue warning is triggered. Head-down warning: If head-down is detected for more than 1.0 second, an alarm is triggered. Left and right look warning: If the driver's gaze shifts for more than 1.0 second, an alarm is triggered. Eye-closing detection is divided into two levels: 0.8 seconds of eye-closing triggers a level 1 warning, and 2 seconds of eye-closing triggers a level 2 warning. Yawning warning: If yawning is detected for more than 0.8 seconds, an alarm is triggered. The infrared camera cannot detect a face to trigger a driver-seat unmanned warning.

[0059] The present invention also provides a driver distraction and fatigue detection system based on an infrared camera, comprising:

[0060] The image acquisition module is used to acquire images of the driver;

[0061] The image data processing module is used to process the acquired driver images.

[0062] The facial feature extraction module is used to extract facial features from images after data processing.

[0063] The face detection module is used to perform face detection on the extracted facial features and input them into the PFLD face 98 key point detection model;

[0064] Replace the model backbone module with the ResNet_18 backbone extraction module, and replace the loss function with the smoothL1 loss function.

[0065] Remove auxiliary network modules: This step removes neural network modules in the auxiliary network of the PFLD neural network that are related to the output yaw angle, rotation angle, and roll angle, and only retains modules related to facial key points.

[0066] The learning rate adjustment strategy module is used to adjust the learning rate strategy of Cosine AnnealingLR to replace the original PFLD face 98 key point detection model when a specific metric is used.

[0067] The output module is used to output 98 facial key points in an infrared image after the optimized PFLD model has been trained and tested. Among them, there are 9 key points for the nose, 18 key points for the mouth, 9 key points for the left and right eyes, 9 key points for the left and right eyebrows, and the remaining key points are set along the facial contour.

[0068] Driver status assessment module: It calculates the offset relative to the face detection box based on the 9 key points of the nose to determine whether the driver is distracted; it calculates the degree of eye opening and closing based on the 18 key points of the eyes and the degree of mouth opening and closing based on the 18 key points of the mouth to determine whether the driver is fatigued.

[0069] The image acquisition module uses an infrared camera to capture images. Infrared cameras are less affected by environmental factors and can adapt to various lighting environments, providing high-quality data even at night or in backlit conditions. The image acquisition module also triggers a driver-side unmanned warning when the infrared camera fails to detect a face.

[0070] The image data processing module employs bilateral filtering, grayscale conversion of color images, and transformation of images in the HSV color space.

[0071] Among them, the facial feature extraction module extracts facial features from the processed images using the RetinaFace face detection model.

[0072] The face detection module uses the retainface model to detect faces in the image. After obtaining the face region, the image of that part is cropped and input into the PFLD face 98 key point detection model.

[0073] The main module of the model is replaced. Compared to the feature extraction backbone of MobileNet_V2, the ResNet_18 backbone has a relatively stronger ability to extract image features. The core of the ResNet model is to establish "shortcuts" (skip connections) between previous and subsequent layers for residual learning. This helps with backpropagation of gradients during training, thus enabling the training of deeper CNN networks. The loss function is replaced with the SmoothL1 loss function. SmoothL1 Loss is a commonly used loss function, typically used in regression tasks. Its advantage over the Mean Squared Error (MSE) loss function is that it penalizes outliers (such as excessively large or small outliers) less, making the model more robust. The formula for SmoothL1 Loss is: Here, x and y are the model's output and label, respectively, and |xy| represents the difference between them; when |xy| is less than 1, squared error is used; otherwise, linear error is used. This makes SmoothL1Loss more robust than MSE, meaning it responds more smoothly to outliers.

[0074] The removal of auxiliary network modules is used to remove neural network modules related to the output yaw angle, rotation angle, and roll angle in the auxiliary network of the PFLD neural network, retaining only the modules related to facial key points. This is because the yaw angle test results under the camera's perspective change frequently and fluctuate greatly, making it extremely inaccurate to rely solely on the yaw angle to determine whether the driver is looking ahead.

[0075] The learning rate adjustment strategy module, specifically the Cosine AnnealingLR strategy, uses a cosine function where the cosine value initially decreases slowly as x increases, then accelerates, and finally decreases slowly again. This decreasing pattern, combined with the learning rate, produces excellent results through a highly efficient calculation method. The Cosine AnnealingLR strategy is used to replace specific metrics in the original PFLD face 98 keypoint detection model.

[0076] In the driver status judgment module, the following warnings are triggered: head down warning: an alarm is triggered if the driver's head is down for more than 1.0 second; looking left and right warning: an alarm is triggered if the driver's gaze is deviated for more than 1.0 second; closed eyes detection is divided into two levels: closed eyes for 0.8 seconds triggers a level 1 warning, and closed eyes for 2 seconds triggers a level 2 warning; yawning warning: an alarm is triggered if yawning for more than 0.8 seconds is detected.

[0077] It is worth noting that fatigue and distraction are determined by the driver's head posture, which indicates whether the driver is paying sufficient attention to the road ahead. If the frequency of time during which the driver does not see the road ahead exceeds a predetermined threshold, a warning signal is generated through distraction detection. If the driver is looking straight ahead and the percentage of time spent with their eyes closed exceeds a predetermined threshold within a specific time period, a warning signal is also generated by the fatigue detection system.

[0078] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the driver distraction and fatigue detection method based on an infrared camera.

[0079] The present invention provides a driver distraction fatigue detection method and system based on an infrared camera, which has the following advantages: 1. The infrared camera is less affected by environmental factors and can adapt to various light source environments, that is, it can provide high-quality data even at night or in backlight conditions; 2. The model has high detection accuracy; 3. Using multiple key points to jointly judge distraction fatigue is more accurate; 4. The optimized algorithm model has strong robustness; 5. The algorithm model occupies less NPU resources and can be applied to chips with low computing power, reducing costs.

[0080] This infrared camera-based driver distraction and fatigue detection method uses multiple key points to jointly determine distraction and fatigue, achieving high detection accuracy. It studies a real-time system for driver state based on driver facial image processing. First, it mines facial expression features in a fatigued state, then quantifies these qualitative facial features, extracting facial feature points and indicator values ​​as judgment criteria. Next, it combines experimental data to summarize a recognition method based on these parameters. Finally, it inputs the acquired state data for recognition and judgment. Fatigue and distraction are determined by the driver's head posture, indicating whether the driver is paying sufficient attention to the road ahead. If the driver does not see the road ahead for a period of time at a frequency exceeding a predetermined threshold, a warning signal is generated through distraction detection. If the driver is looking forward and the percentage of time with eyes closed exceeds a predetermined threshold within a specific time period, the fatigue detection system also generates a warning signal.

[0081] In summary, the technical solution of this invention can fully and effectively achieve the above-mentioned objectives. Furthermore, the structure and functional principles of this invention have been fully verified in the embodiments, achieving the expected effects and objectives. Without departing from the principles and essence of this invention, various changes or modifications can be made to the embodiments. Therefore, this invention includes all substitutions within the scope mentioned in the patent application claims, and any equivalent changes made within the scope of this patent application are within the scope of the patent application.

Claims

1. A method for detecting driver distraction and fatigue based on an infrared camera, characterized in that, The method comprises the following steps: (1) image acquisition: an infrared camera is used to obtain a driver's face image to obtain a picture; (2) image data processing: bilateral filtering, color picture grayscale and picture transformation in the hsv color space are used in sequence; (3) face feature extraction: the picture processed in step (2) is extracted by a face detection model retinaface to obtain face features; (4) face detection: retainface model is used to detect faces on the image of the face features extracted in step (3), and the face region is obtained and input into a PFLD face 98 key point detection model; (5) replace the model backbone: the key point regression model part in the PFLD face 98 key point detection model is replaced by ResNet_18 to extract the backbone, and the loss function part is replaced by a smoothL1 loss function; (6) remove the auxiliary network: the pose estimation part in the PFLD face 98 key point detection model is removed from the auxiliary network of the PFLD neural network, and the neural network modules related to the output yaw angle, roll angle and roll angle are removed, and only the modules related to the face key points are retained; (7) modify the gradient descent adjustment learning rate strategy: the CosineAnnealingLR adjustment learning rate strategy is used to replace the original PFLD face 98 key point detection model when the specific metric is reached; (8) calculate the model output key point coordinate position: after the optimized PFLD model is trained, 98 face key points of the infrared image are output, including 9 key points of the nose part, 18 key points of the mouth part, 9 key points of the left eye part and 9 key points of the right eye part, 9 key points of the left eyebrow part and 9 key points of the right eyebrow part, and the remaining key points are arranged along the face contour; (9) judge the driver's state: judge whether the driver is distracted: judge the head posture according to the offset of the 9 key points of the nose part relative to the face detection frame; judge whether the driver is tired: calculate the degree of opening of the glasses according to the 18 key points of the eye part and the degree of opening of the mouth according to the 18 key points of the mouth part.

2. The infrared camera based driver distraction and fatigue detection method of claim 1, wherein: In step (9), the 9 key points of the nose are calculated relative to the center of the face detection frame to determine the head posture, and when the offset exceeds the left preset value and / or the right preset value, the left and right looking warning is triggered, and when the offset exceeds the downward preset value, the looking down warning is triggered; The coordinate distance between the upper and lower key points of each eye is calculated to determine the degree of opening of the eyes, and the opening frequency of the eyes is calculated; when the eye opening frequency exceeds the preset value, the fatigue warning is triggered; the coordinate distance between the upper and lower key points of the mouth is calculated to determine the degree of opening of the mouth, and the opening frequency of the mouth is calculated; when the mouth opening frequency exceeds the preset value, the fatigue warning is triggered.

3. The infrared camera based driver distraction and fatigue detection method of claim 2, wherein: The low head warning is triggered when the low head is detected for more than 1.0 second; the left and right look warning is triggered when the driver's line of sight is detected to deviate for more than 1.0 second; the eye closure detection is divided into two levels: the first level warning is triggered when the eyes are closed for 0.8 second, and the second level warning is triggered when the eyes are closed for 2 seconds; and the yawn warning is triggered when the yawn is detected for more than 0.8 second.

4. The infrared camera based driver distraction and fatigue detection method of claim 1, wherein: The driver seat no one warning can be triggered when the infrared camera cannot detect a face.

5. The infrared camera based driver distraction and fatigue detection method of claim 1, wherein: The formula of SmoothL1Loss in step (5) is: where x and y are the output of the model and the label respectively, and |x-y| represents the difference between them; when |x-y| is less than 1, the square error is adopted; otherwise, the linear error is adopted.

6. A driver distraction and fatigue detection system based on infrared camera, characterized in that, The method comprises the following steps: an image acquisition module is configured to acquire images of the driver; an image data processing module is configured to process the acquired images of the driver; a face feature extraction module is configured to extract face features from the processed images; a face detection module is configured to detect the extracted face features and input them into a PFLD face 98-keypoint detection model; a replacement model backbone module is configured to extract a backbone from ResNet_18 and replace a loss function part with a smoothL1 loss function; a removal auxiliary network module is configured to remove neural network modules related to output yaw, roll and pitch angles in an auxiliary network of the PFLD neural network, and only keep modules related to face key points; an adjustment learning rate strategy module is configured to replace a specific metric in the original PFLD face 98-keypoint detection model with a CosineAnnealingLR adjustment learning rate strategy; an output module is configured to output 98 face key points of an infrared image after the optimized PFLD model is trained and tested, wherein 9 key points are located at a nose part, 18 key points are located at a mouth part, 9 key points are located at a left eye part and a right eye part, 9 key points are located at a left eyebrow part and a right eyebrow part, and the remaining key points are located along a face contour; a driver state judgment module is configured to calculate a deviation relative to a face detection frame based on the positions of the 9 key points at the nose part, to determine whether the driver is in a distraction state, and to calculate the opening and closing degrees of glasses based on the positions of the 18 key points at the eye part and the opening and closing degrees of the mouth based on the positions of the 18 key points at the mouth part, to determine whether the driver is in a fatigue state.

7. The infrared camera based driver distraction and fatigue detection system of claim 6, wherein: In the driver state judgment module, the low head warning is triggered when the low head is detected for more than 1.0 second; the left and right look warning is triggered when the driver's line of sight is detected to deviate for more than 1.0 second; the eye closure detection is divided into two levels: the first level warning is triggered when the eyes are closed for 0.8 second, and the second level warning is triggered when the eyes are closed for 2 seconds; and the yawn warning is triggered when the yawn is detected for more than 0.8 second.

8. The infrared camera based driver distraction and fatigue detection system of claim 6, wherein: The image acquisition module is further configured to trigger the driver seat no one warning when the infrared camera cannot detect a face.

9. The infrared camera based driver distraction and fatigue detection system of claim 6, wherein: The formula of the SmoothL1Loss is as follows: wherein x and y are respectively the output of the model and the label, and |x-y| represents the difference between them; when |x-y| is less than 1, the square error is adopted; otherwise, the linear error is adopted.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the driver distraction and fatigue detection method based on the infrared camera as claimed in any one of claims 1 to 4.

Images