Driver fatigue status detection method and system

Abstract

The present invention provides a driver fatigue state detection method and system, comprising: collecting a driver's facial image; inputting the driver's facial image into a deep learning fast face detection algorithm to obtain left eye center coordinates, right eye center coordinates and binocular distance; calculating a left eye circumscribed rectangular frame and a right eye circumscribed rectangular frame according to the left eye center coordinates, right eye center coordinates and binocular distance; determining an eye rectangular frame according to the left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame; inputting the eye rectangular frame into a pupil detection model to obtain pupil area coordinates; calculating an adaptive threshold according to the pupil area coordinates; calculating a mean aspect ratio within a preset time period; comparing the mean aspect ratio with the adaptive threshold, and determining the current state of the driver according to the comparison result; and solving the problems of inaccurate fatigue detection and weak robustness caused by factors such as individual differences of drivers or complex lighting in the prior art.

Description

Technical Field

[0001] The present invention relates to the technical field of new energy vehicles, and in particular to a method and system for detecting a driver's fatigue state. Background Art

[0002] At present, the driver fatigue detection methods mainly include: based on the driver's physiological signals, such as electrocardiogram and electroencephalogram, various sensors are attached to the driver, and whether the driver is in a fatigue state is determined by obtaining the driver's physiological data. This method has high detection accuracy, but it is expensive and requires the cooperation of the driver, and the operation is cumbersome; based on visual technology to analyze the driver's facial information, such as judging the degree of eye closure through the eyelid aspect ratio, thereby judging whether the driver is fatigued driving, but the existing eyelid aspect ratio method is greatly affected by individual differences of drivers, complex lighting and other factors. For example, the driver has small eyes, suddenly squinting under strong light, glasses factors, the driver turns his head, etc., which are easy to cause false detection and missed detection.

[0003] In summary, the above detection methods mainly include sensor detection and image detection. Although sensor detection has high detection accuracy, it is expensive and complicated to operate. Image detection judges the degree of closure of human eyes by analyzing the aspect ratio of eyelids. Although it has high accuracy, it is greatly affected by individual differences of drivers and complex lighting, and is prone to false detection and missed detection. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a method and system for detecting driver fatigue status, which solves the problems of inaccurate and weak robustness of fatigue detection in the prior art caused by factors such as individual differences in drivers or complex lighting.

[0005] In a first aspect, an embodiment of the present invention provides a method for detecting a driver's fatigue state, the method comprising:

[0006] Collecting a driver's facial image, inputting the driver's facial image into a multi-scale face detection algorithm of deep learning to obtain face positioning information under different lighting conditions;

[0007] Inputting the driver's facial image into a fast face detection algorithm of deep learning to obtain the left eye center coordinates, the right eye center coordinates and the binocular distance;

[0008] Calculate a left eye circumscribed rectangular frame and a right eye circumscribed rectangular frame according to the left eye center coordinates, the right eye center coordinates and the binocular distance;

[0009] Determine an eye rectangular frame according to the left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame;

[0010] Input the eye rectangular frame into the pupil detection model to obtain the pupil area coordinates;

[0011] Calculating an adaptive threshold according to the pupil area coordinates;

[0012] Calculate the average value of aspect ratio within a preset time period;

[0013] The aspect ratio mean is compared with the adaptive threshold, and the current state of the driver is determined according to the comparison result.

[0014] Further, calculating the adaptive threshold according to the pupil area coordinates includes:

[0015] Extracting a pupil area from the pupil area coordinates and outputting bounding box parameters; wherein the bounding box parameters include the center coordinates of the bounding box, the height of the bounding box, and the width of the bounding box;

[0016] Calculating the aspect ratio of the pupil according to the height of the bounding box and the width of the bounding box;

[0017] Taking the aspect ratio of the pupil as a statistic;

[0018] The adaptive threshold is calculated according to the statistic.

[0019] Further, calculating the adaptive threshold according to the statistic includes:

[0020] When the driver is driving normally, the first t0 minutes of the initial driving journey are called, and the aspect ratio of the pupil of each frame is calculated;

[0021] The aspect ratio of the pupil in each frame is averaged to obtain the adaptive threshold.

[0022] Furthermore, the average value of the aspect ratio within a preset time period is calculated, including:

[0023] Detect the first t1 seconds at time t;

[0024] If within the first t1 seconds, the average of the aspect ratios for consecutive t2 seconds is calculated;

[0025] Wherein, the t is greater than the t1, and the t1 is greater than t2.

[0026] Further, comparing the aspect ratio mean with the adaptive threshold, and determining the current state of the driver according to the comparison result, including:

[0027] When the aspect ratio mean is less than the adaptive threshold, the driver is in a fatigue state;

[0028] When the aspect ratio average is greater than or equal to the adaptive threshold, the driver is in a normal state.

[0029] Furthermore, before inputting the driver's facial image into a fast face detection algorithm of deep learning, the method further includes:

[0030] Performing image resizing on the driver's facial image to obtain a processed driver's facial image;

[0031] The processed driver's facial image is normalized to obtain a normalized driver's facial image.

[0032] In a second aspect, an embodiment of the present invention provides a driver fatigue state detection system, the system comprising:

[0033] An acquisition module is used to acquire a driver's facial image, and input the driver's facial image into a multi-scale face detection algorithm of deep learning to obtain face location information under different lighting conditions;

[0034] A first input module, used for inputting the driver's facial image into a fast face detection algorithm of deep learning to obtain the left eye center coordinates, the right eye center coordinates and the binocular distance;

[0035] A circumscribed rectangular frame calculation module, used for calculating a left eye circumscribed rectangular frame and a right eye circumscribed rectangular frame according to the left eye center coordinates, the right eye center coordinates and the binocular distance;

[0036] A determination module, configured to determine an eye rectangular frame according to the left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame;

[0037] A second input module is used to input the eye rectangular frame into the pupil detection model to obtain pupil area coordinates;

[0038] An adaptive threshold calculation module, used for calculating an adaptive threshold according to the pupil area coordinates;

[0039] A statistics module, used to calculate the average value of aspect ratio within a preset time period;

[0040] A comparison module is used to compare the aspect ratio mean with the adaptive threshold and determine the current state of the driver according to the comparison result.

[0041] Furthermore, the adaptive threshold calculation module is specifically used for:

[0042] Extracting a pupil area from the pupil area coordinates and outputting bounding box parameters; wherein the bounding box parameters include the center coordinates of the bounding box, the height of the bounding box, and the width of the bounding box;

[0043] Calculating the aspect ratio of the pupil according to the height of the bounding box and the width of the bounding box;

[0044] Taking the aspect ratio of the pupil as a statistic;

[0045] The adaptive threshold is calculated according to the statistic.

[0046] Furthermore, the adaptive threshold calculation module is specifically used for:

[0047] When the driver is driving normally, the first t0 minutes of the initial driving journey are called, and the aspect ratio of the pupil of each frame is calculated;

[0048] The aspect ratio of the pupil in each frame is averaged to obtain the adaptive threshold.

[0049] Furthermore, the statistical module is specifically used for:

[0050] Detect the first t1 seconds at time t;

[0051] If within the first t1 seconds, the average of the aspect ratios for consecutive t2 seconds is calculated;

[0052] Wherein, the t is greater than the t1, and the t1 is greater than t2.

[0053] Furthermore, the comparison module is specifically used for:

[0054] When the aspect ratio mean is less than the adaptive threshold, the driver is in a fatigue state;

[0055] When the aspect ratio average is greater than or equal to the adaptive threshold, the driver is in a normal state.

[0056] Furthermore, before inputting the driver's facial image into the deep learning fast face detection algorithm, the system further includes:

[0057] A preprocessing module, used for performing image resizing on the driver's facial image to obtain a processed driver's facial image;

[0058] The normalization processing module is used to normalize the processed driver's facial image to obtain a normalized driver's facial image.

[0059] In a third aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and the processor implements the above-mentioned driver fatigue status detection method when executing the computer program.

[0060] In a fourth aspect, an embodiment of the present invention provides a computer-readable medium having a non-volatile program code executable by a processor, wherein the program code enables the processor to execute the driver fatigue status detection method.

[0061] The embodiment of the present invention provides a driver fatigue status detection method and system, including: collecting a driver's facial image; inputting the driver's facial image into a deep learning fast face detection algorithm to obtain the left eye center coordinates, the right eye center coordinates and the binocular distance; calculating the left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame according to the left eye center coordinates, the right eye center coordinates and the binocular distance; determining the eye rectangular frame according to the left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame; inputting the eye rectangular frame into a pupil detection model to obtain pupil area coordinates; calculating an adaptive threshold according to the pupil area coordinates; counting the aspect ratio mean within a preset time period; comparing the aspect ratio mean with the adaptive threshold, and determining the driver's current state according to the comparison result; solving the problem of inaccurate fatigue detection and weak robustness caused by factors such as individual differences of drivers or complex lighting in the prior art.

[0062] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0063] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 A schematic diagram of an application scenario of a driver fatigue status detection system provided in Embodiment 1 of the present invention;

[0066] Figure 2 A schematic diagram of another application scenario of a driver fatigue status detection system provided in Embodiment 2 of the present invention;

[0067] Figure 3 A flowchart of a method for detecting driver fatigue status provided in Embodiment 3 of the present invention;

[0068] Figure 4 A schematic diagram of a human pupil detection bounding box provided in Embodiment 3 of the present invention;

[0069] Figure 5A schematic diagram of a process for calculating the mean aspect ratio provided in Embodiment 3 of the present invention;

[0070] Figure 6 A schematic diagram of a driver fatigue status detection system provided in Embodiment 4 of the present invention;

[0071] Figure 7 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention.

[0072] Icons: 1-infrared camera; 2-vehicle system equipment. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] At present, road traffic accidents are one of the most important causes of harm to the health of the global public, and road traffic accidents caused by fatigue driving account for more than half. Studies have shown that mild fatigue accounts for 36.8% of accidents and severe fatigue accounts for 31.1%. When a driver is fatigued, his reaction speed, judgment ability and attention will be greatly reduced. By detecting the driver's fatigue, it can be discovered and reminded in time, thereby effectively preventing traffic accidents and ensuring driving safety; at the same time, fatigue detection can also help drivers understand their physical condition in time, avoid overwork, and reduce driver health risks.

[0075] Existing detection methods mainly include sensor detection and image detection. Although sensor detection has high detection accuracy, it is expensive and complicated to operate. Image detection determines the degree of eye closure by analyzing the aspect ratio of the eyelid. Although it has a high accuracy rate, it is greatly affected by individual differences of drivers and complex lighting, and is prone to false detection and missed detection. The present application aims to provide a driver fatigue status detection method and system which is widely applicable to different drivers and has strong robustness, comprising: collecting a driver's facial image; inputting the driver's facial image into a deep learning fast face detection algorithm to obtain the left eye center coordinates, the right eye center coordinates and the binocular distance; calculating the left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame according to the left eye center coordinates, the right eye center coordinates and the binocular distance; determining the eye rectangular frame according to the left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame; inputting the eye rectangular frame into a pupil detection model to obtain the pupil area coordinates; calculating an adaptive threshold according to the pupil area coordinates; calculating the average aspect ratio within a preset time period; comparing the average aspect ratio with the adaptive threshold, and determining the current state of the driver according to the comparison result; and solving the problems of inaccurate fatigue detection and weak robustness caused by factors such as individual differences of drivers or complex lighting in the prior art.

[0076] To facilitate understanding of this embodiment, the embodiment of the present invention is described in detail below.

[0077] Embodiment 1:

[0078] Figure 1 A schematic diagram of an application scenario of a driver fatigue status detection system provided in Embodiment 1 of the present invention.

[0079] Reference Figure 1 , the driver's face image is captured in real time by an infrared camera installed on the A-pillar (driver's side) of the vehicle. The image size can be 1280 pixels × 800 pixels × 3 channels. The image size can be set according to requirements.

[0080] The vehicle-mounted system equipment pre-processes the captured driver's facial image, that is, adjusts the image size, and then normalizes the image to make it suitable for input into the deep learning network-based visual model. The deep learning network-based visual model includes a deep learning fast face detection algorithm and a pupil detection model.

[0081] Image normalization refers to the process of transforming an image into a fixed standard form by performing a series of standard processing transformations on the image. This standard image is called a normalized image. For example, normalizing the pixel value range of an image from [0,255] to [0,1] can better process the image. The specific calculation formula is X' = (X - X_min) / (X_max -X_min)

[0082] Among them, X' is the normalized data, X is the original data, and X_min and X_max are the minimum and maximum values ​​of the original data respectively.

[0083] Input the driver's facial image into the fast face detection algorithm of deep learning to obtain the left eye center coordinates, the right eye center coordinates and the distance between the two eyes; calculate the left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame according to the left eye center coordinates, the right eye center coordinates and the distance between the two eyes; determine the eye rectangular frame according to the left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame; input the eye rectangular frame into the pupil detection model to obtain the pupil area coordinates; calculate the adaptive threshold according to the pupil area coordinates; calculate the mean aspect ratio within a preset time period; compare the mean aspect ratio with the adaptive threshold, when the mean aspect ratio is less than the adaptive threshold, the driver is in a fatigue state; when the mean aspect ratio is greater than or equal to the adaptive threshold, the driver is in a normal state. The above process overcomes the shortcomings of inaccurate fatigue detection and weak robustness caused by factors such as individual differences of drivers or complex lighting in the prior art, and proposes a driver fatigue state detection system with wider applicability and stronger robustness.

[0084] Embodiment 2:

[0085] Figure 2 A schematic diagram of an application scenario of another driver fatigue status detection system provided in Embodiment 2 of the present invention.

[0086] Reference Figure 2 , deploy the trained YOLOv8 fast face detection algorithm and pupil detection model into the vehicle system, and accelerate inference through the vehicle NPU.

[0087] The infrared camera is installed in the middle of the A-pillar (driver's side) to capture the driver's facial image with a size of 1280 pixels × 800 pixels × 3 channels. The facial image is preprocessed, the image size is adjusted, and then normalized.

[0088] The vehicle-mounted system equipment performs face detection through the YOLOv8 fast face detection algorithm, and outputs the left eye center coordinates (x1, y1), the right eye center coordinates (x2, y2) and the binocular distance L.

[0089] The bounding rectangles of the left eye and the right eye are calculated using the binocular distance and their center coordinates, and the eye rectangles of the region of interest are output. The image is then resized and normalized to meet the requirements of the pupil detection model.

[0090] The eye rectangle is input into the pupil detection model of YOLOv8, the eye information is detected and analyzed, and the pupil area coordinates are output.

[0091] The pupil area is extracted from the pupil area coordinates, and the bounding box parameters are output; wherein the bounding box parameters include the center coordinates (x, y) of the bounding box, the height h, and the width w of the bounding box.

[0092] The aspect ratio of the pupil is taken as the statistic z=h / w. When the driver is driving normally, the first t0 minutes of the driving journey are called, the pupil aspect ratio of each frame is calculated and averaged, and the adaptive threshold z0=h0 / w0 suitable for different individual drivers is obtained.

[0093] Detect whether the driver is fatigued at time t, call the previous t1 seconds at time t, if there is a continuous t2 seconds of the average pupil aspect ratio z(t) lower than the adaptive threshold z0, judge the driver's current state is fatigue driving; if there is no continuous t2 seconds of the average pupil aspect ratio z(t) lower than the adaptive threshold z0, judge the driver's current state is normal.

[0094] Embodiment three:

[0095] Figure 3 This is a flow chart of a method for detecting driver fatigue status provided in Embodiment 3 of the present invention.

[0096] Reference Figure 3 , the method comprises the following steps:

[0097] Step S101, collecting a facial image of the driver; wherein the facial image of the driver is input into a multi-scale face detection algorithm of deep learning to obtain face location information under different lighting conditions;

[0098] Here, the driver's facial image is captured in real time by an infrared camera installed on the A-pillar (driver's side) of the vehicle, and the image size can be 1280 pixels × 800 pixels × 3 channels. The image size can be set as required.

[0099] The captured driver's facial image is preprocessed, that is, the image size is adjusted; and then the image is normalized to be suitable for input into the deep learning network-based visual model. The deep learning network-based visual model includes a deep learning fast face detection algorithm and a pupil detection model.

[0100] In addition, the driver's facial image is input into a deep learning multi-scale face detection algorithm to improve the accuracy of face location under different lighting conditions. The algorithm should be able to adapt to face images of different sizes and be able to handle the impact of lighting changes on feature extraction.

[0101] Step S102, inputting the driver's facial image into a fast face detection algorithm of deep learning to obtain the left eye center coordinates, the right eye center coordinates and the binocular distance;

[0102] Specifically, the pre-processed driver's facial image is input into the vehicle system that deploys a fast face detection algorithm based on deep learning for processing. The fast face detection algorithm is deployed in the vehicle system equipment, and the vehicle NPU model inference acceleration technology is used to enable the deep learning model trained on a high-performance server to be deployed in the vehicle system equipment with limited computing power and power.

[0103] The pre-processed image of the driver's face image is detected and analyzed using the fast face detection algorithm based on deep learning, and the left eye center coordinates (x1, y1), the right eye center coordinates (x2, y2) and the binocular distance L are output. The fast face detection algorithm based on deep learning uses the YOLOv8 model framework, which collects face data in actual application scenarios for training, obtains model weights, and deploys them to the vehicle system equipment.

[0104] Step S103, calculating the left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame according to the left eye center coordinates, the right eye center coordinates and the binocular distance;

[0105] Step S104, determining an eye rectangular frame according to the left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame;

[0106] Here, after determining the eye rectangle, the image needs to be resized and normalized to meet the requirements of the pupil detection model.

[0107] Step S105, inputting the eye rectangular frame into the pupil detection model to obtain the pupil area coordinates;

[0108] Specifically, the eye rectangular frame is normalized and then input into the constructed pupil detection model to output the pupil area coordinates.

[0109] The pupil detection model also uses the YOLOv8 model framework. In actual application scenarios, pupil data is collected for training to obtain model weights, and the pupil detection model is deployed to the vehicle-mounted system equipment.

[0110] Step S106, calculating an adaptive threshold according to pupil area coordinates;

[0111] Step S107, calculating the average value of the aspect ratio within a preset time period;

[0112] Step S108, comparing the aspect ratio mean with the adaptive threshold, and determining the current state of the driver according to the comparison result.

[0113] Specifically, the pupil area aspect ratio is used as a statistic to calculate the pupil aspect ratio z=h / w. The aspect ratio distribution in a preset time period is used to calculate the adaptive threshold for judging eye closure. The aspect ratio mean is compared with the adaptive threshold, and the comparison result is used to determine whether the driver is in a fatigue state.

[0114] Further, step S106 includes the following steps:

[0115] Step S201, extracting the pupil area from the pupil area coordinates and outputting bounding box parameters; wherein the bounding box parameters include the center coordinates of the bounding box, the height of the bounding box, and the width of the bounding box;

[0116] Step S202, calculating the aspect ratio of the pupil according to the height and width of the bounding box;

[0117] Step S203, taking the aspect ratio of the pupil as a statistic;

[0118] Step S204, calculating an adaptive threshold according to the statistics.

[0119] Specifically, the pupil area is extracted from the pupil area coordinates, and the bounding box parameters are output; the bounding box parameter format is xywh, where xy represents the center coordinates of the bounding box (x, y), and wh represents the width and height of the bounding box (width, height). Figure 4 This format describes the position and size of the bounding box relative to the image, making the position and size information of the detected object more intuitive and convenient for subsequent processing.

[0120] Further, step S204 includes the following steps:

[0121] Step S301, when the driver is driving normally, calling the first t0 minutes of the initial driving journey and calculating the aspect ratio of the pupil in each frame;

[0122] Step S302, averaging the aspect ratio of the pupil in each frame to obtain an adaptive threshold.

[0123] Specifically, the pupil area aspect ratio is used as a statistic to calculate the pupil aspect ratio z=h / w. The aspect ratio distribution within a preset time period is used to calculate the adaptive threshold for judging eye closure.

[0124] When the driver is driving normally, the first t0 minutes of the driving journey are called, the aspect ratio of each frame of the pupil is calculated, and the average is performed to obtain the adaptive threshold z0=h0 / w0 suitable for different drivers. For details, refer to Figure 5 .

[0125] Further, step S107 includes the following steps:

[0126] Step S401, detecting the first t1 seconds at time t;

[0127] Step S402, if within the first t1 seconds, the average of the aspect ratios for t2 consecutive seconds is calculated;

[0128] Among them, t is greater than t1, and t1 is greater than t2.

[0129] Specifically, it is detected whether the driver is fatigued at time t, and the previous t1 seconds at time t are called; if within the previous t1 seconds, there are t2 consecutive seconds of the average pupil aspect ratio z(t) being less than the adaptive threshold z0, it is judged that the driver is fatigued driving at this moment.

[0130] Further, step S108 includes the following steps:

[0131] Step S501, when the aspect ratio mean is less than the adaptive threshold, the driver is in a fatigue state;

[0132] Step S502: When the aspect ratio mean is greater than or equal to the adaptive threshold, the driver is in a normal state.

[0133] Furthermore, before inputting the driver's facial image into the fast face detection algorithm of deep learning, the method further includes the following steps:

[0134] Step S601, performing image resizing processing on the driver's facial image to obtain a processed driver's facial image;

[0135] Step S602, normalize the processed driver's facial image to obtain a normalized driver's facial image. Image normalization refers to a process of performing a series of standard processing transformations on an image to transform it into a fixed standard form, and the standard image is called a normalized image.

[0136] The embodiment of the present invention provides a method for detecting a driver's fatigue state, comprising: collecting a facial image of the driver; inputting the facial image of the driver into a fast face detection algorithm of deep learning to obtain the left eye center coordinates, the right eye center coordinates and the binocular distance; calculating the left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame according to the left eye center coordinates, the right eye center coordinates and the binocular distance; determining the eye rectangular frame according to the left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame; inputting the eye rectangular frame into a pupil detection model to obtain pupil area coordinates; calculating an adaptive threshold according to the pupil area coordinates; calculating the average aspect ratio within a preset time period; comparing the average aspect ratio with the adaptive threshold, and determining the current state of the driver according to the comparison result; and solving the problems of inaccurate and weak robustness of fatigue detection in the prior art caused by factors such as individual differences of drivers or complex lighting.

[0137] In addition, the driver's fatigue state detection can also be achieved in the following way: when the vehicle is driving, the driver's mental state is first detected to timely find out whether the driver is in a fatigue driving state, so as to remind the driver of fatigue driving. Specifically, by obtaining the current driving information of the target vehicle to determine the continuous driving situation of the target vehicle in the current driving process, when the current driving information meets the preset conditions, it is determined whether it is necessary to perform fatigue detection on the driver to detect whether the driver's mental state is normal, and fatigue detection voice is generated according to the current driving information, and the fatigue detection voice is broadcast to the driver of the target vehicle, thereby realizing the detection of the driver's mental state by voice interaction with the driver. Among them, the current driving information includes but is not limited to mileage, driving speed and continuous driving time, and can also be other information. The preset condition can be to determine whether the continuous driving mileage of the current driving process exceeds the preset driving threshold, or whether the continuous driving time in the current driving process exceeds the preset driving time threshold.

[0138] Embodiment 4:

[0139] Figure 6 This is a schematic diagram of a driver fatigue status detection system provided in Embodiment 4 of the present invention.

[0140] Reference Figure 6 , the system comprises:

[0141] A collection module, used for collecting the driver's facial image;

[0142] Specifically, the driver's facial image is captured in real time by an infrared camera installed on the A-pillar (driver's side) of the vehicle, and the image size can be 1280 pixels × 800 pixels × 3 channels. The image size can be set according to requirements.

[0143] The captured driver's facial image is preprocessed, that is, the image size is adjusted; and then the image is normalized to be suitable for input into the deep learning network-based visual model. The deep learning network-based visual model includes a deep learning fast face detection algorithm and a pupil detection model.

[0144] The first input module is used to input the driver's facial image into a fast face detection algorithm of deep learning to obtain the left eye center coordinates, the right eye center coordinates and the binocular distance;

[0145] Specifically, the pre-processed driver's facial image is input into the vehicle system that deploys a fast face detection algorithm based on deep learning for processing. The fast face detection algorithm is deployed in the vehicle system equipment, and the vehicle NPU model inference acceleration technology is used to enable the deep learning model trained on a high-performance server to be deployed in the vehicle system equipment with limited computing power and power.

[0146] The pre-processed image of the driver's face image is detected and analyzed using the fast face detection algorithm based on deep learning, and the left eye center coordinates (x1, y1), the right eye center coordinates (x2, y2) and the binocular distance L are output. The fast face detection algorithm based on deep learning uses the YOLOv8 model framework, which collects face data in actual application scenarios for training, obtains model weights, and deploys them to the vehicle system equipment.

[0147] The module for calculating the circumscribed rectangular frame is used to calculate the circumscribed rectangular frame of the left eye and the circumscribed rectangular frame of the right eye according to the center coordinates of the left eye, the center coordinates of the right eye and the distance between the two eyes.

[0148] A determination module, used for determining an eye rectangular frame according to a left eye circumscribed rectangular frame and a right eye circumscribed rectangular frame;

[0149] The second input module is used to input the eye rectangular frame into the pupil detection model to obtain the pupil area coordinates;

[0150] Specifically, the eye rectangular frame is normalized and then input into the constructed pupil detection model to output the pupil area coordinates.

[0151] The pupil detection model also uses the YOLOv8 model framework. In actual application scenarios, pupil data is collected for training to obtain model weights, and the pupil detection model is deployed to the vehicle-mounted system equipment.

[0152] An adaptive threshold calculation module, used for calculating an adaptive threshold according to pupil area coordinates;

[0153] A statistics module, used to calculate the average value of aspect ratio within a preset time period;

[0154] The comparison module is used to compare the aspect ratio mean with the adaptive threshold value and determine the current state of the driver according to the comparison result.

[0155] Specifically, the pupil area aspect ratio is used as a statistic to calculate the pupil aspect ratio z=h / w. The aspect ratio distribution in a preset time period is used to calculate the adaptive threshold for judging eye closure. The aspect ratio mean is compared with the adaptive threshold, and whether the driver is in a fatigue state is determined based on the comparison result.

[0156] Furthermore, the adaptive threshold calculation module is specifically used for:

[0157] Extracting the pupil area from the pupil area coordinates and outputting bounding box parameters; wherein the bounding box parameters include the center coordinates of the bounding box, the height of the bounding box, and the width of the bounding box;

[0158] Calculating the aspect ratio of the pupil according to the height of the bounding box and the width of the bounding box;

[0159] The aspect ratio of the pupil is used as a statistic;

[0160] Calculate an adaptive threshold based on a statistic.

[0161] Specifically, the pupil area is extracted from the pupil area coordinates, and the bounding box parameters are output; the bounding box parameter format is xywh, where xy represents the center coordinates of the bounding box (x, y), and wh represents the width and height of the bounding box (width, height). This format describes the position and size of the bounding box relative to the image, making the position and size information of the detected object more intuitive and convenient for subsequent processing.

[0162] Furthermore, the adaptive threshold calculation module is specifically used for:

[0163] When the driver is driving normally, the first t0 minutes of the initial driving trip are called and the aspect ratio of the pupil is calculated for each frame;

[0164] The aspect ratio of the pupil in each frame is averaged to obtain an adaptive threshold.

[0165] Specifically, the pupil area aspect ratio is used as a statistic to calculate the pupil aspect ratio z=h / w. The aspect ratio distribution within a preset time period is used to calculate the adaptive threshold for judging eye closure.

[0166] When the driver is driving normally, the first t0 minutes of the driving journey are called, the aspect ratio of the pupil of each frame is calculated and averaged to obtain the adaptive threshold z0=h0 / w0 suitable for different individual drivers.

[0167] Furthermore, the statistics module is specifically used for:

[0168] Detect the first t1 seconds at time t;

[0169] If within the first t1 seconds, the average of the aspect ratios for consecutive t2 seconds is calculated;

[0170] Among them, t is greater than t1, and t1 is greater than t2.

[0171] Specifically, it is detected whether the driver is fatigued at time t, and the previous t1 seconds at time t are called; if within the previous t1 seconds, there are t2 consecutive seconds of the average pupil aspect ratio z(t) being less than the adaptive threshold z0, it is judged that the driver is fatigued driving at this moment.

[0172] Furthermore, the comparison module is specifically used for:

[0173] When the average aspect ratio is less than the adaptive threshold, the driver is in a fatigue state;

[0174] When the aspect ratio mean is greater than or equal to the adaptive threshold, the driver is in a normal state.

[0175] Furthermore, before inputting the driver's facial image into the deep learning fast face detection algorithm, the system also includes:

[0176] A preprocessing module (not shown), used for performing image resizing on the driver's facial image to obtain a processed driver's facial image;

[0177] The normalization processing module (not shown) is used to normalize the processed driver's facial image to obtain a normalized driver's facial image. Image normalization refers to a process of performing a series of standard processing transformations on an image to transform it into a fixed standard form. The standard image is called a normalized image.

[0178] The embodiment of the present invention provides a driver fatigue status detection system, comprising: collecting a driver's facial image; inputting the driver's facial image into a deep learning fast face detection algorithm to obtain the left eye center coordinates, the right eye center coordinates and the binocular distance; calculating the left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame according to the left eye center coordinates, the right eye center coordinates and the binocular distance; determining the eye rectangular frame according to the left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame; inputting the eye rectangular frame into a pupil detection model to obtain pupil area coordinates; calculating an adaptive threshold according to the pupil area coordinates; calculating the average aspect ratio within a preset time period; comparing the average aspect ratio with the adaptive threshold, and determining the current state of the driver according to the comparison result; solving the problems of inaccurate fatigue detection and weak robustness caused by factors such as individual differences of drivers or complex lighting in the prior art.

[0179] Embodiment five:

[0180] Figure 7 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention.

[0181] Reference Figure 7 The electronic device 100 includes one or more processors 102, one or more storage devices 104, an input device 106, an output device 108, and an image acquisition device 110, and these components are interconnected via a bus system 112 and / or other forms of connection mechanisms (not shown). It should be noted that Figure 7 The components and structures of the electronic device 100 shown are merely exemplary and non-limiting. The electronic device may also have other components and structures as required.

[0182] The processor 102 can be implemented in at least one hardware form of a digital signal processor (DSP), a field programmable gate array (FPGA), and a programmable logic array (PLA). The processor 102 can be a central processing unit (CPU) or one or a combination of other forms of processing units with data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device 100 to perform desired functions.

[0183] The storage device 104 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may run the program instructions to implement the client functions (implemented by the processor) and / or other desired functions in the embodiments of the present invention described below. Various applications and various data, such as various data used and / or generated by the application, may also be stored in the computer-readable storage medium.

[0184] The input device 106 may be a device used by a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.

[0185] The output device 108 may output various information (eg, images or sounds) to the outside (eg, a user), and may include one or more of a display, a speaker, and the like.

[0186] The image acquisition device 110 can capture images desired by the user (eg, photos, videos, etc.), and store the captured images in the storage device 104 for use by other components.

[0187] An embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the driver fatigue status detection method provided in the above embodiment are implemented.

[0188] An embodiment of the present invention also provides a computer-readable medium having a non-volatile program code executable by a processor, wherein a computer program is stored on the computer-readable medium, and when the computer program is executed by the processor, the steps of the driver fatigue status detection method of the above embodiment are executed.

[0189] The computer program product provided in the embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be found in the method embodiment and will not be repeated here.

[0190] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0191] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0192] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0193] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0194] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for detecting driver fatigue status, characterized in that: The method comprises: Collecting a driver's facial image, inputting the driver's facial image into a multi-scale face detection algorithm of deep learning to obtain face positioning information under different lighting conditions; Inputting the driver's facial image into a fast face detection algorithm of deep learning to obtain the left eye center coordinates, the right eye center coordinates and the binocular distance; Calculate a left eye circumscribed rectangular frame and a right eye circumscribed rectangular frame according to the left eye center coordinates, the right eye center coordinates and the binocular distance; Determine an eye rectangular frame according to the left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame; Input the eye rectangular frame into the pupil detection model to obtain the pupil area coordinates; Calculating an adaptive threshold according to the pupil area coordinates; Calculate the average value of aspect ratio within a preset time period; Comparing the aspect ratio mean with the adaptive threshold, and determining the current state of the driver according to the comparison result; wherein the fast face detection algorithm adopts the YOLOv8 model framework; Calculate the average aspect ratio within a preset time period, including: Detect the first t1 seconds at time t; If within the first t1 seconds, the average of the aspect ratios for consecutive t2 seconds is calculated; Wherein, the t is greater than the t1, and the t1 is greater than t2.

2. The method for detecting driver fatigue state according to claim 1, characterized in that: Calculating an adaptive threshold according to the pupil area coordinates includes: Extracting a pupil area from the pupil area coordinates and outputting bounding box parameters; wherein the bounding box parameters include the center coordinates of the bounding box, the height of the bounding box, and the width of the bounding box; Calculating the aspect ratio of the pupil according to the height of the bounding box and the width of the bounding box; Taking the aspect ratio of the pupil as a statistic; The adaptive threshold is calculated according to the statistic.

3. The method for detecting driver fatigue state according to claim 2, characterized in that: Calculating the adaptive threshold according to the statistic includes: When the driver is driving normally, the first t0 minutes of the initial driving journey are called, and the aspect ratio of the pupil of each frame is calculated; The aspect ratio of the pupil in each frame is averaged to obtain the adaptive threshold.

4. The method for detecting driver fatigue state according to claim 1, characterized in that: The aspect ratio mean is compared with the adaptive threshold, and the current state of the driver is determined according to the comparison result, including: When the aspect ratio mean is less than the adaptive threshold, the driver is in a fatigue state; When the aspect ratio average is greater than or equal to the adaptive threshold, the driver is in a normal state.

5. The method for detecting driver fatigue state according to claim 1, characterized in that: Before inputting the driver's facial image into the deep learning fast face detection algorithm, the method further includes: Performing image resizing on the driver's facial image to obtain a processed driver's facial image; The processed driver's facial image is normalized to obtain a normalized driver's facial image.

6. A driver fatigue state detection system, characterized in that: The system comprises: An acquisition module is used to acquire a driver's facial image, and input the driver's facial image into a multi-scale face detection algorithm of deep learning to obtain face location information under different lighting conditions; A first input module, used for inputting the driver's facial image into a fast face detection algorithm of deep learning to obtain the left eye center coordinates, the right eye center coordinates and the binocular distance; A circumscribed rectangular frame calculation module, used for calculating a left eye circumscribed rectangular frame and a right eye circumscribed rectangular frame according to the left eye center coordinates, the right eye center coordinates and the binocular distance; A determination module, configured to determine an eye rectangular frame according to the left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame; A second input module is used to input the eye rectangular frame into the pupil detection model to obtain pupil area coordinates; An adaptive threshold calculation module, used for calculating an adaptive threshold according to the pupil area coordinates; A statistics module, used to calculate the average value of aspect ratio within a preset time period; A comparison module, used for comparing the aspect ratio mean with the adaptive threshold, and determining the current state of the driver according to the comparison result; wherein the fast face detection algorithm adopts the YOLOv8 model framework; The statistical module is specifically used for: Detect the first t1 seconds at time t; If within the first t1 seconds, the average of the aspect ratios for consecutive t2 seconds is calculated; Wherein, the t is greater than the t1, and the t1 is greater than t2.

7. The driver fatigue state detection system according to claim 6, characterized in that: The adaptive threshold calculation module is specifically used for: Extracting a pupil area from the pupil area coordinates and outputting bounding box parameters; wherein the bounding box parameters include the center coordinates of the bounding box, the height of the bounding box, and the width of the bounding box; Calculating the aspect ratio of the pupil according to the height of the bounding box and the width of the bounding box; Taking the aspect ratio of the pupil as a statistic; The adaptive threshold is calculated according to the statistic.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the driver fatigue state detection method described in any one of claims 1 to 5 is implemented.

9. A computer-readable medium having a non-volatile program code executable by a processor, characterized in that: The program code enables the processor to execute the driver fatigue state detection method as described in any one of claims 1 to 5.

Images