Fatigue driving detection method, detection system, detection device and computer-readable storage medium

Abstract

The present invention provides a fatigue driving detection system, which relates to the field of intelligent vehicle control. The detection method includes: acquiring real-time facial image data of a driver, the facial image data including multiple eye feature points; determining the driver's eye opening and blinking frequency based on the spacing between the eye feature points; determining a frequency threshold based on the eye opening, and determining that the driver is fatigued when the eye opening is less than the opening threshold and the blinking frequency is greater than the frequency threshold, wherein the frequency threshold is positively correlated with the eye opening; and outputting a control signal when the driver is determined to be fatigued. This detection system can improve the accuracy of fatigue driving detection. The present invention also provides a fatigue driving detection system, a detection device, and a computer-readable storage medium.

Description

Technical Field

[0001] The present invention relates to the field of vehicle intelligent control, and in particular to a fatigue driving detection method, a detection system, a detection device and a computer-readable storage medium. Background Art

[0002] Driver fatigue driving is one of the factors that cause traffic accidents. A fatigue driving detection system is needed to identify the driver's fatigue state. The relevant fatigue detection system uses facial information such as the driver's head posture and eye posture to identify fatigue. This fatigue detection system has insufficient recognition accuracy and may miss recognition. Summary of the Invention

[0003] The present invention provides a fatigue driving detection method, a detection system, a detection device and a computer-readable storage medium to solve the problem of how to improve the recognition accuracy of fatigue driving and reduce the possibility of missed recognition.

[0004] An embodiment of the present invention provides a method for detecting fatigue driving, which includes: acquiring real-time facial image data of a driver, the facial image data including multiple eye feature points; determining the driver's eye opening and the driver's blinking frequency based on the spacing between each of the eye feature points; determining a frequency threshold based on the eye opening, and determining that the driver is in a fatigue state when the eye opening is less than the opening threshold and the blinking frequency is greater than the frequency threshold, wherein the frequency threshold is positively correlated with the eye opening; and outputting a control signal when it is determined that the driver is in a fatigue state.

[0005] Furthermore, the opening threshold includes a first opening threshold and a second opening threshold, and the first opening threshold is greater than the second opening threshold; when the eye opening is less than the opening threshold and the blinking frequency is greater than the frequency threshold, determining that the driver is in a fatigue state includes: when the eye opening is greater than the first opening threshold and the blinking frequency is greater than the frequency threshold, determining that the driver is in a mild fatigue state; when the eye opening is less than the first opening threshold and greater than the second opening threshold, and the blinking frequency is greater than the frequency threshold, determining that the driver is in a moderate fatigue state; when the eye opening is less than the second threshold and the duration of the eye opening being less than the second threshold is greater than a preset duration, determining that the driver is in a severe fatigue state.

[0006] Furthermore, before obtaining the real-time facial image data of the driver, the detection method also includes: determining the blinking opening based on the facial image data of the driver; determining the blinking opening as the second threshold; determining the eye opening of the driver and the blinking frequency of the driver based on the spacing between each of the eye feature points, including: when the eye opening of the driver is less than the blinking opening, determining that the driver blinked once, and determining the number of times the driver blinks within a preset time period as the blinking frequency.

[0007] Furthermore, before obtaining the real-time facial image data of the driver, the detection method also includes: inputting the facial image data of the driver into a trained neural network model to obtain the blinking opening; determining the eye opening of the driver and the blinking frequency of the driver based on the spacing between each of the eye feature points, including: when the eye opening of the driver is less than the blinking opening, determining that the driver blinked once, and determining the number of blinks of the driver within a preset time period as the blinking frequency.

[0008] Furthermore, determining the eye opening of the driver based on the spacing between each of the eye feature points includes: determining the length dimension of the eye and the width dimension of the eye based on the eye feature points, and obtaining the eye width-to-length ratio based on the width dimension and the length dimension, and determining the eye width-to-length ratio as the eye opening.

[0009] Furthermore, the outputting of control signals when it is determined that the driver is in a fatigue state includes: when it is determined that the driver is in the mild fatigue state, outputting a first control signal to output a prompt signal in the vehicle cabin; when it is determined that the driver is in the moderate fatigue state, outputting a second control signal to limit the maximum speed of the vehicle; when it is determined that the driver is in the severe fatigue state, outputting a third control signal to cause the vehicle to continuously sound the horn and keep the hazard warning lights on.

[0010] Furthermore, after outputting a first control signal to output a prompt signal in the vehicle cabin upon determining that the driver is in the mild fatigue state, the detection method also includes: stopping outputting the prompt signal when a stop command triggered by the driver is detected; multiplying the first opening threshold by a preset correction coefficient to obtain a corrected first opening threshold, and updating the corrected opening threshold to the first opening threshold, wherein the correction coefficient is less than 1.

[0011] An embodiment of the present invention also provides a fatigue driving detection system, which includes: an acquisition module for acquiring real-time facial image data of the driver, the facial image data including multiple eye feature points; an eye processing module for determining the eye opening and the blinking frequency of the driver based on the spacing between each of the eye feature points; the fatigue recognition module for determining a frequency threshold based on the eye opening, the frequency threshold being positively correlated with the eye opening, and determining that the driver is in a fatigue state when the eye opening is less than the opening threshold and the blinking frequency is greater than the frequency threshold, wherein the frequency threshold is positively correlated with the eye opening; and an output module for outputting a control signal when it is determined that the driver is in a fatigue state.

[0012] Furthermore, the opening threshold includes a first opening threshold and a second opening threshold, and the first opening threshold is greater than the second opening threshold. The fatigue recognition module is also used to determine that the driver is in a mild fatigue state when the eye opening is greater than the first opening threshold and the blinking frequency is greater than the frequency threshold; to determine that the driver is in a moderate fatigue state when the eye opening is less than the first opening threshold and greater than the second opening threshold, and the blinking frequency is greater than the frequency threshold; and to determine that the driver is in a severe fatigue state when the eye opening is less than the second threshold and the duration of the eye opening being less than the second threshold is greater than a preset duration.

[0013] Furthermore, the detection system also includes a blink training model configured to input the driver's facial image data into a trained neural network model to determine a blink width. The eye processing module is further configured to determine that the driver has blinked once when the driver's eye width is less than the blink width, and to determine the number of blinks by the driver within a preset time period as the blink frequency.

[0014] Furthermore, the eye processing module is further configured to determine the blink opening degree as the second opening degree threshold.

[0015] Furthermore, the eye processing module is also used to determine the length dimension and the width dimension of the eye based on the eye feature points, and obtain the eye width-to-length ratio based on the width dimension and the length dimension, and determine the eye width-to-length ratio as the eye opening.

[0016] Furthermore, the output module is also used to output a first control signal to output a prompt signal in the vehicle cabin when it is determined that the driver is in the mild fatigue state; output a second control signal to limit the maximum speed of the vehicle when it is determined that the driver is in the moderate fatigue state; and output a third control signal to make the vehicle continue to sound the horn and keep the hazard lights on when it is determined that the driver is in the severe fatigue state.

[0017] An embodiment of the present invention also provides a fatigue driving detection device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the fatigue driving detection method provided in the above embodiment is implemented.

[0018] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the fatigue driving detection method provided in the above embodiment.

[0019] An embodiment of the present invention provides a method for detecting fatigue driving, which includes acquiring real-time facial image data of a driver, wherein the facial image data includes a plurality of eye feature points; determining the driver's eye opening and blinking frequency based on the spacing between the eye feature points; determining a frequency threshold based on the eye opening, and determining that the driver is in a fatigue state when the eye opening is less than the opening threshold and the blinking frequency is greater than the frequency threshold, wherein the frequency threshold is positively correlated with the eye opening; and outputting a control signal when it is determined that the driver is in a fatigue state. The driver's fatigue state is comprehensively determined by the eye opening and the blinking frequency, and is determined to be in a fatigue state when the eye opening is less than the opening threshold and the blinking frequency is greater than the frequency threshold, wherein the frequency threshold is positively correlated with the eye opening. Thus, even when the driver cannot open his eyes and the blinking frequency decreases, the driver's fatigue state can be identified, thereby improving the accuracy of fatigue driving and reducing the possibility of missed identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a method for detecting fatigue driving provided by an embodiment of the present invention;

[0021] Figure 2 A schematic flow chart of another method for detecting fatigue driving provided by an embodiment of the present invention;

[0022] Figure 3 A schematic flow chart of another method for detecting fatigue driving provided by an embodiment of the present invention;

[0023] Figure 4A schematic flow chart of another method for detecting fatigue driving provided by an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of the position distribution of eye feature points on the eye provided by an embodiment of the present invention;

[0025] Figure 6 A schematic flow chart of another method for detecting fatigue driving provided by an embodiment of the present invention;

[0026] Figure 7 A schematic structural diagram of a fatigue driving detection system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0028] The various specific technical features described in the specific embodiments may be combined in any suitable manner, unless they are inconsistent. For example, different embodiments and technical solutions may be formed by combining different specific technical features. To avoid unnecessary repetition, the various possible combinations of the specific technical features in the present invention will not be described separately.

[0029] In the following description, the terms "first, second, ..." are used solely to distinguish different objects and do not imply any similarities or connections between the objects. It should be understood that the directional descriptions "above," "below," "outside," and "inside" refer to directions during normal use. The directions "left" and "right" refer to the left-right directions shown in the corresponding schematic diagrams, which may or may not be the left-right directions during normal use.

[0030] It should be noted that the terms "include," "comprises," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising that element. The term "connected," unless otherwise specified, includes both direct and indirect connections.

[0031] It should be noted that the fatigue driving detection method in the following specific implementation manner can be applied to any type of vehicle. For example, the vehicle can be a car, and it can also be applied to a truck. For the sake of convenience, the following is an example of the application of the detection method to a car, and the specific steps of the detection system are explained by way of example.

[0032] In some embodiments, as Figure 1 As shown, Figure 1 A flowchart of a method for detecting fatigue driving provided by an embodiment of the present invention includes the following steps:

[0033] Step S101: Acquire real-time facial image data of the driver.

[0034] It can be understood that the image acquisition device installed in the cockpit is called by a control signal to obtain real-time facial image data of the driver. The image acquisition device can be, for example, a camera or a video camera. The type of facial image data obtained varies depending on the type of image acquisition device. For example, the camera periodically captures the driver's face, and the facial image data obtained is multiple pictures of the driver's face arranged in chronological order. For example, the video camera continuously captures the driver's face, and the facial image data obtained is video data of the driver's face, that is, the driver's facial image in each frame arranged in chronological order. The facial image data includes multiple eye feature points. The eye feature points can be feature points extracted from the facial image data through image processing that can reflect the degree of openness of the driver's eyes. It can be understood that the feature points are multiple points arranged at intervals around the edge of the space enclosed by the driver's eye sockets and eyelids. It should be noted that the eye feature points can be obtained in any way. For example, the feature points can be determined according to the preset position of the eye outer contour after the outer contour of the driver's eyes is obtained through image recognition; for example, the feature points can also be obtained through neural network model training. For example, the facial image of the driver with the required feature points that have been calibrated is input into the neural network model for training until the distance between the marked position of the feature point output by the neural network model and the marked position of the calibrated feature point is less than the preset threshold, thereby obtaining a trained neural network model. The driver's facial graphic data obtained in real time is input into the trained neural network model, and the neural network model can output a facial image with eye feature points.

[0035] Step S102: determining the eye opening degree and blinking frequency of the driver based on the distance between the eye feature points.

[0036] For the sake of convenience, the line connecting the two corners of the eye will be called the length direction of the eye, and the blinking direction of the eye will be called the width direction of the eye. The eye opening can be understood as the distance from the eyelid to the lower eye socket in the width direction. The larger the eye opening, the smaller the area of ​​the eyeball covered by the eyelid.

[0037] Among them, determining the driver's eye opening by the spacing between each eye feature point can be achieved in a variety of ways. Depending on the arrangement of the feature points, the method of determining the driver's eye opening by the spacing between the feature points is also different. For example, the feature points are a first point located at the highest position of the outer contour of the eyelid and a second point located at the lowest position of the outer contour of the lower eye socket, and the spacing between the first point and the second point is the eye opening; for example, the feature points are multiple pairs of feature points located at the outer contour of the eye socket, and each pair of feature points is spaced apart along the length direction of the eye, and in the width direction of the eye, two feature points in a pair of feature points are located at the lower edge of the eyelid and the upper edge of the lower eye socket, respectively, and the two feature points are arranged in a targeted manner in the width direction, and the average value of the spacing between each pair of feature points is the eye opening.

[0038] At the same time, determining the driver's blinking frequency based on the spacing of each eye feature point can be achieved in a variety of ways. For example, the eye opening is determined by the spacing of each eye feature point, and a blink is determined when the eye opening is less than a preset value, and the number of blinks within a preset time period is determined as the blinking frequency; for example, the eye opening is determined by the spacing of each eye feature point, and whether the driver blinks is determined by the change pattern of the eye opening in the real-time facial image data of the driver arranged in chronological order, for example, the dividing point where the eye opening trend changes from an increasing trend to a decreasing trend is used as a periodic dividing point, and a blink is determined each time the periodic dividing point is detected, and the number of blinks within a preset time period is determined as the blinking frequency.

[0039] Step S103: determining a frequency threshold based on the eye opening degree, and determining that the driver is in a fatigue state when the eye opening degree is less than the opening threshold and the blinking frequency is greater than the frequency threshold.

[0040] Among them, the frequency threshold is positively correlated with the eye opening. It can be understood that the driver's fatigue state is determined comprehensively by the eye opening and the blinking frequency. That is, when the eye opening is less than the opening threshold, the frequency threshold is determined according to the eye opening. The smaller the eye opening, the lower the blinking frequency threshold for judging that the driver is in a fatigue state, thereby more accurately identifying the driver's fatigue state. The following is an illustrative explanation of the principle that the accuracy of fatigue identification can be improved by eye opening and blinking frequency, combined with the law of eye changes when the driver is in a fatigue state.

[0041] When the driver begins to feel tired, the driver will start to blink frequently, but as the degree of fatigue continues to deepen, the driver will be unable to open his eyes. At this time, the eye opening decreases but the blinking frequency begins to decrease. If the driver's fatigue state is detected only by the eye opening, a driver who is not in a fatigued state may be mistakenly identified as being in a fatigued state due to the driver's frowning, squinting and other expressions; if the driver's fatigue state is detected only by the blinking frequency, then when the driver's fatigue level continues to deepen and he begins to be unable to open his eyes, the driver's blinking frequency decreases, and the driver who is in a fatigued state may be mistakenly identified as not being in a fatigued state due to the driver's blinking frequency decreasing; the driver's fatigue state is determined comprehensively by the eye opening and the blinking frequency. When the eye opening is less than the threshold, the frequency threshold is determined according to the eye opening. The smaller the eye opening, the smaller the determined frequency threshold. Therefore, even when the driver is in a fatigued state where he cannot open his eyes and the blinking frequency decreases, the driver's fatigue state can be accurately identified.

[0042] Among them, the opening threshold can be determined in a variety of ways. For example, the opening threshold can be a preset vertical. For example, the opening threshold can also be obtained through neural network model training. For example, a picture of a driver marked with fatigue status is input into the neural network model, and the neural network model outputs the eye opening threshold and fatigue status. When the fatigue status output by the neural network model for multiple consecutive times is consistent with the marked fatigue status, it is determined that the neural network model has completed training. During use, the driver's facial image is input into the trained neural network model in real time or the driver's facial image is input into the trained neural network model in advance, and the neural network model can output the opening threshold.

[0043] Step S104: output a control signal when it is determined that the driver is in a fatigue state.

[0044] Among them, the control signal is used to control the vehicle to output a prompt signal or control the vehicle to reduce the hazards that may be caused by fatigue driving. For example, the control signal is used to control the vehicle to output a prompt voice in the cockpit to prompt the driver to find a nearby service area to rest; for example, the control signal is used to control the vehicle to limit the maximum speed of the vehicle, thereby providing a longer reaction time for the driver in a fatigued state, and if a traffic accident occurs, it can also reduce the impact of the traffic accident.

[0045] An embodiment of the present invention provides a method for detecting fatigue driving, which includes acquiring real-time facial image data of a driver, wherein the facial image data includes a plurality of eye feature points; determining the driver's eye opening and blinking frequency based on the spacing between the eye feature points; determining a frequency threshold based on the eye opening, and determining that the driver is in a fatigue state when the eye opening is less than the opening threshold and the blinking frequency is greater than the frequency threshold, wherein the frequency threshold is positively correlated with the eye opening; and outputting a control signal when it is determined that the driver is in a fatigue state. The driver's fatigue state is comprehensively determined by the eye opening and the blinking frequency, and is determined to be in a fatigue state when the eye opening is less than the opening threshold and the blinking frequency is greater than the frequency threshold, wherein the frequency threshold is positively correlated with the eye opening. Thus, even when the driver cannot open his eyes and the blinking frequency decreases, the driver's fatigue state can be identified, thereby improving the accuracy of fatigue driving and reducing the possibility of missed identification.

[0046] In some embodiments, the opening threshold includes a first threshold and a second threshold, and the first threshold is greater than the second threshold. Figure 2 As shown, Figure 2 A flowchart of another method for detecting fatigue driving provided by an embodiment of the present invention is provided. Figure 1 The difference between the detection methods shown is that Figure 1 Step S103 in the embodiment includes:

[0047] Step S201: When the eye opening is greater than the first opening threshold and the blinking frequency is greater than the frequency threshold, it is determined that the driver is in a mild fatigue state; when the eye opening is less than the first threshold and greater than the second threshold, and the blinking frequency is greater than the frequency threshold, it is determined that the driver is in a moderate fatigue state; when the eye opening is less than the second threshold and the duration of the eye opening being less than the second threshold is greater than a preset duration, it is determined that the driver is in a moderate fatigue state.

[0048] It can be understood that by dividing the eye opening threshold into a first threshold and a second threshold, the fatigue degree can be divided more finely through different eye openings. While the fatigue state can be more accurately identified through eye opening and blinking frequency, the fatigue state recognition accuracy can be improved through different opening thresholds, that is, it can be identified whether the driver is in a mild fatigue state or a moderate fatigue state; at the same time, when the eye opening is less than the second threshold, the fatigue state is identified by the duration of the eye opening less than the second threshold rather than the blinking frequency. This is because it is difficult to identify blinking and difficult to accurately identify multiple consecutive blinks when the eye opening is very small. Therefore, when the eye opening is very small, it is determined whether the driver is in a state of long-term eye closure by the duration of the small opening, and then the driver's moderate fatigue state is identified with less recognition difficulty.

[0049] Optionally, when different degrees of fatigue are identified, different types of control signals corresponding to the fatigue levels may be output, for example, Figure 2 As shown, Figure 1 Step S104 in the embodiment includes:

[0050] Step S301: When it is determined that the driver is in a state of mild fatigue, a first control signal is output to output a prompt signal in the vehicle cabin; when it is determined that the driver is in a state of moderate fatigue, a second control signal is output to limit the maximum speed of the vehicle; when it is determined that the driver is in a state of severe fatigue, a third control signal is output to make the vehicle continue to sound the horn and keep the double flash warning lights on.

[0051] It can be understood that, upon recognizing that the driver is in a state of mild fatigue, the first control signal is used to control the vehicle to send a prompt signal in the cabin to alert the driver. The prompt signal can be a prompt voice played in the cabin, or a prompt image or text displayed on the central control display screen, that is, the driver is reminded to take a rest through non-compulsory control means; when the driver is in a state of moderate fatigue, the second control signal is used to limit the maximum speed of the vehicle, thereby increasing the driver's reaction time when the driver is in a state of moderate fatigue, that is, the driver's reaction speed will slow down when the driver is in a state of moderate fatigue. Limiting the maximum speed can increase the driver's reaction time, so that the driver has enough decision-making time when the driver encounters a situation where he needs to make a traffic decision, thereby reducing the possibility of traffic accidents due to fatigue; when the driver is in a state of severe fatigue, the third control signal is used to make the vehicle horn continue to honk and the vehicle's double flash warning lights remain on, that is, when the driver is in a state of severe fatigue, it is difficult to ensure traffic safety only through the driver's self-adjustment, so the vehicle horn and double flash warning lights are used to remind surrounding vehicles to pay attention and avoid. In some embodiments, such as Figure 3 As shown, Figure 3 A flow chart of another method for detecting fatigue driving provided by an embodiment of the present invention, based on Figure 1 The detection method shown, Figure 3 The detection method shown is the same as Figure 1 The difference between the processes shown is that Figure 1 Before step S101, the fatigue driving detection method further includes:

[0052] Step S401: Determine the blink degree based on the driver's facial image data.

[0053] It can be understood that the blinking aperture is determined by obtaining the eye opening degree of the driver in the blinking state through the driver's facial image. It should be noted that the blinking aperture can be determined based on the driver's facial image data according to any method. Two methods of determining the blinking aperture through the driver's facial data image are given below. Those skilled in the art should understand that the method of determining the blinking aperture based on the driver's facial image data can also be other methods besides the following two methods.

[0054] A first method for determining the eye opening degree based on the driver's facial image data includes: obtaining facial image data of the driver in a blinking state, and facial image data of the driver near the time point of the blinking state, for example, facial image data of the driver in a time period from 0.5 seconds before the blinking state to 0.5 seconds after the blinking state, and determining the average value of the eye opening degree in the driver's facial image data in the time period as the blink opening degree.

[0055] A second method for determining the blinking degree based on facial image data of a driver includes: determining whether the driver blinks by means of the eye opening degree, and determining a threshold value of the eye opening degree for whether the driver blinks, that is, the blinking degree is obtained by a trained neural network model. Specifically, a calibrated facial image of the driver is input into the neural network model to train the neural network model, wherein the calibrated facial image includes the facial image of the driver and whether the driver is in a blinking state. The neural network model outputs the blinking degree after computational processing and determines whether the driver is in a blinking state based on the blinking degree. Parameters of each node of the neural network model are adjusted according to the correctness of the judgment result. The above input and adjustment process is repeated multiple times until the number of consecutive times that the blinking judgment result output by the neural network model matches the marked blinking state reaches a preset number threshold, and the training of the neural network model is considered complete. During use, the facial data of the driver to be identified is input into the trained neural network model in real time, or the pre-stored facial data of the driver is input into the trained neural network model, thereby obtaining the blinking degree.

[0056] Optionally, after obtaining the blink opening, the blink opening is determined as the second opening threshold, that is, the blink opening is the degree of opening of the driver's eyes reduced to a certain extent. After the eye opening is reduced to this extent, it can be considered that the driver has reached the state of closing his eyes. The blink opening output by the neural network is determined as the second opening threshold, which can more accurately identify the driver's closed eyes state and blinking state, and obtain two thresholds at the same time through one step, reducing the time required for calculation.

[0057] at the same time, Figure 1 Determining the driver's blinking frequency based on the distance between each eye feature point in step S102 includes:

[0058] Step S402: When the driver's eye opening is less than the blink opening, it is determined that the driver has blinked once, and the number of blinks of the driver within a preset time period is determined as the blink frequency.

[0059] It can be understood that the blinking frequency obtained by the blinking training model is used as a benchmark for judging whether the driver blinks, the driver's eye opening is obtained by the spacing of the driver's eye feature points, and when the driver's eye opening is less than the blink opening, it is determined that the driver blinks once, and the number of times the driver blinks within a preset time length is determined as the blinking frequency. For example, the preset time length can be one minute.

[0060] In some embodiments, as Figure 4 As shown, Figure 4 A flow chart of another method for detecting fatigue driving provided by an embodiment of the present invention, based on Figure 1 The detection method shown, Figure 4 The detection method shown is the same as Figure 1 The difference between the detection methods shown is that Figure 1 The step S102 shown in FIG. 102 , determining the driver's eye opening degree based on the distance between each eye feature point, includes:

[0061] Step S501: Determine the length and width of the eye based on the eye feature points, obtain the eye width-to-length ratio based on the width and length, and determine the eye width-to-length ratio as the eye opening.

[0062] Among them, the width dimension of the eye is the distance from the lower edge of the eyelid to the lower eye socket, and the length dimension is the distance between the two corners of the eye. Determining the ratio of the width dimension to the length dimension of the eye as the driver's eye opening can reduce the impact of the driver's eye size on fatigue identification. For example, using the ratio of the width dimension to the length dimension of the eye as the eye opening can reduce the possibility of drivers with very small eyes in a natural state being misidentified as fatigue driving. At the same time, when the width-to-length ratio of the eye is determined as the eye opening, the opening threshold is also reflected in the form of the width-to-length ratio of the eye, so that the fatigue state is identified by the width-to-length ratio. It should be noted that the method of calculating the width-to-length ratio of the eye is different according to the distribution and number of eye feature points. The following is an example of the method of calculating the width-to-length ratio of the eye based on six eye feature points. Figure 5 As shown, six eye feature points P1 to P6 are spaced around the outer contour of the eye, where eye feature point P1 and eye feature point P4 are located at the two corners of the eye, eye feature point P6 is located directly below eye feature point P2, and eye feature point P5 is located directly below eye feature point P3. After the eye feature points are identified, the coordinates of each eye feature point in the preset coordinate system are recorded. The width-to-length ratio of the eye is calculated according to the following formula:

[0063]

[0064] In the above formula, EAR represents the aspect ratio of the eye, ||P2-P6|| is the Euclidean norm of the difference between the coordinates of the eye feature point P2 and the coordinates of the eye feature point P6. For example, the coordinates of the eye feature point P2 are (x2, y2), and the coordinates of the eye feature point P6 are (x6, y6). x2 is the abscissa value of the eye feature point P2, x6 is the abscissa value of the eye feature point P6, y2 is the ordinate value of the eye feature point P2, and y6 is the ordinate value of the eye feature point P6. ||P2-P6|| can be expressed as: It can be understood that ||P2-P6|| reflects the distance between the eye feature point P2 and the eye feature point P6. Similarly, ||P3-P5|| is the Euclidean norm of the difference between the coordinates of the eye feature point P3 and the coordinates of the eye feature point P5, which reflects the distance between the eye feature point P3 and the eye feature point P5; ||P1-P4|| is the Euclidean norm of the difference between the coordinates of the eye feature point P1 and the coordinates of the eye feature point P4, which reflects the distance between the eye feature point P1 and the eye feature point P4.

[0065] Optionally, when the opening threshold includes a first opening threshold and a second opening threshold, the first opening threshold and the second opening threshold are both expressed in the form of the aspect ratio of the eye, that is, when a more precise judgment of the opening of the eye is required, the opening of the eye is determined by the aspect ratio of the eye, which can more accurately distinguish the openings of different eyes, thereby more accurately identifying the degree of fatigue of the driver.

[0066] Optionally, when determining whether the driver blinks by using a blink threshold, Figure 3 The blinking degree obtained in step S401 is also expressed as the eye's width-to-length ratio. That is, when blinking is identified using eye opening, it is necessary to identify blinking even when the eye opening is small. Using the eye's width-to-length ratio as the eye opening for identification allows for more accurate identification and differentiation of driver blinking even when the eye opening is small, thereby making blink identification more accurate. Simultaneously, the driver's fatigue state is identified using blink frequency, and the frequency threshold is positively correlated with eye opening. When the eye opening is small, the frequency threshold is smaller. When the eye opening is small, misidentification of blinks within a preset duration has a greater impact on fatigue level identification results, thus requiring more accurate blink identification. By expressing the blink threshold as the eye's width-to-length ratio and using the eye's width-to-length ratio as the eye opening, driver blinking can be more accurately identified, thereby improving the accuracy of driver fatigue identification.

[0067] In some embodiments, as Figure 6As shown, Figure 6 A flow chart of another method for detecting fatigue driving provided by an embodiment of the present invention, based on Figure 2 The detection method shown, Figure 6 The detection method shown is the same as Figure 2 The difference between the detection methods shown is that Figure 2 After step S301, the detection method further includes:

[0068] S601 : When a stop command triggered by the driver is detected, stop outputting the prompt signal.

[0069] It can be understood that if the driver is misidentified as being in a mild fatigue state, the driver can trigger a stop command to stop the prompt signal output in the vehicle cabin, thereby reducing the interference of the prompt signal in the misidentified state on the driver's normal driving. The driver can trigger the stop command in various ways. For example, the driver can trigger the stop command by a button set on the control panel. For example, the driver can also trigger the stop command by voice. Optionally, the step of the driver triggering the stop command through operation can include: upon receiving the driver-triggered stop request instruction, outputting a preset question in the vehicle cabin via voice. The preset question can be any question that can be answered quickly and has a clear answer, such as the driver's name, the driver's birthday, or a mathematical operation within two digits; if the correct answer is received from the driver via voice within a preset time period, the stop command is triggered, and the prompt signal output is stopped in response to the stop command. If the correct answer is not received from the driver via voice within the preset time period, or the answer sent by the driver via voice is incorrect, the stop command is not triggered, and the prompt signal continues to be output. It should be noted that the driver is only allowed to turn off the prompt signal based on his or her actual fatigue situation when it is identified that the driver is in a state of mild fatigue. If the driver is identified as being in a state of moderate fatigue or severe fatigue, the driver is not allowed to turn off the corresponding anti-fatigue driving measures to prevent the driver in a state of moderate fatigue or severe fatigue from forcibly insisting on fatigue driving.

[0070] Step S602: multiply the first opening threshold by a preset correction coefficient to obtain a corrected first opening threshold, and update the corrected opening threshold to the first opening threshold.

[0071] Among them, the correction coefficient is less than 1, that is, when the driver is misidentified as being slightly fatigued, the first opening threshold is corrected, and the first opening threshold is reduced by the correction coefficient to reduce the possibility of the driver being misidentified as being slightly fatigued during subsequent use.

[0072] The following continues to describe the exemplary structure of the fatigue driving detection device provided by the embodiment of the present invention implemented as a software module, that is, the structure of the fatigue driving detection system is described exemplarily. In some embodiments, such as Figure 7 As shown, the detection system includes: an acquisition module 100 , an eye processing module 200 , a fatigue recognition module 300 and an output module 400 .

[0073] The acquisition module 100 is used to acquire real-time facial image data of the driver, wherein the facial image data includes a plurality of eye feature points.

[0074] The eye processing module 200 is configured to determine the driver's eye opening and blinking frequency based on the distance between each eye feature point.

[0075] a fatigue identification module 300 for determining a frequency threshold based on eye opening, and determining that the driver is in a fatigue state when the eye opening is less than the opening threshold and the blinking frequency is greater than the frequency threshold, wherein the frequency threshold is positively correlated with the eye opening;

[0076] The output module 400 is configured to output a control signal when it is determined that the driver is in a fatigue state.

[0077] In some embodiments, the opening threshold includes a first opening threshold and a second opening threshold, and the first opening threshold is greater than the second opening threshold. Figure 7 As shown, the fatigue recognition module 300 is also used to determine that the driver is in a mild fatigue state when the eye opening is greater than a first opening threshold and the blinking frequency is greater than a frequency threshold; to determine that the driver is in a moderate fatigue state when the eye opening is less than the first opening threshold and greater than the second opening threshold, and the blinking frequency is greater than the frequency threshold; and to determine that the driver is in a severe fatigue state when the eye opening is less than the second threshold and the duration of the eye opening being less than the second threshold is greater than a preset duration.

[0078] In some embodiments, as Figure 7 As shown, the detection system also includes a blink training module 500 for inputting the driver's facial image data into a trained neural network model to determine the blink opening. Furthermore, the eye processing module 200 is further configured to determine that the driver has blinked once when the driver's eye opening is less than the blink opening, and to determine the number of blinks within a preset time period as the blink frequency.

[0079] In some embodiments, as Figure 7 As shown, the blink training module 500 is further configured to determine the blink opening degree as the second opening degree threshold.

[0080] In some embodiments, as Figure 7As shown, the eye processing module 200 is further used to determine the length and width of the eye based on the eye feature points, obtain the eye width-to-length ratio based on the width and length dimensions, and determine the eye width-to-length ratio as the eye opening.

[0081] In some embodiments, as Figure 7 As shown, the output module 400 is also used to output a first control signal to output a prompt signal in the vehicle cabin when it is determined that the driver is in a light fatigue state; output a second control signal to limit the maximum speed of the vehicle when it is determined that the driver is in a moderate fatigue state; and output a third control signal to make the vehicle continue to sound the horn and keep the double flash warning lights on when it is determined that the driver is in a severe fatigue state.

[0082] The following is an exemplary description of the embodiment of the present invention, wherein the fatigue driving detection device is implemented as a hardware structure. The hardware structure may be a vehicle computer or a fatigue detection device installed in the vehicle cabin. For example, the detection device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following can be achieved: Figures 1 to 6 A method for identifying fatigue driving as shown in any one of the figures.

[0083] The embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes Figures 1 to 6 A method for identifying fatigue driving as shown in any one of the figures.

[0084] In some embodiments, the storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EE PROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0085] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0086] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0087] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0088] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A method for detecting fatigue driving, characterized in that: The detection method comprises: Acquiring real-time facial image data of the driver, wherein the facial image data includes a plurality of eye feature points; determining the eye opening of the driver and the blinking frequency of the driver based on the distance between the eye feature points; determining a frequency threshold based on the eye opening, and determining the fatigue state of the driver based on the eye opening, the opening threshold, the blinking frequency, and the frequency threshold, wherein the frequency threshold is positively correlated with the eye opening; and the opening threshold includes a first opening threshold; When it is determined that the driver is in a mild fatigue state, a first control signal is output to output a prompt signal in the vehicle cabin; the mild fatigue state is a state in which the eye opening is greater than the first opening threshold and the blinking frequency is greater than the frequency threshold; When a driver-triggered stop command is received, a preset question is output in the vehicle cabin via voice; the preset question includes four mathematical operations within two digits; If a correct answer is received from the driver via voice within a preset time period, a stop command is triggered, and output of the prompt signal is stopped in response to the stop command; The first opening threshold is multiplied by a preset correction coefficient to obtain a corrected first opening threshold, and the corrected first opening threshold is updated to the first opening threshold, wherein the correction coefficient is less than 1.

2. The detection method according to claim 1, wherein The opening threshold further includes a second opening threshold, and the first opening threshold is greater than the second opening threshold; In a state where the eye opening is less than the first opening threshold and greater than the second opening threshold, and the blinking frequency is greater than the frequency threshold, determining that the driver is in a moderate fatigue state; In a state where the eye opening is less than the second opening threshold and the duration for which the eye opening is less than the second opening threshold is greater than a preset duration, it is determined that the driver is in a severe fatigue state.

3. The detection method according to claim 2, characterized in that Before acquiring the real-time facial image data of the driver, the detection method further includes: determining a blink opening degree based on facial image data of the driver; determining the blink opening degree as the second opening degree threshold; The determining of the eye opening degree and the blinking frequency of the driver based on the distance between the eye feature points includes: When the eye opening of the driver is less than the blink opening, it is determined that the driver has blinked once, and the number of blinks of the driver within a preset time period is determined as the blink frequency.

4. The detection method according to claim 1, wherein Before acquiring the real-time facial image data of the driver, the detection method further includes: Inputting the driver's facial image data into a trained neural network model to obtain the blink opening; Determining the eye opening of the driver and the blinking frequency of the driver based on the distance between the eye feature points includes: When the eye opening of the driver is less than the blink opening, it is determined that the driver has blinked once, and the number of blinks of the driver within a preset time period is determined as the blink frequency.

5. The detection method according to any one of claims 1 to 4, characterized in that The determining the eye opening of the driver based on the distance between the eye feature points includes: The length and width of the eye are determined based on the eye feature points, and the eye width-to-length ratio is obtained based on the width and length, and the eye width-to-length ratio is determined as the eye opening.

6. The detection method according to claim 2, characterized in that The method further comprises: When it is determined that the driver is in the moderate fatigue state, a second control signal is output to limit the maximum speed of the vehicle; when it is determined that the driver is in the severe fatigue state, a third control signal is output to make the vehicle continue to sound the horn and keep the double flash warning lights on.

7. A fatigue driving detection system, characterized in that: The detection system comprises: An acquisition module, configured to acquire real-time facial image data of the driver, wherein the facial image data includes a plurality of eye feature points; an eye processing module, configured to determine the eye opening of the driver and the blinking frequency of the driver based on the distance between the eye feature points; a fatigue recognition module, configured to determine a frequency threshold based on the eye opening degree, and determine the fatigue state of the driver based on the eye opening degree, the opening degree threshold, the blinking frequency, and the frequency threshold, wherein the frequency threshold is positively correlated with the eye opening degree; and the opening degree threshold includes a first opening degree threshold; An output module is used to output a first control signal to output a prompt signal in the vehicle cabin when it is determined that the driver is in a state of mild fatigue; the mild fatigue state is a state in which the eye opening is greater than the first opening threshold and the blinking frequency is greater than the frequency threshold; when a stop command triggered by the driver is received, a preset question is output in the vehicle cabin by voice; the preset question includes four mathematical operations within two digits; if a correct answer is received from the driver by voice within a preset time, a stop command is triggered, and the output of the prompt signal is stopped in response to the stop command; the first opening threshold is multiplied by a preset correction coefficient to obtain a corrected first opening threshold, and the corrected first opening threshold is updated to the first opening threshold, wherein the correction coefficient is less than 1.

8. A fatigue driving detection device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the fatigue driving detection method as claimed in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the fatigue driving detection method according to any one of claims 1 to 6.

Images