Image-based fatigue detection method, device and equipment

Abstract

The invention relates to an image-based fatigue detection method, device and equipment. The method comprises the following steps: acquiring an image acquired at a current time point; using a pre-trained first image recognition model to recognize the image to obtain a face image marked with a plurality of key points and a head attitude angle; when the head attitude angle meets a preset condition, an identification result of the current time point is determined according to the face image and a pre-trained second image identification model, and the second image identification model is obtained through training based on different face images; according to the identification result of the current time point, determining an identification result sequence corresponding to a time window taking the current time point as an end point; and performing fatigue driving detection on the driver according to the recognition result sequence. The method can avoid the influence of the individual difference of the driver on the recognition result to a certain extent, and further accurately determine the fatigue state of the driver.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device and equipment for fatigue detection based on images. Background Art

[0002] With the development of technology, long-term driving is becoming more and more common. Drivers are prone to fatigue after long-term driving, which will seriously affect driving safety and even cause serious safety accidents. Therefore, driver fatigue detection is very important for driving safety.

[0003] In the related art, the vehicle control system collects images of the driver's position to obtain an image to be processed, inputs the image into a pre-trained facial key point model to obtain coordinate information of the eye key points, and calculates the eye aspect ratio based on the coordinate information of the eye key points to judge the driver's fatigue state according to the eye aspect ratio.

[0004] Since the eyes of different drivers are quite different, resulting in different eye aspect ratios, judging fatigue status based on eye aspect ratio is easily affected by individual differences of drivers, and thus cannot accurately determine the fatigue status of the driver. Summary of the invention

[0005] The present application provides an image-based fatigue detection method, device and equipment, which can avoid the influence of individual differences of drivers on recognition results to a certain extent, and thus accurately determine the driver's fatigue status.

[0006] In a first aspect, the present application provides a method for fatigue detection based on an image, the method comprising:

[0007] Get the image collected at the current time point;

[0008] Using a pre-trained first image recognition model, the image is recognized to obtain a face image and a head posture angle annotated with a plurality of key points;

[0009] When the head posture angle meets a preset condition, determining a recognition result at the current time point according to the face image and a pre-trained second image recognition model, wherein the second image recognition model is trained based on different face images;

[0010] Determine, based on the recognition result at the current time point, a recognition result sequence corresponding to a time window with the current time point as an end point;

[0011] The driver's fatigue driving detection is performed according to the recognition result sequence.

[0012] Optionally, the first image recognition model includes a first detection module and a second detection module, and the use of the pre-trained first image recognition model to recognize the image to obtain a face image annotated with multiple key points and a head posture angle includes:

[0013] Inputting the image into a first detection module to obtain a face image;

[0014] The face image is input into a second detection module to obtain a face image with a plurality of key points and a head posture angle.

[0015] Optionally, determining the recognition result at the current time point according to the facial image and a pre-trained second image recognition model includes:

[0016] According to the key points on the face image, the face image is cropped to obtain a target image;

[0017] The target image is recognized using a pre-trained second image recognition model to obtain a recognition result, where the target image is a mouth image and an eye image.

[0018] Optionally, the step of cropping the facial image according to key points on the facial image to obtain a target image includes:

[0019] Determine a reference area in the face image according to the key points on the face image, wherein the reference area includes a mouth area and an eye area;

[0020] According to the head posture angle, the reference area is expanded to obtain a target area;

[0021] A target image is obtained according to the target area.

[0022] Optionally, the head posture angle includes a head pitch angle and a head yaw angle, and the step of expanding the reference area according to the head posture angle to obtain a target area includes:

[0023] When the head pitch angle is greater than the pitch angle threshold, a first outward expansion ratio is acquired, and longitudinal expansion is performed with the reference area as the center to obtain a target area;

[0024] When the head yaw angle is greater than the yaw angle threshold, a second outward expansion ratio is obtained, and horizontal expansion is performed with the reference area as the center to obtain a target area;

[0025] When the head pitch angle is not greater than the pitch angle threshold, and the head yaw angle is not greater than the yaw angle threshold, a third outward expansion ratio is acquired, and according to the third outward expansion ratio, the reference area is expanded outward to obtain a target area.

[0026] Optionally, obtaining a target image according to the target area includes:

[0027] When the target area corresponds to the eye area, cropping the target area to obtain a cropped image;

[0028] Performing image enhancement on the cropped image to obtain an image-enhanced cropped image;

[0029] A target image is determined according to the cropped image after image enhancement.

[0030] Optionally, the method further includes:

[0031] When the head posture angle does not meet the preset condition, determining the recognition result at the current time point as the first recognition result, the first recognition result being that the driver's face is blocked;

[0032] The step of performing fatigue driving detection on the driver according to the recognition result sequence includes:

[0033] Determining the number of times the first recognition result appears in the recognition result sequence;

[0034] When the number of times is greater than or equal to a first preset number of times, an occlusion alarm is issued to the driver;

[0035] When the number is less than a first preset number, the fatigue level of the driver is determined according to the recognition result, and a fatigue reminder strategy to be used is determined according to the fatigue level of the driver.

[0036] Optionally, determining, based on the recognition result at the current time point, a recognition result sequence corresponding to a time window with the current time point as an end point includes:

[0037] Adjusting the position of the time window on the time axis so that the end point of the time window is the current time point;

[0038] Obtaining the recognition result corresponding to each time point covered by the time window;

[0039] The recognition results are sorted according to the time point corresponding to each recognition result to obtain a recognition result sequence corresponding to a time window with the current time point as the end point.

[0040] In a second aspect, the present application provides a fatigue detection device based on an image, the device comprising:

[0041] An acquisition unit, used to acquire an image collected at a current time point;

[0042] A recognition unit, configured to recognize the image using a pre-trained first image recognition model to obtain a face image and a head posture angle annotated with a plurality of key points;

[0043] A first determination unit, configured to determine a recognition result at the current time point according to the face image and a pre-trained second image recognition model when the head posture angle meets a preset condition, wherein the second image recognition model is trained based on different face images;

[0044] A second determining unit, configured to determine, based on the recognition result at the current time point, a recognition result sequence corresponding to a time window with the current time point as an end point;

[0045] The detection unit is used to perform fatigue driving detection on the driver according to the recognition result sequence.

[0046] Optionally, the first image recognition model includes a first detection module and a second detection module, and the recognition unit is used to:

[0047] Inputting the image into a first detection module to obtain a face image;

[0048] The face image is input into a second detection module to obtain a face image with a plurality of key points and a head posture angle.

[0049] Optionally, the first determining unit is configured to:

[0050] According to the key points on the face image, the face image is cropped to obtain a target image;

[0051] The target image is recognized using a pre-trained second image recognition model to obtain a recognition result, where the target image is a mouth image and an eye image.

[0052] Optionally, the first determining unit is configured to:

[0053] Determine a reference area in the face image according to the key points on the face image, wherein the reference area includes a mouth area and an eye area;

[0054] According to the head posture angle, the reference area is expanded to obtain a target area;

[0055] A target image is obtained according to the target area.

[0056] Optionally, the head posture angle includes a head pitch angle and a head yaw angle, and the first determining unit is used to:

[0057] When the head pitch angle is greater than the pitch angle threshold, a first outward expansion ratio is acquired, and longitudinal expansion is performed with the reference area as the center to obtain a target area;

[0058] When the head yaw angle is greater than the yaw angle threshold, a second outward expansion ratio is obtained, and horizontal expansion is performed with the reference area as the center to obtain a target area;

[0059] When the head pitch angle is not greater than the pitch angle threshold, and the head yaw angle is not greater than the yaw angle threshold, a third outward expansion ratio is acquired, and according to the third outward expansion ratio, the reference area is expanded outward to obtain a target area.

[0060] Optionally, the first determining unit is configured to:

[0061] When the target area corresponds to the eye area, cropping the target area to obtain a cropped image;

[0062] Performing image enhancement on the cropped image to obtain an image-enhanced cropped image;

[0063] A target image is determined according to the cropped image after image enhancement.

[0064] Optionally, the device further includes a third determining unit:

[0065] When the head posture angle does not meet the preset condition, determining the recognition result at the current time point as the first recognition result, the first recognition result being that the driver's face is blocked;

[0066] The detection unit is used for:

[0067] Determining the number of times the first recognition result appears in the recognition result sequence;

[0068] When the number of times is greater than or equal to a first preset number of times, an occlusion alarm is issued to the driver;

[0069] When the number is less than a first preset number, the fatigue level of the driver is determined according to the recognition result, and a fatigue reminder strategy to be used is determined according to the fatigue level of the driver.

[0070] Optionally, the second determining unit is used to:

[0071] Adjusting the position of the time window on the time axis so that the end point of the time window is the current time point;

[0072] Obtaining the recognition result corresponding to each time point covered by the time window;

[0073] The recognition results are sorted according to the time point corresponding to each recognition result to obtain a recognition result sequence corresponding to a time window with the current time point as the end point.

[0074] In a third aspect, the present application provides an image-based fatigue detection device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; at least one memory connected to the at least one bus, wherein the processor is configured to:

[0075] Get the image collected at the current time point;

[0076] Using a pre-trained first image recognition model, the image is recognized to obtain a face image and a head posture angle annotated with a plurality of key points;

[0077] When the head posture angle meets a preset condition, determining a recognition result at the current time point according to the face image and a pre-trained second image recognition model, wherein the second image recognition model is trained based on different face images;

[0078] Determine, based on the recognition result at the current time point, a recognition result sequence corresponding to a time window with the current time point as an end point;

[0079] The driver's fatigue driving detection is performed according to the recognition result sequence.

[0080] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the above-mentioned image-based fatigue detection method is implemented.

[0081] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: in the embodiment of the present application, an image captured at the current time point is obtained; the image is recognized using a pre-trained first image recognition model to obtain a face image annotated with multiple key points and a head posture angle; when the head posture angle meets the preset conditions, the recognition result at the current time point is determined according to the face image and the pre-trained second image recognition model, and the second image recognition model is trained using images including different eyes; according to the recognition result at the current time point, a recognition result sequence corresponding to a time window with the current time point as the end point is determined; according to the recognition result sequence, a fatigue driving detection is performed on the driver. Since the recognition result in the embodiment of the present application is obtained based on the pre-trained second image recognition model, and the second image recognition model is obtained based on training with different face images, that is, the influence of individual differences on the results has been considered during the training process, the present application can avoid the influence of individual differences of drivers on the recognition results to a certain extent, and thus accurately determine the fatigue state of the driver. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0083] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0084] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0085] Figure 1 A schematic diagram of a process of performing fatigue detection based on an image provided in an embodiment of the present application;

[0086] Figure 2 A flowchart of an image recognition method provided in an embodiment of the present application;

[0087] Figure 3 A flowchart of an image processing method provided in an embodiment of the present application;

[0088] Figure 4 A schematic diagram of a flow chart of a target image determination method provided in an embodiment of the present application;

[0089] Figure 5 A schematic diagram of a human face provided in an embodiment of the present application;

[0090] Figure 6 A flowchart of another target image determination method provided in an embodiment of the present application;

[0091] Figure 7 A schematic diagram of a flow chart of a method for area expansion provided in an embodiment of the present application;

[0092] Figure 8 A flowchart of another method for detecting fatigue driving provided in an embodiment of the present application;

[0093] Fig. 9 A flowchart of another method for determining a recognition result sequence provided in an embodiment of the present application;

[0094] Fig.10 A schematic diagram of a process flow of an image-based fatigue detection device provided in an embodiment of the present application;

[0095] Fig.11 A schematic diagram of an image-based fatigue detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0096] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0097] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0098] With the development of science and technology, long-term driving trips have become more and more common. However, drivers are prone to fatigue when driving for a long time, which will seriously affect driving safety and even cause serious safety accidents. For this reason, driver fatigue detection is very important for driving safety. In the related art, the vehicle control system collects images of the driver's position to obtain an image to be processed, inputs the image into a pre-trained face key point model, obtains the coordinate information of the eye key points, and calculates the eye aspect ratio based on the coordinate information of the eye key points to judge the driver's fatigue state according to the eye aspect ratio. Due to the large differences in the eyes of different drivers, the eye aspect ratios of different drivers are different. Therefore, judging the fatigue state based on the eye aspect ratio is easily affected by individual differences of drivers, and thus the driver's fatigue state cannot be accurately determined.

[0099] In order to solve the above problems, an image captured at the current time point is obtained; a pre-trained first image recognition model is used to recognize the image to obtain a face image and a head posture angle marked with multiple key points; when the head posture angle meets the preset conditions, the recognition result at the current time point is determined according to the face image and a pre-trained second image recognition model, and the second image recognition model is trained using images including different eyes; according to the current time point and the recognition result at the current time point, a recognition result sequence corresponding to a time window with the current time point as the end point is determined; according to the recognition result sequence, a fatigue driving detection is performed on the driver. Since the recognition result in the embodiment of the present application is obtained based on the pre-trained second image recognition model, and the second image recognition model is obtained based on training of different face images, that is, the influence of individual differences on the results has been considered during the training process, the present application can avoid the influence of individual differences of drivers on the recognition results to a certain extent, and thus accurately determine the fatigue state of the driver.

[0100] In summary, the present application provides an image-based fatigue detection method, which can avoid the influence of individual differences of drivers on the recognition results to a certain extent, and thus accurately determine the fatigue state of the driver, such as Figure 1 As shown, the specific steps include:

[0101] Step 101, obtaining an image captured at the current time point.

[0102] In this step, a camera is provided in the vehicle, and the camera can be provided at a position where the driver's face can be photographed, and periodic image acquisition is performed through the camera. When the vehicle is in a starting state and the vehicle speed is greater than a certain threshold, the camera can send the image acquired at the current time point to the vehicle control system, so that the vehicle control system receives the image and performs subsequent processing based on the image.

[0103] It should be noted that the execution subject of the present application may be a vehicle control system, so that the vehicle control system can execute the steps described in the present application. Of course, in order to save the computing resources of the vehicle control system, after the present application sends the image to the vehicle control system, the vehicle control system can send the image to the server through the network, so that the server can execute the steps described in the present application.

[0104] Step 102: Use a pre-trained first image recognition model to recognize the image and obtain a face image and head posture angles marked with multiple key points.

[0105] The first image recognition model can output a face image with key points marked based on the image to determine the eye area and the mouth area based on the key points. At the same time, the first image recognition model can also output the head posture angle corresponding to the face on the face image.

[0106] In this step, the vehicle control system can input the image into a pre-trained first image recognition model to obtain a facial image and head posture angles with eyes and mouth annotated with multiple key points.

[0107] Step 103, when the head posture angle meets the preset conditions, the recognition result at the current time point is determined according to the face image and the pre-trained second image recognition model.

[0108] The second image recognition model is trained based on images of different faces. The head posture angle includes a head pitch angle and a head yaw angle, and the preset condition is that the head pitch angle is less than a first value and the head yaw angle is less than a second value.

[0109] In this step, the head posture angle is used to determine whether the driver's face is blocked. When the head posture meets the preset conditions, the vehicle control system determines that the driver's face is not blocked, and when the head posture does not meet the preset conditions, the vehicle control system determines that the driver's face is blocked. Since the driver's face is blocked, the facial image cannot be recognized based on the second image recognition model. Therefore, when the driver's face is not blocked, the vehicle control system determines the recognition result at the current time point based on the facial image and the pre-trained second image recognition model.

[0110] Furthermore, the vehicle control system can directly use the second image recognition model to recognize the face image and obtain the recognition result at the current time point. Of course, the embodiment of the present application can also further process the face image to obtain a processed image, and use the second image recognition model to process the processed image to obtain the recognition result at the current time point.

[0111] It should be noted that when the head posture does not meet the preset conditions, the recognition result at the current time point can be directly determined as the driver's face being blocked, so as to perform processing based on the recognition result at the current time point.

[0112] Step 104: Determine, based on the recognition result at the current time point, a recognition result sequence corresponding to a time window with the current time point as the end point.

[0113] The time window is a time period with the current time point as the end point and a preset time length. Each recognition result in the recognition result sequence is sorted according to its corresponding time point.

[0114] In this step, the vehicle control system determines a time window with the current time point as the end point and a preset duration, and searches for the recognition results corresponding to all time points in the time window, and sorts these recognition results in the order of their corresponding time points to obtain a recognition result sequence.

[0115] Step 105: Perform a fatigue driving test on the driver according to the recognition result sequence.

[0116] In this step, the vehicle control system can perform fatigue driving detection on the driver based on the recognition result sequence.

[0117] In an embodiment of the present application, an image captured at the current time point is obtained; the image is recognized using a pre-trained first image recognition model to obtain a face image annotated with multiple key points and a head posture angle; when the head posture angle meets a preset condition, the recognition result at the current time point is determined based on the face image and a pre-trained second image recognition model, the second image recognition model is trained using images including different eyes; based on the recognition result at the current time point, a recognition result sequence corresponding to a time window with the current time point as the end point is determined; based on the recognition result sequence, a fatigue driving detection is performed on the driver. Since the recognition result in the embodiment of the present application is obtained based on a pre-trained second image recognition model, and the second image recognition model is obtained based on training with different face images, that is, the influence of individual differences on the results has been considered during the training process, the present application can avoid the influence of individual differences of drivers on the recognition results to a certain extent, and thus accurately determine the fatigue state of the driver.

[0118] In the embodiment of the present application, when the first image recognition model includes a face detection module, a key point recognition module, and a posture angle recognition module, the vehicle control system can use the pre-trained first image recognition model to recognize the image and obtain a face image with multiple key points and a head posture angle. Therefore, the embodiment of the present application provides an image recognition method, which is a further limitation of step 102, such as Figure 2 As shown, the specific steps are:

[0119] Step 201: input an image into a first detection module to obtain a face image.

[0120] Among them, the first detection module is a face recognition module, which is used to identify faces in images. This module is a machine learning module. Specifically, this module is a lightweight face detection model. This model builds a lightweight face detection model by acquiring images collected by the DMS camera, and uses a single-step face detector RetinaFace to detect face information. In order to facilitate the embedded deployment of the model, MobileNet v3 with a width factor of 0.25 is used as the backbone network. MobileNetv3 is based on technologies such as deep separable convolution and inverted residual modules, SE modules, and neural architecture search to provide a lightweight, efficient and accurate deep learning model that can minimize computing resource requirements and latency while maintaining high performance. It is very suitable for application scenarios with high requirements for real-time and efficiency. In order to increase the generalization of the model, a face detection dataset is produced based on open source datasets and images collected by DMS cameras. Data enhancement methods such as random rotation, random noise addition, random blurring, and random erasing are used to process the data in the dataset.

[0121] In this step, the vehicle control system can input the image into the first detection module and output the face image.

[0122] Step 202: input the face image into the second detection module to obtain the face image and head posture angle annotated with multiple key points.

[0123] Among them, the head posture angle includes the head pitch angle and the head yaw angle. The head pitch angle is the angular deviation caused by the up and down movement of the head, and the head yaw angle is the angular deviation caused by the left and right movement of the head.

[0124] The second detection module is a key point labeling and head posture angle recognition module, which is used to label the key points of the mouth area and eye area in the face image, and is also a machine learning model. In this step, a lightweight face 25 key point and head posture angle model is built, and an adaptive key point loss function is designed. Considering that eye and mouth detection requires eye and mouth key point information, and the number of key points will affect the model effect and post-processing time complexity, this application uses MobileNet v3 with a width factor of 0.25 as the backbone network of the face 25 key point model. Because it is difficult to judge the driver's fatigue state when the head deflection angle is too large, this application adds a head posture angle branch to the face key point model to identify the head posture angle. In the face key point detection, some key points (such as side face, key points under extreme posture) are often difficult to predict and prone to large errors. Wing Loss has a stronger correction effect on these large errors, helping the model to better handle these difficult-to-predict points. For most key points that are easy to predict, MSE Loss provides an efficient optimization approach. For this purpose, a weighted sum of two loss functions is used as the key point model loss function:

[0125] Total Loss=α·MSE Loss+(1-α)·Wing Loss

[0126]

[0127] Where w and ∈ are the weights of the control error, which can be set according to the needs of the technicians, and p i is the predicted value of the i-th key point, g i is the true value of the ith key point, C is a constant used to maintain smoothness and is set according to the needs of the technician, and α is the weight parameter of the loss function.

[0128] In order to improve the training effect, the weight of the loss function is automatically adjusted according to the performance of the model at different training stages; in the early stages of training, the model error is usually large, so Wing Loss can account for a larger proportion to help the model correct large errors as quickly as possible. In the later stages of training, when the model gradually converges, the weight of MSE Loss can be increased to fine-tune small errors. For this purpose, a progress factor t of the training process is defined:

[0129]

[0130] α(t)=1-t

[0131] Among them, current epoch is the number of times training has been completed, and total epoch is the total number of times training has been completed.

[0132] In this way, by combining MSE Loss and Wing Loss, the model can not only quickly optimize most of the key points that are easy to predict, but also perform better on the key points that are difficult to predict, thereby improving the overall prediction accuracy.

[0133] In addition, based on the WLFW and AFLW open source data sets, we can obtain 29 key points of the face and head posture information to build a training set of the face key point model. The model input size is 112x112. During the training process, data enhancement methods such as random erasing, random changes in brightness contrast and saturation are used. An exponential decay learning rate regulator with hot start and AdamW optimizer are used to train the model for 100 Epochs.

[0134] Optionally, since the key points on the face image are marked with eyes and mouth, the vehicle control system can also crop the face image according to the key points marked in the face image to obtain eye images and mouth images, and then use the pre-trained second image recognition model to process these images to determine the recognition result at the current time point. Therefore, the embodiment of the present application provides an image processing method, which method is as follows Figure 3 As shown, the specific steps include:

[0135] Step 301, cropping the face image according to key points on the face image to obtain a target image.

[0136] Among them, the target images are human eye images and human mouth images.

[0137] In this step, since the key points on the face image are marked with eyes and mouth, the eye area and eye area in the face image can be found based on these key points, and then the face image can be cropped according to the eye area and mouth area to obtain eye image and eye image.

[0138] Step 302: Use the pre-trained second image recognition model to recognize the target image and obtain a recognition result.

[0139] Among them, the recognition results are whether the driver is yawning and whether the driver's eyes are open.

[0140] In this step, since the target image is an eye image that only includes human eyes and a mouth image that only includes human mouth, when the second image recognition model recognizes the target image, the interference of other images on the recognition process can be reduced, thereby making the recognition result more accurate. The vehicle control system can use the pre-trained second image recognition model to identify whether the driver is yawning and whether the driver's eyes are open in the target image to obtain the recognition result. The second image recognition model is trained based on different facial images, which can avoid the influence of individual differences on the recognition result to a certain extent.

[0141] It should be noted that after obtaining the recognition result, it is also possible to determine whether the driver's eyes are closed based on the highest key point and the lowest key point of the left eye and the highest key point and the lowest key point of the right eye, and to determine whether the driver's mouth is closed based on the highest key point and the lowest key point of the mouth. The recognition result is further verified based on whether the driver's eyes are closed and whether the driver's mouth is closed.

[0142] In the embodiment of the present application, the vehicle control system can first determine the mouth area and the eye area according to the key points on the face image, and then crop the face image according to the mouth area and the eye area to obtain the target image. Therefore, the embodiment of the present application provides a method for determining a target image, as described in 4, and the specific steps include:

[0143] Step 401, determining a reference area in the face image according to key points on the face image.

[0144] The reference area includes the mouth area and the eye area.

[0145] In this step, generally speaking, a plurality of key points are marked on the face image, and the number of these key points is mainly determined by the first image recognition model used. For example, if the first image recognition model is a face 25 key point model, the face image is marked with 25 key points, and if the first image recognition model is a face 27 key point model, the face image is marked with 27 key points. No matter which model is used, the face model outputted therein has key points corresponding to the left corner of the eye, the highest key point and the lowest key point of the left eye, the key points corresponding to the right corner of the eye, the highest key point and the lowest key point of the right eye, the key points corresponding to the left corner of the mouth, the key points corresponding to the right corner of the mouth, the highest key point of the human mouth and the lowest key point of the human mouth. In this way, the key points corresponding to the left corner of the eye, the highest key point and the lowest key point of the left eye, the key points corresponding to the right corner of the eye, the highest key point and the lowest key point of the right eye can be determined to determine the minimum circumscribed rectangle of these key points. According to the key points corresponding to the left corner of the mouth, the key points corresponding to the right corner of the mouth, the highest key point of the human mouth and the lowest key point of the human mouth, the minimum circumscribed rectangle of these key points can be determined. Finally, the areas corresponding to these minimum bounding rectangles are determined as reference areas.

[0146] Furthermore, in order to determine whether the key point is a key point of the minimum bounding rectangle, the first image recognition model can also output the label of each key point and pre-store the preset label corresponding to the required key point. In this way, when it is necessary to perform this step, the label of each key point can be obtained according to the key points on the face image, and then the key points corresponding to the preset label are selected from these key points, and finally the minimum bounding rectangle corresponding to these key points is determined, and the corresponding area is determined as the reference area.

[0147] For example, Figure 5 As shown, the figure is a face image with 25 key points annotated, which is the output of the first image recognition model of the face annotated with 25 key points, wherein the preset numbers are 7, 8, 11, 12, 13, 14, 19, 20, 21 and 24. After the model outputs the face image, among the key points on the face image, key points 7, 8, 11, 12, 13 and 14 are determined, the minimum bounding rectangles of these key points are obtained, and the corresponding areas are used as reference areas, and the minimum bounding rectangles of these key points are determined according to key points 19, 20, 21, and 24, and the corresponding areas are used as reference areas.

[0148] Step 402: Expand the reference area according to the head posture angle to obtain the target area.

[0149] In this step, the vehicle control system can predict the movement direction of the human eyes and mouth in the image based on the head posture angle, and expand the reference area according to the prediction result to obtain the target area.

[0150] Step 403, obtaining a target image according to the target area.

[0151] In this step, the vehicle control system can directly crop the target area to obtain a target image. Since the eye image is not easy to identify, the cropped eye image can also be enhanced to obtain an image enhanced, and then the target image is obtained based on the image.

[0152] In summary, the vehicle control system can first detect whether the target area corresponds to the eye. When the target area does not correspond to the eye, the target area can be directly cropped to obtain a target image. When the target area corresponds to the eye, the target area needs to be cropped to obtain a cropped image, and the cropped image is enhanced to obtain an enhanced cropped image. The target image is obtained based on the enhanced cropped image and the cropped image. Therefore, the present application provides a target image determination method, which method is as follows: Figure 6 As shown, the specific steps include:

[0153] Step 601: When the target area corresponds to the eye area, the target area is cropped to obtain a cropped image.

[0154] In this step, the vehicle control system may crop the target area to obtain a cropped image.

[0155] Step 602: Perform image enhancement on the cropped image to obtain an enhanced cropped image.

[0156] In this step, the vehicle control system can copy the cropped image to save the original cropped image. After that, the vehicle control system can use an edge detection algorithm to detect the contour of the eye area in the cropped image, and use an adaptive histogram equalization method to enhance the contrast of the image so that the contour of the eye is more obvious. Finally, by applying a sharpening filter, the edge of the eye is further enhanced, and finally the cropped image after image enhancement is obtained.

[0157] Among them, edge detection algorithms are used to detect the contour of the eye area, such as the Canny edge detection algorithm. Sharpening filters can highlight the edge information of objects while denoising, keep the edge information of the image unchanged or enhance the edge information, such as the Laplacian operator, custom convolution kernel, etc.

[0158] Step 603: determine the target image according to the cropped image after image enhancement.

[0159] In this step, the vehicle control system can directly determine the cropped image after image enhancement as the target image. Of course, in order to avoid image information loss during the image enhancement process, the vehicle control system can merge the cropped image and the cropped image after image enhancement to generate a new image, and determine the new image as the target image to highlight the eye contour while retaining the details and color information of the original image.

[0160] In the embodiment of the present application, when the driver's head turns too much up and down and left and right, since the position of the driver's face on the image collected in step 101 moves up and down and left and right, the effective areas of the eyes and mouth can be determined based on the face. The embodiment of the present application provides a method for expanding the area. Figure 7 As shown, the specific steps include:

[0161] Step 701: When the head pitch angle is greater than the pitch angle threshold, a first outward expansion ratio is obtained, and longitudinal expansion is performed with the reference area as the center to obtain a target area.

[0162] In this step, when the head pitch angle is greater than the pitch angle threshold, it means that the driver is lowering or raising his head. In this case, in order to adapt to the up and down movement of the human eyes and mouth in the image, the vehicle control system can obtain the first outward expansion ratio and perform longitudinal expansion with the reference area as the center to obtain the target area.

[0163] Step 702: When the head yaw angle is greater than the yaw angle threshold, a second outward expansion ratio is obtained, and horizontal expansion is performed with the reference area as the center to obtain a target area.

[0164] In this step, when the head yaw angle is greater than the yaw angle threshold, it means that the driver is tilting his head. In this case, in order to adapt to the left and right movement of the human eyes and mouth in the image, the vehicle control system can obtain a second expansion ratio and expand horizontally with the reference area as the center to obtain the target area.

[0165] Step 703: when the head pitch angle is not greater than the pitch angle threshold and the head yaw angle is not greater than the yaw angle threshold, a third expansion ratio is obtained, and the target area is expanded outward with the reference area as the center according to the third expansion ratio.

[0166] In this step, when the head pitch angle is not greater than the pitch angle threshold, and the head yaw angle is not greater than the yaw angle threshold, it means that the movement of the human eyes and the human mouth is not large. However, in order to avoid the reference area not being able to completely cover the eye area and the mouth area, the vehicle control system can also expand the reference area according to a smaller third expansion ratio to obtain the target area.

[0167] It should be noted that the above expansion ratios are set according to actual needs. When the head pitch angle is not greater than the pitch angle threshold, and the head yaw angle is not greater than the yaw angle threshold, it means that the driver's head swing is not large, so among the above expansion ratios, the third expansion ratio is the smallest.

[0168] In the embodiment of the present application, when the head posture angle does not meet the preset conditions, the vehicle control system determines the recognition result at the current time point as the first recognition result, and then generates a recognition result sequence based on the recognition result at the current time point, and then performs fatigue driving detection on the driver based on the recognition result sequence. Among them, the first recognition result is that the driver's face is blocked. Therefore, the embodiment of the present application provides a fatigue driving detection method, which is as follows Figure 8 As shown, the specific steps include:

[0169] Step 801: Determine the number of times a first recognition result appears in a recognition result sequence.

[0170] Step 802: When the number of times is greater than or equal to the first preset number of times, an obstruction alarm is issued to the driver.

[0171] Among them, the first preset time is set by the technician based on experience.

[0172] In this step, the vehicle control system can detect the number of times the first recognition result appears in the recognition result sequence. When the number is greater than or equal to the first preset number, an occlusion alarm is issued to the driver. When the number is less than the first preset number, step 802 is executed.

[0173] Step 803: When the number of times is less than the first preset number of times, the fatigue level of the driver is determined according to the recognition result, and the fatigue reminder strategy to be used is determined according to the fatigue level of the driver.

[0174] In this step, when the number is less than the first preset number, the vehicle control system can also detect the number of times the second recognition result appears in the recognition result sequence. When the number is greater than or equal to the second preset number, the driver's fatigue level is determined to be level three fatigue. When the number is less than the second preset number, the number of times the third recognition result appears in the recognition result sequence is continued to be detected. When the number is greater than or equal to the third preset number, and the recognition results corresponding to these numbers are not continuous in the recognition result sequence, the driver's fatigue level is determined to be level two fatigue. When the number is greater than or equal to the third preset number, and the recognition results corresponding to these numbers are continuous in the recognition result sequence, the driver's fatigue level is determined to be level one. After that, the fatigue reminder strategy to be used corresponding to the fatigue level can be determined according to the correspondence between the preset level and the fatigue reminder strategy.

[0175] Among them, the second recognition result is that the driver's mouth is open, and the third recognition result is that the driver's eyes are closed.

[0176] In the embodiment of the present application, the vehicle control system can also determine the recognition result sequence corresponding to the time window with the current time point as the end point according to the recognition result at the current time point. Therefore, the embodiment of the present application provides a method for determining the recognition result sequence. Fig. 9 As shown, the specific steps include:

[0177] Step 901: adjust the position of the time window on the time axis so that the end point of the time window is the current time point.

[0178] The time window is a time period with a preset time period, which is set according to the needs of the technician.

[0179] In this step, a time axis is set in the vehicle control system, and there are multiple time points on the time axis, and each time point corresponds to a recognition result. The vehicle control system can adjust the position of the time window on the time axis so that the end point of the time window corresponds to the current time point.

[0180] Step 902: Obtain the recognition result corresponding to each time point covered by the time window.

[0181] In this step, when the end point of the time window corresponds to the current time point, the vehicle control system can obtain all the time points covered by the time window and obtain the recognition results corresponding to each time point.

[0182] Assume that the length of the time window is 2 seconds, the current time point is the 4th second, and there are the 1st second, 2nd second, 3rd second, and 4th second on the time axis. Adjust the position of the time window so that the end point of the time window corresponds to the current time point. At this time, the time points covered by the time window are the 2nd second, the 3rd second, and the 4th second. At this time, obtain the recognition results corresponding to these time points respectively.

[0183] Step 903, sorting the recognition results according to the time point corresponding to each recognition result, and obtaining a recognition result sequence corresponding to the time window with the current time point as the end point.

[0184] For example, based on step 902, the recognition result A corresponding to the 2nd second, the recognition result B corresponding to the 3rd second, and the recognition result C corresponding to the 4th second are obtained. At this time, the recognition results corresponding to each time point are sorted in the order of the 2nd second, the 3rd second, and the 4th second, and the obtained recognition result sequence is the recognition result A, the recognition result B, and the recognition result C.

[0185] like Fig.10 As shown, the embodiment of the present application provides a fatigue detection device based on an image, which corresponds to the method embodiment and specifically includes:

[0186] An acquisition unit 1001 is used to acquire an image collected at a current time point;

[0187] The recognition unit 1002 is used to recognize the image using a pre-trained first image recognition model to obtain a face image and a head posture angle annotated with a plurality of key points;

[0188] A first determining unit 1003 is used to determine the recognition result at the current time point according to the face image and a pre-trained second image recognition model when the head posture angle meets a preset condition, wherein the second image recognition model is trained based on different face images;

[0189] A second determining unit 1004 is used to determine, according to the recognition result at the current time point, a recognition result sequence corresponding to a time window with the current time point as an end point;

[0190] The detection unit 1005 is used to perform fatigue driving detection on the driver according to the recognition result sequence.

[0191] Optionally, the first image recognition model includes a first detection module and a second detection module, and the recognition unit 1002 is used to:

[0192] Inputting the image into a first detection module to obtain a face image;

[0193] The face image is input into a second detection module to obtain a face image with a plurality of key points and a head posture angle.

[0194] Optionally, the first determining unit 1003 is configured to:

[0195] According to the key points on the face image, the face image is cropped to obtain a target image;

[0196] The target image is recognized using a pre-trained second image recognition model to obtain a recognition result, where the target image is a mouth image and an eye image.

[0197] Optionally, the first determining unit 1003 is configured to:

[0198] Determine a reference area in the face image according to the key points on the face image, wherein the reference area includes a mouth area and an eye area;

[0199] According to the head posture angle, the reference area is expanded to obtain a target area;

[0200] A target image is obtained according to the target area.

[0201] Optionally, the head posture angle includes a head pitch angle and a head yaw angle, and the first determining unit 1003 is used to:

[0202] When the head pitch angle is greater than the pitch angle threshold, a first outward expansion ratio is acquired, and longitudinal expansion is performed with the reference area as the center to obtain a target area;

[0203] When the head yaw angle is greater than the yaw angle threshold, a second outward expansion ratio is obtained, and horizontal expansion is performed with the reference area as the center to obtain a target area;

[0204] When the head pitch angle is not greater than the pitch angle threshold, and the head yaw angle is not greater than the yaw angle threshold, a third outward expansion ratio is acquired, and according to the third outward expansion ratio, the reference area is expanded outward to obtain a target area.

[0205] Optionally, the first determining unit 1003 is configured to:

[0206] When the target area corresponds to the eye area, cropping the target area to obtain a cropped image;

[0207] Performing image enhancement on the cropped image to obtain an image-enhanced cropped image;

[0208] A target image is determined according to the cropped image after image enhancement.

[0209] Optionally, the device further includes a third determining unit 1006:

[0210] When the head posture angle does not meet the preset condition, determining the recognition result at the current time point as the first recognition result, the first recognition result being that the driver's face is blocked;

[0211] The detection unit 1005 is used for:

[0212] Determining the number of times the first recognition result appears in the recognition result sequence;

[0213] When the number of times is greater than or equal to a first preset number of times, an occlusion alarm is issued to the driver;

[0214] When the number is less than a first preset number, the fatigue level of the driver is determined according to the recognition result, and a fatigue reminder strategy to be used is determined according to the fatigue level of the driver.

[0215] Optionally, the second determining unit 1004 is configured to:

[0216] Adjusting the position of the time window on the time axis so that the end point of the time window is the current time point;

[0217] Obtaining the recognition result corresponding to each time point covered by the time window;

[0218] The recognition results are sorted according to the time point corresponding to each recognition result to obtain a recognition result sequence corresponding to a time window with the current time point as the end point.

[0219] like Fig.11 As shown, the embodiment of the present application provides an image-based fatigue detection device, including a processor 1101, a communication interface 1102, a memory 1103 and a communication bus 1104, wherein the processor 1101, the communication interface 1102, and the memory 1103 communicate with each other through the communication bus 1104.

[0220] Memory 1103, used for storing computer programs;

[0221] In one embodiment of the present application, the processor 1101 is used to execute the program stored in the memory 1103 to implement the image-based fatigue detection method provided by any of the aforementioned method embodiments, including:

[0222] Get the image collected at the current time point;

[0223] Using a pre-trained first image recognition model, the image is recognized to obtain a face image and a head posture angle annotated with a plurality of key points;

[0224] When the head posture angle meets a preset condition, determining a recognition result at the current time point according to the face image and a pre-trained second image recognition model, wherein the second image recognition model is trained based on different face images;

[0225] Determine, based on the recognition result at the current time point, a recognition result sequence corresponding to a time window with the current time point as an end point;

[0226] The driver's fatigue driving detection is performed according to the recognition result sequence.

[0227] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps performed by the image-based fatigue detection method provided in any of the aforementioned method embodiments are implemented.

[0228] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0229] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0230] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0231] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A fatigue detection method based on an image, characterized in that: The method comprises: Get the image collected at the current time point; Using a pre-trained first image recognition model, the image is recognized to obtain a face image and a head posture angle annotated with a plurality of key points; When the head posture angle meets the preset condition, determining the recognition result at the current time point according to the face image and a pre-trained second image recognition model, wherein the second image recognition model is trained based on different face images; Determine, based on the recognition result at the current time point, a recognition result sequence corresponding to a time window with the current time point as an end point; The driver's fatigue driving detection is performed according to the recognition result sequence.

2. The method according to claim 1, characterized in that: The first image recognition model includes a first detection module and a second detection module. The first image recognition model trained in advance is used to recognize the image to obtain a face image with multiple key points and a head posture angle, including: Inputting the image into a first detection module to obtain a face image; The face image is input into a second detection module to obtain a face image with a plurality of key points and a head posture angle.

3. The method according to claim 1, characterized in that: The determining the recognition result at the current time point according to the face image and the pre-trained second image recognition model includes: According to the key points on the face image, the face image is cropped to obtain a target image; The target image is recognized using a pre-trained second image recognition model to obtain a recognition result, where the target image is a mouth image and an eye image.

4. The method according to claim 3, characterized in that: The step of cropping the face image according to the key points on the face image to obtain a target image includes: Determine a reference area in the face image according to the key points on the face image, wherein the reference area includes a mouth area and an eye area; According to the head posture angle, the reference area is expanded to obtain a target area; A target image is obtained according to the target area.

5. The method according to claim 4, characterized in that: The head posture angle includes a head pitch angle and a head yaw angle, and the reference area is expanded according to the head posture angle to obtain a target area, including: When the head pitch angle is greater than the pitch angle threshold, a first outward expansion ratio is acquired, and longitudinal expansion is performed with the reference area as the center to obtain a target area; When the head yaw angle is greater than the yaw angle threshold, a second outward expansion ratio is obtained, and horizontal expansion is performed with the reference area as the center to obtain a target area; When the head pitch angle is not greater than the pitch angle threshold, and the head yaw angle is not greater than the yaw angle threshold, a third outward expansion ratio is acquired, and according to the third outward expansion ratio, the reference area is expanded outward to obtain a target area.

6. The method according to claim 4, characterized in that: According to the target area, a target image is obtained, including: When the target area corresponds to the eye area, cropping the target area to obtain a cropped image; Performing image enhancement on the cropped image to obtain an image-enhanced cropped image; A target image is determined according to the cropped image after image enhancement.

7. The method according to claim 3, characterized in that: The method further comprises: When the head posture angle does not meet the preset condition, determining the recognition result at the current time point as the first recognition result, the first recognition result being that the driver's face is blocked; The step of performing fatigue driving detection on the driver according to the recognition result sequence includes: Determining the number of times the first recognition result appears in the recognition result sequence; When the number of times is greater than or equal to a first preset number of times, an occlusion alarm is issued to the driver; When the number is less than a first preset number, the fatigue level of the driver is determined according to the recognition result, and a fatigue reminder strategy to be used is determined according to the fatigue level of the driver.

8. The method according to claim 1, characterized in that: The step of determining, based on the recognition result at the current time point, a recognition result sequence corresponding to a time window with the current time point as an end point comprises: Adjusting the position of the time window on the time axis so that the end point of the time window is the current time point; Obtaining the recognition result corresponding to each time point covered by the time window; The recognition results are sorted according to the time point corresponding to each recognition result to obtain a recognition result sequence corresponding to a time window with the current time point as the end point.

9. A fatigue detection device based on an image, characterized in that: The device comprises: An acquisition unit, used to acquire an image collected at a current time point; A recognition unit, configured to use a pre-trained first image recognition model to recognize the image, and obtain a face image and a head posture angle annotated with a plurality of key points; A first determination unit, configured to determine a recognition result at the current time point according to the face image and a pre-trained second image recognition model when the head posture angle meets a preset condition, wherein the second image recognition model is trained based on different face images; A second determining unit, configured to determine, based on the recognition result at the current time point, a recognition result sequence corresponding to a time window with the current time point as an end point; The detection unit is used to perform fatigue driving detection on the driver according to the recognition result sequence.

10. An image-based fatigue detection device, characterized in that: include: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; at least one memory connected to the at least one bus, wherein the processor is configured to: Get the image collected at the current time point; Using a pre-trained first image recognition model, the image is recognized to obtain a face image and a head posture angle annotated with a plurality of key points; When the head posture angle meets the preset condition, determining the recognition result at the current time point according to the face image and a pre-trained second image recognition model, wherein the second image recognition model is trained based on different face images; Determine, based on the recognition result at the current time point, a recognition result sequence corresponding to a time window with the current time point as an end point; The driver's fatigue driving detection is performed according to the recognition result sequence.

Images