A fatigue monitoring method applied to a driver of a vehicle

Abstract

The application provides a kind of fatigue monitoring method applied to car driver, specifically includes: through camera acquisition in-car image, first data analysis processing is carried out to image, obtains the face feature of person to be monitored;Extract the key feature in face feature for representing fatigue degree to evaluate the fatigue level of driver;According to fatigue level, control vehicle to execute fatigue reminding process.The technical scheme provided in the application includes that the face image of driver can be continuously searched and positioned in the image collected subsequently to carry out real-time monitoring, whether the image is compensated according to the light-dependent resistance to determine the current light condition, overcome the influence of light condition, ensure real-time monitoring while improving the accuracy of fatigue degree judgment.

Description

TECHNICAL FIELD

[0001] The present application relates to safe driving of a vehicle, and in particular to a method for monitoring fatigue of a driver of a vehicle. BACKGROUND

[0002] Improving public safety and reducing accidents is an important goal of intelligent transportation systems (ITS). One of the most important factors in traffic accidents, especially on rural roads, is driver fatigue. Fatigue reduces the driver's ability to perceive and make decisions to control the vehicle. Studies have shown that drivers usually feel tired after driving for 1 hour. Drivers are more tired and drowsy in the afternoon after lunch and in the late night than at other times. Statistics on accidents caused by fatigue driving and distraction vary from country to country. In single vehicle collisions (accidents involving only one damaged vehicle) or collisions involving heavy vehicles, up to 50% of accidents are related to the lack of alertness of the driver. According to current research, it is estimated that the use of driver face monitoring systems will reduce the number of car accidents by 10% to 20%. Therefore, many engineering and technical personnel in the prior art have attempted to detect driver fatigue. Currently, the methods for monitoring driver distraction and fatigue driving behavior at home and abroad are mainly divided into four categories, namely, driving behavior monitoring based on driving data, driving behavior monitoring based on physiological signals, driving behavior monitoring based on image processing, and driving behavior monitoring based on deep learning. Driving behavior based on driving data requires the collection of a large amount of data, such as vehicle speed, steering wheel angle, vehicle lateral and longitudinal displacement, and the data is complex, so the accuracy is low. The driving behavior monitoring technology based on physiological signals mainly collects physiological signal data related to the driver's distraction state and fatigue state, such as electroencephalogram, electrocardiogram, electrooculogram, and electromyogram, by wearing physiological signal collection devices on the driver. Machine learning methods are used to classify the data. The collection of physiological signals makes the driver feel uncomfortable, and many drivers reject this detection method. The driving behavior monitoring technology based on deep learning is mainly divided into two categories. The first category uses a deep convolutional neural network model to extract the spatial features of the driver's image to classify the driver's driving behavior. This method requires high hardware costs, and it is difficult for a car-grade chip embedded SOC to maintain high-load operation.

[0003] The driving behavior monitoring technology based on image processing mainly extracts the facial features or body posture features of the driver collected by the vehicle-mounted camera through image processing and traditional machine learning methods to monitor the driving behavior of the driver. Compared with the above three methods, the hardware configuration is relatively simple, and only a camera and a processor are needed to achieve it. However, the existing image-based driver fatigue driving method is affected by complex lighting conditions and other external environments when obtaining the driver's facial information, and only simple threshold judgment is used for judgment, which can lead to low accuracy. SUMMARY

[0004] Based on the defects existing in the prior art, the application provides a fatigue monitoring method applied to a driver of a vehicle, at least comprising:

[0005] The image is collected by a camera, and first data analysis and processing are performed on the image to obtain a face feature of a person to be monitored; a key feature for representing a fatigue degree is extracted from the face feature; a driver fatigue level is evaluated according to the key feature of the fatigue degree; a vehicle is controlled to perform a fatigue reminding process according to the fatigue level; the face of the person to be detected is positioned

[0006] The first data analysis and processing include: performing image preprocessing, obtaining a face existing in an initial frame of the image, obtaining a face image of the driver from the existing face, and continuously searching for and positioning the face image of the driver in subsequently collected images to monitor and extract a key feature coefficient for fatigue monitoring.

[0007] The key feature coefficient is trained and classified by using a support vector machine and a logistic regression to obtain a fatigue monitoring model, and the model is used to monitor real-time collected images and obtain a fatigue level of the driver.

[0008] A fatigue monitoring method applied to a driver of a vehicle, and further optionally, face feature recognition includes the following steps:

[0009] HOG feature operators of positive samples and negative samples are obtained; the positive and negative samples are trained by using a support vector machine algorithm to obtain a trained model; negative sample difficult example recognition is performed by using the model; the difficult example samples are retrained to obtain a final classification model through repeated training;

[0010] The final trained classifier is used to detect a face picture, different sizes of the picture are scanned by sliding, HOG features are extracted, and the classifier is classified.

[0011] If the detection is determined to be a face, the face is marked, and a final positioned face is obtained by using non-maximum suppression.

[0012] The number of data in the negative sample data set is greater than the number of samples in the positive sample data set.

[0013] The positive sample image includes: images of different scales obtained by splicing after random cropping and random scaling of pictures containing faces;

[0014] The negative sample image includes: spliced images obtained by random cropping of pictures not containing faces.

[0015] The model is used to identify difficult negative samples. Specifically, it includes: continuously scaling the negative samples in the training set until they match the template position; searching for matching through the template sliding serial port; selecting the image that identifies the negative sample as a positive sample and its corresponding confidence level; selecting the image with a confidence level greater than a preset threshold; and the image composed of randomly cropped, randomly scaled and stitched images; in addition, regularization is added.

[0016] A fatigue monitoring method for automobile drivers, optionally, further includes key facial features used to characterize fatigue levels, such as key eye features, including eye closure frequency η and blink frequency. To characterize;

[0017] Statistical continuous N L For each frame image, the eye closing coefficients (Leye_c) and blinking coefficients (Leye_o) are stored in a circular list. A first-in-first-out (FIFO) mechanism is used, and when the circular list is full, the old data will be replaced by the new data.

[0018] A fatigue monitoring method for automobile drivers, further optionally, in N L The formula for calculating the eye closure frequency η in a frame image is:

[0019]

[0020] In N L Blinking frequency in frame images The calculation formula is:

[0021]

[0022] Where i represents the i-th frame image, and the eye-closing coefficients Leye_c and Leye_o are the eye-blinking coefficients;

[0023] The blink coefficient Leye_o is represented by +1 and -1 to indicate the start and stop frames of the eye-closing event, respectively, with Leye_o = 0 under other conditions;

[0024] The formula for calculating the eye blink coefficient (Leye_o) is:

[0025]

[0026] A fatigue monitoring method for automobile drivers, optionally, further includes key facial features used to characterize fatigue levels, such as the mouth state feature coefficient Lmo.

[0027] When the mouth is open, Lmo = 1; when the mouth is closed, Lmo = 0.

[0028] Statistical continuous NL frame image, store Lmo of each frame image in a circular list, adopt a first-in first-out mechanism, when the circular list space is full, the old data will be replaced by new data;

[0029] The mouth closing frequency is defined as:

[0030]

[0031] A fatigue monitoring method applied to a driver of a vehicle, further optionally, obtaining a face image of the driver from the existing face includes:

[0032] Obtaining the number of faces in the image, obtaining the minimum circumscribed rectangle of all detected faces;

[0033] Obtaining the horizontal and vertical coordinates P(x i , y i ), i = 1, 2, 3, …, n of the center point of the minimum circumscribed rectangle of each face, wherein n represents the number of faces in the image;

[0034] Obtaining the maximum value x k of the horizontal coordinates in all minimum circumscribed rectangles;

[0035] x k = max(x1, x2, x3, …, x n )

[0036] The face identified by the minimum circumscribed rectangle corresponding to P(x k , y k ) is the face to be detected;

[0037] The face to be detected locates the face of the driver;

[0038] Or pre-obtain a face image of the driver, and match the pre-obtained face image of the driver in the existing face image to locate the driver image through module matching.

[0039] A fatigue monitoring method applied to a driver of a vehicle, further optionally, searching for and locating a face image of the driver includes:

[0040] The face to be detected is positioned, and feature points of the face to be detected are stored; a pre-processed current frame image is acquired, and the same feature points as the face to be detected are searched for and matched in the current frame image; when the feature points are not lost, a next frame image is acquired and the next frame image is used to replace the current frame image to continue the repeated identification process; when the feature points are lost, the unmatched feature points are deleted, and the matched feature points are saved; a next frame image is acquired, the face to be detected is positioned, and new feature points are saved; a frame image after the next frame image is acquired, and the matched feature points are searched for; the new feature points are selected to replace the lost feature points, the new feature points are saved and updated.

[0041] Or the following method is used:

[0042] A pre-processed previous frame image is acquired, and face feature points are detected in the previous frame image by using a face recognition algorithm;

[0043] A pre-processed current frame image is acquired, and the same feature points as the face to be detected are searched for and matched in the current frame image to acquire the feature points;

[0044] It is judged whether the feature points are lost, and when the feature points are not lost, the current frame image is replaced by the previous frame image, a next frame image is acquired, and the next frame image is used to replace the current frame image to perform cyclic matching;

[0045] When the feature points are lost, the unmatched feature points are deleted, and the matched feature points are saved, and new feature points are acquired in subsequent preset multiple images to replace the matched feature points.

[0046] The new feature points acquired in the subsequent preset multiple images to replace the matched feature points specifically include:

[0047] A next frame image is acquired, the face to be detected is positioned, and new feature points are saved; a frame image after the next frame image is acquired, and the matched feature points are searched for;

[0048] It is judged whether the matched feature points contain the number of deleted feature points, and if yes, the new feature points are selected to replace the lost feature points, the new feature points are saved and updated;

[0049] If not, the matched feature points are continuously acquired in multiple frame images; and if the number of cycles exceeds a preset value, the new feature points are re-determined.

[0050] A fatigue monitoring method applied to a driver of a vehicle, and further optionally, the first data analysis processing includes: judging a light condition of a current vehicle environment, and performing image preprocessing on an image according to the light condition;

[0051] A kind of fatigue monitoring method applied to car driver, further optional, judging the illumination condition of current vehicle interior environment includes: according to photoresistor judging current illumination intensity, if illumination intensity is lower than threshold value, first image pre-processing is carried out to the image collected.

[0052] A kind of fatigue monitoring method applied to car driver, further optional, when illumination is poor, using image pre-processing method includes:

[0053] F (x) = CDF (x) 2 +K (x) ×2 × CDF (x) × (1-CDF (x));

[0054] Wherein,

[0055] Y=F (x) ×255;

[0056] Y is the converted pixel value, Pm{.} represents the probability of event occurrence, X is the gray value of image, x indicates the classification value of image pixel value;CDF (x) indicates cumulative distribution function;PDF (u) is the probability density distribution function of X pixel value in image;K (x) is weight coefficient, for representing the uncertainty degree of event probability density.

[0057] A kind of fatigue monitoring method applied to car driver, further optional, when illumination is poor, using image pre-processing method includes:

[0058]

[0059]

[0060]

[0061] Convert R (x, y) into the range of 0~255;

[0062] Wherein, (x, y) is pixel coordinate, b brightness is adjustment parameter, R (x, y) is the converted pixel value, S (x, y) is the pixel value before conversion, g i (x, y) is two-dimensional Gaussian distribution function, Indicates standard deviation.

[0063] The method for searching and positioning the monitored face image includes:

[0064] Obtaining the minimum circumscribed rectangle Arect of the monitored face contour in the previous frame image; respectively obtaining the minimum circumscribed rectangle B(i)rect of all face contours in the current frame image, i=1, 2, …, k; calculating the spatial correlation coefficient T(i)s according to Arect and B(i)rect, i=1, 2, …, k; comparing the spatial correlation coefficient T(i)s with a preset threshold, and obtaining all spatial correlation coefficients T(i)s less than the preset threshold;

[0065] Solving the minimum value Tmin_s in all spatial correlation coefficients T(i)s less than the threshold, then obtaining the corresponding B(i)rect in the current frame image according to Tmin_s, and the face surrounded by B(i)rect is the monitored face image for search positioning.

[0066] Beneficial effects:

[0067] 1. In the prior art fatigue recognition method, only the case where only the driver exists in the camera is considered, but in the actual imaging process, different imaging ranges exist in different positions of the camera, and most positions can often shoot other members of the vehicle. In the fatigue monitoring process of the present application, the position of the driver's face is accurately positioned from a large number of faces, and in the subsequent monitoring process, the position of the driver in the image is continuously positioned, avoiding false monitoring of the face and thus false alarm;

[0068] 2. In the fatigue monitoring method of the intelligent vehicle-mounted box of the present application, the traditional method is improved, and for the key technical features of the eyes, only the closing frequency of the eyes is calculated, and when the closing of the eyes is counted to reach a preset value within a period of time, it is considered that the driver is in a fatigue state. However, it is ignored that the blinking of the driver will affect the accuracy. Therefore, in addition to the closing frequency of the eyes, the blinking frequency of the eyes is also considered in the present application, which is more reasonable to evaluate and calculate whether the driver is in a fatigue closed eye or a closed eye operation caused by facial expression change. Moreover, instead of simply counting the number, the classification model is obtained through feature extraction, model training by support vector machine or logistic regression, so that the judgment result is more reasonable. The method has small calculation amount and can ensure real-time performance.

[0069] 4. The present application needs to collect the same driver's face in different images, when there are multiple face targets, it needs to find the same face in different face positions in different images for feature comparison, in order to avoid the change of coordinate position caused by the movement of the people in the car, which leads to the inconsistency of the recognized face. In addition, the traditional method will lead to feature comparison error under different light environments, which will lead to recognition error. The present application identifies the fixed face of different image frames, and uses the spatial correlation coefficient of the upper frame face image and the current frame face image to determine the face to be matched, which can overcome the positioning error problem caused by light change. BRIEF DESCRIPTION OF DRAWINGS

[0070] The following drawings only illustrate and explain the present application, and do not limit the scope of the present application.

[0071] Figure 1 The algorithm logic diagram for fatigue monitoring of the detected target in an embodiment of the present application.

[0072] Figure 2 The algorithm logic diagram for continuous face positioning of the detected target in an embodiment of the present application.

[0073] Figure 3 The schematic diagram of eye feature structure in an embodiment of the present application.

[0074] Figure 4 The schematic diagram of mouth feature structure in an embodiment of the present application. DETAILED DESCRIPTION

[0075] In order to have a clearer understanding of the technical features, objects and effects of the present application, the specific embodiments of the present application will be described with reference to the drawings, and the same reference numerals in the drawings represent the same parts. In order to make the drawings simple, the relevant parts of the present application are schematically shown in the drawings, but not represent the actual structure of the product. In addition, in order to make the drawings simple and easy to understand, in some drawings, only one of the parts with the same structure or function is schematically shown, or only one of them is marked.

[0076] As is well known to those skilled in the art of control systems, functional modules, and application programs (APP), they can take any appropriate form, either hardware or software, either discrete multiple functional modules or multiple functional units integrated into one hardware. As the simplest form, the control system can be a controller, such as a combination logic controller, a microprogram controller, etc., as long as it can realize the operation described in the present application. Of course, the control system can also be integrated into a physical device as different modules, which does not deviate from the basic principles and protection scope of the present application.

[0077] In this invention, "connection" can include direct connection, indirect connection, communication connection, and electrical connection, unless otherwise specified.

[0078] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly specifies otherwise. It will also be understood that, when used in the specification, the terms “comprising” and / or “including” mean the presence of the stated features, values, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, values, steps, operations, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the listed related items.

[0079] It should be understood that the term "vehicle" or "of a vehicle" or other similar terms as used herein generally includes motor vehicles, such as passenger cars including sport utility vehicles (SUVs), buses, trucks, various commercial vehicles, boats including various vessels, aircraft, etc., and includes hybrid vehicles, electric vehicles, plug-in hybrid electric vehicles, hydrogen-powered vehicles, and other vehicles powered by alternative fuels (e.g., fuels derived from non-petroleum energy sources). As mentioned herein, a hybrid vehicle is a vehicle with two or more power sources, such as both gasoline and electric power.

[0080] The first embodiment of the present invention provides a method for monitoring driver fatigue, see below. Figures 1 to 4 Specifically, this includes:

[0081] Images are captured by a camera, and the images undergo initial data analysis to obtain the facial features of the driver to be monitored.

[0082] Extract key facial features used to characterize fatigue levels;

[0083] Assess driver fatigue levels based on key characteristics of fatigue levels;

[0084] Based on the fatigue level, control the vehicle to execute a fatigue reminder process.

[0085] The first data analysis and processing includes determining the current light intensity based on the photoresistor, and if the light intensity is lower than the threshold, performing a first image preprocessing on the acquired image.

[0086] Due to poor lighting, the imaging quality is poor, and the black area increases when the pixel is displayed, which affects the subsequent face positioning and face feature extraction. Therefore, in order to accurately extract the face positioning and face feature extraction in the poor lighting environment in the vehicle, the current illumination is detected by using a photoresistor, and if the lighting is poor, the first image preprocessing is started to make up for the insufficient exposure.

[0087] Specifically, the first image preprocessing is used only when the exposure is insufficient, which is different from the prior art. In the current image recognition process, in order to improve the robustness of the algorithm, the same image preprocessing is used for the case of sufficient light or insufficient light. The applicant found during the research and development that if the same image preprocessing scheme is used for the case of sufficient light or insufficient light, the imaging of the sufficient light condition will cause the brightness to concentrate, which will cause the subsequent face detection feature extraction to be inaccurate, especially in the eye area.

[0088] Therefore, the photoresistor is used to detect the lighting condition, and the first image preprocessing method is used when the lighting is poor. The first image preprocessing method includes:

[0089] F(x)=CDF(x) 2 +K(x)×2×CDF(x)×(1-CDF(x));

[0090] Wherein,

[0091] Y=F(x)×255;

[0092] Pm{.} represents the probability of the event occurring, X is the gray value of the image, x represents the classification value of the image pixel value; CDF(x) represents the cumulative distribution function; PDF(u) is the probability density distribution function of the X pixel value in the image; K(x) is a weight coefficient, used to represent the uncertainty degree of the event probability density. Y is the converted pixel value; through the above simple image conversion, the pixel distribution in the image can be improved when the light is insufficient, the dark pixel value in the image is improved as a whole, which can help the subsequent face positioning and face feature extraction.

[0093] Specifically, the above method has good processing effect and very fast speed, and is the preferred scheme for embedded devices, but some color values are lost.

[0094] The first image preprocessing further includes a second scheme, which specifically includes:

[0095]

[0096]

[0097]

[0098] R(x,y) is converted into the range of 0~255, so that it can be displayed.

[0099] wherein (x,y) is a pixel coordinate, b is a brightness adjustment parameter, R(x,y) is a converted pixel value, S(x,y) is a pixel value before conversion, g i (x,y) is a two-dimensional Gaussian distribution function, Different values are taken, and the pixel range of image display can be adjusted:

[0100] By introducing w(x,y), a larger image pixel range can be converted into a smaller range of output, balancing the pixel range of the image and increasing the imaging contrast of insufficient illumination.

[0101] The first data analysis processing further includes face positioning and face positioning of the person to be detected;

[0102] In the prior art, the fatigue recognition method only considers the case where only the driver exists in the camera, but in the actual imaging process, different imaging ranges exist at different positions of the camera, and most positions can often capture other members of the vehicle. How to accurately capture the driver in the image becomes one of the technical problems to be solved by the embodiment;

[0103] Referring to Figure 1 In the embodiment, the camera is mounted on the streaming media mirror of the front seat; preferably, in the embodiment, the smart car box mounted with the camera is mounted on the streaming media mirror to observe the driver or other positions in the vehicle, such as the front of the driver's cabin and the roof installed at a certain angle.

[0104] The smart car box is applied to image processing, so that the fatigue monitoring does not consume the resources of the central control host and is isolated from the host system.

[0105] Face positioning includes: recognizing all faces in the image, specifically:

[0106] A Haar classifier based on the AdaBoost algorithm is used to detect faces using Haar wavelet features and integral image method;

[0107] Prepare target and non-target samples, calculate feature values and integral images, filter out T excellent feature values; put the T weak classifiers into AdaBoost for training; construct a cascade classifier for detection to obtain the position of the face and the position of the eyes and mark the table;

[0108] Another method is used to detect faces;

[0109] Extract HOG features from the positive sample data set, obtain the HOG feature operator;

[0110] Extract HOG features from the negative sample data set, obtain the HOG feature operator, wherein the data amount in the negative sample data set is greater than the sample amount in the positive sample data set;

[0111] Specifically, in the actual model, the number of negative samples is much larger than the number of positive samples, and if the negative samples are not selected properly, the accuracy of the model will be reduced, and the training time will also be increased.

[0112] The positive sample image includes: using a picture containing a face to randomly crop and randomly scale to splice into images of different scales;

[0113] The negative sample image includes: using a picture without a face to randomly crop;

[0114] In the process of recognizing the face features, the number of negative samples is controlled, and the negative sample image and the positive sample image and the difficult example image are optimized, such as splicing multiple images containing a face after random cropping into an image and randomly cropping an image without a face, so that a single image can express more rich features. Through the above improvement of negative sample selection, the training speed of the model can be improved, and the convergence of the model can be accelerated.

[0115] Through the above improvement of positive sample and negative sample selection, the training speed of the model can be improved, the convergence of the model can be accelerated, and the accuracy can be improved.

[0116] The positive and negative samples are trained by using a support vector machine algorithm to obtain a trained model;

[0117] The model is used for negative sample difficult example recognition, specifically including: continuously scaling the negative samples in the training set until matching the template position, searching for matching through template sliding serial port, if the non-face area is misdetected by the classifier, the part of the image is intercepted and added to the negative sample, and / or the image and the corresponding confidence degree are selected from the model training, the image with a confidence degree greater than a preset threshold is selected, and the image after random cropping, random scaling and splicing is composed. In addition, the regularization is increased. Through difficult example recognition, the difficult example samples are fully mined, and the classification ability of the final model can be improved.

[0118] The difficult example samples are retrained to obtain a final classification model.

[0119] The final trained classifier is applied to detect a face picture, different sizes of the picture are scanned, HOG features are extracted, and the classifier is used for classification;

[0120] If the detection determines a face, the face is calibrated, and a final positioned face is obtained by using non-maximum suppression.

[0121] The face positioning of the person to be detected specifically includes:

[0122] The number of faces in each frame of image is obtained, and the minimum circumscribed rectangle of all detected faces is obtained;

[0123] The horizontal coordinate and the vertical coordinate P(x i ,y i ), i = 1, 2, 3, …, n of the center point of the minimum circumscribed rectangle of each face are obtained, wherein n represents the number of faces in the image;

[0124] The maximum value x k of the horizontal coordinates in all minimum circumscribed rectangles is obtained;

[0125] x k = max(x1, x2, x3, …, x n )

[0126] The face identified by the circumscribed matrix corresponding to P(x k ,y k ) is the face to be detected;

[0127] Specifically, in the embodiment, the case where multiple faces exist in the image is considered, and through the above simple operation, the face of the driver can be quickly identified from the multiple faces;

[0128] Specifically, after identifying the face, key features in the face need to be extracted for fatigue degree identification.

[0129] The position coordinates of the eyes and the mouth in the face to be detected are extracted;

[0130] In the prior art, the threshold values of the eyes EAR and the mouth MAR are commonly used to determine whether fatigue occurs: if the EAR value is less than the EAR threshold value or if the MAR value is greater than the MAR threshold value for a number of frames, the driver is reminded to take a rest. However, although it is not easy to set a global threshold value due to the difference in facial features among different people, natural expressions such as speaking or smiling or surprise often reduce the EAR and increase the MAR.

[0131] Specifically, the calculation formulas of EAR and MAR are as follows, see Figure 3 , Figure 4 :

[0132]

[0133]

[0134] In this embodiment, a support vector machine is used as the classification learning algorithm instead of using a threshold.

[0135] The SVM is then trained using the EAR and MAR features to classify the state of the driver, and three classes can be detected: awake, fatigue, and closed eyes;

[0136] In order to avoid detecting blinking as a fatigue state of the driver when the eyes are closed, a new feature is calculated, namely the duration of the closed eyes.

[0137] Therefore, each blink is identified and distinguished from a nap or a doze;

[0138] When a yawn is predicted, an alarm is activated to alert the driver, and a notification is displayed on the screen.

[0139] In particular, in this embodiment, the representation of the eye closure or blinking of the prior art is modified for calculating the fatigue judgment mode, and not simply as in the prior art by counting the number of times the eyes are closed for a period of time. The eye closure is influenced by various factors, and therefore in order to more reasonably assess whether the driver is in a fatigue closed eye or a closed eye operation caused by a change in expression, the present embodiment uses the following method:

[0140] In particular, the eye feature coefficient Leye_c is defined, where Leye_c = 1 when the eyes are open, and Leye_c = 0 when the eyes are closed.

[0141] The number of consecutive N L frames of images is counted, and the Leye_c of each frame of image is stored in a circular list using a first-in, first-out mechanism. When the circular list space is full, the old data is replaced by the new data;

[0142] In N L , the formula for calculating the eye closure frequency η is:

[0143]

[0144] where i represents the i-th frame of image;

[0145] The eye blink frequency is defined as the blink rate within a given duration of time. The eye feature coefficient Leye_o is defined, where Leye_o = +1 and -1 represent the start frame and stop frame of the closed eye event, respectively, and Leye_o = 0 for other cases.

[0146] The eye blink frequency is calculated as follows:

[0147]

[0148]

[0149] The mouth state feature coefficient is Lmo, Lmo=1 when the mouth is open, and Lmo=0 when the mouth is closed; the mouth opening and closing frequency of the driver is calculated by counting the number of times the mouth is open and closed in the last N L frames of images, and the Lmo of each frame of image is stored in a circular list. When the circular list space is full, the old data will be replaced by new data using a first-in, first-out mechanism;

[0150] The mouth closing frequency is defined as:

[0151]

[0152] Let the matrix be M The data set composed of M is trained using a support vector machine or a logistic regression to obtain a trained model;

[0153] The trained model is used to identify a new image to determine the fatigue degree of the driver;

[0154] Preferably, w1, w2, and w3 are weight systems, w1+w2+w3=1, and w3<w1, w3<w2

[0155] Let The data set composed of M' is trained using a support vector machine or a logistic regression to obtain a trained model;

[0156] The trained model is used to identify a new image to determine the fatigue degree of the driver.

[0157] Specifically, the embodiment provides a continuous face positioning algorithm method for a to-be-detected target, specifically comprising:

[0158] An image of a previous frame after preprocessing is obtained;

[0159] Face feature points are detected in the image of the previous frame using a face recognition algorithm;

[0160] The face to be detected is positioned;

[0161] The feature points of the face to be detected are stored;

[0162] An image of a current frame after preprocessing is obtained, and the same feature points as the face to be detected are searched for and matched in the image of the current frame;

[0163] It is determined whether the feature points are lost;

[0164] Specifically, the feature point loss judgment matches the feature points of the current frame image with the saved feature points, and calculates the number of matched feature points. If the feature points of the current frame image are all matched with the saved feature points, there is no loss of feature points; if there is a phenomenon of unmatched feature points in the current frame image or the saved feature points have no matching object, there is a loss of feature points.

[0165] When there is no loss of feature points, the next frame image is obtained and the next frame image is used to replace the current frame image to continue the recognition process.

[0166] When there is a loss of feature points, the unmatched feature points are deleted, and the matched feature points are saved.

[0167] The next frame image is obtained, the face of the person to be detected is positioned, and the new feature points are saved.

[0168] The next frame image is obtained, the face of the person to be detected is positioned, and the new feature points are saved.

[0169] The next frame image is obtained, the face of the person to be detected is positioned, and the new feature points are saved.

[0170] The next frame image is obtained, the face of the person to be detected is positioned, and the new feature points are saved.

[0171] Specifically, since the same driver's face needs to be collected in different video images, when there are multiple face targets, the same face needs to be found in different face positions in different images for feature comparison. In order to avoid the change of coordinate position caused by the movement of the person in the car, the recognized face is inconsistent. In this embodiment, the traditional face recognition method is not used, that is, the features of the previous face are compared with the features of the next face, and the same is considered as the same person. When this traditional method is used in different light environments, it can cause feature comparison error and thus recognition error. In this embodiment, the following method is used:

[0172] Arect represents the minimum circumscribed rectangle of the face to be detected in the previous frame image.

[0173] Brect represents the minimum circumscribed rectangle of the face to be detected in the current frame image.

[0174] The spatial correlation coefficient Ts is defined as:

[0175] Ts = 1 - Q(Arect, Brect) + D(Acenter, Bcenter) / Cmax,

[0176] Q(Arect, Brect) = (S Arect∩Brect ) / (S Arect +S Brect -S Arect∩Brect ),

[0177] SArect∩Brect denotes the area of the overlapping region of rectangle Arect and rectangle Brect, S Arect denotes

[0178] D(Acenter, Bcenter) is the distance between the center of the minimum circumscribed rectangle of the face to be detected in the previous frame and the minimum circumscribed rectangle of the face to be detected in the current frame;

[0179] Cmax defines the diagonal length of the minimum rectangle that can simultaneously cover rectangle Arect and rectangle Brect;

[0180] Ts< Thresh, wherein Thresh is a preset threshold, the value of which can be obtained by training with sample numbers. Obviously, the smaller the value of the spatial correlation coefficient Ts, the closer the face position in the previous frame and the face position in the current frame, and the more likely it is that the same face is recognized. The above method does not use feature comparison and can overcome insufficient lighting conditions, has low computational complexity, fast speed, and real-time performance.

[0181] Specifically, the above method is simple to calculate, fast, and can overcome the influence of different lighting conditions on image formation. However, if there are many people in the car and the relative change in position is relatively large, the above method will increase the number of times of misjudgment or failure to find the target object. Therefore, in order to solve this technical problem, the present embodiment also provides a method for recognizing fixed faces in different image frames, which has higher recognition accuracy than the above method and can overcome the misdetection or failure to detect caused by position changes, but has slightly larger computational complexity than the above method, and the specific calculation is as follows:

[0182] A(xi, yi) represents the pixel value of the face in the previous frame image at coordinates (xi, yi), and B(xi + Δx, yi + Δy) represents the pixel value of the face in the current frame image at coordinates (xi + Δx, yi + Δy);

[0183] Set the objective function as:

[0184]

[0185] wherein xi, yi are the coordinates of the pixels of the known face, Δx, Δy are the coordinates to be solved, and Z is the face recognition region of the previous frame image;

[0186] Define the function as:

[0187] Ri(dx, dy) = A(xi, yi) - B(xi + dx, yi + dy)

[0188] Then the objective function is: U(dx, dy) = ∑ xi,yi∈z ||Ri(dx, dy)||2 = R(dx, dy) T R(dx, dy)

[0189] optimizing the objective function to obtain an optimal solution of the objective function;

[0190] R(dx, dy) is expanded according to the first derivative at the point (xo, yo) to obtain the following equation:

[0191]

[0192] Similarly:

[0193] Definition: Then

[0194] R(dx, dy) is substituted into the cost function, and the condition of solving the optimal solution of the cost function is:

[0195] I T IΔdx = -I T m,

[0196] where I and m are known, therefore, Δdx is solved;

[0197] dx is updated, and the updating method is: dx = dx + Δdx

[0198] The updated dx is substituted into the objective function to calculate the error, if the error is greater than the preset threshold, then the above process is repeated until the error is less than the threshold.

[0199] Then the region of B(xi + dx, yi + dy) in the current frame image is the new face to be monitored, wherein xi, yi ∈ Z.

[0200] In the process of calculating the partial derivative, since the image pixel value only has a value when the coordinates are integers, when the coordinates xi + dx, yi + dy obtained in the current frame image are not integer values, the pixel value at the current coordinate position is calculated by interpolation method.

[0201] Specifically, the pixel is a discrete value, which cannot be differentiated. In the embodiment, the partial derivative is replaced by the gradient of the image, for example: the partial derivative of x is converted to the gradient of x in the x direction, and the partial derivative of y is converted to the gradient of y in the y direction.

[0202] Specifically, in the actual execution process, the extreme condition still appears the situation that the objective function cannot converge. In order to solve the problem and maintain the continuity of monitoring, the following method is adopted in the embodiment:

[0203] When the current frame image t does not converge with the t-1th frame image, the current frame image t is discarded, the t-1th frame image and the t+1th frame image are subjected to target recognition, if not converging, the t-1th frame image continues to be subjected to target recognition with the t+2th frame image, if converging, the result of the t+2th frame is taken as a reference for next frame target recognition; if not converging, the above loop is continuously executed, when the number of loops exceeds a preset threshold, the current recognition result is ended, and the face recognition process of the to-be-monitored person is restarted.

[0204] The above only describes the preferred embodiments of the present application, and the present application is not limited to the above embodiments. Those skilled in the art can clearly understand that the forms in the embodiments are not limited thereto, and the adjustment modes are also not limited thereto. It can be understood that other improvements and changes directly derived or thought of by those skilled in the art without departing from the basic concept of the present application should be considered to be included in the protection scope of the present application.

Claims

1. A method for monitoring fatigue of a driver of a vehicle, characterized by, At least comprising: Collecting images in the vehicle, performing first data analysis on the images to obtain facial features of the monitored person; extracting key features in the facial features for representing the degree of fatigue; According to the key features, evaluating the fatigue level of the monitored person; According to the fatigue level, controlling the vehicle to perform a fatigue reminding process; Wherein, the first data analysis processing includes: image preprocessing; obtaining the face image of the driver from the existing face in the initial frame image, and continuously searching and positioning the face image of the driver in the subsequently collected images for monitoring; Obtaining the face image of the driver from the existing face includes: Obtaining the number of faces in the image, and obtaining the minimum bounding rectangle of all detected faces; Obtain the horizontal coordinate and the vertical coordinate P(x i ,y i ) of the center point of the minimum circumscribed rectangle of each face, i = 1, 2, 3, …, n, wherein n represents the number of faces in the image; obtaining the maximum value x of the horizontal coordinates in all the minimum bounding rectangles k ; x k = max(x1,x2,x3,…,x n ) P(x k ,y k ) corresponds to the minimum circumscribed matrix identified by the face to be detected face; The detected face locates the face of the driver; Or, pre-acquire the face image of the driver, and match the pre-acquired face image of the driver in the existing face image to locate the driver image.

2. A method for monitoring fatigue of a driver of a vehicle as claimed in claim 1, wherein, The face feature recognition includes the following steps: Obtain the HOG feature operator of the positive and negative samples; train the positive and negative samples by using the support vector machine algorithm to obtain the trained model; use the model to identify the difficult examples of the negative samples; retrain the model by collecting the difficult example samples, and repeatedly train to obtain the final classification model; Apply the finally trained classifier to detect the face picture, slide scan the different sizes of the picture, extract the HOG feature, and classify by using the classifier; If the detection determines that it is a face, it is marked, and the final positioned face is obtained by using non-maximum suppression.

3. A method for monitoring fatigue of a driver of a vehicle as claimed in claim 1, wherein, The key features in the facial features for representing the fatigue degree include: eye key features, with eye closure frequency η and eye blink frequency characterization; Statistical consecutive N L frame image, the eye closed coefficient Leye_c, Leye_o eye blinking coefficient of each frame image is stored in the circular list, and the first-in first-out mechanism is adopted. When the circular list space is full, the old data will be replaced by the new data.

4. A method for monitoring fatigue of a driver of a vehicle as claimed in claim 3, wherein, In N L The formula for calculating the frequency of eye closure η in the frame image is: In N L In the frame image, the blinking frequency of the eye The calculation formula is Wherein, i represents the i-th frame image, the eye closed eye coefficient Leye_c, the eye blinking coefficient Leye_o; The eye blinking coefficient Leye_o respectively represents the start frame and stop frame of the closed eye event with +1 and-1, and other conditions Leye_o=0; The calculation formula of the eye blinking coefficient Leye_o is:

5. A method for monitoring fatigue of a driver of a vehicle as claimed in claim 1, wherein, The key features in the face features for representing the degree of fatigue include: the closing frequency of the mouth mo, and the calculation formula is as follows: Wherein, the mouth state feature coefficient Lmo, when the mouth is open, Lmo=1, when the mouth is closed, Lmo=0; Statistical consecutive N L frame image, Lmo of each frame image is stored in a circular list, and a first-in first-out mechanism is adopted, when the circular list space is full, the old data will be replaced by new data.

6. A method for monitoring fatigue of a driver of a vehicle as claimed in claim 1, wherein, Searching and positioning the face image of the driver includes: Obtain the last frame image after preprocessing, and detect the face feature points in the last frame image by using the face recognition algorithm; position the face to be detected, and store the feature points of the face to be detected; Obtain the current frame image after preprocessing, search for the same feature points as the face to be detected in the current frame image and match to obtain the feature points; Determine whether the feature points are lost, when the feature points are not lost, replace the current frame image with the last frame image, obtain the next frame image and replace the current frame image with the next frame image for cyclic matching; When the feature points are lost, delete the unmatched feature points and save the matched feature points, and replace the matched feature points with new feature points obtained in subsequent preset multiple images.

7. A method for monitoring fatigue of a driver of a vehicle as claimed in claim 1, wherein, The first data analysis processing includes: judging the light condition of the current vehicle environment, and performing image preprocessing on the image according to the light condition.

8. A method for monitoring fatigue of a driver of a vehicle as claimed in claim 7, wherein, The judgment of the light condition of the current vehicle interior environment comprises: judging the current light intensity according to the photoresistor, and if the light intensity is lower than a threshold, performing first image preprocessing on the collected image.

9. A method for monitoring fatigue of a driver of a vehicle as claimed in claim 7, wherein, When the lighting is poor, the image preprocessing method comprises: F(x) = CDF(x) 2 + K(x) x 2 x CDF(x) x (1 - CDF(x)); wherein Y = F(x) * 255; Y is the converted pixel value, Pm{.} represents the probability of event occurrence, X is the gray value of the image, x represents the classification value of the image pixel value; CDF(x) represents the cumulative distribution function; PDF(u) is the probability density distribution function of the X pixel value in the image; K(x) is a weight coefficient, used to represent the uncertainty degree of the event probability density.

10. A method for monitoring fatigue of a driver of a vehicle as claimed in claim 7, wherein, When the lighting is poor, the image preprocessing method comprises: Converting R(x, y) into the range of 0-255; where (x, y) is a pixel coordinate, b is a luminance adjustment parameter, R(x, y) is a converted pixel value, S(x, y) is a pixel value before conversion, g i (x, y) is a two-dimensional Gaussian distribution function, denotes a standard deviation.

Images