Intelligent vehicle-mounted box and driving fatigue monitoring method

Through intelligent vehicle-mounted boxes to collect and analyze the driver's eyelid and mouth characteristics, combined with machine learning models, the problems of large hardware resources and low accuracy in the prior art are solved, and accurate fatigue monitoring is achieved under complex lighting conditions.

CN113971796BActive Publication Date: 2024-11-01SHANGHAI HEQIAN ELECTRONICS TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111134902.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-27
Publication Date
2024-11-01
Estimated Expiration
2041-09-27

AI Technical Summary

Technical Problem

The existing automotive driving fatigue monitoring technology has problems such as high hardware resource consumption, inconsistent hardware configuration, and low accuracy, especially under complex lighting conditions.

Method used

Using intelligent car-mounted boxes to collect images through cameras, extract eyelid closure frequency, blink frequency and mouth closure frequency in facial features as fatigue evaluation indicators, and train them using support vector machines or logistic regression models, combining first-in-first-out mechanisms and image preprocessing technology to improve monitoring accuracy and real-time.

Benefits of technology

Accurate driver fatigue monitoring under different lighting conditions is achieved, false alarms are avoided, and different vehicle hardware configurations are adapted to real-time and accurate monitoring.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113971796B_ABST
    Figure CN113971796B_ABST
Patent Text Reader

Abstract

The present invention provides an intelligent vehicle-mounted box and a driving fatigue monitoring method, which uses a camera on the intelligent vehicle-mounted box to collect images of the driver, analyzes and processes the images, extracts key features of facial features for characterizing the degree of fatigue, and evaluates the driver's fatigue level. The key features characterizing the degree of fatigue include: eye closure frequency and eye blinking frequency, and the key features of the mouth include the closure frequency of the mouth. The technical solution provided by the present invention includes accurately capturing the driver's face from multiple faces in images of different frames and extracting key features, which can reduce the misjudgment rate of fatigue monitoring while ensuring real-time monitoring.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to automobile safe driving, and in particular to an intelligent vehicle-mounted box and a driving fatigue monitoring method. Background Art

[0002] Traffic accidents are one of the most important factors in the occurrence of accidents, especially on rural roads and highways. Fatigue can reduce the driver's perception and decision-making ability to control the vehicle. Studies have shown that drivers usually feel tired after driving for 1 hour. In single-vehicle collisions (accidents in which only one vehicle is damaged) or collisions involving heavy vehicles, up to 50% of accidents are related to the driver's lack of vigilance. At this stage, more fatigue detection technologies are applied to automobiles for monitoring, but after research by the applicant, it was found that the fatigue detection system is carried out using a central control host or an in-vehicle entertainment host mounted on the vehicle. This will undoubtedly consume the resources of the central control host or the in-vehicle entertainment host, and will inevitably increase the host's compliance. Different models have high configurations, and the vehicle hardware configuration is different. It may not have the computing power and hardware conditions for fatigue detection, resulting in the driver's demand being unable to be installed due to hardware or system support. In addition, the current image processing-based driving behavior monitoring technology mainly uses image processing and traditional machine learning methods to extract the driver's facial features or body posture features collected by the on-board camera to monitor the driver's driving behavior. The hardware configuration is relatively simple, and only a camera and a processor are required to achieve it. However, existing image-based driver fatigue driving methods are affected by external environments such as complex lighting conditions when obtaining driver facial information, and only use simple thresholds for judgment, which leads to low accuracy. Summary of the invention

[0003] Based on the defects in the prior art, the present invention provides a driving fatigue monitoring method of an intelligent vehicle-mounted box, which at least includes:

[0004] The camera mounted on the smart vehicle box collects images, performs a first data analysis on the images to obtain facial features of the monitored person; extracts key features from the facial features for characterizing the degree of fatigue; and evaluates the fatigue level of the monitored person based on the key features of the degree of fatigue;

[0005] Among them, the key features that characterize fatigue level include: eye closure frequency η and eye blinking frequency The closing frequency of the mouth, mo;

[0006] The mo in each frame image, η forms a training data set, and uses support vector machine or logistic regression for training to obtain the trained classification model;

[0007] Use the trained classification model to classify and identify new images to obtain the fatigue level of the monitored person;

[0008] Statistics continuous N L Frame image, the eye closure coefficient Leye_c, eye blink coefficient Leye_o, and mouth state characteristic coefficient Lmo of each frame image are stored in a circular list, using a first-in-first-out mechanism. When the circular list space is full, the old data will be replaced by the new data.

[0009] In N L In the frame image, the calculation formula of eye closure frequency η is:

[0010]

[0011] Eye blink frequency The calculation formula is

[0012]

[0013] Among them, i represents the i-th frame image, eye closing coefficient Leye_c, Leye_o eye blinking coefficient;

[0014] The eye blink coefficient Leye_o represents the start frame and stop frame of the eye closing event with +1 and -1 respectively, and Leye_o=0 for other conditions;

[0015] The calculation formula of eye blink coefficient Leye_o is:

[0016]

[0017] The key features used to characterize the degree of fatigue in facial features include: mouth state feature coefficient Lmo,

[0018] When the mouth is open, Lmo=1, when the mouth is closed, Lmo=0;

[0019] Statistics continuous N L Frame image, store the Lmo of each frame image in a circular list, using a first-in-first-out mechanism. When the circular list space is full, the old data will be replaced by the new data.

[0020] The closing frequency of the mouth is then defined as:

[0021]

[0022] The monitored persons include the driver or persons in designated areas within the vehicle.

[0023] The driving fatigue monitoring method of the intelligent vehicle box is further preferably that the first data analysis process includes: acquiring a human face existing in an image of an initial frame, acquiring a monitored human face image from the existing human faces, and continuously searching and locating the monitored human face image in subsequent acquired images in a loop;

[0024] The driving fatigue monitoring method of the intelligent vehicle-mounted box is further preferably that obtaining the monitored face image from the existing face includes: obtaining the monitored face image in advance, matching the pre-acquired monitored face image with the existing face image in the current image through module matching to locate the monitored person image;

[0025] Or when the monitored face is a driver, obtaining the monitored face image from the existing faces includes:

[0026] Get the number of faces in the image and the minimum bounding rectangle of all detected faces;

[0027] Get the horizontal coordinate and median coordinate P(x i ,y i ), i = 1, 2, 3, ..., n, where n represents the number of faces in the image;

[0028] Get the maximum value x of the horizontal coordinates in all minimum bounding rectangles k ;

[0029] x k =max(x1, x2, x3, ..., x n )

[0030] P(x k ,y k ) The face identified by the minimum circumscribed matrix corresponding to is the monitored face;

[0031] The monitored face is determined to be the driver's face.

[0032] The driving fatigue monitoring method of the intelligent vehicle box is further preferably that the first data analysis process includes: facial feature recognition, and the facial feature recognition includes:

[0033] Obtain the HOG feature operators of positive and negative samples; use the support vector machine algorithm to train positive and negative samples to obtain the trained model; use the model to identify difficult examples of negative samples; collect difficult samples to retrain the model, and repeatedly train to obtain the final classification model;

[0034] The final trained classifier is used to detect face images, and the images of different sizes are scanned, HOG features are extracted, and then classified using the classifier;

[0035] If the detection is determined to be a face, it will be calibrated and non-maximum suppression will be used to obtain the final located face.

[0036] The driving fatigue monitoring method of the intelligent vehicle box is further preferably, obtaining the minimum circumscribed rectangle Arect of the monitored face contour in the previous frame image; obtaining the minimum circumscribed rectangles B(i)rect of all face contours in the current frame image, i=1, 2, ..., k; calculating the position space correlation coefficient T(i)s according to Arect and B(i)rect, i=1, 2, ..., k; comparing the space correlation coefficient T(i)s with a preset threshold, and obtaining all space correlation coefficients T(i)s less than the preset threshold;

[0037] Solve the minimum value Tmin_s among all spatial correlation coefficients T(i)s that are less than the threshold, and then obtain the corresponding B(i)rect in the current frame image according to Tmin_s. The face surrounded by B(i)rect is the monitored face image to be searched and located.

[0038] Among them, T(i)s=1-Q(Arect, B(i)rect)+D(Acenter, B(i)center) / Cmax,

[0039] Where, Q(Arect, B(i)rect)=(S Arect∩B(i)rect ) / (S Arect +S B(i)rect -S Arect∩B(i)rect ),

[0040] S Arect∩B(i)rect represents the area of ​​the overlap between the matrix Arect and the rectangle B(i)rect, S Arect express

[0041] D(Acenter, B(i)center) is the distance between the center of the minimum bounding rectangle of the face to be tested in the previous frame and the minimum bounding rectangle of the face to be tested in the current frame image;

[0042] Cmax defines the diagonal length of the smallest rectangle that can cover both rectangle Arect and rectangle B(i)rect.

[0043] The driving fatigue monitoring method of the intelligent vehicle box, further preferably, another method for searching and locating the driver's face image includes:

[0044] Obtain the last frame of image after preprocessing, and use the face recognition algorithm to detect the facial feature points in the last frame of image; locate the monitored face, and store the feature points of the monitored face;

[0045] Obtain the preprocessed current frame image, search for feature points identical to the monitored face in the current frame image, and perform matching to obtain the feature points;

[0046] Determine whether there is loss of feature points. If there is no loss of feature points, replace the current frame image with the previous frame image, obtain the next frame image and use the next frame image to replace the current frame image for cyclic matching;

[0047] When there is a loss of feature points, the unmatched feature points are deleted and the matched feature points are saved, and new feature points are obtained in the subsequent preset multi-images to replace the matched feature points.

[0048] The method of obtaining new feature points in subsequent preset multi-images to replace the matched feature points specifically includes:

[0049] Get the next frame of image, locate the monitored face, and save new feature points; get the next frame of image after the next frame of image, and search for matching feature points;

[0050] Determine whether the matched feature points include the number of deleted feature points. If so, select new feature points to replace the lost feature points, save the new feature points and update them;

[0051] If not, continue to loop through multiple frames of images to obtain matching feature points; if the number of loops exceeds a preset value, start again to determine new feature points.

[0052] The driving fatigue monitoring method of the intelligent vehicle box is further preferably that searching and locating the driver's face image includes:

[0053] Get the pixel value A(xi, yi) of the monitored face coordinate area in the previous frame image, and get the pixel value B(xi+Δx, yi+Δx) of the monitored face coordinate area in the current frame image; where xi and yi are the pixel coordinates of the face area

[0054] Set the objective function:

[0055] U(dx,dy)=∑ xi,yi∈z ||Ri(dx,dy)|| 2 =R(dx,dy) T R(dx,dy)

[0056] Wherein, dx, dy are optimized coordinates, which are used to indicate the relative position of the coordinates of the monitored face region in the current frame image relative to the coordinates of the monitored face region in the previous frame image;

[0057] Among them, Ri(dx,dy)=A(xi,yi)-B(xi+dx,yi+dy), where xi,yi are the pixel coordinates of the face area, and Ri(dx,dy) represents the pixel difference between the previous frame image and the current frame image;

[0058] Substitute the formula of Ri(dx, dy) expanded at the point (xi, yi) according to the first-order derivative into the objective function;

[0059] Optimize the objective function to obtain the optimal solution Δdx of the objective function;

[0060] Update dx, update method: dx = dx + Δdx

[0061] Substitute the updated dx into the objective function and calculate the error. If the error is greater than the preset threshold, the objective function is optimized repeatedly until the error is less than the threshold.

[0062] The driving fatigue monitoring method of the smart vehicle box is further preferably that the first data analysis and processing includes: judging the lighting conditions of the current in-vehicle environment, and performing image preprocessing on the image according to the lighting conditions.

[0063] The driving fatigue monitoring method of the intelligent vehicle box is further preferably, when the lighting is poor, the image preprocessing method includes:

[0064] F(x)=CDF(x) 2 +K(x)×2×CDF(x)×(1-CDF(x));

[0065] in,

[0066] Y = F(x) × 255;

[0067] Y is the converted pixel value, Pm{.} represents the probability of the event, X is the gray value of the image, and x represents the classification value of the image pixel value; CDF(x) represents the cumulative distribution function; PDF(u) is the probability density distribution function of the X pixel value in the image; K(x) is the weight coefficient, which is used to characterize the uncertainty of the event probability density;

[0068] Or use image preprocessing methods including:

[0069]

[0070]

[0071]

[0072] Convert R(x, y) to the range of 0 to 255;

[0073] Among them, (x, y) is the pixel coordinate, b brightness is the adjustment parameter, R(x, y) is the pixel value after transformation, S(x, y) is the pixel value before transformation, g i (x, y) is a two-dimensional Gaussian distribution function, σ i 2 Represents standard deviation.

[0074] An intelligent vehicle-mounted box, used for monitoring fatigue of a vehicle driver, comprises: a vehicle-mounted intelligent terminal body, a PCBA circuit board installed on the body;

[0075] The PCBA circuit board is provided with: a communication bus, a SoC chip, a camera, and a DSP chip, wherein the SoC chip, the camera, and the DSP chip are connected via the communication bus; at least one camera is used to collect an image of the driver in the car in real time;

[0076] The DSP chip is used for data processing of fatigue detection images;

[0077] It also includes a driving fatigue monitoring method for the above-mentioned smart vehicle-mounted box.

[0078] A vehicle-mounted box further comprises: a display screen and at least one multi-function trigger button arranged on the vehicle-mounted intelligent terminal body;

[0079] The vehicle-mounted intelligent terminal body is provided with a fixing component for mounting the vehicle-mounted intelligent terminal body at a preset position in the vehicle.

[0080] Beneficial effects:

[0081] 1. In the prior art, the fatigue recognition method only considers the situation where only the driver is present in the camera. However, in the actual imaging process, different imaging ranges exist in different positions of the camera, and most positions are often able to capture other members of the parking space. In the fatigue monitoring process of the present invention, the accurate positioning of the driver's face position from a large number of faces is considered, and in the subsequent monitoring process, the driver's position in the image is continuously positioned to avoid incorrect monitoring of the face, thereby generating false alarms.

[0082] 2. In the driving fatigue monitoring method of the smart car box of the present invention, the traditional method is improved. For the key technical features of the eyes, only the closing frequency of the eyes is used for calculation. When the eye closure statistics within a period of time reach a preset value, it is considered that the driver is in a fatigued state. However, it ignores the fact that the driver's blinking will affect the accuracy. Therefore, in addition to the eye closure frequency, the present invention also considers the eye blinking frequency, and more reasonably evaluates and calculates whether the driver is in a state of fatigue or a closed eye operation caused by facial expression changes. It is not a simple statistical number of times, but after extracting features, a model is trained by a support vector machine or a logistic regression to obtain a classification model, so that its judgment result is more reasonable. This method has a small amount of calculation and can ensure real-time performance.

[0083] 3. Since the present invention needs to collect the face of the same driver in different images, when there are multiple face targets, it is necessary to find the same face in different face positions in different images for feature comparison in order to avoid inconsistent recognized faces due to changes in coordinate positions caused by the movement of people on the vehicle. In addition, traditional methods will cause feature comparison errors under different lighting environments, thereby causing recognition errors. The present invention recognizes fixed faces in different image frames, and uses the spatial correlation coefficient between the previous frame face image and the current frame face image to determine the face to be matched, which can overcome the problem of positioning errors caused by lighting changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] The following drawings are only used to schematically illustrate and explain the present invention, and do not limit the scope of the present invention.

[0085] Figure 1 This is an algorithm logic diagram for fatigue monitoring of a monitored target in one embodiment of the present invention.

[0086] Figure 2 This is a logic diagram of a face location algorithm for a continuously monitored target in one embodiment of the present invention.

[0087] Figure 3 Schematic diagram of eye feature structure in one embodiment of the present invention.

[0088] Figure 4 Schematic diagram of the structure of the mouth feature in one embodiment of the present invention.

[0089] Figure 5 The figure is a schematic diagram of the structure of the intelligent vehicle box in one embodiment of the present invention.

[0090] Figure 6 The figure is a schematic diagram of the apparent structure of the smart vehicle box in one embodiment of the present invention. DETAILED DESCRIPTION

[0091] In order to have a clearer understanding of the technical features, purposes and effects of this invention, the specific implementation methods of the present invention are now described with reference to the accompanying drawings, and the same reference numerals in each figure represent the same parts. In order to make the drawings concise, the parts related to the present invention are schematically shown in each figure, and do not represent the actual structure of the product. In addition, in order to make the drawings concise and easy to understand, in some figures, only one of the parts with the same structure or function is schematically drawn or marked.

[0092] Regarding control systems, functional modules, and application programs (APP), it is well known to those skilled in the art that they can take any appropriate form, either hardware or software, either multiple discrete functional modules or multiple functional units integrated into one hardware. In the simplest form, the control system can be a controller, such as a combinational logic controller, a microprogram controller, etc., as long as the operations described in this application can be implemented. Of course, the control system can also be integrated into a physical device as different modules, which does not deviate from the basic principles and protection scope of the present invention.

[0093] The term “connection” in the present invention may include direct connection, indirect connection, communication connection, and electrical connection, unless otherwise specified.

[0094] The terms used herein are for the purpose of describing specific embodiments only and are not intended to limit the present disclosure. As used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will also be understood that when used in the specification, the terms "include" and / or "comprise" refer to the presence of stated features, values, steps, operations, elements and / or components, but do not exclude the presence or additional addition of one or more other features, values, steps, operations, elements, components and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the listed related items.

[0095] It should be understood that the term "vehicle" or "vehicular" or other similar terms used herein generally include motor vehicles, such as passenger cars including sport utility vehicles (SUVs), buses, trucks, various commercial vehicles, ships including various boats, marine vessels, aircraft, etc., and include hybrid vehicles, electric vehicles, plug-in hybrid electric vehicles, hydrogen-powered vehicles, and other alternative fuel vehicles (e.g., fuels derived from non-petroleum energy sources). As referred to herein, a hybrid vehicle is a vehicle having two or more power sources, such as both gasoline-powered and electric-powered vehicles.

[0096] The first embodiment of the present invention provides a fatigue monitoring method of an intelligent vehicle box, which is used to monitor the fatigue state of a vehicle driver. Figures 1 to 4 , specifically including:

[0097] Capturing images through a camera, and performing a first data analysis process on the images to obtain facial features of the driver to be monitored;

[0098] Extract key features from facial features to characterize fatigue levels;

[0099] Assess driver fatigue level based on key characteristics of fatigue level;

[0100] According to the fatigue level, the vehicle is controlled to execute the fatigue reminder process.

[0101] The first data analysis process includes determining the current light intensity according to the photoresistor, and if the light intensity is lower than a threshold, performing a first image preprocessing on the collected image.

[0102] Due to poor imaging quality when the lighting is poor, the number of black areas increases when the pixels are displayed, which affects the subsequent face positioning and face feature extraction. Therefore, in order to accurately extract face positioning and face feature extraction when the lighting environment in the car is poor, if the lighting is poor, the first image preprocessing is started to compensate for the underexposure.

[0103] Specifically, the first image preprocessing is only used when the exposure is insufficient, which is different from the prior art. In the current image recognition process, in order to improve the robustness of the algorithm, the same image preprocessing is used for situations with or without illumination. During the research and development process, the applicant found that if the same image preprocessing scheme is used for situations with or without illumination, the imaging of conditions with sufficient illumination will cause brightness concentration, making the subsequent feature extraction of face detection inaccurate, especially in the eye area.

[0104] Therefore, in this embodiment, the lighting condition of the acquired image is judged, and the image enhancement process is performed only when the lighting is insufficient.

[0105] Specifically, the image illumination condition judgment includes: firstly, converting the RGB image into the HSV color space, obtaining the standard deviation S of the V channel 2 , according to the standard deviation S of the V channel 2 Distribution map to determine the lighting conditions;

[0106] The judgment is made based on the time of the current area. For example, in summer, the sun rises early. At 6:30 in Beijing, the light intensity is considered sufficient. After 5:00 in the afternoon, the light intensity is considered insufficient. By setting a mapping table of time, geographical location, and illumination, the current lighting conditions are judged according to the mapping table, and image enhancement is started according to the lighting conditions.

[0107] In addition, it is more accurate, but it will also bring the problem of increased calculation amount. The neural network model is used for judgment. Specifically: by collecting images with sufficient lighting and insufficient lighting conditions, and marking the images, the convolutional neural network is used for training and the classification model is obtained.

[0108] When the lighting is poor, a first image preprocessing method is used, and the first image preprocessing method includes:

[0109] F(x)=CDF(x) 2 +K(x)×2×CDF(x)×(1-CDF(x));

[0110]

[0111] Y = F(x) × 255;

[0112] Pm{.} represents the probability of an event, X represents the grayscale value of the image, and x represents the classification value of the image pixel value; CDF(x) represents the cumulative distribution function; PDF(u) is the probability density distribution function of the X pixel value in the image; K(x) is the weight coefficient, which is used to characterize the uncertainty of the event probability density. Y is the converted pixel value; through the above simple image conversion, it is possible to overcome the problem of improving the pixel distribution in the image when the light is insufficient, and improve the overall value of the dark pixels in the image, which can help with the subsequent face positioning and face feature extraction.

[0113] Specifically, although the above method has a good processing effect and is very fast, it is a preferred solution for embedded devices, but it will lose a certain color value.

[0114] Therefore, this embodiment provides another solution for processing an image with insufficient lighting, which specifically includes:

[0115]

[0116]

[0117]

[0118] Convert R(x, y) to a range of 0 to 255 so that it can be displayed.

[0119] Among them, (x, y) is the pixel coordinate, b brightness is the adjustment parameter, R(x, y) is the pixel value after transformation, S(x, y) is the pixel value before transformation, g i (x, y) is a two-dimensional Gaussian distribution function, By taking different values, you can adjust the pixel range of the image display:

[0120] By introducing w(x, y), a larger image pixel range can be converted into a smaller output range, the pixel range of the image can be balanced, and the imaging contrast in insufficient light can be increased.

[0121] The first data analysis and processing also includes face location and face location of the monitored person;

[0122] In the prior art, fatigue recognition methods only consider the situation where only the driver is in the camera. However, in the actual imaging process, different imaging ranges exist at different positions of the camera, and most positions can often capture other members of the parking space. How to accurately capture the driver in the image becomes one of the technical problems to be solved in this embodiment.

[0123] It should be noted that, in this embodiment, the monitored person may be the driver, or other persons in the vehicle, such as the person in the back seat or the person in the front seat;

[0124] Generally, the person being monitored is the driver;

[0125] See also Figure 1 , the camera in this embodiment is mounted on the streaming media mirror of the front seat; preferably, in this embodiment, a smart car box with a camera is mounted on the streaming media mirror to observe the driver or other positions in the car, such as the front of the cab, or the roof can be installed at a certain angle;

[0126] The smart car box is used for image processing, so that fatigue monitoring does not consume the resources of the central control host and is isolated from the host system.

[0127] Face localization includes: identifying all faces in an image, specifically:

[0128] Use Haar classifier based on AdaBoost algorithm, Haar-like wavelet features and integral image method for face detection;

[0129] Prepare target and non-target samples, calculate eigenvalues ​​and integral graphs, and filter out T excellent eigenvalues; put these T weak classifiers into AdaBoost for training; build a cascade classifier for detection to obtain the position of the face and the position of the eyes and mark them in the table;

[0130] This embodiment also provides another method for detecting a face;

[0131] Extract HOG features from the positive sample data set and obtain the HOG feature operator;

[0132] Extract HOG features from the negative sample data set and obtain the HOG feature operator, wherein the amount of data in the negative sample data set is set to be greater than the number of samples in the positive sample data set;

[0133] Specifically, in actual models, the number of negative samples is much larger than the number of positive samples. If the negative samples are not selected properly, the accuracy of the model will be reduced and the training time will be increased. During the training process, the model often recognizes some negative samples as positive samples with a high confidence level;

[0134] Positive sample images include: randomly cropped and randomly scaled pictures containing faces and spliced ​​into images of different scales;

[0135] Negative sample images include: randomly cropping images without faces;

[0136] Use the support vector machine algorithm to train positive and negative samples to obtain the trained model;

[0137] The model is used to identify difficult negative examples, specifically including: continuously scaling the negative samples in the training set until they match the template, searching for matches through the template sliding serial port, selecting images that identify negative samples as positive samples and the corresponding confidence levels, selecting images with confidence levels greater than a preset threshold, and randomly cropping, scaling, and splicing images to form the resulting image; in addition, regularization is added.

[0138] Compared with the traditional method, this embodiment improves the positive samples, negative samples and difficult samples. Through the above-mentioned selection improvements of positive samples, negative samples and difficult example recognition, the training speed of the model can be increased, the convergence of the model can be accelerated, and the accuracy can be improved.

[0139] The model is retrained by collecting difficult samples and repeatedly trained to obtain the final classification model.

[0140] The finally trained classifier is used to detect the face image, and the image is scanned at different sizes to extract the HOG features and then classified using the classifier.

[0141] If the detection is determined to be a face, it will be calibrated and non-maximum suppression will be used to obtain the final located face.

[0142] The facial positioning of the monitored person specifically includes:

[0143] Get the number of faces in each frame and the minimum bounding rectangle of all detected faces;

[0144] Get the horizontal and vertical coordinates of the center point of the minimum circumscribed rectangle of each face P(x i ,y i ), i = 1, 2, 3, ..., n, where n represents the number of faces in the image;

[0145] Get the maximum value x of the horizontal coordinates in all minimum bounding rectanglesk ;

[0146] x k =max(x1, x2, x3, ..., x n )

[0147] P(x k ,y k ) The face identified by the external matrix corresponding to is the monitored face;

[0148] Specifically, in this embodiment, the situation where there are multiple faces in the image is considered, and the driver's face can be quickly identified from the multiple faces through the above simple calculation;

[0149] Specifically, after identifying the face, it is necessary to extract key features from the face to identify the fatigue level.

[0150] Extract the position coordinates of the eyes and mouth in the monitored face;

[0151] In the prior art, it is common to use the thresholds of eye EAR and mouth MAR to judge whether fatigue occurs: if the EAR value is less than the EAR threshold or if the MAR value is greater than the MAR threshold for a few frames, the driver is reminded to take a break. However, although it is not easy to set a global threshold because facial features vary from person to person, natural expressions, such as talking, smiling, or being surprised, tend to reduce the EAR and increase the MAR.

[0152] Specifically, the calculation formulas for EAR and MAR are as follows, see Figure 3 , Figure 4 :

[0153]

[0154]

[0155] In this embodiment, a support vector machine is used as a classification learning algorithm instead of using a threshold.

[0156] Then, the EAR and MAR features are used to train the SVM to classify the driver’s status, and three categories can be detected: awake, fatigued, and eyes closed;

[0157] Therefore, recognize each blink and distinguish it from a nap or drowsiness;

[0158] When a yawn is predicted, an alarm is activated to alert the driver and a notification is displayed on the screen.

[0159] Specifically, in this embodiment, the prior art characterization of eye closure or blinking is modified to calculate the fatigue judgment method, rather than simply using the number of times the eyes are closed continuously within a period of time as in the prior art. Eye closure is affected by many factors, so in order to more reasonably evaluate and calculate whether the driver is closing his eyes due to fatigue or the closing operation caused by expression changes, this embodiment adopts the following method:

[0160] Specifically, the eye characteristic coefficient Leye_c is defined, when the eyes are open, Leye_c=1, when the eyes are closed, Leye_c=0;

[0161] Statistics continuous N L Frame image, store Leye_c of each frame image in a circular list, using a first-in-first-out mechanism. When the circular list space is full, the old data will be replaced by the new data.

[0162] In N L In the calculation formula of eye closure frequency η, the eye closure frequency η is:

[0163]

[0164] Wherein, i represents the i-th frame image;

[0165] Define eye blink frequency Defined as the blink rate within a given duration; define the eye characteristic coefficient Leye_o, where Leye_o represents the start frame and stop frame of the eye closing event with +1 and -1 respectively, and Leye_o = 0 in other cases;

[0166] Eye blink frequency The calculation of is as follows:

[0167]

[0168]

[0169] The mouth state characteristic coefficient is Lmo, when the mouth is open, Lmo = 1, when the mouth is closed, Lmo = 0; statistics of continuous N L Frame image, store the Lmo of each frame image in a circular list, using a first-in-first-out mechanism. When the circular list space is full, the old data will be replaced by the new data.

[0170] The closing frequency of the mouth is then defined as:

[0171]

[0172] Let matrix The data set composed of M is trained using a support vector machine to obtain a training model;

[0173] Use the trained model to identify new images and determine the driver's fatigue level;

[0174] Preferably, w1, w2, w3 are weighted systems, w1+w2+w3=1, and w3<w1, w3<w2

[0175] set up The data set composed of M' is trained using a support vector machine to obtain the training model;

[0176] The trained model is used to recognize new images and determine the driver's fatigue level.

[0177] Specifically, this embodiment provides a continuous face location algorithm method for a monitored target, which specifically includes:

[0178] Get the last frame of image after preprocessing;

[0179] Detect facial feature points using a face recognition algorithm in the previous frame image;

[0180] Locate the monitored face;

[0181] The feature points of the monitored face are stored;

[0182] Obtain the preprocessed current frame image, search for feature points identical to the monitored face in the current frame image and perform matching;

[0183] Determine whether the feature points are lost;

[0184] Specifically, the feature point loss is determined by matching the feature points of the current frame image with the saved feature points, and calculating the number of matched feature points. If the feature points of the current frame image match the saved feature points one by one, then there is no feature point loss; if there is an unmatched feature point in the current frame image or the saved feature points have no matching objects, then there is a feature point loss;

[0185] When there is no loss of feature points, the next frame of image is obtained and the next frame of image is used to replace the current frame of image to continue the recognition process;

[0186] When there is loss of feature points, the unmatched feature points are deleted and the matched feature points are saved;

[0187] Get the next frame of image, locate the monitored face, and save the new feature points;

[0188] Get the next frame of the next frame and search for matching feature points;

[0189] Select new feature points to replace the lost feature points;

[0190] Save the feature points and update them.

[0191] Specifically, since the face of the same driver needs to be captured in different video images, when there are multiple face targets, it is necessary to find the same face in different face positions in different images for feature comparison, in order to avoid inconsistent recognized faces due to changes in coordinate positions caused by the movement of people on the vehicle. This embodiment does not use the traditional face recognition method, that is, comparing the features of the previous face with the features of the next face, and if they are the same, they are considered to be the same person. This traditional method will lead to feature comparison errors under different lighting environments, thereby leading to recognition errors. In this embodiment, the following method is used:

[0192] Set Arect to represent the minimum bounding rectangle of the face to be tested in the previous frame image;

[0193] Set Brect to represent the minimum bounding rectangle of the face to be tested in this frame image;

[0194] The characterization function Ts is defined as:

[0195] Ts=1-Q(Arect, Brect)+D(Acenter, Bcenter) / Cmax,

[0196] Where, Q(Arect, Brect)=(S Arect∩Brect ) / (S Arect +S Brect -S Arect∩Brect ),

[0197] S Arect∩Brect S represents the area of ​​the overlap between Arect and rectangle Brect. Arect express

[0198] D(Acenter, Bcenter) is the distance between the center of the minimum bounding rectangle of the face to be tested in the previous frame and the minimum bounding rectangle of the face to be tested in the current frame image;

[0199] Cmax defines the diagonal length of the smallest rectangle that can cover both rectangle Arect and rectangle Brect;

[0200] Specifically, when there are multiple face images, the minimum circumscribed rectangle of all detected face contours in the current frame image needs to be involved in the above calculation to obtain the optimal solution.

[0201] Get the minimum enclosing rectangle Arect of the monitored face contour in the previous frame image;

[0202] Obtain the minimum circumscribed rectangles B(i)rect of all face contours in the current frame image, i=1, 2, ..., k;

[0203] Calculate the position space correlation coefficient T(i)s according to Arect and B(i)rect, where i=1, 2, ..., k;

[0204] Compare T(i)s with a preset threshold, and obtain all T(i)s that are less than the preset threshold;

[0205] Solve for the minimum value Tmin_s among all values ​​less than the threshold T(i)s, and then obtain the corresponding B(i)rect in the current frame image according to Tmin_s. The face surrounded by B(i)rect is the face image of the driver being searched and located.

[0206] Among them, T(i)s=1-Q(Arect, B(i)rect)+D(Acenter, B(i)center) / Cmax,

[0207] Where, Q(Arect, B(i)rect)=(S Arect∩B(i)rect ) / (S Arect +S B(i)rect -S Arect∩B(i)rect ),

[0208] S Arect∩B(i)rect represents the area of ​​the overlap between the matrix Arect and the rectangle B(i)rect, S Arect express

[0209] D(Acenter, B(i)center) is the distance between the center of the minimum bounding rectangle of the face to be tested in the previous frame and the minimum bounding rectangle of the face to be tested in the current frame image;

[0210] Cmax defines the diagonal length of the smallest rectangle that can cover both rectangle Arect and rectangle B(i)rect;

[0211] Ts<Thresh, where Thresh is a preset threshold value, and its value can be obtained by training with the number of samples. Obviously, the smaller the value of Ts, the closer the face position of the previous frame image is to the face position of the current frame image, indicating that the same face is recognized. The above method does not use feature comparison, can overcome the situation of insufficient lighting, has low computational complexity, fast speed, and is real-time.

[0212] Specifically, the above method is simple to calculate, fast, and can overcome the influence of different lighting conditions on image formation. However, if there are many people in the car and the position movement changes relatively greatly, the above method will have more misjudgments or fail to find the target object. Therefore, in order to solve this technical problem, this implementation also provides a method for identifying fixed faces in different image frames. The recognition accuracy is higher than the above method, and it can overcome the misdetection or failure to detect caused by position changes. However, the amount of calculation is slightly larger than that of the previous method, specifically:

[0213] A(xi, yi) represents the pixel value of the face at the coordinate (xi, yi) in the previous frame image, and B(xi+Δx, yi+Δx) represents the pixel value of the face at the coordinate (xi+Δx, yi+Δy) in the current frame image;

[0214] Set the objective function:

[0215]

[0216] Among them, xi, yi are the coordinates of the pixels of the known face, Δx, Δy are the coordinates to be solved, and Z is the face recognition area of ​​the previous frame image;

[0217] Define the function:

[0218] Ri(dx,dy)=A(xi,yi)-B(xi+dx,yi+dy)

[0219] Then the objective function is: U(dx,dy)=∑ xi,yi ∈z||Ri(dx,dy)|| 2 =R(dx,dy) T R(dx,dy)

[0220] Optimize the objective function to obtain the optimal solution of the objective function;

[0221] Expand R(dx, dy) at the point (xo, yo) according to the first-order derivative to obtain the following formula:

[0222]

[0223] Similarly:

[0224] definition: but

[0225] Substitute the formula (1) after R(dx, dy) is expanded into the objective function. The condition for obtaining the optimal solution by solving the objective function is:

[0226] I T IΔdx=-I Tm,

[0227] Among them, I and m are known, so solve for Δdx;

[0228] Update dx in the following way: dx = dx + Δdx

[0229] Substitute the updated dx into the objective function and calculate the error. If the error is greater than the preset threshold, continue to repeat the above calculation process until the error is less than the threshold;

[0230] At this time, the area B(xi+dx,yi+dy) in the current frame image is the new face to be monitored, where xi,yi∈Z.

[0231] In the process of partial derivative calculation, since the image pixel value has a value only when the coordinate is an integer, when the coordinates xi+dx, yi+dy obtained in the current frame image are not integer values, the pixel value of the current coordinate position is calculated by interpolation.

[0232] Specifically, pixels are discrete values, and partial derivatives cannot be calculated. In this embodiment, partial derivatives are replaced by gradients of the image. For example, partial derivatives of x are converted to gradients of x in the x direction, and partial derivatives of y are converted to gradients of y in the y direction.

[0233] Specifically, in the actual execution process, the objective function may still fail to converge under extreme conditions. To solve this problem and maintain the continuity of monitoring, the following method is adopted in this embodiment:

[0234] When the current frame image t and the t-1 frame image do not converge, the current frame image t is discarded, and the t-1 frame and the t+1 frame image are used for target recognition. If they do not converge, the t-1 frame image continues to be used for target recognition with the t+2 frame image. If they converge, the result of the t+2 frame is used as the basis for the next frame target recognition. If they do not converge, continue to execute the above cycle. When the number of cycles exceeds the preset threshold, the current recognition result is terminated and the face recognition process of the monitored person is restarted.

[0235] The second embodiment of the present invention provides an intelligent vehicle-mounted box, in which the fatigue monitoring algorithm is installed in the SoC system in the box for monitoring. Figure 5 and Figure 6 ,

[0236] The smart vehicle box includes: a vehicle-mounted smart terminal body, a PCBA circuit board 01 installed on the body;

[0237] The PCBA circuit board is provided with: a communication bus, a SoC chip 107, a camera 103, a wireless communication module 106, and a DSP chip 109, wherein the SoC chip 107, the camera 103, the DSP chip 109, and the wireless communication module 106 are connected via the communication bus;

[0238] The camera 103 includes at least one or more cameras, at least one of which is installed on the front of the body; when there are multiple cameras, the multiple cameras are installed on the front and back of the body respectively;

[0239] The DSP chip is specifically used for the calculation of fatigue detection algorithm;

[0240] A display screen and at least one multi-function trigger button are arranged on the body;

[0241] The main body is provided with a fixing component for mounting the main body at a preset position in the vehicle.

[0242] Specifically, the fixing component is used to mount the body at a preset position in the vehicle, and the preset position may be a position in front of the driver or a position of a streaming media mirror or a roof;

[0243] The fixing parts are hooks and slots but are not limited to the above two types. For example, specially designed fixing parts that cooperate with each other are located in the car and in the box body. The two parts cooperate with each other to achieve fixation.

[0244] The fixed parts are mainly designed for easy disassembly. If the user needs, the in-vehicle intelligent terminal can be disassembled and carried with them for outdoor photography, or placed in the office for meetings, live broadcasts, and viewing pictures or photos stored in the in-vehicle intelligent terminal.

[0245] The smart car box is provided with a charging power supply 102 and a solar panel 100 for supplying power to the smart car box; when the smart car box is in a closed state, the solar panel charges the charging power supply. The purpose of setting the solar panel and the charging power supply in this embodiment is to supply power to the smart car box through the charging power supply when it is raining or cloudy and the solar panel is insufficient to supply power. When the smart car box is in a closed state, the solar panel charges the charging power supply. This ensures that the car box is always in a powered state and increases the endurance time of fatigue monitoring.

[0246] The above is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiment. It is clear to those skilled in the art that the form in the embodiment is not limited thereto, and the adjustable manner is not limited thereto. It is understood that other improvements and changes directly derived or associated with those skilled in the art without departing from the basic concept of the present invention should be considered to be included in the protection scope of the present invention.

Claims

1. Driving fatigue monitoring method of smart car box, It is characterized in that At least: The camera mounted on the smart vehicle box collects images, performs a first data analysis on the images to obtain facial features of the monitored person; and extracts key features from the facial features for characterizing the degree of fatigue; Assess the fatigue level of the monitored personnel based on the key characteristics of fatigue level; Among them, the key features to characterize the degree of fatigue include: eye closure frequency η and eye blinking frequency The closing frequency of the mouth, mo; The first data analysis process includes: acquiring a human face existing in an image of an initial frame, acquiring a monitored human face image from the existing human faces, and continuously searching and locating the monitored human face image in subsequent acquired images in a loop; The people being monitored include the driver or people in designated areas of the vehicle; The method for searching and locating the monitored face image includes: Obtain the pixel value A(xi, yi) of the monitored face coordinate area in the previous frame image, and obtain the pixel value B(xi+dx, yi+dy) of the monitored face coordinate area in the current frame image; wherein xi and yi are the pixel coordinates of the face area; Set the objective function: Wherein, dx, dy are optimized coordinates, which are used to indicate the relative position of the coordinates of the monitored face region in the current frame image relative to the coordinates of the monitored face region in the previous frame image; Among them, Ri(dx,dy)=A(xi,yi)-B(xi+dx,yi+dy), where xi,yi are the pixel coordinates of the face area, and Ri(dx,dy) represents the pixel difference between the previous frame image and the current frame image; Substitute the formula of Ri(dx, dy) expanded at the point (xi, yi) according to the first-order derivative into the objective function; Optimize the objective function to obtain the optimal solution Δdx of the objective function; Update dx in the following way: dx = dx + Δdx Substitute the updated dx into the objective function and calculate the error. If the error is greater than the preset threshold, continue to optimize the objective function repeatedly until the error is less than the threshold.

2. The driving fatigue monitoring method of the smart vehicle box according to claim 1, It is characterized in that The mo in each frame image, η forms a training data set, and uses support vector machine or logistic regression for training to obtain the trained classification model; The trained classification model is used to classify and identify new images to obtain the fatigue level of the monitored person.

3. The driving fatigue monitoring method of the smart vehicle box according to claim 1, It is characterized in that The first data analysis process includes: facial feature recognition, which includes: Obtain the HOG feature operators of positive and negative samples; use the support vector machine algorithm to train positive and negative samples to obtain the trained model; use the model to identify difficult examples of negative samples; collect difficult samples to retrain the model, and repeatedly train to obtain the final classification model; The final trained classifier is used to detect face images, and the images of different sizes are scanned, HOG features are extracted, and then classified using the classifier; If the detection is determined to be a face, it will be calibrated and non-maximum suppression will be used to obtain the final located face.

4. The driving fatigue monitoring method of the smart vehicle box according to claim 1, It is characterized in that The first data analysis process includes: determining the lighting conditions of the current in-vehicle environment, and performing image preprocessing on the image according to the lighting conditions.

5. The driving fatigue monitoring method of the smart vehicle box according to claim 1, It is characterized in that When the lighting is poor, the image preprocessing methods used include: F(x)=CDF(x) 2 +K(x)×2×CDF(x)×(1-CDF(x)); in, Y = F(x) × 255; Y is the converted pixel value, Pm{.} represents the probability of the event, X is the gray value of the image, and x represents the classification value of the image pixel value; CDF(x) represents the cumulative distribution function; PDF(u) is the probability density distribution function of the X pixel value in the image; K(x) is the weight coefficient, which is used to characterize the uncertainty of the event probability density.

6. An intelligent vehicle-mounted box for monitoring driver fatigue. It is characterized in that include: The vehicle-mounted intelligent terminal body and the PCBA circuit board installed on the vehicle-mounted intelligent terminal body; The PCBA circuit board is provided with: a communication bus, a SoC chip, a camera, and a DSP chip, wherein the SoC chip, the camera, and the DSP chip are connected via the communication bus; at least one camera is used to collect an image of the driver in the vehicle in real time; The DSP chip is used for data processing of fatigue detection images; It also includes the driving fatigue monitoring method of the smart vehicle box as described in any one of claims 1 to 5.

7. A smart car box as claimed in claim 6, It is characterized in that Also includes: A display screen and at least one multi-function trigger button arranged on the vehicle-mounted intelligent terminal body; The vehicle-mounted intelligent terminal body is provided with a fixing component for mounting the vehicle-mounted intelligent terminal body at a preset position in the vehicle.