Intelligent recognition method of vehicle driver fatigue state based on multi-modal physiological information fusion
Through multi-mode physiological information fusion and adaptive lighting equalization technology, the fatigue characteristics of the on-board driver are extracted and the detection model is trained, which solves the problems of the difficulty of accurately detecting under different lighting conditions in traditional methods, achieving more accurate and individual characteristics Identification of fatigue state and early warning.
Patent Information
- Application Number
- CN202410232670.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-01
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-03-01
AI Technical Summary
In an on-board environment, traditional image processing methods are difficult to accurately detect the driver's fatigue state under different lighting conditions, and fail to fully consider individual differences, resulting in insufficient accuracy and robustness of fatigue detection.
Using multi-mode physiological information fusion method, the driver's facial video is continuously recorded to obtain normal and fatigue images, and using adaptive lighting equalization technology and adaptive threshold segmentation method to extract the degree of eye movement, eye closure degree and eye closing frequency. and use convolutional neural network to train the fatigue detection model to generate fatigue index for early warning.
It improves the accuracy and personalization of fatigue detection, can better adapt to the characteristics of different drivers, enhances the accuracy of image quality and feature point detection, provides direct vibration feedback warning, and improves driving safety.
Smart Images

Figure CN118279878B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an intelligent identification method for vehicle-mounted driver fatigue status by fusing multi-modal physiological information. Background Art
[0002] Long-term driving is not only a common life scenario, but also a potential traffic safety hazard. When the driver is fatigued, his attention to the road traffic environment and his ability to respond quickly decrease, which may lead to traffic accidents and pose a serious threat to driving safety. Therefore, early detection and effective prevention of fatigue driving behavior is crucial to improving road safety and the driver's own driving safety.
[0003] However, fatigue driving detection in a vehicle environment faces a series of challenges. One of them is the influence of lighting conditions. Changes in lighting not only directly affect the acquisition of facial images, but also cause trouble for the accurate acquisition and analysis of physiological information. Traditional image processing methods are easily disturbed by brightness and contrast when processing images under different lighting conditions, which may lead to instability in physiological feature extraction and reduce the accuracy and robustness of fatigue detection systems.
[0004] Existing technologies have certain deficiencies in addressing individual differences. Many image processing methods use global thresholds or general models, which fail to fully consider the differences in physiological characteristics between different drivers. Individual differences include but are not limited to factors such as facial structure, skin color, and light intensity, which may have a significant impact on the collection and analysis of physiological characteristics. Therefore, a fatigue detection method that can personalize the processing of driver fatigue state images is needed to improve the applicability and accuracy of the system. Summary of the invention
[0005] The present application aims to solve at least one of the technical problems in the related art to a certain extent. Based on the above background technology, one purpose of the present application is to propose an intelligent identification method of a vehicle-mounted driver fatigue state by integrating multi-modal physiological information, thereby improving the accuracy and personalization of the identification of the driver fatigue state.
[0006] One aspect of the present application provides a method for intelligently identifying a driver's fatigue state by integrating multi-modal physiological information, including:
[0007] Step S100: continuously recording the driver's facial video to obtain M consecutive frames of normal images and M consecutive frames of fatigue images;
[0008] The method for obtaining M continuous frames of normal images and M continuous frames of fatigue images is as follows: a camera is installed in the car to capture a facial video of the driver, and a normal video clip and a fatigue video clip of the driver are obtained by manually selecting video clips containing a normal state and a fatigue state, respectively, and M continuous frames of normal images and M continuous frames of fatigue images are obtained according to the normal video clip and the fatigue video clip;
[0009] The physiological characteristics of the normal image and the fatigue image include: freedom of eyeball movement, degree of eye closure, and frequency of eye closure;
[0010] The normal image refers to: a continuous frame image extracted from the captured facial video of the driver, corresponding to the normal state of the driver;
[0011] The fatigue image refers to: a continuous frame image extracted from the captured facial video of the driver, corresponding to the fatigue state of the driver;
[0012] The angular mapping range of the eyeball mobility is [0, 1], where 0 is the minimum eyeball mobility in the horizontal or vertical direction, and 1 is the maximum eyeball mobility in the horizontal or vertical direction;
[0013] The range of the eye closure degree is [0, 1], where 0 represents completely closed eyes and 1 represents completely open eyes;
[0014] The eye closing frequency refers to the ratio of the eye closing duration within the cycle time t to the cycle time t;
[0015] Step S200: using an adaptive illumination equalization technique to adjust the brightness and contrast of the fatigue image to obtain a second fatigue image;
[0016] The specific method of using the adaptive illumination equalization technology to adjust the brightness and contrast of the fatigue image to obtain the second fatigue image is:
[0017] Step S210: graying the fatigue image into a grayscale fatigue image;
[0018] The specific method of graying the fatigue image and converting it into a gray fatigue image is: reading the pixels of the fatigue image to be processed, calculating the gray value of each pixel, and obtaining the gray fatigue image, and the gray value calculation formula is: H = 0.299 × R + 0.587 × G + 0.114 × B, where R, G, B are the pixel values of the red, green, and blue channels in the fatigue image, and H is the gray value of each pixel;
[0019] Step S220: using the threshold adjustment model to predict the grayscale fatigue image to obtain a probability image;
[0020] The probability image refers to a grayscale fatigue image containing the probability that each pixel belongs to the fatigue area;
[0021] Step S230: Select a b×b window to slide on the probability image, calculate the adaptive threshold of the small block probability image in each window, and perform adaptive threshold segmentation;
[0022] The method of the adaptive threshold segmentation is:
[0023] Step S231: select a window of size b×b, set the sliding step size, and slide on the probability image;
[0024] Step S232: dynamically adjusting the adaptive threshold for the small probability image in each window through the threshold adjustment model;
[0025] The training process of the threshold adjustment model is as follows: using grayscale fatigue images under different brightness and contrast conditions, marking the target threshold for each grayscale fatigue image, using a convolutional neural network as the initial model of the threshold adjustment model, the initial model uses the grayscale fatigue image as input, and generates a probability image in the output layer; using the grayscale fatigue image and the probability image as input data for training the threshold adjustment model, using the dynamically adjusted threshold as output data of the threshold adjustment model, accurately outputting the target threshold as the prediction target of the threshold adjustment model, and minimizing the difference between the dynamically adjusted threshold and the target threshold as the loss function;
[0026] The calculation formula of the loss function is: Where M is the number of grayscale fatigue images used as training samples, represents the dynamically adjusted threshold of the j-th small probability image on the i-th grayscale fatigue image, represents the target threshold of the j-th small block probability image on the i-th grayscale fatigue image;
[0027] The target threshold labeling method is as follows: a professional or a domain expert manually checks each grayscale fatigue image and determines the target threshold according to the color and texture features of the fatigue area;
[0028] Step S233: binarizing the probability image according to the dynamically adjusted adaptive threshold y to obtain a binary image of the grayscale area;
[0029] The specific method of binarizing the probability image according to the dynamically adjusted adaptive threshold to obtain the binary image of the grayscale area is: comparing the grayscale values of the pixels in the probability image with the dynamically calculated threshold, defining the pixels greater than the threshold as the first pixel and marked as white, and defining the pixels less than or equal to the threshold as the second pixel and marked as black, to obtain the binary image of the grayscale area;
[0030] Step S234: repeating steps S231 to S233, repeatedly sliding the window on the probability image, and processing the entire probability image;
[0031] Step S240: combining the binary images processed in each window and performing global contrast enhancement to obtain a second fatigue image;
[0032] Step S300: using a facial key point detection algorithm to process the normal image and the second fatigue image respectively to obtain normal feature points and fatigue feature points;
[0033] The specific method of using the facial key point detection algorithm to process the normal image and the second fatigue image respectively to obtain the normal feature points and the fatigue feature points is:
[0034] Step S310: using a face detection algorithm to locate the faces in the normal image and the second fatigue image, and determine the positions and frames of the faces;
[0035] Step S320: performing key point detection on the located face using a face key point detection algorithm;
[0036] Step S330: extracting the facial coordinate information obtained by key point detection to obtain a normal feature point set and a fatigue feature point set;
[0037] The normal feature point set is composed of r normal feature points (k i ,q i ),(k i ,q i ) refers to the i-th normal feature point;
[0038] The fatigue feature point set is composed of r fatigue feature points (c i ,w i ) is composed of (c i ,w i ) refers to the i-th fatigue characteristic point;
[0039] Step S400: extracting the normal physiological characteristics and fatigue physiological characteristics of the driver according to the normal feature points and fatigue feature points;
[0040] The normal physiological characteristics refer to the physiological characteristics of the driver in a normal image;
[0041] The fatigue physiological characteristics refer to the physiological characteristics of the driver during fatigue driving in the second fatigue image;
[0042] The specific calculation method for extracting the normal physiological characteristics and fatigue physiological characteristics of the driver based on the normal feature points and fatigue feature points is:
[0043] Get the highest feature point (k) of the upper eyelid in the eye area of the driver's normal image or the second fatigue image i ,q i ) and the lowest feature point of the lower eyelid (k' i ,q' i ), obtain the highest feature point of the upper eyelid in the eye area of the second fatigue image and the lowest feature point of the lower eyelid The position of the eyes is calculated to obtain the degree of eye closure;
[0044] The calculation formula of the eye closure degree α is:
[0045] Track the movement of the key points of the eyeballs in the continuous frames, obtain the eyeball movement speed v1 on the normal image, and the eyeball movement speed v2 on the normal image or the second fatigue image, and calculate the freedom of eyeball movement;
[0046] The calculation formula of the eyeball movement freedom β is:
[0047] By tracking the state of the eye area, capturing the time node t1 of eye closure and the time node t2 of eye opening in the normal image or the second fatigue image in the continuous frames, the eye closure duration is obtained, and the eye closure frequency is calculated;
[0048] The calculation formula of the eye closing frequency γ is: Where t is the cycle time;
[0049] Step S500: using normal physiological characteristics and fatigue physiological characteristics as training data, training a fatigue detection model for outputting a fatigue index;
[0050] The training process of the fatigue detection model is as follows: normalizing normal physiological characteristics and fatigue physiological characteristics to obtain standardized normal physiological characteristics and standardized fatigue physiological characteristics, respectively marking the standardized normal physiological characteristics and the standardized fatigue physiological characteristics with corresponding fatigue index labels, using the standardized normal physiological characteristics and the standardized fatigue physiological characteristics as input data of training data, using the fatigue index labels as output data of training data, training the fatigue detection model to accurately output the driver's fatigue index as the prediction target of the fatigue detection model, and minimizing the fatigue loss function between the real fatigue index label and the predicted fatigue index as the training target of the fatigue detection model;
[0051] The calculation formula of the fatigue loss function is: where x i represents the true fatigue index label of the i-th training data, represents the fatigue index of the predicted output of the i-th training data;
[0052] The method of marking the corresponding fatigue index labels of the standardized normal physiological characteristics and the standardized fatigue physiological characteristics is as follows: professionals observe the normal image and the fatigue image of the driver, and every δ professionals mark the fatigue index of the same image, and calculate the average fatigue index as the fatigue index label corresponding to each image;
[0053] Step S600: obtaining a fatigue index using a fatigue detection model, and issuing a warning to the driver when the fatigue index exceeds a dangerous threshold;
[0054] The danger threshold is Q, which is set manually based on actual experience and practical needs;
[0055] When the fatigue index exceeds the danger threshold, the driver is given a warning. When the fatigue index x is greater than the danger threshold Q, it indicates that the current driver is driving fatigued and continuing to drive will pose a safety risk. Vibration feedback is set according to the fatigue index. The fatigue index is directly proportional to the vibration feedback. The higher the fatigue index, the stronger the vibration feedback of the seat.
[0056] One aspect of the present application provides an intelligent identification system for vehicle-mounted driver fatigue status by integrating multi-modal physiological information, including:
[0057] An image collection module is used to continuously record the driver's facial video and obtain M consecutive frames of normal images and M consecutive frames of fatigue images;
[0058] An illumination equalization module, used for adjusting the brightness and contrast of the fatigue image using an adaptive illumination equalization technique to obtain a second fatigue image;
[0059] A feature point detection module, used to process the normal image and the second fatigue image respectively using a face key point detection algorithm to obtain normal feature points and fatigue feature points;
[0060] A feature extraction module, used for extracting the normal physiological features and fatigue physiological features of the driver according to the normal feature points and fatigue feature points;
[0061] A model training module, used to use normal physiological characteristics and fatigue physiological characteristics as training data to train a fatigue detection model for outputting a fatigue index;
[0062] The danger warning module is used to obtain a fatigue index using a fatigue detection model, and to issue a warning to the driver when the fatigue index exceeds a danger threshold.
[0063] Compared with the prior art, the intelligent identification method of vehicle driver fatigue state based on multi-modal physiological information fusion proposed in this application has the following advantages:
[0064] The method takes into account multiple physiological characteristics such as eyeball movement freedom, eye closure degree and eye closure frequency, so that fatigue detection is more comprehensive and can more accurately reflect the driver's fatigue state. The brightness and contrast of the fatigue image are adjusted by adaptive illumination equalization technology, which helps to improve the image quality and thus improve the accuracy of feature point detection. An adaptive threshold segmentation method is used in image processing. By dynamically adjusting the threshold, it better adapts to the brightness distribution of different regions and improves the accuracy of image segmentation. The idea of longitudinal comparison is introduced to compare the fatigue state of each driver with his own normal state, rather than horizontally comparing the fatigue index between different drivers, which is more in line with individual differences and improves the personalization of fatigue detection. A fatigue detection model is introduced, which learns from the physiological characteristics of normal and fatigue through a machine learning algorithm, and can more flexibly adapt to the characteristics of different drivers, thereby improving the accuracy and universality of fatigue detection. When fatigue driving is detected, a vibration feedback method is used for early warning. This feedback method does not rely on the driver's vision or hearing. It is a more direct early warning method that can quickly attract the driver's attention and helps to improve driving safety.
[0065] The multi-modal physiological information fusion-based intelligent identification method for vehicle-mounted driver fatigue status provides a comprehensive and effective solution for fatigue driving detection by integrating technologies from multiple fields such as image processing, physiological feature extraction, machine learning, and early warning systems, thereby improving the accuracy and personalization of intelligent identification of driver fatigue status. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] Figure 1 A method flow chart of the method for intelligent identification of vehicle driver fatigue status by fusion of multi-modal physiological information of the present application;
[0067] Figure 2 This is a flow chart of the adaptive threshold segmentation method of the vehicle driver fatigue state intelligent identification method based on multi-modal physiological information fusion of the present application;
[0068] Figure 3 A schematic diagram of facial feature point extraction for this application;
[0069] Figure 4 This is a functional module diagram of the vehicle-mounted driver fatigue status intelligent recognition system based on multi-modal physiological information fusion of the present application. DETAILED DESCRIPTION
[0070] In order to better understand the present application, a more detailed description will be made of various aspects of the present application with reference to the accompanying drawings. It should be understood that these detailed descriptions are only descriptions of exemplary embodiments of the present application, and are not intended to limit the scope of the present application in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.
[0071] In the accompanying drawings, the size, dimensions and shapes of the elements have been slightly adjusted for ease of illustration. The drawings are for illustration only and are not strictly drawn to scale. As used herein, the terms "substantially", "approximately" and similar terms are used as terms of approximation, not as terms of degree, and are intended to illustrate the inherent deviations in measurements or calculations that will be recognized by those of ordinary skill in the art. In addition, in this application, the order in which the steps are described does not necessarily represent the order in which these processes occur in actual operation, unless otherwise specified or can be derived from the context.
[0072] It should also be understood that expressions such as "include", "including", "have", "contain" and / or "comprising" are open rather than closed expressions in this specification, which indicate the presence of the stated features, elements and / or components, but do not exclude the presence of one or more other features, elements, components and / or combinations thereof. In addition, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire list of features rather than just the individual elements in the list. In addition, when describing embodiments of the present application, "may" is used to mean "one or more embodiments of the present application". And, the term "exemplary" is intended to refer to an example or illustration.
[0073] Unless otherwise specified, all words (including engineering terms and scientific and technological terms) used in this article have the same meaning as those commonly understood by ordinary technicians in the field to which this application belongs. It should also be understood that, unless clearly stated in this application, words defined in common dictionaries should be interpreted as having the same meaning as their meaning in the context of the relevant technology, and should not be interpreted in an idealized or overly formal sense.
[0074] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0075] Example 1
[0076] like Figure 1 As shown, it is a method flow chart of the method for intelligent identification of vehicle driver fatigue state by fusion of multi-modal physiological information provided by the present application, including:
[0077] Step S100: continuously recording the driver's facial video to obtain M consecutive frames of normal images and M consecutive frames of fatigue images;
[0078] The method for obtaining M continuous frames of normal images and M continuous frames of fatigue images is as follows: a camera is installed in the car to capture a facial video of the driver, and a normal video clip and a fatigue video clip of the driver are obtained by manually selecting video clips containing a normal state and a fatigue state, respectively, and M continuous frames of normal images and M continuous frames of fatigue images are obtained according to the normal video clip and the fatigue video clip;
[0079] The physiological characteristics of the normal image and the fatigue image include: freedom of eyeball movement, degree of eye closure, and frequency of eye closure;
[0080] The normal image refers to: a continuous frame image extracted from the captured facial video of the driver, corresponding to the normal state of the driver;
[0081] The normal image reflects the physiological characteristics of the driver under normal conditions, for example, normal degree of eye closure, normal freedom of eye movement, and normal frequency of eye closure;
[0082] The fatigue image refers to: a continuous frame image extracted from the captured facial video of the driver, corresponding to the fatigue state of the driver;
[0083] The fatigue image reflects the physiological characteristics of the driver in a fatigued state, such as high degree of eye closure, abnormal freedom of eye movement, and low frequency of eye closure;
[0084] The angular mapping range of the eyeball mobility is [0, 1], where 0 is the minimum eyeball mobility in the horizontal or vertical direction, and 1 is the maximum eyeball mobility in the horizontal or vertical direction;
[0085] The range of the eye closure degree is [0, 1], where 0 represents completely closed eyes and 1 represents completely open eyes;
[0086] The eye closing frequency refers to the ratio of the eye closing duration within the cycle time t to the cycle time t;
[0087] This step obtains the physiological characteristic data of the driver in normal and fatigue states by capturing M consecutive frames of normal images and fatigue images, providing comprehensive physiological characteristic data for identifying the driver's fatigue state and providing a basis for subsequent image processing, feature extraction and training of fatigue detection models.
[0088] Step S200: using an adaptive illumination equalization technique to adjust the brightness and contrast of the fatigue image to obtain a second fatigue image;
[0089] The specific method of using the adaptive illumination equalization technology to adjust the brightness and contrast of the fatigue image to obtain the second fatigue image is:
[0090] Step S210: graying the fatigue image into a grayscale fatigue image;
[0091] The specific method of graying the fatigue image and converting it into a gray fatigue image is: reading the pixels of the fatigue image to be processed, calculating the gray value of each pixel, and obtaining the gray fatigue image, and the gray value calculation formula is: H = 0.299 × R + 0.587 × G + 0.114 × B, where R, G, B are the pixel values of the red, green, and blue channels in the fatigue image, and H is the gray value of each pixel;
[0092] Step S220: using the threshold adjustment model to predict the grayscale fatigue image to obtain a probability image;
[0093] The probability image refers to a grayscale fatigue image containing the probability that each pixel belongs to the fatigue area;
[0094] Step S230: Select a b×b window to slide on the probability image, calculate the adaptive threshold of the small block probability image in each window, and perform adaptive threshold segmentation;
[0095] The adaptive threshold segmentation is an improved image processing method. The difference from the prior art is that the local threshold is calculated according to the brightness distribution of different regions of the image, rather than calculating the global threshold of the image.
[0096] like Figure 2 As shown, it is a flow chart of the adaptive threshold segmentation method of the vehicle driver fatigue state intelligent identification method based on multi-modal physiological information fusion of the present application, and the adaptive threshold segmentation method is:
[0097] Step S231: select a window of size b×b, set the sliding step size, and slide on the probability image;
[0098] Step S232: dynamically adjusting the adaptive threshold for the small probability image in each window through the threshold adjustment model;
[0099] The training process of the threshold adjustment model is as follows: using grayscale fatigue images under different brightness and contrast conditions, marking the target threshold for each grayscale fatigue image, using a convolutional neural network as the initial model of the threshold adjustment model, the initial model uses the grayscale fatigue image as input, and generates a probability image in the output layer; using the grayscale fatigue image and the probability image as input data for training the threshold adjustment model, using the dynamically adjusted threshold as output data of the threshold adjustment model, accurately outputting the target threshold as the prediction target of the threshold adjustment model, and minimizing the difference between the dynamically adjusted threshold and the target threshold as the loss function;
[0100] The calculation formula of the loss function is: Where M is the number of grayscale fatigue images used as training samples, represents the dynamically adjusted threshold of the j-th small probability image on the i-th grayscale fatigue image, represents the target threshold of the j-th small block probability image on the i-th grayscale fatigue image;
[0101] The target threshold labeling method is as follows: a professional or a domain expert manually checks each grayscale fatigue image and determines the target threshold according to the color and texture features of the fatigue area;
[0102] Step S233: binarizing the probability image according to the dynamically adjusted adaptive threshold y to obtain a binary image of the grayscale area;
[0103] The specific method of binarizing the probability image according to the dynamically adjusted adaptive threshold to obtain the binary image of the grayscale area is: comparing the grayscale values of the pixels in the probability image with the dynamically calculated threshold, defining the pixels greater than the threshold as the first pixel and marked as white, and defining the pixels less than or equal to the threshold as the second pixel and marked as black, to obtain the binary image of the grayscale area;
[0104] Step S234: repeating steps S231 to S233, repeatedly sliding the window on the probability image, and processing the entire probability image;
[0105] Step S240: combining the binary images processed in each window and performing global contrast enhancement to obtain a second fatigue image;
[0106] The global contrast enhancement refers to improving the contrast of the combined binary image so that the obtained second fatigue image is clearer;
[0107] This step uses adaptive illumination equalization technology to adjust the brightness and contrast of the fatigue image to make the image clearer and easier to analyze. The fatigue image is grayed out and a probability image is generated, which helps to highlight the eye features in the image. The local threshold is calculated based on the brightness distribution of different areas of the image, which helps to more accurately segment the image into fatigue and non-fatigue areas, improves the sensitivity to eye features, optimizes the fatigue image, and provides better image quality for subsequent feature extraction and analysis.
[0108] Step S300: using a facial key point detection algorithm to process the normal image and the second fatigue image respectively to obtain normal feature points and fatigue feature points;
[0109] The specific method of using the facial key point detection algorithm to process the normal image and the second fatigue image respectively to obtain the normal feature points and the fatigue feature points is:
[0110] Step S310: using a face detection algorithm to locate the faces in the normal image and the second fatigue image, and determine the positions and frames of the faces;
[0111] Step S320: performing key point detection on the located face using a face key point detection algorithm;
[0112] Step S330: extracting the facial coordinate information obtained by key point detection to obtain a normal feature point set and a fatigue feature point set;
[0113] The normal feature point set is composed of r normal feature points (k i ,q i ),(k i ,q i ) refers to the i-th normal feature point;
[0114] The fatigue feature point set is composed of r fatigue feature points (c i ,w i ) is composed of (c i ,w i ) refers to the i-th fatigue characteristic point;
[0115] This step successfully locates the face position and border in the normal image and the second fatigue image through the face detection algorithm, providing accurate face areas for subsequent key point detection, helping to focus on the extraction of facial features, more accurately analyzing the physiological characteristics in normal and fatigue states, improving the sensitivity and accuracy of physiological characteristics in normal and fatigue states, and providing a reliable foundation for the training of fatigue detection models.
[0116] Step S400: extracting the normal physiological characteristics and fatigue physiological characteristics of the driver according to the normal feature points and fatigue feature points;
[0117] The normal physiological characteristics refer to the physiological characteristics of the driver in a normal image;
[0118] The fatigue physiological characteristics refer to the physiological characteristics of the driver during fatigue driving in the second fatigue image;
[0119] The specific calculation method for extracting the normal physiological characteristics and fatigue physiological characteristics of the driver based on the normal feature points and fatigue feature points is:
[0120] like Figure 3 As shown in the figure, it is a schematic diagram of extracting facial feature points of the present application, and the highest feature point (k i ,q i ) and the lowest feature point of the lower eyelid (k' i ,q' i ), obtain the highest feature point of the upper eyelid in the eye area of the second fatigue image and the lowest feature point of the lower eyelid The position of the eyes is calculated to obtain the degree of eye closure;
[0121] The calculation formula of the eye closure degree α is:
[0122] Track the movement of the key points of the eyeballs in the continuous frames, obtain the eyeball movement speed v1 on the normal image, and the eyeball movement speed v2 on the normal image or the second fatigue image, and calculate the freedom of eyeball movement;
[0123] The calculation formula of the eyeball movement freedom β is:
[0124] By tracking the state of the eye area, capturing the time node t1 of eye closure and the time node t2 of eye opening in the normal image or the second fatigue image in the continuous frames, the eye closure duration is obtained, and the eye closure frequency is calculated;
[0125] The calculation formula of the eye closing frequency γ is: Where t is the cycle time;
[0126] The normal physiological characteristics and fatigue physiological characteristics are calculated by the calculation formula given above. Since each person's facial features are different, whether different drivers are driving fatigued can only be judged by longitudinal comparison, not by transverse comparison. The longitudinal comparison refers to judging whether each driver is in a fatigue state by comparing it with his normal state, and the transverse comparison refers to comparing different drivers;
[0127] This step quantifies the driver's normal physiological characteristics and fatigue physiological characteristics by calculating the degree of eye closure, freedom of eye movement and frequency of eye closure. By comparing with the driver's normal state, it can more accurately determine whether the driver is in a fatigue state. This longitudinal comparison is helpful for personalized fatigue driving detection. Taking into account that each driver may have different facial features and physiological characteristics, it provides an effective data basis for subsequent fatigue state judgment, which helps to improve the accuracy and personalization of fatigue driving detection.
[0128] Step S500: using normal physiological characteristics and fatigue physiological characteristics as training data, training a fatigue detection model for outputting a fatigue index;
[0129] The training process of the fatigue detection model is as follows: normalizing normal physiological characteristics and fatigue physiological characteristics to obtain standardized normal physiological characteristics and standardized fatigue physiological characteristics, respectively marking the standardized normal physiological characteristics and the standardized fatigue physiological characteristics with corresponding fatigue index labels, using the standardized normal physiological characteristics and the standardized fatigue physiological characteristics as input data of training data, using the fatigue index labels as output data of training data, training the fatigue detection model to accurately output the driver's fatigue index as the prediction target of the fatigue detection model, and minimizing the fatigue loss function between the real fatigue index label and the predicted fatigue index as the training target of the fatigue detection model;
[0130] The calculation formula of the fatigue loss function is: where x i represents the true fatigue index label of the i-th training data, represents the fatigue index of the predicted output of the i-th training data;
[0131] The method of marking the corresponding fatigue index labels of the standardized normal physiological characteristics and the standardized fatigue physiological characteristics is as follows: professionals observe the normal image and the fatigue image of the driver, and every δ professionals mark the fatigue index of the same image, and calculate the average fatigue index as the fatigue index label corresponding to each image;
[0132] In this step, a fatigue detection model is obtained by training by using normal physiological characteristics and fatigue physiological characteristics as training data. The model can learn the relationship between the input physiological characteristics and the fatigue index, so as to evaluate the driver's fatigue state. Standardizing normal physiological characteristics and fatigue physiological characteristics helps to improve the training effect of the model. Professionals observe normal images and fatigue images and annotate the images with fatigue index. Using the observation results of multiple professionals and calculating the average fatigue index helps to reduce subjective annotation errors and improve the accuracy of labels, providing a trained fatigue detection model for subsequent operations, and improving the performance of the model and the accurate evaluation of the driver's fatigue state by means of standardization, definition of loss function and adaptive label generation.
[0133] Step S600: obtaining a fatigue index using a fatigue detection model, and issuing a warning to the driver when the fatigue index exceeds a dangerous threshold;
[0134] The danger threshold is Q, which is set manually based on actual experience and practical needs;
[0135] When the fatigue index exceeds the danger threshold, the driver is given a warning. When the fatigue index x is greater than the danger threshold Q, it indicates that the driver is currently driving in fatigue and continuing to drive will be a safety risk. Vibration feedback is set according to the fatigue index. The fatigue index is in direct proportion to the vibration feedback. The higher the fatigue index, the stronger the vibration feedback of the seat.
[0136] This step uses an artificially set danger threshold Q, which can be adjusted individually according to actual experience and real needs. Different drivers may show safety hazards at different fatigue levels. Therefore, personalized setting of the danger threshold helps to more accurately reflect the driver's fatigue state. The design of a proportional relationship between the fatigue index and the vibration feedback can make the feedback more sensitive and personalized. By monitoring the fatigue index in real time, when the driver's fatigue level reaches the danger threshold, vibration feedback is used to remind the driver, thereby providing timely safety protection when fatigue driving may lead to an accident.
[0137] Example 2
[0138] like Figure 4 As shown, it is a functional module diagram of the vehicle-mounted driver fatigue state intelligent recognition system based on multi-modal physiological information fusion provided by this application, including:
[0139] An image collection module is used to continuously record the driver's facial video and obtain M consecutive frames of normal images and M consecutive frames of fatigue images;
[0140] An illumination equalization module, used for adjusting the brightness and contrast of the fatigue image using an adaptive illumination equalization technique to obtain a second fatigue image;
[0141] A feature point detection module, used to process the normal image and the second fatigue image respectively using a face key point detection algorithm to obtain normal feature points and fatigue feature points;
[0142] A feature extraction module, used for extracting the normal physiological features and fatigue physiological features of the driver according to the normal feature points and fatigue feature points;
[0143] A model training module, used to use normal physiological characteristics and fatigue physiological characteristics as training data to train a fatigue detection model for outputting a fatigue index;
[0144] The danger warning module is used to obtain a fatigue index using a fatigue detection model, and to issue a warning to the driver when the fatigue index exceeds a danger threshold.
[0145] Furthermore, the illumination balancing module includes:
[0146] A grayscale module, used for grayscale the fatigue image and converting it into a grayscale fatigue image;
[0147] A probability image conversion module is used to predict the grayscale fatigue image using a threshold adjustment model to obtain a probability image;
[0148] The adaptive threshold segmentation module is used to select a b×b window to slide on the probability image, calculate the adaptive threshold of the small block probability image in each window, and perform adaptive threshold segmentation;
[0149] A global enhancement module, used for combining the binary images processed by each window and performing global contrast enhancement to obtain a second fatigue image;
[0150] Furthermore, the adaptive threshold segmentation module also includes:
[0151] The sliding selection module is used to select a window of size b×b, set the sliding step size, and slide on the probability image;
[0152] An adaptive threshold adjustment module is used to dynamically adjust the adaptive threshold of a small probability image in each window through a threshold adjustment model;
[0153] A binarization module is used to binarize the probability image according to the dynamically adjusted adaptive threshold y to obtain a binary image of the grayscale area;
[0154] An iterative processing module, used to repeatedly slide the window on the probability image to process the entire probability image;
[0155] Furthermore, the feature point detection module also includes:
[0156] A face positioning module, used to locate the faces in the normal image and the second fatigue image using a face detection algorithm, and determine the positions and frames of the faces;
[0157] A key point detection module is used to perform key point detection on the located face using a face key point detection algorithm;
[0158] The feature point set module is used to extract the facial coordinate information obtained by key point detection to obtain a normal feature point set and a fatigue feature point set.
[0159] In addition, the parts of the above-mentioned technical solutions provided in the embodiments of the present application that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive redundancy.
[0160] The specific implementation modes as described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation mode of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. An intelligent identification method for vehicle driver fatigue status based on multi-modal physiological information fusion, characterized in that: include: Continuously record the driver's facial video to obtain M consecutive frames of normal images and M consecutive frames of fatigue images; Using adaptive illumination equalization technology to adjust the brightness and contrast of the fatigue image to obtain a second fatigue image; The specific method of using the adaptive illumination equalization technology to adjust the brightness and contrast of the fatigue image to obtain the second fatigue image is: Graying the fatigue image and converting it into a gray fatigue image; the specific method of graying the fatigue image and converting it into a gray fatigue image is: reading the pixels of the fatigue image to be processed, calculating the gray value of each pixel, and obtaining the gray fatigue image, the gray value calculation formula is: H = 0.299 × R + 0.587 × G + 0.114 × B, where R, G, B are the pixel values of the red, green, and blue channels in the fatigue image, and H is the gray value of each pixel; The grayscale fatigue image is predicted using the threshold adjustment model to obtain a probability image; Select a b×b window to slide on the probability image, calculate the adaptive threshold of the small block probability image in each window, and perform adaptive threshold segmentation; The binary images processed in each window are combined and global contrast enhanced to obtain a second fatigue image; The normal image and the second fatigue image are processed respectively using a facial key point detection algorithm to obtain normal feature points and fatigue feature points; Extracting the normal physiological characteristics and fatigue physiological characteristics of the driver according to the normal feature points and fatigue feature points; Using normal physiological characteristics and fatigue physiological characteristics as training data, a fatigue detection model for outputting a fatigue index is trained; The fatigue detection model is used to obtain the fatigue index. When the fatigue index exceeds the dangerous threshold, a warning is issued to the driver. The specific calculation method for extracting the normal physiological characteristics and fatigue physiological characteristics of the driver based on the normal feature points and fatigue feature points is: Get the highest feature point (k) of the upper eyelid in the eye area of the driver's normal image or the second fatigue image i ,q i ) and the lowest feature point of the lower eyelid (k' i ,q' i ), obtain the highest feature point of the upper eyelid in the eye area of the second fatigue image and the lowest feature point of the lower eyelid The position of the eyes is calculated to obtain the degree of eye closure; The calculation formula of the eye closure degree α is: Track the movement of the eyeball key points in continuous frames, obtain the eyeball movement speed v1 on the normal image, and the eyeball movement speed v2 on the normal image or the second fatigue image, and calculate the eyeball movement freedom; the calculation formula of the eyeball movement freedom β is: By tracking the state of the eye area, capturing the time node t1 of eye closure and the time node t2 of eye opening in the normal image or the second fatigue image in the continuous frames, the eye closure duration is obtained, and the eye closure frequency is calculated; the calculation formula of the eye closure frequency γ is: Where t is the cycle time.
2. The method for intelligently identifying the fatigue state of a vehicle driver based on multi-modal physiological information fusion according to claim 1, characterized in that: The method for obtaining M consecutive frames of normal images and M consecutive frames of fatigue images is as follows: A camera is installed in the car to capture the driver's facial video. By manually selecting video clips containing normal and fatigue states, normal video clips and fatigue video clips of the driver are obtained respectively. Based on the normal video clips and fatigue video clips, M continuous frames of normal images and M continuous frames of fatigue images are obtained.
3. The method for intelligently identifying the fatigue state of a vehicle driver by fusion of multi-modal physiological information as claimed in claim 2, characterized in that: The method of the adaptive threshold segmentation is: Step S231: select a window of size b×b, set the sliding step size, and slide on the probability image; Step S232: dynamically adjusting the adaptive threshold for the small probability image in each window through the threshold adjustment model; Step S233: binarizing the probability image according to the dynamically adjusted adaptive threshold y to obtain a binary image of the grayscale area; Step S234: repeat steps S231 to S233, repeatedly sliding the window on the probability image, and processing the entire probability image.
4. The method for intelligently identifying the fatigue state of a vehicle driver by fusion of multi-modal physiological information as claimed in claim 3, characterized in that: The training process of the threshold adjustment model is: Using grayscale fatigue images with different brightness and contrast conditions, marking the target threshold for each grayscale fatigue image, using a convolutional neural network as the initial model of the threshold adjustment model, the initial model takes the grayscale fatigue image as input and generates a probability image at the output layer; The grayscale fatigue image and the probability image are used as input data for training the threshold adjustment model, the dynamically adjusted threshold is used as the output data of the threshold adjustment model, the accurate output target threshold is used as the prediction target of the threshold adjustment model, and the minimization of the difference between the dynamically adjusted threshold and the target threshold is used as the loss function; the calculation formula of the loss function is: Where M is the number of grayscale fatigue images used as training samples, represents the dynamically adjusted threshold of the j-th small probability image on the i-th grayscale fatigue image, Represents the target threshold of the j-th small block probability image on the i-th grayscale fatigue image.
5. The method for intelligently identifying the fatigue state of a vehicle driver based on multi-modal physiological information fusion as claimed in claim 4, characterized in that: The specific method of using the facial key point detection algorithm to process the normal image and the second fatigue image respectively to obtain the normal feature points and the fatigue feature points is: Use a face detection algorithm to locate the face in the normal image and the second fatigue image, and determine the position and border of the face; Use the face key point detection algorithm to detect key points of the located face; The face coordinate information obtained by key point detection is extracted to obtain a normal feature point set and a fatigue feature point set; the normal feature point set is composed of r normal feature points (k i ,q i ),(k i ,q i ) refers to the i-th normal feature point; the fatigue feature point set is composed of r fatigue feature points (c i ,w i ) is composed of (c i ,w i ) refers to the i-th fatigue characteristic point.
6. The method for intelligently identifying the fatigue state of a vehicle driver by fusion of multi-modal physiological information as claimed in claim 5, characterized in that: The training process of the fatigue detection model is as follows: normalizing normal physiological characteristics and fatigue physiological characteristics to obtain standardized normal physiological characteristics and standardized fatigue physiological characteristics, respectively marking the standardized normal physiological characteristics and standardized fatigue physiological characteristics with corresponding fatigue index labels, using the standardized normal physiological characteristics and the standardized fatigue physiological characteristics as input data of training data, using the fatigue index label as output data of training data, training the fatigue detection model to accurately output the driver's fatigue index as the prediction target of the fatigue detection model, and minimizing the fatigue loss function between the real fatigue index label and the predicted fatigue index as the training target of the fatigue detection model; the calculation formula of the fatigue loss function is: where x i represents the true fatigue index label of the i-th training data, Represents the fatigue index of the predicted output of the i-th training data.
7. The method for intelligently identifying the fatigue state of a vehicle driver by fusion of multi-modal physiological information as claimed in claim 6, characterized in that: When the fatigue index exceeds the danger threshold, the driver is given a warning. When the fatigue index x is greater than the danger threshold Q, it indicates that the current driver is driving fatigued and continuing to drive will pose a safety risk. Vibration feedback is set according to the fatigue index. The fatigue index is directly proportional to the vibration feedback. The higher the fatigue index, the stronger the vibration feedback of the seat.
8. The vehicle-mounted driver fatigue state intelligent recognition system based on multi-modal physiological information fusion is characterized by: include: An image collection module is used to continuously record the driver's facial video and obtain M consecutive frames of normal images and M consecutive frames of fatigue images; An illumination equalization module, used for adjusting the brightness and contrast of the fatigue image using an adaptive illumination equalization technique to obtain a second fatigue image; The specific method of using the adaptive illumination equalization technology to adjust the brightness and contrast of the fatigue image to obtain the second fatigue image is: Graying the fatigue image and converting it into a gray fatigue image; the specific method of graying the fatigue image and converting it into a gray fatigue image is: reading the pixels of the fatigue image to be processed, calculating the gray value of each pixel, and obtaining the gray fatigue image, the gray value calculation formula is: H = 0.299 × R + 0.587 × G + 0.114 × B, where R, G, B are the pixel values of the red, green, and blue channels in the fatigue image, and H is the gray value of each pixel; The grayscale fatigue image is predicted using the threshold adjustment model to obtain a probability image; Select a b×b window to slide on the probability image, calculate the adaptive threshold of the small block probability image in each window, and perform adaptive threshold segmentation; The binary images processed in each window are combined and global contrast enhanced to obtain a second fatigue image; A feature point detection module, used to process the normal image and the second fatigue image respectively using a face key point detection algorithm to obtain normal feature points and fatigue feature points; A feature extraction module, used for extracting the normal physiological features and fatigue physiological features of the driver according to the normal feature points and fatigue feature points; A model training module, used to use normal physiological characteristics and fatigue physiological characteristics as training data to train a fatigue detection model for outputting a fatigue index; The danger warning module is used to obtain the fatigue index using the fatigue detection model, and issue a warning to the driver when the fatigue index exceeds the danger threshold; The specific calculation method for extracting the normal physiological characteristics and fatigue physiological characteristics of the driver based on the normal feature points and fatigue feature points is: Get the highest feature point (k) of the upper eyelid in the eye area of the driver's normal image or the second fatigue image i ,q i ) and the lowest feature point of the lower eyelid (k' i ,q' i ), obtain the highest feature point of the upper eyelid in the eye area of the second fatigue image and the lowest feature point of the lower eyelid The position of the eyes is calculated to obtain the degree of eye closure; The calculation formula of the eye closure degree α is: Track the movement of the eyeball key points in continuous frames, obtain the eyeball movement speed v1 on the normal image, and the eyeball movement speed v2 on the normal image or the second fatigue image, and calculate the eyeball movement freedom; the calculation formula of the eyeball movement freedom β is: By tracking the state of the eye area, capturing the time node t1 of eye closure and the time node t2 of eye opening in the normal image or the second fatigue image in the continuous frames, the eye closure duration is obtained, and the eye closure frequency is calculated; the calculation formula of the eye closure frequency γ is: Where t is the cycle time.
Citation Information
Cited By
Method for detecting fatigue driving state of driver based on head 3D modeling
CN120340005A
A method for detecting driver fatigue based on 3D head modeling
CN120340005B
Driver fatigue detection method and system based on deep learning
NL2039485B1
