A fatigue driving detection method based on non-contact heart rate detection and eye feature extraction

Abstract

The application discloses a kind of fatigue driving detection methods based on non-contact heart rate detection and eye feature extraction, which comprises: real-time acquisition of driver face video data, positioning intercepts apple muscle and eye site;The apple muscle and eye data collected are pretreated;Using remote photoelectric plethysmography to the apple muscle data after pretreatment Feature extraction is obtained Heart rate characteristic signal;Using the CLIP multimodal model after fine-tuning on eye data Extracts preprocessed eye visual features;Heart rate characteristic signal and eye feature matrix are fused to obtain fusion features, and the driver's fatigue score is obtained using the fusion features, and the final fatigue driving detection result is determined by threshold value.The application realizes non-contact fatigue driving detection by effectively utilizing various features of the driver's face.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of fatigue driving detection, and in particular to a fatigue driving detection method based on non-contact heart rate detection and eye feature extraction. BACKGROUND

[0002] The working time of people is gradually extended, and insufficient rest leads to more and more fatigue driving phenomenon. When the driver is in a fatigue driving state, the attention and reaction speed will sharply decrease, the decision-making ability will be affected, and the accident risk will be increased, which brings a serious threat to the safety of people's life and property, and also brings a great challenge to the urban public security management. Automatically detecting whether the driver is in a fatigue driving state and timely prompting the driver to rest is an effective way to avoid such safety problems, and can also provide data support for the responsibility identification after the accident. The fatigue detection methods for the state of the driver mainly include the contact detection by wearing instruments to monitor the physiological indexes such as the pulse and the electrocardiogram signal of the driver, and the non-contact detection by image detection to detect the facial features of the driver. The contact detection has a high cost and may affect the normal driving, and the detection and recognition algorithm based on the image detection is complex, the feature extraction is difficult, and the requirement for the technology is high. SUMMARY

[0003] In view of the problems in the prior art, the present application discloses a fatigue driving detection method based on non-contact heart rate detection and eye feature extraction, which comprehensively evaluates the heart rate features and the eye fatigue information to determine whether the driver is in a fatigue driving state.

[0004] The technical scheme of the present application is as follows:

[0005] A fatigue driving detection method based on non-contact heart rate detection and eye feature extraction, comprising the following steps:

[0006] S1: Real-time collection of driver facial video data, positioning and interception of apple muscle and eye parts;

[0007] S2: Preprocessing of the collected apple muscle and eye part images;

[0008] S3: Feature extraction of the preprocessed apple muscle data in S2 using remote photoplethysmography (rPPG), to obtain a heart rate feature signal;

[0009] S4: Extraction of the eye visual features of the preprocessed data in S2 using the CLIP multi-modal model fine-tuned on the eye data;

[0010] S5: Fuse the heart rate feature signal and the eye feature matrix obtained in S3 and S4 to obtain a fusion feature, use the fusion feature to obtain a driver fatigue score, and determine a final fatigue driving detection result through a threshold value.

[0011] S1 includes the following steps:

[0012] S11: Obtain video data of a driver driving state in a non-contact manner through a vehicle-mounted camera;

[0013] S12: Automatically frame the driver driving state video obtained by shooting;

[0014] S13: Use a face detector of a dlib library to detect the face of the driver in the video frame, locate and intercept the cheek apple muscle and eye parts of the driver.

[0015] S2 includes the following steps:

[0016] S21: Preprocess the intercepted cheek apple muscle sample frame using Fast Fourier Transform (FFT), convert the signal of the color change of the apple muscle blood vessels between the continuous sample frames into a frequency signal, and the FFT calculation formula is as follows:

[0017]

[0018] Wherein, x(n) is the nth sample value in the time domain, that is, the signal intensity of the color change of the blood vessels, and N is the total number of samples in the time domain. is the core complex exponential function of Fourier transform, wherein e is the base of natural logarithm, and j is the imaginary unit, represents the phase angle. X(k) is the complex amplitude of the kth frequency component in the frequency domain. Therefore, the frequency components in the blood vessel color change signal can be analyzed and identified.

[0019] S22: Use a Gamma correction algorithm to correct the illumination of the intercepted eye part sample frame to reduce the negative impact on the detection result caused by too strong sunlight and poor light at night, and the formula of the Gamma correction algorithm used is as follows:

[0020] f(I)=I γ

[0021] Wherein, I is the input pixel value, ranging from 0 to 1, representing the original illumination intensity in the sample frame of the eye part intercepted from the image. f(I) represents the process of gamma correction. γ is the gamma correction coefficient, and its value is usually greater than 0. When γ < 1, the overall brightness is improved, and when γ > 1, the effect is reversed.

[0022] S3 includes the following steps:

[0023] The rPPG signal of the S21 frequency signal is extracted using remote photoelectric plethysmography, and converted into a heart rate signal feature;

[0024] S4 includes the following steps:

[0025] S41: Collect eye image data of the driver in the driving state according to S22, including normal driving and different degrees of fatigue driving categories, and construct a data set (L, V) corresponding to the eye image and its corresponding driving state type, wherein V is the eye image data, and L is the corresponding language description. (L, V) contains two categories: normal and fatigue;

[0026] S42: Fine-tune the CLIP multimodal model using the eye image data set (L, V) in S41, first use ChatGPT to prompt the language description L in the data set to obtain Provide more detailed language descriptions of the eye image data corresponding to the driving state. The expansion process is completed using a prompt template, as follows:

[0027] “What are the characteristics of a driver's eyes when driving under{Class}?”

[0028] Generate more detailed descriptions by replacing {Class} with specific categories in S41 to achieve the purpose of expansion. Then use the language description represented by to fine-tune the part of the CLIP model. The fine-tuned model adapts to the distribution of human eye features, and the fine-tuning process is optimized in a contrast learning manner that matches the similarity of text and image features:

[0029]

[0030] where p(y=i|v) is the probability that the random variable y is equal to the class i given the vector v. cos(v, l) is the cosine similarity between vectors v and l. τ is the temperature parameter, the lower the τ, the more the output distribution tends to have a higher probability of class, otherwise the output distribution is smoother. is a normalization constant, ensuring that the sum of all p(y=i|v) is 1.

[0031] S43: Input the preprocessed eye image of S2 into the fine-tuned CLIP multimodal model to extract the eye feature matrix of the driver.

[0032] S5 includes the following steps:

[0033] S51: The heart rate signal features and eye feature matrices obtained in S3 and S4 are fused, specifically, the heart rate signal features and eye feature matrices are first reduced to the same dimension by principal component analysis (PCA), and then the two features are fused by a certain proportion to obtain fusion features containing both physiological indicators and visual features of the driver;

[0034] S52: A multi-layer perception (MLP) model is used to predict the fatigue score of the fusion features, and a threshold value θ is used as the judgment standard for whether the driver is in fatigue driving. The MLP is obtained by learning, and the fusion features obtained in S51 are used to train the MLP model, and the trained MLP is used to judge whether the driver state belongs to fatigue driving. If the score of fatigue driving is higher than the threshold value θ, it is considered that there is fatigue driving, and the decision process is as follows:

[0035]

[0036] Wherein, R represents the final decision, z represents the fatigue score, when the fatigue score is higher than the threshold value θ, 1 represents that the driver is in fatigue driving, and 0 represents that the driver is not in fatigue driving.

[0037] Compared with the prior art, the present application has the following advantages and effects: in the overall scheme design, the physiological indicators and eye visual features of the driver are comprehensively considered, and the judgment result is more accurate. In the recognition of eye visual features, the CLIP multi-modal model is introduced, and ChatGPT is used for prompt learning, so that the visual feature recognition ability is stronger; in the acquisition of the heart rate signal of the driver, the rPPG technology is used to extract the non-contact mode from the color change of the blood vessels of the apple muscle of the driver in the video, which reduces the interference to the driver. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is the basic flowchart of the method of the present application;

[0039] Figure 2 is the technical flowchart of the method of the present application. DETAILED DESCRIPTION

[0040] In order to make the technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application:

[0041] According to the flowchart shown in Figure 1 , the fatigue driving detection method based on non-contact heart rate detection and eye feature extraction proposed in this embodiment has the technical flowchart shown in Figure 2 , and the specific steps are as follows:

[0042] S1: Real-time collection of driver face video data, positioning and intercepting apple muscle and eye parts;

[0043] S2: Preprocessing of the collected apple muscle and eye part images;

[0044] S3: Feature extraction of the preprocessed apple muscle data in S2 using remote photoplethysmography (rPPG), obtaining a heart rate feature signal;

[0045] S4: Extracting eye visual features of the preprocessed data in S2 using a CLIP multi-modal model fine-tuned on eye data;

[0046] S5: Fusing the heart rate feature signal and eye feature matrix obtained in S3 and S4 to obtain a fusion feature, using the fusion feature to obtain the driver's fatigue score, and determining the final fatigue driving detection result through a threshold.

[0047] S1 includes the following steps:

[0048] S11: Acquiring video data of the driver's driving state in a non-contact manner through a vehicle-mounted camera;

[0049] S12: Automatically extracting frames of the captured driver driving state video using the FFmpeg algorithm library;

[0050] S13: Using the face detector of the dlib library to detect the face of the driver in the video frame, positioning and intercepting the driver's facial apple muscle and eye parts.

[0051] S2 includes the following steps:

[0052] S21: Preprocessing the extracted facial apple muscle sample frame using Fast Fourier Transform (FFT), converting the signal of the apple muscle blood vessel color change between consecutive sample frames into a frequency signal, and the FFT calculation formula is as follows:

[0053]

[0054] Where x(n) is the nth sample value in the time domain, i.e. the signal intensity of the blood vessel color change, N is the total number of samples in the time domain, and in this embodiment, N is set to 600, which is about 20 seconds at a standard frame rate. is the core complex exponential function of Fourier transform, where e is the base of natural logarithm, and j is the imaginary unit, represents the phase angle. The result X(k) is the complex amplitude of the kth frequency component in the frequency domain. Thus, the frequency components in the blood vessel color change signal can be analyzed and identified.

[0055] S22: The sampled frame of the intercepted eye part is subjected to illumination correction using a Gamma correction algorithm to reduce the negative impact on the detection result caused by excessive sunlight and poor light at night. The formula of the Gamma correction algorithm used is as follows:

[0056] f(I) = I γ

[0057] where I is the input pixel value, ranging from 0 to 1, representing the original illumination intensity in the sampled frame of the intercepted eye part. Gamma is the gamma correction coefficient, usually greater than 0. When gamma is less than 1, the overall brightness is increased, and when gamma is greater than 1, the effect is reversed.

[0058] S3 includes the following steps:

[0059] The rPPG signal in the frequency signal in S21 is extracted using remote photoplethysmography, and converted into a heart rate signal feature;

[0060] S4 includes the following steps:

[0061] S41: Collect eye image data when the driver is in a driving state, and construct a data set (L, V) corresponding to the eye image and its corresponding driving state type, where V is the eye image data and L is the corresponding language description. (L, V) contains two categories: normal and fatigue, and each category contains 500 eye images.

[0062] S42: Fine-tune the CLIP multi-modal model using the eye image data set in S41. First, use ChatGPT to prompt and expand the language description L in the data set to obtain The expansion process is achieved through the interaction of the prompt template and ChatGPT. The prompt template is "What are the characteristics of a driver's eyes when driving under {Class}?" The specific interaction process is shown in Table 1. This can provide more detailed language descriptions of eye image data corresponding to driving states.

[0063] Table 1 ChatGPT expanded language description

[0064]

[0065] Use The CLIP is fine-tuned with the language description represented by V and the eye image represented by V. In the implementation, the CLIP model of the ResNet50 version is used, and only the last attention pooling layer of the image encoder and the last feedforward neural network of the text encoder are updated. The fine-tuning process is optimized in a contrast learning manner that matches the similarity of text and image features:

[0066]

[0067] where p(y = i | v) is the probability that the random variable y equals the class i given the vector v. cos(v, l) is the cosine similarity between the vector v and the vector l. τ is a temperature parameter, which is set to 0.1 in this embodiment. is a normalization constant, which ensures that the sum of all p(y = i | v) is 1. The fine-tuning process is performed for 100 epochs, and the fine-tuned model adapts to the distribution of human eye features.

[0068] S43: input the eye image preprocessed in S2 into the image encoder of the fine-tuned CLIP model to extract the eye feature matrix of the driver, which has a size of 197x1024, where 197 represents 1 class label and 196 visual labels, and 1024 is the dimension of each label.

[0069] S5 includes the following steps:

[0070] S51: fuse the heart rate signal feature and the eye feature matrix obtained in S3 and S4, and apply principal component analysis (PCA) to reduce the dimension of the heart rate signal feature and the eye feature matrix to the same dimension. In the implementation, the eye feature matrix is first pooled along the channel dimension by global average pooling, and the class label is added to the visual label to obtain the overall representation of the eye feature, which is a vector of length 1024 at this time. Since the number of samples in the time domain is set to 600 in S21, the heart rate feature is a vector of length 600, representing the heart rate feature signal for about 20 consecutive seconds. We use PCA to reduce the dimension of the eye feature and the heart rate signal, and the dimension after reduction is 512. Then, the two features are fused by a certain proportion, which is set to 20% for the heart rate signal feature and 80% for the eye feature through experimental evaluation, and the two are added. In this way, a fusion feature containing both the physiological indicators and the visual features of the driver is obtained, which is a vector of length 512;

[0071] S52: Use the multi-layer perception (MLP) model to predict the fatigue score of the fusion features, and use the threshold θ as the judgment standard for whether the driver is in fatigue driving. The MLP is implemented as two fully connected layers, the input and output dimensions of the first layer are 512 and 256 respectively, and the input and output dimensions of the second layer are 256 and 2 respectively. Among them, 2 represents the number of categories, that is, whether to be tired. The MLP is learned by using the vector after fusing the heart rate signal features and eye features as input, and using the discrete label of the category to which the eye features belong as supervision to learn the MLP within 20 epochs. The learned MLP is used to judge whether the driver belongs to fatigue driving, if it is judged that the fatigue driving score exceeds the threshold θ, it is considered that there is fatigue driving, the decision process is as follows:

[0072]

[0073] Among them, R represents the final decision, z represents the fatigue score, and the threshold θ is set to 0.7 when implemented.

[0074] Compared with the prior art, the method proposed in the embodiment considers the non-contact heart rate detection which is not considered in the previous fatigue driving detection method, and comprehensively evaluates the heart rate features and eye fatigue information to determine whether the driver is in a fatigue driving state. In the extraction of eye visual features, the CLIP model is fine-tuned to make it understand eye image data, and ChatGPT is used for language description expansion to improve the extraction ability of visual features. In addition, the rPPG technology is used to extract the changes of heart rate features to avoid the problem of affecting normal driving in the contact fatigue driving detection method. Finally, through verification, this method realizes an identification accuracy of 98.32% on actual fatigue driving detection data, which is higher than the prior art.

Claims

1. A fatigue driving detection method based on non-contact heart rate detection and eye feature extraction, characterized in that, Includes the following steps: S1: Real-time acquisition of driver's facial video data, locating and cropping the cheekbones and eyes; S2: Preprocess the collected images of the cheekbones and eyes; S3: Use remote photoplethysmography (rPPG) to extract features from the preprocessed apple cheek data in S2 to obtain heart rate feature signals; S4: Use the CLIP multimodal model, finely tuned on the eye data, to extract the eye visual features from the preprocessed data in S2; S5: The heart rate feature signal and eye feature matrix obtained in S3 and S4 are fused to obtain the fused feature. The driver's fatigue score is obtained using the fused feature, and the final fatigue driving detection result is determined by the threshold. S1 includes the following steps: S11: Acquire video data of the driver's driving status in a non-contact manner using an in-vehicle camera; S12: Automatically extract frames from the captured video of the driver's driving status; S13: Use the face detector of the dlib library to detect the driver's face in the video frame, locate and crop the driver's cheek and eye areas; S2 includes the following steps: S21: The sampled frames of the facial cheek area are preprocessed using Fast Fourier Transform (FFT) to convert the signal of color change of blood vessels in the cheek area between consecutive sampled frames into a frequency signal. The FFT calculation formula is as follows: Where x(n) is the nth sample value in the time domain, i.e. the signal intensity of the blood vessel color change, and N is the total number of samples in the time domain; It is the core complex exponential function of the Fourier transform, where e is the base of the natural logarithm and j is the imaginary unit. X(k) represents the phase angle; X(k) is the complex amplitude of the k-th frequency component in the frequency domain; therefore, it is possible to analyze and identify the frequency components in the blood vessel color change signal. S22: The gamma correction algorithm is used to correct the illumination of the sampled frames of the eye area to reduce the negative impact of strong sunlight and poor lighting at night on the detection results. The formula of the gamma correction algorithm used is as follows: f(I)=I γ Where I is the input pixel value, ranging from 0 to 1, representing the original illumination intensity in the sampled frame of the eye area extracted from the image; f(I) represents the gamma correction process; γ is the gamma correction coefficient; S3 includes the following steps: The rPPG signal of the frequency signal in S21 was extracted using remote photoplethysmography and converted into heart rate signal characteristics. S4 includes the following steps: S41: Collect eye image data of the driver while driving according to S22, including normal driving and driving under different degrees of fatigue. Construct a dataset (L,V) by corresponding the eye images and their corresponding driving state types, where V is the eye image data and L is the corresponding language description; (L,V) contains two categories: normal and fatigued. S42: Fine-tuning and optimizing the CLIP multimodal model using the eye image dataset (L,V) from S41. First, ChatGPT is used to augment the language description L in the dataset with prompts. Provides richer verbal descriptions of driving states corresponding to eye image data; the expansion process is completed using prompt templates, as follows: "What are the characteristics of a driver's eyes when driving under{Class}?" The expansion is achieved by replacing {Class} with specific categories in S41 to generate a more detailed description; subsequently, it is used... The language description and the eye image represented by V are used to fine-tune some layers of CLIP; the fine-tuned model adapts to the distribution of human eye features, and the fine-tuning process is optimized through contrastive learning by matching text and image feature similarity. Where p(y=i|v) is the probability that the random variable y equals class i given vector v; cos(v,l) is the cosine similarity between vector v and vector l; τ is a temperature parameter, the lower the τ, the more the output distribution tends to be of the class with higher probability, and vice versa. It is a normalization constant that ensures that the sum of all p(y=i|v) is 1; S43: Input the preprocessed eye image from S2 into the fine-tuned CLIP multimodal model to extract the driver's eye feature matrix; S5 includes the following steps: S51: The heart rate signal features and eye feature matrices obtained in S3 and S4 are fused. Specifically, principal component analysis is first applied to the heart rate signal features and eye feature matrices respectively to reduce their dimensionality to the same dimension. Then, the two features are fused in a certain proportion to obtain a fused feature that simultaneously contains the driver's physiological indicators and visual features. S52: A multilayer perceptron (MLP) model is used to predict fatigue scores based on the fused features, and a threshold θ is used as the criterion for judging whether the driver is driving while fatigued. The MLP is learned and trained using the fused features obtained in S51. The trained MLP is then used to determine whether the driver's state is fatigued. If the score indicating fatigue exceeds the threshold θ, then fatigued driving is considered to exist. The decision-making process is as follows: Where R represents the final decision and z represents the fatigue score. When the fatigue score is higher than the threshold θ, 1 indicates that the driver is driving while fatigued, and 0 indicates that the driver is not driving while fatigued.

Images