Driver heart rate detection method and device based on signal quality evaluation

Abstract

The invention discloses a driver heart rate detection method and device based on signal quality evaluation, and belongs to the technical field of video non-contact heart rate detection. The method comprises the following steps: acquiring a face video signal of a driver, and obtaining a face detection area through the face video signal; dividing a face detection area, obtaining a face pulse signal based on the face detection area, and processing the face pulse signal; performing signal quality evaluation on the processed face pulse signal; and carrying out heart rate detection by utilizing the evaluated face pulse signal. According to the method, signal quality evaluation is introduced, the collected face pulse signal quality of the driver is evaluated in real time, the heart rate detection strategy is adjusted according to the evaluation result, the detection strategy can be adjusted in a self-adaptive mode in different vehicle-mounted environments, motion interference and illumination interference of heart rate detection in the vehicle-mounted environments are overcome, and the robustness of heart rate detection is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video non-contact heart rate detection, and more specifically to a driver heart rate detection method and device based on signal quality evaluation. Background Art

[0002] Human factors are the main cause of road traffic accidents, and continuous monitoring of the driver's physiological state can effectively prevent accidents. The driver monitoring system can monitor the driver's behavioral characteristics in real time, and its heart rate changes are an effective way to assess the driver's fatigue and concentration, which plays a vital role in ensuring driving safety.

[0003] When using an electrocardiogram or pulse oximeter to measure heart rate, the device needs to be in direct contact with the driver, which will cause discomfort to the driver during driving. With the rapid development of computer vision, video photoplethysmography can be used as a technical basis for measuring human vital signs by detecting subtle color changes caused by blood pulsation on the human face. Since the near-infrared camera in the cockpit increases the light reflected from the face, pulse measurement can be achieved without interfering with the driver's driving state. However, the near-infrared video of the driver obtained by the on-board near-infrared camera is monochrome, and the near-infrared light has a longer wavelength than visible light. When it penetrates the skin tissue, it will be more affected by motion artifacts, and there is an inaccurate heart rate measurement problem caused by the technology itself. In addition, due to motion artifacts caused by vehicle movement, driver dialogue, facial expressions, and rearview mirror viewing, the driver's facial ambient light distribution is uneven during driving, and the face has rapid and drastic brightness changes. The motion interference and light interference caused by the acquisition environment will lead to inaccurate extraction of the driver's heart rate.

[0004] After searching, the Chinese patent application, application number 202410156801.0, published on June 7, 2024, discloses a method and device for in-vehicle physical examination of the driver. The method points out that the on-board DMS camera can be used to detect the driver's heart rate, heart rate variability, respiratory rate, blood pressure when the vehicle is static, and form a health check report. However, this method lacks a signal quality evaluation link, resulting in poor accuracy of the detection indicators, and mainly detects the driver's heart rate in a stationary vehicle environment, which cannot solve the problem of robust detection of the vehicle's heart rate while driving. From the above, it can be seen that the relevant technology does not provide any technical inspiration for the problem of insufficient accuracy of driver heart rate detection in a dynamic environment. Summary of the invention

[0005] 1. Technical problems to be solved

[0006] Aiming at the problem of insufficient accuracy in detecting the driver's heart rate in the existing in-vehicle DMS system, the present invention provides a method and device for detecting the driver's heart rate based on signal quality assessment, which realizes highly accurate driver heart rate detection through a comprehensive signal filtering and quality assessment method.

[0007] 2. Technical solution

[0008] The object of the present invention is achieved through the following technical solutions.

[0009] A method for detecting the driver's heart rate based on signal quality assessment includes the following steps:

[0010] Collect the facial video signal of the driver and obtain the facial detection area through the facial video signal;

[0011] Divide the facial detection area, obtain the facial pulse signal based on the facial detection area, and process the facial pulse signal;

[0012] Perform signal quality assessment on the processed facial pulse signal;

[0013] Use the evaluated facial pulse signal to detect the heart rate.

[0014] As a further improvement of the present invention, divide the facial detection area through a face recognition tool, extract the key area in the facial detection area, and further divide the key area into several sub-areas.

[0015] As a further improvement of the present invention, extract the original facial pulse signal from several sub-areas and use a band-pass filter to remove the noise interference in the original facial pulse signal.

[0016] As a further improvement of the present invention, convert the denoised original facial pulse signal into a multi-dimensional vector sequence through delay coordinate transformation.

[0017] As a further improvement of the present invention, extract the facial pulse signal from the multi-dimensional vector sequence based on the orthogonal matrix image transformation filtering method, specifically including: in the multi-dimensional vector sequence, let A represent the n×3 matrix of the original pulse signal, n represents the signal window length, 3 represents the number of delayed channels, perform QR decomposition through Householder reflection, Householder reflection iteratively transforms the n×3 matrix A of the original pulse signal into an upper triangular matrix R, accumulate the orthogonal transformation into the decomposed orthogonal matrix Q, and the column vectors of the decomposed orthogonal matrix Q include q1, q2, and q3, and the q1, q2, and q3 form an orthonormal basis of the column space of the n×3 matrix A of the original pulse signal.

[0018] As a further improvement of the present invention, the noise component captured by q1 is removed, and a projection matrix P is constructed to project the original facial pulse signal onto a subspace orthogonal to q1. The projection matrix P is defined as:

[0019] P = I n - SS T

[0020] Wherein, I n represents an n×n identity matrix, S = q1, and S T represents the transpose operation of q1.

[0021] As a further improvement of the present invention, after obtaining the projection matrix P, the n×3 matrix A of the original pulse signal is projected onto the orthogonal subspace to eliminate the influence of the noise component, and the facial pulse signal is obtained; the orthogonal projection is expressed as:

[0022] Y = PA

[0023] Wherein, Y represents the facial pulse signal of the n×3 matrix.

[0024] As a further improvement of the present invention, the signal quality of the processed facial pulse signal is evaluated. The signal quality is evaluated by analyzing the characteristics of the facial pulse signal, and the comprehensive score of the characteristics is calculated by using the weighted average method;

[0025] Suppose n features F 1 , F 2 ,..., F n are extracted, and each feature corresponds to a score S 1 , S 2 ,..., S n . The comprehensive score is calculated by using the weighted average method, and the calculation formula is:

[0026] S total = w 1 ·S 1 + w 2 ·S 2 +…+ w n ·S n

[0027] Wherein, S total represents the comprehensive score, and w 1 , w 2 , w n represent the weights corresponding to each feature.

[0028] As a further improvement of the present invention, the heart rate main frequency of each region signal of the facial pulse signal obtained for each region is Fourier-transformed to the frequency domain, and the heart rate of each region is calculated through the heart rate main frequency. The calculation formula is:

[0029]

[0030] Among them, t represents the signal acquisition time, i represents the i-th facial area, and HR i [t] represents the heart rate result measured in the i-th facial area, and Freq i represents the main frequency of the facial pulse signal obtained by Fourier transform, fps represents the video frame rate, and N represents the total number of signal frames.

[0031] A driver heart rate detection device based on signal quality assessment, comprising:

[0032] A signal acquisition module that acquires the facial video signal of the driver and obtains the facial detection area through the facial video signal;

[0033] A signal processing module that divides the facial detection area, obtains the facial pulse signal based on the facial detection area, and processes the facial pulse signal;

[0034] A signal quality assessment module that assesses the signal quality of the processed facial pulse signal;

[0035] A heart rate detection module that uses the evaluated facial pulse signal for heart rate detection.

[0036] 3. Beneficial effects

[0037] Compared with the prior art, the advantages of the present invention are as follows:

[0038] The driver heart rate detection method and device based on signal quality assessment provided by the present invention introduce a signal quality assessment link to real-time assess the quality of the acquired infrared facial signal of the driver, and adjust the heart rate detection strategy according to the assessment result, which can adaptively adjust the detection strategy in different vehicle-mounted environments. The signal quality assessment of the present invention is based on multi-level feature extraction and real-time feedback mechanism, which can adaptively adjust the detection strategy in different vehicle-mounted environments, overcome the motion interference and light interference of heart rate detection in vehicle-mounted environments, improve the robustness of heart rate detection, and has strong practicability and wide applicability. Brief description of the drawings

[0039] Figure 1 It is a schematic flow chart of the driver heart rate detection method in the embodiment of the present invention;

[0040] Figure 2 It is a schematic diagram of the comparison between the driver heart rate detection value and the reference value in the embodiment of the present invention. Detailed implementation manners

[0041] The present invention will be described in detail below in conjunction with the specification drawings and specific embodiments.

[0042] Embodiment

[0043] As shown in Figure 1 Figure [not shown], a driver heart rate detection method based on signal quality assessment provided in this embodiment includes the following steps: collecting a facial video signal of the driver, and obtaining a facial detection area from the facial video signal; dividing the facial detection area, obtaining a facial pulse signal based on the facial detection area, and processing the facial pulse signal; performing signal quality assessment on the processed facial pulse signal; and performing heart rate detection using the evaluated facial pulse signal.

[0044] Specifically, in this embodiment, a driver's facial infrared video is collected by an in-vehicle infrared camera under various conditions such as stationary, low-speed daytime driving, low-speed night driving, and driver facial movement.

[0045] Specifically, a monochrome CMOS image sensor of the in-vehicle system is used, and two near-infrared light-emitting diodes with a wavelength of 940 nm are equipped as illumination sources. The driver's facial infrared video is obtained in real time through the vehicle head unit, and the video is stored as a data set at the same time. In this embodiment, the camera is located below the steering wheel, facing the driver directly, with an average distance from the driver of 0.3 m to 0.5 m. The video is recorded at a resolution of 1280 pixels × 800 pixels, the video frame rate is 30 frames, and the video duration is > 1 minute.

[0046] Thus, the collected driver's facial infrared video contains common driving scenario data, ensuring the diversity of the data. The facial video signal of the driver is collected from the obtained facial infrared video, and the facial detection area is obtained from the collected facial video signal.

[0047] Further, through the existing seetaface face recognition tool, the entire face is divided into multiple detection areas, including the forehead, cheeks, eyes, etc. The key areas in the facial detection area are extracted, and the key areas are further divided into several sub-areas. In this embodiment, the key areas include the cheek area and the chin area, excluding signal-noisy areas such as facial features and the easily covered forehead. It should be noted that since there are some motion and light interferences during driving, which may affect the signal extraction quality, and the sub-areas can extract signals from each divided area respectively, and the facial video signal in the most stable area is used to calculate the heart rate. In this embodiment, the key area is divided into 42 sub-areas. Since each sub-area corresponds to a different part of the face, when facing different driving environments, the sub-areas of different parts provide blood flow information with different stabilities, and the original pulse signal is dynamically extracted from the facial video signal within the 42 sub-areas using the existing video photoplethysmography.

[0048] Furthermore, a band-pass filter is used to remove the noise interference in the original facial pulse signal. Specifically, in a vehicle environment, due to factors such as the minute movements of the driver's face, vehicle vibrations, and ambient light changes, the extracted original pulse signal contains strong noise interference, which affects the accuracy of heart rate extraction. Since a band-pass filter allows signals within a specific frequency range to pass through while suppressing frequency components outside that range and can also filter out strong interference in the signal. Therefore, in this embodiment, in vehicle heart rate detection, the frequency range of the band-pass filter is set between the heart rate fluctuation frequencies of 0.7 Hz and 2.5 Hz to remove the noise interference in the original facial pulse signal.

[0049] Due to the non-linearity of physiological signals, the use of the delay embedding method to reconstruct the system allows for the extraction of complex dynamic features hidden in a single signal. The time-delay embedding theorem reconstructs from a single observable signal, which is equivalent to a high-dimensional state space of the true state space of the system. Therefore, in this embodiment, the denoised original facial pulse signal is converted into a multi-dimensional vector sequence through delay coordinate transformation. Specifically, for the single-channel signal Ak[n] in the k-th key region, through delay coordinate transformation, it is converted into a multi-dimensional vector sequence. A delay time and an embedding dimension are set to construct a high-dimensional vector space, which is represented as:

[0050] X k [n] = [a k [n], a k [n + τ], a k [n + 2τ], … a k [n + (m - 1)τ]]

[0051] where n and k are both natural numbers, X k [n] represents the high-dimensional vector space, a k [n] represents the average pixel value within the key region of the n-th frame, m represents the embedding dimension, and τ represents the delay time, which can be selected according to the time series characteristics.

[0052] The embedding matrix can be constructed from multiple consecutive delay vectors and is represented as:

[0053]

[0054] where A k represents the embedding matrix, Q represents an n×3 orthogonal matrix, Q = H 1 H 2 … H k , H k represents a Householder reflector used to zero out the elements below the diagonal in each column of the n×3 matrix A of the original pulse signal.

[0055] Furthermore, the facial pulse signal is extracted from the multi-dimensional vector sequence based on the orthogonal matrix image transformation filtering method. Specifically, in the multi-dimensional vector sequence, let A represent the n×3 matrix of the original pulse signal, where n represents the signal window length and 3 represents the number of delayed channels. QR decomposition is achieved through Householder reflection. The Householder reflection iteratively transforms the n×3 matrix A of the original pulse signal into an upper triangular matrix R, and accumulates the orthogonal transformation into the decomposed orthogonal matrix Q. The decomposition process is expressed as:

[0056] A = QR

[0057] After three iterations (k = 3), the product of the Householder reflectors produces the orthogonal matrix Q. The column vectors of the orthogonal matrix Q include q1, q2, and q3, and q1, q2, and q3 form an orthonormal basis for the column space of the n×3 matrix A of the original pulse signal.

[0058] In the driving scenario, the energy changes such as external environmental lighting changes, sensor noise, and facial movements are relatively strong. The direction that causes the most significant changes in the facial pulse signal space captured by q1 usually represents noise. Therefore, the noise component captured by q1 is removed, and the projection matrix P is constructed to project the original facial pulse signal onto the subspace orthogonal to q1. The projection matrix P is defined as:

[0059] P = I n - SS T

[0060] where I n represents the n×n identity matrix, S = q1, and S T represents the transpose operation on q1.

[0061] After obtaining the projection matrix P, the n×3 matrix A of the original pulse signal is projected onto the orthogonal subspace to eliminate the influence of the noise component, and the facial pulse signal is obtained; the orthogonal projection is expressed as:

[0062] Y = PA

[0063] where Y represents the n×3 matrix of the facial pulse signal, and the first column has been effectively suppressed. The second column is taken as the filtered facial pulse signal, which retains the physiological changes related to blood volume changes and minimizes the irrelevant noise.

[0064] The signal quality of the filtered facial pulse signal is evaluated. The signal quality is evaluated by analyzing the characteristics of the facial pulse signal. In this embodiment, the characteristics of the facial pulse signal include characteristics such as signal-to-noise ratio, spectral characteristics, and time-domain stability. The comprehensive score of the characteristics is calculated by using a weighted average method;

[0065] Set to extract n features F1 , F 2 ,..., F n , each feature corresponds to a score S 1 , S 2 ,..., S n , calculate the comprehensive score by weighted average, and the calculation formula is:

[0066] S total = w 1 ·S 1 + w 2 ·S 2 + … + w n ·S n

[0067] Among them, S total represents the comprehensive score, and w 1 , w 2 , w n represent the weights corresponding to each feature.

[0068] Furthermore, use the evaluated facial pulse signal for heart rate detection. Cluster analyze the heart rate results of several sub-regions, and according to the short-term stationarity of the heart rate, select the results with relatively stable heart rate as the heart rate results of the sub-regions. Then, fuse the heart rate information of multiple regions to obtain the final heart rate estimation value. In the clustering results, the heart rate should occupy the main component. Therefore, count the number of samples in each cluster, eliminate the cluster component with the smallest number and the classes with too few data points, and remove obvious noise points. Specifically, Fourier transform the facial pulse signal obtained for each region to the heart rate main frequency of the signal in each region in the frequency domain, and calculate the heart rate of each region through the heart rate main frequency. The calculation formula is:

[0069]

[0070] Among them, t represents the signal acquisition time, i represents the i-th facial region, and HR i [t] represents the heart rate result measured in the i-th facial region, Freq i represents the main frequency of the facial pulse signal obtained by Fourier transform, fps represents the video frame rate, N represents the total number of frames of the signal. In this embodiment, the total number of frames of the signal is 512 frames, and the heart rate result for a continuous period of time can be obtained. In this embodiment, an existing multi-region heart rate fusion algorithm based on Gaussian mixture model is used to synthesize the heart rate information of each sub-region, and the most representative heart rate detection result is screened and output.

[0071] Such as Figure 2As shown, the detection results of the heart rate are presented. Specifically, they are the results of continuously detecting the heart rate for a set of 10-minute driving data, including the heart rate reference value recorded by a pulse oximeter and the heart rate measurement value calculated by the detection method provided in this embodiment. From Figure 2 it can be seen that when the heart rate reference value changes, the heart rate measurement value calculated in this embodiment is in good synchronization with the heart rate reference value.

[0072] A driver heart rate detection method based on signal quality assessment provided in this embodiment introduces a signal quality assessment method to real-time evaluate the quality of the collected infrared facial signals of the driver. According to the evaluation results, the heart rate detection strategy is adjusted. Low-quality signals are directly removed, and high-quality signals are filtered and then the heart rate is calculated. Its signal quality assessment is based on multi-level feature extraction and a real-time feedback mechanism, which can adaptively adjust the detection strategy in different vehicle-mounted environments, overcome the motion interference and light interference in heart rate detection in vehicle-mounted environments, and improve the robustness of heart rate detection.

[0073] This embodiment also provides a driver heart rate detection device based on signal quality assessment, including a signal acquisition module, a signal processing module, a signal quality assessment module, and a heart rate detection module. The signal acquisition module collects the facial video signal of the driver and obtains the facial detection area through the facial video signal. The signal processing module divides the facial detection area, obtains the facial pulse signal based on the facial detection area, and processes the facial pulse signal. The signal quality assessment module assesses the signal quality of the processed facial pulse signal. The heart rate detection module uses the evaluated facial pulse signal to detect the heart rate. The driver heart rate detection device based on signal quality assessment provided in this embodiment can implement any of the methods of the driver heart rate detection method based on signal quality assessment, and the specific working process of the driver heart rate detection device based on signal quality assessment can refer to the corresponding process in the embodiment of the driver heart rate detection method based on signal quality assessment. The methods and devices provided in this embodiment can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for example, the division of a certain module is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed connections or communication connections to each other can be indirect coupling or communication connections through some interfaces, devices or units, and can also be electrical, mechanical or other forms of connections.

[0074] This embodiment also provides a computer device. A computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the method for detecting a driver's heart rate based on signal quality assessment as described above.

[0075] This embodiment also provides a computer-readable storage medium. A computer-readable storage medium stores a computer program. When the computer program is run by a processor, it executes the method for detecting a driver's heart rate based on signal quality assessment in this embodiment. Among them, a computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device; the program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination of the above.

[0076] The present invention and its implementation manners are schematically described above. This description is not restrictive. Without departing from the spirit or basic characteristics of the present invention, the present invention can be implemented in other specific forms. What is shown in the drawings is only one of the implementation manners of the present invention, and the actual structure is not limited thereto. Any reference numeral in the claims should not limit the claimed rights. Therefore, if those of ordinary skill in the art are inspired by it and design similar structural manners and embodiments to this technical solution without creative efforts without departing from the purpose of this creation, they should all fall within the protection scope of the present invention. In addition, the word "including" does not exclude other elements or steps, and the word "a" before an element does not exclude including "a plurality of" such elements. The plurality of elements stated in the product claims can also be implemented by one element through software or hardware. The words "first", "second", etc. are used to indicate names and do not indicate any specific order.

Claims

1. A driver heart rate detection method based on signal quality evaluation, comprising the following steps: Collecting the driver's facial video signal, and obtaining the facial detection area through the facial video signal; Dividing a facial detection area, obtaining a facial pulse signal based on the facial detection area, and processing the facial pulse signal; Performing signal quality assessment on the processed facial pulse signal; Heart rate detection is performed using the evaluated facial pulse signal.

2. A driver heart rate detection method based on signal quality evaluation according to claim 1, characterized in that: The face detection area is divided by a face recognition tool, and the key area in the face detection area is extracted, and the key area is further divided into several sub-areas.

3. A driver heart rate detection method based on signal quality evaluation according to claim 2, characterized in that: The original facial pulse signal is extracted from several sub-regions, and the noise interference in the original facial pulse signal is removed by using a bandpass filter.

4. The driver heart rate detection method based on signal quality evaluation according to claim 3 is characterized in that: The denoised original facial pulse signal is converted into a multi-dimensional vector sequence through delayed coordinate transformation.

5. The driver heart rate detection method based on signal quality evaluation according to claim 4, characterized in that: The invention relates to extracting facial pulse signals from a multidimensional vector sequence based on an orthogonal matrix image transformation filtering method, which specifically includes: in the multidimensional vector sequence, A is set to represent an n×3 matrix of an original pulse signal, n represents a signal window length, 3 represents the number of delayed channels, and QR decomposition is implemented through Householder reflection. The Householder reflection transforms the n×3 matrix A of the original pulse signal into an upper triangular matrix R through iteration, and the orthogonal transformation is accumulated into an orthogonal matrix Q after decomposition. The column vectors of the decomposed orthogonal matrix Q include q1, q2 and q3, and the q1, q2 and q3 form a standard orthogonal basis of the column space of the n×3 matrix A of the original pulse signal.

6. The driver heart rate detection method based on signal quality evaluation according to claim 5, characterized in that: Remove the noise component captured by q1, construct the projection matrix P, and project the original facial pulse signal onto a subspace orthogonal to q1. The projection matrix P is defined as: P=I n -SS T Among them, I n represents the n×n identity matrix, S=q1, S T Indicates the transpose operation on q1.

7. The driver heart rate detection method based on signal quality evaluation according to claim 6, characterized in that: After obtaining the projection matrix P, the n×3 matrix A of the original pulse signal is projected onto the orthogonal subspace to eliminate the influence of the noise component and obtain the facial pulse signal; the orthogonal projection is expressed as: Y=PA Wherein, Y represents the facial pulse signal of n×3 matrix.

8. The driver heart rate detection method based on signal quality evaluation according to claim 7, characterized in that: The signal quality of the processed facial pulse signal is evaluated by analyzing the characteristics of the facial pulse signal and a comprehensive score of the characteristics is calculated by weighted average method; Set to extract n features F1, F2, ..., F n , each feature corresponds to a score S1,S2,...,S n , the comprehensive score is calculated by weighted average, and the calculation formula is: S total =w1·S1+w2·S2+…+w n ·S n Among them, S total Indicates the comprehensive score, w1, w2, w n Indicates the weight corresponding to each feature.

9. The driver heart rate detection method based on signal quality evaluation according to claim 8, characterized in that: The facial pulse signal obtained in each area is Fourier transformed into the main frequency of the heart rate of each area signal in the frequency domain, and the heart rate of each area is calculated by the main frequency of the heart rate. The calculation formula is: Where t represents the signal acquisition time, i represents the i-th facial area, HR i [t] represents the heart rate result measured in the i-th facial area, Freq i represents the main frequency of the facial pulse signal obtained by Fourier transform, fps represents the video frame rate, and N represents the total number of frames of the signal.

10. A driver heart rate detection device based on signal quality evaluation, characterized in that: include: A signal acquisition module collects the driver's facial video signal and obtains a facial detection area through the facial video signal; A signal processing module divides the facial detection area, obtains a facial pulse signal based on the facial detection area, and processes the facial pulse signal; A signal quality evaluation module, which performs signal quality evaluation on the processed facial pulse signal; The heart rate detection module uses the evaluated facial pulse signal to perform heart rate detection.

Images