Fatigue driving monitoring system based on millimeter wave radar

Abstract

The invention belongs to the technical field of fatigue driving monitoring, and particularly relates to a fatigue driving monitoring system based on a millimeter wave radar. The invention discloses a fatigue driving monitoring system based on a millimeter-wave radar. The millimeter-wave radar is installed at a preset position of a vehicle and transmits a collected radar echo signal of a driver to a data return module; the fatigue driving detection module carries out detection through the face fatigue detection module and the physical sign fatigue detection module. And the fatigue index calculation module brings the obtained first data and second data into a fatigue index calculation formula to obtain a fatigue index FI of the driver. The fatigue driving monitoring system based on the millimeter wave radar disclosed by the invention is not influenced by the environment, can accurately detect the fatigue state of a driver during vehicle driving in real time, and is high in detection efficiency and high in detection precision.

Description

Technical Field

[0001] The present invention belongs to the technical field of fatigue driving monitoring, and particularly relates to a fatigue driving monitoring system based on a millimeter-wave radar. Background Art

[0002] With the continuous improvement of economic development and transportation networks, the vehicle ownership has increased explosively, and road traffic has become increasingly busy. According to the reports released by authoritative traffic data statistics agencies, the number of casualties caused by traffic accidents remains high globally every year, and fatigue driving is one of the key factors leading to traffic accidents. In scenarios such as long-distance transportation and night driving, the proportion of accidents caused by fatigue driving is particularly prominent, bringing disasters to countless families and imposing a heavy economic burden on society.

[0003] The current mainstream fatigue driving detection technologies are facing a series of problems that need to be solved urgently. For example, in the detection system based on behavior analysis, it mainly relies on cameras to capture the facial expressions and eye movements of drivers to judge the fatigue state. However, the actual driving environment is complex and changeable. Drivers may interfere with the normal recognition of the camera due to factors such as wearing sunglasses, vehicle interior decorations blocking the line of sight, and frequently adjusting the sitting posture, resulting in deviations in the detection results. Moreover, in extreme light conditions such as backlight and low light, the image quality collected by the camera deteriorates severely, making it difficult for the fatigue detection algorithm based on image analysis to operate accurately. On the other hand, users generally worry about the possible privacy infringement of the camera.

[0004] Another type of technology that relies on monitoring the physiological signals of drivers, such as obtaining data such as heart rate and brain waves through wearable devices to analyze the degree of fatigue. Although it can theoretically reflect the fatigue state more accurately, in actual use, drivers often have a resistance to wearing additional devices, which greatly limits the popularization and application of such technologies. In addition, the battery life of these devices is limited, and the signal transmission is also easily interfered by in-vehicle electronic devices, and the stability is difficult to guarantee.

[0005] The rise of millimeter-wave radar technology has brought hope for solving the problem of fatigue driving detection. Millimeter-wave radar uses electromagnetic waves in the millimeter-wave band for target detection, and its unique physical characteristics endow it with many advantages. It can work normally under harsh weather conditions such as rain, snow, and fog, and is not affected by light changes, with excellent environmental adaptability. In the complex in-vehicle environment, millimeter-wave radar can effectively penetrate objects such as clothing and seats, accurately detect the physiological characteristics such as the body micro-movement and breathing frequency of the driver, as well as the dynamic changes of the head and limbs. By deeply analyzing these data, the fatigue state of the driver can be evaluated in real time and accurately.

[0006] Although millimeter-wave radar shows great potential in the field of fatigue driving detection, there are still some technical bottlenecks in its current applications. For example, how to further optimize the signal processing algorithm of millimeter-wave radar to improve the recognition accuracy of drivers' subtle movements and physiological signals; how to reduce the cost of millimeter-wave radar systems to make them more conducive to large-scale commercial applications; and how to better integrate millimeter-wave radar with other vehicle-mounted systems to achieve more comprehensive and intelligent fatigue driving warning and intervention. These issues need to be further studied and resolved to promote the wide application of millimeter-wave radar in the field of fatigue driving detection and ensure road traffic safety.

[0007] Therefore, in view of the above problems, further improvements are made. Summary of the Invention

[0008] The main object of the present invention is to provide a fatigue driving monitoring system based on millimeter-wave radar, which is not affected by the environment, can accurately detect the fatigue state of drivers in moving vehicles in real time, and has high detection efficiency and high detection accuracy.

[0009] To achieve the above object, the present invention provides a fatigue driving monitoring system based on millimeter-wave radar, including a millimeter-wave radar, a data transmission module, a fatigue driving detection module, a fatigue index calculation module, and a status discrimination and warning module, wherein:

[0010] The millimeter-wave radar is installed at a preset position of the vehicle and transmits the collected radar echo signals of the driver to the data transmission module, and the data transmission module transmits the radar echo signals to the fatigue driving detection module;

[0011] The fatigue driving detection module performs detections respectively through a facial fatigue detection module and a physical sign fatigue detection module, wherein:

[0012] The facial fatigue detection module periodically counts the nodding and blinking times of the driver respectively through a head movement frequency monitoring algorithm and a periodic eyelid vibration analysis algorithm to obtain first data (including nodding and blinking);

[0013] The physical sign fatigue detection module respectively detects the heart rate and respiration frequencies of the driver through a micro-motion heart rate extraction algorithm and a periodic respiration detection algorithm to obtain second data (including heart rate and respiration);

[0014] The fatigue index calculation module substitutes the obtained first data and second data into the fatigue index calculation formula to obtain the fatigue index FI of the driver;

[0015] The state discrimination and early warning module compares the driver's fatigue index FI with the fatigue state index range. If it is determined that the driver is in a fatigued state, early warning reminders will be given according to the fatigue level. If it is not detected that the driver is fatigued, the detection and determination will continue.

[0016] As a further preferred technical solution of the above technical solution, the millimeter-wave radar includes a first millimeter-wave radar and a second millimeter-wave radar. Both the first millimeter-wave radar and the second millimeter-wave radar are equipped with a data transmission module, where:

[0017] The first millimeter-wave radar is installed at the top of the vehicle windshield, and the angle is adjusted to align with the driver's chest cavity. The first data transmission module continuously collects echo data, including information on breathing, heartbeat, blinking, and nodding. Among them, the information on breathing and heartbeat is obtained through the chest vibration data in the echo data (i.e., the original data of breathing and heartbeat, and the breathing rate and heart rate will be calculated based on this data later);

[0018] The second millimeter-wave radar is installed in the front left of the driver's seat and monitors the upper body of the driver, monitoring the limb movement information of the driver including the arms.

[0019] As a further preferred technical solution of the above technical solution, the specific implementation of the head movement frequency monitoring algorithm of the facial fatigue detection module is as follows:

[0020] Use the first millimeter-wave radar to obtain the I / Q quadrature signal in real time and calculate the signal amplitude:

[0021]

[0022] Among them, I(t) is the in-phase component, representing the real part of the radar echo; Q(t) is the quadrature component, representing the imaginary part of the radar echo. Apply a band-pass filter with a frequency range of 0.1 Hz - 1 Hz to the signal A(t) to suppress background noise and other interferences, and retain the low-frequency components corresponding to the small head displacements;

[0023] Perform STFT on the filtered signal. The time window is 1 - 2 seconds, and use the Hann window for smoothing processing. The formula is:

[0024]

[0025] Among them, STFT nod (t,f) is the energy distribution of the nodding-related signal in the time t and frequency f dimensions; t is the current time point; f is the frequency; A nod (t′) is the amplitude of the preprocessed nodding signal; w(t′ - t) is the window function, used to intercept the local signal; e -j2πft′ is the complex exponential carrier, which realizes frequency modulation;

[0026] Integrate the STFT result in the frequency band of 0.1 Hz - 1 Hz to generate a nodding power curve. The formula is as follows:

[0027]

[0028] Among them, P 点头 (t) is the energy corresponding to the nodding motion at time t, and |STFT nod (t,f)| 2 is the energy density at frequency f;

[0029] Detect valid peaks through the find_peaks algorithm. Each peak corresponds to a nodding motion, and the total number of times is counted.

[0030] As a further preferred technical solution of the above technical solution, the specific implementation of the periodic eyelid vibration analysis algorithm of the facial fatigue detection module is as follows:

[0031] Use the first millimeter-wave radar to obtain the I / Q quadrature signal in real time and calculate the signal amplitude:

[0032]

[0033] Among them, I(t) is the in-phase component, representing the real part of the radar echo; Q(t) is the quadrature component, representing the imaginary part of the radar echo. Apply a band-pass filter of 0.2 Hz - 2 Hz to the signal A(t) to retain the high-frequency components of the rapid micro-movement of the eyes;

[0034] Perform STFT on the filtered signal. The time window is 1 - 2 seconds and is smoothed using a Hann window. The formula is as follows:

[0035]

[0036] Among them, STFT bink (t,f) is the energy distribution of the blink-related signal in the time t and frequency f dimensions. t is the current time point; f is the frequency; A bink (t′) is the amplitude of the preprocessed blink signal; w(t′ - t) is the window function used to intercept the local signal; e -j2πft′ is the complex exponential carrier to achieve frequency modulation;

[0037] Integrate the STFT result in the frequency band of 0.2 Hz - 2 Hz to generate a blink power curve. The formula is as follows:

[0038]

[0039] Among them, P 眨眼 (t) is the energy corresponding to the blink motion at time t, and |STFT bink (t,f)|2 is the energy density at frequency f;

[0040] Blinking frequency = number of detected blinks / time length.

[0041] As a further preferred technical solution of the above technical solution, the physical sign fatigue detection module is specifically implemented as:

[0042] First, use the limb influence elimination algorithm to remove the influence generated when the driver makes limb movements and the vehicle vibrates and bumps. The specific implementation is as follows:

[0043] First, through the limb influence elimination algorithm, eliminate the interference of large displacements and fine movements on the vital sign signals. To form a digital beam, first calculate the channel impulse response of different distance-azimuth angles. The formula is:

[0044] h(r,θ,m) = s H (θ)h r,l (m) + ∈(m);

[0045] where s H (θ) is the conjugate transpose of the steering vector, h r,l (m) is the multi-antenna received signal, h(r,θ,m) is the signal received by the radar in the direction of distance r, angle θ, and time m; ∈(m) is the noise;

[0046] Calculate the background average signal and subtract it to remove clutter. The formula is:

[0047]

[0048] where is the processed signal; h(r,θ,m) is the original received signal; M is the number of signal frames participating in background estimation; is to estimate the static background signal by averaging the past M frames of signals and subtract it from the current signal to achieve background suppression;

[0049] Use 2D cross-correlation to align consecutive CIRs for large displacement compensation. The formula is:

[0050]

[0051] where is to search for the values of x and y for horizontal displacement x and vertical displacement y to find the x and y values that maximize the subsequent summation result, corresponding to the displacement amount between the previous and subsequent frames of signals; CIR t (i,j) represents the channel impulse response value of the radar at the distance unit i and azimuth unit j at the current time t;

[0052] Use a smooth spline to fit and remove the phase trend to achieve fine motion cancellation. The formula is:

[0053]

[0054] where y(t) is the phase sequence, A is the roughness penalty matrix, is the objective function value of the final estimate, and I is the identity matrix;

[0055] Finally, detect the periodic signal through the autocorrelation function, screen the range-azimuth angle unit where the vital signal is located, and perform vital signal positioning;

[0056] Second, for the data where the influence of limb movement and vehicle bump has been eliminated, the next step is to perform a micro-motion heart rate extraction algorithm to extract heart rate parameters, where:

[0057] First, enhance the heartbeat harmonics through first-order / second-order differentiation:

[0058]

[0059] where y(t) is the thoracic displacement signal; y′(t) is the first derivative of the thoracic displacement signal; t is the time point; Δt is the time step;

[0060] Select the differentiation result with high amplitude spectrum sparsity to suppress respiratory harmonic interference;

[0061] Use db5 wavelet for 7-layer decomposition, extract the sub-signal in the second harmonic frequency band of the heartbeat, and perform weighted reconstruction. The formula is:

[0062]

[0063] where A i is the sub-signal energy, estimate the heart rate through the power spectrum, S heart (t) is the reconstructed heartbeat signal; ∑A j is the sum of the sub-signal energies A j of all sub-signals participating in the fusion, and S i (t) is the i-th sub-signal after wavelet packet decomposition;

[0064] Perform Welch power spectrum estimation on the reconstructed signal S heart (t), and the peak frequency corresponds to the heart rate f h . The formula is:

[0065]

[0066] where P(f) is the power spectral density value used to locate the peak frequency corresponding to the heart rate; L is the number of signal segments; w(n) is the window function; S heart,i(n) is the heartbeat signal of the i-th segment; XU is the normalization factor for the window function;

[0067] Third, for the data with the influence of limb movement and vehicle bump eliminated, a periodic breathing detection algorithm should also be performed to extract breathing parameters, where:

[0068] The periodic chest displacement caused by breathing is modeled as:

[0069]

[0070] where, A r is the breathing amplitude, f r is the breathing frequency, p controls the waveform shape, x r (t) is the periodic chest displacement function caused by breathing;

[0071] Subsequently, a low-pass filter with a cut-off frequency of 0.5 Hz is used to extract the breathing signal, and the formula is:

[0072] S breath (t) = LPF(y(t));

[0073] where, S breath (t) is the breathing signal extracted after low-pass filtering; LPF is the low-pass filter; y(t) is the original signal input to the low-pass filter;

[0074] Perform Welch power spectrum estimation on the breathing signal S breath (t), and the peak frequency corresponds to the breathing frequency. The formula is:

[0075]

[0076] where, P(f) is the power spectral density value used to locate the peak frequency corresponding to the heart rate; L is the number of signal segments; w(n) is the window function; S braeth,i (n) is the breathing signal of the i-th segment; XU is the normalization factor for the window function;

[0077] Extract the breathing period through peak detection and calculate the breathing frequency.

[0078] As a further preferred technical solution of the above technical solution, the fatigue index calculation module is specifically implemented as:

[0079] The calculation formula of the fatigue index FI is: FI = w N ·N + w HR ·HR + w B ·B + w BR ·BR;

[0080] where:

[0081] N is the number of nods, unit: times / minute;

[0082] HR is the heart rate, unit: times / minute;

[0083] B is the number of blinks, unit: times / minute;

[0084] BR is the breathing rate, unit: times / minute;

[0085] w N 、w HR 、w B and w BR represent the corresponding weights;

[0086] The driver's fatigue index FI is obtained through the above formula.

[0087] As a further preferred technical solution of the above technical solution, the state discrimination and warning module is specifically implemented as:

[0088] When the fatigue index FI is in the first interval, it is determined as a non-fatigued state;

[0089] When the fatigue index FI is in the second interval, it is determined as a mild fatigue state;

[0090] When the fatigue index FI is in the third interval, it is determined as a moderate fatigue state;

[0091] When the fatigue index FI is in the fourth interval, it is determined as a severe fatigue state. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 is a schematic diagram of the fatigue driving monitoring system based on millimeter wave radar of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0093] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles defined in the following description can be applied to other embodiments, variations, improvements, equivalent embodiments, and other technical solutions without departing from the spirit and scope of the present invention.

[0094] In the preferred embodiment of the present invention, those skilled in the art should note that the millimeter wave radar and the like involved in the present invention can be regarded as prior art.

[0095] Preferred embodiment.

[0096] Such as Figure 1As shown in the figure, the present invention discloses a fatigue driving monitoring system based on a millimeter-wave radar, which includes a millimeter-wave radar, a data transmission module, a fatigue driving detection module, a fatigue index calculation module, and a status determination and warning module, where:

[0097] The millimeter-wave radar is installed at a preset position of the vehicle and transmits the collected radar echo signals of the driver to the data transmission module, and the data transmission module transmits the radar echo signals to the fatigue driving detection module;

[0098] The fatigue driving detection module performs detections respectively through a facial fatigue detection module and a physical fatigue detection module, where:

[0099] The facial fatigue detection module respectively performs periodic statistics on the nodding and blinking times of the driver through a head movement frequency monitoring algorithm and a periodic eyelid vibration analysis algorithm, so as to obtain the first data (including nodding and blinking);

[0100] The physical fatigue detection module respectively performs frequency detections on the heart rate and breathing of the driver through a micro-motion heart rate extraction algorithm and a periodic breathing detection algorithm, so as to obtain the second data (including heart rate and breathing);

[0101] The fatigue index calculation module substitutes the obtained first data and second data into the fatigue index calculation formula to obtain the driver's fatigue index FI;

[0102] The status determination and warning module compares the driver's fatigue index FI with the fatigue status index range. If it is determined to be in a fatigue state, a warning reminder is given according to the fatigue level. If the driver is not detected to be in fatigue, the detection and determination continue.

[0103] Specifically, the millimeter-wave radar includes a first millimeter-wave radar (i.e., millimeter-wave radar 1) and a second millimeter-wave radar (i.e., millimeter-wave radar 2), and both the first millimeter-wave radar and the second millimeter-wave radar are equipped with a data transmission module, where:

[0104] The first millimeter-wave radar is installed at the top of the vehicle windshield, and the angle is adjusted to align with the driver's chest cavity. The first data transmission module continuously collects echo data, including information on breathing, heartbeat, blinking, and nodding. Among them, the information on breathing and heartbeat is obtained through the chest vibration data in the echo data (i.e., the original data of breathing and heartbeat, and the breathing and heart rate are calculated according to this data later);

[0105] The second millimeter-wave radar is installed at the left front (60-degree position) of the driver's seat and monitors the upper body of the driver, monitoring the limb movement information of the driver including the arms (performing limb interference removal and separation processing on the physiological data collected by the radar).

[0106] More specifically, the head movement frequency monitoring algorithm of the facial fatigue detection module is specifically implemented as follows:

[0107] Use the first millimeter-wave radar to obtain the I / Q quadrature signal in real time and calculate the signal amplitude:

[0108]

[0109] Among them, I(t) is the in-phase component, representing the real part of the radar echo; Q(t) is the quadrature component, representing the imaginary part of the radar echo. Apply a band-pass filter with a frequency range of 0.1 Hz - 1 Hz to the signal A(t) to suppress background noise and other interferences and retain the low-frequency components corresponding to the small head displacements.

[0110] Perform STFT (Short-Time Fourier Transform) on the filtered signal. The time window is 1 - 2 seconds (50% overlap), and use the Hann window for smoothing processing. The formula is as follows:

[0111]

[0112] Among them, STFT nod (t, f) is the energy distribution of the nodding-related signal in the time t and frequency f dimensions; t is the current time point; f is the frequency; A nod (t′) is the amplitude of the nodding signal after preprocessing; w(t′ - t) is the window function used to intercept the local signal; e -j2πft′ is the complex exponential carrier to achieve frequency modulation;

[0113] Integrate the STFT result in the frequency band of 0.1 Hz - 1 Hz to generate the nodding power curve. The formula is as follows:

[0114]

[0115] Among them, P 点头 (t) is the energy corresponding to the nodding action at time t, and |STFT nod (t, f)| 2 is the energy density at frequency f;

[0116] (Set parameters: peak amplitude threshold: greater than the mean + 2 × standard deviation; minimum interval between adjacent peaks: 0.5 seconds, maximum interval: 5 seconds, used to filter false actions that are too fast or too slow);

[0117] Detect valid peaks through the find_peaks algorithm. Each peak corresponds to a nodding action, and count the total number of times.

[0118] Furthermore, the periodic eyelid vibration analysis algorithm of the facial fatigue detection module is specifically implemented as follows:

[0119] Use the first millimeter-wave radar to obtain I / Q quadrature signals in real time and calculate the signal amplitude:

[0120]

[0121] Among them, I(t) is the in-phase component, representing the real part of the radar echo; Q(t) is the quadrature component, representing the imaginary part of the radar echo. Apply a band-pass filter of 0.2 Hz - 2 Hz to the signal A(t) to retain the high-frequency components of the rapid micro-movements of the eyes;

[0122] Perform STFT on the filtered signal, with a time window of 1 - 2 seconds (50% overlap), and use a Hann window for smoothing. The formula is:

[0123]

[0124] Among them, STFT bink (t, f) is the energy distribution of the blink-related signal in the time t and frequency f dimensions, t is the current time point; f is the frequency; A bink (t′) is the amplitude of the preprocessed blink signal; w(t′ - t) is the window function, used to intercept the local signal; e -j2πft′ is the complex exponential carrier, which realizes frequency modulation;

[0125] Integrate the STFT result in the frequency band of 0.2 Hz - 2 Hz to generate a blink power curve. The formula is:

[0126]

[0127] Among them, P 眨眼 (t) is the energy corresponding to the blink action at time t, |STFT bink (t, f)| 2 is the energy density at frequency f;

[0128] (Perform parameter settings: peak amplitude threshold: greater than the mean + 2 × standard deviation mean + 2 × standard deviation; minimum interval between adjacent peaks: 0.2 seconds, used to adapt to the high-frequency characteristics of blinking);

[0129] Blinking frequency = number of detected blinks / time length.

[0130] Furthermore, for the specific implementation of the physical fatigue detection module:

[0131] First, use a limb influence elimination algorithm to remove the influence generated when the driver makes limb movements and the vehicle vibrates and jolts. The specific implementation is:

[0132] First, through the limb influence elimination algorithm, eliminate the interference of large displacements and fine movements on the vital sign signals. To form a digital beam, first calculate the channel impulse response (CIR) at different distances - azimuth angles, with the formula:

[0133] h(r,θ,m)=s H (θ)h r,l (m)+∈(m);

[0134] where s H (θ) is the conjugate transpose of the steering vector, h r,l (m) is the multi - antenna received signal, h(r,θ,m) is the signal received by the radar at distance r, angle θ direction, and time m; ε(m) is the noise;

[0135] Calculate the background average signal and subtract it to remove clutter, with the formula:

[0136]

[0137] where is the processed signal; h(r,θ,m) is the original received signal; M is the number of signal frames participating in background estimation; is to estimate the static background signal by averaging the past M - frame signals and subtract it from the current signal to achieve background suppression;

[0138] Use 2D cross - correlation to align consecutive CIRs for large displacement compensation, with the formula:

[0139]

[0140] where is to search for the values of x and y for horizontal displacement x and vertical displacement y that maximize the subsequent summation result, corresponding to the displacement amount between the previous and the current two frames of signals; CIR t (i,j) represents the channel impulse response value of the radar at distance cell i and azimuth cell j at the current time t;

[0141] Use smooth spline to fit and remove the phase trend to achieve fine movement elimination, with the formula:

[0142]

[0143] where y(t) is the phase sequence, A is the roughness penalty matrix, is the finally estimated objective function value, I is the identity matrix;

[0144] Finally, detect the periodic signal through the autocorrelation function (ACF), screen the distance - azimuth angle cells where the vital signal is located, and perform vital signal positioning;

[0145] Second, for the data that has eliminated the effects of limb movement and vehicle bumps, the next step is to use the micro-motion heart rate extraction algorithm to extract the heart rate parameters, where:

[0146] First enhance the heartbeat harmonics by first / second order differentiation:

[0147]

[0148] Where y(t) is the chest displacement signal; y′(t) is the first-order derivative of the chest displacement signal; t is the time point; Δt is the time step;

[0149] Select the differential result with high amplitude spectrum sparsity (second-order differential is preferred) to suppress respiratory harmonic interference;

[0150] Use db5 wavelet to perform 7-layer decomposition, extract the heartbeat second harmonic frequency band (2-4Hz) sub-signal, and reconstruct it by weight. The formula is:

[0151]

[0152] Among them, A i is the sub-signal energy, and the heart rate is estimated by the power spectrum, S heart (t) is the reconstructed heartbeat signal; ∑A j is the energy A of all sub-signals involved in the fusion j Sum, S i (t) is the i-th sub-signal after wavelet packet decomposition;

[0153] For the reconstructed signal S heart (t) Welch power spectrum estimation, the peak frequency corresponds to the heart rate f h , the formula is:

[0154]

[0155] Where P(f) is the power spectrum density value, which is used to locate the peak frequency corresponding to the heart rate; L is the number of signal segments; w(n) is the window function; S heart,i (n) is the heartbeat signal of the i-th segment; XU is the normalization factor of the window function; FFT stands for fast Fourier transform, which is used to transform the windowed heartbeat signal segment w(n)·S heart,i (n) Convert from time domain to frequency domain;

[0156] Third, for the data that has been cleared of the effects of limb movement and vehicle bumps, a periodic breathing detection algorithm is also used to extract breathing parameters, where:

[0157] The periodic chest displacement caused by breathing is modeled as:

[0158]

[0159] Among them, A r is the respiratory amplitude, f r is the respiratory frequency, p controls the waveform shape, and x r (t) is the periodic thoracic displacement function caused by respiration;

[0160] Subsequently, a low-pass filter with a cut-off frequency of 0.5 Hz is used to extract the respiratory signal, and the formula is:

[0161] S breath (t) = LPF(y(t));

[0162] Among them, S breath (t) is the respiratory signal extracted after low-pass filtering; LPF is the low-pass filter; y(t) is the original signal input to the low-pass filter;

[0163] Perform Welch power spectrum estimation on the respiratory signal S breath (t), and the peak frequency corresponds to the respiratory frequency. The formula is:

[0164]

[0165] Among them, P(f) is the power spectral density value used to locate the peak frequency corresponding to the heart rate; L is the number of signal segments; w(n) is the window function; S braeth,i (n) is the respiratory signal of the i-th segment; XU is the normalization factor of the window function;

[0166] Extract the respiratory cycle through peak detection and calculate the respiratory frequency.

[0167] Preferably, for the fatigue index calculation module, it is specifically implemented as:

[0168] The calculation formula of the fatigue index FI is: FI = w N ·N + w HR ·HR + w B ·B + w BR ·BR;

[0169] Among them:

[0170] N is the number of nods, unit: times / minute;

[0171] HR is the heart rate, unit: times / minute;

[0172] B is the number of blinks, unit: times / minute;

[0173] BR is the respiratory frequency, unit: times / minute;

[0174] w N 、w HR 、wB and w BR represents the corresponding weight (standard weight setting: w N = 0.35; w HR = 0.25; w B = 0.3; w BR = 0.1); The fatigue index FI of the driver is obtained through the above formula;

[0175] It is worth mentioning that the normal physical characteristics of adults:

[0176] Nodding frequency: 0 - 2 times per minute (very low in non-fatigued state)

[0177] Trend of fatigue state appearance: When the driver shows fatigue, the number of nodding times will increase significantly.

[0178] Heartbeat frequency: 60 - 100 times per minute (in the quiet state of an adult).

[0179] Trend of fatigue state appearance: Exceeding 100 times per minute can be judged as fatigue driving (fatigue causes an increase in heartbeat frequency).

[0180] Blinking frequency: 12 - 20 times per minute (normal blinking frequency).

[0181] Trend of fatigue state appearance: Exceeding 20 times per minute can be judged as fatigue driving (fatigue causes an increase in blinking frequency).

[0182] Respiration frequency: 12 - 20 times per minute (ordinary respiration frequency range).

[0183] Trend of fatigue state appearance: Exceeding 20 times per minute can be judged as fatigue driving (fatigue causes an increase in respiration frequency).

[0184] Conclusion:

[0185] Respectively take the highest normal value and the lowest normal value of the nodding frequency, heartbeat frequency, blinking frequency, and respiration frequency, and calculate the fatigue index range of an adult during normal driving: 19.8 - 33.7 (non-fatigue driving condition)

[0186] Because the ages and physical health conditions of different drivers are different, the middle value of the normal range is taken as the benchmark, that is, the most normal fatigue index value: 26.75.

[0187] Preferably, the state discrimination and warning module is specifically implemented as:

[0188] When the fatigue index FI is in the first interval (FI ≤ 26.75), it is determined to be in a non-fatigued state (in this state, the driver is in good condition, all physiological indicators are normal, and can continue driving completely; no alarm is required, the device is in normal monitoring state, and can intermittently record data for subsequent analysis and comparison. At the same time, it shows that the monitored person is in good condition, all physiological indicators and facial features are normal, and there are no signs of fatigue);

[0189] When the fatigue index FI is in the second interval (26.75 < FI ≤ 36.75), it is determined to be in a mild fatigue state (in this state, initial signs of fatigue appear, such as a slightly increased nodding frequency, a faster blinking speed, etc., but it is still within the safe driving range; a first-level alarm is issued, a short ding sound is emitted with a soft reminder tone, and a text prompt "You have entered a mild fatigue state, please pay attention to taking appropriate rest" appears on the device screen to remind the driver);

[0190] When the fatigue index FI is in the third interval (36.75 < FI ≤ 46.75), it is determined to be in a moderate fatigue state (in this state, the fatigue signal is more obvious, such as frequent nodding or irregular breathing, less / faster blinking, which may lead to a decrease in reaction ability; a second-level alarm is issued, a continuous "buzz" sound and a prominent pop-up text prompt "Moderate fatigue! Please stop the current activity as soon as possible and rest and relax" are emitted. At the same time, some devices on the vehicle connected to the system can automatically pause some non-essential functions such as playing music to give a reminder);

[0191] When the fatigue index FI is in the fourth interval (FI > 46.75), it is determined to be in a severe fatigue state (in this state, the fatigue index rises significantly, which may be manifested as strong tiredness (irregular breathing and heart rate, frequent head-down), and there are major safety risks; a third-level alarm is issued, a sharp alarm sound is emitted, and through continuous vibration of the device and a pop-up display of red emergency text reminder "Severe fatigue! Your body is extremely exhausted, and you must immediately stop all activities, seek a safe place to rest and rest". In addition, if it is in a dangerous scenario such as driving, the system should automatically link with relevant safety systems, such as the vehicle automatically decelerating and pulling over to the side of the road, and the safety equipment at the workplace starts a locking mechanism to prevent the monitored person from continuing to operate dangerous machinery, etc., to ensure the safety of the driver's life and avoid accidents caused by severe fatigue);

[0192] It is worth mentioning that technical features such as millimeter-wave radars involved in this invention patent application should be regarded as prior art. The specific structures, working principles, and possible control methods and spatial arrangement methods of these technical features can be selected conventionally in the art, and should not be regarded as the invention points of this invention patent, and this invention patent will not be further specifically elaborated.

[0193] For those skilled in the art, it is still possible to modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A fatigue driving monitoring system based on a millimeter-wave radar, characterized in that, It includes a millimeter-wave radar, a data transmission module, a fatigue driving detection module, a fatigue index calculation module, and a status discrimination and warning module, where: The millimeter-wave radar is installed at a preset position of the vehicle and transmits the collected radar echo signals of the driver to the data transmission module, and the data transmission module transmits the radar echo signals to the fatigue driving detection module; The fatigue driving detection module performs detections through a facial fatigue detection module and a physical sign fatigue detection module respectively, where: The facial fatigue detection module periodically counts the nodding and blinking times of the driver through a head movement frequency monitoring algorithm and a periodic eyelid vibration analysis algorithm respectively, so as to obtain the first data; The physical sign fatigue detection module detects the heart rate and breathing frequencies of the driver through a micro-motion heart rate extraction algorithm and a periodic breathing detection algorithm respectively, so as to obtain the second data; The fatigue index calculation module substitutes the obtained first data and second data into the fatigue index calculation formula to obtain the fatigue index FI of the driver; The status discrimination and warning module compares the fatigue index FI of the driver with the fatigue status index range. If it is determined to be in a fatigue state, a warning reminder is given according to the fatigue level. If the driver is not detected to be in fatigue, the detection and determination continue.

2. The fatigue driving monitoring system based on millimeter wave radar according to claim 1, characterized in that, The millimeter-wave radar includes a first millimeter-wave radar and a second millimeter-wave radar, and both the first millimeter-wave radar and the second millimeter-wave radar are equipped with a data transmission module, where: The first millimeter-wave radar is installed at the top of the vehicle windshield, adjusts the angle to align with the driver's chest cavity, and continuously collects echo data through the first data transmission module, including information on breathing, heartbeat, blinking, and nodding. Among them, the information on breathing and heartbeat is obtained through the chest vibration data in the echo data; The second millimeter-wave radar is installed in the front left of the driver's seat and monitors the upper body of the driver, monitoring the limb movement information of the driver including the arms.

3. The fatigue driving monitoring system based on millimeter wave radar according to claim 2, wherein The specific implementation of the head movement frequency monitoring algorithm of the facial fatigue detection module is: Use the first millimeter-wave radar to obtain the I / Q quadrature signal in real time and calculate the signal amplitude: Among them, I(t) is the in-phase component, representing the real part of the radar echo; Q(t) is the quadrature component, representing the imaginary part of the radar echo. Apply a band-pass filter of 0.1Hz - 1Hz to the signal A(t) to suppress background noise and other interferences and retain the low-frequency components corresponding to the small head displacements; Perform STFT on the filtered signal, the time window is 1 - 2 seconds, and use the Hann window for smoothing processing. The formula is: Among them, STFT nod (t,f) is the energy distribution of the head-nodding related signal in the time t and frequency f dimensions; t is the current time point; f is the frequency; A nod (t′) is the amplitude of the preprocessed head-nodding signal; w(t′ - t) is a window function used to intercept local signals; e -j2πft′ is a complex exponential carrier wave for realizing frequency modulation; Integrate the STFT result in the frequency band of 0.1Hz - 1Hz to generate a nodding power curve. The formula is: Among them, P 点头 (t) is the energy corresponding to the nodding motion at time t, |STFT nod (t,f)| 2 is the energy density at frequency f; Detect valid peaks through the find_peaks algorithm. Each peak corresponds to a nodding action, and the total number is counted.

4. The fatigue driving monitoring system based on millimeter-wave radar according to claim 3, characterized in that, The specific implementation of the periodic eyelid vibration analysis algorithm of the facial fatigue detection module is: Use the first millimeter-wave radar to obtain the I / Q quadrature signal in real time and calculate the signal amplitude: Among them, I(t) is the in-phase component, representing the real part of the radar echo; Q(t) is the quadrature component, representing the imaginary part of the radar echo. Apply a band-pass filter of 0.2 Hz - 2 Hz to the signal A(t) to retain the high-frequency components of the rapid micro-movements of the eyes. Perform STFT on the filtered signal with a time window of 1 - 2 seconds and smooth it using a Hann window. The formula is: Among them, STFT bink (t, f) is the energy distribution of the blink-related signal in the time t and frequency f dimensions, where t is the current time point; f is the frequency; A bink (t′) is the amplitude of the preprocessed blink signal; w(t′ - t) is a window function used to intercept the local signal; e -j2πft′ is a complex exponential carrier to achieve frequency modulation; Integrate the STFT result in the frequency band of 0.2 Hz - 2 Hz to generate a blinking power curve. The formula is: Among them, P 眨眼 (t) is the energy corresponding to the blinking action at time t, and |STFT bink (t,f)| 2 is the energy density at frequency f; Blinking frequency = number of detected blinks / time length.

5. The fatigue driving monitoring system based on millimeter wave radar according to claim 4, characterized in that, The specific implementation of the physical sign fatigue detection module is as follows: First, use a limb movement influence elimination algorithm to remove the influence generated when the driver makes limb movements and the vehicle vibrates and jolts. The specific implementation is as follows: First, through the limb movement influence elimination algorithm, eliminate the interference of large displacements and fine movements on the vital sign signal. To form a digital beam, first calculate the channel impulse response of different range - azimuth angles. The formula is: h(r, θ, m) = s H (θ)h r,l (m) + ∈(m); where s H (θ) is the conjugate transpose of the steering vector, h r,l (m) is the multi-antenna received signal, and h(r, θ, m) is the signal received by the radar at range r, angle θ, and time m; ∈(m) is the noise; Calculate the background average signal and subtract it to remove clutter. The formula is: Among them, is the processed signal; h(r, θ, m) is the original received signal; M is the number of signal frames participating in background estimation; estimates the static background signal by averaging the past M frames of signals and subtracts it from the current signal to achieve background suppression; Use 2D cross-correlation to align the continuous CIR for large displacement compensation. The formula is: Among them, search for the values of x and y for the horizontal displacement x and the vertical displacement y to find the x and y values that maximize the subsequent summation result, corresponding to the displacement amounts of the front and rear two-frame signals; CIR t (i, j) represents the channel impulse response value of the radar at the range cell i and the azimuth cell j at the current time t; Use a smoothing spline to fit and remove the phase trend to achieve fine movement elimination. The formula is: where y(t) is the phase sequence, A is the roughness penalty matrix, is the objective function value of the final estimation, and I is the identity matrix; Finally, detect the periodic signal through the autocorrelation function, screen the range - azimuth angle unit where the vital signal is located, and perform vital signal positioning. Second, for the data after the influence of limb movement and vehicle jolting has been eliminated, the next step is to use a micro-motion heart rate extraction algorithm to extract heart rate parameters. Among them: First, enhance the heartbeat harmonics through first-order / second-order differentiation: Among them, y(t) is the thoracic displacement signal; y′(t) is the first derivative of the thoracic displacement signal; t is the time point; Δt is the time step. Select the differentiation result with high amplitude spectrum sparsity to suppress the interference of respiratory harmonics. Use db5 wavelet for 7-layer decomposition, extract the sub-signal in the second harmonic frequency band of the heartbeat, and perform weighted reconstruction. The formula is: Among them, A i is the sub-signal energy, and the heart rate is estimated through the power spectrum. S heart (t) is the reconstructed heartbeat signal; ∑A j is the summation of the sub-signal energies A j of all sub-signals participating in the fusion, and S i (t) is the i-th sub-signal after wavelet packet decomposition; Perform Welch power spectrum estimation on the reconstructed signal S heart (t), and the peak frequency corresponds to the heart rate f h , and the formula is: where P(f) is the power spectral density value used to locate the peak frequency corresponding to the heart rate; L is the number of signal segments; w(n) is the window function; S heart,i (n) is the heartbeat signal of the i-th segment; XU is the normalization factor of the window function; Third, for the data after the influence of limb movement and vehicle jolting has been eliminated, at the same time, a periodic breathing detection algorithm is also used to extract breathing parameters. Among them: Model the periodic thoracic displacement caused by breathing as: Where, A r is the respiratory amplitude, f r is the respiratory frequency, p controls the waveform shape, and x r (t) is the periodic thoracic displacement function caused by respiration; Subsequently, use a low-pass filter with a cut-off frequency of 0.5 Hz to extract the breathing signal. The formula is: S breath (t) = LPF(y(t)); Among them, S breath (t) is the respiratory signal extracted after low-pass filtering; LPF is the low-pass filter; y(t) is the original signal input to the low-pass filter; Perform Welch power spectrum estimation on the respiratory signal S breath (t), and the peak frequency corresponds to the respiratory frequency. The formula is as follows: where P(f) is the power spectral density value used to locate the peak frequency corresponding to the heart rate; L is the number of signal segments; w(n) is the window function; S braeth,i (n) is the respiratory signal of the i-th segment; XU is the normalization factor of the window function; Extract the breathing cycle through peak detection and calculate the breathing frequency.

6. The fatigue driving monitoring system based on millimeter-wave radar according to claim 5, characterized in that, The specific implementation of the fatigue index calculation module is as follows: The calculation formula for the fatigue index FI is: FI = w N ·N + w HR ·HR + w B ·B + w BR ·BR; Among them: N is the number of nods, unit: times / minute; HR is the heart rate, unit: times / minute; B is the number of blinks, unit: times / minute; BR is the breathing frequency, unit: times / minute; w N 、w HR 、w B and w BR represents the corresponding weight; Obtain the driver's fatigue index FI through the above formula.

7. The fatigue driving monitoring system based on millimeter wave radar according to claim 6, characterized in that, The specific implementation of the state discrimination and warning module is as follows: When the fatigue index FI is in the first interval, it is determined as a non-fatigued state. When the fatigue index FI is in the second interval, it is determined as a mildly fatigued state. When the fatigue index FI is in the third interval, it is determined as a moderately fatigued state. When the fatigue index FI is in the fourth interval, it is determined as a severely fatigued state.

Images