Dangerous driving monitoring method and system based on optical camera and millimeter wave radar

Abstract

The invention discloses a dangerous driving monitoring method and system based on an optical camera and a millimeter-wave radar, and the method comprises the steps: adjusting the optical camera and the millimeter-wave radar to enable the image centers of the optical camera and the millimeter-wave radar to be aligned, and then maintaining the relative pose to be fixed; based on the position of the cardiopulmonary part on the optical image, adjusting the pose of the camera and the radar carrier to enable the antenna center of the millimeter wave radar to be aligned with the cardiopulmonary part, and adjusting the focal length of the optical camera to enable the eyes of the driver to be within the view field range; continuous optical images and echo signals are respectively obtained by using an optical camera and a radar; judging whether the eyes are closed or not by calculating the aspect ratio of the eyes in the optical image, and judging whether the driver is in fatigue driving or not according to the eye closing duration; the breathing and heartbeat frequencies of the driver are extracted according to the radar echo signals, and then whether the driver currently drives in danger or not is judged again. The problem that the detection precision of an optical camera is limited due to the illumination condition is effectively solved, and dangerous driving behaviors can be accurately recognized.

Description

Technical Field

[0001] The present invention belongs to the technical field of radar monitoring, and in particular relates to a dangerous driving monitoring method and system based on an optical camera and a millimeter-wave radar. Background Art

[0002] At present, the dangerous driving behavior identification method based on the driver's physiological parameters usually uses contact devices such as electrocardiogram and electroencephalogram. Although such devices can accurately obtain the driver's physiological parameters and can deeply analyze the driver's physiological state, the devices have complex connection lines. During driving, these intricate connection lines may not only interfere with the driver's vision and operation, but also become a constraint in an emergency, affecting the driver's ability to respond quickly. In addition, wearing these devices for a long time may also cause discomfort to the driver, thereby affecting his driving state and indirectly increasing driving risks.

[0003] In addition, the core process of the dangerous driving behavior recognition technology based on optical cameras first focuses on collecting the driver's facial image. Subsequently, using advanced deep learning algorithms, the system can accurately analyze key physiological indicators such as the degree of eye closure and blinking frequency, so as to effectively identify and determine whether the driver has potential dangerous driving behavior. However, this method faces a significant challenge in practical applications: the sensitivity of optical cameras to lighting conditions. Especially when driving at night or in environments with poor lighting conditions, the imaging quality of optical cameras will drop significantly, directly leading to a decrease in detection accuracy. This decrease in accuracy will further affect the system's accurate judgment of dangerous driving behavior, and may not be able to detect dangerous conditions such as driver fatigue and distraction in a timely manner, thereby increasing the risk of road traffic accidents. Summary of the invention

[0004] The present invention provides a dangerous driving monitoring method and system based on an optical camera and a millimeter-wave radar, which effectively solves the problem of limited detection accuracy of optical cameras due to lighting conditions, and accurately identifies and prevents dangerous driving behaviors.

[0005] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0006] A dangerous driving monitoring method based on an optical camera and a millimeter-wave radar, comprising:

[0007] Step 1: adjust the optical camera and the millimeter-wave radar until the center of the optical image is aligned with the center of the radar image, and then fix the relative posture of the optical camera and the millimeter-wave radar;

[0008] Step 2: Based on the position of the heart and lungs on the optical image, the position of the camera and the radar carrier is adjusted so that the center of the millimeter-wave radar antenna is aligned with the heart and lungs, and the focal length of the optical camera is adjusted so that the driver's eyes are within the field of view;

[0009] Step 3, using an optical camera to photograph the driver to obtain a continuous optical image, and using a millimeter wave radar to collect an echo signal;

[0010] Step 4, extracting the eye part in the optical image, determining whether the eye is closed by calculating the aspect ratio of the eye, and counting the blinking frequency and the duration of each eye closure;

[0011] Step 5, extracting the vibration frequency of the driver's heart and lungs based on the echo signal collected by the millimeter wave radar, and extracting the breathing frequency and heart rate through filtering;

[0012] Step 6, comprehensively consider the blinking frequency, the duration of each eye closure, the breathing rate and the heart rate to determine whether the driver is currently driving dangerously.

[0013] Furthermore, the optical camera and the millimeter wave radar are adjusted so that the center of the optical image is aligned with the center of the radar image, which is expressed as:

[0014]

[0015] In the formula, is the center coordinate of the radar image, is the center coordinate of the optical image, is the spatial calibration matrix, To adjust the rotation angle of the optical camera, To adjust the displacement of the optical camera along the negative x-axis and negative y-axis of the radar coordinate system.

[0016] Furthermore, in step 2, based on the position of the heart and lungs on the optical image, the center of the millimeter wave radar antenna is adjusted to align with the heart and lungs. for:

[0017]

[0018] In the formula, is the field of view of the optical camera, is the number of pixel columns of the optical camera, is the column coordinate of the corresponding pixel of the heart and lung parts in the optical image, is the column coordinate of the center pixel of the optical image.

[0019] Furthermore, when the aspect ratio of the eyes is less than a preset aspect ratio threshold, the eyes are judged to be closed; if the number of optical images judged to be closed within a period of time exceeds a preset number threshold, the driver is judged to be currently fatigued.

[0020] Furthermore, step 5 extracts the vibration frequency of the driver's heart and lungs based on the echo signal collected by the millimeter wave radar, specifically:

[0021] set up The distance from the driver to the radar, including the motion components of breathing and heartbeat, is constructed as:

[0022]

[0023] In the formula, The distance from the radar to the driver when breathing and heartbeat are not taken into account. are the amplitude and frequency of breathing, are the amplitude and frequency of the heartbeat respectively;

[0024] According to the relationship between phase and distance, the phase model is constructed as follows:

[0025]

[0026] Add a DC signal to the echo signal to get the signal model for:

[0027]

[0028] Where A is the amplitude of the echo signal, is the amplitude of the DC signal, is the carrier frequency of the echo signal, The phase of the echo signal; adding a DC signal to the echo signal can improve the signal-to-noise ratio, enhance the anti-interference ability to noise, improve the problem of low-amplitude signals being submerged by noise in complex environments, and effectively extract small amplitudes. In addition, adding a DC signal in the actual system is simple.

[0029] Demodulate the echo signal by mixing Extract the baseband signal and remove the DC component to obtain the dynamic component in the echo signal :

[0030]

[0031] The phase change is extracted as:

[0032]

[0033] The extracted phase signal Perform fast Fourier transform to get the spectrum :

[0034]

[0035] There will be two low-frequency and high-frequency peaks on the spectrum, corresponding to the respiratory frequency and heart rate in the phase model respectively. The respiratory signal and heart rate signal are then extracted using a bandpass filter.

[0036] Furthermore, the respiratory frequency and the heart rate are extracted by filtering, the band-pass filtering range of the respiratory frequency is 0.1-0.5 Hz, and the band-pass filtering range of the heart rate is 0.8-2 Hz.

[0037] Furthermore, the integrated blinking frequency, the duration of each eye closure, the breathing frequency and the heart rate are used to determine whether the driver is currently driving dangerously, specifically:

[0038] When the light reaches the preset conditions: if the blinking frequency reaches the preset value and the duration of each eye closure exceeds the preset value, the driver is judged to be driving fatigued; if the breathing frequency is lower than 0.2Hz or the heart rate is lower than 1Hz, the driver is judged to be driving fatigued; if the breathing frequency is higher than 0.35Hz or the heart rate is higher than 1.7Hz, the driver is judged to be driving dangerously;

[0039] When the light does not meet the preset conditions: if the breathing rate is lower than 0.2Hz or the heart rate is lower than 1Hz, the driver is judged to be driving fatigued; if the breathing rate is higher than 0.35Hz or the heart rate is higher than 1.7Hz, the driver is judged to be driving dangerously.

[0040] A dangerous driving monitoring system based on an optical camera and a millimeter-wave radar comprises an optical camera, a millimeter-wave radar and a data processing and control center; the data processing and control center is used to implement the dangerous driving monitoring method based on an optical camera and a millimeter-wave radar described above.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention can avoid the cumbersome connection wires of traditional contact monitoring equipment to collect breathing and heart rate, thereby eliminating potential interference with the driver's driving operation and significantly improving driving safety. At the same time, the present invention effectively solves the problem of limited detection accuracy of optical cameras under poor lighting conditions (especially when driving at night) by introducing millimeter-wave radar technology. In addition, the present invention uses a camera to guide the millimeter-wave radar antenna to accurately align with the driver's heart and lungs, thereby achieving high-quality collection of vital signs data without affecting the driver's driving experience. Therefore, the present invention uses the characteristics of millimeter-wave radar with strong penetration and unaffected by light, and can stably capture the driver's breathing and heart rate, forming complementary advantages with optical cameras, and jointly constructing a comprehensive, high-precision dangerous driving behavior monitoring system, which not only improves the accuracy and reliability of recognition, but also brings users a safer and more comfortable driving experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of the dangerous driving monitoring method based on optical camera and millimeter wave radar described in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The following is a detailed description of an embodiment of the present invention. This embodiment is based on the technical solution of the present invention, and provides a detailed implementation method and a specific operation process to further explain the technical solution of the present invention.

[0045] Example 1

[0046] The embodiment of the present invention provides a dangerous driving monitoring method based on an optical camera and a millimeter wave radar. Figure 1 As shown, including:

[0047] Step 1: adjust the optical camera and the millimeter-wave radar until the center of the optical image is aligned with the center of the radar image, and then fix the relative posture of the optical camera and the millimeter-wave radar.

[0048] When monitoring key vital signs such as breathing and heartbeat, it is crucial to ensure the accuracy and reliability of data collection, which requires precise data capture of specific reaction areas of breathing and heartbeat. However, although millimeter-wave radar technology has shown great potential in the field of monitoring due to its non-contact and penetrating properties, the results of its imaging processing often have low resolution and lack detailed texture information and details such as the fine contours and colors of the target, which makes it particularly difficult to accurately locate the ideal detection area such as the heart and lungs directly from the radar image. Once the collection area deviates, it will not only directly lead to a significant decrease in data quality, but may also introduce additional interference factors, which will seriously weaken the accuracy of the monitoring results.

[0049] In order to overcome this technical bottleneck, the present invention proposes a strategy of combining optical cameras to guide the positioning of millimeter-wave radar antennas. With its excellent image capture capabilities and rich texture information, optical cameras can easily and accurately identify the heart and lung areas of the body that need to be monitored. Through this visual guidance mechanism, it can ensure that the center of the millimeter-wave radar antenna can be accurately aligned with the key areas of the heart and lungs, thereby achieving high-quality collection of vital signs such as breathing rate and heart rate without affecting the driver's driving experience.

[0050] In this embodiment, the optical camera and the millimeter-wave radar are first spatially calibrated, that is, the optical camera and the millimeter-wave radar are adjusted so that the center of the optical image is aligned with the center of the radar image, which is expressed as:

[0051]

[0052] In the formula, is the center coordinate of the radar image, is the center coordinate of the optical image, is the spatial calibration matrix, To adjust the rotation angle of the optical camera, To adjust the displacement of the optical camera along the x-axis and y-axis respectively.

[0053] Step 2: Based on the position of the heart and lungs on the optical image, adjust the position of the camera and the radar carrier so that the center of the millimeter-wave radar antenna is aligned with the heart and lungs, and adjust the focal length of the optical camera so that the driver's eyes are within the field of view.

[0054] In this embodiment, YoloV8 is first used to process the optical image captured by the camera to determine whether there is a living organism. If there is a living organism, the pixel coordinates of the center of the radar monitoring area are returned, and the posture of the camera and the radar carrier are adjusted by the following method so that the center of the radar antenna is aligned with the cardiopulmonary area to collect echo signals.

[0055] Among them, based on the position of the heart and lungs on the optical image, the posture of the camera and the radar carrier is adjusted so that the center of the millimeter-wave radar antenna is aligned with the heart and lungs. The specific adjustment angle for:

[0056]

[0057] In the formula, is the field of view of the optical camera, is the number of pixel columns of the optical camera, is the column coordinate of the corresponding pixel of the heart and lung parts in the optical image, is the column coordinate of the center pixel of the optical image.

[0058] Step 3: Use an optical camera to photograph the driver to obtain a continuous optical image, and use a millimeter-wave radar to collect the echo signal.

[0059] Step 4, extract the eye part in the optical image, determine whether the eyes are closed by calculating the aspect ratio of the eyes, and count the blinking frequency and the duration of each eye closure.

[0060] Facemesh is used to extract the key points of the eyes of the optical image, and then the eye aspect ratio is used to determine whether the eyes are closed. When the eye aspect ratio is close to or equal to 0, it proves that the life form has begun to close its eyes, so it is determined to be closed. In order to avoid false alarms caused by blinking, this embodiment determines whether the eyes are closed by the eye aspect ratio of the time series, and counts the blinking frequency and the duration of each eye closure within a period of time.

[0061] Step 5: extract the vibration frequency of the driver's heart and lungs based on the echo signal collected by the millimeter wave radar, and extract the breathing frequency and heart rate through filtering.

[0062] Although millimeter-wave radar does not have rich texture information, it has the ability to measure the amplitude and frequency of target vibrations, so it can monitor heartbeat and breathing by detecting the amplitude and frequency changes in the heart and lungs. The current resolution of millimeter-wave radar is at the centimeter level and cannot detect small amplitude vibrations in the heart and lungs. Therefore, this embodiment specifically uses a phase-based method for detection.

[0063] set up The distance from the target to the radar, including the motion components of breathing and heartbeat, is constructed as:

[0064]

[0065] In the formula, are the amplitude and frequency of breathing, are the amplitude and frequency of the heartbeat respectively;

[0066] According to the relationship between phase and distance, the phase model is constructed as follows:

[0067]

[0068] Add a DC signal to the echo signal to get the signal model for:

[0069]

[0070] Where A is the amplitude of the echo signal, is the amplitude of the DC signal, is the carrier frequency of the echo signal, is the phase of the echo signal;

[0071] Demodulate the echo signal by mixing Extract the baseband signal and remove the DC component to obtain the dynamic component in the echo signal :

[0072]

[0073] The phase change is extracted as:

[0074]

[0075] The extracted phase signal Perform fast Fourier transform to get the spectrum :

[0076]

[0077] There will be two low-frequency and high-frequency peaks on the spectrum, corresponding to the respiratory frequency and heart rate in the phase model respectively. The respiratory signal and heart rate signal are then extracted using a bandpass filter.

[0078] Finally, according to the normal adult's breathing and heart rate, the corresponding frequency filter is designed to complete the extraction of breathing frequency and heart rate. The bandpass filter range of breathing frequency is 0.1-0.5Hz, and the bandpass filter range of heart rate is 0.8-2Hz.

[0079] Step 6, comprehensively consider the blinking frequency, the duration of each eye closure, the breathing rate and the heart rate to determine whether the driver is currently driving dangerously.

[0080] Optical cameras, with their excellent image capture capabilities and rich texture information, have shown significant advantages in well-lit environments, especially when detecting eye status to identify whether the driver is drowsy or asleep. However, when the ambient light becomes insufficient, such as at night or on a cloudy day, the performance of optical cameras will be significantly affected, resulting in a significant drop in recognition accuracy, thus limiting their ability to work all day as a single monitoring method.

[0081] In contrast, millimeter-wave radar has shown unique advantages. It is not limited by lighting conditions and can work stably in various lighting environments, providing all-weather monitoring capabilities. Although millimeter-wave radar is not as rich in image texture information as optical cameras, its strong penetrating properties enable it to maintain high monitoring accuracy in complex environments.

[0082] In view of the advantages and limitations of optical cameras and millimeter wave radars, the present invention cleverly adopts a method of joint monitoring of the two. In this embodiment, the blinking frequency, the duration of each eye closure, the breathing rate and the heart rate are combined to determine whether the driver is currently driving dangerously, specifically:

[0083] When the light reaches the preset conditions (for example, during the day or when the light is relatively good), the monitoring scheme is mainly based on optical cameras and assisted by millimeter-wave radars: if the blinking frequency reaches the preset value and the duration of each eye closure exceeds the preset value (set to 1-2 seconds in this embodiment), the driver is judged to be driving fatigued; if the breathing frequency is lower than 0.2Hz or the heart rate is lower than 1Hz, the driver is judged to be driving fatigued; if the breathing frequency is higher than 0.35Hz or the heart rate is higher than 1.7Hz, it means that the driver is in a state of tension, anxiety or anger, and the driver is judged to be driving dangerously;

[0084] When the light does not meet the preset conditions (for example, at night or when the light is poor): if the breathing rate is lower than 0.2Hz or the heart rate is lower than 1Hz, the driver is judged to be driving fatigued; if the breathing rate is higher than 0.35Hz or the heart rate is higher than 1.7Hz, it means that the driver is in a state of tension, anxiety or anger, and the driver is judged to be driving dangerously.

[0085] By using an optical camera to provide high-precision eye status detection in good light, and with the assistance of millimeter-wave radar to achieve all-weather vital sign monitoring, the present invention not only overcomes the limitations of a single technology, but also builds a more comprehensive and reliable driving behavior monitoring system through complementary advantages. Such a design not only improves the accuracy and real-time nature of monitoring, but also provides a more solid guarantee for driving safety.

[0086] Example 2

[0087] This embodiment provides a dangerous driving monitoring system based on an optical camera and a millimeter-wave radar, including an optical camera, a millimeter-wave radar, and a data processing and control center; the data processing and control center is used to implement the dangerous driving monitoring method based on an optical camera and a millimeter-wave radar described in Example 1.

[0088] The above embodiments are preferred embodiments of the present application. Ordinary technicians in this field can also make various changes or improvements on this basis. Without departing from the overall concept of the present application, these changes or improvements should fall within the scope of protection required by the present application.

Claims

1. A dangerous driving monitoring method based on optical camera and millimeter wave radar, characterized in that: include: Step 1: adjust the optical camera and the millimeter-wave radar until the center of the optical image is aligned with the center of the radar image, and then fix the relative posture of the optical camera and the millimeter-wave radar; Step 2: Based on the position of the heart and lungs on the optical image, the position of the camera and the radar carrier is adjusted so that the center of the millimeter-wave radar antenna is aligned with the heart and lungs, and the focal length of the optical camera is adjusted so that the driver's eyes are within the field of view; Step 3, using an optical camera to photograph the driver to obtain a continuous optical image, and using a millimeter wave radar to collect an echo signal; Step 4, extracting the eye part in the optical image, determining whether the eyes are closed by calculating the aspect ratio of the eyes, and counting the blinking frequency and the duration of each eye closure; Step 5, extracting the vibration frequency of the driver's heart and lungs based on the echo signal collected by the millimeter wave radar, and extracting the breathing frequency and heart rate through filtering; Step 6, comprehensively consider the blinking frequency, the duration of each eye closure, the breathing rate and the heart rate to determine whether the driver is currently driving dangerously.

2. The dangerous driving monitoring method based on optical camera and millimeter wave radar according to claim 1 is characterized in that: The optical camera and the millimeter wave radar are adjusted so that the center of the optical image is aligned with the center of the radar image, which is expressed as: ； In the formula, is the center coordinate of the radar image, is the center coordinate of the optical image, is the spatial calibration matrix, To adjust the rotation angle of the optical camera, To adjust the displacement of the optical camera along the negative x-axis and negative y-axis of the radar coordinate system.

3. The dangerous driving monitoring method based on optical camera and millimeter wave radar according to claim 1 is characterized in that: Step 2: Based on the position of the heart and lungs on the optical image, adjust the center of the millimeter wave radar antenna to align with the heart and lungs. for: ； In the formula, is the field of view of the optical camera, is the number of pixel columns of the optical camera, is the column coordinate of the corresponding pixel of the heart and lung parts in the optical image, is the column coordinate of the center pixel of the optical image.

4. The dangerous driving monitoring method based on optical camera and millimeter wave radar according to claim 1 is characterized in that: When the aspect ratio of the eyes is less than a preset aspect ratio threshold, it is determined to be eye closed.

5. The dangerous driving monitoring method based on optical camera and millimeter wave radar according to claim 1, characterized in that: Step 5 extracts the vibration frequency of the driver's heart and lungs based on the echo signal collected by the millimeter wave radar, specifically: set up The distance from the driver to the radar, including the motion components of breathing and heartbeat, is constructed as: ； In the formula, The distance from the radar to the driver when breathing and heartbeat are not taken into account. are the amplitude and frequency of breathing, are the amplitude and frequency of the heartbeat respectively; According to the relationship between phase and distance, the phase model is constructed as follows: ； Add a DC signal to the echo signal to get the signal model for: ； Where A is the amplitude of the echo signal, is the amplitude of the DC signal, is the carrier frequency of the echo signal, is the phase of the echo signal; Demodulate the echo signal by mixing Extract the baseband signal and remove the DC component to obtain the dynamic component in the echo signal : ； The phase change is extracted as: ； The extracted phase signal Perform fast Fourier transform to get the spectrum : ； There will be two low-frequency and high-frequency peaks on the spectrum, corresponding to the respiratory frequency and heart rate in the phase model respectively. The respiratory signal and heart rate signal are then extracted using a bandpass filter.

6. The dangerous driving monitoring method based on optical camera and millimeter wave radar according to claim 1, characterized in that: The respiratory frequency and heart rate are extracted by filtering, the bandpass filter range of the respiratory frequency is 0.1-0.5Hz, and the bandpass filter range of the heart rate is 0.8-2Hz.

7. The dangerous driving monitoring method based on optical camera and millimeter wave radar according to claim 1 is characterized in that: The comprehensive blinking frequency, the duration of each eye closure, the breathing frequency and the heart rate are used to determine whether the driver is currently driving dangerously, specifically: When the light reaches the preset conditions: if the blinking frequency reaches the preset value and the duration of each eye closure exceeds the preset value, the driver is judged to be driving fatigued; if the breathing frequency is lower than 0.2Hz or the heart rate is lower than 1Hz, the driver is judged to be driving fatigued; If the breathing rate is higher than 0.35Hz or the heart rate is higher than 1.7Hz, the driver is judged to be driving dangerously; When the light does not meet the preset conditions: if the breathing rate is lower than 0.2Hz or the heart rate is lower than 1Hz, the driver is judged to be driving fatigued; if the breathing rate is higher than 0.35Hz or the heart rate is higher than 1.7Hz, the driver is judged to be driving dangerously.

8. A dangerous driving monitoring system based on an optical camera and a millimeter-wave radar, characterized in that: It comprises an optical camera, a millimeter-wave radar and a data processing and control center; the data processing and control center is used to implement the dangerous driving monitoring method based on an optical camera and a millimeter-wave radar as described in any one of claims 1-7.

Images