Fatigue nap identification and intervention method and system based on radar and wearable device

Abstract

The embodiment of the invention provides a fatigue nap identification and intervention method and system based on radar and wearable equipment, and belongs to the technical field of safety monitoring. According to the invention, attitude data of a target monitoring person and vehicle operation data can be obtained; acquiring a physiological signal of a target monitoring person; performing fatigue nap identification according to the posture data, the vehicle operation data and the physiological signal to obtain fatigue state judgment data; obtaining fatigue state grading information according to the fatigue state judgment data; and completing intervention measures according to the fatigue state grading information. The fatigue and nap behaviors of the driver can be accurately monitored and effectively intervened, and the driving safety is remarkably improved. The system has high accuracy, comfort and multiple intervention means, and is suitable for various drivers.

Description

Technical Field

[0001] The present application relates to the field of safety monitoring technology, and in particular to a method and system for identifying and intervening in fatigue and dozing based on radar and wearable devices. Background Art

[0002] When drivers are fatigued, their attention is easily distracted, and they may easily ignore traffic signals, road signs, or the behavior of other vehicles, increasing the risk of accidents. Existing fatigue monitoring technologies often rely on single technical means, such as visually judging fatigue status. These methods are susceptible to environmental interference and individual differences. These technologies are limited in their practical application and user acceptance due to defects such as high misjudgment rates, single intervention methods, poor adaptability, insufficient real-time performance, poor user experience, and poor system scalability. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to provide a method and system for identifying and intervening in fatigue and dozing based on radar and wearable devices.

[0004] The technical solution adopted by the present invention is:

[0005] In one aspect, an embodiment of the present invention provides a method for identifying and intervening in fatigue and dozing based on radar and wearable devices. The method comprises the following steps:

[0006] Obtaining the target monitoring personnel's posture data and vehicle operation data;

[0007] Acquire physiological signals of target monitoring personnel;

[0008] performing fatigue dozing identification based on the posture data, the vehicle operation data, and the physiological signal to obtain fatigue state judgment data;

[0009] Obtaining fatigue state classification information according to the fatigue state judgment data;

[0010] According to the fatigue status classification information, intervention measures are completed.

[0011] Furthermore, the acquisition of the target monitoring person's posture data and vehicle operation data includes the following steps:

[0012] Obtain data collected by vehicle-mounted radar;

[0013] Preprocessing the collected data of the vehicle-mounted radar to extract the posture data of the target monitoring person and the vehicle operation data;

[0014] The posture data includes facial image data, eye state data, head posture data, and chest and abdomen movement data;

[0015] The vehicle operation data includes driving speed data, braking data, and steering wheel control data.

[0016] Furthermore, the step of obtaining the physiological signal of the target monitored person includes the following steps:

[0017] Acquiring collected data from a wearable device; the wearable device includes a wearable flexible material device;

[0018] Performing data preprocessing on the collected data of the wearable device to extract the physiological signals of the target monitored person;

[0019] The physiological signals include brain wave data, heart rate data and skin electrical response data.

[0020] Furthermore, the fatigue dozing identification is performed based on the posture data, the vehicle operation data, and the physiological signal to obtain fatigue state judgment data, including the following steps:

[0021] Performing time stamp alignment on the posture data, the vehicle operation data, and the physiological signal using a dynamic time warping method to obtain a time stamp alignment matrix;

[0022] Combined with the current vehicle speed, radar visual flow is obtained;

[0023] Acquire physiological signal streams;

[0024] According to the timestamp alignment matrix, a multi-branch feature extraction network and a cross-modal attention mechanism are used to fuse the radar visual stream, the physiological signal stream, and the vehicle operation data to generate comprehensive feature data;

[0025] The comprehensive feature data is analyzed using a trained deep learning model to obtain fatigue state judgment data.

[0026] Furthermore, the dynamic time warping method is used to perform timestamp alignment on the posture data, the vehicle operation data, and the physiological signal to obtain a timestamp alignment matrix, and the formula used includes:

[0027]

[0028] Among them, T sync Align the matrix for timestamps; is the i-th radar monitoring data; the radar monitoring data includes the posture data and the vehicle operation data; is the i-th physiological signal; N is the total number of features monitored by radar and wearable devices.

[0029] Furthermore, the formula used to obtain the radar visual flow in combination with the current vehicle speed includes:

[0030]

[0031] ω velocity =1 / (1+e -0.1v );

[0032] Among them, PERCLOS is the radar visual flow; v is the current driving speed; t eyelid≥80％ T is the time when the eyelid coverage area is greater than or equal to 80%; window is the monitoring time window; ω velocity is the speed weight factor.

[0033] Furthermore, obtaining fatigue state classification information according to the fatigue state judgment data includes the following steps:

[0034] Setting fatigue level standards; the fatigue level standards include mild fatigue, moderate fatigue, and severe fatigue;

[0035] According to the fatigue state judgment data and the fatigue degree standard, the fatigue state classification information of the target monitored person is obtained.

[0036] Furthermore, the method for identifying and intervening in fatigue and dozing based on radar and wearable devices further includes the following steps:

[0037] Adjust the algorithm's accuracy and response mechanism based on feedback from target monitors and their historical fatigue status.

[0038] On the other hand, an embodiment of the present invention further provides a system for identifying and intervening in fatigue and dozing based on radar and wearable devices, which is used to implement the aforementioned method for identifying and intervening in fatigue and dozing based on radar and wearable devices. The system for identifying and intervening in fatigue and dozing based on radar and wearable devices includes:

[0039] The vehicle-mounted radar module is used to collect information about the target person's facial expressions, eye opening and closing status, head posture, and chest and abdominal movements, as well as vehicle speed, braking, and steering control.

[0040] Wearable devices used to collect physiological signals of target monitoring personnel;

[0041] A data processing module, configured to analyze data collected by the vehicle-mounted radar module and the wearable device;

[0042] The intervention module is used to intervene and wake up the target monitored person when fatigue or dozing is detected.

[0043] On the other hand, an embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions, which are used to enable a computer to execute the fatigue doze identification and intervention method based on radar and wearable devices as described above.

[0044] The embodiments of the present application include at least the following beneficial effects: The present application provides a method and system for identifying and intervening in fatigue and dozing based on radar and wearable devices. The present invention can obtain the posture data and vehicle operation data of the target monitoring personnel; obtain the physiological signals of the target monitoring personnel; identify fatigue and dozing based on the posture data, vehicle operation data and physiological signals, and obtain fatigue state judgment data; obtain fatigue state classification information based on the fatigue state judgment data; and complete intervention measures based on the fatigue state classification information. The present invention can accurately monitor and effectively intervene in driver fatigue and dozing behavior, significantly improving driving safety. The system has high accuracy, comfort and multiple intervention methods, and is suitable for all types of drivers. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flowchart of a method for identifying and intervening in fatigue and dozing based on radar and wearable devices, provided by an embodiment of the present invention;

[0046] Figure 2 Schematic diagram of a multi-branch feature extraction network provided by an embodiment of the present invention;

[0047] Figure 3 is a schematic diagram of micro-expression cycle analysis provided by an embodiment of the present invention;

[0048] Figure 4 Schematic diagram of the system structure provided by an embodiment of the present invention;

[0049] Figure 5 It is a flow chart of the working principle provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are merely examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0051] It will be understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0052] The terms "at least one", "plurality", "each", "any", etc. used in this application include "at least one", "two" or more, "plurality" or "each", "any" or "any one", "each" or "any one" as used herein.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0054] Before explaining the embodiments of the present application in detail, some of the nouns and terms involved in the embodiments of the present application are first explained. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0055] 1) EEG, which stands for electroencephalogram, is a method of recording electrical activity from the scalp using electrodes to analyze the brain's electrophysiological activity.

[0056] 2) HR, which is the heart rate, the number of heart beats per minute;

[0057] 3) GSR, which stands for Galvanic Skin Response, reflects changes in emotions or physiological states by measuring changes in skin conductivity;

[0058] 4) PERCLOS algorithm, which is the percentage of eyelid closure time algorithm (Percentage of Eyelid Closure Over the Pupil), calculates the proportion of eyelid closure time to total time;

[0059] 5) HRV, which stands for Heart Rate Variability, is the fluctuation of heart rate over time and is used to analyze the activity of the autonomic nervous system;

[0060] 6) Morlet Wavelet Transform, a time-frequency analysis method;

[0061] 7) FGSM method, which is the Fast Gradient Sign Method, is a method for generating adversarial examples;

[0062] 8) STFT, short-time Fourier transform, a time-frequency analysis method;

[0063] 9) FMCW radar is a frequency-modulated continuous wave radar (Frequency-Modulated Continuous Wave Radar), a radar technology;

[0064] 10) CAN bus, Controller Area Network Bus, a network protocol used for communication between electronic devices within a vehicle;

[0065] 11) HMM, Hidden Markov Model, a statistical model;

[0066] 12) LightGBM, a Light Gradient Boosting Machine, a machine learning algorithm;

[0067] 13) AR-HUD, which stands for Augmented Reality Head-Up Display, is an in-vehicle display technology.

[0068] 14) TSN, which stands for Time Synchronization Network, is a technology used for time synchronization of multimodal data.

[0069] The embodiments of the present invention are further described below with reference to the accompanying drawings.

[0070] On the one hand, the embodiment of the present invention provides a method for identifying and intervening in fatigue and dozing based on radar and wearable devices, referring to Figure 1 The fatigue nap recognition and intervention method based on radar and wearable devices includes the following steps:

[0071] S100, acquiring posture data of the target monitoring person and vehicle operation data;

[0072] S200, obtaining physiological signals of the target monitored person;

[0073] S300, identifying fatigue and dozing based on posture data, vehicle operation data, and physiological signals to obtain fatigue state judgment data;

[0074] S400, obtaining fatigue status classification information based on fatigue status judgment data;

[0075] S500: Complete intervention measures based on fatigue status classification information.

[0076] S100 disclosed in the embodiment of the present invention obtains the posture data of the target monitoring person and the vehicle operation data, including the following steps:

[0077] S110, acquiring data collected by the vehicle-mounted radar;

[0078] S120, preprocessing the collected data of the vehicle-mounted radar to extract the posture data of the target monitoring person and the vehicle operation data;

[0079] S130, posture data includes facial image data, eye state data, head posture data, and chest and abdomen movement data;

[0080] S140. Vehicle operation data includes driving speed data, braking data, and steering wheel control data.

[0081] As an optional implementation method, the vehicle-mounted radar module collects data such as the driver's facial image, eye status, head posture, chest and abdominal movement in real time, and also monitors vehicle operation data such as driving speed, braking, and steering wheel control.

[0082] S200 disclosed in the embodiment of the present invention obtains the physiological signal of the target monitored person, including the following steps:

[0083] S210, acquiring collected data from a wearable device; the wearable device includes a wearable flexible material device;

[0084] S220, performing data preprocessing on the collected data of the wearable device to extract the physiological signals of the target monitored person;

[0085] S230. Physiological signals include brain wave data, heart rate data, and skin electrical response data.

[0086] As an optional implementation, a wearable flexible material device collects the driver's physiological signals, including electroencephalogram (EEG), heart rate (HR), galvanic skin response (GSR), etc., to reflect the driver's physiological state in real time.

[0087] S300 disclosed in the embodiment of the present invention performs fatigue dozing identification based on posture data, vehicle operation data, and physiological signals to obtain fatigue state judgment data, including the following steps:

[0088] S310, using a dynamic time warping method to perform timestamp alignment on the posture data, vehicle operation data, and physiological signals to obtain a timestamp alignment matrix;

[0089] S320, obtaining radar visual flow based on the current vehicle speed;

[0090] S330, obtaining a physiological signal stream;

[0091] S340: Based on the timestamp alignment matrix, a multi-branch feature extraction network and a cross-modal attention mechanism are used to fuse radar visual stream, physiological signal stream, and vehicle operation data to generate comprehensive feature data.

[0092] S350. Analyze the comprehensive feature data using the trained deep learning model to obtain fatigue state judgment data.

[0093] S310 disclosed in the embodiment of the present invention uses a dynamic time warping method to align the timestamps of the posture data, vehicle operation data, and physiological signals to obtain a timestamp alignment matrix. The formula used includes:

[0094]

[0095] Among them, T sync Align the matrix for timestamps; is the i-th radar monitoring data; radar monitoring data includes posture data and vehicle operation data; is the i-th physiological signal; N is the total number of features monitored by radar and wearable devices.

[0096] S320 disclosed in the embodiment of the present invention combines the current vehicle speed to obtain the radar visual flow, and the formula used includes:

[0097]

[0098] ω velocity =1 / (1+e -0.1v );

[0099] Among them, PERCLOS is the radar visual flow; v is the current driving speed; t eyelid≥80％ T is the time when the eyelid coverage area is greater than or equal to 80%; window is the monitoring time window; ω velocity is the speed weight factor.

[0100] As an optional implementation, eyelid coverage is a key visual feature used to determine driver fatigue, analyzed using facial image data collected by the on-board radar module. Eyelid coverage refers to the extent to which the upper eyelid covers the eyeball when the driver's eyes are closed. This metric can be quantified and analyzed using image processing techniques and computer vision algorithms, and combined with other features (such as blink frequency and eye opening / closing status) to comprehensively determine the driver's fatigue state.

[0101] S400 disclosed in the embodiment of the present invention obtains fatigue state classification information based on fatigue state judgment data, including the following steps:

[0102] S410, setting fatigue level standards; fatigue level standards include mild fatigue, moderate fatigue, and severe fatigue;

[0103] S420. Obtain fatigue status classification information of the target monitored person based on the fatigue status judgment data and the fatigue degree standard.

[0104] As an optional implementation manner, the fatigue level standard of the embodiment of the present invention is:

[0105] Conditions classified as mild fatigue include: mild abnormalities in brain waves, mild changes in heart rate variability (HRV), and slight deviations from normal visual features (such as the PERCLOS algorithm).

[0106] Conditions classified as moderate fatigue include: a sharp increase in EEG theta waves, unstable steering wheel control, and large changes in vehicle speed.

[0107] Conditions classified as severe fatigue include: a significant increase in theta waves in the brain, extremely unstable driver behavior, and excessive frequency of eye closure.

[0108] As an optional implementation, the embodiment of the present invention takes corresponding interventions for different fatigue levels based on fatigue status classification information:

[0109] Mild fatigue intervention: Triggering mild intervention measures such as seat vibration (5Hz), air conditioning cooling by 2°C, speed limit +5%, etc.

[0110] Moderate fatigue intervention: activates moderate interventions such as seat belt tightening (50N), fragrance release, and increasing following distance (+20%).

[0111] Severe fatigue intervention: trigger electrical stimulation (3mA), automatic opening of the sunroof, automatic side parking and other strong interventions.

[0112] The radar and wearable device-based fatigue dozing identification and intervention method disclosed in the embodiment of the present invention further includes the following steps:

[0113] S600: Adjust the accuracy and response mechanism of the algorithm based on the feedback of the target monitoring personnel and their historical fatigue status.

[0114] As an optional implementation, in the embodiment of the present invention, the algorithm can automatically adapt and adjust its judgment and intervention strategy based on individual differences of drivers to ensure that each driver can receive appropriate monitoring and intervention.

[0115] On the other hand, an embodiment of the present invention further provides a system for identifying and intervening in fatigue and dozing based on radar and wearable devices, which is used to implement the aforementioned method for identifying and intervening in fatigue and dozing based on radar and wearable devices. The system for identifying and intervening in fatigue and dozing based on radar and wearable devices includes:

[0116] The vehicle-mounted radar module is used to collect information about the target person's facial expressions, eye opening and closing status, head posture, and chest and abdominal movements, as well as vehicle speed, braking, and steering control.

[0117] Wearable devices used to collect physiological signals of target monitoring personnel;

[0118] Data processing module, used to analyze data collected by the vehicle radar module and wearable devices;

[0119] The intervention module is used to intervene and wake up the target monitored person when fatigue or dozing is detected.

[0120] As an optional implementation, an embodiment of the present invention provides a fatigue and dozing identification and intervention system based on artificial intelligence radar monitoring of the eyes combined with wearable flexible materials to monitor vital signs. The system uses on-board radar to monitor the driver's facial expressions, eye opening and closing status (such as blinking frequency, eye closure duration), head posture and chest and abdominal activities in real time, and combines wearable flexible material equipment to monitor the driver's physiological signals (such as brain waves, heart rate, and skin electrical response) to achieve layered monitoring and management of fatigue status. The system can accurately determine whether the driver is in a state of fatigue or dozing, and ensure driving safety through multi-level reminders, alarms and intervention measures.

[0121] The system of the present invention comprises:

[0122] 1. Vehicle-mounted radar module: Installed inside the vehicle, it monitors the driver's facial expressions, eye opening and closing, head posture, and chest and abdominal movements in real time. It also dynamically monitors the driver's driving speed, braking, and steering control.

[0123] 2. Wearable flexible material device: Made of flexible materials, it fits on the driver's head or wrist and is used to monitor the driver's physiological signals in real time, such as electroencephalogram (EEG), heart rate (HR), and galvanic skin response (GSR).

[0124] 3. Data Processing Unit: This includes a microprocessor and artificial intelligence algorithm module, which analyzes data collected by onboard radar and wearable devices to determine whether the driver is fatigued or dozing. The algorithm module uses deep learning technology to adapt to the physiological characteristics of different drivers, improving judgment accuracy.

[0125] 4. Intervention module: includes a sound reminder, vibrator and electrical stimulator, which is used to wake up the driver through sound, vibration or mild electrical stimulation when fatigue or dozing is detected to ensure that the driver remains awake.

[0126] As an optional implementation, a method for identifying and intervening in fatigue and dozing based on radar and wearable devices according to an embodiment of the present invention includes:

[0127] 1. Equipment installation and wearing:

[0128] The on-board radar module is installed inside the vehicle to ensure that it can clearly capture the driver's facial expressions, eye conditions, head posture, and chest and abdominal movements; at the same time, it can dynamically monitor the driver's driving speed, braking, and steering wheel control.

[0129] The driver wears the wearable flexible material device on the head or wrist, ensuring that the flexible sensor is in close contact with the skin.

[0130] 2. Signal acquisition:

[0131] The on-board radar module collects the driver's facial image, head posture, chest and abdominal activity data in real time; as well as the driver's driving speed, braking and steering wheel control conditions.

[0132] Wearable flexible material devices collect the driver's physiological signals in real time, such as brain waves, heart rate and skin electrical response.

[0133] 3. Signal processing

[0134] The data processing unit pre-processes the data collected by the vehicle-mounted radar and wearable devices, and extracts characteristic parameters such as eye opening and closing status, blinking frequency, the ratio of alpha waves and theta waves in the brain waves, heart rate variability and changes in skin electrical response.

[0135] 4. Determining Fatigue and Dozing:

[0136] The data processing unit uses a deep learning algorithm to determine whether the driver is fatigued or dozing off based on characteristic parameters. The algorithm has been trained with extensive data and can accurately identify early signs of fatigue and dozing off.

[0137] Specifically, the training data includes facial expressions, eye movements, head posture, chest and abdominal movements detected by radar, as well as physiological signals from wearable devices, such as brain waves, heart rate, and galvanic skin response. Furthermore, in-vehicle data includes speed, braking, and steering control. This multimodal data requires fusion processing.

[0138] The embodiment of the present invention utilizes a multimodal data preprocessing framework, including:

[0139] 1. Data source synchronization mechanism

[0140] Create a timestamp alignment matrix:

[0141]

[0142] Millisecond-level synchronization is achieved through dynamic time warping (DTW), with the error controlled within ±50ms. sync Align the matrix for timestamps; is the i-th radar monitoring data; radar monitoring data includes posture data and vehicle operation data; is the i-th physiological signal; N is the total number of features monitored by radar and wearable devices.

[0143] 2. Feature Engineering

[0144] ① Radar visual stream (sampling rate 60Hz):

[0145] Eye dynamic features: using the improved PERCLOS algorithm:

[0146]

[0147] The speed weight factor ω velocity =1 / (1+e -0.1v ), PERCLOS is the radar visual flow; v is the current driving speed; t eyelid≥80％ T is the time when the eyelid coverage area is greater than or equal to 80%; window is the monitoring time window (e.g. 1-2 minutes).

[0148] ②Physiological signal stream (sampling rate 256Hz):

[0149] EEG feature extraction: based on Morlet wavelet transform:

[0150]

[0151] The focus is on monitoring the energy ratio of theta waves (4-7Hz) and alpha waves (8-12Hz). Where CWT(a,b) is the result of continuous wavelet transform; a is the scale parameter that controls the frequency component of the wavelet; b is the translation parameter that controls the position of the wavelet on the time axis; EEG(t) is the brain wave signal; t is the time point of the signal; ψ * is the complex conjugate of the wavelet basis function.

[0152] 3. Data Augmentation Strategy

[0153] Spatiotemporal perturbation enhancement: Gaussian noise ∈ ~N(0,0.1) is applied to the training data;

[0154] Adversarial sample generation: Improving model robustness through the FGSM method;

[0155]

[0156] Where x is the original input data (such as image, signal, etc.); ∈ is the perturbation coefficient, which obeys the Gaussian distribution ∈~N(0,0.1); sign() is the gradient sign function; is the gradient of the loss function with respect to the input data, indicating the direction in which the input data affects the loss; θ is the model parameter; y is the true label; x adv is a generated adversarial example.

[0157] The embodiment of the present invention utilizes a hybrid deep learning architecture, including:

[0158] 1.Reference Figure 2 , the present invention adopts a multi-branch feature extraction network:

[0159] Visual branch: self.vis_net, using the pre-trained 3D-ResNet34 model.

[0160] Physiology branch: self.bio_lstm, using the bidirectional LSTM model.

[0161] Vehicle dynamics branch: self.drive_fc, using a fully connected layer.

[0162] The network output is a joint feature vector.

[0163] 2. Attention Fusion Module:

[0164] Designing a cross-modal attention mechanism:

[0165]

[0166] Among them, α i is the attention weight of the i-th modal feature; q iis the query vector of the i-th modal feature; K is the key vector, which is a learnable parameter; is the dot product of the query vector and the key vector, indicating relevance; It takes the exponential function of the dot product result and maps it to the positive number range.

[0167] The joint training strategy of the embodiment of the present invention includes:

[0168] 1. Progressive training process

[0169] Phase 1: Single-modal pre-training:

[0170] The model is trained on the pre-collected single-modal data respectively.

[0171] Phase 2: Multimodal Joint Training:

[0172] Adopt a curriculum learning strategy and gradually increase the difficulty of samples.

[0173] Set dynamic loss weights:

[0174]

[0175] Among them, λ t is the dynamic loss weight at step t; t is the current number of training steps; T max is the total number of training steps.

[0176] 2. Composite Loss Function

[0177]

[0178] in, is the composite loss function; is the cross entropy loss; is the triplet loss; is the KL divergence, which is used to constrain the consistency of multimodal prediction; i is the i-th category of the true label; p i The probability of the i-th category predicted by the model.

[0179] Calculate the triplet loss: max(d(a,p)-d(a,n)+α,0), where a is the anchor sample; p is the positive sample; n is the negative sample; d(a,p) is the distance between the anchor sample and the positive sample; d(a,n) is the distance between the anchor sample and the negative sample; α is the interval parameter.

[0180] V. Intervention Measures

[0181] 1. Different intervention measures will be taken for different levels of fatigue. The following are the graded quantitative indicators:

[0182] ① EEG grading index (based on EEG time-frequency analysis):

[0183]

[0184] Among them, θ is the θ wave; β is the β wave; α is the α wave.

[0185] ②Heart rate variability (HRV):

[0186] Time domain indicators: SDNN < 50ms is severe, 50-100ms is moderate, and > 100ms is mild;

[0187] Frequency domain indicators: LF / HF ratio >3.0 triggers a severe warning.

[0188] 2. Visual behavior feature group:

[0189] ① Improved PERCLOS classification:

[0190]

[0191] in, Where v is the vehicle speed (km / h).

[0192] ②Micro-expression cycle analysis:

[0193] refer to Figure 3 , by analyzing the driver's facial features (such as the height-to-width ratio of the lips) to detect yawning behavior, thereby judging the driver's fatigue state.

[0194] Calculate the lip aspect ratio: Extract the lip aspect ratio from each frame image and generate a lip_ratio sequence.

[0195] Yawn detection: The frequency domain characteristics of the lip_ratio sequence are analyzed by short-time Fourier transform (STFT) to detect whether the peak value of the 0.5-2 Hz frequency band exceeds the threshold.

[0196] Return result: Returns whether yawning behavior is detected based on the detection result.

[0197] 3. Driving behavior characteristic group:

[0198] ① Steering correction entropy value:

[0199] H steer =-∑p(x)log p(x)

[0200] Among them, H steer is the steering correction entropy value, which is used to quantify the change law of the steering wheel angle; x is the piecewise statistic of the steering wheel angle differential signal; p(x) is the probability distribution of the piecewise statistic of the steering wheel angle differential signal; when H steer>4.2 triggers a moderate warning.

[0201] ②Speed maintenance:

[0202]

[0203] Where Δv std is the standard deviation of vehicle speed maintenance; v t is the vehicle speed at time t; is the average speed of the vehicle; N is the total number of samples in the time window.

[0204] Classification threshold:

[0205] Mild: Δv std <5km / h;

[0206] Moderate: 5 ≤ Δv std <10km / h;

[0207] Severe: Δv std ≥10km / h.

[0208] The patient's fatigue state is defined using a sequential state machine model:

[0209] 'Awake → Mild': 'reaching the mild threshold for 3 consecutive cycles (18s)',

[0210] 'Mild→Moderate': 'The current state lasts for 90 seconds and the steering wheel entropy increases',

[0211] 'Moderate → Severe': 'EEG θ / β increases by 50% and lasts for 30 seconds',

[0212] 'Any → awake': 'Not reaching any threshold for 5 consecutive minutes'.

[0213] Refer to Table 1, for different fatigue states detected, an interactive response mechanism is adopted:

[0214] Table 1

[0215] Fatigue level Physical intervention Environmental Regulation Vehicle Control Mild Seat vibration (5Hz) Air conditioning cools down by 2℃ Speed limit +5% Moderate Seat belt tightening (50N) Fragrance release Following distance +20% severe Electrical stimulation (3 mA) Automatic sunroof opening Automatic pull-over parking

[0216] Advantages of the present invention:

[0217] 1. High Accuracy: Combining on-board radar and wearable flexible material equipment, and adopting deep learning algorithms, it can accurately identify the driver's fatigue and dozing state, reducing the misjudgment rate.

[0218] 2. Comfort: Wearable flexible material equipment fits the skin, is comfortable to wear, and does not affect driving operations.

[0219] 3. Multiple intervention methods: Provides multiple intervention methods such as sound, vibration and electrical stimulation to meet the needs of different drivers.

[0220] 4. Adaptability: The algorithm module can adapt to the physiological characteristics of different drivers and improve the personalization level of monitoring and intervention.

[0221] As an optional implementation, refer to Figure 4 The system structure of the embodiment of the present invention includes:

[0222] 1. Multimodal data acquisition layer: with intelligent radar sensing system, wearable flexible devices, and vehicle dynamic bus.

[0223] The intelligent radar sensing system includes a 60GHz FMCW radar, a binocular camera, and a thermal imaging sensor. The 60GHz FMCW radar monitors the driver's facial expressions, eye state, head posture, and chest and abdominal movements; the binocular camera captures the driver's facial image; and the thermal imaging sensor detects changes in the driver's body temperature. The wearable flexible device includes a flexible inductor array, a three-axis accelerometer, and a bioimpedance sensor. The flexible inductor array monitors EEG, ECG, and galvanic skin response (EDA); the three-axis accelerometer detects the driver's body motion index; and the bioimpedance sensor measures physiological indicators such as blood oxygen saturation. The vehicle dynamics bus includes CAN bus data, which is used to collect driving behavior data such as vehicle speed, braking status, and steering angle.

[0224] Timestamp synchronization at the multimodal data acquisition layer: ensures the temporal consistency of multimodal data.

[0225] 2. Edge computing processing layer: has visual signal processing unit, physiological signal processing unit, and vehicle signal processing unit.

[0226] The functions of the visual signal processing unit include:

[0227] 3D-CNN real-time parsing: Extract features such as facial expressions and eye states from visual data.

[0228] PERCLOS calculation: calculates the proportion of eyelid coverage to determine the degree of eye closure.

[0229] Micro-expression detection: Identify changes in the driver's micro-expressions.

[0230] The functions of the physiological signal processing unit include:

[0231] Wavelet denoising: Reduce the noise of physiological signals to improve signal quality.

[0232] HRV time-frequency analysis: Analyze heart rate variability (HRV) to determine fatigue status.

[0233] EEG rhythm decomposition: analyzing electrical brain activity.

[0234] The functions of the vehicle signal processing unit include:

[0235] Kalman filter: Filters vehicle dynamic data to improve data accuracy.

[0236] Behavioral pattern recognition: Analyze driving behavior (such as steering angle and speed changes) to determine driving risks.

[0237] The edge computing processing layer can perform cross-modal feature fusion, and the data can be anonymized and uploaded to the multimodal data collection layer for iterative model training.

[0238] 3. Hierarchical decision engine: with feature fusion module, timing state machine, and hierarchical decision model.

[0239] The functions of the feature fusion module include:

[0240] Attention weight allocation: Dynamically allocate weights of different modal data through the attention mechanism.

[0241] Spatiotemporal alignment: aligning data of different modalities in time and space.

[0242] The functions of the sequential state machine include:

[0243] Based on Hidden Markov Model (HMM): Model the evolution path of fatigue state and analyze the driver's state transition from alertness to fatigue.

[0244] The functions of the hierarchical decision model include:

[0245] LightGBM Decision Tree: Use the LightGBM model to classify fatigue status.

[0246] Confidence calibration: Determine the light / moderate / severe fatigue status based on the confidence level of the model output.

[0247] 4. Multi-level intervention execution system: with human-computer interaction interface, vehicle control bus, and emergency avoidance system.

[0248] The functions of the human-computer interaction interface include:

[0249] AR-HUD projection: Provides visual warnings to the driver through augmented reality head-up display (AR-HUD).

[0250] Biofeedback micrometry: Remind the driver through biofeedback devices (such as seat vibration).

[0251] Multimodal alarm: combines sound, light, touch and other methods to provide multi-dimensional warnings.

[0252] The functions of the vehicle control bus include:

[0253] Electronic Power Steering: Adjusts steering assistance based on fatigue status.

[0254] Adaptive cruise control: adjusts vehicle speed to maintain a safe following distance.

[0255] Power Limitation: Limit vehicle power output to reduce driving risks.

[0256] The functions of the emergency avoidance system include:

[0257] AEB Pre-charge: Preloads automatic emergency braking to prepare for emergencies.

[0258] Double flash light trigger: trigger the double flash light in an emergency to alert other vehicles.

[0259] Cloud-based early warning: Upload fatigue status and driving risk information to the cloud for remote early warning.

[0260] As an optional implementation, refer to Figure 5 The working principle process of the embodiment of the present invention includes:

[0261] Multimodal data acquisition: Through the vehicle bus, wearable devices and radar sensor system, the posture data, vehicle operation data and physiological signals of the target monitoring personnel are collected in real time.

[0262] Edge processing: The vehicle processing unit, physiological processing unit, and visual processing unit are used to filter and extract features from the collected data, and time alignment of multimodal data is achieved through the time synchronization network (TSN).

[0263] Decision engine: Through the attention weight allocation mechanism, hidden Markov model and LightGBM classifier, the fatigue state of the target monitored personnel is judged and the classification result is calculated.

[0264] Execution system: Take appropriate intervention measures based on the fatigue status classification results.

[0265] Optimize system performance: Continuously optimize system performance through anonymized storage, model updates, and incremental learning.

[0266] On the other hand, an embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions, which are used to enable a computer to execute the above-mentioned fatigue dozing identification and intervention method based on radar and wearable devices.

[0267] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0268] On the other hand, an embodiment of the present invention further provides a device for identifying and intervening in fatigue and dozing based on radar and wearable devices, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for identifying and intervening in fatigue and dozing based on radar and wearable devices as described above is implemented.

[0269] The processor and the memory can be connected via a bus or other means. The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0270] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A method for identifying and intervening in fatigue and dozing based on radar and wearable devices, characterized in that: The method for identifying and intervening in fatigue and napping based on radar and wearable devices includes the following steps: Obtaining the target monitoring personnel's posture data and vehicle operation data; Acquire physiological signals of target monitoring personnel; performing fatigue dozing identification based on the posture data, the vehicle operation data, and the physiological signal to obtain fatigue state judgment data; obtaining fatigue state classification information according to the fatigue state judgment data; According to the fatigue status classification information, intervention measures are completed.

2. The method for identifying and intervening in fatigue and napping based on radar and wearable devices according to claim 1 is characterized in that: The step of obtaining the posture data and vehicle operation data of the target monitoring person includes the following steps: Obtain data collected by vehicle-mounted radar; Performing data preprocessing on the collected data of the vehicle-mounted radar to extract the posture data of the target monitoring person and the vehicle operation data; The posture data includes facial image data, eye state data, head posture data, and chest and abdomen movement data; The vehicle operation data includes driving speed data, braking data, and steering wheel control data.

3. The method for identifying and intervening in fatigue and napping based on radar and wearable devices according to claim 1, characterized in that: The step of obtaining the physiological signal of the target monitored person includes the following steps: Acquiring collected data from a wearable device; the wearable device includes a wearable flexible material device; Performing data preprocessing on the collected data of the wearable device to extract the physiological signals of the target monitored person; The physiological signals include brain wave data, heart rate data and skin electrical response data.

4. The method for identifying and intervening in fatigue and napping based on radar and wearable devices according to claim 1, characterized in that: The step of identifying fatigue and dozing based on the posture data, the vehicle operation data, and the physiological signal to obtain fatigue state judgment data includes the following steps: Performing time stamp alignment on the posture data, the vehicle operation data, and the physiological signal using a dynamic time warping method to obtain a time stamp alignment matrix; Combined with the current vehicle speed, radar visual flow is obtained; Acquire physiological signal streams; According to the timestamp alignment matrix, a multi-branch feature extraction network and a cross-modal attention mechanism are used to fuse the radar visual stream, the physiological signal stream, and the vehicle operation data to generate comprehensive feature data; The comprehensive feature data is analyzed using a trained deep learning model to obtain fatigue state judgment data.

5. The method for identifying and intervening in fatigue and dozing based on radar and wearable devices according to claim 4 is characterized in that: The dynamic time warping method is used to align the timestamps of the posture data, the vehicle operation data, and the physiological signal to obtain a timestamp alignment matrix. The formula used includes: Among them, T sync Align the matrix for timestamps; is the i-th radar monitoring data; the radar monitoring data includes the posture data and the vehicle operation data; is the i-th physiological signal; N is the total number of features monitored by radar and wearable devices.

6. The method for identifying and intervening in fatigue and dozing based on radar and wearable devices according to claim 4, characterized in that: The formula used to obtain the radar visual flow in combination with the current vehicle speed includes: oh velocity =1 / (1+e -0.1v ); Among them, PERCLOS is the radar visual flow; v is the current driving speed; t eyelid≥80％ T is the time when the eyelid coverage area is greater than or equal to 80%; window is the monitoring time window; ω velocity is the speed weight factor.

7. The method for identifying and intervening in fatigue and napping based on radar and wearable devices according to claim 1, characterized in that: The step of obtaining fatigue state classification information based on the fatigue state judgment data includes the following steps: Setting fatigue level standards; the fatigue level standards include mild fatigue, moderate fatigue, and severe fatigue; According to the fatigue state judgment data and the fatigue degree standard, the fatigue state classification information of the target monitored person is obtained.

8. The method for identifying and intervening in fatigue and napping based on radar and wearable devices according to claim 1, characterized in that: The method for identifying and intervening in fatigue and napping based on radar and wearable devices further includes the following steps: Adjust the algorithm's accuracy and response mechanism based on feedback from target monitors and their historical fatigue status.

9. A system for identifying and intervening in fatigue and dozing based on radar and wearable devices, for implementing the method for identifying and intervening in fatigue and dozing based on radar and wearable devices as claimed in any one of claims 1 to 8, characterized in that: The radar and wearable device-based fatigue and dozing identification and intervention system includes: The vehicle-mounted radar module is used to collect information about the target person's facial expressions, eye opening and closing status, head posture, and chest and abdominal movements, as well as vehicle speed, braking, and steering control. Wearable devices used to collect physiological signals of target monitoring personnel; A data processing module, configured to analyze data collected by the vehicle-mounted radar module and the wearable device; The intervention module is used to intervene and wake up the target monitored person when fatigue or dozing is detected.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the fatigue and doze identification and intervention method based on radar and wearable devices as described in any one of claims 1 to 8.

Images