Hierarchical security intervention method and system, electronic equipment and computer readable medium

Abstract

The invention provides a graded safety intervention method, and belongs to the field of intelligent driving. Calculating a fatigue index according to the multi-mode signal; determining an intervention level according to the fatigue index in combination with a scene adaptation engine; according to the method, cross validation is carried out through signals of two completely different principles of behavior and physiology (capacitance) and nerve reflex and physical sign (vision), extremely high reliability of judgment is ensured, and serious accidents caused by failure or false alarm of a single sensor are completely eradicated.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, and in particular to a graded safety intervention method, system, electronic device, and computer-readable medium. Background Technology

[0002] With the development of intelligent driving technology, driver fatigue monitoring systems (DMS) have become crucial for improving road safety. Current technologies mainly rely on two types of solutions:

[0003] 1. Vision-based DMS: It detects the driver's facial features (such as eyelid closure and yawning) through a camera, but it is greatly affected by lighting and occlusion, and cannot distinguish between intentional eye closure and true unconsciousness.

[0004] 2. Vehicle behavior-based DMS: It infers fatigue by observing steering wheel angle fluctuations or lane departures, but the response is delayed and it cannot identify unconscious states (such as sudden illness).

[0005] The false alarm rate of a single sensor is high (>30%), and existing intervention strategies are mostly uniform horn blaring or emergency braking, lacking precise response mechanisms based on different scenarios and levels, especially when the driver is unconscious and cannot take autonomous avoidance. Summary of the Invention

[0006] The present invention aims to solve at least one of the technical problems existing in the prior art, and proposes a graded safety intervention method and system.

[0007] In a first aspect, embodiments of the present invention provide a graded security intervention method, comprising:

[0008] Acquire multimodal signals;

[0009] The fatigue index is calculated based on the multimodal signals.

[0010] The intervention level is determined based on the fatigue index and the scene adaptation engine.

[0011] Implement tiered intervention measures based on the intervention level.

[0012] In some embodiments, the step of acquiring multimodal signals includes extracting visual modal features and extracting capacitive modal features.

[0013] In some embodiments, the step of extracting visual modal features includes: inputting an infrared camera video stream, using a MediaPipe facial mesh model to track 468 facial feature points in real time, and then outputting PERCLOS and yawn frequency;

[0014] The steps for extracting capacitive modal features include: inputting the original signal from the steering wheel capacitive sensing array, decomposing the original signal using wavelet transform, filtering out high-frequency noise caused by hand sweat and minor movements, reconstructing a clean signal, extracting low-frequency components from the filtered capacitive signal as an EDA signal, and calculating its mean or standard deviation as feature values; extracting periodic fluctuations synchronized with the heartbeat from the capacitive signal, obtaining the RR interval sequence through a peak detection algorithm, and then calculating time-domain or frequency-domain features, normalizing them into a comprehensive HRV stress index.

[0015] In some embodiments, the step of calculating the fatigue index based on the multimodal signal includes:

[0016] Submodal score normalization: All extracted original features (PERCLOS, yawn frequency, EDA, HRV) are normalized by min-max normalization or Z-score and mapped to the [0,1] interval, where 1 represents the most fatigued state;

[0017] Intramodal weighted fusion: Define visual modality score V = 0.7 * normalization (PERCLOS) + 0.3 * normalization (yawn frequency), and physiological modality score P = 0.5 * normalization (EDA) + 0.5 * normalization (HRV);

[0018] Intermodal weighted fusion: Define the fatigue index F = α * V + β * P, where α = 0.6 and β = 0.4;

[0019] Confidence check and intervention trigger: The F-value is calculated in real time, and the intervention process is triggered if and only if F > 0.8.

[0020] In some embodiments, the step of determining the intervention level based on the fatigue index and in conjunction with the scene adaptation engine includes:

[0021] Receive the real-time fatigue index F;

[0022] The F-value is directly mapped to intervention levels L1-L4; where: F=0.81 maps to intervention level L1, F=0.85 maps to intervention level L2, F=0.90 maps to intervention level L3, and F=0.95 maps to intervention level L4.

[0023] Based on the scenario, the above decisions are adapted as follows: if the vehicle speed is >80km / h: skip L1 and go directly to L2; if it is raining: disable window lowering and start the air conditioning external circulation.

[0024] In some embodiments, the step of performing graded interventions according to the intervention level includes:

[0025] L1 intervention: Releases eucalyptus oil or other invigorating aromas;

[0026] L2 intervention: 3Hz lumbar support pressure, 5cm window reduction and voice prompts;

[0027] L3 intervention: After multiple attempts to wake up the user failed and no abnormalities were found in the user's physiological signs, L2.9 level autonomous driving takeover was activated and the user was navigated to the nearest service area to stop.

[0028] L4 intervention: Detects driver unconsciousness, enables autonomous parking, and triggers ECALL emergency call.

[0029] In some embodiments, detecting whether a driver is unconscious includes: detecting whether a driver is unconscious by capacitive signals, and determining that the driver is unconscious when there is zero grip strength and a sudden drop in skin conductance for 10 seconds; and detecting whether a driver is unconscious by visual signals, and determining that the driver is unconscious when there is pupil dilation and no blinking reflex.

[0030] Secondly, the present invention also provides a graded safety intervention system, comprising:

[0031] The acquisition unit is used to acquire multimodal signals;

[0032] A calculation unit is used to calculate the fatigue index based on the multimodal signal;

[0033] An intervention unit is used to determine the intervention level based on the fatigue index and in conjunction with the scene adaptation engine.

[0034] An execution unit is used to execute graded intervention measures according to the intervention level.

[0035] Thirdly, the present invention also provides an electronic device, comprising:

[0036] One or more processors;

[0037] Memory, used to store one or more programs;

[0038] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the methods.

[0039] Fourthly, the present invention also provides a computer-readable medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps in any of the methods described.

[0040] The graded safety intervention method provided by this invention collects multimodal signals; calculates a fatigue index based on the multimodal signals; determines the intervention level based on the fatigue index and in conjunction with a scene adaptation engine; and executes graded intervention measures according to the intervention level. This invention cross-validates signals based on two completely different principles: behavioral and physiological (capacitance) signals and neural reflexes and physical signs (visual) signals. This ensures extremely high reliability of the judgment and eliminates serious accidents caused by the failure or false alarm of a single sensor. At the same time, the system completes a fully automatic closed loop from "perception" (detecting unconsciousness) to "decision" (triggering L4) and then to "execution" (pushing over to the side of the road and calling for rescue). In critical moments when the driver is completely incapacitated, the system becomes a guardian of life. Attached Figure Description

[0041] Figure 1 This is a schematic diagram illustrating the steps of an embodiment of the graded safety intervention method of the present invention;

[0042] Figure 2 This is a schematic diagram illustrating the principle of one embodiment of the decision tree of the present invention;

[0043] Figure 3 This is a schematic diagram illustrating the principle of one embodiment of the graded safety intervention system of the present invention;

[0044] Figure 4 This is a schematic diagram of the structure of an embodiment of the electronic device of the present invention. Detailed Implementation

[0045] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0046] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0047] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0048] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0049] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0050] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0051] Among related technologies, the false alarm rate of a single sensor is high (>30%), and existing intervention strategies are mostly uniform horn blaring or emergency braking, lacking precise response mechanisms based on different scenarios and levels, especially when the driver is unconscious and cannot autonomously avoid danger.

[0052] To address at least one of the technical problems existing in the aforementioned related technologies, the present invention provides a graded safety intervention method. Figure 1 The flowchart illustrates the steps of a graded safety intervention method provided in an embodiment of the present invention.

[0053] like Figure 1 As shown, this graded safety intervention method includes the following steps:

[0054] Step S10: Acquire multimodal signals.

[0055] In this embodiment, the step of acquiring multimodal signals includes extracting visual modal features and extracting capacitive modal features.

[0056] In the visual modality feature extraction step, an infrared camera (850nm, 30fps) video stream is input. 468 facial feature points are tracked in real-time using a MediaPipe facial mesh model. Output features include:

[0057] PERCLOS: The percentage of frames in a unit of time (e.g., 3 minutes) where the eyelids are closed more than 80%. It is calculated from the Euclidean distance of eyelid feature points (e.g., left eye: points 33, 133, 145, 153; right eye: points 362, 263, 386, 374).

[0058] Yawning frequency: By calculating the aspect ratio between characteristic points at the corners of the mouth (such as points 61, 291, 0, and 17), a threshold is set, the number of yawns per unit time is counted, and normalized to the frequency (times / minute).

[0059] In the capacitive mode feature extraction step, the raw signal from the steering wheel capacitive sensing array (0.5mm pitch ITO electrodes, 100Hz sampling rate) is input. Wavelet transform (e.g., db4 wavelet) is used to decompose the raw signal, filtering out high-frequency noise caused by hand sweat and minor movements, reconstructing a clean signal. The following feature separation is then performed:

[0060] EDA (Electrodermal Activity): Extract low-frequency components (0.05-0.2Hz) from the filtered capacitance signal as EDA signals, and calculate their mean or standard deviation as characteristic values.

[0061] HRV (Heart Rate Variability): Extracts periodic fluctuations synchronized with the heartbeat from the capacitance signal, obtains the RR interval sequence through peak detection algorithm, and then calculates time-domain (such as SDNN) or frequency-domain (LF / HF power) features, normalizing them into a comprehensive HRV stress index.

[0062] It is understood that this embodiment transforms raw, unstructured sensor data into quantifiable feature indicators with physiological and behavioral significance, in preparation for subsequent fusion computing.

[0063] Step S20: Calculate the fatigue index based on the multimodal signal.

[0064] The step of calculating the fatigue index based on the multimodal signal includes:

[0065] Submodal score normalization: All extracted original features (PERCLOS, yawn frequency, EDA, HRV) are normalized by min-max normalization or Z-score and mapped to the [0,1] interval, where 1 represents the most fatigued state.

[0066] Intramodal weighted fusion: Define the visual modality score V = 0.7 * normalized (PERCLOS) + 0.3 * normalized (yawn frequency). Since PERCLOS is recognized as the most reliable visual fatigue indicator, it is given higher weight; yawn frequency serves as an effective supplement.

[0067] Define the physiological modality score P = 0.5 * normalized (EDA) + 0.5 * normalized (HRV)

[0068] Since both EDA and HRV can effectively reflect the stress and fatigue state of the autonomic nervous system, they are temporarily considered to be of equal importance.

[0069] Intermodal weighted fusion: The fatigue index is defined as F = α * V + β * P, where α = 0.6 and β = 0.4. This indicates that the system relies more on visual behavioral signals, with physiological signals serving as verification and supplementation. These parameters can be preset initially and then fine-tuned (i.e., trained) using collected driver data in the machine learning model.

[0070] Confidence check and intervention trigger: The system calculates the F-value in real time. The intervention process is triggered only if F > 0.8. This high threshold (0.8) ensures that the system only acts when it is highly certain that the driver is fatigued, greatly reducing the interference of false alarms.

[0071] As can be understood, this embodiment uses a carefully designed weighting formula to fuse signals from different modalities and with different dimensions into a unified and interpretable fatigue index. The confidence fusion is reflected in the fact that only when the fused comprehensive index F exceeds a high threshold (0.8) is the detection result considered "high confidence," thus initiating intervention. This solves the problem of easy misjudgment in single-modal analysis.

[0072] Step S30: Determine the intervention level based on the fatigue index and in conjunction with the scene adaptation engine.

[0073] In this embodiment, the step of determining the intervention level based on the fatigue index and in conjunction with the scene adaptation engine includes:

[0074] Input: Fatigue index F (from step S20) and real-time scene data, including vehicle speed (CAN bus), weather (rain sensor / V2X), location / GPS, and time (day and night).

[0075] Basic level mapping: Based on the F value, it is directly mapped to intervention levels L1-L4, where: F=0.81 is mapped to intervention level L1, F=0.85 is mapped to intervention level L2, F=0.90 is mapped to intervention level L3, and F=0.95 is mapped to intervention level L4.

[0076] The above decisions should be adapted to the specific scenario.

[0077] Specifically, the scenario rules cover the following:

[0078] Rule 1: IF vehicle speed > 80 km / h AND base level == L1 THEN intervention level = L2

[0079] Understandably, in high-speed environments, mild interventions (such as fragrance) may be ineffective and inappropriate, requiring a more proactive approach.

[0080] Rule 2: IF Rainfall == True THEN Disable "Lower Window" function

[0081] Understandable, safety is the priority, and it's important to prevent rainwater from getting into the car.

[0082] Rule 3: IF F >= 0.9 AND Location on highway THEN Automated driver takeover target = nearest service area.

[0083] Understandably, on highways, the goal of Level 3 intervention is to safely guide passengers to service areas, rather than to the roadside.

[0084] Please see Figure 2 The flowchart of the decision tree provided in this embodiment specifically includes the following steps:

[0085] The system receives the real-time fatigue index F. Based on the calculated fatigue index F, the system's decision layer has preliminarily determined whether to initiate L1 (mild) or L2 (moderate) intervention. The role of this decision tree is not to decide whether to use L1 or L2, but to fine-tune and optimize the preliminarily determined L1 or L2 intervention plan according to the specific scenario.

[0086] How to determine if the vehicle speed is greater than 80 km / h?

[0087] Understandably, the core of this judgment is the trade-off between safety and efficiency, ensuring that mild interventions that may be ineffective or distracting are not used when driving at high speeds.

[0088] The judgment conditions include: the system obtains the current vehicle speed in real time through the CAN bus and compares it with a preset threshold (80 km / h).

[0089] When the "Yes" option is selected (vehicle speed > 80 km / h): Skip L1 intervention. At high speeds, drivers need to maintain a high level of concentration. L1 interventions (such as releasing fragrance) are relatively mild and slow-acting, and may not quickly alleviate existing fatigue, thus failing to promptly avoid high-risk situations at high speeds. Therefore, the system considers L1 intervention insufficient in terms of both "cost-effectiveness" and "timeliness" in this scenario. Regardless of whether the initial judgment is L1 or L2, as long as the vehicle speed exceeds 80 km / h, the system will directly prepare to execute L2-level intervention measures and proceed to the next level of judgment.

[0090] "No" Branch (Vehicle speed ≤ 80 km / h): Execute L1 intervention. In low-speed, urban, or congested traffic conditions, the risk is relatively controllable. The system first performs the least invasive L1 intervention (such as releasing fragrance), a low-cost, non-disruptive approach that aligns with the "minimum intervention" principle. If fatigue is alleviated at this stage, more aggressive measures are avoided. While performing L1 intervention, the system continues to monitor the fatigue index. If the index decreases, the process ends; if the index continues to worsen, the system escalates to L2 intervention and proceeds to the next level of assessment.

[0091] Determining whether rainfall is imminent? The core of this assessment is environmental adaptability, ensuring that intervention measures do not create secondary risks or reduce comfort when implemented.

[0092] The judgment criteria include: the system determines whether it is currently in a rainy state by using rain sensors or online weather information.

[0093] "Yes" branch (Rainy Day): Disable window lowering, activate air conditioning external circulation. This is an intelligent overlay of the "lower window 5cm" command in L2 intervention. Disabling window lowering prevents rainwater from entering the vehicle, affecting the driver, damaging the interior, or causing short circuits in electronic devices. Activating air conditioning external circulation is an alternative; it introduces fresh air at a lower temperature from outside, achieving the same ventilation and refreshing effect while avoiding the problems caused by rain. Final execution: L2 intervention (rainy day version) = 3Hz lumbar support press + activate air conditioning external circulation + voice prompt.

[0094] "No" option (no rain): Execute L2 window lowering. In good weather, directly lowering the window is the most direct and effective way to quickly introduce fresh air and stimulate the driver using wind noise and ambient sounds. Final execution: L2 intervention (standard version) = 3Hz lumbar support press + 5cm window lowering + voice prompt.

[0095] Step S40: Implement tiered intervention measures according to the intervention level.

[0096] In this embodiment, the step of implementing graded intervention measures according to the intervention level includes:

[0097] L1 Intervention: IF Intervention Level == L1 THEN. Activate aromatherapy diffuser (eucalyptus oil, 0.1ppm).

[0098] L2 Intervention: IF Intervention Level == L2 THEN {Activate seat massage module (3Hz, lumbar support); Invoke window controller (lowers by 5cm); Play voice prompt ("You are fatigued, please be alert!")}

[0099] L3 intervention:

[0100] IF Intervention Level == L3 THEN {

[0101] 1. Continuously monitor the F value. If F > 0.85 for three consecutive periods, then:

[0102] 2. Send a takeover request via the L2.9 level autonomous driving API.

[0103] 3. Plan a route to the nearest safe area (service area / parking area).

[0104] 4. After the vehicle has come to a safe stop, keep the alarm on.

[0105] }

[0106] L4 intervention:

[0107] IF Intervention Level == L4 THEN {

[0108] 1. Verify pupil dilation via camera + heart rate <40 bpm via capacitive signal.

[0109] 2. Trigger the emergency pullover protocol (activate hazard lights and avoid obstacles).

[0110] 3. Once the vehicle has come to a complete stop, immediately activate the ECALL rescue module and send the precise location and emergency code via 4G / 5G V2X.

[0111] }

[0112] In this embodiment, detecting whether the driver is unconscious includes: detecting whether the driver is unconscious by capacitive signals, and determining whether the driver is unconscious when there is zero grip strength and a sudden drop in skin conductance for 10 seconds; and detecting whether the driver is unconscious by visual signals, and determining whether the driver is unconscious when there is pupil dilation and no blinking reflex.

[0113] It is understandable that the system uses capacitive and visual signals for cross-verification, and only triggers the final rescue when the two conclusions are consistent. This dual verification mechanism minimizes the risk of sudden vehicle braking caused by false triggering (such as the driver briefly closing their eyes or adjusting their grip).

[0114] A. Determination of capacitance signals (physiological and behavioral indicators)

[0115] Zero grip for 10 seconds: The steering wheel's capacitive sensor array continuously monitors the hand contact area and pressure distribution. When the system detects that the capacitance value of all electrodes is below a preset threshold (indicating that the hands are completely off or only slightly touching, without generating effective grip force), and this state lasts for a full 10 seconds, it is determined to be "zero grip force." In non-autonomous driving mode, it is highly abnormal for a driver's hands to be completely off the steering wheel for an extended period of time, and is a key behavioral indicator of loss of control. The 10-second duration is to filter out normal behaviors such as the driver briefly picking up items.

[0116] Sudden drop in skin conductance: EDA (electrodermal activity) data extracted from the same capacitance signal. When a driver is unconscious, the autonomic nervous system may malfunction, causing skin conductance levels to drop sharply from a certain level to an extremely low baseline, which is distinctly different from the slow change pattern when asleep or relaxed. The sudden disappearance of EDA, combined with zero grip strength, strongly suggests that the driver no longer has a normal physiological stress response, supporting the determination of unconsciousness.

[0117] B. Visual signal determination (physiological and neural reflex indicators)

[0118] Mydriasis (diameter > 5mm): An infrared camera and MediaPipe model are used to locate and measure the pupil diameter. In a normally lit vehicle interior, the pupil diameter is typically less than 4mm. Mydriasis is defined as a pupil diameter consistently greater than 5mm without any response to light. The pupillary light reflex is an important indicator of brainstem function. In a coma, severe brain damage often leads to paralysis of the pupillary sphincter, resulting in bilateral pupillary dilation and loss of light reflex. This is a very serious medical emergency.

[0119] Absence of blink reflex: The system continuously monitors blinking. Normal individuals blink regularly (approximately 15-20 times per minute) even when not fatigued. When the system fails to detect any complete blink within 10-15 seconds, it is defined as "absence of blink reflex." Blinking is a fundamental reflex that protects the eyeball and maintains the tear film. Prolonged lack of blinking, especially under strong light or wind stimulation (if the window may have been lowered previously), is medically strong evidence of loss of consciousness.

[0120] The graded safety intervention method provided by this invention cross-verifies signals based on two completely different principles: behavior and physiology (capacitance) and nerve reflexes and signs (vision). This ensures extremely high reliability of the judgment and eliminates serious accidents caused by the failure or false alarm of a single sensor.

[0121] Furthermore, the system achieves a closed loop from detection to execution: from "perception" (detecting unconsciousness), to "decision" (triggering L4), and then to "execution" (pushing over and calling for help). In critical moments when the driver is completely incapacitated, the system becomes a lifesaver. This design also adheres to the safety objectives of automotive functional safety standards (such as ISO 26262), which state that when the system detects a serious malfunction or hazard, it should be able to automatically enter or maintain a safe state. Here, "unconscious driving" is the ultimate danger, while "safely pulling over and calling for help" is the corresponding safe state.

[0122] Please see Figure 3 The present invention also provides a graded security intervention system. Applied to the graded security intervention method provided in the above embodiments, it specifically includes:

[0123] The acquisition unit is used to acquire multimodal signals, and its corresponding signal acquisition layer.

[0124] In this embodiment, the step of acquiring multimodal signals includes extracting visual modal features and extracting capacitive modal features.

[0125] In the visual modality feature extraction step, an infrared camera (850nm, 30fps) video stream is input. 468 facial feature points are tracked in real-time using a MediaPipe facial mesh model. Output features include:

[0126] PERCLOS: The percentage of frames in a unit of time (e.g., 3 minutes) where the eyelids are closed more than 80%. It is calculated from the Euclidean distance of eyelid feature points (e.g., left eye: points 33, 133, 145, 153; right eye: points 362, 263, 386, 374).

[0127] Yawning frequency: By calculating the aspect ratio between characteristic points at the corners of the mouth (such as points 61, 291, 0, and 17), a threshold is set, the number of yawns per unit time is counted, and normalized to the frequency (times / minute).

[0128] In the capacitive mode feature extraction step, the raw signal from the steering wheel capacitive sensing array (0.5mm pitch ITO electrodes, 100Hz sampling rate) is input. Wavelet transform (e.g., db4 wavelet) is used to decompose the raw signal, filtering out high-frequency noise caused by hand sweat and minor movements, reconstructing a clean signal. The following feature separation is then performed:

[0129] EDA (Electrodermal Activity): Extract low-frequency components (0.05-0.2Hz) from the filtered capacitance signal as EDA signals, and calculate their mean or standard deviation as characteristic values.

[0130] HRV (Heart Rate Variability): Extracts periodic fluctuations synchronized with the heartbeat from the capacitance signal, obtains the RR interval sequence through peak detection algorithm, and then calculates time-domain (such as SDNN) or frequency-domain (LF / HF power) features, normalizing them into a comprehensive HRV stress index.

[0131] It is understood that this embodiment transforms raw, unstructured sensor data into quantifiable feature indicators with physiological and behavioral significance, in preparation for subsequent fusion computing.

[0132] The calculation unit is used to calculate the fatigue index based on the multimodal signal, and its corresponding data processing layer.

[0133] The step of calculating the fatigue index based on the multimodal signal includes:

[0134] Submodal score normalization: All extracted original features (PERCLOS, yawn frequency, EDA, HRV) are normalized by min-max normalization or Z-score and mapped to the [0,1] interval, where 1 represents the most fatigued state.

[0135] Intramodal weighted fusion: Define the visual modality score V = 0.7 * normalized (PERCLOS) + 0.3 * normalized (yawn frequency). Since PERCLOS is recognized as the most reliable visual fatigue indicator, it is given higher weight; yawn frequency serves as an effective supplement.

[0136] Define the physiological modality score P = 0.5 * normalized (EDA) + 0.5 * normalized (HRV)

[0137] Since both EDA and HRV can effectively reflect the stress and fatigue state of the autonomic nervous system, they are temporarily considered to be of equal importance.

[0138] Intermodal weighted fusion: The fatigue index is defined as F = α * V + β * P, where α = 0.6 and β = 0.4. This indicates that the system relies more on visual behavioral signals, with physiological signals serving as verification and supplementation. These parameters can be preset initially and then fine-tuned (i.e., trained) using collected driver data in the machine learning model.

[0139] Confidence check and intervention trigger: The system calculates the F-value in real time. The intervention process is triggered only if F > 0.8. This high threshold (0.8) ensures that the system only acts when it is highly certain that the driver is fatigued, greatly reducing the interference of false alarms.

[0140] As can be understood, this embodiment uses a carefully designed weighting formula to fuse signals from different modalities and with different dimensions into a unified and interpretable fatigue index. The confidence fusion is reflected in the fact that only when the fused comprehensive index F exceeds a high threshold (0.8) is the detection result considered "high confidence," thus initiating intervention. This solves the problem of easy misjudgment in single-modal analysis.

[0141] An intervention unit is used to determine the intervention level based on the fatigue index and in conjunction with the scene adaptation engine, corresponding to the decision-making layer.

[0142] In this embodiment, the step of determining the intervention level based on the fatigue index and in conjunction with the scene adaptation engine includes:

[0143] Input: Fatigue index F (from step S20) and real-time scene data, including vehicle speed (CAN bus), weather (rain sensor / V2X), location / GPS, and time (day and night).

[0144] Basic level mapping: Based on the F value, it is directly mapped to intervention levels L1-L4, where: F=0.81 is mapped to intervention level L1, F=0.85 is mapped to intervention level L2, F=0.90 is mapped to intervention level L3, and F=0.95 is mapped to intervention level L4.

[0145] The above decisions should be adapted to the specific scenario.

[0146] Specifically, the scenario rules cover the following:

[0147] Rule 1: IF vehicle speed > 80 km / h AND base level == L1 THEN intervention level = L2

[0148] Understandably, in high-speed environments, mild interventions (such as fragrance) may be ineffective and inappropriate, requiring a more proactive approach.

[0149] Rule 2: IF Rainfall == True THEN Disable "Lower Window" function

[0150] Understandable, safety is the priority, and it's important to prevent rainwater from getting into the car.

[0151] Rule 3: IF F >= 0.9 AND Location on highway THEN Automated driver takeover target = nearest service area.

[0152] Understandably, on highways, the goal of Level 3 intervention is to safely guide passengers to service areas, rather than to the roadside.

[0153] Please see Figure 2 The flowchart of the decision tree provided in this embodiment specifically includes the following steps:

[0154] The system receives the real-time fatigue index F. Based on the calculated fatigue index F, the system's decision layer has preliminarily determined whether to initiate L1 (mild) or L2 (moderate) intervention. The role of this decision tree is not to decide whether to use L1 or L2, but to fine-tune and optimize the preliminarily determined L1 or L2 intervention plan according to the specific scenario.

[0155] How to determine if the vehicle speed is greater than 80 km / h?

[0156] Understandably, the core of this judgment is the trade-off between safety and efficiency, ensuring that mild interventions that may be ineffective or distracting are not used when driving at high speeds.

[0157] The judgment conditions include: the system obtains the current vehicle speed in real time through the CAN bus and compares it with a preset threshold (80 km / h).

[0158] When the "Yes" option is selected (vehicle speed > 80 km / h): Skip L1 intervention. At high speeds, drivers need to maintain a high level of concentration. L1 interventions (such as releasing fragrance) are relatively mild and slow-acting, and may not quickly alleviate existing fatigue, thus failing to promptly avoid high-risk situations at high speeds. Therefore, the system considers L1 intervention insufficient in terms of both "cost-effectiveness" and "timeliness" in this scenario. Regardless of whether the initial judgment is L1 or L2, as long as the vehicle speed exceeds 80 km / h, the system will directly prepare to execute L2-level intervention measures and proceed to the next level of judgment.

[0159] "No" Branch (Vehicle speed ≤ 80 km / h): Execute L1 intervention. In low-speed, urban, or congested traffic conditions, the risk is relatively controllable. The system first performs the least invasive L1 intervention (such as releasing fragrance), a low-cost, non-disruptive approach that aligns with the "minimum intervention" principle. If fatigue is alleviated at this stage, more aggressive measures are avoided. While performing L1 intervention, the system continues to monitor the fatigue index. If the index decreases, the process ends; if the index continues to worsen, the system escalates to L2 intervention and proceeds to the next level of assessment.

[0160] Determining whether rainfall is imminent? The core of this assessment is environmental adaptability, ensuring that intervention measures do not create secondary risks or reduce comfort when implemented.

[0161] The judgment criteria include: the system determines whether it is currently in a rainy state by using rain sensors or online weather information.

[0162] "Yes" branch (Rainy Day): Disable window lowering, activate air conditioning external circulation. This is an intelligent overlay of the "lower window 5cm" command in L2 intervention. Disabling window lowering prevents rainwater from entering the vehicle, affecting the driver, damaging the interior, or causing short circuits in electronic devices. Activating air conditioning external circulation is an alternative; it introduces fresh air at a lower temperature from outside, achieving the same ventilation and refreshing effect while avoiding the problems caused by rain. Final execution: L2 intervention (rainy day version) = 3Hz lumbar support press + activate air conditioning external circulation + voice prompt.

[0163] "No" option (no rain): Execute L2 window lowering. In good weather, directly lowering the window is the most direct and effective way to quickly introduce fresh air and stimulate the driver using wind noise and ambient sounds. Final execution: L2 intervention (standard version) = 3Hz lumbar support press + 5cm window lowering + voice prompt.

[0164] An execution unit is used to execute graded intervention measures according to the intervention level, and its corresponding execution layer.

[0165] In this embodiment, the step of implementing graded intervention measures according to the intervention level includes:

[0166] L1 Intervention: IF Intervention Level == L1 THEN. Activate aromatherapy diffuser (eucalyptus oil, 0.1ppm).

[0167] L2 Intervention: IF Intervention Level == L2 THEN {Activate seat massage module (3Hz, lumbar support); Invoke window controller (lowers by 5cm); Play voice prompt ("You are fatigued, please be alert!")}

[0168] L3 intervention:

[0169] IF Intervention Level == L3 THEN {

[0170] 1. Continuously monitor the F value. If F > 0.85 for three consecutive periods, then:

[0171] 2. Send a takeover request via the L2.9 level autonomous driving API.

[0172] 3. Plan a route to the nearest safe area (service area / parking area).

[0173] 4. After the vehicle has come to a safe stop, keep the alarm on.

[0174] }

[0175] L4 intervention:

[0176] IF Intervention Level == L4 THEN {

[0177] 1. Verify pupil dilation via camera + heart rate <40 bpm via capacitive signal.

[0178] 2. Trigger the emergency pullover protocol (activate hazard lights and avoid obstacles).

[0179] 3. Once the vehicle has come to a complete stop, immediately activate the ECALL rescue module and send the precise location and emergency code via 4G / 5G V2X.

[0180] }

[0181] In this embodiment, detecting whether the driver is unconscious includes: detecting whether the driver is unconscious by capacitive signals, and determining whether the driver is unconscious when there is zero grip strength and a sudden drop in skin conductance for 10 seconds; and detecting whether the driver is unconscious by visual signals, and determining whether the driver is unconscious when there is pupil dilation and no blinking reflex.

[0182] It is understandable that the system uses capacitive and visual signals for cross-verification, and only triggers the final rescue when the two conclusions are consistent. This dual verification mechanism minimizes the risk of sudden vehicle braking caused by false triggering (such as the driver briefly closing their eyes or adjusting their grip).

[0183] A. Determination of capacitance signals (physiological and behavioral indicators)

[0184] Zero grip for 10 seconds: The steering wheel's capacitive sensor array continuously monitors the hand contact area and pressure distribution. When the system detects that the capacitance value of all electrodes is below a preset threshold (indicating that the hands are completely off or only slightly touching, without generating effective grip force), and this state lasts for a full 10 seconds, it is determined to be "zero grip force." In non-autonomous driving mode, it is highly abnormal for a driver's hands to be completely off the steering wheel for an extended period of time, and is a key behavioral indicator of loss of control. The 10-second duration is to filter out normal behaviors such as the driver briefly picking up items.

[0185] Sudden drop in skin conductance: EDA (electrodermal activity) data extracted from the same capacitance signal. When a driver is unconscious, the autonomic nervous system may malfunction, causing skin conductance levels to drop sharply from a certain level to an extremely low baseline, which is distinctly different from the slow change pattern when asleep or relaxed. The sudden disappearance of EDA, combined with zero grip strength, strongly suggests that the driver no longer has a normal physiological stress response, supporting the determination of unconsciousness.

[0186] B. Visual signal determination (physiological and neural reflex indicators)

[0187] Mydriasis (diameter > 5mm): An infrared camera and MediaPipe model are used to locate and measure the pupil diameter. In a normally lit vehicle interior, the pupil diameter is typically less than 4mm. Mydriasis is defined as a pupil diameter consistently greater than 5mm without any response to light. The pupillary light reflex is an important indicator of brainstem function. In a coma, severe brain damage often leads to paralysis of the pupillary sphincter, resulting in bilateral pupillary dilation and loss of light reflex. This is a very serious medical emergency.

[0188] Absence of blink reflex: The system continuously monitors blinking. Normal individuals blink regularly (approximately 15-20 times per minute) even when not fatigued. When the system fails to detect any complete blink within 10-15 seconds, it is defined as "absence of blink reflex." Blinking is a fundamental reflex that protects the eyeball and maintains the tear film. Prolonged lack of blinking, especially under strong light or wind stimulation (if the window may have been lowered previously), is medically strong evidence of loss of consciousness.

[0189] The graded safety intervention system provided by this invention cross-verifies signals based on two completely different principles: behavior and physiology (capacitance) and nerve reflexes and signs (vision). This ensures extremely high reliability of the judgment and eliminates serious accidents caused by the failure or false alarm of a single sensor.

[0190] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Figure 4 As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement any of the hierarchical security intervention methods described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0191] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0192] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0193] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0194] This invention also provides a computer-readable medium. The computer-readable medium stores a computer program, which, when executed by a processor, implements the steps of any of the graded security intervention methods described in the above embodiments. The computer-readable storage medium can be volatile or non-volatile.

[0195] This invention also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described graded security intervention method.

[0196] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0197] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0198] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0199] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0200] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0201] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0202] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0203] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0204] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0205] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A graded safety intervention method, characterized in that, It includes: Acquire multimodal signals; The fatigue index is calculated based on the multimodal signals. The intervention level is determined based on the fatigue index and the scene adaptation engine. Implement tiered intervention measures based on the intervention level.

2. The graded safety intervention method according to claim 1, characterized in that, The steps of acquiring multimodal signals include extracting visual modal features and extracting capacitive modal features.

3. The graded safety intervention method according to claim 2, characterized in that, The steps for extracting visual modal features include: inputting an infrared camera video stream, using the MediaPipe facial mesh model to track 468 facial feature points in real time, and then outputting PERCLOS and yawn frequency; The steps for extracting capacitive modal features include: inputting the original signal from the steering wheel capacitive sensing array, decomposing the original signal using wavelet transform, filtering out high-frequency noise caused by hand sweat and minor movements, reconstructing a clean signal, extracting low-frequency components from the filtered capacitive signal as an EDA signal, and calculating its mean or standard deviation as feature values; extracting periodic fluctuations synchronized with the heartbeat from the capacitive signal, obtaining the RR interval sequence through a peak detection algorithm, and then calculating time-domain or frequency-domain features, normalizing them into a comprehensive HRV stress index.

4. The graded safety intervention method according to claim 3, characterized in that, The step of calculating the fatigue index based on the multimodal signal includes: Submodal score normalization: All extracted original features are normalized by min-max normalization or Z-score and mapped to the [0,1] interval, where 1 represents the most fatigued state. The original features include PERCLOS, yawn frequency, EDA and HRV. Intramodal weighted fusion: Define visual modality score V = 0.7 * normalization (PERCLOS) + 0.3 * normalization (yawn frequency), and physiological modality score P = 0.5 * normalization (EDA) + 0.5 * normalization (HRV); Intermodal weighted fusion: Define the fatigue index F = α * V + β * P, where α = 0.6 and β = 0.4; Confidence check and intervention trigger: The F-value is calculated in real time, and the intervention process is triggered if and only if F > 0.

8.

5. The graded safety intervention method according to claim 4, characterized in that, The step of determining the intervention level based on the fatigue index and in conjunction with the scene adaptation engine includes: Receive the real-time fatigue index F; The F value is directly mapped to intervention levels L1-L4; where: F=0.81 is mapped to intervention level L1, F=0.85 is mapped to intervention level L2, F=0.90 is mapped to intervention level L3, and F=0.95 is mapped to intervention level L4. Based on the scenario, the above decisions are adapted as follows: if the vehicle speed is >80km / h: skip L1 and go directly to L2; if it is raining: disable window lowering and start the air conditioning external circulation.

6. The graded safety intervention method according to claim 5, characterized in that, The step of implementing graded intervention measures according to the intervention level includes: L1 intervention: Releases eucalyptus oil or other invigorating aromas; L2 intervention: 3Hz lumbar support pressure, 5cm window reduction and voice prompts; L3 intervention: After multiple attempts to wake up the user failed and no abnormalities were found in the user's physiological signs, L2.9 level autonomous driving takeover was activated and the user was navigated to the nearest service area to stop. L4 intervention: Detects if the driver is unconscious, enables autonomous parking, and provides ECALL assistance.

7. The graded safety intervention method according to claim 6, characterized in that, Detecting whether a driver is unconscious includes: detecting whether the driver is unconscious through capacitive signals, and determining whether the driver is unconscious when there is zero grip strength and a sudden drop in skin conductance for 10 seconds; and detecting whether the driver is unconscious through visual signals, and determining whether the driver is unconscious when there is pupil dilation and no blinking reflex.

8. A graded safety intervention system, characterized in that, include: The acquisition unit is used to acquire multimodal signals; A calculation unit is used to calculate the fatigue index based on the multimodal signal; An intervention unit is used to determine the intervention level based on the fatigue index and in conjunction with the scene adaptation engine. An execution unit is used to execute graded intervention measures according to the intervention level.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images