Vehicle-machine multi-source fusion driver state monitoring and intelligent intervention system

Abstract

The invention discloses an in-vehicle multi-source fusion driver state monitoring and intelligent intervention system, which relates to the field of safe driving and comprises a hardware sensing layer, a software processing layer, an early warning intervention layer and a cloud collaboration layer. The hardware sensing layer is used for collecting facial features, physiological data, operation behavior data and environmental parameter data of a driver; the software processing layer is used for carrying out preprocessing, multi-source fusion calculation, state judgment and risk level evaluation on the collected data; the early warning and intervention layer is used for triggering graded early warning and intervention response based on the risk grade; and the cloud collaboration layer is used for model iteration updating, abnormal data storage and remote assistance triggering.

Description

Technical Field

[0001] This application relates to the field of safe driving, and in particular to a vehicle-machine multi-source fusion driver status monitoring and intelligent intervention system. Background Technology

[0002] Existing vehicle-mounted driver monitoring technology has the following core shortcomings:

[0003] Single monitoring dimension: Most systems rely on a single camera to collect facial data, which can only identify basic fatigue characteristics such as closed eyes and yawning, and cannot cover multi-dimensional risk states such as "hands off the steering wheel", "eyes off the road", and "abnormal heart rate".

[0004] Low data fusion: The data does not combine onboard sensors (such as steering wheel pressure sensors and vehicle speed sensors) with physiological data (such as heart rate and blood oxygen), resulting in a high misjudgment rate (such as misjudging a driver's normal blinking as fatigue).

[0005] Delayed early warning and intervention: Only voice prompts are used to issue single warnings, without matching dynamic intervention measures according to the risk level (e.g., only a reminder for mild fatigue, and a forced reduction in speed for severe fatigue).

[0006] Poor adaptability: The monitoring accuracy is greatly reduced due to the influence of light (such as backlight, night) and driver's clothing (such as glasses, masks), and no environmental adaptive adjustment mechanism has been formed.

[0007] Existing technologies cannot achieve a closed loop of "multi-dimensional perception - accurate judgment - graded intervention", making it difficult to fundamentally avoid safety risks caused by abnormal driver conditions. Summary of the Invention

[0008] The purpose of this invention is to provide a vehicle-machine multi-source fusion driver status monitoring and intelligent intervention system, which solves at least one of several technical problems.

[0009] Current technologies have limited monitoring dimensions, relying solely on a single camera to collect basic facial features, failing to cover multiple driving risks such as hands leaving the steering wheel, abnormal heart rate, and abnormal posture. They also lack integration of vehicle sensor data and physiological data, resulting in low data fusion and high misjudgment rates (e.g., misjudging normal blinking as fatigue). Early warning and intervention methods are simplistic and delayed, failing to match dynamic intervention measures to risk levels and thus unable to specifically mitigate different levels of safety risks. Furthermore, they are susceptible to variations in lighting (backlight, nighttime) and driver attire (glasses, masks), leading to poor monitoring adaptability and significantly reduced accuracy. Finally, the lack of a complete closed loop of "multi-dimensional perception - accurate judgment - tiered intervention - cloud optimization" makes it difficult to fundamentally prevent safety hazards caused by abnormal driver conditions.

[0010] This invention provides the following solution:

[0011] According to a first aspect of the present invention, a vehicle-mounted multi-source fusion driver state monitoring and intelligent intervention system is provided, comprising:

[0012] Hardware perception layer, software processing layer, early warning and intervention layer, and cloud collaboration layer;

[0013] The hardware perception layer is used to collect driver facial features, physiological data, operational behavior data, and environmental parameter data.

[0014] The software processing layer is used for preprocessing the collected data, multi-source fusion calculation, status determination, and risk level assessment.

[0015] The early warning and intervention layer is used to trigger tiered early warning and intervention responses based on risk levels.

[0016] The cloud-based collaboration layer is used for model iteration and updates, abnormal data storage, and remote assistance triggering.

[0017] Furthermore, including:

[0018] The hardware perception layer includes an in-vehicle perception module, a physiological monitoring module, an operational behavior module, and an environmental parameter module;

[0019] The in-vehicle perception module includes a binocular camera and an infrared camera. The binocular camera is used to collect facial features such as blink frequency, pupil diameter, and gaze direction, while the infrared camera is used for supplemental lighting in night / backlight environments and to eliminate interference from clothing.

[0020] The physiological monitoring module includes a steering wheel-integrated photoelectric sensor and a seat pressure sensor. The photoelectric sensor is used to collect heart rate and blood oxygen data, and the seat pressure sensor is used to determine whether the driver's sitting posture is abnormal.

[0021] The operation behavior module includes a steering wheel pressure sensor, a steering angle sensor, and a pedal travel sensor. The steering wheel pressure sensor uses a preset pressure as a standard to determine if the hands are off the steering wheel. The steering angle sensor uses a preset rotation frequency as a standard to identify distraction behavior. The pedal travel sensor is used to monitor the frequency of emergency braking / acceleration.

[0022] The environmental parameter module includes an in-vehicle light sensor and a vehicle speed sensor. The light sensor is used to adjust the camera's exposure parameters, and the vehicle speed sensor is used to determine the risk level based on the vehicle speed.

[0023] Furthermore, including:

[0024] The in-vehicle light sensor works in conjunction with the binocular camera and infrared camera to dynamically adjust the camera exposure parameters and perform monitoring in all weather conditions and in all wearable scenarios.

[0025] Furthermore, including:

[0026] The software processing layer includes a data preprocessing unit, a multi-source data fusion unit, a status determination model unit, and a risk level assessment unit;

[0027] The data preprocessing unit is used to filter sensor noise data and standardize data formats.

[0028] The multi-source data fusion unit uses a weighted fusion algorithm to perform fusion calculations based on the following weights: facial features 40%, physiological data 30%, operational behavior 20%, and environmental parameters 10%, and outputs a comprehensive feature value.

[0029] The state determination model unit is a deep learning model trained on a preset number of abnormal driver state samples to determine fatigue state, distraction state or abnormal state.

[0030] The risk level assessment unit matches the risk level to mild, moderate, or severe based on the condition type and vehicle speed.

[0031] Furthermore, including:

[0032] The decision rules for the state determination model unit are as follows:

[0033] Fatigue state: blinking frequency < 5 times / minute + pupil diameter > 6mm + heart rate < 60 beats / minute;

[0034] Distracted state: eyes off the road for more than 3 seconds + hands off the steering wheel for more than 2 seconds + frequent corrections to the steering angle;

[0035] Abnormal condition: Heart rate >120 beats / minute or <50 beats / minute + abnormal sitting posture with seat pressure distribution deviation >30%.

[0036] Furthermore, including:

[0037] The assessment rules for the risk level assessment unit are as follows:

[0038] Mild risk: Continuous visual deviation for 1-3 seconds at speeds below 40 km / h;

[0039] Moderate risk: At speeds of 40-80 km / h, the initial stage of fatigue is characterized by a blinking frequency of 5-8 times per minute and a sustained gaze deviation of 3-5 seconds;

[0040] Severe risk: severe fatigue at speeds greater than 80 km / h, blinking frequency less than 5 times / minute, abnormal physiological state, and continuous gaze deviation for more than 5 seconds.

[0041] Furthermore, including:

[0042] The standardization process of the data preprocessing unit includes unifying data timestamps and units of measurement, and the noise filtering of the multi-source data fusion unit includes removing instantaneous fluctuation data of steering wheel pressure.

[0043] Furthermore, including:

[0044] The early warning and intervention layer includes a primary early warning module, a secondary early warning module, and a tertiary intervention module, each corresponding to a risk level.

[0045] The Level 1 warning module includes a voice prompt with a preset warning message, a flashing green light on the dashboard, and a single-frequency slight vibration of the seat.

[0046] The Level 2 warning module includes a second preset warning message with enhanced voice prompts, a constantly lit yellow light on the dashboard, continuous seat vibration at 3 frequencies, and automatic reduction of music volume;

[0047] The Level 3 intervention module includes a third preset warning voice emergency prompt, a flashing red light on the dashboard, automatic search for the nearest service area and push navigation, limiting the vehicle speed to a maximum of 60km / h, and sending the location to preset emergency contacts.

[0048] Furthermore, including:

[0049] Based on user-preset authorization, the navigation push, speed limit, and emergency contact location sending functions of the three-level intervention module are activated.

[0050] Furthermore, including:

[0051] The cloud-based collaboration layer includes a model iteration and update unit, an anomaly data storage unit, and a remote assistance triggering unit;

[0052] The model iteration and update unit is used to collect abnormal status data of multiple vehicles and pushes out model update packages every preset period.

[0053] An abnormal data storage unit is used to de-identify and back up driver abnormal status data, allowing users to view historical records;

[0054] The remote assistance trigger unit is used to trigger a connection to cloud-based customer service to contact rescue if a severe risk persists for a preset time without being alleviated.

[0055] The above solution achieves the following beneficial technical effects:

[0056] This application utilizes a four-dimensional system of "facial features + physiological data + operational behavior + environmental parameters" to cover more than 10 types of risk states, completely solving the problem of single-dimensional monitoring in traditional technologies and achieving multi-dimensional comprehensive monitoring.

[0057] This application significantly reduces the false positive rate and improves the accuracy of judgment by using a weighted fusion algorithm (with clear weight allocation) and a deep learning model trained on 100,000+ samples, combined with multi-source data collaborative judgment.

[0058] This application triggers a three-level early warning intervention based on the linkage between risk level and vehicle speed, ranging from minor alerts to mandatory speed limits and navigation push notifications, achieving "risk-adaptive response" and solving the problem of delayed intervention. The tiered intervention is more targeted.

[0059] This application utilizes a dual-camera system combining a binocular camera and an infrared camera, along with dynamic parameter adjustment via a light sensor, to adapt to nighttime, backlighting, and scenarios where glasses / masks are worn. This significantly improves adaptability by 80% compared to traditional technologies.

[0060] This application achieves monthly model iterations through cloud collaboration, anonymized storage of abnormal data, and remote assistance for severe risks, continuously optimizing system performance, fundamentally avoiding driving safety risks, and forming a complete closed-loop management system.

[0061] This application requires users to pre-authorize functions such as vehicle speed limits and location transmission in the three-level intervention process, balancing safety intervention with users' autonomy and control, and ensuring users' right to know and right to choose. Attached Figure Description

[0062] Figure 1 This is a structural diagram of a vehicle-machine multi-source fusion driver status monitoring and intelligent intervention system provided by one or more embodiments of the present invention.

[0063] Figure 2 This is a schematic diagram of the vehicle-machine multi-source fusion driver status monitoring and intelligent intervention system architecture provided in a specific embodiment of the present invention. Detailed Implementation

[0064] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are one module of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] Figure 1 This is a structural diagram of a vehicle-machine multi-source fusion driver status monitoring and intelligent intervention system provided by one or more embodiments of the present invention.

[0066] like Figure 1 The vehicle-mounted multi-source fusion driver status monitoring and intelligent intervention system shown includes:

[0067] Hardware perception layer, software processing layer, early warning and intervention layer, and cloud collaboration layer;

[0068] The hardware perception layer is used to collect driver facial features, physiological data, operational behavior data, and environmental parameter data.

[0069] The software processing layer is used for preprocessing the collected data, multi-source fusion calculation, status determination, and risk level assessment.

[0070] The early warning and intervention layer is used to trigger tiered early warning and intervention responses based on risk levels.

[0071] The cloud-based collaboration layer is used for model iteration and updates, abnormal data storage, and remote assistance triggering.

[0072] In this embodiment, it includes:

[0073] The hardware perception layer includes an in-vehicle perception module, a physiological monitoring module, an operational behavior module, and an environmental parameter module;

[0074] The in-vehicle perception module includes a binocular camera and an infrared camera. The binocular camera is used to collect facial features such as blink frequency, pupil diameter, and gaze direction, while the infrared camera is used for supplemental lighting in night / backlight environments and to eliminate interference from clothing.

[0075] The physiological monitoring module includes a steering wheel-integrated photoelectric sensor and a seat pressure sensor. The photoelectric sensor is used to collect heart rate and blood oxygen data, and the seat pressure sensor is used to determine whether the driver's sitting posture is abnormal.

[0076] The operation behavior module includes a steering wheel pressure sensor, a steering angle sensor, and a pedal travel sensor. The steering wheel pressure sensor uses a preset pressure as a standard to determine if the hands are off the steering wheel. The steering angle sensor uses a preset rotation frequency as a standard to identify distraction behavior. The pedal travel sensor is used to monitor the frequency of emergency braking / acceleration.

[0077] The environmental parameter module includes an in-vehicle light sensor and a vehicle speed sensor. The light sensor is used to adjust the camera's exposure parameters, and the vehicle speed sensor is used to determine the risk level based on the vehicle speed.

[0078] In this embodiment, it includes:

[0079] The in-vehicle light sensor works in conjunction with the binocular camera and infrared camera to dynamically adjust the camera exposure parameters and perform monitoring in all weather conditions and in all wearable scenarios.

[0080] In this embodiment, it includes:

[0081] The software processing layer includes a data preprocessing unit, a multi-source data fusion unit, a status determination model unit, and a risk level assessment unit;

[0082] The data preprocessing unit is used to filter sensor noise data and standardize data formats.

[0083] The multi-source data fusion unit uses a weighted fusion algorithm to perform fusion calculations based on the following weights: facial features 40%, physiological data 30%, operational behavior 20%, and environmental parameters 10%, and outputs a comprehensive feature value.

[0084] The state determination model unit is a deep learning model trained on a preset number of abnormal driver state samples to determine fatigue state, distraction state or abnormal state.

[0085] The risk level assessment unit matches the risk level to mild, moderate, or severe based on the condition type and vehicle speed.

[0086] In this embodiment, it includes:

[0087] The decision rules for the state determination model unit are as follows:

[0088] Fatigue state: blinking frequency < 5 times / minute + pupil diameter > 6mm + heart rate < 60 beats / minute;

[0089] Distracted state: eyes off the road for more than 3 seconds + hands off the steering wheel for more than 2 seconds + frequent corrections to the steering angle;

[0090] Abnormal condition: Heart rate >120 beats / minute or <50 beats / minute + abnormal sitting posture with seat pressure distribution deviation >30%.

[0091] In this embodiment, it includes:

[0092] The assessment rules for the risk level assessment unit are as follows:

[0093] Mild risk: Continuous visual deviation for 1-3 seconds at speeds below 40 km / h;

[0094] Moderate risk: At speeds of 40-80 km / h, the initial stage of fatigue is characterized by a blinking frequency of 5-8 times per minute and a sustained gaze deviation of 3-5 seconds;

[0095] Severe risk: severe fatigue at speeds greater than 80 km / h, blinking frequency less than 5 times / minute, abnormal physiological state, and continuous gaze deviation for more than 5 seconds.

[0096] In this embodiment, it includes:

[0097] The standardization process of the data preprocessing unit includes unifying data timestamps and units of measurement, and the noise filtering of the multi-source data fusion unit includes removing instantaneous fluctuation data of steering wheel pressure.

[0098] In this embodiment, it includes:

[0099] The early warning and intervention layer includes a primary early warning module, a secondary early warning module, and a tertiary intervention module, each corresponding to a risk level.

[0100] The Level 1 warning module includes a voice prompt with a preset warning message, a flashing green light on the dashboard, and a single-frequency slight vibration of the seat.

[0101] The Level 2 warning module includes a second preset warning message with enhanced voice prompts, a constantly lit yellow light on the dashboard, continuous seat vibration at 3 frequencies, and automatic reduction of music volume;

[0102] The Level 3 intervention module includes a third preset warning voice emergency prompt, a flashing red light on the dashboard, automatic search for the nearest service area and push navigation, limiting the vehicle speed to a maximum of 60km / h, and sending the location to preset emergency contacts.

[0103] In this embodiment, it includes:

[0104] Based on user-preset authorization, the navigation push, speed limit, and emergency contact location sending functions of the three-level intervention module are activated.

[0105] In this embodiment, it includes:

[0106] The cloud-based collaboration layer includes a model iteration and update unit, an anomaly data storage unit, and a remote assistance triggering unit;

[0107] The model iteration and update unit is used to collect abnormal status data of multiple vehicles and pushes out model update packages every preset period.

[0108] An abnormal data storage unit is used to de-identify and back up driver abnormal status data, allowing users to view historical records;

[0109] The remote assistance trigger unit is used to trigger a connection to cloud-based customer service to contact rescue if a severe risk persists for a preset time without being alleviated.

[0110] Specifically, in an embodiment of a vehicle-machine multi-source fusion driver state monitoring and intelligent intervention method, the method includes:

[0111] Step S1, Multi-dimensional data collection: Through the in-vehicle perception module, physiological monitoring module, operation behavior module and environmental parameter module of the hardware perception layer, the driver's facial features, physiological data, operation behavior data and vehicle environmental parameters are collected simultaneously.

[0112] Step S2, Data Preprocessing and Fusion: After noise filtering and format standardization of the collected multi-dimensional data, a weighted fusion algorithm is used to calculate the comprehensive feature value;

[0113] Step S3, Status Determination and Risk Assessment: Based on a deep learning model, determine the driver's fatigue, distraction or abnormal state according to the comprehensive feature value, and match the mild, moderate or severe risk level with the vehicle speed.

[0114] Step S4, Tiered Early Warning and Intervention: Trigger corresponding early warning and intervention measures according to the risk level, and link with the vehicle infotainment system and vehicle control system to execute early warning and / or risk avoidance actions;

[0115] Step S5, Cloud Collaborative Optimization: Upload abnormal state data to the cloud for model iteration and data backup, and trigger remote assistance warnings or / and risk avoidance actions when the severe risk persists for a preset period without relief.

[0116] The facial features described in step S1 include those collected by a binocular camera inside the vehicle cabin, including blink frequency, pupil diameter, and gaze direction; supplemental lighting is provided by an infrared camera in nighttime or backlight conditions to eliminate interference from the driver wearing glasses or a mask; physiological data include heart rate and blood oxygen data collected by an integrated photoelectric sensor in the steering wheel, and abnormal sitting posture detected by a seat pressure sensor; operational behavior data includes hand detachment detection by a steering wheel pressure sensor, distraction detection by a steering angle sensor, and emergency braking / acceleration frequency monitored by a pedal travel sensor; environmental parameters include light intensity collected by an in-vehicle light sensor and vehicle speed data collected by a vehicle speed sensor.

[0117] In step S2, noise filtering includes removing instantaneous fluctuation data of steering wheel pressure, and format standardization includes unifying data timestamps and units of measurement. The weight allocation of the weighted fusion algorithm is as follows: facial features account for 40%, physiological data accounts for 30%, operational behavior data accounts for 20%, and environmental parameters account for 10%.

[0118] In step S3, the deep learning model is trained based on 100,000+ abnormal driver state samples. The fatigue state judgment rule is blinking frequency < 5 times / minute + pupil diameter > 6mm + heart rate < 60 beats / minute. The distraction state judgment rule is gaze deviation from the road for > 3 seconds + hand off the steering wheel for > 2 seconds + frequent steering angle correction. The abnormal state judgment rule is heart rate > 120 beats / minute or < 50 beats / minute + seat pressure distribution deviation > 30%. The risk level matching rule is: short-term distraction (gaze deviation for 1-3 seconds) at low speed < 40km / h is mild risk; initial fatigue or continuous distraction (gaze deviation for 3-5 seconds) at medium speed 40-80km / h is moderate risk; and severe fatigue, abnormal physiological state or distraction > 5 seconds at high speed > 80km / h is severe risk.

[0119] In step S4, a mild risk triggers a voice prompt + flashing green light on the dashboard + single-frequency seat vibration; a moderate risk triggers an enhanced voice prompt + a constantly lit yellow light on the dashboard + three-frequency seat vibrations + automatic reduction of music volume; a severe risk triggers an emergency voice prompt + flashing red light on the dashboard + navigation push to the nearest service area / hospital + speed limit to a maximum of 60km / h + sending location to a preset emergency contact. The navigation push, speed limit, and location sending functions for severe risks must be based on user-preset authorization.

[0120] In step S5, the cloud pushes a model update package monthly to optimize the local judgment model; abnormal data is backed up after being desensitized and made available for users to view historical records; if a severe risk persists for 10 seconds without being alleviated, the cloud customer service is automatically connected to assist in contacting rescue.

[0121] It is worth noting that although this system / device only discloses the above-mentioned modules / units, it does not mean that this system / device is limited to the above-mentioned basic functional modules. On the contrary, what this invention intends to express is that, based on the above-mentioned basic functional modules, those skilled in the art can add one or more functional modules in combination with the prior art to form an infinite number of embodiments or technical solutions. That is to say, this system is open rather than closed. It cannot be assumed that the scope of protection of the claims of this invention is limited to the above-disclosed basic functional modules just because this embodiment only discloses a few basic functional modules.

[0122] In one specific embodiment, a method as follows is disclosed Figure 2 The diagram shows the architecture of a vehicle-machine multi-source fusion driver status monitoring and intelligent intervention system.

[0123] The system architecture includes a hardware perception layer, a software processing layer, an early warning and intervention layer, and a cloud collaboration layer. The system block diagram is shown above, and the functions and connections of each module are also shown above.

[0124] Hardware perception layer (data acquisition end, communicating with the vehicle's main control chip via CAN bus / USB):

[0125] In-vehicle sensing module: binocular camera in the cabin (collects facial features: blink frequency, pupil diameter, gaze direction), infrared camera (supplementary lighting in night / backlight environments, and eliminates interference from clothing);

[0126] Physiological monitoring module: Steering wheel integrated photoelectric sensor (collects heart rate and blood oxygen), seat pressure sensor (determines whether the driver's sitting posture is abnormal);

[0127] Operation behavior module: Steering wheel pressure sensor (detects whether the hand is off the steering wheel, pressure value <5N is judged as off the wheel), steering angle sensor (identifies frequent small angle correction behavior, correction times >10 times within 1 minute are judged as distraction), pedal travel sensor (monitors the frequency of emergency braking / acceleration).

[0128] Environmental parameter module: in-vehicle light sensor (adjusts camera exposure parameters), vehicle speed sensor (determines risk based on vehicle speed, such as "line of sight deviation > 3 seconds" when driving at high speed, indicating an increased risk level).

[0129] Software processing layer (core of local data processing in the vehicle):

[0130] Data preprocessing unit: filters sensor noise data (such as instantaneous fluctuations in steering wheel pressure) and standardizes data format (unified timestamp and unit);

[0131] Multi-source data fusion unit: Employs a weighted fusion algorithm to fuse and calculate "facial features (weight 40%) + physiological data (weight 30%) + operational behavior (weight 20%) + environmental parameters (weight 10%)", and outputs a comprehensive feature value;

[0132] State determination model unit: Based on a deep learning model (training data includes 100,000+ driver abnormal state samples), determining state type:

[0133] Fatigue state: blinking frequency < 5 times / minute + pupil diameter > 6mm + heart rate < 60 beats / minute;

[0134] Distracted state: eyes off the road for more than 3 seconds + hands off the steering wheel for more than 2 seconds + frequent corrections to the steering angle;

[0135] Abnormal condition: Heart rate >120 beats / minute or <50 beats / minute + abnormal sitting posture (seat pressure distribution deviation >30%).

[0136] Risk level assessment unit: Risk level is matched based on condition type and vehicle speed.

[0137] Mild risk (Level 1): Brief distraction at low speeds (<40km / h) (view of sight deviates for 1-3 seconds);

[0138] Moderate risk (Level 2): ​​Initial stage of fatigue at medium speed (40-80km / h) (blinking frequency 5-8 times / minute), continuous distraction (eyes deviating from focus for 3-5 seconds);

[0139] Severe risk (Level 3): Severe fatigue at high speed (>80km / h) (blinking frequency <5 times / minute), abnormal physiological state, distraction >5 seconds.

[0140] Early warning and intervention layer (triggered response based on risk level, linked with vehicle infotainment and vehicle control systems):

[0141] Level 1 warning module: voice prompt ("Please pay attention") + dashboard light flashing (green) + slight seat vibration (single frequency);

[0142] Level 2 warning module: enhanced voice prompt ("Fatigue detected, it is recommended to rest nearby") + constant yellow dashboard light + continuous seat vibration (3 times) + automatic music volume reduction;

[0143] Level 3 intervention module: voice emergency prompt ("Severe risk detected, safety intervention will be triggered") + flashing red light on the dashboard + automatic search for the nearest service area and push navigation from the vehicle system + speed limit (maximum speed reduced to 60km / h) + sending location to emergency contacts (user-preset authorization required).

[0144] Cloud-based collaboration layer (communicating with the vehicle's infotainment system via 5G / V2X):

[0145] Model Iteration and Update Unit: Collects abnormal status data of multiple vehicles and optimizes the local judgment model (monthly update package pushed);

[0146] Abnormal data storage unit: Backs up driver abnormal status data (de-identified) for users to view historical records;

[0147] Remote assistance trigger unit: If a severe risk persists for 10 seconds without relief, it will automatically connect to cloud-based customer service to assist in contacting rescue.

[0148] System Flow:

[0149] Example 1: High-speed driving fatigue monitoring (severe risk)

[0150] Hardware sensing: The vehicle speed sensor detects a vehicle speed of 90km / h, the binocular camera captures "blinking frequency of 4 times / minute and pupil diameter of 7mm", and the steering wheel pressure sensor detects "hand detachment for 3 seconds";

[0151] Software processing: The data fusion unit calculates the comprehensive feature value (fatigue features account for 65%), the state determination model determines it as "severe fatigue", and the risk level assessment is level three;

[0152] Early warning intervention: Triggering the three-level intervention module - emergency voice prompt + flashing red light on the dashboard + automatic search for service areas within 5km and push navigation to the vehicle system + limiting vehicle speed to 60km / h + sending location to preset emergency contacts;

[0153] Cloud-based collaboration: The abnormal data is stored in the cloud, and a "Highway Fatigue Driving Prevention" reminder will be pushed out subsequently.

[0154] Example 2: Monitoring of Distraction Status on Urban Roads (Medium Risk)

[0155] Hardware perception: The vehicle speed sensor detects a vehicle speed of 50km / h, the binocular camera captures "the line of sight deviates from the road for 4 seconds", and the steering angle sensor detects "12 corrections within 1 minute";

[0156] Software processing: The comprehensive feature value shows that the distraction feature accounts for 55%, which is judged as "moderate distraction", with a risk level of level 2;

[0157] Warning intervention: Triggering the Level 2 warning module - enhanced voice prompt ("Please focus on driving and reduce eye deviance") + constant yellow light on the instrument panel + seat vibration 3 times + automatic reduction of music volume to 30%.

[0158] Example 3: Monitoring of Abnormal Physiological States During Nighttime Driving (Severe Risk)

[0159] Hardware sensing: The ambient light sensor detects a light intensity of 20 lux (nighttime), the infrared camera is activated, the steering wheel photoelectric sensor collects "heart rate 130 beats / minute", and the seat pressure sensor detects "sitting posture deviation 40%";

[0160] Software processing: The comprehensive feature value shows that the proportion of abnormal physiological characteristics is 70%, which is judged as "abnormal state" with a risk level of three;

[0161] Warning and intervention: Triggering the three-level intervention module - voice prompt "Abnormal heart rate detected, it is recommended to stop immediately" + red light flashing on the dashboard + automatic search for the nearest hospital and push navigation + limit vehicle speed to 40km / h + cloud customer service proactively calls the vehicle's phone to inquire about the situation.

[0162] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the meaning consistent with their meaning in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined.

[0163] For the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0164] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, or the module that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or certain modules of the embodiments of this application.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to the modules or all technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A vehicle-mounted multi-source fusion driver status monitoring and intelligent intervention system, characterized in that, include: Hardware perception layer, software processing layer, early warning and intervention layer, and cloud collaboration layer; The hardware perception layer is used to collect driver facial features, physiological data, operational behavior data, and environmental parameter data; The software processing layer is used for preprocessing the collected data, multi-source fusion calculation, status determination, and risk level assessment. The early warning and intervention layer is used to trigger graded early warning and intervention responses based on risk levels; The cloud collaboration layer is used for model iteration and updates, abnormal data storage, and remote assistance triggering.

2. The vehicle-machine multi-source fusion driver status monitoring and intelligent intervention system according to claim 1, characterized in that, include: The hardware perception layer includes an in-vehicle perception module, a physiological monitoring module, an operational behavior module, and an environmental parameter module. The in-vehicle perception module includes a binocular camera and an infrared camera. The binocular camera is used to collect facial features such as blink frequency, pupil diameter, and gaze direction. The infrared camera is used for supplemental lighting in night / backlight environments and to eliminate interference from clothing. The physiological monitoring module includes a steering wheel-integrated photoelectric sensor and a seat pressure sensor. The photoelectric sensor is used to collect heart rate and blood oxygen data, and the seat pressure sensor is used to determine whether the driver's sitting posture is abnormal. The operation behavior module includes a steering wheel pressure sensor, a steering angle sensor, and a pedal travel sensor. The steering wheel pressure sensor uses a preset pressure as a standard to determine if the hand leaves the steering wheel. The steering angle sensor uses a preset rotation frequency as a standard to identify distraction behavior. The pedal travel sensor is used to monitor the frequency of emergency braking / acceleration. The environmental parameter module includes an in-vehicle light sensor and a vehicle speed sensor. The light sensor is used to adjust the camera exposure parameters, and the vehicle speed sensor is used to determine the risk level in conjunction with the vehicle speed.

3. The vehicle-machine multi-source fusion driver status monitoring and intelligent intervention system according to claim 1, characterized in that, include: The software processing layer includes a data preprocessing unit, a multi-source data fusion unit, a state determination model unit, and a risk level assessment unit. The data preprocessing unit is used to filter sensor noise data and standardize data formats; The multi-source data fusion unit uses a weighted fusion algorithm to perform fusion calculations based on the following weights: facial features (40%), physiological data (30%), operational behavior (20%), and environmental parameters (10%), and outputs a comprehensive feature value. The state determination model unit is a deep learning model trained on a preset number of abnormal driver state samples to determine fatigue state, distraction state or abnormal state. The risk level assessment unit matches mild risk, moderate risk, or severe risk based on the state type and vehicle speed.

4. The vehicle-machine multi-source fusion driver status monitoring and intelligent intervention system according to claim 3, characterized in that, include: The determination rule of the state determination model unit is as follows: Fatigue state: blinking frequency < 5 times / minute + pupil diameter > 6mm + heart rate < 60 beats / minute; Distracted state: eyes off the road for more than 3 seconds + hands off the steering wheel for more than 2 seconds + frequent corrections to the steering angle; Abnormal condition: Heart rate >120 beats / minute or <50 beats / minute + abnormal sitting posture with seat pressure distribution deviation >30%.

5. The vehicle-machine multi-source fusion driver status monitoring and intelligent intervention system according to claim 3, characterized in that, include: The assessment rules for the risk level assessment unit are as follows: Mild risk: Continuous visual deviation for 1-3 seconds at speeds below 40 km / h; Moderate risk: At speeds of 40-80 km / h, the initial stage of fatigue is characterized by a blinking frequency of 5-8 times per minute and a sustained gaze deviation of 3-5 seconds; Severe risk: severe fatigue at speeds greater than 80 km / h, blinking frequency less than 5 times / minute, abnormal physiological state, and continuous gaze deviation for more than 5 seconds.

6. The vehicle-machine multi-source fusion driver status monitoring and intelligent intervention system according to claim 3, characterized in that, include: The standardization process of the data preprocessing unit includes unifying data timestamps and units of measurement, and the noise filtering of the multi-source data fusion unit includes removing instantaneous fluctuation data of steering wheel pressure.

7. The vehicle-machine multi-source fusion driver status monitoring and intelligent intervention system according to claim 1, characterized in that, include: The early warning and intervention layer includes a first-level early warning module, a second-level early warning module, and a third-level intervention module, each corresponding to a risk level. The first-level warning module includes a voice prompt with a first preset warning message, a flashing green light on the dashboard, and a single-frequency slight vibration of the seat. The secondary warning module includes a second preset warning message with enhanced voice prompts, a constantly lit yellow light on the dashboard, continuous seat vibration at 3 frequencies, and automatic reduction of music volume. The three-level intervention module includes a third preset warning voice emergency prompt, a flashing red light on the dashboard, automatic search for the nearest service area and push navigation, limiting the vehicle speed to a maximum of 60km / h, and sending the location to preset emergency contacts.

8. The vehicle-machine multi-source fusion driver status monitoring and intelligent intervention system according to claim 7, characterized in that, include: Based on user-preset authorization, the navigation push, speed limit, and emergency contact location sending functions of the three-level intervention module are activated.

9. The vehicle-machine multi-source fusion driver status monitoring and intelligent intervention system according to claim 1, characterized in that, include: The cloud collaboration layer includes a model iteration and update unit, an abnormal data storage unit, and a remote assistance triggering unit. The model iterative update unit is used to collect abnormal status data of multiple vehicles and push out model update packages in each preset period. The abnormal data storage unit is used to de-identify and back up driver abnormal status data for users to view historical records; The remote assistance triggering unit is used to trigger a connection to cloud-based customer service to assist in contacting rescue when a severe risk persists for a preset period of time without being alleviated.

10. A method for multi-source fusion driver status monitoring and intelligent intervention in a vehicle, characterized in that, include: Step S1, Multi-dimensional data collection: Through the in-vehicle perception module, physiological monitoring module, operation behavior module and environmental parameter module of the hardware perception layer, the driver's facial features, physiological data, operation behavior data and vehicle environmental parameters are collected simultaneously. Step S2, Data Preprocessing and Fusion: After noise filtering and format standardization of the collected multi-dimensional data, a weighted fusion algorithm is used to calculate the comprehensive feature value; Step S3, Status Determination and Risk Assessment: Based on a deep learning model, determine the driver's fatigue, distraction or abnormal state according to the comprehensive feature value, and match the mild, moderate or severe risk level with the vehicle speed. Step S4, Tiered Early Warning and Intervention: Trigger corresponding early warning and intervention measures according to the risk level, and link with the vehicle infotainment system and vehicle control system to execute early warning and / or risk avoidance actions; Step S5, Cloud Collaborative Optimization: Upload abnormal state data to the cloud for model iteration and data backup, and trigger remote assistance warnings or / and risk avoidance actions when the severe risk persists for a preset period without relief.

Images