Security sensing method and device, electronic equipment and storage medium

Abstract

The invention provides a safety perception method and device, electronic equipment and a storage medium, and the method comprises the steps: determining an environment risk level based on the out-cabin environment information, and adjusting in-cabin monitoring parameters based on the environment risk level, thereby enabling the sensitivity of in-cabin monitoring to be matched with the real risk of the external environment; meanwhile, the driver state level is determined based on the in-cabin state information, and the out-cabin control parameters are adjusted based on the driver state level, so that the out-cabin sensing strategy can make up the possible insufficient sensing ability when the driver state is poor; the mismatching condition between the driver attention and the external environment risk can be accurately recognized, the problems that in-cabin and out-cabin information isolation causes safety strategy solidification and cannot adapt to a dynamic driving scene in a traditional scheme are solved, the accuracy and timeliness of safety perception are improved, and the driving safety is improved. And the driving safety in the intelligent driving scene is ensured.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, and in particular to a safety perception method, device, electronic device, and storage medium. Background Technology

[0002] With the rapid development of intelligent driving technology, vehicle active safety systems have evolved from single-function safety protection to intelligent protection systems that integrate multiple sensors. Current active safety systems mainly fall into two categories: one focuses on technologies for perceiving the external environment, such as millimeter-wave radar, lidar, and cameras for identifying vehicles, pedestrians, and obstacles; the other focuses on technologies for monitoring the driver's condition inside the cabin, such as vision-based fatigue detection and distraction warnings. However, in existing technological architectures, the in-cabin and external systems often operate independently, forming two "information silos" that lack deep information interaction and collaborative decision-making capabilities.

[0003] In real-world driving scenarios, the external environment of a vehicle is closely related to the driver's internal state. For example, when a vehicle is traveling at high speed or in a complex traffic environment, even a brief moment of distraction by the driver, such as looking down at a mobile phone, can pose a significant safety risk. Conversely, when the system detects that the driver is fatigued or distracted, it should promptly increase its sensitivity to the surrounding environment to compensate for the risks that may arise from human error.

[0004] However, traditional systems lack this two-way collaborative capability, cannot dynamically adjust cabin monitoring strategies based on external environmental risks, and cannot utilize driver status information to optimize external perception algorithms, thus resulting in deficiencies in the timeliness and accuracy of safety decisions. Summary of the Invention

[0005] This invention provides a safety perception method, device, electronic device, and storage medium to address the shortcomings of existing traditional systems in lacking this two-way collaborative capability, being unable to dynamically adjust cabin monitoring strategies based on external environmental risks, and being unable to optimize external perception algorithms using driver status information, thus resulting in deficiencies in the timeliness and accuracy of safety decisions.

[0006] This invention provides a security sensing method, comprising the following steps: Acquire information about the external environment of the vehicle and information about the internal state of the vehicle that characterizes the state of the driver inside the vehicle; The environmental risk level is determined based on the external environmental information, and the monitoring parameters inside the cabin are adjusted according to the environmental risk level. The pilot's status level is determined based on the cabin status information, and the external control parameters are adjusted according to the pilot's status level. The system acquires in-cabin monitoring results and out-of-cabin perception results. The in-cabin monitoring results are collected by the monitoring equipment inside the vehicle based on the adjusted in-cabin monitoring parameters, and the out-of-cabin perception results are collected by the perception equipment outside the vehicle based on the out-of-cabin control parameters.

[0007] According to a safety perception method provided by the present invention, determining the environmental risk level based on the external environment information includes: The external environment information is analyzed to obtain vehicle speed data, traffic flow density data, environmental weather data, and road complexity data; The environmental risk level is determined based on the vehicle speed data, traffic flow density data, environmental weather data, and road complexity data.

[0008] According to a safety perception method provided by the present invention, determining the pilot's state level based on the cabin state information includes: The cabin status information is parsed to obtain at least two of the following: driver behavior data, driver facial status data, driver hand status data, body posture data, and cabin passenger behavior data. Multimodal fusion analysis is performed on at least two of the driver behavior data, driver facial state data, driver hand state data, body posture data, and cabin passenger behavior data to obtain a comprehensive driver state model that characterizes the driver's current state. Based on the comprehensive driver state model, the driver state level is determined.

[0009] According to a safety perception method provided by the present invention, the step of performing multimodal fusion analysis on at least two of the driver behavior data, driver facial state data, driver hand state data, body posture data, and cabin passenger behavior data to obtain a comprehensive driver state model characterizing the driver's current state includes: The driver behavior data is analyzed to obtain driver behavior analysis results; Visual monitoring is performed on the driver's facial state data to obtain visual monitoring results; The driver's hand status data is subjected to a hands-off detection to obtain the hands-off detection result; The body posture data is subjected to posture monitoring to obtain abnormal movement recognition results; Interactive behavior detection is performed on the passenger behavior data in the cabin to obtain interactive behavior detection results; The comprehensive driver state model is determined based on at least two of the driver behavior analysis results, the visual monitoring results, the hands-off detection results, the abnormal action recognition results, and the interactive behavior detection results.

[0010] According to a safety perception method provided by the present invention, the step of visually monitoring the driver's facial state data to obtain a visual monitoring result includes: Visual distraction monitoring is performed on the driver's facial state data to obtain visual distraction monitoring results; The driver's facial state data was used to monitor fatigue state, and fatigue state monitoring results were obtained. The visual monitoring results are determined based on the visual distraction detection results and / or the fatigue state monitoring results.

[0011] According to a safety perception method provided by the present invention, the step of monitoring the driver's facial state data for fatigue state to obtain fatigue state monitoring results includes: Based on the eye opening and closing state, the percentage of eyelid closure time per unit time, blinking frequency, and gaze direction corresponding to a preset number of frames in the driver's facial state data, fatigue state monitoring is performed to obtain the fatigue state monitoring results.

[0012] According to a safety perception method provided by the present invention, after acquiring the in-cabin monitoring results and the external perception results, the method further includes: The vehicle is controlled based on the in-cabin monitoring results and the external perception results.

[0013] According to a safety perception method provided by the present invention, controlling the vehicle based on the in-cabin monitoring results and the external perception results includes: Based on the external sensing results, an environmental risk distribution map is constructed to characterize the distribution of external environmental risks. Based on the cabin monitoring results, an attention distribution map is constructed to characterize the driver's gaze points. The environmental risk distribution map and the attention distribution map are compared and analyzed to determine the attention mismatch areas; Generate attention guidance instructions, and control the vehicle based on the attention guidance instructions; the attention guidance instructions are used to guide the driver's attention to the attention mismatch area.

[0014] The present invention also provides a security sensing device, comprising the following units: The first acquisition unit is used to acquire information about the external environment of the vehicle and information about the internal state of the vehicle that characterizes the state of the driver inside the vehicle. The first determining unit is used to determine the environmental risk level based on the external environment information, so as to adjust the internal monitoring parameters according to the environmental risk level. The second determining unit is used to determine the pilot's status level based on the cabin status information, so as to adjust the external control parameters according to the pilot's status level. The second acquisition unit is used to acquire in-cabin monitoring results and external perception results. The in-cabin monitoring results are collected by the monitoring equipment inside the vehicle based on the adjusted in-cabin monitoring parameters, and the external perception results are collected by the perception equipment outside the vehicle based on the external control parameters.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the security awareness methods described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the security awareness method as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the security awareness method as described above.

[0018] The safety perception method, device, electronic equipment, and storage medium provided by this invention determine the environmental risk level based on external environmental information and adjust the in-cabin monitoring parameters based on the environmental risk level, thereby matching the sensitivity of in-cabin monitoring with the actual risks of the external environment. Simultaneously, by determining the driver's state level based on in-cabin state information and adjusting external control parameters based on the driver's state level, the external perception strategy can compensate for the insufficient perception capabilities that may occur when the driver is in poor condition. Furthermore, by obtaining the bidirectional adjusted in-cabin monitoring results and external perception results, it can accurately identify mismatches between driver attention and external environmental risks. This solves the problem of traditional solutions where the isolation of in-cabin and external information leads to rigid safety strategies and an inability to adapt to dynamic driving scenarios, improving the accuracy and timeliness of safety perception and ensuring driving safety in intelligent driving scenarios. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is one of the flowcharts of the security perception method provided by the present invention.

[0021] Figure 2 This is a schematic diagram of the dynamic sensitivity adjustment of the external information-assisted internal monitoring provided by the present invention.

[0022] Figure 3 This is a schematic diagram of the extravehicular information-assisted multidimensional monitoring inside the cabin provided by the present invention.

[0023] Figure 4 This is a schematic diagram of the in-cabin distraction state transmission and external perception enhancement provided by the present invention.

[0024] Figure 5 This is a schematic diagram of visual attention-guided external perception and in-cabin early warning provided by the present invention.

[0025] Figure 6 This is the second flowchart of the security perception method provided by the present invention.

[0026] Figure 7 This is a structural schematic diagram of the safety sensing device provided by the present invention.

[0027] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] In related technologies, although intelligent driving systems and intelligent cockpit systems are developing rapidly in their respective fields, they are still two independent "information islands". This separation limits the further improvement of their performance and brings potential safety risks.

[0030] Intelligent driving systems can already perceive the external environment through multi-sensor fusion, and even achieve advanced functions such as automatic following, lane keeping, and automatic lane changing in specific scenarios. However, because their initial design focuses primarily on external environment perception and vehicle control, they generally lack the ability to perceive and understand the real-time state of the driver and passengers inside the vehicle. This makes the intelligent driving system like a "driver with excellent vision but no sense of cues": it may be able to see obstacles 100 meters away, but it cannot know whether the person in the driver's seat is tired, distracted, or even in a state of semi-sleep. When the system encounters an extreme scenario that it cannot handle and prompts for takeover, if the driver fails to respond in time due to fatigue or other reasons, the risk of an accident increases sharply. For example, in an accident involving intelligent driving on the XX Expressway, although the system issued a takeover warning, the driver failed to react in time. At the same time, because it cannot know the driver's state, such as whether a novice driver is nervous or an experienced driver is relaxed, the intelligent driving system also finds it difficult to personalize its driving strategy, such as following distance and the degree of aggressiveness in lane changes, making it difficult to find the best balance between ensuring safety and improving the experience.

[0031] Meanwhile, with advancements in AI technology, intelligent cockpit systems are increasingly capable of enhanced in-cabin perception, utilizing built-in cameras, microphone arrays, and biosensors to achieve driver identification, fatigue monitoring, distraction warnings, and even emotion detection. However, this valuable in-cabin data is often used for in-cabin interaction, entertainment, or basic safety alerts, without deep, real-time integration with the intelligent driving system. This makes the cockpit system resemble a "co-pilot who is keenly aware but powerless to intervene": it may accurately detect every yawn or glance from the driver, but it struggles to directly drive the vehicle to take substantive safety measures. For example, when the system determines the driver is fatigued, it may at most issue an alarm, but it cannot proactively coordinate with the intelligent driving system to increase following distance, plan earlier arrival at service areas, or trigger seat vibrations or adjust the air conditioning temperature to refresh the driver. Furthermore, because the in-cabin system lacks information about complex external road conditions, its warning and interaction strategies may lack contextual relevance, leading to false alarms, missed warnings, or disturbances to the driver, potentially distracting their attention.

[0032] This "cabin-driver separation" architecture means that the intelligent driving system lacks consideration of the "human" core factor when making decisions. The rich in-cabin perception data of the intelligent cockpit system cannot empower vehicle control to achieve higher-level active safety, resulting in resource waste and functional bottlenecks. It is difficult to cope with truly complex hybrid interaction scenarios and also restricts the upper limit of overall driving safety and personalized comfort experience.

[0033] To address the aforementioned problems, this invention provides a safety perception method. This method establishes a two-way interaction and coordination mechanism between the driver's in-cabin state information and the external environment information, breaking down the barriers of isolation between in-cabin and external information processing in traditional solutions. This enables dynamic and collaborative perception and control of the vehicle's safety status. This method can be applied to intelligent connected vehicles, autonomous vehicles, or vehicles equipped with Advanced Driver-Assistance Systems (ADAS). The method can be executed by an onboard computing unit, such as an intelligent cockpit domain controller, an autonomous driving domain controller, or a central computing unit. Figure 1 This is one of the flowcharts illustrating the security sensing method provided by the present invention. Figure 2 This is a schematic diagram of the dynamic sensitivity adjustment of the extravehicular information-assisted intravehicular monitoring provided by the present invention, as shown in the figure. Figure 1 , Figure 2 As shown, the method includes steps 110, 120, 130 and 140.

[0034] Step 110: Obtain information about the external environment of the vehicle and information about the internal state of the vehicle that characterizes the driver's state.

[0035] Specifically, firstly, it can acquire information about the external environment of the vehicle and information about the internal state of the vehicle, which characterizes the driver's state inside the vehicle.

[0036] Here, external environment information is used to reflect the external world conditions in which the vehicle is located while driving. External environment information may include image or video data collected by onboard visual sensors, such as forward-facing cameras and surround-view cameras; point cloud data or target list data collected by millimeter-wave radar and LiDAR (Light Detection and Ranging), vehicle status data transmitted via the Controller Area Network (CAN) bus or Ethernet, weather data obtained from external weather services or roadside units through onboard communication modules, such as V2X (Vehicle-to-Everything) modules, and information such as vehicle geographical location, road curvature, and slope obtained from GPS and high-precision map services. This embodiment of the invention does not specifically limit this information.

[0037] Here, the image or video data collected by the vehicle-mounted vision sensor can be used to identify lane type, road grade, obstacles, traffic signs, pedestrians, other vehicles, etc., and the embodiments of the present invention do not specifically limit this.

[0038] Here, point cloud data or target list data collected by millimeter-wave radar and lidar can be used to detect surrounding obstacle information, vehicle density, relative speed, distance and azimuth angle, etc., and the embodiments of the present invention do not specifically limit this.

[0039] Here, the vehicle's own status data transmitted by the vehicle controller local area network bus or Ethernet, such as current vehicle speed, acceleration, yaw rate, turn signal status, etc., are not specifically limited in this embodiment of the invention.

[0040] Here, weather data such as sunny, rainy, snowy, foggy, and real-time traffic flow data are obtained from external weather services or roadside units via the vehicle communication module.

[0041] Among them, cabin status information refers to various types of data that can reflect the physiological, behavioral, and intentional states of the pilot.

[0042] Here, the cabin status information may include driver behavior data, driver facial status data, driver hand status data, body posture data, and cabin passenger behavior data, etc., and the embodiments of the present invention do not specifically limit this.

[0043] It should be noted that acquiring information about the external environment of the vehicle and information about the internal state of the driver can be performed periodically by the onboard computing unit, for example, every 100 milliseconds, or it can be triggered by a specific event, such as when the vehicle starts or enters a specific driving mode. The acquisition methods include, but are not limited to, actively collecting, receiving, or reading data from various sensors or data buses within the vehicle.

[0044] Step 120: Determine the environmental risk level based on the external environmental information, and adjust the internal monitoring parameters according to the environmental risk level.

[0045] Specifically, after obtaining information about the external environment, the environmental risk level can be determined based on this information, and the monitoring parameters inside the cabin can be adjusted accordingly. This step enables unidirectional adjustment from outside to inside the cabin, allowing the sensitivity of the internal monitoring system to match the level of danger in the external environment.

[0046] Here, environmental risk level is an indicator used to quantitatively characterize the degree of danger of the external environment in which a vehicle is currently located. It can be a continuous numerical value; for example, the environmental risk level can be an Environmental Risk Coefficient (ERC) ranging from 0 to 1.0, where 0 represents the lowest risk environment, such as an open parking lot, and 1.0 represents the highest risk environment, such as a high-speed curve with heavy traffic in severe weather. Alternatively, environmental risk level can also be a discrete classification, such as low-risk, medium-risk, and high-risk environments.

[0047] The process of determining the environmental risk level based on external environmental information involves the comprehensive analysis and processing of acquired multi-source external environmental information. This can be achieved through a pre-defined rule engine or a trained machine learning model. For example, the machine learning model takes various types of external environmental information as input and outputs the environmental risk level. For instance, when the system detects a high vehicle speed and rainy weather, it can determine the current environmental risk level as high-risk.

[0048] Here, cabin monitoring parameters refer to the various thresholds or configuration parameters used by the Driver Monitoring System (DMS) to make algorithmic judgments. These parameters directly affect the sensitivity and rigor of the system's judgment of the driver's status.

[0049] Here, the cabin monitoring parameters may include time threshold parameters, state threshold parameters, and sensor configuration parameters, etc., and the embodiments of the present invention do not specifically limit them.

[0050] Among these, time threshold parameters could be the maximum allowable duration for which the driver's gaze is diverted from the road, constituting visual distraction, and the maximum allowable duration for which the driver's hands are removed from the steering wheel, constituting hands-off driving. State threshold parameters could be the percentage of eyelid closure time required to determine if the driver is in a fatigued state, and the magnitude of body deviation from the center position required to determine if the driver is in an abnormal posture. Sensor configuration parameters could be the image acquisition frame rate and image resolution of the DMS camera, but this embodiment of the invention does not specifically limit these parameters.

[0051] Adjusting cabin monitoring parameters based on environmental risk levels refers to establishing a mapping relationship between environmental risk levels and cabin monitoring parameters. Generally, the higher the environmental risk level, the more stringent the cabin monitoring parameter settings should be to achieve more sensitive monitoring. Vehicle speed signal is one of the key parameters for sensitivity adjustment. When the vehicle speed > 10 km / h, the system automatically enters active monitoring mode, dynamically adjusting the sensitivity thresholds for hands-off and eyes-off detection based on the ERC value. For example, in a low-risk environment (ERC < 0.3), the threshold allowing the driver's gaze to deviate from the road can be set to a longer 2.0 seconds, and the steering wheel hands-off detection threshold can be set to 3 consecutive seconds without detected grip force. When the environmental risk level rises to a medium-risk environment (0.3 ≤ ERC < 0.7), such as on urban expressways, this threshold can be automatically adjusted to a shorter 1.2 seconds, and the steering wheel hands-off detection threshold can be adjusted to 2 consecutive seconds without detected grip force. When the environmental risk level is high (ERC≥0.7), such as on highways and accompanied by rain or snow, the threshold can be further shortened to 0.8 seconds, the steering wheel off-hand detection is changed to zero tolerance mode (i.e., a level one warning is triggered as soon as the hands are released), and the sampling frequency of the DMS camera can be moderately increased to 30fps to ensure that more subtle changes in the driver's state can be captured.

[0052] Through this dynamic adjustment, the method of this embodiment of the invention ensures that in truly dangerous scenarios, the system can monitor the driver with the highest sensitivity and detect potential risks in a timely manner; while in safe scenarios, the monitoring is appropriately relaxed to reduce excessive interference with the driver and improve the driving experience.

[0053] Step 130: Determine the pilot's status level based on the cabin status information, and adjust the external control parameters according to the pilot's status level.

[0054] Specifically, after obtaining the cabin status information, the driver's status level can be determined based on this information, and the external control parameters can be adjusted accordingly. This step achieves reverse adjustment from inside the cabin to outside, enabling the external perception system's capabilities to adaptively enhance based on the driver's actual state.

[0055] Here, the driver state level is an indicator used to quantitatively characterize the driver's current level of attention and fatigue. Similar to the environmental risk level, the driver state level can be a continuous numerical value or a discrete level. For example, the driver state level can be attentive, slightly distracted, severely fatigued, unconscious, etc. This embodiment of the invention does not specifically limit this.

[0056] The process of determining the pilot's condition level based on cabin status information can be achieved by analyzing one or more of the pilot's vital signs. For example, the pilot's fatigue level can be determined by analyzing the pilot's eyelid closure and yawning frequency; or the pilot's distraction level can be determined by analyzing the pilot's gaze direction and hand position.

[0057] Here, external control parameters refer to the various algorithm parameters used in the external perception and decision-making system. These parameters affect the vehicle's perception range, perception accuracy, and the conservatism of subsequent decisions regarding the external environment.

[0058] Here, the external control parameters may include sensing algorithm parameters, sensor operating parameters, decision planning parameters, etc., and the embodiments of the present invention do not specifically limit them.

[0059] The perception algorithm parameters can be the confidence threshold of the target detection algorithm. Lowering the confidence threshold means that the system is more sensitive to the identification of potential targets; even small or distant objects with low confidence may be reported as obstacles. The sensor operating parameters can be the scanning mode of the lidar or the detection range mode of the millimeter-wave radar. The decision planning parameters can be the safe distance threshold, risk tolerance, etc., when performing path planning; this embodiment of the invention does not specifically limit these parameters.

[0060] Adjusting external control parameters based on the driver's condition level refers to establishing a mapping relationship between the driver's condition level and the external control parameters. Generally, the worse the driver's condition level, such as being more fatigued or distracted, the more conservative and sensitive the external control parameters should be, in order to compensate for any possible loss of perception and reaction ability by the driver.

[0061] For example, when the driver's condition is determined to be attentive, the external perception system can operate with standard parameter configurations. When the driver's condition is determined to be mildly distracted, such as when the driver briefly looks away from the road to look at the central control screen, the system can appropriately enhance its ability to detect obstacles in front and to the sides, for example, by lowering the confidence threshold for target detection from 0.9 to 0.8. When the driver's condition is determined to be severely fatigued, such as when the driver closes their eyes for an extended period or their head is continuously drooping, the system can activate a defensive perception mode, significantly reducing various detection thresholds, expanding the perception range, and adopting a more conservative decision-making strategy to prepare for possible emergency takeover by the system.

[0062] Through this adjustment, embodiments of the present invention can dynamically increase the perception and decision-making weights of the system based on the driver's condition. When the system determines that the driver's poor condition leads to an increased level of operational risk, it will automatically enhance the monitoring and intervention of the assistance system, thereby providing important functional redundancy for vehicle safety.

[0063] Step 140: Obtain the in-cabin monitoring results and the external sensing results. The in-cabin monitoring results are collected by the monitoring equipment inside the vehicle based on the adjusted in-cabin monitoring parameters, and the external sensing results are collected by the sensing equipment outside the vehicle based on the external control parameters.

[0064] Specifically, the system acquires both in-cabin monitoring results and external perception results. The in-cabin monitoring results are collected by monitoring equipment within the vehicle, controlled based on adjusted in-cabin monitoring parameters. The external perception results are collected by external perception equipment, controlled based on external control parameters. In other words, the external perception results are the latest perceptions of the external environment under adjusted, more sensitive, or standardized external control parameters, such as a rapidly approaching vehicle to the right rear of the vehicle.

[0065] In one embodiment, if external perception indicates a potential collision risk to the left front of the vehicle, and internal monitoring shows that the driver's attention is also focused on the left front, no command may need to be generated. However, if external perception indicates a potential collision risk to the left front of the vehicle, but internal monitoring shows the driver is turning their head to look out the right window, this constitutes an attention mismatch. In this case, the system can generate an attention guidance command to control the vehicle.

[0066] Among them, attention guidance instructions can be implemented by overlaying a bright visual symbol on the left front risk area through the instrument panel, central control screen, or augmented reality head-up display (AR-HUD), or by emitting sound prompts from the left front through directional acoustic technology, in order to guide the driver's attention back to the source of risk.

[0067] In addition, risk warning commands and vehicle assistance control commands can be generated. Risk warning commands are warnings issued through sound, light, electricity, or seat vibration that are more urgent or prominent than regular collision warnings. Vehicle assistance control commands, in some advanced autonomous driving systems, can even trigger a slight assisted steering or braking to avoid danger if the system determines that the driver cannot respond in time.

[0068] The method provided in this invention determines the environmental risk level based on external environmental information and adjusts the in-cabin monitoring parameters based on the environmental risk level, thereby matching the sensitivity of in-cabin monitoring with the actual risks of the external environment. Simultaneously, by determining the driver's state level based on in-cabin state information and adjusting external control parameters based on the driver's state level, the external perception strategy can compensate for the insufficient perception capabilities that may occur when the driver is in a poor state. Furthermore, by obtaining the bidirectional adjusted in-cabin monitoring and external perception results, it can accurately identify the mismatch between driver attention and external environmental risks. This solves the problem of traditional solutions where the isolation of in-cabin and external information leads to rigid safety strategies and an inability to adapt to dynamic driving scenarios, improving the accuracy and timeliness of safety perception and ensuring driving safety in intelligent driving scenarios.

[0069] Based on the above embodiments, step 120, which involves determining the environmental risk level based on the external environment information, includes: Step 121: Analyze the external environment information to obtain vehicle speed data, traffic flow density data, environmental weather data, and road complexity data; Step 122: Determine the environmental risk level based on the vehicle speed data, the traffic flow density data, the environmental weather data, and the road complexity data.

[0070] Specifically, firstly, the external environmental information can be analyzed to obtain vehicle speed data, traffic flow density data, environmental weather data, and road complexity data.

[0071] Among them, vehicle speed data can be directly obtained from the vehicle's CAN bus or Ethernet, and is the most direct parameter reflecting the vehicle's dynamic risks. Vehicle speed data may include vehicle speed, acceleration, and yaw angle, etc., and this embodiment of the invention does not specifically limit this.

[0072] Traffic flow density data can be obtained through visual cameras, millimeter-wave radar, and lidar, or through visual cameras and millimeter-wave radar, or through visual cameras and lidar. This embodiment of the invention does not specifically limit the specific methods used.

[0073] For example, visual algorithms can be used to identify and count the number of vehicles in the lanes ahead and to the sides, or radar data can be used to count the number of targets within a unit distance (e.g., 100 meters ahead). When there are more than three vehicles per 100 meters on a highway, the traffic flow density is considered high. Furthermore, traffic flow density data can also include the identification and counting of pedestrians, non-motorized vehicles, and other road users.

[0074] Here, environmental weather data can be obtained through multi-sensor fusion. For example, visual cameras can identify rain and snow by analyzing image contrast, brightness, and raindrop patterns; millimeter-wave radar can determine precipitation intensity by analyzing the strength of echo signals; at the same time, data from vehicle-mounted rain sensors or forecast information obtained from external weather services via the network can be fused to form a comprehensive judgment of weather conditions.

[0075] Here, road complexity data can be obtained through visual cameras, millimeter-wave radar, and lidar; alternatively, it can be obtained through visual cameras and millimeter-wave radar, or visual cameras and lidar. This embodiment of the invention does not specifically limit the method used. For example, high-precision maps can directly provide information such as the current road's classification (e.g., highway, urban road, rural road), curvature, and slope. Visual cameras can identify lane line types in real time, such as solid lines and dashed lines, assisting in determining the constraint strength of the driving scenario. For instance, when a high-precision map warns of a sharp bend 200-300 meters ahead, the road complexity will increase accordingly.

[0076] Secondly, based on vehicle speed data, traffic flow density data, environmental weather data, and road complexity data, the environmental risk level is determined. For example, the vehicle speed data, traffic flow density data, environmental weather data, and road complexity data can be normalized to obtain their respective risk coefficients, for example, ranging from 0 to 1. Then, a weighted sum is performed, as follows: Environmental Risk Rating (ERC) = w1 × Vehicle Speed ​​Data + w2 × Traffic Flow Density Data + w3 × Environmental Weather Data + w4 × Road Complexity Data; Where w1, w2, w3, and w4 are the weights of each item, and their sum is 1. Here, the weights of each item can be set to w1=0.3, w2=0.2, w3=0.2, and w4=0.3.

[0077] Understandably, the speed coefficient is positively correlated with the speed data, meaning the higher the speed data, the larger the speed coefficient; the traffic flow density coefficient is positively correlated with the number of surrounding traffic participants, meaning the more vehicles and pedestrians there are, the larger the traffic flow density coefficient; the weather coefficient is positively correlated with the severity of weather, such as heavy rain or snow, the weather coefficient increases accordingly; and the road complexity coefficient is positively correlated with the degree of road curvature, grade, and lane line constraint strength, the more complex the road, the higher the road complexity coefficient value.

[0078] The method provided in this invention analyzes external environmental information into vehicle speed data, traffic flow density data, environmental weather data, and road complexity data. Based on this structured data, a comprehensive calculation is performed to determine the environmental risk level. This transforms the environmental risk assessment process from a subjective and vague qualitative judgment to an objective and precise quantitative assessment. This approach addresses the problem that traditional methods often rely on a single information source or general judgments, leading to inaccurate and unstable risk level assessments, thus improving the accuracy and objectivity of risk level assessments.

[0079] Based on the above embodiments, step 130, which involves determining the pilot's state level based on the cabin state information, includes: Step 131: Analyze the cabin status information to obtain at least two of the following: driver behavior data, driver facial status data, driver hand status data, body posture data, and cabin passenger behavior data. Step 132: Perform multimodal fusion analysis on at least two of the driver behavior data, driver facial state data, driver hand state data, body posture data, and cabin passenger behavior data to obtain a comprehensive driver state model that characterizes the driver's current state. Step 133: Determine the driver state level based on the comprehensive driver state model.

[0080] Specifically, firstly, the cabin status information is analyzed to obtain at least two of the following: driver behavior data, driver facial status data, driver hand status data, body posture data, and cabin passenger behavior data.

[0081] Here, driver behavior data refers to data collected by DMS cameras and processed by behavior recognition algorithms to detect whether the driver is performing specific actions that affect safe driving, such as smoking, talking on a phone, or drinking water.

[0082] Here, the driver's facial state data is obtained through the DMS camera in the A-pillar of the cabin, and through facial key point detection, head posture estimation and gaze tracking algorithms, the driver's gaze direction, head turning angle, eyelid opening and closing degree, blinking frequency, yawning state and other data are obtained.

[0083] The driver's hand status data can be obtained through two or more methods. One method is to use an in-cabin camera and a hand joint recognition algorithm to determine whether the hand is located in the steering wheel area; another method is to use a capacitive or pressure sensor built into the steering wheel to directly detect the contact and grip strength of the hand.

[0084] Body posture data is obtained by using a wide-angle camera (OMS) or a DMS camera and a monocular 3D human posture estimation algorithm to reconstruct the driver's skeletal model, thereby monitoring for abnormal postures, such as excessive turning to the side to pick up items or frequently turning around to communicate with rear passengers.

[0085] Monitoring the behavior of front passenger and rear passengers is an important extension of multi-dimensional monitoring. The system uses an OMS wide-angle camera to detect interactions between passengers and the driver, i.e., in-cabin passenger behavior data, such as physical contact, frequent conversations, or passing items. If "excessive interaction" is detected (such as a passenger continuously obstructing the driver's view or prolonged physical contact), the system will trigger a level two warning (audio and visual alert) and record the event data for subsequent risk analysis. This function is particularly suitable for family travel scenarios, where children or pets may distract the driver.

[0086] Secondly, multimodal fusion analysis is performed on at least two of the driver behavior data, driver facial state data, driver hand state data, body posture data, and cabin passenger behavior data to obtain a comprehensive driver state model that represents the driver's current state.

[0087] The core of multi-dimensional monitoring lies in information fusion and collaborative decision-making. Data generated from each monitoring dimension undergoes feature-level and decision-level fusion, combined with external environmental risk assessment, to ultimately generate early warnings using different human body monitoring signals. The entire multi-dimensional monitoring system works closely with external environmental perception to achieve a risk-matching intelligent monitoring strategy, greatly improving the accuracy and reliability of driver status monitoring.

[0088] Here, the comprehensive driver state model is the output of multimodal fusion analysis. It is not a single numerical value or level, but a comprehensive model that can fully describe the driver's physiological state, attention state, and behavioral intentions. For example, the comprehensive driver state model can include multiple sub-dimensions such as fatigue index, distraction index, behavioral risk index, and takeover ability index.

[0089] Finally, based on the comprehensive driver state model, the driver state level is determined. For example, rules can be set such that when the fatigue index in the comprehensive model exceeds threshold A, or the distraction index exceeds threshold B, the driver state level is determined to be severely distracted / fatigued; when all indices are within the normal range, it is determined to be attentive.

[0090] The method provided in this invention analyzes at least two of the driver behavior data, facial state data, hand state data, body posture data, and cabin passenger behavior data, and performs multimodal fusion analysis to construct a comprehensive driver state model. This enables a more comprehensive assessment of the driver's state, captures complex distraction or fatigue states that cannot be effectively identified by single-dimensional monitoring, and improves the accuracy and robustness of driver state level determination.

[0091] Based on the above embodiments, step 132 includes: Step 1321: Perform driving behavior analysis on the driver behavior data to obtain driver behavior analysis results; Step 1322: Visually monitor the driver's facial state data to obtain visual monitoring results; Step 1323: Perform a hands-off detection on the driver's hand status data to obtain the hands-off detection result; Step 1324: Perform posture monitoring on the body posture data to obtain abnormal action recognition results; Step 1325: Perform interactive behavior detection on the in-cabin passenger behavior data to obtain interactive behavior detection results; Step 1326: Determine the comprehensive driver state model based on at least two of the driver behavior analysis results, the visual monitoring results, the hands-off detection results, the abnormal action recognition results, and the interactive behavior detection results.

[0092] Specifically, Figure 3 This is a schematic diagram of the extravehicular information-assisted multidimensional monitoring inside the cabin provided by the present invention, as shown below. Figure 3 As shown, driver behavior data refers to data collected by DMS cameras and processed by behavior recognition algorithms to detect whether the driver is performing specific actions that affect safe driving, such as smoking, talking on a phone, or drinking water. Driver behavior data is then analyzed to obtain driver behavior analysis results. For example, if an image recognition model detects that the driver is holding a mobile phone close to their ear, the driver behavior analysis result is that they are making a phone call, and a higher risk score is assigned. Similarly, detecting smoking or drinking water will also output corresponding behavior analysis results.

[0093] Here, driver facial status data is obtained through a DMS camera located on the A-pillar inside the cabin. This data is processed using facial landmark detection, head pose estimation, and eye-tracking algorithms to acquire information such as the driver's gaze direction, head tilt angle, eyelid opening and closing, blinking frequency, and yawning status. Visual monitoring of the driver's facial status data yields visual monitoring results. These results can include whether the driver's gaze is on the road and whether there are signs of fatigue; in other words, visual monitoring can include detection of eye distraction. For example, if the driver's gaze is detected to be fixed on the central control screen for an extended period, such as more than 1.2 seconds, a result of visual distraction is output; if the eyelid closure time exceeds 40% within a unit of time, a result of moderate fatigue is output.

[0094] The system performs hand-off detection on driver hand position data to obtain the hand-off detection result. For example, by fusing camera visual detection and steering wheel capacitance / pressure sensor data, if no hand is visually visible on the steering wheel and the sensor does not detect a grip force signal for two consecutive seconds, the hand-off detection result is confirmed. This multimodal fusion approach can effectively handle complex scenarios such as drivers lightly gripping the steering wheel or obstructed light, improving detection accuracy.

[0095] Body posture data is monitored to obtain abnormal movement identification results. For example, by analyzing the driver's three-dimensional posture skeleton model, if a tilt of more than 30 degrees or a large backward rotation of the upper body is detected, the abnormal movement identification result is that there is an abnormal body movement.

[0096] Interaction behavior detection is performed on passenger behavior data in the cabin to obtain interaction behavior detection results. For example, if the OMS camera detects continuous physical contact between the front passenger and the driver, or the front passenger passing items to the driver, the interaction behavior detection result is that excessive interaction exists.

[0097] Finally, based on at least two results from driver behavior analysis, eye state monitoring, hands-off detection, abnormal action recognition, and interactive behavior detection, a comprehensive driver state model is constructed using a data fusion algorithm. Specifically, a quantitative scoring mechanism can be designed to map each detection result to a corresponding risk score. Typical detection indicators include behaviors such as holding a phone, eye deviation, and hands off the steering wheel. The system calculates a comprehensive risk score using weighted summation or averaging. The comprehensive risk score and its corresponding sub-detection results together constitute the output of the comprehensive driver state model, used to accurately characterize the driver's real-time risk level. For example, if the system simultaneously detects eye deviation and hands off the steering wheel, the model will output a comprehensive score representing an extremely high-risk state.

[0098] Based on the above embodiments, step 1322 includes: Step 1322-1: Perform visual distraction monitoring on the driver's facial state data to obtain the visual distraction monitoring results; Step 1322-2: Perform fatigue state monitoring on the driver's facial state data to obtain fatigue state monitoring results; Step 1322-3: Determine the visual monitoring result based on the visual distraction detection monitoring result and / or the fatigue state monitoring result.

[0099] Specifically, Figure 4 This is a schematic diagram of the in-cabin distraction state transmission and external perception enhancement provided by the present invention, as shown below. Figure 4 As shown, in-cabin driver status information plays a crucial role in enhancing the perception of the external environment, especially in the co-driving mode of the autonomous driving system. This system establishes a distraction status transmission channel, transmitting the driver's status in real time to the environmental perception module of the autonomous driving system. This allows the autonomous driving system to intelligently adjust its perception strategies and decision-making mechanisms based on the driver's status. When the system detects driver distraction, fatigue, or emotional fluctuations, it automatically enhances the perception of the external environment, increases safety redundancy, and prepares for potential intervention delays.

[0100] Distraction state quantification and transmission are achieved through multimodal sensor fusion. The system calculates the distraction level using driver facial data. The distraction state is encoded into standardized data packets, which are transmitted to the environmental perception module of the autonomous driving system via CAN bus or Ethernet. The data packets contain fields such as the distraction level, including short-term distraction, long-term distraction, and unconsciousness, as well as estimated recovery time and confidence level. Upon receiving this information, the autonomous driving system adjusts its perception and decision-making strategies.

[0101] First, visual distraction monitoring is performed on the driver's facial expression data to obtain the visual distraction monitoring results. The core of visual distraction monitoring is to determine whether the driver's gaze is focused on the area related to the driving task.

[0102] Here, visual distraction monitoring can include eye tracking and distraction judgment. Eye tracking can be achieved by capturing images of the driver's eyes through a DMS camera, using eye-tracking algorithms to accurately calculate the orientation of the gaze in three-dimensional space, and determining the gaze point inside or outside the vehicle, such as whether it is on the road in front, the rearview mirror, the central control screen, or the side window.

[0103] For visual distraction (such as prolonged deviance of the driver's gaze from the road), the autonomous driving system enhances its ability to detect obstacles in front and to the sides, increasing the frequency and range of perception. The system temporarily allocates more computing resources to visual perception tasks, lowers the detection confidence threshold, and increases sensitivity. When the distraction level is prolonged or an unconscious state, the system activates the following enhancement strategies: the confidence thresholds for LiDAR point cloud and millimeter-wave radar detection are appropriately lowered to enhance the ability to detect small obstacles at a distance (such as tire debris hundreds of meters away). Simultaneously, the system activates predictive algorithms to estimate the time required for the driver to regain attention and formulates risk control strategies based on this estimate. If the estimated recovery time exceeds a safe threshold, the system further enhances its perception capabilities and may even prepare to take over vehicle control.

[0104] Here, distraction detection compares the driver's gaze point with a predefined safe driving area, such as the lower part of the windshield. If the driver's gaze is detected to linger in an unsafe driving area for an extended period, such as continuously staring at the center console screen for more than a preset time threshold, the visual distraction detection result is determined to indicate the presence of visual distraction.

[0105] Secondly, fatigue monitoring was performed on the driver's facial expression data to obtain fatigue monitoring results. Fatigue monitoring aims to identify changes in physiological indicators caused by driver drowsiness.

[0106] Fatigue monitoring can include physiological feature extraction and fatigue assessment. Physiological feature extraction involves extracting key fatigue-related features from the driver's facial data, such as the degree of eyelid opening and closing, blinking frequency per unit time, and whether the head posture shows drooping or swaying. This embodiment of the invention does not specifically limit these features.

[0107] Fatigue assessment is based on the changing trends of one or more of the above characteristics to determine the degree of fatigue. For example, if the driver's eye closure time percentage increases continuously, or slow eye movements lasting more than 0.5 seconds occur frequently, or yawning more than 3 times per minute is detected, then the fatigue monitoring results are judged to indicate the presence of fatigue.

[0108] The perception enhancement mechanism is specifically implemented at both the sensor and algorithm levels. At the sensor level, the system dynamically adjusts the operating modes and parameters of each sensor. For example, when driver distraction is detected, the algorithm lowers the confidence threshold for target detection, enabling the system to detect smaller and more distant obstacles; at the decision-making level, it reduces risk tolerance and adopts a more conservative decision-making strategy. A multi-hypothesis tracking algorithm is introduced into the target tracking module, generating multiple trajectory hypotheses for each detected target to avoid missed or false tracking due to driver distraction. Simultaneously, the vision recognition module lowers the classification threshold, such as adjusting the pedestrian recognition confidence from 0.9 to 0.7, ensuring that potential risks are not filtered out.

[0109] It's important to note that while lowering the detection threshold and increasing detection sensitivity enhances the vehicle's perception capabilities, it doesn't affect driving control. Instead of preventing sudden lane changes due to false detections, driver alerts are provided through visual and audible cues. The synergy between distraction and enhanced perception is achieved through a multi-layered control architecture. The perception algorithm layer modifies detection and tracking strategies, while the decision-making and planning layer adjusts safety thresholds. This vertically integrated architecture ensures that distraction information comprehensively influences the perception and decision-making processes of the autonomous driving system, creating perception capabilities that match the driver's state and significantly improving the safety of human-machine co-driving.

[0110] Finally, based on the visual distraction monitoring results and / or fatigue state monitoring results, a visual monitoring result is determined. In one embodiment, the visual monitoring result may simultaneously include information from both distraction and fatigue dimensions. For example, the visual monitoring result could be visual distraction with a fatigue level of mild. In another embodiment, a final comprehensive conclusion can be output based on a preset priority or risk level. For example, regardless of whether distraction is detected, if severe fatigue is detected, the final visual monitoring result is directly determined to be a dangerous state: severe fatigue.

[0111] The method provided in this invention subdivides distraction monitoring into visual distraction monitoring and fatigue state monitoring, and determines the final visual monitoring result based on the results of these two. This enables the system to clearly distinguish the root cause of the driver's lack of concentration, improves the accuracy of state diagnosis, and makes human-computer interaction intervention more efficient and humanized.

[0112] Based on the above embodiments, step 1322-2 includes: Based on the eye opening and closing state, the percentage of eyelid closure time per unit time, blinking frequency, and gaze direction corresponding to a preset number of frames in the driver's facial state data, fatigue state monitoring is performed to obtain the fatigue state monitoring results.

[0113] Specifically, fatigue monitoring can be performed based on a preset number of frames in the driver's facial state data, corresponding to the eye opening and closing state, the proportion of eyelid closure time per unit time, blinking frequency, and gaze direction, to obtain fatigue state monitoring results. For example, assuming the DMS camera has a frame rate of 30fps, the preset number of frames can be set to 300 frames, meaning the system will continuously analyze the driver's eye state data over the past 10 seconds to capture the trend of changes in the state, rather than relying solely on instantaneous states.

[0114] For each frame within the window, a facial landmark detection algorithm is first used to locate the upper and lower eyelid contours of the driver's eyes. The eye opening / closing state for that frame is obtained by calculating the distance between the upper and lower eyelids or the ratio of that distance to the eye width. The eye opening / closing state can be a continuous value or a discrete state of open, half-closed, or closed. To ensure accuracy under different lighting conditions or when the driver is wearing glasses, techniques such as 3D eye modeling can be used to improve the robustness of eye state detection.

[0115] Within a preset number of frames, the system counts the total duration of eye closure within a given time window and calculates its percentage of the total duration within that time window, thus obtaining the percentage of eyelid closure time per unit time. For example, within a 10-second analysis window, if the cumulative eye closure time is 2.5 seconds, then the percentage of eyelid closure time per unit time is 25%.

[0116] Here, blink frequency refers to the number of blinks a driver makes per unit of time, and its value is positively correlated with the number of blinks detected. A complete blink event is defined as the rapid, continuous process of the eye opening, closing, and then opening again. Normal blink frequency remains within a certain range. When a driver begins to fatigue, their blink frequency may initially decrease, followed by compensatory frequent blinking. Therefore, the trend in blink frequency is an important reference for assessing early fatigue.

[0117] Here, in fatigue monitoring, gaze direction specifically refers to the quantitative analysis of the dynamic characteristics of a driver's eye movements. Its core is to assess the trajectory and stability of the gaze point within a time window. In fatigue monitoring, the analysis of gaze direction focuses on its dynamic characteristics. The system assesses the driver's attention level by calculating the trajectory and stability of the driver's gaze point within a specific time window. In a normal state, a driver's gaze will frequently and regularly scan key areas such as the road ahead and rearview mirrors; however, when fatigue sets in, eye movement patterns will change significantly, manifesting as slowed eye movements, prolonged fixed gaze, or aimless gaze drift. These characteristics all indicate a decline in cognitive alertness.

[0118] After acquiring the analytical data of the above four characteristics, the system performs a fusion decision to obtain the final fatigue state monitoring result. This decision-making process can be based on a preset rule engine or a trained machine learning model. For example, no fatigue: eyelid closure time accounts for less than 5% of the total time, blinking frequency is normal, and the gaze direction shows an active saccadic pattern. Mild fatigue: eyelid closure time accounts for 5%-15% of the total time, or blinking frequency is abnormal (too high or too low), or brief gazing behavior begins to appear, i.e., early fatigue characteristics, allowing the system to intervene before obvious symptoms appear. Moderate fatigue: eyelid closure time accounts for 15%-40% of the total time, accompanied by prolonged gazing behavior or slow eye movements (eyelids in a semi-closed state for a long time).

[0119] Severe fatigue: The proportion of eyelid closure time per unit time is higher than 40%, or a microsleep event lasting more than 2 seconds (i.e., complete eye closure) is detected. At this time, gaze direction analysis is meaningless.

[0120] In one embodiment, the system detects fatigue by identifying features such as eye opening <50%, frequent yawning, or persistent head drooping. When the fatigue level (mild, moderate, severe) reaches moderate or severe, the external perception system simultaneously activates a "defensive perception mode": lowering the detection threshold to ensure early response to vehicles cutting in from the side and long-distance road events. Simultaneously, based on historical data and real-time traffic flow, the system predicts the future trajectories of surrounding vehicles and marks high-risk targets (such as vehicles frequently changing lanes) as priority targets, calculating avoidance strategies in advance.

[0121] Based on the above embodiments, step 140 further includes: Step 141: Based on the in-cabin monitoring results and the external perception results, control the vehicle.

[0122] Specifically, after obtaining the in-cabin monitoring results and the external perception results, the vehicle can be controlled based on these results. For example, attention guidance commands can be generated based on the in-cabin monitoring results and the external perception results, and vehicle control can be performed based on these attention guidance commands.

[0123] The method provided in this invention performs vehicle control based on in-cabin monitoring results and external perception results, thereby improving the accuracy and reliability of vehicle control.

[0124] Based on the above embodiments, step 141 includes: Step 1411: Based on the external sensing results, construct an environmental risk distribution map to characterize the distribution of external environmental risks; Step 1412: Based on the cabin monitoring results, construct an attention distribution map to characterize the driver's gaze points; Step 1413: Compare and analyze the environmental risk distribution map with the attention distribution map to determine the attention mismatch area; Step 1414: Generate attention guidance instructions and control the vehicle based on the attention guidance instructions; the attention guidance instructions are used to guide the driver's attention to the attention mismatch area.

[0125] Specifically, Figure 5 This is a schematic diagram of visual attention-guided external perception and internal early warning provided by the present invention, as shown below. Figure 5 As shown, firstly, based on the results of external perception, an environmental risk distribution map is constructed to characterize the distribution of external environmental risks. The environmental risk distribution map can be understood as a virtual map centered on the vehicle, overlaid with dynamic risk information. The construction process involves projecting the results of external perception—for example, all obstacles detected by the fusion of lidar, millimeter-wave radar, and cameras—including their position, speed, and type, onto a two-dimensional or three-dimensional coordinate system. Then, based on the degree of danger of each obstacle, it is visually marked on the environmental risk distribution map. High-risk targets, such as rapidly approaching vehicles, may be marked as prominent red areas on the map, while low-risk targets, such as stationary vehicles in the distance, may be marked yellow or left unmarked. Here, the environmental risk distribution map intuitively displays the distribution of risks around the vehicle.

[0126] Secondly, based on the driver's gaze data output by the in-cabin monitoring system, an attention distribution map can be constructed to characterize the spatial distribution of their visual attention. This attention distribution map can use a unified coordinate system with the environmental risk distribution map to form a virtual attention mapping layer superimposed on the vehicle's surrounding environment. Its construction principle involves continuously mapping the coordinates of the driver's gaze points, monitored in real time, onto this unified coordinate system. By statistically analyzing and visually rendering the duration of gaze points, attention focal points can be generated in the attention distribution map in the form of a heatmap. The color depth or brightness is positively correlated with the duration of gaze, intuitively reflecting the area of ​​the driver's focused attention. For example, when the driver's gaze is continuously focused on the left rearview mirror, the corresponding area on the attention distribution map will appear as a bright hotspot.

[0127] Here, the attention distribution map can be a real-time attention distribution map, and this embodiment of the invention does not specifically limit it.

[0128] Next, a comparative analysis of the environmental risk distribution map and the attention distribution map is conducted to identify attention mismatch areas. These areas are marked as high-risk on the environmental risk distribution map but do not receive sufficient driver attention on the attention distribution map, or vice versa. For example, if the environmental risk distribution map shows a motorcycle suddenly crossing from the right (high-risk area), while the attention distribution map shows the driver's focus is on the left-hand center console screen, then the area where the motorcycle is located is an attention mismatch area.

[0129] Finally, attention guidance commands are generated, and the vehicle is controlled based on these commands. These commands direct the driver's attention to the area of ​​attention mismatch. This guidance can be visual, such as projecting a flashing arrow, highlighted frame, or other warning symbol onto the windshield at the real-world location corresponding to the area of ​​attention mismatch using an AR-HUD; or acoustic, such as using the directional sound technology of the in-vehicle audio system to emit warning sounds or voice prompts in the direction of the area of ​​attention mismatch, for example, issuing a "Please be aware of vehicles approaching from the right" warning from the front right speaker.

[0130] It should be noted that while lowering the detection threshold and increasing detection sensitivity enhances the vehicle's perception capabilities, this does not affect driving control. Instead of preventing sudden lane changes due to false detections, driver alerts are provided through visual and auditory cues. Attention guidance commands primarily serve to remind and guide the driver and are typically not accompanied by direct vehicle control actions such as automatic steering or braking. This avoids abrupt vehicle movements that may result from misjudgments, ensuring smooth and predictable driving.

[0131] The method provided in this invention constructs an environmental risk distribution map representing external risks and an attention distribution map representing the driver's focus points. By comparing the two maps in real time, it accurately locates attention mismatch areas and generates attention guidance instructions pointing to those areas. This transforms the generation of vehicle control instructions from blind global warnings to precise visual or auditory guidance for specific risk points, improving the efficiency of human-computer interaction and proactive safety. It upgrades system intervention from simple risk notification to intelligent attention assistance, greatly optimizing the driver's situational awareness and response speed in complex scenarios.

[0132] Based on any of the above embodiments Figure 6 This is the second flowchart illustrating the security sensing method provided by the present invention, as shown below. Figure 6 As shown, this method mainly consists of four parts: External Information Assists In-Cabin Monitoring (Dynamic Sensitivity Adjustment): How to dynamically adjust the DMS monitoring sensitivity using external environmental data, including multi-dimensional perception methods based on vision and vehicle status signals, and sensitivity adjustment strategies for different driving scenarios.

[0133] External information assists in-cabin monitoring (multi-dimensional monitoring): This section describes how multi-sensor fusion technology enables comprehensive monitoring of the driver's condition, dynamically adjusts the warning level based on the external risk level, including multi-level warning mechanisms and personalized safety strategies, and highlights the extension of fine-grained monitoring in high-risk environments.

[0134] In-cabin information assists external perception (distraction state transmission and perception enhancement): This study analyzes the interaction mechanism between the driver's distraction state and the autonomous driving system, including the quantification and transmission of distraction state, perception enhancement strategies, and system fault-tolerant control design, and emphasizes the enhancement of overall perception sensitivity by distraction and fatigue states.

[0135] In-cabin information assists external perception (visual attention guidance): The AR-HUD-based visual attention guidance system enhances the driver's situational awareness and reduces cognitive load through dynamic information projection and adaptive interface design.

[0136] Traditional systems lack two-way collaborative capabilities, cannot dynamically adjust cabin monitoring strategies based on external environmental risks, and cannot utilize driver status information to optimize external perception algorithms, thus resulting in deficiencies in the timeliness and accuracy of safety decisions.

[0137] It is understood that the embodiments of the present invention break through the limitation of the isolation between the in-cabin and out-of-cabin information processing in traditional automotive safety systems. By establishing a two-way interactive and collaborative decision-making intelligent mechanism, it achieves earlier, more accurate, and more personalized identification and intervention of driving safety risks.

[0138] The present invention aims to provide an active safety perception and early warning system that integrates information inside and outside the cabin. Its core lies in breaking the isolation between information processing inside and outside the cabin in the traditional system, and realizing the dynamic adjustment of safety early warning function by establishing a two-way information interaction and collaborative decision-making mechanism.

[0139] The safety sensing device provided by the present invention is described below. The safety sensing device described below and the safety sensing method described above can be referred to in correspondence.

[0140] Based on any of the above embodiments, the present invention provides a security sensing device. Figure 7 This is a structural schematic diagram of the safety sensing device provided by the present invention, as shown below. Figure 7 As shown, the device includes: The first acquisition unit 710 is used to acquire external environmental information of the vehicle and internal state information that characterizes the state of the driver inside the vehicle. The first determining unit 720 is used to determine the environmental risk level based on the external environment information, so as to adjust the internal monitoring parameters according to the environmental risk level. The second determining unit 730 is used to determine the pilot's status level based on the cabin status information, so as to adjust the external control parameters according to the pilot's status level. The second acquisition unit 740 is used to acquire in-cabin monitoring results and out-of-cabin perception results. The in-cabin monitoring results are collected by the monitoring equipment inside the vehicle based on the adjusted in-cabin monitoring parameters, and the out-of-cabin perception results are collected by the perception equipment outside the vehicle based on the out-of-cabin control parameters.

[0141] The device provided in this invention determines the environmental risk level based on external environmental information and adjusts the in-cabin monitoring parameters based on the environmental risk level, thereby matching the sensitivity of in-cabin monitoring with the actual risks of the external environment. Simultaneously, by determining the driver's state level based on in-cabin state information and adjusting external control parameters based on the driver's state level, the external perception strategy can compensate for insufficient perception capabilities that may occur when the driver is in a poor state. Furthermore, by acquiring the bidirectional adjusted in-cabin monitoring and external perception results, it can accurately identify mismatches between driver attention and external environmental risks. This solves the problem of traditional solutions where the isolation of in-cabin and external information leads to rigid safety strategies and an inability to adapt to dynamic driving scenarios, improving the accuracy and timeliness of safety perception and ensuring driving safety in intelligent driving scenarios.

[0142] Based on any of the above embodiments, the first determining unit 720 is specifically used for: The external environment information is analyzed to obtain vehicle speed data, traffic flow density data, environmental weather data, and road complexity data; The environmental risk level is determined based on the vehicle speed data, traffic flow density data, environmental weather data, and road complexity data.

[0143] Based on any of the above embodiments, the second determining unit 730 specifically includes: The parsing unit is used to parse the cabin state information to obtain at least two of the following: driver behavior data, driver facial state data, driver hand state data, body posture data, and cabin passenger behavior data. The fusion analysis unit is used to perform multimodal fusion analysis on at least two of the driver behavior data, driver facial state data, driver hand state data, body posture data and cabin passenger behavior data to obtain a comprehensive driver state model that characterizes the driver's current state. A level determination unit is used to determine the driver's state level based on the comprehensive driver state model.

[0144] Based on any of the above embodiments, the fusion analysis unit specifically includes: A driving behavior analysis unit is used to perform driving behavior analysis on the driver behavior data and obtain driver behavior analysis results. A visual monitoring unit is used to perform visual monitoring on the driver's facial state data and obtain visual monitoring results; The hands-off detection unit is used to detect hands-off status data of the driver and obtain hands-off detection results; The posture monitoring unit is used to monitor the body posture data and obtain abnormal action recognition results. An interactive behavior detection unit is used to detect interactive behavior on the passenger behavior data in the cabin and obtain interactive behavior detection results. A model unit is defined to determine the comprehensive driver state model based on at least two of the driver behavior analysis results, the visual monitoring results, the hands-off detection results, the abnormal action recognition results, and the interactive behavior detection results.

[0145] Based on any of the above embodiments, the visual monitoring unit specifically includes: A visual distraction monitoring unit is used to perform visual distraction monitoring on the driver's facial state data and obtain visual distraction monitoring results. A fatigue state monitoring unit is used to monitor the driver's facial state data for fatigue state and obtain fatigue state monitoring results. A monitoring result determination unit is used to determine the visual monitoring result based on the visual distraction monitoring result and / or the fatigue state monitoring result.

[0146] Based on any of the above embodiments, the fatigue state monitoring unit is specifically used for: Based on the eye opening and closing state, the percentage of eyelid closure time per unit time, blinking frequency, and gaze direction corresponding to a preset number of frames in the driver's facial state data, fatigue state monitoring is performed to obtain the fatigue state monitoring results.

[0147] Based on any of the above embodiments, a vehicle control unit is further included, wherein the vehicle control unit specifically includes: A control subunit is used to control the vehicle based on the in-cabin monitoring results and the external perception results.

[0148] Based on any of the above embodiments, the control subunit is specifically used for: Based on the external sensing results, an environmental risk distribution map is constructed to characterize the distribution of external environmental risks. Based on the cabin monitoring results, an attention distribution map is constructed to characterize the driver's gaze points. The environmental risk distribution map and the attention distribution map are compared and analyzed to determine the attention mismatch areas; Generate attention guidance instructions, and control the vehicle based on the attention guidance instructions; the attention guidance instructions are used to guide the driver's attention to the attention mismatch area.

[0149] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 8 As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a safety perception method, which includes: acquiring external environmental information of the vehicle and internal state information characterizing the driver's state within the vehicle; determining an environmental risk level based on the external environmental information, and adjusting internal monitoring parameters according to the environmental risk level; determining a driver state level based on the internal state information, and adjusting external control parameters according to the driver state level; acquiring internal monitoring results and external perception results, wherein the internal monitoring results are collected by monitoring devices inside the vehicle controlled based on the adjusted internal monitoring parameters, and the external perception results are collected by sensing devices outside the vehicle controlled based on the external control parameters.

[0150] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0151] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the safety perception method provided by the above methods. The method includes: acquiring external environmental information of the vehicle and internal state information characterizing the driver's state inside the vehicle; determining an environmental risk level based on the external environmental information, and adjusting internal monitoring parameters according to the environmental risk level; determining a driver state level based on the internal state information, and adjusting external control parameters according to the driver state level; acquiring internal monitoring results and external perception results, wherein the internal monitoring results are collected by monitoring equipment inside the vehicle controlled based on the adjusted internal monitoring parameters, and the external perception results are collected by sensing equipment outside the vehicle controlled based on the external control parameters.

[0152] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the safety perception method provided by the methods described above. This method includes: acquiring external environmental information of a vehicle and internal state information characterizing the driver's state within the vehicle; determining an environmental risk level based on the external environmental information, and adjusting internal monitoring parameters according to the environmental risk level; determining a driver state level based on the internal state information, and adjusting external control parameters according to the driver state level; acquiring internal monitoring results and external perception results, wherein the internal monitoring results are collected by monitoring equipment inside the vehicle controlled based on the adjusted internal monitoring parameters, and the external perception results are collected by sensing equipment outside the vehicle controlled based on the external control parameters.

[0153] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0154] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A security sensing method, characterized in that, include: Acquire information about the external environment of the vehicle and information about the internal state of the vehicle that characterizes the state of the driver inside the vehicle; The environmental risk level is determined based on the external environmental information, and the monitoring parameters inside the cabin are adjusted according to the environmental risk level. The pilot's status level is determined based on the cabin status information, and the external control parameters are adjusted according to the pilot's status level. The system acquires in-cabin monitoring results and out-of-cabin perception results. The in-cabin monitoring results are collected by the monitoring equipment inside the vehicle based on the adjusted in-cabin monitoring parameters, and the out-of-cabin perception results are collected by the perception equipment outside the vehicle based on the out-of-cabin control parameters.

2. The security sensing method according to claim 1, characterized in that, The determination of the environmental risk level based on the external environment information includes: The external environment information is analyzed to obtain vehicle speed data, traffic flow density data, environmental weather data, and road complexity data; The environmental risk level is determined based on the vehicle speed data, traffic flow density data, environmental weather data, and road complexity data.

3. The security perception method according to claim 1, characterized in that, Determining the pilot's status level based on the cabin status information includes: The cabin status information is parsed to obtain at least two of the following: driver behavior data, driver facial status data, driver hand status data, body posture data, and cabin passenger behavior data. Multimodal fusion analysis is performed on at least two of the driver behavior data, driver facial state data, driver hand state data, body posture data, and cabin passenger behavior data to obtain a comprehensive driver state model that characterizes the driver's current state. Based on the comprehensive driver state model, the driver state level is determined.

4. The security perception method according to claim 3, characterized in that, The process involves performing multimodal fusion analysis on at least two of the driver behavior data, driver facial state data, driver hand state data, body posture data, and cabin passenger behavior data to obtain a comprehensive driver state model characterizing the driver's current state, including: The driver behavior data is analyzed to obtain driver behavior analysis results; Visual monitoring is performed on the driver's facial state data to obtain visual monitoring results; The driver's hand status data is subjected to a hands-off detection to obtain the hands-off detection result; The body posture data is subjected to posture monitoring to obtain abnormal movement recognition results; Interactive behavior detection is performed on the passenger behavior data in the cabin to obtain interactive behavior detection results; The comprehensive driver state model is determined based on at least two of the driver behavior analysis results, the visual monitoring results, the hands-off detection results, the abnormal action recognition results, and the interactive behavior detection results.

5. The security sensing method according to claim 4, characterized in that, The step of visually monitoring the driver's facial state data to obtain visual monitoring results includes: Visual distraction monitoring is performed on the driver's facial state data to obtain visual distraction monitoring results; The driver's facial state data was used to monitor fatigue state, and fatigue state monitoring results were obtained. The visual monitoring results are determined based on the visual distraction detection results and / or the fatigue state monitoring results.

6. The security perception method according to claim 5, characterized in that, The fatigue state monitoring of the driver's facial state data to obtain fatigue state monitoring results includes: Based on the eye opening and closing state, the percentage of eyelid closure time per unit time, blinking frequency, and gaze direction corresponding to a preset number of frames in the driver's facial state data, fatigue state monitoring is performed to obtain the fatigue state monitoring results.

7. The security sensing method according to any one of claims 1 to 6, characterized in that, The process of acquiring the in-cabin monitoring results and the external sensing results then includes: The vehicle is controlled based on the in-cabin monitoring results and the external perception results.

8. The security perception method according to claim 7, characterized in that, The control of the vehicle based on the in-cabin monitoring results and the external perception results includes: Based on the external sensing results, an environmental risk distribution map is constructed to characterize the distribution of external environmental risks. Based on the cabin monitoring results, an attention distribution map is constructed to characterize the driver's gaze points. The environmental risk distribution map and the attention distribution map are compared and analyzed to determine the attention mismatch areas; Generate attention guidance instructions, and control the vehicle based on the attention guidance instructions; the attention guidance instructions are used to guide the driver's attention to the attention mismatch area.

9. A safety sensing device, characterized in that, include: The first acquisition unit is used to acquire information about the external environment of the vehicle and information about the internal state of the vehicle that characterizes the state of the driver inside the vehicle. The first determining unit is used to determine the environmental risk level based on the external environment information, so as to adjust the internal monitoring parameters according to the environmental risk level. The second determining unit is used to determine the pilot's status level based on the cabin status information, so as to adjust the external control parameters according to the pilot's status level. The second acquisition unit is used to acquire in-cabin monitoring results and external perception results. The in-cabin monitoring results are collected by the monitoring equipment inside the vehicle based on the adjusted in-cabin monitoring parameters, and the external perception results are collected by the perception equipment outside the vehicle based on the external control parameters.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the security awareness method as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the security awareness method as described in any one of claims 1 to 8.

Images