Driving state monitoring method, device and equipment and computer readable storage medium

Abstract

The embodiment of the invention provides a driving state monitoring method, device and equipment and a computer readable storage medium. The method comprises the following steps: acquiring a current facial image of a driver; according to the current facial image, identifying a real-time facial feature value of the driver; acquiring a current driving scene of the vehicle; according to the current driving scene, obtaining a corresponding personalized driving state threshold value; and determining the driving state of the driver according to the real-time facial feature value and the personalized driving state threshold. In this way, the personalized driving state threshold value can be obtained according to the current driving scene instead of obtaining the preset fixed threshold value, so that the driving state of the driver can be accurately judged according to the driving scene, the driving state judgment mode can adapt to different drivers, false alarms caused by too sensitive threshold value setting are avoided, and the driving experience of the driver is improved. And potential safety hazards caused by missing report due to over-high threshold setting are also avoided.

Description

Technical Field

[0001] This disclosure relates to the field of active safety for automobiles, and more particularly to the field of driving condition monitoring technology. Background Technology

[0002] With the continuous advancement of automotive intelligence technology, Driver Monitoring Systems (DMS) have become an important component of driver assistance and active safety systems. Currently, most mainstream DMS systems are based on vision sensors, using in-vehicle cameras deployed inside the cockpit to capture real-time images of the driver's face and extracting key features such as eye opening and closing (EAR), gaze direction, head posture, and blink frequency using computer vision algorithms. Through real-time calculation of these feature parameters, the system determines whether the driver is fatigued, distracted, or in other abnormal states based on preset fixed thresholds.

[0003] However, this method of judging driver status using preset fixed thresholds lacks adaptability to individual driver differences. For example, for some drivers, the system may generate false alarms due to overly sensitive threshold settings, frequently issuing unnecessary alerts, thereby interfering with normal driving and reducing user trust in the system; while for other drivers, the threshold may not be able to effectively capture their fatigue or distraction characteristics, leading to missed alarms and creating potential safety hazards.

[0004] Secondly, the system performs poorly in different driving scenarios. Changes in the driving environment (such as day-night cycles, highway cruising, and urban congestion) can significantly affect the driver's behavior patterns and facial features, and fixed thresholds cannot dynamically respond to such situational changes.

[0005] Therefore, accurately judging the driver's driving status has become an urgent problem to be solved. Summary of the Invention

[0006] This disclosure provides a driving status monitoring method, apparatus, device, and storage medium.

[0007] According to a first aspect of this disclosure, a method for monitoring driving status is provided. The method includes:

[0008] Obtain the driver's current facial image; Based on the current facial image, identify the driver's real-time facial feature values; Obtain the current driving scenario of the vehicle; Based on the current driving scenario, obtain the corresponding personalized driving state threshold; The driver's driving status is determined based on the real-time facial feature values ​​and the personalized driving status threshold.

[0009] In addition to the aspects and any possible implementations described above, a further implementation is provided in which obtaining the current driving scenario of the vehicle includes: Acquire an image of the road ahead of the vehicle; The vehicle's current dynamic driving data is read via the vehicle's CAN bus; Based on the image of the road ahead and the current dynamic driving data, the current driving scenario is confirmed.

[0010] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein confirming the current driving scenario based on the forward road image and the current dynamic driving data includes: If multiple scenarios are identified based on the road image ahead and the current dynamic driving data; Determine the priority of each scenario among the multiple scenarios; Based on the priority of each scenario, select several scenarios from the plurality of scenarios; The current driving scenario is obtained by combining the aforementioned scenarios.

[0011] In addition to the aspects and any possible implementations described above, a further implementation is provided in which the real-time facial feature values ​​include: the driver's eye opening degree, the driver's gaze direction, the driver's head posture angle, the driver's mouth shape, and the yawning frequency calculated based on the changes in the mouth shape.

[0012] In addition to the aspects and any possible implementations described above, a further implementation is provided in which determining the driver's driving state based on the real-time facial feature value and the personalized driving state threshold includes: If the real-time facial feature value includes multiple facial feature values, then the driving state threshold corresponding to each of the multiple facial feature values ​​is determined from the personalized driving state threshold. The plurality of facial feature values ​​are compared with the corresponding driving state thresholds to obtain the driving state comparison results corresponding to each of the plurality of facial feature values; Determine whether the driving state comparison results corresponding to each of the multiple facial feature values ​​meet the corresponding preset driving state comparison results; If the driving state comparison result corresponding to any of the multiple facial feature values ​​satisfies the corresponding preset driving state comparison result, and the duration of the satisfaction is greater than the preset duration, then the driving state of the driver is determined based on any of the facial feature values.

[0013] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein the method further includes: Based on any of the facial feature values, the driver's state level is output; Based on the driver's status level, a corresponding warning is triggered.

[0014] In addition to the aspects and any possible implementations described above, a further implementation is provided to obtain the corresponding personalized driving state threshold, including: Obtain the anonymous identity of the driver; The current driving scenario and the anonymous identity are uploaded to the cloud. The cloud obtains each historical driving scenario uploaded by the driver, the facial feature value corresponding to each historical driving scenario, and the anonymous identity. The cloud uses a Gaussian mixture model to fit each historical driving scenario and the facial feature value corresponding to each historical driving scenario to obtain the personalized driving state threshold corresponding to each historical driving scenario of the anonymous identity. The personalized driving state threshold corresponding to each historical driving scenario is stored in correspondence with the anonymous identity. Receive the personalized driving status thresholds sent from the cloud that correspond to the current driving scenario and the anonymous identity.

[0015] According to a second aspect of this disclosure, a driving condition monitoring device is provided. The device includes: The first acquisition module is used to acquire the driver's current facial image; The recognition module is used to recognize the driver's real-time facial feature values ​​based on the current facial image; The second acquisition module is used to acquire the current driving scenario of the vehicle; The third acquisition module is used to acquire the corresponding personalized driving state threshold based on the current driving scenario. The determination module is used to determine the driver's driving state based on the real-time facial feature values ​​and the personalized driving state threshold.

[0016] According to a third aspect of this disclosure, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described above.

[0017] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method according to a first aspect of this disclosure.

[0018] In this disclosure, after obtaining the driver's current facial image, the driver's real-time facial feature value can be identified based on the current facial image, and the current driving scenario of the vehicle can be obtained. Then, based on the current driving scenario, a corresponding personalized driving state threshold can be obtained. Subsequently, the driver's driving state can be determined based on the real-time facial feature value and the personalized driving state threshold. In this way, a personalized driving state threshold can be obtained based on the current driving scenario, rather than obtaining a preset fixed threshold. Moreover, the personalized driving state threshold corresponds to the driving scenario. Therefore, the driver's driving state can be accurately judged based on the driving scenario. This driving state judgment method can adapt to different drivers, avoid false alarms caused by overly sensitive threshold settings, and also avoid false alarms caused by excessively high threshold settings, thus preventing potential safety hazards.

[0019] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0020] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A framework diagram of a driving condition monitoring system according to an embodiment of the present disclosure is shown; Figure 2 It shows Figure 1 Workflow diagram of the personalized baseline application module; Figure 3 It shows Figure 1 Workflow diagram of the local processing unit; Figure 4 It shows Figure 1 Workflow diagram of the personalized modeling server in Zhongyun. Figure 5 A flowchart of a driving state monitoring method according to an embodiment of the present disclosure is shown; Figure 6 A block diagram of a driving status monitoring device according to an embodiment of the present disclosure is shown; Figure 7 A block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0022] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0023] Figure 5 A flowchart of a driving state monitoring method 500 according to an embodiment of the present disclosure is shown. Method 500 may include: Step 510: Obtain the driver's current facial image; Step 520: Based on the current facial image, identify the driver's real-time facial feature values; Step 530: Obtain the current driving scenario of the vehicle; Step 540: Based on the current driving scenario, obtain the corresponding personalized driving state threshold; Step 550: Determine the driver's driving status based on the real-time facial feature value and the personalized driving status threshold.

[0024] After acquiring the driver's current facial image, the system can identify the driver's real-time facial feature values ​​based on the current facial image and obtain the vehicle's current driving scenario. Then, based on the current driving scenario, a corresponding personalized driving state threshold can be obtained. Subsequently, the driver's driving state can be determined based on the real-time facial feature values ​​and the personalized driving state threshold. In this way, a personalized driving state threshold can be obtained based on the current driving scenario, rather than a preset fixed threshold. Moreover, the personalized driving state threshold corresponds to the driving scenario. Therefore, the driver's driving state can be accurately judged based on the driving scenario. This driving state judgment method can adapt to different drivers, avoiding false alarms caused by overly sensitive threshold settings, and also avoiding missed alarms caused by excessively high threshold settings, thus preventing potential safety hazards.

[0025] In some embodiments, obtaining the current driving scenario of the vehicle includes: Acquire an image of the road ahead of the vehicle; A high-definition camera (usually a DMS camera) can capture images of the road ahead of the vehicle. This camera should have a resolution of at least 720P and a frame rate of at least 30fps to ensure clear facial images are captured even in low-light conditions. The camera is preferably mounted on the A-pillar, steering column, or above the dashboard in the driver's seat area. The angle between the line connecting its optical center axis and the center of the driver's face and the vehicle's direction of travel should be between 15° and 45° to ensure the optimal shooting angle.

[0026] The vehicle's current dynamic driving data is read via the vehicle's CAN bus; The CAN bus is the communication line connected to the vehicle's CAN bus interface. This interface connects to the vehicle's CAN network and is used to read the vehicle's current dynamic driving data in real time, including but not limited to: vehicle speed, latitude and longitude location information (from GPS), timestamp, turn signal signals, and high beam signals. This data is used to help determine the driving scenario (for example, combining vehicle speed and location information to determine whether it is on a highway or in urban areas; combining timestamp and light signals to determine lighting conditions).

[0027] Based on the image of the road ahead and the current dynamic driving data, the current driving scenario is confirmed.

[0028] The scene recognition engine analyzes images from a forward-facing camera using a pre-trained image recognition model to identify lighting conditions (e.g., low light, normal light, dusk), road types (e.g., highways, city roads, rural roads, parking lots), and traffic conditions (e.g., congestion, free flow). Simultaneously, the engine integrates vehicle speed and location (GPS) data from the CAN bus. For example, if the vehicle speed consistently exceeds 80 km / h and GPS location indicates a highway, the current scene is identified as a "highway"; if the vehicle speed is below 20 km / h with frequent stops and starts, and the forward-facing camera detects numerous vehicle taillights, it is determined to be a "congested urban area." (It should be noted that in real-world driving environments, the aforementioned scene dimensions are not mutually exclusive, and multiple dimensions often occur simultaneously. For example, a vehicle may be simultaneously in a "low-light environment," a "highway," and a "familiar commuting route." To address this issue, the scene recognition engine is further configured to handle composite scenes: Firstly, the system pre-sets priority rules for different scene dimensions (e.g., low light > highway > urban congestion > familiar commuting). When multiple scene dimensions are identified simultaneously, the engine will select the one or two dimensions with the highest priority and combine them into a composite external environment scene label (e.g., "low light_highway"), and...) The composite label is used for subsequent personalized threshold indexing. Other dimensional information can be used as auxiliary parameters in local decision-making or uploaded to the cloud for model training. Secondly, the image recognition model can be trained to directly output composite external environment scene labels. This model takes multi-dimensional raw data as input and directly maps it to preset composite scene classifications (such as "nighttime highway," "daytime congested city," etc.) through deep learning. The cloud-based personalized modeling server 105 also supports establishing and issuing corresponding personalized judgment thresholds for these composite driving external environment scene labels (such as "low light_highway"), thereby achieving more refined scene adaptation. In some embodiments, determining the current driving scenario based on the image of the road ahead and the current dynamic driving data includes: If multiple scenarios are identified based on the road image ahead and the current dynamic driving data; Determine the priority of each scenario among the multiple scenarios; Based on the priority of each scenario, select several scenarios from the plurality of scenarios; The current driving scenario is obtained by combining the aforementioned scenarios.

[0029] If multiple scenarios are identified based on the road image ahead and the current dynamic driving data, the priority of each scenario can be determined. Then, based on the priority of each scenario, several scenarios are selected from the multiple scenarios, i.e., partial scenarios are selected. These several scenarios are then combined to accurately obtain the current driving scenario.

[0030] The system has pre-defined priority rules for different scenarios (e.g., low light > highway > urban congestion > familiar commuting). When multiple scenario dimensions are identified at the same time, the engine will select one or two of the highest priority dimensions to combine into a composite external environment scenario label (e.g., under the above 5 scenarios, according to the above priority, 2 scenarios "low light_highway" can be selected as the current driving scenario).

[0031] When multiple scene dimensions are identified simultaneously, the engine will select the one or two dimensions with the highest priority to combine into a composite external environment scene label (such as "low light_highway"), and use this composite label for subsequent personalized threshold indexing.

[0032] In some embodiments, the real-time facial feature values ​​include: the driver's eye opening angle (EAR), the driver's gaze direction, the driver's head posture angle, the driver's mouth shape, and the yawning frequency calculated based on the changes in the mouth shape.

[0033] Head attitude angles include pitch, yaw, and roll angles.

[0034] In some embodiments, determining the driver's driving state based on the real-time facial feature value and the personalized driving state threshold includes: If the real-time facial feature value includes multiple facial feature values, then the driving state threshold corresponding to each of the multiple facial feature values ​​is determined from the personalized driving state threshold. Personalized driving state thresholds may include thresholds corresponding to eye opening and closing angles, and distraction thresholds corresponding to head yaw angles.

[0035] The plurality of facial feature values ​​are compared with the corresponding driving state thresholds to obtain the driving state comparison results corresponding to each of the plurality of facial feature values; Determine whether the driving state comparison results corresponding to each of the multiple facial feature values ​​meet the corresponding preset driving state comparison results; If the driving state comparison result corresponding to any of the multiple facial feature values ​​satisfies the corresponding preset driving state comparison result, and the duration of the satisfaction is greater than the preset duration, then the driving state of the driver is determined based on any of the facial feature values.

[0036] If the real-time facial feature value includes multiple facial feature values, then the driving state thresholds corresponding to the multiple facial feature values ​​are determined from the personalized driving state thresholds. Then, the multiple facial feature values ​​are compared with the corresponding driving state thresholds to obtain the driving state comparison results corresponding to each of the multiple facial feature values. Then, it is determined whether the driving state comparison results corresponding to each of the multiple facial feature values ​​meet the corresponding preset driving state comparison results. If the driving state comparison result corresponding to any one of the multiple facial feature values ​​meets the corresponding preset driving state comparison result, and the duration of the satisfaction is greater than the preset duration, then the driver's driving state is accurately determined based on any one of the facial feature values.

[0037] For example, the preset driving state comparison results include head yaw angle continuously exceeding the distraction threshold and eye opening degree continuously falling below the personalized fatigue threshold (e.g., EAR < 0.25). If the driving state comparison result corresponding to head yaw angle is that head yaw angle continuously exceeds the distraction threshold, then the driver is confirmed to be distracted based on head yaw angle; if the driving state comparison result corresponding to eye opening degree is that eye opening degree continuously falls below the personalized fatigue threshold, then the driver is confirmed to be fatigued based on eye opening degree.

[0038] In some embodiments, the method further includes: Based on any of the facial feature values, the driver's state level is output; For example, if the eye opening and closing degree remains below the personalized fatigue threshold for 2.0 seconds, it can be judged as "mild fatigue"; if it remains below the threshold for more than 4.0 seconds, it can be judged as "severe fatigue".

[0039] Based on the driver's status level, a corresponding warning is triggered.

[0040] Based on any facial feature value, the driver's status level can also be output, and then the corresponding warning can be automatically and accurately triggered based on the driver's status level.

[0041] In some embodiments, obtaining the corresponding personalized driving state threshold includes: Obtain the anonymous identity of the driver; The current driving scenario and the anonymous identity are uploaded to the cloud. The cloud obtains each historical driving scenario uploaded by the driver, the facial feature value corresponding to each historical driving scenario, and the anonymous identity. The cloud uses a Gaussian mixture model to fit each historical driving scenario and the facial feature value corresponding to each historical driving scenario to obtain the personalized driving state threshold corresponding to each historical driving scenario of the anonymous identity. The personalized driving state threshold corresponding to each historical driving scenario is stored in correspondence with the anonymous identity. The cloud receives anonymized data packets uploaded from a large number of vehicles. Each data packet contains an anonymous driver ID, a sequence of facial feature values ​​(such as a set of EAR values), an external environment scene label (i.e., driving scenario, such as "nighttime highway"), and a driver driving status label (such as "normal" or "fatigued").

[0042] The received data undergoes cleaning and preprocessing to remove outliers caused by signal interference. The massive amounts of data are then strictly categorized and stored according to two key dimensions: "Driver ID" and "External Environment Scene Label." For newly registered drivers or "driver-scene" combinations not yet encountered by the system, due to a lack of historical data, the cloud server will temporarily use a default threshold configuration file based on large-scale population data statistics and distribute it to the vehicle to ensure the system functions correctly in the initial stage. Once a sufficient amount of "normal" state data (e.g., 10 minutes of continuous normal driving) of data from the driver in that scene is collected, the personalized modeling process described below will be initiated.

[0043] For each unique “driver-scenario” combination, extract the sequence of feature values ​​(e.g., all EAR values) collected during periods when the system did not trigger an alarm from its historical data, and use these data as the “normal behavior sample” set for that combination.

[0044] Using these normal behavior samples as input, a Gaussian mixture model (GMM) is run to estimate the probability density and fit the probability distribution of the driver's normal characteristic values ​​in this scenario.

[0045] Calculate a specific low quantile (e.g., the 5th quantile) of this probability distribution. Determine the eigenvalue corresponding to this quantile as the driver's personalized fatigue baseline threshold for this scenario (e.g., the EAR value of the 5th quantile as the fatigue threshold). This means that the driver has a 95% probability that their eigenvalues ​​under normal conditions are better than this threshold, and if the eigenvalues ​​consistently fall below this threshold, they are considered abnormal.

[0046] Receive the personalized driving status thresholds sent from the cloud that correspond to the current driving scenario and the anonymous identity.

[0047] Figure 1 This is a system architecture diagram of the present invention. (For example...) Figure 1 As shown, the system includes: an in-vehicle perception module 101, an external environment perception module 102, a local processing unit 103, a vehicle-to-everything (V2X) communication module (T-Box) 104, and a cloud-based personalized modeling server 105. The modules interact with each other via a vehicle bus (such as CAN, LIN, or Ethernet) or a high-speed data interface.

[0048] The in-vehicle perception module 101 mainly includes a driver-facing, built-in infrared-illuminated high-definition camera (typically a DMS camera) with a resolution of at least 720P and a frame rate of at least 30fps to ensure clear facial images can be captured even in low-light conditions. This camera is preferably mounted on the A-pillar, steering column, or above the dashboard, with the angle between the line connecting its optical center axis and the center of the driver's face and the vehicle's direction of travel between 15° and 45° to ensure the optimal shooting angle. This module is responsible for continuously acquiring the driver's facial video stream and transmitting it to the local processing unit 103.

[0049] The external environment perception module 102 is used to acquire information about the vehicle's external environment, and it includes two sub-units: Forward-facing camera: Installed near the rearview mirror inside the windshield, it is used to acquire images of the road environment in front of the vehicle. Its field of view should cover at least three lanes and road signs in front of the vehicle. The image data from this camera is mainly used for visual scene recognition.

[0050] Vehicle CAN bus interface: This interface connects to the vehicle's CAN network and is used to read the vehicle's dynamic data in real time, including but not limited to: vehicle speed, latitude and longitude location information (from GPS), timestamp, turn signal signals, and high beam signals. This data is used to help determine the driving scenario (for example, combining vehicle speed and location information to determine whether it is on a highway or in urban areas; combining timestamp and light signals to determine lighting conditions).

[0051] The local processing unit 103 can be integrated into the domain controller of the in-vehicle infotainment system (IVI) or a standalone intelligent driving domain controller. It possesses certain computing capabilities, typically including a multi-core CPU based on ARM or x86 architecture and an AI acceleration core (such as an NPU). It receives raw data from 101 and 102 and has three built-in core engines for real-time processing: Scene Recognition Engine: This engine operates based on data from the vehicle's external environment perception module 102. It analyzes images from the forward-facing camera using a pre-trained image recognition model to identify lighting conditions (e.g., low light, normal light, dusk), road types (e.g., highways, urban roads, rural roads, parking lots), and traffic conditions (e.g., congestion, smooth traffic). Simultaneously, the engine integrates vehicle speed and location (GPS) data from the CAN bus. For example, if the vehicle speed consistently exceeds 80 km / h and GPS location indicates a highway, the current scene is identified as a "highway"; if the vehicle speed is below 20 km / h with frequent starts and stops, and the forward-facing camera detects numerous vehicle taillights, it is determined to be a "congested urban area." (It should be noted that in real-world driving environments, the aforementioned scene dimensions are not mutually exclusive, and multiple dimensions often occur simultaneously. For example, a vehicle may be simultaneously in a "low-light environment," a "highway," and a "familiar commuting route." To address this issue, the scene recognition engine is further configured to handle composite scenes: Firstly, the system pre-sets priority rules for different scene dimensions (e.g., low light > highway > urban congestion > familiar commuting). When multiple scene dimensions are identified simultaneously, the engine will select the one or two dimensions with the highest priority and combine them into a composite external environment scene label (e.g., "low light_highway"), and...) The composite label is used for subsequent personalized threshold indexing. Other dimensional information can be used as auxiliary parameters in local decision-making or uploaded to the cloud for model training. Secondly, the image recognition model can be trained to directly output composite external environment scene labels. This model takes multi-dimensional raw data as input and directly maps it to preset composite scene classifications (such as "nighttime highway," "daytime congested city," etc.) through deep learning. The cloud-based personalized modeling server 105 also supports establishing and issuing corresponding personalized judgment thresholds for these composite driving external environment scene labels (such as "low light_highway"), thereby achieving more refined scene adaptation. Real-time feature extraction module: This module runs a lightweight facial feature detection algorithm based on the facial image data and extracts a series of feature values ​​related to the driver's state. These feature values ​​include: eye opening angle (EAR), gaze direction (pupil center position), head posture angles (pitch, yaw, roll angles), and yawning frequency calculated based on mouth shape changes. These feature values ​​are calculated and output at a certain frequency (e.g., 1Hz).

[0052] Personalized Baseline Application Module: This module is the core decision-making component of the local processing unit, configured to execute a dynamically adaptive driver state determination algorithm. This module achieves high-precision judgment through the following mechanism.

[0053] Contextualized threshold mapping: The module receives real-time external environment scene labels (such as "low_light_highway") from the scene recognition engine, combines them with the current driver identity identifier, generates a unique environment-driver matching key, and indexes a specific threshold parameter group issued by the cloud from the local cache accordingly.

[0054] Parallel comparison of multiple features: The module synchronously processes multi-dimensional physiological feature stream data from the feature extraction module (including but not limited to eye opening and closing EAR value, head Euler angle, and gaze offset), and compares each feature value with the corresponding dimension's personalized threshold in real time.

[0055] Sequential state machine decision: To avoid misjudgments caused by transient interference, the module incorporates decision logic based on time windows. When the eye opening and closing angle remains below the personalized fatigue threshold (e.g., EAR < 0.25) for a preset duration (e.g., 2.0 seconds), a fatigue status indicator is triggered. When the head yaw angle continuously exceeds the distraction threshold (e.g., |yaw|>30°) for a preset duration (e.g., 3.0 seconds), the distraction status flag is triggered; Supports multi-feature composite decision strategies (such as priority handling when fatigue and distraction coexist). Module 103 ultimately outputs a structured driver state assessment result. This result not only includes the state category (e.g., normal, mild fatigue, severe fatigue, distraction) but also associates it with the main characteristics that triggered the state, its duration, or the degree of deviation, so that the subsequent warning system can provide graded and categorized alerts. The logic for state assessment can be defined according to actual needs. For example, if the eye opening and closing degree remains below the personalized fatigue threshold for 2.0 seconds, it can be judged as "mild fatigue"; if it remains below for more than 4.0 seconds, it can be judged as "severe fatigue".

[0056] The vehicle-to-everything (V2X) communication module (T-Box) 104 serves as the gateway connecting the vehicle to external networks. It connects to the internet via a 4G / 5G cellular network. The local processing unit 103 sends anonymized data packets generated during the current driving process (containing feature value sequences, corresponding external environment scene labels, and the final driver driving status label; all information that can directly identify an individual must be anonymized) to the T-Box 104 via the vehicle network. The T-Box 104 then encrypts the data and uploads it to the cloud-based personalized modeling server 105. Conversely, the T-Box 104 also receives personalized baseline threshold configuration files from the cloud 105 and forwards them to the local processing unit 103 for storage in its local cache.

[0057] The cloud-based personalized modeling server 105 is deployed in the data center, possessing large-scale data storage and distributed computing capabilities. Its workflow is as follows: Figure 4 As shown: Step S401, Receive data: Receive anonymized data packets from thousands of vehicles.

[0058] Step S402, Data Cleaning and Classification: Clean the data, remove invalid and outlier values, and then classify the massive data according to the unique driver ID (an anonymous identifier generated by the system, which does not contain real identity information) and external environment scene labels (such as "nighttime highway" and "daytime urban congestion").

[0059] Steps 403 and 404, Model Training: For each “driver-scenario” combination, sufficient feature data accumulated (usually data from stable driving periods when the system does not trigger an alarm, which are regarded as samples of the driver’s normal behavior in that scenario) are used to run a Gaussian mixture model (GMM) for unsupervised probability density estimation, and then the probability distribution representing the normal feature value is fitted to the output.

[0060] Step S405, Threshold Calculation: Calculate a specific low quantile (e.g., the 5th quantile) for this probability distribution. For features such as eye opening and closing angle (EAR), this quantile value is determined as a personalized fatigue threshold (i.e., an EAR value consistently below this threshold indicates fatigue); for features such as head yaw angle, this quantile value is determined as a personalized distraction threshold (i.e., an absolute yaw angle consistently above this threshold indicates distraction). This threshold means that, under normal conditions, the probability that the driver's corresponding feature value is better than (higher for EAR, lower for yaw angle) this threshold is 95%.

[0061] Step S406, Threshold distribution: Package all generated personalized thresholds into a configuration file and distribute it to the vehicle where the driver is located via the network.

[0062] Figure 2 This is a complete workflow diagram of personalized baseline generation and application according to the present invention. The method includes the following steps: When the system first serves a driver, or encounters a "driver-scenario" combination for which a personalized model has not yet been established, the system will enter a cold start phase. At this time, the personalized baseline application module will use a set of pre-defined default thresholds for status determination. These default thresholds are derived from the following two aspects: General population statistical baseline: Based on massive amounts of anonymous driver data, feature thresholds applicable to most drivers were obtained through statistical analysis. Conservative safety principle: To ensure safety in the initial stage, the default threshold is usually set relatively sensitively (e.g., slightly higher than the general statistical threshold) to avoid false negatives before the personalized model is established. The personalized threshold is then optimized and adjusted after data accumulation.

[0063] After the system starts, step S201 involves data acquisition. The in-vehicle perception module 101 and the external environment perception module 102 begin working, continuously acquiring images of the driver's face and dynamic data of the vehicle's external environment.

[0064] In step S202, the scene recognition engine in the local processing unit 103 analyzes the external environment data and identifies the current driving scene (e.g., "nighttime - highway").

[0065] In step S203, the real-time feature extraction module in the local processing unit 103 extracts feature values ​​of eye opening and closing, gaze direction, head posture angle, and yawning frequency from the facial image.

[0066] In step S204, the personalized baseline application module queries the local cache or requests the personalized fatigue and distraction baseline thresholds that match the "current driver ID" and "current external environment scene label" from the cloud personalized modeling server 105 through the vehicle network communication module 104.

[0067] In step S205, the personalized baseline application module compares the real-time extracted feature values ​​with the obtained personalized thresholds, and outputs the driver status judgment result (normal, fatigued, or distracted) according to the preset logic (such as the feature value exceeding the threshold for a certain period of time, for example: exceeding the threshold in M ​​cycles within N consecutive detection cycles).

[0068] Step S206: The data from this driving process (including feature value sequences, corresponding external environment scene labels, and driver driving status labels ultimately recorded by the system or confirmed by the driver) is anonymized, packaged into a data package, and uploaded to the cloud server 105 via T-Box 104.

[0069] In step S207, the cloud server 105 uses a large amount of data collected from a large cluster of vehicles to periodically (e.g., every 24 hours) update the personalized model and thresholds for each driver in different scenarios, and distributes them to the corresponding vehicles via the network to complete closed-loop learning.

[0070] Figure 3 This is a detailed flowchart illustrating the internal workings of the local processing unit 103 of the present invention. The workings of this unit are as follows: Step S301: Receive raw data from the in-vehicle perception module 101 and the external environment perception module 102 in real time.

[0071] Step S302: The scene recognition engine works, based on the image from the forward-facing camera and CAN bus signals (vehicle speed, GPS location, time, light signals), and uses a pre-trained deep learning model to determine the current scene (such as one or more of the following: low-light road section, highway section, urban congested road section, familiar commuting route section).

[0072] Step S303: The real-time feature extraction module operates, based on the DMS camera image, runs the facial key point detection algorithm, and calculates and outputs the feature values ​​of the driver's eye opening degree (EAR), gaze direction, head posture angle, and yawning behavior frequency in real time.

[0073] Step S304: The personalized baseline application module, based on the external environment scene label output in step S302, calls the personalized threshold that is pre-downloaded from the cloud and matches the driver and the scene, which is stored in the local cache. If the threshold is not available locally, it requests it from the cloud via T-Box 104.

[0074] Step S305: Compare the real-time feature value output in step S303 with the personalized threshold obtained in step S304.

[0075] Step S306: Output the driver's state level based on the comparison results. For example, if the eye opening and closing degree remains below the personalized fatigue threshold for 2 seconds, the "fatigue" state is output.

[0076] Step S307: Based on the output status level, trigger the corresponding level of local warning prompt, such as triggering an audible reminder for level one fatigue and triggering a strong vibration seat reminder for level two fatigue.

[0077] Figure 4 This is a detailed flowchart illustrating the internal workflow of the cloud-based personalized modeling server 105 of the present invention. The server's workflow is as follows: Step S401: Receive anonymized data packets uploaded from a large number of vehicles. Each data packet contains a driver ID, a sequence of feature values ​​(such as a set of EAR values), an external environment scene label (such as "nighttime highway"), and a driver driving status label (such as "normal" or "fatigued").

[0078] Step S402: Clean and preprocess the received data, removing outliers caused by signal interference, and strictly classify and store the massive data according to the two key dimensions of "driver ID" and "external environment scene label". For newly registered drivers or "driver-scene" combinations that the system has not yet encountered, due to the lack of historical data, the cloud server will temporarily use a default threshold configuration file based on large-scale population data statistics and distribute it to the vehicle to ensure that the system can work normally in the initial stage. After collecting a sufficient amount of "normal" state data of the driver in the scene (such as continuous normal driving for 10 minutes), the personalized modeling process described below will be started.

[0079] Step S403: For each unique “driver-scenario” combination, extract the sequence of feature values ​​(e.g., all EAR values) collected during the period when the system did not alarm from its historical data, and use these data as the “normal behavior sample” set for that combination.

[0080] Step S404: Using these normal behavior samples as input, run a Gaussian mixture model (GMM) to estimate the probability density and fit the probability distribution of the driver's normal characteristic values ​​in this scenario.

[0081] Step S405: Calculate a specific low quantile (e.g., the 5th quantile) of the probability distribution. Determine the feature value corresponding to this quantile as the driver's personalized fatigue baseline threshold for this scenario (e.g., the EAR value of the 5th quantile as the fatigue threshold). This means that the driver has a 95% probability that their feature value under normal conditions is better than this threshold, and once the feature value is consistently worse than this threshold, it is judged as abnormal.

[0082] Step S406: Package all the generated personalized thresholds into a configuration file and send it to the vehicle where the driver is located via the network for the local processing unit 103 to call, so as to complete the update of the personalized model.

[0083] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0084] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0085] Figure 6A block diagram of a driving condition monitoring device 600 according to an embodiment of the present disclosure is shown. Figure 6 As shown, the device 600 includes: The first acquisition module 610 is used to acquire the driver's current facial image; The recognition module 620 is used to recognize the real-time facial feature values ​​of the driver based on the current facial image; The second acquisition module 630 is used to acquire the current driving scenario of the vehicle; The third acquisition module 640 is used to acquire the corresponding personalized driving state threshold according to the current driving scenario; The determination module 650 is used to determine the driver's driving state based on the real-time facial feature value and the personalized driving state threshold.

[0086] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0087] According to embodiments of the present disclosure, the present disclosure also provides an electronic device and a non-transitory computer-readable storage medium storing computer instructions.

[0088] Figure 7 A schematic block diagram of an electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0089] Device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0090] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0091] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as method 500. For example, in some embodiments, method 500 may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of method 500 described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform method 500 by any other suitable means (e.g., by means of firmware).

[0092] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0093] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0094] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0095] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0096] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0097] Computing systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0098] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0099] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for monitoring driving status, characterized in that, include: Obtain the driver's current facial image; Based on the current facial image, identify the driver's real-time facial feature values; Obtain the current driving scenario of the vehicle; Based on the current driving scenario, obtain the corresponding personalized driving state threshold; The driver's driving status is determined based on the real-time facial feature values ​​and the personalized driving status threshold.

2. The method as described in claim 1, characterized in that, The acquisition of the vehicle's current driving scenario includes: Acquire an image of the road ahead of the vehicle; The vehicle's current dynamic driving data is read via the vehicle's CAN bus; Based on the image of the road ahead and the current dynamic driving data, the current driving scenario is confirmed.

3. The method as described in claim 1, characterized in that, The step of confirming the current driving scenario based on the image of the road ahead and the current dynamic driving data includes: If multiple scenarios are identified based on the road image ahead and the current dynamic driving data; Determine the priority of each scenario among the multiple scenarios; Based on the priority of each scenario, select several scenarios from the plurality of scenarios; The current driving scenario is obtained by combining the aforementioned scenarios.

4. The method as described in claim 1, characterized in that, The real-time facial feature values ​​include: the driver's eye opening degree, the driver's gaze direction, the driver's head posture angle, the driver's mouth shape, and the yawning frequency calculated based on the changes in the mouth shape.

5. The method as described in claim 1, characterized in that, Determining the driver's driving state based on the real-time facial feature values ​​and the personalized driving state threshold includes: If the real-time facial feature value includes multiple facial feature values, then the driving state threshold corresponding to each of the multiple facial feature values ​​is determined from the personalized driving state threshold. The plurality of facial feature values ​​are compared with the corresponding driving state thresholds to obtain the driving state comparison results corresponding to each of the plurality of facial feature values; Determine whether the driving state comparison results corresponding to each of the multiple facial feature values ​​meet the corresponding preset driving state comparison results; If the driving state comparison result corresponding to any of the multiple facial feature values ​​satisfies the corresponding preset driving state comparison result, and the duration of the satisfaction is greater than the preset duration, then the driving state of the driver is determined based on any of the facial feature values.

6. The method as described in claim 5, characterized in that, The method further includes: Based on any of the facial feature values, the driver's state level is output; Based on the driver's status level, a corresponding warning is triggered.

7. The method according to any one of claims 1 to 6, characterized in that, Obtain the corresponding personalized driving status thresholds, including: Obtain the anonymous identity of the driver; The current driving scenario and the anonymous identity are uploaded to the cloud. The cloud obtains each historical driving scenario uploaded by the driver, the facial feature value corresponding to each historical driving scenario, and the anonymous identity. The cloud uses a Gaussian mixture model to fit each historical driving scenario and the facial feature value corresponding to each historical driving scenario to obtain the personalized driving state threshold corresponding to each historical driving scenario of the anonymous identity. The personalized driving state threshold corresponding to each historical driving scenario is stored in correspondence with the anonymous identity. Receive the personalized driving status thresholds sent from the cloud that correspond to the current driving scenario and the anonymous identity.

8. A driving status monitoring device, characterized in that, include: The first acquisition module is used to acquire the driver's current facial image; The recognition module is used to recognize the driver's real-time facial feature values ​​based on the current facial image; The second acquisition module is used to acquire the current driving scenario of the vehicle; The third acquisition module is used to acquire the corresponding personalized driving state threshold based on the current driving scenario. The determination module is used to determine the driver's driving state based on the real-time facial feature values ​​and the personalized driving state threshold.

9. An electronic device, characterized in that, include: Memory and processor The memory stores a computer program, and when the processor executes the program, it implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor corresponding to the electronic device, the electronic device is able to implement the driving state monitoring method as described in any one of claims 1-7.

Images