Methods, devices, equipment, media, and products for identifying driving status

By integrating geometric feature analysis and facial expression recognition into a driver state recognition method, the problem of low accuracy of single-modal solutions in complex environments is solved, achieving accurate recognition and stable output of driver state, thereby improving vehicle safety and user experience.

CN123071982APending Publication Date: 2026-11-10SHENZHEN HONGHE INNOVATION INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611329039.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-28
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing single-modal driver state recognition solutions have low accuracy in complex vehicle environments, which can easily lead to missed or false alarms and fail to trigger timely and reliable active safety interventions.

Method used

By combining geometric feature analysis and facial expression recognition, a state probability distribution is generated, and weighted fusion is performed based on confidence level. Hysteresis state machine and conflict detection rules are introduced to dynamically adjust the fusion weights, ensuring the accuracy and stability of the recognition results.

Benefits of technology

It significantly improves the accuracy and robustness of driving status recognition, avoids false alarms and missed alarms, and enhances the reliability of vehicle active safety systems and user trust.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123071982A_ABST
    Figure CN123071982A_ABST
Patent Text Reader

Abstract

This application relates to the field of computer technology, and particularly to a method, apparatus, device, medium, or product for recognizing driving states. The method includes: acquiring an image frame sequence; performing geometric feature analysis on multiple frames of images including the face of a target person, obtaining the geometric feature analysis result for each frame, and generating a first state probability distribution for the image based on the geometric feature analysis result; performing facial expression recognition processing on each frame to generate a second state probability distribution and a corresponding confidence level for the image; weightedly fusing the first and second state probability distributions based on the confidence level to obtain a state score for each frame; and determining the preset driving state with the highest state score in the image as the driving state of the target person. This method can solve the problem of low accuracy in recognizing driving states in related technologies.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application belongs to the field of computer technology, and in particular relates to a method, device, equipment, medium, or product for identifying driving status. Background Technology

[0002] Real-time driver status monitoring and timely alerts during driving can improve driving safety and the intelligent cockpit interaction experience. Currently, two main approaches are commonly used to perceive driver status: The first approach is based on voice interaction or command recognition, where the vehicle system passively receives and executes control commands issued by the driver. While this approach achieves basic interactive functions, it lacks the ability to actively perceive the driver's current physiological state and cannot intervene when the driver does not input commands. The second approach is a single-modal expression classification scheme based on deep convolutional neural networks. This scheme directly extracts and classifies features from the driver's facial images to output semantic probabilities corresponding to different states. However, in actual in-vehicle applications, in-vehicle cameras often face extreme conditions such as low light at night, drivers turning their faces significantly to the side, direct sunlight, or obscured faces. In these complex environments, the reliability of the deep neural network output drops sharply, leading to frequent misjudgments and a significantly increased false detection rate.

[0003] It is evident that the relevant technologies struggle to accurately and reliably identify a user's true driving state in complex in-vehicle dynamic environments, which can easily lead to missed or false alarms of dangerous states such as fatigue and distraction. Consequently, the vehicle's active safety intervention system cannot be triggered in a timely and reliable manner, posing a safety hazard. Summary of the Invention

[0004] This application provides a method, device, equipment, medium, and product for identifying driving states, which can solve the problem of low accuracy of single-modal recognition schemes under complex driving conditions.

[0005] In a first aspect, embodiments of this application provide a method for identifying driving states, the method comprising: Acquire an image frame sequence, which contains multiple frames including the face of the target person; Geometric feature analysis is performed on multiple frames of images to obtain the geometric feature analysis results of each frame in the multiple frames of images. Based on the geometric feature analysis results, a first state probability distribution of the image is generated. The first state probability distribution is used to predict the probability of the target person in the image under various preset driving states. Perform facial expression recognition processing on each frame of a multi-frame image to generate the second state probability distribution of the image and the corresponding confidence level; The probability distributions of the first and second states are weighted and fused based on the confidence level to obtain the state score of each frame in the multi-frame images. The state score includes the prediction score for each preset driving state among multiple preset driving states. The preset driving state with the highest score in the state score of the image is determined as the driving state of the target person in the image.

[0006] The first aspect of the beneficial effects: By fusing the first-state probability distribution generated from geometric feature analysis with the second-state probability distribution generated from facial expression recognition, heterogeneous data complementarity between low-level physical interpretability and high-level semantic expressiveness is achieved. Simultaneously, the fusion weights are dynamically adjusted based on classification confidence, effectively balancing the reliability differences between the two sources. This solves the problem of missed detections or false positives in complex in-vehicle scenarios using single-modal solutions, thus significantly improving the accuracy and robustness of driving state recognition results.

[0007] In one possible implementation, geometric feature analysis is performed on multiple frames of images to obtain the geometric feature analysis results for each frame in the multiple frames, and a first state probability distribution for the image is generated based on the geometric feature analysis results, including: Extract the static geometric features of each frame in a multi-frame image; Based on the static geometric features of each frame in a multi-frame image and the static geometric features of its preceding image, the dynamic geometric features of the image are determined. The preceding image refers to at least two frames that are temporally preceding and adjacent to the image, and the at least two frames are consecutive. Static and dynamic geometric features are defined as the results of geometric feature analysis; there are multiple static geometric features and multiple dynamic geometric features. For each preset driving state among multiple preset driving states, the geometric features in the geometric feature analysis results are weighted, mapped and normalized to obtain the probability of each preset driving state, and a first state probability distribution is generated.

[0008] In this implementation, dynamic geometric features are determined by introducing static geometric features from previous consecutive frames of images. This makes the generation of the first state probability distribution no longer depend solely on the current single-frame data, but fully incorporates the physical behavior inertia of the driver over a period of time. This effectively eliminates the interference of instantaneous actions (such as brief blinking or rubbing the eyes) on single-frame judgment, and improves the temporal smoothness and noise resistance of geometric state evaluation.

[0009] In one possible implementation, the confidence level includes sub-confidence levels corresponding to each preset driving state; The first and second state probability distributions are weighted and fused based on confidence levels to obtain the state score for each frame in the multi-frame image, including: Obtain the historical smoothing probability distribution and corresponding preset historical weights for each frame in a multi-frame image. The historical smoothing probability distribution is generated based on the state probability distribution of the preceding image of that image. Based on the sub-confidence corresponding to each preset driving state, a first weight and a second weight corresponding to that preset driving state are determined; the first weight is a fusion weight assigned to the probability distribution of the first state, and the second weight is a fusion weight assigned to the probability distribution of the second state. The first probability of the preset driving state in the first state probability distribution is multiplied by the first weight to obtain the first product; The second probability of the preset driving state in the second state probability distribution is multiplied by the second weight to obtain the second product; The third product is obtained by multiplying the historical smooth probability of the preset driving state in the historical smooth probability distribution with the preset historical weight; The predicted score for the preset driving state is determined based on the sum of the first product, the second product, and the third product. A state score is determined based on the predicted score for each preset driving state.

[0010] In this implementation, by configuring independent sub-confidence and corresponding weights for each preset driving state, and combining this with a historical smooth probability distribution based on preceding images, fine-grained weighted fusion at the state level is achieved. This effectively avoids abrupt changes in state scores, ensures temporal consistency of the fused output state scores, and further improves the stability of the system's recognition.

[0011] In one possible implementation, the method further includes: If the geometric feature analysis results of the image and the probability distribution of the second state meet the preset conflict judgment conditions, the state score of the image is corrected to obtain the corrected state score. The preset driving state with the highest score in the state score of this image is determined as the driving state of the target person in this image, including: The preset driving state with the highest score in the corrected state score of the image is determined as the driving state of the target person in the image.

[0012] In this implementation, the state score is forcibly corrected by a preset conflict judgment condition, which can correct the deviation in time when there is a significant logical discrepancy between the geometric physical indicators and the facial expression recognition results, thereby improving the recognition accuracy under complex driving conditions.

[0013] In one possible implementation, multiple preset driving states include at least normal focus and fatigue; the geometric feature analysis results include at least eye opening and closing; the first state probability distribution includes the first probability of each preset driving state, the second state probability distribution includes the second probability of each preset driving state, and the state score includes the predicted score of each preset driving state. If the geometric feature analysis results of the image and the probability distribution of the second state meet the preset conflict judgment conditions, the state score of the image is corrected to obtain the corrected state score, including: If the second probability of a normal state of focus exceeds the first threshold and the duration of eye opening and closing less than the first opening and closing threshold both reach the first duration threshold, then the predicted score of the fatigue state is corrected to the first value, which is the maximum value between the predicted score of the fatigue state and the first probability of the fatigue state. If the duration of the second probability of a normal focused state exceeding the first threshold reaches the first duration threshold, and the proportion of eye opening less than or equal to the eye opening threshold within the preset time window is greater than the preset proportion threshold, then the predicted score of the fatigue state will be corrected to the first value.

[0014] In one possible implementation, the preset driving states also include distraction, anger, and surprise; the geometric feature analysis results also include head deflection angle, geometric change, and number of yawns; If the geometric feature analysis results of the image and the probability distribution of the second state meet the preset conflict judgment conditions, the state score of the image is corrected to obtain the corrected state score, including: If the second probability of the normal focused state exceeds the second threshold and the duration of the head deflection angle being greater than the second angle threshold reaches the second duration threshold, then the predicted score corresponding to the distracted state will be corrected to the second value, which is the maximum value between the predicted score corresponding to the distracted state and the preset probability value. If the second probability of the surprised state exceeds the third threshold and the geometric change is less than the preset change threshold, then the predicted score of the surprised state is corrected to the third value. The third value is the product of the predicted score of the surprised state and the weight reduction weight, which is less than 1. If the confidence level is lower than the low confidence threshold, and the highest value of the second probability in the second state probability distribution is lower than the fourth threshold, then the first state probability distribution is used as the corrected state score.

[0015] In both implementation methods described above, conflict detection and correction rules based on geometric analysis results and facial expression semantic probabilities are constructed for preset states such as normal focus, fatigue, distraction, anger, and surprise. When logical conflicts such as misjudging fatigue, omitting distraction, or misreporting surprise are detected in the facial expression recognition results, the system can forcibly execute geometric probability fallback correction or weight reduction processing, breaking the limitations of simple weighted fusion and significantly enhancing the system's fault tolerance and decision reliability when the confidence of facial expression recognition is insufficient or there are logical anomalies.

[0016] In one possible implementation, the method further includes: If the second probability of the angry state exceeds the fifth threshold, and the number of yawns within the preset time window reaches the first threshold, and the proportion of eye opening less than or equal to the eye opening threshold within the preset time window is greater than the preset proportion threshold, then the one with the higher prediction score between the fatigue state and the angry state is determined as the target person's driving state in the image.

[0017] In this implementation, for scenarios where the driver experiences a combination of fatigue and anger, the higher of the fatigue prediction score and the anger prediction score is determined as the final driving state. This composite determination strategy effectively compensates for the blind spots of a single emotion classifier when faced with multiple overlapping states, improving the system's accuracy in emotion recognition in complex human-computer interaction scenarios.

[0018] In one possible implementation, the method further includes: If the target person is in a fatigued state in the current frame image, and the prediction score of the fatigued state is greater than the first score threshold, and the duration of the fatigued state reaches the first anti-shake duration, then it is determined that the target person has entered a fatigued state, and a fatigue intervention command is output. Alternatively, after determining that the target person has entered a fatigue state, if the duration of the fatigue state's predicted score being less than the second score threshold reaches the second anti-shake duration, then the target person is determined to exit the fatigue state, and a fatigue intervention cancellation command is output. Alternatively, if the target person is in an angry driving state in the current frame image, and the prediction score of the angry state is greater than the third score threshold, and the duration of the angry state reaches the third anti-shake duration, then it is determined that the target person has entered an angry state, and an anger intervention command is output.

[0019] In this implementation, a hysteresis state machine mechanism is introduced to perform time-series counting and determination of the duration of entering and exiting a specific driving state. The corresponding command is only output after the continuous duration of the state meets the first anti-shake duration, or the continuous duration below the threshold meets the second anti-shake duration. This mechanism avoids the problem of frequent bouncing of vehicle intervention commands due to momentary misjudgments, significantly improving the smoothness of the active intervention system's execution in real-world vehicle environments and user trust.

[0020] In one possible implementation, the method further includes: The first time period is the period corresponding to the preset calibration duration after the camera starts capturing images. Based on multiple consecutive calibration images, determine the average eye opening and mouth opening of the target person in a normal state; Based on the average eye opening and closing angle and the average mouth opening and closing angle, a personal geometric feature threshold is generated for the target person. The personal geometric feature threshold is used for geometric feature analysis. Geometric feature analysis is performed on multiple frames of images to obtain the geometric feature analysis results for each frame, including: Geometric feature analysis is performed on multiple frames of images based on individual geometric feature thresholds to obtain the geometric feature analysis results for each frame of the multiple images.

[0021] In this implementation, by generating a unique personal geometric feature threshold for the target person based on multiple consecutive calibration images within a preset calibration period after the camera is turned on, personalized calibration of the driver's eye and mouth physical state is achieved. This setting overcomes the judgment benchmark deviation caused by the differences in the facial features of different individual drivers due to the uniform threshold scheme, making the physical scale of all subsequent geometric feature analyses accurately adapted to the current user, and greatly improving the targeting of feature extraction.

[0022] Secondly, embodiments of this application provide an apparatus, comprising: The acquisition module is used to acquire image frame sequences, which contain multiple frames of images including the face of the target person. The geometric analysis module is used to perform geometric feature analysis on multiple frames of images, obtain the geometric feature analysis results of each frame of the multiple frames of images, and generate the first state probability distribution of the image based on the geometric feature analysis results. The first state probability distribution is used to predict the probability of the target person in the image under various preset driving states. The facial expression recognition module is used to perform facial expression recognition processing on each frame of a multi-frame image, and generate the second state probability distribution of the image and the corresponding confidence level. The fusion module is used to perform weighted fusion of the first state probability distribution and the second state probability distribution according to the confidence level to obtain the state score of each frame in the multi-frame images. The state score includes the prediction score for each preset driving state among multiple preset driving states. The state recognition module is used to identify the preset driving state with the highest state score in the image as the driving state of the target person in the image.

[0023] Thirdly, this application also provides an electronic device. The electronic device includes a memory, one or more processors, and a computer program stored in the memory and executable on the processor. The electronic device executes the computer program to implement any of the implementations of the first aspect described above.

[0024] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method of any of the implementations of the first aspect described above.

[0025] Fifthly, this application also provides a computer program product, including a computer program and a method for implementing any of the implementation methods of the first aspect when the computer program is executed by a processor.

[0026] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect above, and will not be repeated here. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of a driving state recognition method provided in an embodiment of this application; Figure 2 This is a schematic diagram of 468 key points of a face provided in an embodiment of this application; Figure 3 This is a schematic diagram of a driving state recognition processing flow provided in an embodiment of this application; Figure 4 This is a structural block diagram of a driving state recognition device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] This solution provides a method for recognizing driving states, aiming to address the limitations of single-modal recognition methods in terms of limited perception capabilities and insufficient reliability. By fusing facial geometric feature analysis (including static and dynamic geometric features) with expression recognition output across modalities, and dynamically adjusting the fusion weights based on recognition confidence, this solution combines a pre-defined set of conflict resolution rules to forcibly correct contradictory logic. Furthermore, it introduces a hysteresis state machine with entry and exit stabilization durations, thereby enabling accurate and stable output of the target person's state recognition results, including multi-dimensional state probabilities such as normal focus, fatigue, distraction, anger, and surprise.

[0030] This solution can be widely applied to various smart hardware scenarios that require monitoring the status of people or users and have cameras and edge computing capabilities.

[0031] For example, in intelligent driving scenarios, cameras deployed inside vehicles collect images of the driver in real time. This solution processes these images, identifies the driver's driving status, and outputs results, providing a basis for intervention by the vehicle's active safety system to avoid traffic accidents caused by fatigue, distraction, etc. At the same time, it can intelligently adjust the cabin atmosphere based on emotional states such as anger and anxiety, realizing active interaction on the vehicle side.

[0032] In the scenario of a study desk or smart desk lamp, the camera deployed inside the desk lamp (or an external camera) monitors the student's status in real time while doing homework or reading. Once it detects that the student is distracted, fatigued from slouching at the desk for a long time, or irritable, the smart desk lamp can automatically adjust the brightness and color temperature of the light, or guide the student to rest through voice prompts, thereby improving learning efficiency and protecting eyesight.

[0033] In remote education scenarios or online classes, it can monitor students' facial expressions in real time, helping teachers to understand students' concentration, fatigue, or emotional fluctuations. This allows teachers to ask questions, adjust the pace, or issue warnings after class, thereby improving the interactivity and effectiveness of online education.

[0034] In addition, this solution can be extended to application scenarios such as children's companion robots and elderly care for those living alone. By accurately identifying users' emotions and fatigue states, it can achieve humanized interaction and safety protection.

[0035] The specific implementation method of this solution will be described in detail below.

[0036] Figure 1 This is a flowchart of a driving state recognition method provided in an exemplary embodiment of this application. The method can be applied to an in-vehicle terminal, which is connected to an in-vehicle camera to acquire video frames captured by the in-vehicle camera in real time and apply the method to them. The method includes the following steps.

[0037] Step 101: Obtain the image frame sequence.

[0038] The image frame sequence contains multiple frames including the face of the target person. The image frame sequence is a series of consecutive image frames captured by the camera.

[0039] This embodiment uses a driving scenario as an example for specific explanation. In this scenario, the vehicle-mounted camera can continuously capture consecutive images containing the driver at a preset frame rate (e.g., 15 to 30 frames / second, i.e., capturing 15 to 30 images per second), thus obtaining an image frame sequence.

[0040] Each time an image frame is acquired, the method in the subsequent steps of this embodiment is used to analyze the image frame in real time, thereby realizing dynamic recognition and tracking of the driver's driving state.

[0041] It is worth noting that not every frame in an image frame sequence can include the face of the target person. For example, if the target person briefly leaves or turns their head, the captured image may be too blurry to recognize the target person's face.

[0042] If the face of the target person is not recognized, the subsequent processing steps 102 to 105 cannot be performed. In other words, it is meaningless to perform the subsequent processing steps 102 to 105 on images that do not contain the face of the target person. Therefore, when processing the image frame sequence using the method of the subsequent steps of this embodiment, the face recognition processing of the target person must be performed on any image. Only the images that can recognize the face of the target person are used for driving status recognition and detection.

[0043] However, the proposed solution cannot be implemented if only one frame containing the target person's face is detected. Therefore, after obtaining the image frame sequence, face recognition processing is performed on each frame in the image frame sequence to identify multiple frames containing the target person's face, and each frame in the multiple frames is then processed.

[0044] When the image frame sequence is a sequence of images acquired in real time, each acquired image undergoes face recognition processing. If the current acquired image contains the face of the target person, it is determined to belong to a multi-frame image; if the current acquired image does not contain the face of the target person, it is determined not to belong to a multi-frame image.

[0045] If the target person's face is not present in the currently acquired image, and if no target person's face has been detected in any of the K consecutive images in the image frame sequence, including the currently acquired image, then the target person is considered to have left the frame. Driving status recognition and detection for the currently acquired image and subsequent images is then cancelled until a detectable target person's face reappears. K is a positive integer. If the number of consecutive images in the image frame sequence, including the currently acquired image, where no target person's face has been detected is less than K frames, then the target person's face from the last detected image is reused for subsequent detections, ensuring the continuity of the scheme even when a face is temporarily lost.

[0046] In this embodiment, the target person is the driver in the current vehicle's driver's seat. The image of the driver's seat area is captured by the vehicle's onboard camera, and the face detection model is used to detect the face in the image to lock the driver's face area, which serves as the basis for subsequent geometric feature analysis and expression recognition processing.

[0047] As mentioned above, the application scenarios of this application are not limited to driving scenarios, but can be applied to devices with camera and edge computing capabilities, such as smart desk lamps, online learning terminals, and children's companion robots.

[0048] When applied to other scenarios mentioned above, or when the image to be detected contains multiple target individuals, the face recognition algorithm can be used to track and locate each target user sequentially, crop out sub-images containing each face region, and independently call the subsequent geometric feature analysis, expression recognition, weighted fusion, conflict resolution, and hysteresis state machine processes of this embodiment to perform independent state recognition and subsequent intervention decisions for the target individuals in each sub-image.

[0049] For driving scenarios, since the target is usually a single driver, the focus is on recognizing and responding to this single target. The implementation principles for other non-driving scenarios are the same as the core ideas in the embodiments of this application, and will not be elaborated here.

[0050] Step 102: Perform geometric feature analysis on the multi-frame images to obtain the geometric feature analysis results of each frame in the multi-frame images, and generate the first state probability distribution of the image based on the geometric feature analysis results.

[0051] The first state probability distribution is used to predict the probability of a target person in the image being in multiple preset driving states. It is a set of predicted probability values ​​corresponding to each preset driving state. The sum of the predicted probability values ​​of each preset driving state is 1. The higher the predicted probability value of a certain state, the greater the probability that the target person is in that state.

[0052] The purpose of geometric feature analysis is to objectively analyze users' facial closure and head posture from a physical and physiological perspective, thereby inferring their current mental state and fatigue level. However, due to significant individual differences in facial physiological structures such as the eyes, mouth, and skull among different users, using a fixed threshold for uniform judgment can easily lead to systematic bias. For example, even when a user with a larger palpebral fissure is in a state of normal focus, their natural squinting may easily be misjudged as fatigued eye closure by a fixed threshold; similarly, even when a user with a smaller palpebral fissure is experiencing slight fatigue, their natural eye opening may easily be misjudged as normal focus by a fixed threshold. To eliminate the recognition errors caused by these individual physiological differences, this solution can optionally perform personalized calibration in the first phase before formal analysis, customizing exclusive geometric judgment benchmarks for different users, thereby effectively improving the adaptation accuracy and judgment accuracy of geometric feature analysis for different users.

[0053] Optionally, multiple consecutive calibration images of the target person can be acquired within the first time period, which is the time period corresponding to the preset calibration duration after the camera starts acquiring data.

[0054] For example, the preset calibration period is 60 seconds. That is, during the first 60 seconds after the camera starts capturing images, the system does not perform the subsequent state recognition and analysis steps 102 to 105. Instead, it designates this 60-second period as the calibration period and uses the continuous multiple frames of images captured during this period to establish exclusive calibration data for the target person.

[0055] Based on multiple consecutive calibration images, the average eye opening and mouth opening of the target person in a normal state are determined; based on the average eye opening and mouth opening, the corresponding personal geometric feature threshold is generated, and the personal geometric feature threshold is used for geometric feature analysis.

[0056] For example, a face mesh model (such as MediaPipe Face Mesh) is used to output the coordinates of 468 three-dimensional key points of the face, Pi=(xi, yi, zi) (i=1…468); the eye aspect ratio (EAR) and mouth aspect ratio (MAR) are calculated based on the coordinates of the three-dimensional key points, and the individual geometric feature threshold is generated according to the following formula.

[0057] Formula 1: θ_e=μEAR'-a×σ.

[0058] Formula 2: θ_m=μMAR'+b×σ.

[0059] The meanings of the variables in Formula 1 and Formula 2 are as follows.

[0060] μEAR': refers to the average eye opening and closing angle of the target person under normal conditions during the calibration period (first time period). Among them, EAR is the original eye aspect ratio, EAR' is the eye opening and closing angle, which is EAR divided by the target person's face width W, and μEAR' is the average of all EAR' collected during the calibration period.

[0061] μMAR': refers to the average mouth opening and closing angle of the target person in a normal state during the calibration period (first time period). Among them, MAR is the original mouth aspect ratio, MAR' is the mouth opening and closing angle, which is MAR divided by the target person's face width W, and μMAR' is the average of all MAR' collected during the calibration period.

[0062] σ: refers to the standard deviation of the above eye opening and closing degree and mouth opening and closing degree within the calibration period (the first time period), which is used to measure the natural physiological fluctuation range of the target person under normal conditions.

[0063] θ_e: The generated eye opening / closing threshold. When the measured eye opening / closing is lower than this value, the current state is determined to be "closed eyes".

[0064] θ_m: The threshold for the generated mouth opening degree. When the measured mouth opening degree is higher than this value, the current event is determined to be "open mouth (yawning)".

[0065] The eye opening threshold and mouth opening threshold together constitute the personal geometric feature threshold of the target person.

[0066] a and b: These are preset magnification coefficients (preset multipliers). Their function is to incorporate a specific multiple of the standard deviation onto the mean benchmark, so that the threshold can both sensitively capture fatigue characteristics and effectively exclude interference from slight physiological fluctuations and random blinking in the user's normal state. a can be 1.2, and b can be 2.5.

[0067] Optionally, geometric feature analysis is performed on multiple frames of images based on an individual geometric feature threshold to obtain the geometric feature analysis results for each frame in the multiple frames. This process includes the following sub-steps S01-S04.

[0068] Indicative, such as Figure 2 As shown, Figure 2 This is a schematic diagram of 468 key points on a human face. Each key point P corresponds to a different position on the face.

[0069] S01, extract the static geometric features of each frame in the multi-frame image.

[0070] That is, by using 468 three-dimensional key points output by the face mesh model, the physical geometric values ​​such as the opening and closing of the eyes, the opening and closing of the mouth, the head posture, and the facial muscle movement units of the target person in the current single frame are calculated.

[0071] Static geometric features can be derived solely from the data of the current frame of the image and are used to reflect the user's physiological and physical state at that instant.

[0072] S02, determine the dynamic geometric features of the image based on the static geometric features of each frame in the multi-frame image and the static geometric features of its preceding image.

[0073] Among them, the preceding image refers to at least two adjacent frames that are located before the current image in time sequence, and the at least two frames are consecutive.

[0074] It should be noted that the number of preceding images required to determine each dynamic geometric feature is not fixed, but is determined by the length of the time window or statistical requirements for calculating each dynamic geometric feature. Furthermore, not all static geometric features are used to calculate dynamic geometric features.

[0075] In practical implementation, it can be understood that the system maintains a sliding window buffer, stores the static geometric feature results calculated for each frame into this data pool, and combines historical data to complete the statistics of dynamic features.

[0076] S03, static and dynamic geometric features are determined as the results of geometric feature analysis.

[0077] There are multiple static geometric features and multiple dynamic geometric features. The static geometric feature data of a single frame extracted in step S01 is integrated with the dynamic geometric feature data of the preceding image obtained in step S02, and then compiled into a geometric feature analysis result containing multiple geometric features.

[0078] Among them, the geometric feature analysis results include at least the eye opening degree, because the eye opening degree is the most intuitive and core physical feature that characterizes the driver's fatigue state. During driving, the fatigue state is the key identification state in the driving scenario. The driver's actions such as closing his eyes or squinting due to drowsiness will be directly reflected in the numerical change of the eye opening degree. By monitoring this value in real time, the most direct and stable underlying data support can be provided for the subsequent determination of fatigue state.

[0079] In an optional embodiment, the geometric feature analysis results also include head deflection angle, geometric change, and the number of yawns. Head deflection angle can be used to identify distracted states caused by drivers looking down at their phones, turning their heads to look in the rearview mirror, or talking to passengers while driving. Geometric change can be used to capture instantaneous changes in the driver's facial muscles to help identify sudden emotions such as surprise. The number of yawns is an important auxiliary feature for determining whether the driver is drowsy or fatigued; it complements eye opening and closing, jointly improving the accuracy of fatigue detection.

[0080] To more comprehensively depict the physiological and behavioral characteristics of drivers in real-world driving scenarios, this embodiment uses a geometric feature analysis result comprising 20 geometric features (the sum of static and dynamic geometric features) as an example. This geometric feature analysis result can be represented as a 20-dimensional geometric feature vector, where each bit represents the value of a geometric feature. The specific meanings, classifications (static / dynamic), and calculation formulas of these 20 features are explained in detail below.

[0081] I. Static geometric characteristics.

[0082] 1. Eye opening and closing degree EAR': The average of the vertical and horizontal ratio of the left and right eyes divided by the face width W. The smaller the value, the deeper the eyes are closed.

[0083] 2. Mouth Opening Degree MAR': Vertical distance between the lips divided by the face width W. The larger the value, the wider the mouth can open.

[0084] 3~5. Head attitude angles: pitch, yaw, and roll: These are head pitch, yaw, and roll angles calculated using the PnP (Perspective-n-Point) algorithm, and are used to determine actions such as head tilting, turning, and turning.

[0085] 6. Change in eyebrow distance Δbrow: The change in the current eyebrow distance relative to the individual baseline divided by the face width W, used to determine anger features such as frowning.

[0086] 7. Lip asymmetry: The degree of symmetry between the left and right corners of the mouth relative to the vertical center line of the face, divided by the face width W.

[0087] 8-11. Facial Action Unit (AU) approximations: AU12+ (upturned corners of the mouth), AU15+ (downturned corners of the mouth), AU25+ (parted lips, slightly open mouth), AU26+ (drooping jaw, open mouth): The degree of change of facial muscle action units such as upturned corners of the mouth, downturned corners of the mouth, and open mouth is divided by the face width W. The geometric change mentioned earlier refers to the weighted sum of the changes in the approximations of each facial action unit between frames (between the previous and current frames). The weights corresponding to each facial action unit approximation are preset values ​​and are not limited here; the total weight is 1.

[0088] 12. Gaze_off distance: The distance the iris is offset from the center of the pupil divided by the face width W, used to determine the direction of the gaze.

[0089] 13. Squinting degree (eye_squint): The change in distance from the upper eyelid to the eyebrow divided by the face width W, used to help determine emotions such as anger.

[0090] II. Dynamic geometric features.

[0091] 1. Percentage of time with eyes closed (PERCLOS_w): Within a preset sliding window (e.g., 60 seconds), the percentage of time during which the eye opening angle EAR' is less than or equal to the target person's eye opening angle threshold θ_e. This feature is based on the eye-closed state of a single frame.

[0092] 2. Blink rate: The total number of complete "eye-closing-opening" events detected by the statistical system within a preset 60-second time window. This feature is based on the eye opening and closing changes in a single frame.

[0093] 3. Yawn Count (yawn_cnt): Within a preset sliding window, the system counts the number of complete yawning events where the mouth opening angle (MAR') is greater than the target person's mouth opening angle threshold (θ_m). This feature is based on the mouth opening angle of a single frame.

[0094] 4. Continuous eye closure duration (closure_dur): Based on a duration counter, this records the cumulative duration (in seconds) of the current continuous eye closure (i.e., EAR'<θ_e). This feature is the predecessor of the single-frame eye closure state.

[0095] 5. Head Nod Count (nod_cnt): Within a sliding window, counts the cumulative number of consecutive head-down events (accompanied by head-up reset) where the head pitch angle is greater than a preset angle (e.g., 30°). This feature is a precursor to single-frame head pose.

[0096] 6. Head movement amplitude head_move: Calculate the spatial displacement of the nose tip between two adjacent frames and divide it by the face width W.

[0097] 7. Facial microshake variance: Calculates the variance of the difference between adjacent frames in the distance between the eyebrows Δbrow, used to determine the driver's facial microshake (related to anxiety).

[0098] When calculating the aforementioned static geometric features, the eye aspect ratio (EAR) and mouth aspect ratio (MAR) are the most critical feature indicators. To illustrate the calculation process more clearly, the specific formulas are given here, along with explanations of their meanings.

[0099] Formula 3:

[0100]

[0101] Here, EAR is the aspect ratio of the eye, which is essentially calculated by dividing the height of the eye when it is open by the width of the eye. P159 and P158: represent the center points of the upper eyelid margins of the right and left eyes, respectively. P145 and P153: represent the center points of the lower eyelid margins of the right and left eyes, respectively. P33 and P133: represent the key points of the inner and outer corners of the right eye (used to calculate the horizontal span of the right eye, i.e., the eye width). This embodiment uses the right eye as an example. Since the width of the left and right eyes is generally the same, twice the width of the right eye can be regarded as the sum of the width of the left and right eyes.

[0102] W refers to the width in pixels of the driver's face in the current image. It is usually obtained directly from the width of the face detection box or calculated from the key points of the cheekbones on both sides of the face.

[0103] EAR' is the eye opening angle. Dividing EAR by W aims to eliminate the influence of camera distance on the judgment, achieving normalization.

[0104] Formula 4:

[0105]

[0106] MAR is the aspect ratio of the mouth, which is essentially the height of the mouth when it is open divided by the width of the mouth. P13 and P14 represent the center points of the upper and lower lips (midpoints of the upper and lower lips), respectively, used to calculate the vertical distance of the mouth; P61 and P291 represent the left and right corners of the mouth, respectively (used to calculate the horizontal span of the mouth, i.e., the width of the mouth).

[0107] W refers to the width in pixels of the current driver's face in the current image.

[0108] MAR' is the degree of mouth opening. Dividing MAR by W aims to eliminate the influence of camera distance (i.e., image scaling) on ​​the judgment, thus achieving normalization.

[0109] Therefore, in Formulas 3 and 4 above, dividing by the face width W allows EAR' and MAR' to remain stable.

[0110] S04. For each preset driving state among multiple preset driving states, perform weighted mapping and normalization on each geometric feature in the geometric feature analysis results to obtain the probability of each preset driving state and generate the first state probability distribution.

[0111] In other words, when calculating the probability of each preset driving state, some geometric features (including at least one of static and dynamic geometric features) are selected from the geometric feature analysis results according to the type of the preset driving state, and weighted mapping and normalization are performed to obtain the probabilities corresponding to each of the multiple preset driving states and generate the first state probability distribution.

[0112] After obtaining the aforementioned geometric feature analysis results containing 20 dimensions, the system needs to convert these physical quantities into predictable high-level semantic probabilities.

[0113] Among these, the preset driving states include at least normal focus and fatigue. Normal focus is the baseline state for safe driving and also the reference point for the system to determine abnormal states; while fatigue is the most common and most dangerous state during driving, often accompanied by physiological phenomena such as slow reaction and inattention, requiring timely warning and intervention.

[0114] In one optional embodiment, the preset driving states also include distraction, anger, and surprise. Distraction typically corresponds to behaviors such as looking down at a phone, glancing in the rearview mirror, or talking to passengers while driving, which can easily lead to operational errors. Anger corresponds to road rage caused by traffic congestion or being cut off, which can easily induce dangerous driving. Surprise corresponds to the brief emotional fluctuation experienced by the driver when encountering unexpected situations, and can be used to help judge sudden changes in road conditions. The introduction of these states can further enhance the system's ability to perceive complex driving scenarios.

[0115] To more comprehensively depict the driver's emotions and physiological state in real-world driving scenarios, this embodiment uses eight preset driving states as examples: normal focus, fatigue, distraction, anger, surprise, anxiety, sadness, and neutral relaxation. The first state probability distribution corresponds to the predicted probability of each of these eight states.

[0116] First, for the target state k (k takes the value of a positive integer from 1 to 8, representing any of the above 8 states, for example, k=2 is the fatigue state), the raw score (Logit) of the state k is calculated using the following formula 5.

[0117] Formula 5:

[0118] The meanings of the variables in this formula are as follows.

[0119] : The actual value of the i-th geometric feature, where i is a positive integer from 1 to 20. For example, g1 might be the eye opening angle EAR'=0.12, and g2 might be the percentage of time the eyes are closed PERCLOS_w=0.28.

[0120] : Piecewise linear normalization function. Since the 20 geometric features have different physical dimensions (some measure angle, some measure pixels, and some measure time ratio), the system uses this function to flatten all the values ​​into the range of 0 to 1 in order to make fair weighted comparisons.

[0121] : Sparse weight. This represents the "importance" of the i-th geometric feature in determining the k-th state. It's important to note that this weight is sparse; that is, when determining a certain state, the system does not need to refer to all 20 features, but only relies on 2 to 5 of the most critical features (for example, when determining fatigue, the system will heavily rely on indicators such as EAR', PERCLOS_w, and yawn_cnt, while ignoring secondary features such as facial tremors). This weight can be positive (the larger the feature, the more similar it is to the state) or negative (the larger the feature, the less similar it is to the state).

[0122] : Bias (intercept). Used to adjust the default initial tendency score for the k-th state. For example, the system can assign a relatively high bias to the "normal focus" state, while the bias value tends to be close to 0 for other abnormal emotional states.

[0123] : The original score for the k-th state. The higher the score, the more likely the geometric feature recognition results believe that the target person is in that state.

[0124] Due to the above The original score has both positive and negative values, and there are no standard proportional boundaries for the values. In order to establish an absolute comparative relationship between the prediction results of the 8 states, the system inputs the original scores of the above 8 states into the Softmax function for normalization processing, as shown in Formula 6 below.

[0125] Formula 6:

[0126] It refers to the predicted probability of the k-th preset driving state in the first state probability distribution, where the sum of the first probabilities of each preset driving state in the first state probability distribution is 1.

[0127] As mentioned earlier, when calculating the probability of the eight preset driving states, it is not necessary to use all 20 geometric features for each state. Each state only depends on a number of geometric features that are directly related to its physical performance.

[0128] For example, the correspondence between each preset driving state and the specific geometric triggering condition is as follows.

[0129] 1. For the fatigue state, the main geometric triggering conditions are: the proportion of time with eyes closed (PERCLOS_w) is greater than or equal to 0.20; the duration of continuous eye closure (closure_dur) is greater than or equal to 0.8 seconds; the number of yawns (yawn_cnt) within the preset time window is greater than or equal to 1; and the eye opening and closing degree (EAR') is less than 0.18.

[0130] 2. For the distraction state, the main geometric triggering conditions are: the head yaw angle is greater than 25° and lasts for a certain period of time; the gaze offset distance is greater than 0.40; and the head movement amplitude is high.

[0131] 3. For the angry state, the main geometric triggering conditions are: the change in the distance between the eyebrows Δbrow compression is greater than 15%; the degree of squinting (eye_squint) increases; and the degree of mouth opening (MAR') is at a moderate level.

[0132] 4. For the anxiety state, the main geometric triggering conditions are: blink rate greater than 30 times / minute, eye opening and closing angle EAR' remains normal; and facial microshake variance is high.

[0133] 5. For the sad state, the main geometric triggering conditions are: a higher facial motion unit AU15+ and a lower facial motion unit AU12+.

[0134] 6. For the surprise state, the main geometric triggering conditions are: inter-frame bursts of facial motion units AU25+ or AU26+; and inter-frame bursts of eye opening / closing EAR'.

[0135] 7. For the normal focus state, the main geometric triggering conditions are: the eye opening angle EAR', head yaw angle, head pitch angle, and gaze distance gaze_off are all within the normal range.

[0136] 8. For the neutral_relaxed state, the main geometric triggering condition is that all geometric features are close to the narrow band of the target person's personal baseline.

[0137] Step 103: Perform facial expression recognition processing on each frame of the multi-frame images to generate the second state probability distribution of the image and the corresponding confidence level.

[0138] The second-state probability distribution is used to predict the probability of a target person in the image under various preset driving states. The difference between the first-state and second-state probability distributions is that the first-state probability distribution represents the prediction result based on geometric feature analysis (i.e., physical geometric measurement and statistics), while the second-state probability distribution represents the prediction result based on facial expression recognition processing (i.e., deep neural network image semantic classification). Both have the same form, being a set of probability values ​​corresponding to various preset driving states of the same type, with the sum of the predicted probability values ​​for each preset driving state being 1.

[0139] Optionally, each frame of the multi-frame image is input into YOLOv8-cls (or an equivalent lightweight facial expression classification network) for facial expression recognition processing, and x second probabilities (where x is the number of multiple preset driving states) and their corresponding confidence scores are output. The set of x second probabilities is determined as the second state probability distribution.

[0140] The confidence level includes sub-confidence levels corresponding to each preset driving state; the sub-confidence level corresponding to each preset driving state is used to characterize the confidence level of the expression classification network in predicting the second probability corresponding to that preset driving state.

[0141] To further improve the input quality of geometric feature analysis and facial expression recognition while taking into account computational efficiency, the following preprocessing steps can be performed before steps 102 and 103.

[0142] A lightweight object detection network (such as YOLOv8n-face) is used to detect faces in each frame of a multi-frame image, locate the face of the target person, and obtain the face bounding box. Each frame of the image is cropped according to the face bounding box, and the cropped image containing the face of the target person is scaled to a first preset resolution (e.g., 192×192 pixels) and a second preset resolution (e.g., 224×224 pixels) to obtain a first region image and a second region image.

[0143] In this process, the first region image is used as input for geometric feature analysis in step 102 to extract static and dynamic geometric features; the second region image is used as input for facial expression recognition processing in step 103, so that the facial expression classification network can perform facial expression classification prediction. Since steps 102 and 103 process two region images with different resolutions but the same content source, they can be executed in parallel. Therefore, in specific implementations, the execution order of steps 102 and 103 can be arbitrary and does not restrict each other.

[0144] It is worth noting that steps 102 and 103 are two independent steps that can be implemented separately or simultaneously. Therefore, the execution order of steps 102 and 103 can be arbitrary.

[0145] Step 104: The first state probability distribution and the second state probability distribution are weighted and fused according to the confidence level to obtain the state score of each frame in the multi-frame images.

[0146] The state score includes a predicted score for each of the various preset driving states.

[0147] Optionally, the historical smoothing probability distribution of each frame in the multi-frame images and the corresponding preset historical weights are obtained. The historical smoothing probability distribution is generated based on the state probability distribution of the preceding images of that image.

[0148] Based on the sub-confidence level corresponding to each preset driving state, a first weight and a second weight are determined for that preset driving state. The first weight is a fusion weight assigned to the probability distribution of the first state, and the second weight is a fusion weight assigned to the probability distribution of the second state. The first probability of the preset driving state in the probability distribution of the first state is multiplied by the first weight to obtain a first product; the second probability of the preset driving state in the probability distribution of the second state is multiplied by the second weight to obtain a second product; the historical smoothed probability of the preset driving state in the historical smoothed probability distribution is multiplied by a preset historical weight to obtain a third product; the sum of the first, second, and third products is used to determine the predicted score of the preset driving state; and the state score is determined based on the predicted score of each preset driving state.

[0149] Thus, we obtained three different dimensions of prediction results: the first state probability distribution, the second state probability distribution, and the state score. For ease of understanding, we will uniformly name the specific quantitative values ​​for each preset driving state in the above three output results: the predicted probability corresponding to each preset driving state in the first state probability distribution is called the "first probability"; the predicted probability corresponding to each preset driving state in the second state probability distribution is called the "second probability"; and the final score corresponding to each preset driving state in the state score is called the "prediction score".

[0150] That is, the first state probability distribution includes the first probability of each preset driving state, the second state probability distribution includes the second probability of each preset driving state, and the state score includes the predicted score of each preset driving state.

[0151] For example, the first weight and the second weight mentioned above can be implemented by the following formula 7.

[0152] Formula 7:

[0153]

[0154] The meanings of the variables and symbols in Formula 7 above are as follows.

[0155] : Represents the first weight (i.e., the fusion weight assigned to the first state probability distribution, used to weight the first probability output by the geometric feature analysis).

[0156] : indicates the second weight (i.e. the fusion weight assigned to the second state probability distribution, used to weight the second probability of the expression recognition output).

[0157] : This represents the preset minimum value of the second weight. When the confidence level of the expression recognition network output is extremely low, the second weight will decay to this value, at which point the first weight of the geometric features will dominate.

[0158] : Indicates the preset maximum value of the second weight. When the confidence level of the expression recognition network output is extremely high, the second weight will rise to this value, at which point the second weight of expression recognition will dominate.

[0159] : Represents the Sigmoid activation function. This function generates a smooth S-shaped curve based on the confidence level of the input, preventing abrupt changes in weights and thus achieving a smooth transition from geometry-driven to expression recognition-driven approaches.

[0160] : Represents the steepness adjustment coefficient. Used to control the slope of the Sigmoid function curve near the confidence threshold (i.e., how fast the weight transitions).

[0161] : Represents the confidence level of the facial expression recognition network output.

[0162] : Indicates the confidence threshold (inflection point). When the confidence level is below this threshold, the system tends to decrease the second weight (i.e., decrease the weight of AI) and increase the first weight (i.e., increase the weight of geometry); when the confidence level is above this threshold, the system tends to increase the second weight and decrease the first weight.

[0163] Based on Formula 7 above, by setting the sum of the first weight and the second weight to 1, the weights can be adaptively adjusted between geometric physical analysis (first probability) and facial expression semantic judgment (second probability) according to the real-time confidence level of the facial expression recognition network. This ensures that the fused state score has optimal robustness and reliability under different image quality and driving conditions.

[0164] In one alternative implementation, =0.2, =0.75, =0.55, =12.

[0165] For example, the weighted fusion of the probability distributions of the first state and the probability distributions of the second state based on the first weight and the second weight can be calculated using the following formula 8.

[0166] Formula 8:

[0167] The meanings of the variables and symbols in Formula 8 above are as follows.

[0168] : Represents the final state score (predicted score) corresponding to the k-th preset driving state. k is a positive integer between 1 and 8, corresponding to the 8 preset driving states defined above.

[0169] Second weight.

[0170] First weight.

[0171] : Represents the first probability corresponding to the k-th preset driving state (the first state probability distribution output from the geometric feature analysis in step 102).

[0172] : Represents the second probability corresponding to the k-th preset driving state (from the second state probability distribution output by the expression classification network in step 103).

[0173] : Indicates the preset historical weight. Used to control the buffering and influence of historical frame state determination results on the current frame prediction score. This parameter is a fixed constant preset by the system. In an optional embodiment, this value can be set to 0.15.

[0174] : Represents the historical smoothing probability, which is the smoothing probability value corresponding to the k-th preset driving state in the historical smoothing probability distribution obtained by the system. This value is calculated internally by the system using a historical sliding window smoothing algorithm.

[0175] The historical smoothing probability is generated based on the state probability distribution of the preceding images of the current frame. In other words, the historical smoothing probability is generated based on the state scores of the preceding images of the current frame. For example, a historical sliding window of length T frames is maintained in memory, where T corresponds to a time range of 0.5 seconds to 3 seconds (e.g., 15 to 90 frames in a 30 FPS historical buffer). For each preset driving state, the system reads the state score changes over the past T frames and uses either the Exponential Moving Average (EMA) algorithm or a Gated Recurrent Unit (GRU) neural network to model the temporal sequence, ultimately smoothing the historical smoothing probability distribution of the current frame. This feature effectively eliminates the problem of drastic jumps in prediction scores caused by occasional misjudgments in a single frame, greatly improving the stability of the state output and the consistency between consecutive frames.

[0176] Through the aforementioned fusion calculation process, a state score is obtained for each frame of the image. This state score can comprehensively characterize the predictability of the target person under various preset driving states. However, in some embodiments, there may be unreasonable situations where the geometric feature analysis results and the facial expression recognition processing results clearly violate objective physical facts, leading to missed or false alarms in the final driving state judgment. To address these situations and ensure the reliability of the state recognition results, this embodiment further introduces a conflict determination mechanism, which can filter out some obviously unreasonable situations and correct them.

[0177] Optionally, if the geometric feature analysis results of the image and the probability distribution of the second state meet the preset conflict judgment conditions, the state score of the image is corrected to obtain the corrected state score.

[0178] Among them, the results of geometric feature analysis include eye opening and closing, head turning angle, geometric change, number of yawns, etc. These geometric features directly constitute the core indicators for judging whether there is a significant deviation between physical facts and facial expression semantics, and are closely related to the triggering of the preset conflict judgment conditions in this step.

[0179] For example, the preset conflict judgment conditions and corresponding correction processes include the following typical cases.

[0180] C1. If the second probability of a normal focused state exceeds a first threshold and the duration of eye opening less than a first eye opening threshold reaches a first duration threshold, then the predicted score of the fatigue state is corrected to a first value, which is the maximum value between the predicted score of the fatigue state and the first probability of the fatigue state. Alternatively, if the duration of the second probability of a normal focused state exceeding the first threshold reaches a first duration threshold, and the proportion of eye opening less than or equal to the eye opening threshold within a preset time window is greater than a preset proportion threshold, then the predicted score of the fatigue state is corrected to the first value.

[0181] For example, the probability of detecting a target person in the current frame as being in a "normal focused" state is 0.6 (exceeding a preset first threshold such as 0.5), while the geometric feature analysis shows that the current eye opening angle EAR' = 0.10 (less than a preset first opening angle threshold such as 0.18), and both have lasted for more than 0.8 seconds (the first duration threshold). At this point, the expression recognition algorithm outputs a result indicating the target person is in a normal state, but the geometric data clearly indicates the eyes are almost completely closed. Determining a conflict between the two, the predicted score for fatigue is raised to the geometrically calculated level to avoid missing fatigue warnings.

[0182] For example, if the probability of detecting a "normal focused" state of the target person in the current frame exceeds 0.5, and the focused state has lasted for more than 0.8 seconds (the first duration threshold), and the sliding window over the past 60 seconds is analyzed, it is determined that the percentage of time (PERCLOS_w) during which the target person's eye opening is below the eye opening threshold reaches 0.18 (exceeding the preset percentage threshold of 0.15). In this case, even if the expression recognition algorithm misjudges the target person as normal, it can still raise the fatigue score to a geometric probability level, preventing the target person from being overlooked due to "prolonged eye closure".

[0183] C2. If the second probability of the normal focused state exceeds the second threshold and the duration of the head deflection angle being greater than the second angle threshold reaches the second duration threshold, then the predicted score corresponding to the distracted state is corrected to the second value, which is the maximum value between the predicted score corresponding to the distracted state and the preset probability value.

[0184] For example, if the probability of detecting a target person in the current frame being in a "normal focused" state exceeds the second threshold, but the geometric features simultaneously detect that the target person's head yaw angle |yaw| reaches 30 degrees (greater than the preset second angle threshold, such as 25 degrees), and both last for more than 0.5 seconds (the second duration threshold), this usually corresponds to the target person turning their head and not focusing on the road conditions. In this case, the facial expression recognition algorithm is deemed to have missed a detection, and the predicted score for the distracted state is forcibly corrected to the preset probability value (such as 0.6) to ensure that no false positives are detected when the target person is distracted.

[0185] C3. If the second probability of the surprised state exceeds the third threshold and the geometric change is less than the preset change threshold, then the predicted score of the surprised state is corrected to the third value. The third value is the product of the predicted score of the surprised state and the weight reduction weight, which is less than 1.

[0186] For example, the probability of detecting a surprised state output by the facial expression recognition algorithm is as high as 0.75 (exceeding the preset third threshold, such as 0.7). However, the geometric feature analysis results show that the overall change in facial motion units in the current frame is extremely low (less than the preset change threshold), indicating that the facial muscles of the target person have not undergone obvious stretching or exaggerated expressions. In this case, it is considered that the facial expression recognition algorithm has misjudged normal speech or slight micro-movements as "surprise," and a weight reduction correction is immediately triggered, multiplying the prediction score of the surprised state by 0.5 (weight reduction weight), thereby effectively reducing the probability of misjudgment.

[0187] C4. If the confidence level is lower than the low confidence threshold, and the highest value of the second probability in the second state probability distribution is lower than the fourth threshold, then the first state probability distribution is used as the corrected state score.

[0188] For example, in low light conditions at night, when the target person's face is turned significantly to the side, or when the face is obscured, the overall confidence level of the expression recognition algorithm may drop to 0.3 (below the preset low confidence threshold of 0.4), and the highest probability in the second state probability distribution of that frame may also be less than 0.4 (below the fourth threshold). This indicates that the expression recognition algorithm's output in the current frame is unreliable. In this case, the expression recognition algorithm's judgment result is discarded, and the first state probability distribution of that frame (the result of pure geometric measurement) is used directly as the corrected state score.

[0189] C5. If the second probability of the angry state exceeds the fifth threshold, and the number of yawns within the preset time window reaches the first threshold, and the proportion of eye opening and closing less than or equal to the eye opening and closing threshold within the preset time window is greater than the preset proportion threshold, then the one with the higher prediction score between the fatigue state and the angry state is determined as the driving state of the target person in the image.

[0190] For example, if the probability of detecting anger in the target character in the current frame exceeds 0.6 (exceeding the fifth threshold), and the number of yawns within the geometric feature statistics window reaches 2 (reaching the first threshold of 1), and the percentage of eyes with an opening angle below the eye opening angle threshold in the past window (PERCLOS_w) exceeds 0.15 (the preset percentage threshold), this indicates that the target character may be in a combined state of fatigue and anger. In this case, the fatigue score and the anger score are compared, and the higher score is taken as the final comprehensive driving state output.

[0191] It is worth noting that the preset time window mentioned in this application refers to a statistical window that slides continuously on the timeline, used to accumulate and statistically analyze the specific geometric feature states of a target person within a historical time range. For different geometric features, the corresponding preset time window for determining driving status can be configured with specific window durations according to actual needs, and does not need to be exactly the same.

[0192] For illustrative purposes, please refer to Table 1 below, which covers the conflict judgment conditions C1 to C5 and their corresponding correction methods.

[0193] Table 1

[0194] Step 105: Determine the preset driving state with the highest score in the state score of the image as the driving state of the target person in the image.

[0195] Optionally, if the geometric feature analysis results of the image and the probability distribution of the second state meet the preset conflict judgment conditions, the state score of the image is corrected. After obtaining the corrected state score, the preset driving state with the highest score in the corrected state score of the image is determined as the driving state of the target person in the image.

[0196] That is, if the image does not trigger the conflict judgment condition, the preset driving state with the highest score in the state score output in step 104 is directly used as the driving state of the frame. If the image has triggered the conflict judgment condition and has been corrected, the preset driving state with the highest score in the corrected state score is used as the driving state of the target person in the image.

[0197] Given that both fatigue and anger are directly related to driving safety and require timely intervention, this embodiment sets up the following three anti-shake judgment logics for these two states.

[0198] For determining the entry of fatigue state: if the target person is in a fatigue state in the current frame image, and the predicted score of the fatigue state is greater than the first score threshold (e.g., 0.5), and the duration of the fatigue state reaches the first anti-shake duration (e.g., 0.8 seconds), then it is determined that the target person has entered a fatigue state, and a fatigue intervention command is output.

[0199] The calculation logic for the duration of the fatigue state is as follows: After determining that the current frame is in a fatigue state, backtrack frame by frame in the temporal sequence from the current frame to determine the image frame in the nearest non-fatigue state. Then, accumulate the image acquisition time spans corresponding to all consecutive fatigue frames between the current frame and the non-fatigue state image frame; the total duration is the duration of the fatigue state. If no non-fatigue state has appeared in the preceding image frames, the duration is calculated starting from the first frame of the fatigue state sequence.

[0200] Alternatively, for the exit determination of fatigue state: after determining that the target person has entered a fatigue state, if the duration of the predicted score of the fatigue state is less than the second score threshold (e.g., 0.3) reaches the second anti-shake duration (e.g., 3 seconds), then it is determined that the target person has exited the fatigue state, and a fatigue intervention cancellation command is output.

[0201] The calculation logic for the duration during which the predicted fatigue score is less than the second score threshold is as follows: After entering fatigue intervention, continuous detection continues in the subsequent direction, frame by frame. Once a frame's predicted fatigue score is found to be lower than the second score threshold, the system starts a "recovery counter" to accumulate the duration of consecutive frames below the threshold. If, during this process, a frame's predicted fatigue score is detected to be higher than the second score threshold again, the counter is immediately reset to zero and starts accumulating again. Only when the accumulated duration of consecutive frames reaches the second anti-shake duration (e.g., 3 seconds) does the system finally confirm that the target has exited the fatigue state and output a release command.

[0202] Alternatively, for determining the entry of an angry state: if the target person is in an angry state while driving in the current frame image, and the prediction score of the angry state is greater than the third score threshold (e.g., 0.55), and the duration of the angry state reaches the third anti-shake duration (e.g., 2 seconds), then it is determined that the target person has entered an angry state, and an anger intervention command is output.

[0203] The calculation logic for the duration of the angry state can refer to the entry judgment logic of the aforementioned fatigue state, that is, by backtracking the previous consecutive frames to calculate the time span between frames and accumulate it; when a non-angry state frame appears in the middle, the counter will be cleared and re-accumulated until the requirement of the third anti-shake duration (such as 2 seconds) is met.

[0204] For example, the above intervention commands are used to instruct the vehicle's control system to perform corresponding response actions according to different driving states.

[0205] 1. Fatigue intervention command, used to instruct the vehicle to activate fatigue warning response, the specific forms of which include but are not limited to: broadcasting voice reminders through the vehicle audio output unit; displaying a fatigue warning icon on the human-machine interface; automatically opening windows or adjusting air conditioning ventilation to help the driver regain alertness; or controlling the vehicle to decelerate smoothly while the driver assistance function is in operation.

[0206] 2. Fatigue intervention cancellation command, used to instruct the vehicle to cancel the preceding fatigue warning response, which may include, but is not limited to: turning off the voice reminder and warning interface display, and restoring the vehicle's various interactive display interfaces to normal driving mode.

[0207] 3. Anger intervention command, used to instruct the vehicle to perform an emotion-soothing response, which may include, but is not limited to: automatically switching to soothing music playback; adjusting the ambient lighting in the car to a soft, warm tone; activating the in-car fragrance system to release a calming aroma; or displaying a soothing prompt on the interactive interface containing text such as "It is recommended to drive calmly".

[0208] In summary, this embodiment provides a method for recognizing driving states. By fusing a first state probability distribution generated from geometric feature analysis with a second state probability distribution generated from facial expression recognition, it achieves heterogeneous data complementarity between low-level physical interpretability and high-level semantic expressiveness. Simultaneously, it dynamically adjusts the fusion weights based on classification confidence, effectively balancing the reliability differences between the two sources. Furthermore, by forcibly correcting the state scores through preset conflict judgment conditions, it can promptly correct deviations when there are significant logical discrepancies between geometric physical indicators and facial expression recognition results. This solves the problem of missed detections or misjudgments in complex vehicle scenarios using single-modal solutions, thereby significantly improving the accuracy and robustness of driving state recognition results.

[0209] Indicative, such as Figure 3 As shown, Figure 3 This is a schematic diagram of a driving state recognition and processing flow, including the following steps S1-S9, and the processing logic between each step is as follows.

[0210] S1: Image acquisition.

[0211] The vehicle-mounted camera continuously captures real-time video frames containing the driver's face at a preset frame rate (corresponding to step 101).

[0212] S2: Face detection and image preprocessing.

[0213] Face detection is performed on the acquired images, face bounding boxes are located, face ROIs (Regions of Interest) are cropped, and the images are scaled to different preset resolutions to obtain a first region image with a resolution of 192×192 and a second region image with a resolution of 224×224. These images are used for subsequent splitting processing, corresponding to the preprocessing steps 102 and 103.

[0214] The first region image with a resolution of 192×192 is output to stage S3, and the second region image with a resolution of 224×224 is output to stage S4.

[0215] S3: Geometric feature recognition and processing.

[0216] Corresponding to step 102, the static and dynamic geometric features of the target person are extracted, and the first state probability distribution is generated after weighted mapping and normalization.

[0217] S4: Facial expression recognition and processing.

[0218] In step 103, the scaled image is identified using an expression classification network, and the second state probability distribution and the corresponding confidence level are output.

[0219] S3 and S4 are processed in parallel.

[0220] S5: Dynamic fusion.

[0221] In step 104, the parallel output results of S3 and S4 are received. Based on the confidence level of the output of S4, the first probability distribution, the second probability distribution, and the historical smoothing term are dynamically weighted and fused to obtain the state score of the current frame.

[0222] S6, Conflict Detection.

[0223] Step S6 corresponds to the conflict determination content in step 104 mentioned above. It determines whether the result of geometric feature analysis and the result of expression recognition meet the preset conflict triggering conditions (i.e., the rule set C1 to C5 mentioned above). If the judgment conditions are met ("Yes" branch), then proceed to S7 for conflict correction; if the judgment conditions are not met ("No" branch), then skip S7 and send the state score output by S5 directly to S8.

[0224] S7, Conflict Correction.

[0225] Based on the specific conflict rules matched, the corresponding forced correction action is executed (such as taking the maximum value, reducing the weight, or directly using the geometric probability as a catch-all), and the corrected state score is output. Refer to Table 1, and then proceed to S8.

[0226] S8, fatigue and anger trigger exit judgment.

[0227] Receive the final state score output by S6 or S7. Utilize the built-in hysteresis state machine to perform a continuous duration anti-shake determination on the predicted score of the fatigue state or anger state (e.g., determine whether fatigue entry lasts for a first anti-shake duration of 0.8 seconds, and whether fatigue exit lasts for a second anti-shake duration of 3 seconds), corresponding to the anti-shake determination in step 105.

[0228] S9, Output the result.

[0229] Output the final judgment result of the target person's current driving status or the corresponding control intervention command, and send it to the vehicle's human-machine interface or domain controller for the vehicle to execute subsequent active safety intervention or emotional calming response, corresponding to step 105.

[0230] Corresponding to the driving state recognition method in the above embodiments, Figure 4 A structural block diagram of a driving state recognition device provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0231] Reference Figure 4 The device 400 includes: The acquisition module 410 is used to acquire an image frame sequence, wherein the image frame sequence contains multiple frames of images including the face of the target person; The geometric analysis module 420 is used to perform geometric feature analysis on multiple frames of images, obtain the geometric feature analysis results of each frame of the multiple frames of images, and generate the first state probability distribution of the image based on the geometric feature analysis results. The first state probability distribution is used to predict the probability of the target person in the image under various preset driving states. The facial expression recognition module 430 is used to perform facial expression recognition processing on each frame of a multi-frame image, and generate the second state probability distribution of the image and the corresponding confidence level. The fusion module 440 is used to perform weighted fusion of the first state probability distribution and the second state probability distribution according to the confidence level to obtain the state score of each frame in the multi-frame images. The state score includes the prediction score for each preset driving state among multiple preset driving states. The state recognition module 450 is used to determine the preset driving state with the highest state score in the image as the driving state of the target person in the image.

[0232] It should be noted that the information interaction and execution process between the above-mentioned devices / modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0233] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0234] To implement the above embodiments, this application also proposes an electronic device.

[0235] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application.

[0236] like Figure 5 As shown, the above-mentioned electronic device 500 includes: The system includes a memory 510 and at least one processor 520, and a bus 530 connecting different components (including the memory 510 and the processor 520). The memory 510 stores a computer program, and when the processor 520 executes the program, it implements the driving state recognition method of the present application embodiment.

[0237] Bus 530 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0238] Electronic device 500 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by electronic device 500, including volatile and non-volatile media, removable and non-removable media.

[0239] Memory 510 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 540 and / or cache memory 550. Electronic device 500 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 560 can be used to read and write non-removable, non-volatile magnetic media (… Figure 5 Not shown; usually referred to as a "hard drive"). Although Figure 5 As not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 530 via one or more data media interfaces. Memory 510 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0240] A program / utility 580 having a set (at least one) of program modules 570 may be stored, for example, in memory 510. Such program modules 570 include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 570 typically perform the functions and / or methods described in the embodiments of this application.

[0241] Electronic device 500 can also communicate with one or more external devices 590 (e.g., keyboard, pointing device, display 591, etc.), and with one or more devices that enable a user to interact with electronic device 500, and / or with any device that enables electronic device 500 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 595. Furthermore, electronic device 500 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 593. As shown, network adapter 593 communicates with other modules of electronic device 500 via bus 530. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0242] The processor 520 performs various functional applications and data processing by running programs stored in the memory 510.

[0243] It should be noted that the implementation process and technical principles of the electronic device in this embodiment are explained in the foregoing description of the driving state identification method in the embodiments of this application, and will not be repeated here.

[0244] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0245] This application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the steps described in the various method embodiments above.

[0246] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed by a processor, it implements the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some regions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0247] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0248] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0249] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0250] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0251] In the foregoing, specific details such as particular system architectures and techniques have been set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted to avoid unnecessary detail from obscuring the description of this application.

[0252] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0253] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0254] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0255] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0256] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0257] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for recognizing driving states, characterized in that, The method includes: Obtain an image frame sequence, wherein the image frame sequence contains multiple frames of images including the face of the target person; Geometric feature analysis is performed on the multi-frame images to obtain the geometric feature analysis results of each frame in the multi-frame images, and a first state probability distribution of the image is generated based on the geometric feature analysis results. The first state probability distribution is used to predict the probability of the target person in the image under various preset driving states. Perform facial expression recognition processing on each of the multi-frame images to generate the second state probability distribution of the image and the corresponding confidence level; The first state probability distribution and the second state probability distribution are weighted and fused according to the confidence level to obtain the state score of each frame in the multi-frame images. The state score includes the prediction score for each preset driving state among the multiple preset driving states. The preset driving state with the highest score in the state score of the image is determined as the driving state of the target person in the image.

2. The method according to claim 1, characterized in that, The step of performing geometric feature analysis on the multi-frame images to obtain the geometric feature analysis results for each frame in the multi-frame images, and generating a first state probability distribution for the image based on the geometric feature analysis results, includes: Extract the static geometric features of each frame in the multi-frame image; Based on the static geometric features of each frame in the multi-frame image and the static geometric features of its preceding image, the dynamic geometric features of the image are determined. The preceding image refers to at least two frames that are sequentially preceding and adjacent to the image, and the at least two frames are consecutive. The static geometric features and the dynamic geometric features are determined as the geometric feature analysis results; there are multiple static geometric features and multiple dynamic geometric features; For each of the multiple preset driving states, the geometric features in the geometric feature analysis results are weighted, mapped, and normalized to obtain the probability of each preset driving state, and the first state probability distribution is generated.

3. The method according to claim 1, characterized in that, The confidence level includes the sub-confidence level corresponding to each preset driving state; The step of weightedly fusing the first state probability distribution and the second state probability distribution based on the confidence level to obtain the state score of each frame in the multi-frame images includes: The historical smoothing probability distribution and the corresponding preset historical weight of each frame in the multi-frame image are obtained. The historical smoothing probability distribution is generated based on the state probability distribution of the preceding image of the image. Based on the sub-confidence corresponding to each preset driving state, a first weight and a second weight corresponding to that preset driving state are determined; the first weight is a fusion weight assigned to the probability distribution of the first state, and the second weight is a fusion weight assigned to the probability distribution of the second state. The first probability of the preset driving state in the first state probability distribution is multiplied by the first weight to obtain the first product; The second probability of the preset driving state in the second state probability distribution is multiplied by the second weight to obtain the second product; The preset driving state is multiplied by the historical smooth probability in the historical smooth probability distribution and the preset historical weight to obtain a third product; The predicted score for the preset driving state is determined based on the sum of the first product, the second product, and the third product. The state score is determined based on the predicted score for each preset driving state.

4. The method according to claim 1, characterized in that, The method further includes: If the geometric feature analysis results of the image and the probability distribution of the second state meet the preset conflict judgment conditions, the state score of the image is corrected to obtain the corrected state score. The step of determining the preset driving state with the highest score in the state score of the image as the driving state of the target person in the image includes: The preset driving state with the highest score in the corrected state score of the image is determined as the driving state of the target person in the image.

5. The method according to claim 4, characterized in that, The multiple preset driving states include at least normal focus and fatigue; the geometric feature analysis results include at least eye opening and closing; the first state probability distribution includes a first probability of each preset driving state, the second state probability distribution includes a second probability of each preset driving state, and the state score includes a predicted score of each preset driving state. If the geometric feature analysis result of the image and the probability distribution of the second state meet the preset conflict judgment conditions, the state score of the image is corrected to obtain the corrected state score, including: If the second probability of a normal focused state exceeds the first threshold and the duration of the eye opening degree being less than the first opening degree threshold reaches the first duration threshold, then the predicted score of the fatigue state is corrected to the first value, where the first value is the maximum value between the predicted score of the fatigue state and the first probability of the fatigue state. If the duration of the second probability of the normal focus state exceeding the first threshold reaches the first duration threshold, and the proportion of the eye opening degree being less than or equal to the eye opening degree threshold within the preset time window is greater than the preset proportion threshold, then the predicted score of the fatigue state is corrected to the first value.

6. The method according to claim 5, characterized in that, The preset driving states also include distraction, anger, and surprise; the geometric feature analysis results also include head deflection angle, geometric change, and number of yawns. If the geometric feature analysis result of the image and the probability distribution of the second state meet the preset conflict judgment conditions, the state score of the image is corrected to obtain the corrected state score, including: If the second probability of the normal focused state exceeds the second threshold and the duration of the head deflection angle being greater than the second angle threshold reaches the second duration threshold, then the predicted score corresponding to the distracted state is corrected to the second value, which is the maximum value between the predicted score corresponding to the distracted state and the preset probability value. If the second probability of the surprised state exceeds the third threshold, and the geometric change is less than the preset change threshold, then the predicted score of the surprised state is corrected to the third value, which is the product of the predicted score of the surprised state and the weight reduction weight, and the weight reduction weight is less than 1. If the confidence level is lower than the low confidence threshold, and the highest value of the second probability in the second state probability distribution is lower than the fourth threshold, then the first state probability distribution is used as the corrected state score.

7. The method according to claim 6, characterized in that, The method further includes: If the second probability of the angry state exceeds the fifth threshold, and the number of yawns within the preset time window reaches the first threshold, and the proportion of the eye opening degree being less than or equal to the eye opening degree threshold within the preset time window is greater than the preset proportion threshold, then the one with the higher predicted score between the fatigue state and the angry state is determined as the driving state of the target person in the image.

8. The method according to claim 1, characterized in that, The method further includes: If the target person is in a fatigued state while driving in the current frame image, and the predicted score of the fatigued state is greater than the first score threshold, and the duration of the fatigued state reaches the first anti-shake duration, then it is determined that the target person has entered a fatigued state, and a fatigue intervention command is output. Alternatively, after determining that the target person has entered a fatigue state, if the duration of the predicted score of the fatigue state being less than the second score threshold reaches the second anti-shake duration, then the target person is determined to exit the fatigue state, and a fatigue intervention release command is output. Alternatively, if the target person is in an angry driving state in the current frame image, and the prediction score of the angry state is greater than the third score threshold, and the duration of the angry state reaches the third anti-shake duration, then it is determined that the target person has entered an angry state, and an anger intervention command is output.

9. The method according to claim 1, characterized in that, The method further includes: During the first time period, a series of calibration images of the target person are acquired. The first time period is the time period corresponding to the preset calibration duration after the camera starts acquiring images. Based on the continuous multi-frame calibration images, the average eye opening and closing angle and the average mouth opening and closing angle of the target person in a normal state are determined; Based on the average eye opening and closing angle and the average mouth opening and closing angle, a personal geometric feature threshold corresponding to the target person is generated, and the personal geometric feature threshold is used for geometric feature analysis; The geometric feature analysis of the multi-frame images, to obtain the geometric feature analysis results of each frame in the multi-frame images, includes: Geometric feature analysis is performed on the multi-frame images based on the personal geometric feature threshold to obtain the geometric feature analysis results of each frame in the multi-frame images.

10. A driving state recognition device, characterized in that, The device includes: The acquisition module is used to acquire an image frame sequence, wherein the image frame sequence contains multiple frames of images including the face of the target person; The geometric analysis module is used to perform geometric feature analysis on the multi-frame images, obtain the geometric feature analysis results of each frame in the multi-frame images, and generate a first state probability distribution of the image based on the geometric feature analysis results. The first state probability distribution is used to predict the probability of the target person in the image under various preset driving states. The facial expression recognition module is used to perform facial expression recognition processing on each frame of the multi-frame images, and generate the second state probability distribution of the image and the corresponding confidence level. The fusion module is used to perform weighted fusion of the first state probability distribution and the second state probability distribution according to the confidence level to obtain the state score of each frame in the multi-frame images. The state score includes the prediction score for each preset driving state among the multiple preset driving states. The state recognition module is used to determine the preset driving state with the highest state score in the image as the driving state of the target person in the image.

11. An electronic device comprising a memory, one or more processors, and a computer program stored in the memory and executable on the one or more processors, characterized in that, When the one or more processors execute the computer program, the electronic device performs the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 9.