Vehicle safety warning method, vehicle and computer readable storage medium

Abstract

This application provides a vehicle safety warning method, a vehicle, and a computer-readable storage medium. The method includes: acquiring a driving area image sequence; analyzing and processing the driving area image sequence to obtain multi-dimensional feature parameters and target driving behavior categories; determining the current risk level based on the multi-dimensional feature parameters, target driving behavior categories, and a preset risk matrix lookup table; and determining target warning information based on the current risk level and a preset mapping table. This application solves the technical problems of existing driver monitoring systems in situations where facial occlusion occurs, including missed detection of complex risks, coarse warnings, and poor robustness.

Description

Technical Field

[0001] This application relates to the field of intelligent driving and traffic safety technology, specifically to a vehicle safety early warning method, a vehicle, and a computer-readable storage medium. Background Technology

[0002] With the development of intelligent vehicles, Driver Monitoring Systems (DMS) have become a core component in improving driving safety by using visual sensing to identify fatigue and distracted behavior in real time. However, existing technologies mostly focus on detecting single states, specifically for fatigue driving warnings or distracted driving detection. Their functions are relatively limited, and they lack fine-grained classification of fatigue levels and accurate identification of specific distracted behaviors. Furthermore, the systems are highly sensitive to facial occlusions such as sunglasses and masks; failure of key features leads to detection interruption, resulting in poor robustness to environmental obstructions.

[0003] There is currently no good solution to the above problems. Summary of the Invention

[0004] This application provides a vehicle safety early warning method, a vehicle, and a computer-readable storage medium to at least solve the technical problems of existing driver monitoring systems in situations where facial occlusion occurs, such as missed detection of complex risks, crude early warning, and poor robustness.

[0005] According to one aspect of the embodiments of this application, a vehicle safety warning method is provided, comprising: acquiring a driving area image sequence, wherein the driving area image sequence is obtained by preprocessing a driving area video stream, the driving area video stream being used to represent a real-time image sequence corresponding to the driving area; analyzing and processing the driving area image sequence to obtain multi-dimensional feature parameters and a target driving behavior category, wherein the multi-dimensional feature parameters are used to determine the driver's facial state and head posture, and the target driving behavior category is used to represent the behavior category of the driver's attention deviating from the driving task; determining a current risk level based on the multi-dimensional feature parameters, the target driving behavior category, and a preset risk matrix lookup table, wherein the current risk level is used to represent the degree of risk corresponding to the current driving state; and determining target warning information based on the current risk level and a preset mapping table, wherein the target warning information is used to prompt the driver to perform safe driving operations, and the preset mapping table is used to represent the mapping relationship between the current risk level and the target warning information.

[0006] Furthermore, acquiring the driving area image sequence includes: acquiring a driving area video stream, wherein the driving area video stream is obtained by image acquisition through an on-board image acquisition component located above the front structural pillar of the vehicle; and preprocessing the driving area video stream to obtain the driving area image sequence.

[0007] Further, the driving area video stream is preprocessed to obtain a driving area image sequence, including: grayscale conversion of the driving area video stream to obtain a grayscale image sequence; histogram equalization of the grayscale images to obtain an enhanced image sequence; and size normalization of the enhanced image sequence to obtain a driving area image sequence.

[0008] Furthermore, the driving area image sequence is analyzed and processed to obtain multidimensional feature parameters, including: feature extraction of the driving area image sequence based on the target detection algorithm to obtain feature extraction results, wherein the feature extraction results include the head pitch angle and the coordinate values ​​of key facial feature points; and the feature extraction results are analyzed and processed to obtain multidimensional feature parameters.

[0009] Furthermore, the key facial feature points include: multiple key eye feature points and multiple key mouth feature points. The multiple mouth feature points include multiple key points on the outer edge of the upper lip and multiple key points on the outer edge of the lower lip. Multidimensional feature parameters include: eye closure rate, blink frequency, preset number of movements, and nodding frequency. The feature extraction results are analyzed and processed to obtain multidimensional feature parameters, including: analyzing and processing the coordinate values ​​of multiple key eye feature points to obtain eye opening and closing degree, where the eye opening and closing degree includes the left eye opening and closing degree and the right eye opening and closing degree is the average of the left and right eye opening and closing degrees; and analyzing and processing the coordinate values ​​of multiple key mouth feature points. The following steps are taken: The mouth opening / closing degree is obtained, determined based on the coordinates of multiple key points on the outer edge of the upper lip and multiple key points on the outer edge of the lower lip; the eye closure rate is determined based on the duration and a preset time when the eye opening / closing degree is less than a first preset threshold within a preset time; the blinking frequency is determined based on the number of times the eye opening / closing degree is greater than the first preset threshold and less than or equal to a second preset threshold within a preset time; the preset number of actions is determined based on the number of times the mouth opening / closing degree is greater than or equal to a third preset threshold and the duration is greater than or equal to a preset time threshold within a preset time; and the head pitch angle is analyzed to obtain the nodding frequency.

[0010] Furthermore, the driving area image sequence is analyzed and processed to obtain the target driving behavior category, which includes: the driving area image sequence is inferred based on the target inference model to obtain the target driving behavior category. The target inference model is obtained by training an initial inference model based on sample data and the labels corresponding to the sample data. The sample data includes historical driving area image sequences.

[0011] Furthermore, based on multidimensional feature parameters, target driving behavior categories, and a preset risk matrix lookup table, determining the current risk level includes: analyzing and processing the multidimensional feature parameters and preset thresholds to obtain analysis results, wherein the analysis results are used to determine the fatigue level corresponding to the multidimensional feature parameters; and determining the current risk level based on the fatigue level corresponding to the multidimensional feature parameters, the target driving behavior category, and the preset risk matrix lookup table.

[0012] Furthermore, the multidimensional feature parameters include: eye closure rate, blink frequency, preset number of actions, and nodding frequency. The multidimensional feature parameters and preset thresholds are analyzed to obtain the following results: In response to an eye closure rate greater than or equal to a first preset threshold and less than a second preset threshold, and / or a blink frequency less than or equal to a third preset threshold, and / or a preset number of actions equal to a first preset value, the fatigue level is determined to be mild fatigue; In response to an eye closure rate greater than or equal to a second preset threshold and less than a fourth preset threshold, and / or a blink frequency less than or equal to a fifth preset threshold, and / or a preset number of actions equal to a second preset value, and / or a nodding frequency greater than or equal to a sixth preset threshold, the fatigue level is determined to be moderate fatigue, wherein the fifth preset threshold is less than the third preset threshold; In response to an eye closure rate greater than or equal to a seventh preset threshold, and / or a preset number of actions greater than the second preset value, the fatigue level is determined to be severe fatigue, wherein the seventh preset threshold is greater than the fifth preset threshold.

[0013] According to another aspect of the embodiments of this application, a vehicle safety warning device is also provided, comprising: an acquisition module, configured to acquire a driving area image sequence, wherein the driving area image sequence is obtained by preprocessing a driving area video stream, the driving area video stream being used to represent a real-time image sequence corresponding to the driving area; an analysis module, configured to analyze and process the driving area image sequence to obtain multi-dimensional feature parameters and a target driving behavior category, wherein the multi-dimensional feature parameters are used to determine the driver's facial state and head posture, and the target driving behavior category is used to represent the behavior category of the driver's attention deviating from the driving task; a determination module, configured to determine the current risk level based on the multi-dimensional feature parameters, the target driving behavior category, and a preset risk matrix lookup table, wherein the current risk level is used to represent the degree of risk corresponding to the current driving state; and a warning module, configured to determine target warning information based on the current risk level and a preset mapping table, wherein the target warning information is used to prompt the driver to perform safe driving operations, and the preset mapping table is used to represent the mapping relationship between the current risk level and the target warning information.

[0014] Furthermore, the acquisition module is also used to: acquire a driving area video stream, wherein the driving area video stream is obtained by image acquisition through an on-board image acquisition component, which is located above the front structural pillar of the vehicle; and preprocess the driving area video stream to obtain a driving area image sequence.

[0015] Furthermore, the acquisition module is also used to: perform grayscale processing on the video stream of the driving area to obtain a grayscale image sequence; perform histogram equalization processing on the grayscale images to obtain an enhanced image sequence; and perform size normalization processing on the enhanced image sequence to obtain a driving area image sequence.

[0016] Furthermore, the analysis module is also used to: extract features from the image sequence of the driving area based on the target detection algorithm to obtain feature extraction results, including the head pitch angle and the coordinate values ​​of key facial feature points; and analyze and process the feature extraction results to obtain multidimensional feature parameters.

[0017] Furthermore, the key facial feature points include: multiple key eye feature points and multiple key mouth feature points. The multiple mouth feature points include multiple key points on the outer edge of the upper lip and multiple key points on the outer edge of the lower lip. Multidimensional feature parameters include: eye closure rate, blink frequency, preset number of actions, and nodding frequency. The analysis module is also used to: analyze and process the coordinate values ​​of multiple key eye feature points to obtain the eye opening degree, where the eye opening degree includes the left eye opening degree and the right eye opening degree, and the eye opening degree is the average of the left and right eye opening degrees; and analyze and process the coordinate values ​​of multiple key mouth feature points to obtain the mouth opening degree. The determination of mouth opening and closing is based on the coordinate values ​​of multiple key points on the outer edge of the upper lip and multiple key points on the outer edge of the lower lip. The determination of eye closure rate is based on the duration and time of eye opening and closing being less than a first preset threshold within a preset time. The determination of blinking frequency is based on the number of times eye opening and closing is greater than the first preset threshold and less than or equal to the second preset threshold within a preset time. The determination of preset action number is based on the number of times mouth opening and closing is greater than or equal to a third preset threshold and the duration is greater than or equal to a preset time threshold within a preset time. The determination of head nodding frequency is also based on the analysis and processing of head pitch angle.

[0018] Furthermore, the analysis module is also used to: perform reasoning processing on the driving area image sequence based on the target reasoning model to obtain the target driving behavior category, wherein the target reasoning model is obtained by training the initial reasoning model based on the sample data and the labels corresponding to the sample data, and the sample data includes historical driving area image sequences.

[0019] Furthermore, the determination module is also used to: analyze and process multidimensional feature parameters and preset thresholds to obtain analysis results, wherein the analysis results are used to determine the fatigue level corresponding to the multidimensional feature parameters; and determine the current risk level based on the fatigue level corresponding to the multidimensional feature parameters, the target driving behavior category, and the preset risk matrix lookup table.

[0020] Furthermore, the multidimensional feature parameters include: eye closure rate, blink frequency, preset number of actions, and nodding frequency. The determining module is also used to: determine the fatigue level as mild fatigue in response to the eye closure rate being greater than or equal to a first preset threshold and less than a second preset threshold, and / or, the blink frequency being less than or equal to a third preset threshold, and / or, the preset number of actions being equal to a preset first value; determine the fatigue level as moderate fatigue in response to the eye closure rate being greater than or equal to a second preset threshold and less than a fourth preset threshold, and / or, the blink frequency being less than or equal to a fifth preset threshold, and / or, the preset number of actions being equal to a preset second value, and / or, the nodding frequency being greater than or equal to a sixth preset threshold, wherein the fifth preset threshold is less than the third preset threshold; and determine the fatigue level as severe fatigue in response to the eye closure rate being greater than or equal to a seventh preset threshold, and / or, the preset number of actions being greater than the preset second value, wherein the seventh preset threshold is greater than the fifth preset threshold.

[0021] According to another aspect of the embodiments of this application, a vehicle is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0022] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0023] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0024] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0025] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0026] In this embodiment, a driving area image sequence is acquired, which is obtained by preprocessing a driving area video stream, representing the real-time image sequence corresponding to the driving area. The driving area image sequence is then analyzed to obtain multidimensional feature parameters and a target driving behavior category. The multidimensional feature parameters determine the driver's facial state and head posture, and the target driving behavior category represents the type of behavior that deviates from the driver's attention to the driving task. Furthermore, based on the multidimensional feature parameters, the target driving behavior category, and a preset risk matrix lookup table, the current risk level is determined, where the current risk level represents... The system assesses the risk level corresponding to the current driving state. Finally, based on the current risk level and a preset mapping table, it determines the target warning information. The target warning information is used to prompt the driver to perform safe driving operations, and the preset mapping table represents the mapping relationship between the current risk level and the target warning information. This achieves the goal of accurately classifying the driver's fatigue level and simultaneously identifying distracted behavior, thereby triggering the target warning information. This improves the system's accuracy in perceiving complex risk situations, the precision of warnings, and the robustness of warnings. It also solves the technical problems of existing driver monitoring systems in situations with facial occlusion, such as missed detection of complex risks, crude warnings, and poor robustness. Attached Figure Description

[0027] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0028] Figure 1 This is a flowchart of an optional vehicle safety warning method according to an embodiment of this application;

[0029] Figure 2 This is a schematic diagram of an optional plurality of key eye feature points according to an embodiment of this application;

[0030] Figure 3 This is a schematic diagram of an optional vehicle safety warning method according to an embodiment of this application;

[0031] Figure 4 This is a schematic diagram of another optional vehicle safety warning method according to an embodiment of this application;

[0032] Figure 5 This is a structural block diagram of an optional vehicle safety warning device according to an embodiment of this application. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] According to an embodiment of this application, a method embodiment for a vehicle safety warning method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0036] This embodiment provides a vehicle safety warning method. Figure 1 This is a flowchart of an optional vehicle safety warning method according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps:

[0037] Step S11: Obtain the driving area image sequence, wherein the driving area image sequence is obtained by preprocessing the driving area video stream, and the driving area video stream is used to represent the real-time image sequence corresponding to the driving area.

[0038] Step S12: Analyze and process the image sequence of the driving area to obtain multidimensional feature parameters and target driving behavior category. The multidimensional feature parameters are used to determine the driver's facial state and head posture, and the target driving behavior category is used to represent the behavior category of the driver's attention deviating from the driving task.

[0039] Step S13: Determine the current risk level based on multidimensional feature parameters, target driving behavior category and preset risk matrix lookup table, wherein the current risk level is used to represent the degree of risk corresponding to the current driving state;

[0040] Step S14: Based on the current risk level and the preset mapping table, determine the target warning information, wherein the target warning information is used to prompt the driver to perform safe driving operations, and the preset mapping table is used to represent the mapping relationship between the current risk level and the target warning information.

[0041] The above driving area image sequence represents a standardized image set consisting of consecutive image frames obtained by preprocessing the driving area video stream. Each image frame corresponds to a point in time during the vehicle's movement and contains visual information about the driver's face and upper body.

[0042] The aforementioned driving area video stream refers to a continuous sequence of frame images, including the driver's face and upper body, continuously captured by a camera installed inside the vehicle's cockpit. This sequence is used to reflect the driver's posture changes, eye movements, mouth opening and closing, head movements, and hand gestures in real time during driving. The frame rate of the image sequence is no less than 30 frames per second, and the resolution is no less than 1080P, to ensure that sufficiently detailed and smooth facial and behavioral features are captured in dynamic driving scenarios.

[0043] The aforementioned multidimensional feature parameters represent multiple independent visual features extracted from the driver's facial region in the driving area image sequence to quantify the driver's physiological state and behavioral tendencies. These features include eye opening and closing, mouth opening and closing, head pitch angle, head yaw angle, blink frequency, percentage of eyelid closure over the pull over time (PERCLOS), and number of yawns. These parameters collectively constitute a multidimensional and complementary representation of the driver's fatigue state, overcoming the risk of failure of single features under occlusion or abnormal movements, and improving the robustness and accuracy of state determination.

[0044] The aforementioned target driving behavior categories represent types of driver distraction behaviors directly related to driving safety, including making phone calls, drinking water, smoking, and talking to passengers. These four categories constitute a complete coverage of high-risk distraction behaviors, providing structured and quantifiable behavioral inputs for the collaborative decision engine.

[0045] The aforementioned preset risk matrix lookup table represents a rule-based decision mapping table used to integrate driver fatigue levels with target driving behavior categories and output a comprehensive risk level. Its structure is a two-dimensional table, with rows representing the grading results of fatigue status, including no fatigue, mild fatigue, moderate fatigue, and severe fatigue, and columns representing target driving behavior categories, including no behavior, drinking water, smoking, making a phone call, and talking to passengers. Each intersecting cell in the table is preset with a comprehensive risk level, including five risk levels: no risk, low risk, medium risk, high risk, and extremely high risk.

[0046] The above method determines the current risk level based on multidimensional feature parameters, target driving behavior categories, and a preset risk matrix lookup table. This means that the assessment is based on multidimensional feature parameters and target driving behavior categories, combined with a preset risk matrix lookup table. The risk matrix lookup table defines five levels—no risk, low risk, medium risk, high risk, and extremely high risk—based on the combination relationship between fatigue level and behavior category, thus achieving a structured judgment of complex risk scenarios.

[0047] The above-mentioned determination of target warning information based on the current risk level and a preset mapping table means that, according to the current risk level, the corresponding target warning information is mapped through the preset mapping table. This mapping table pre-stores text-based voice prompts matching each risk level to ensure that the warning content accurately corresponds to the risk scenario. For example, if the current risk level is "extremely high risk" and the behavior category is "smoking," the system directly retrieves "Warning! The driver is severely fatigued and smoking, which is extremely likely to cause an accident. Please stop and rest immediately in a safe area" as the target warning information; if the current risk level is "medium risk" and the behavior category is "drinking water," the system retrieves "Driver drinking water detected. Please pay attention to road conditions and hold the steering wheel firmly" as the target warning information. Furthermore, upon receiving a target warning message, the system generates the warning message in text format and outputs it to the in-vehicle text-to-speech (TTS) synthesis module via a Universal Asynchronous Receiver / Transmitter (UART) serial communication interface. The in-vehicle TTS speech synthesis module then synthesizes and plays the corresponding voice warning, completing a full detection and warning process. This process transforms abstract risk levels into action-oriented natural language commands, breaking through the traditional DMS system's reliance on a single feedback mode of audible and visual alarms, significantly improving the effectiveness of information delivery and the driver's willingness to respond.

[0048] Based on the above steps S11 to S14, this embodiment of the application acquires a driving area image sequence, wherein the driving area image sequence is obtained by preprocessing a driving area video stream, the driving area video stream being used to represent the real-time image sequence corresponding to the driving area; then, the driving area image sequence is analyzed and processed to obtain multi-dimensional feature parameters and target driving behavior categories, wherein the multi-dimensional feature parameters are used to determine the driver's facial state and head posture, and the target driving behavior category is used to represent the behavior category of the driver's attention deviating from the driving task; furthermore, based on the multi-dimensional feature parameters, the target driving behavior category, and a preset risk matrix lookup table, the current risk level is determined, wherein the current risk... The risk level is used to indicate the degree of risk corresponding to the current driving state. Finally, based on the current risk level and the preset mapping table, the target warning information is determined. The target warning information is used to prompt the driver to perform safe driving operations. The preset mapping table is used to represent the mapping relationship between the current risk level and the target warning information. This achieves the purpose of accurately classifying the driver's fatigue level and simultaneously identifying distracted behavior, and triggering the target warning information. This improves the system's perception accuracy, warning precision, and robustness in complex risk situations, and solves the technical problems of existing driver monitoring systems in the face-obscured environment, such as missed detection of complex risks, crude warnings, and poor robustness.

[0049] Furthermore, acquiring the driving area image sequence includes: acquiring a driving area video stream, wherein the driving area video stream is obtained by image acquisition through an on-board image acquisition component located above the front structural pillar of the vehicle; and preprocessing the driving area video stream to obtain the driving area image sequence.

[0050] The area above the front structural pillar of the vehicle can be the A-pillar. The in-vehicle image acquisition component can be a monocular RGB camera module. That is, the video stream of the driving area is a color video sequence of the driver's face and upper body area acquired by a monocular RGB camera module installed inside the car's cockpit (above the A-pillar). It is important to note that the recommended resolution for this camera module is 1080P or higher, with a frame rate of at least 30fps to ensure a sufficiently clear and smooth image sequence. Furthermore, the camera's installation position must ensure coverage of the driver's eyes, bridge of the nose, mouth, and head movement range, avoiding obstruction by the steering wheel, dashboard, or roof trim. Simultaneously, the shooting angle should be consistent with the driver's normal gaze direction, thereby acquiring stable, high-resolution image data that conforms to the laws of human vision without interfering with driving operations. This process achieves non-invasive, continuous, and stable visual perception of the driver's facial area, solving problems such as viewing angle shift, frequent obstruction, and uneven lighting caused by traditional systems where the camera is installed on the dashboard or rearview mirror, significantly improving the robustness and consistency of image acquisition.

[0051] The above preprocessing means that for each frame of the original driving area video stream output by the vehicle image acquisition component, grayscale conversion, histogram equalization and size normalization are performed sequentially to provide standardized input data for the subsequent feature extraction module.

[0052] Based on the above optional embodiments, this application embodiment constructs a front-end processing link from physical space image acquisition to digital image sequence generation, realizing the synergistic optimization of optimal acquisition layout above the front structural pillar of the vehicle and lightweight image preprocessing, significantly improving the spatiotemporal consistency, illumination robustness and feature extractability of the driving area image sequence, and providing reliable data support for subsequent multimodal perception and collaborative decision-making without relying on high-cost sensors.

[0053] Further, the driving area video stream is preprocessed to obtain a driving area image sequence, including: grayscale conversion of the driving area video stream to obtain a grayscale image sequence; histogram equalization of the grayscale images to obtain an enhanced image sequence; and size normalization of the enhanced image sequence to obtain a driving area image sequence.

[0054] The grayscale processing described above means performing grayscale processing on each frame of the original driving area video stream output by the vehicle-mounted image acquisition component, converting the red, green, and blue three-channel pixel values ​​into single-channel grayscale values ​​according to a weighted formula, in order to eliminate color information redundancy and reduce the computational complexity of subsequent feature extraction.

[0055] The above histogram equalization process means performing histogram equalization on each frame of the grayscale image sequence. By redistributing the pixel grayscale distribution, it enhances the local contrast of the image, especially under complex lighting conditions such as strong backlight, tunnel entrances and exits, low light at night, or reflections from sunglasses, thereby improving the clarity and distinguishability of key physiological features such as eyelid edges, corners of the mouth, and bridge of the nose.

[0056] The above size normalization process means performing size normalization on the image sequence after histogram equalization, uniformly scaling all images to a fixed resolution of 640×480 pixels, ensuring that the image size input to the subsequent face detection and key point localization models is consistent, and avoiding feature extraction drift and model inference errors caused by differences in image scale.

[0057] Based on the above optional embodiments, this application embodiment constructs a standardized front-end image preprocessing link by sequentially performing grayscale processing, histogram equalization processing, and size normalization processing on the video stream of the driving area. While ensuring computational efficiency, it effectively enhances the feature extractability of the image under complex lighting and acquisition conditions, significantly improves the quality stability and algorithm robustness of the driving area image sequence, and provides reliable data support for subsequent accurate driver status analysis.

[0058] Furthermore, the driving area image sequence is analyzed and processed to obtain multidimensional feature parameters, including: feature extraction of the driving area image sequence based on the target detection algorithm to obtain feature extraction results, wherein the feature extraction results include the head pitch angle and the coordinate values ​​of key facial feature points; and the feature extraction results are analyzed and processed to obtain multidimensional feature parameters.

[0059] The aforementioned feature extraction utilizes a facial landmark detection algorithm to accurately locate key feature points of the face, such as the left and right eyes, mouth, and nose tip. Specifically, in this embodiment, a pre-trained deep learning-based object detection algorithm is used to process each frame of the driving area image sequence. This algorithm extracts semantic features from the image through a convolutional neural network, locates and outputs the coordinates of multiple preset key feature points in the driver's facial region, including the outer corner of the left eye, the inner corner of the left eye, the outer corner of the right eye, the inner corner of the right eye, the midpoint of the upper lip, the midpoint of the lower lip, the nose tip, and the center of the chin, totaling 68 or more two-dimensional coordinate points. Simultaneously, based on the spatial distribution of these key points and combined with a 3D face standard model, the Perspective-n-Point (PnP) algorithm, which combines perspective projection and point-to-point matching, is used to solve for the driver's head posture parameters in three-dimensional space, including head pitch angle, yaw angle, and roll angle.

[0060] The above analysis and processing of the feature extraction results means that mathematical operations are performed on the key point coordinates according to the preset algorithm to extract multi-dimensional feature parameters for state judgment, including eye opening and closing, mouth opening and closing, head pitch angle, blinking frequency, PERCLOS, nodding frequency, etc.

[0061] Based on optional embodiments, this application embodiment achieves high-precision synchronous output of head pitch angle and facial key feature point coordinates by performing feature extraction and subsequent analysis processing based on target detection algorithm on the driving area image sequence, thereby stably generating multi-dimensional feature parameters covering eyes, mouth, and head posture, providing structured and reusable input data for the quantitative analysis of driver state, and significantly improving the comprehensiveness and environmental adaptability of feature expression.

[0062] Furthermore, the key facial feature points include: multiple key eye feature points and multiple key mouth feature points. The multiple mouth feature points include multiple key points on the outer edge of the upper lip and multiple key points on the outer edge of the lower lip. Multidimensional feature parameters include: eye closure rate, blink frequency, preset number of movements, and nodding frequency. The feature extraction results are analyzed and processed to obtain multidimensional feature parameters, including: analyzing and processing the coordinate values ​​of multiple key eye feature points to obtain eye opening and closing degree, where the eye opening and closing degree includes the left eye opening and closing degree and the right eye opening and closing degree is the average of the left and right eye opening and closing degrees; and analyzing and processing the coordinate values ​​of multiple key mouth feature points. The following steps are taken: The mouth opening / closing degree is obtained, determined based on the coordinates of multiple key points on the outer edge of the upper lip and multiple key points on the outer edge of the lower lip; the eye closure rate is determined based on the duration and a preset time when the eye opening / closing degree is less than a first preset threshold within a preset time; the blinking frequency is determined based on the number of times the eye opening / closing degree is greater than the first preset threshold and less than or equal to a second preset threshold within a preset time; the preset number of actions is determined based on the number of times the mouth opening / closing degree is greater than or equal to a third preset threshold and the duration is greater than or equal to a preset time threshold within a preset time; and the head pitch angle is analyzed to obtain the nodding frequency.

[0063] The aforementioned key facial feature points represent a set of structured key coordinate points of the face precisely located from the driving area image sequence using an object detection algorithm. The aforementioned key eye feature points include 12 points at the inner and outer corners of the upper and lower eyelids of the left and right eyes, used to construct the orbital geometry model. The aforementioned key mouth feature points include 16 points at the outer edges of the upper and lower lip, used to accurately depict the mouth contour; their distribution density ensures sub-pixel accuracy in calculating mouth opening.

[0064] The aforementioned eye opening and closing angle represents the arithmetic mean of the left and right eye opening and closing angles. These angles are calculated based on the ratio of the vertical distance between the key points of the upper and lower eyelids of the left and right eyes to the horizontal eye width. The value of this eye opening and closing angle ranges from 0 to 1, used to quantify the degree of eyelid closure and reflect the dynamic changes in the driver's eye fatigue state. 0 represents complete closure, and 1 represents complete opening.

[0065] Specifically, six standardized key ocular feature points were set for each of the left and right eyes, labeled as p1 to p6 respectively, as follows: Figure 2 As shown, p1 and p4 are the outer and inner corners of the eye in the horizontal direction of the eyelids; p2 and p6 are the vertically symmetrical points at the midpoints of the upper and lower eyelids; and p3 and p5 are the vertically symmetrical points at the inner sides of the upper and lower eyelids. The formula for calculating the left / right eye opening ratio (EAR) is:

[0066]

[0067] in, This represents the Euclidean distance between two points. The aforementioned eye opening and closing degree achieves symmetrical and stable measurement of the closed state of both eyes, effectively compensating for detection biases caused by unilateral occlusion, pupil deviation, or the face not being directly facing the camera, and improving the reliability of features under common driving interference conditions such as strong light, wearing sunglasses, and slight head tilt; furthermore, this parameter, as a continuous time series input, provides basic data support for subsequent statistical analysis of eye closure rate and blink frequency, ensuring that fatigue state assessment has high temporal resolution and spatial consistency.

[0068] The mouth opening degree mentioned above represents the ratio of the maximum vertical distance between multiple key points on the outer edges of the upper lip and multiple key points on the outer edges of the lower lip to the horizontal width of the mouth, reflecting the degree of mouth opening and used to identify yawning behavior. The specific calculation process of the mouth opening degree is as follows: First, extract the maximum value of the vertical coordinate of all key points on the outer edges of the upper lip and the minimum value of the vertical coordinate of all key points on the outer edges of the lower lip. The difference between the two is the maximum vertical opening distance of the mouth. Second, calculate the horizontal Euclidean distance between the left and right endpoints of the key points on the outer edges of the upper lip, as the reference for the horizontal width of the mouth. Finally, divide the vertical opening distance of the mouth by the horizontal width of the mouth to obtain the mouth opening degree, which ranges from 0 to 1, where 0 indicates that the mouth is completely closed and 1 indicates that the mouth is completely open. The aforementioned mouth opening and closing degree achieves continuous and stable quantification of dynamic mouth behavior, avoiding local errors that rely solely on single-point distances (such as from the corner of the mouth to the center of the lips). This ensures that even in complex scenarios such as drivers wearing masks, head turning, or changes in lighting, the mouth opening action can still be accurately captured, providing high-precision input for determining the number of preset actions, thereby improving the accuracy and robustness of yawning behavior recognition.

[0069] The aforementioned preset action count represents the number of yawns. The aforementioned third preset threshold represents a pre-set critical value for mouth opening and closing, with a value of 0.5, used to distinguish between genuine yawning actions and ordinary mouth opening behavior. The aforementioned preset time threshold represents a pre-set lower limit for duration, with a value of 1.5 seconds, used to exclude momentary mouth opening interference. In this embodiment, when the mouth opening and closing exceeds the threshold TH_mouth (e.g., 0.5) and lasts for a certain number of frames, it is determined as a yawning action, i.e., one preset action is executed. This embodiment, by introducing a combination of dual thresholds (the third preset threshold and the preset time threshold), achieves physiological rationality screening of yawning behavior, significantly improving the accuracy and anti-interference ability of yawn recognition.

[0070] The aforementioned eye closure rate is a percentage parameter calculated based on the ratio of the duration of eye opening / closing less than a first preset threshold within a preset time period to the preset time itself. It quantifies the proportion of time the eyes are in a closed state, reflecting the persistent eyelid closure trend caused by driver fatigue. The preset time is typically 60 seconds, and the first preset threshold is 0.2. For example, the eye closure rate can be expressed as the percentage of eye closure time per unit time (PERCLOS). The specific calculation process for the eye closure rate is as follows: when the eye opening / closing degree is below 0.2, it is determined to be an eye closure state, and the number of frames lasting in this state is accumulated; when the eye opening / closing degree rises back to above or equal to 0.2, the timing of this closure event is terminated. Finally, the cumulative duration of all closure events is divided by the preset time (60 seconds) and then multiplied by 100% to obtain the eye closure rate, expressed as a percentage (%), ranging from 0% to 100%. The aforementioned eye closure rate enables a time-domain cumulative measurement of eye closure behavior, transforming instantaneous eye state into a physiologically meaningful cumulative fatigue indicator. This effectively distinguishes between brief blinks and continuous eye closure, avoiding misjudgment of fatigue based on a single blink and improving the reliability of fatigue assessment. Furthermore, this parameter serves as a core input for fatigue grading, directly correlated with the eye opening and closing standards in the mild, moderate, and severe fatigue assessments, providing crucial quantitative evidence for achieving accurate grading based on multi-feature fusion.

[0071] The blink frequency mentioned above represents the number of complete cyclical movements of the driver's eyes, from fully open to fully closed and back to open, occurring within a unit of time (e.g., 60 seconds). This is used to quantify physiological changes in blinking patterns caused by fatigue, such as an abnormal increase or decrease in blinking behavior. Specifically, based on continuously acquired eye opening and closing time series, the system identifies single blink events according to the following rules: when the eye opening and closing degree drops from above a second preset threshold (e.g., 0.3) to below a first preset threshold (e.g., 0.2), it is determined as the start of a blink; subsequently, the eye opening and closing degree remains below 0.2 for at least 3 consecutive frames (approximately 0.1 seconds) to eliminate momentary occlusion or noise interference; when the eye opening and closing degree rises back above 0.3 and remains stable, it is determined as the end of a blink, completing one complete blink cycle. Each time a complete blink event is identified, the counter increments by one. Within a continuous 60-second monitoring window, the total number of blinks is counted, which is the blink frequency for that period, expressed in "blinks / minute". The aforementioned blink frequency enables dynamic, adaptive, and false-trigger-resistant detection of blinking behavior. Its design effectively avoids false counts caused by light flicker, blink afterimages, or camera shake through a "hysteresis comparison" mechanism composed of dual thresholds (0.3 and 0.2). At the same time, by setting a minimum closing duration (3 frames), non-physiological instantaneous closing, such as eyelash touch or lens reflection, is filtered out to ensure that only physiologically significant fatigue-related blinks are recorded.

[0072] The aforementioned head-nodding frequency represents the number of times the driver's head undergoes a significant periodic downward and then upward movement along the pitch axis (vertical direction) within a unit of time (e.g., 60 seconds). This is used to quantify the physiological compensatory behavior of maintaining a positive posture due to decreased head control caused by fatigue. Specifically, the system obtains the three-dimensional spatial coordinates of key points such as the tip of the nose, the center of the eyebrows, and the chin through a facial key point detection algorithm, and calculates the pitch angle, yaw angle, and roll angle of the head in three-dimensional space using the PnP algorithm. The pitch angle represents the angle of head rotation around the X-axis; a positive pitch angle indicates the head is raised, and a negative pitch angle indicates the head is lowered. The yaw angle represents the angle of head rotation around the Y-axis; a positive yaw angle indicates the head turns to the right, and a negative yaw angle indicates the head turns to the left. The roll angle represents the angle of head rotation around the Z-axis; a positive roll angle indicates the head tilts to the right (the right ear moves closer to the right shoulder), and a negative roll angle indicates the head tilts to the left (the left ear moves closer to the left shoulder). The rules for identifying head-nodding events are as follows: When the pitch angle is continuously below the third preset threshold (e.g., -15°) for at least 5 frames (approximately 0.17 seconds), it is determined as a "head-down" state; subsequently, when the pitch angle rises above the fourth preset threshold (e.g., -5°) and remains stable, it is determined as a "head-back" completion; a complete "head-down-head-back" cycle is counted as one head-nodding event. Within a 60-second monitoring window, the total number of complete head-nodding cycles that meet the above conditions is counted, which is the head-nodding frequency, measured in "times / minute". The above head-nodding frequency achieves objective quantification of non-eye fatigue signals. By setting a "movement integrity criterion" consisting of dual thresholds and a minimum duration, it effectively distinguishes between genuine fatigue-induced head-nodding and accidental head-down, such as checking the central control screen or adjusting the seat, thus avoiding false triggers. At the same time, this parameter is independent of eye characteristics and can serve as a core alternative indicator when the driver is wearing sunglasses, a mask, or has their face obscured by strong light, ensuring that the fatigue detection system does not interrupt operation due to visual feature failure.

[0073] Based on the above optional embodiments, this application embodiment performs structured analysis on the coordinate values ​​of multiple key eye feature points and multiple key mouth feature points, and combines the calculation rules of preset thresholds and time windows to stably output four types of multi-dimensional feature parameters: eye closure rate, blink frequency, preset number of actions, and nodding frequency. This achieves a quantitative description of the driver's physiological state, significantly improves the accuracy, consistency, and anti-interference ability of feature extraction, and provides a reliable data foundation for subsequent fatigue grading and comprehensive risk assessment.

[0074] Furthermore, the driving area image sequence is analyzed and processed to obtain the target driving behavior category, which includes: the driving area image sequence is inferred based on the target inference model to obtain the target driving behavior category. The target inference model is obtained by training an initial inference model based on sample data and the labels corresponding to the sample data. The sample data includes historical driving area image sequences.

[0075] The aforementioned target inference model can be a lightweight target detection neural network. The inference processing of driving area image sequences based on this model involves inputting the preprocessed image sequence into a structured, trained deep neural network model. This model uses a convolutional neural network as its backbone, combined with a temporal feature extraction module, to extract spatial features frame by frame and model inter-frame dynamic changes. Finally, it outputs a confidence vector for each behavior category, namely the classification confidence for "making a phone call," "drinking water," and "smoking." If the confidence exceeds a threshold (e.g., 0.7), the behavior is determined to have occurred. "Talking to passengers" is determined based on a combination of head yaw angle and duration. Specifically, the system continuously calculates the head yaw angle in each frame. When the absolute value of this angle is greater than or equal to a preset yaw angle threshold (e.g., 30°) and remains at least equal to a preset time threshold (e.g., 1.5 seconds), the system determines that the current behavior pattern matches the typical posture characteristics of "talking to passengers." Specifically, the system first obtains head posture parameters through facial key point detection, then performs sliding window smoothing on the yaw angle sequence to identify the intervals that are continuously and stably deviated from the center; then it calculates the mean square error of the angle values ​​within the interval to ensure that the posture is stable rather than jittery; finally, it counts the number of events that satisfy both angle and time constraints. If a complete event occurs once within a continuous monitoring period, it outputs "talking to passengers" as the target driving behavior category.

[0076] The aforementioned target inference model is obtained by training an initial inference model based on sample data and its corresponding labels. This means that the parameters of the target inference model are optimized from the labeled dataset through supervised learning. The sample data consists of historical driving area image sequences from real road tests and simulated environments, covering driver behavior samples under different lighting conditions, occlusions, postures, and vehicle models. The labels corresponding to the sample data are manually annotated behavior category labels precisely aligned with each image sequence, ensuring that the model learns behavioral semantics rather than irrelevant background features. The sample data includes historical driving area image sequences, indicating that the training data consists of multiple consecutive frame image segments, rather than single static images, with a time span typically between 1 and 3 seconds. This is used to capture the complete process of behavior, such as picking up a phone, holding it to the ear, maintaining the call, and putting it back in its original position—a complete action chain—thereby improving the model's ability to identify the start and end points of behavior and its resistance to interference.

[0077] Based on the above optional embodiments, this application embodiment introduces a supervised training mechanism based on historical driving area image sequences to construct a target reasoning model with temporal perception capabilities. This enables the system to identify complex distracted behaviors with clear start and end boundaries and action logic from a continuous visual stream, rather than relying on instantaneous judgments from static object detection. At the same time, the model is trained using real historical data to ensure that the target reasoning model can maintain stable generalization performance in out-of-distribution scenarios, thereby achieving high-precision, low-false-prone, and highly adaptive automatic identification of dangerous driver behaviors.

[0078] Furthermore, based on multidimensional feature parameters, target driving behavior categories, and a preset risk matrix lookup table, determining the current risk level includes: analyzing and processing the multidimensional feature parameters and preset thresholds to obtain analysis results, wherein the analysis results are used to determine the fatigue level corresponding to the multidimensional feature parameters; and determining the current risk level based on the fatigue level corresponding to the multidimensional feature parameters, the target driving behavior category, and the preset risk matrix lookup table.

[0079] The above analysis and processing of multidimensional feature parameters and preset thresholds yields analysis results. This means that the multidimensional feature parameters collected at each moment are compared with the corresponding preset thresholds item by item. The current state is determined to belong to a certain level of mild fatigue, moderate fatigue, or severe fatigue by rule logic or a lightweight classifier. This process does not rely on a single parameter, but makes decisions based on the combined state of multiple parameters to ensure that the analysis results have stability against single-point failure.

[0080] The above method determines the current risk level based on the fatigue level, target driving behavior category, and preset risk matrix lookup table corresponding to multidimensional feature parameters. This means that the fatigue level and target driving behavior category in the above analysis results are used as indexes and simultaneously input into the preset risk matrix lookup table. The unique corresponding current risk level is obtained directly through the table lookup operation. This process is a deterministic mapping without additional calculations, which ensures the real-time performance and consistency of the decision response.

[0081] The risk matrix lookup table related to driver fatigue and behavior in this embodiment is shown below, which displays the risk levels corresponding to different fatigue levels and target driving behavior categories.

[0082]

[0083] Based on the mapping relationship in the table, a final comprehensive risk level is output.

[0084] The core logic of risk classification in this application is that driver fatigue level is the primary factor, while driver behavior is an additional factor. As the fatigue level increases, the driver's reaction speed, judgment, and attention systematically decline, and the basic probability of an accident increases sharply. Different driving behaviors occupy different degrees of the driver's cognition, vision, and operation. Moreover, at the same fatigue level, the risks of different behaviors are different. Making a phone call is the highest-risk behavior because it simultaneously occupies hearing, cognition, and vision, making it a "comprehensive" distraction. Smoking is riskier than drinking water because it involves more complex operations (lighting, holding the cigarette, flicking ash) and lasts longer. The risk of conversation depends on the content, but it usually requires sustained cognitive input.

[0085] In this embodiment, the assessment of behavioral risk is not isolated to a single action, but rather dynamically determined based on the synergistic effect of fatigue level and distraction behavior. Taking "drinking water," a common distraction behavior, as an example, its risk level varies significantly under different fatigue states, reflecting the systematic consumption of cognitive resources by fatigue and the nonlinear amplification mechanism of the consequences of distraction behavior. When the driver is not fatigued, drinking water only requires a brief single-handed operation and limited attention allocation, which is a low-cognitive-load action. The driver still retains sufficient visual monitoring ability and reaction reserve, and can quickly restore full attention to the road environment. Therefore, this combination is judged as low-risk.

[0086] When a driver enters a state of mild fatigue, their physiological alertness begins to decline, reaction time lengthens, and information processing efficiency decreases. Even a simple action like drinking water at this point consumes already limited cognitive and operational resources, leading to a longer recovery time from a distracted state to effective driving, and a significantly increased probability of delayed responses to emergencies. At this stage, fatigue, as a fundamental risk factor, amplifies the consequences of drinking water, escalating what was initially a low-risk distraction into a medium-risk event.

[0087] When fatigue intensifies to a moderate level, the driver's attention span narrows significantly, visual search ability declines, and hand-eye coordination and stability weaken, potentially even leading to brief periods of confusion. At this point, even a small action like looking down to grab a water bottle can cause the driver's gaze to drift away from the road beyond a safe threshold. Simultaneously, reduced control over the steering wheel makes it highly susceptible to accidents caused by the driver failing to notice the vehicle ahead slowing down or veering out of its lane. At this stage, drinking water is no longer a simple operational distraction but a critical trigger that overwhelms the driver's limited cognitive capacity, thus elevating its risk level to high risk.

[0088] When a driver is severely fatigued, their nervous system is nearing functional impairment, and they may enter a state of microsleep at any time. Their level of consciousness remains consistently low, and their ability to perceive and respond to external stimuli is almost completely lost. At this point, the direct impact of drinking water on driving safety is negligible, because even without any distracting actions, the probability of loss of control caused solely by fatigue is already close to its limit. The system determines that in this state, regardless of whether drinking water, smoking, or talking occurs, the overall risk reaches the highest level—extremely high risk. Essentially, fatigue has become the dominant and irreversible prerequisite for an accident, and any additional actions are superimposed on this extreme risk baseline, no longer having independent risk increment significance.

[0089] Based on the above optional embodiments, this application embodiment uses the analysis and processing results of multi-dimensional feature parameters and preset thresholds as input for fatigue level, combines the identification output of target driving behavior category, and performs unified mapping according to preset risk matrix lookup table. This achieves structured, interpretable, and low-latency assessment of the driver's composite risk state, avoids missed detections or misjudgments caused by independent judgment of fatigue and distraction in traditional systems, and improves the accuracy, consistency, and traceability of risk level determination in real complex driving scenarios.

[0090] Furthermore, the multidimensional feature parameters include: eye closure rate, blink frequency, preset number of actions, and nodding frequency. The multidimensional feature parameters and preset thresholds are analyzed to obtain the following results: In response to an eye closure rate greater than or equal to a first preset threshold and less than a second preset threshold, and / or a blink frequency less than or equal to a third preset threshold, and / or a preset number of actions equal to a first preset value, the fatigue level is determined to be mild fatigue; In response to an eye closure rate greater than or equal to a second preset threshold and less than a fourth preset threshold, and / or a blink frequency less than or equal to a fifth preset threshold, and / or a preset number of actions equal to a second preset value, and / or a nodding frequency greater than or equal to a sixth preset threshold, the fatigue level is determined to be moderate fatigue, wherein the fifth preset threshold is less than the third preset threshold; In response to an eye closure rate greater than or equal to a seventh preset threshold, and / or a preset number of actions greater than the second preset value, the fatigue level is determined to be severe fatigue, wherein the seventh preset threshold is greater than the fifth preset threshold.

[0091] The first preset threshold can be 0.2, used to identify mild eye-closing accumulation in the early stages of fatigue. When the eye closure rate first reaches this value, it indicates that the driver has entered the fatigue accumulation stage. The second preset threshold can be 0.4, serving as the boundary between mild and moderate fatigue. Its value is higher than the first preset threshold, ensuring that the determination of mild fatigue has a clear physiological lower limit. The third preset threshold can be 8 times / minute, used to determine whether the blinking frequency has significantly decreased. When the blinking frequency is less than or equal to this value, it indicates that the driver's neural response is slow, which is a typical behavioral manifestation of fatigue. The first preset value can be 1, representing the minimum number of yawn triggers in the mild fatigue stage. The second preset value can be 2, representing the yawn trigger threshold in the moderate fatigue stage. A value greater than 1 reflects the progression of fatigue level. The fourth preset threshold can be 0.6, serving as the boundary between moderate and severe fatigue. Its value is higher than the second preset threshold, ensuring that the moderate fatigue determination includes more significant eye closure accumulation. The fifth preset threshold can be 5 times / minute, used to determine whether the driver is in a moderate fatigue state, and it is lower than the third preset threshold. The sixth preset threshold can be 7 times / minute, used to capture periodic nodding behavior caused by the decline in head control ability due to fatigue. The seventh preset threshold can be 0.7, serving as the minimum eye closure rate criterion for severe fatigue. Its value is higher than the fifth preset threshold, ensuring that the severe fatigue determination is triggered only in a state of extremely long eye closure or complete eye closure.

[0092] The above response is that the eye closure rate is greater than or equal to a first preset threshold and less than a second preset threshold, and / or the blink frequency is less than or equal to a third preset threshold, and / or the preset number of actions is equal to a preset first value. The fatigue level is determined to be mild fatigue. This means that when any one or more target parameters meet the above combination of conditions, that is, when the PERCLOS (the proportion of frames with more than 50% eye closure per unit time) value is between 0.2 and 0.4, or the blink frequency is less than 8 times / minute, or 1 yawn is detected, the system determines that it is mild fatigue. This logic design allows single-parameter trigger judgment, improves detection sensitivity, and avoids missed judgment due to abnormal local features.

[0093] The above response is that the eye closure rate is greater than or equal to a second preset threshold and the eye closure rate is less than a fourth preset threshold, and / or the blinking frequency is less than or equal to a fifth preset threshold, and / or the preset number of actions is equal to a preset second value, and / or the nodding frequency is greater than or equal to a sixth preset threshold, and the fatigue level is determined to be moderate fatigue. This means that when any one or more target parameters meet a more stringent combination of conditions, i.e., the PERCLOS value is between 0.4 and 0.6, or the blinking frequency is less than 5 times / minute, or 2 yawns are detected, or the nodding frequency is significantly increased, the system determines that the fatigue level is moderate fatigue. This logic achieves accurate capture of the fatigue aggravation state by introducing a lower blinking frequency threshold, a higher nodding frequency threshold, and a higher yawning number threshold.

[0094] The above response is that the eye closure rate is greater than or equal to the seventh preset threshold, and / or the preset number of actions is greater than the preset second value, and the fatigue level is determined to be severe fatigue. This means that when the PERCLOS value is greater than 0.6, or three or more consecutive yawns are detected, or the eye closure lasts for more than 2 seconds, the system immediately determines that it is severe fatigue. This logic adopts the highest threshold and extreme behavior triggering mechanism to ensure timely identification of potential microsleep risks.

[0095] Based on the above optional embodiments, this application embodiment constructs a fatigue level determination rule that includes multi-level thresholds and multi-parameter combination logic, realizing a step-by-step, quantifiable, and interference-resistant identification of mild, moderate, and severe fatigue states. This method breaks through the traditional binary judgment mode, enhances the detection robustness through the "OR" relationship between parameters, and reflects the gradual fatigue process through the progressive threshold levels. It ensures that the system can still complete accurate classification based on alternative features such as head posture, blinking patterns, and yawning behavior even in complex scenarios such as drivers wearing sunglasses, strong light obstruction, or local feature failure. This significantly improves the accuracy, continuity, and physiological relevance of fatigue judgment.

[0096] Figure 3This diagram illustrates an optional vehicle safety warning method according to an embodiment of this application. The diagram visually demonstrates the core hardware architecture and information flow loop of this embodiment. The specific process is as follows: The system uses a monocular RGB camera module installed above the A-pillar of the driver's cabin to collect real-time video streams of the driver's face. After preprocessing, the video streams are input to the central processing unit. This unit is an embedded AI computing platform with a built-in parallel computing architecture, simultaneously executing two independent but collaborative processing channels: one is a driver fatigue monitoring module, which uses a facial key point detection algorithm to detect the head, eye closure, mouth closure, and gaze state, acquiring multi-dimensional features such as eye opening angle (EAR), mouth opening angle, head posture (pitch angle, yaw angle), and gaze direction. After fusion calculation, it outputs a three-level classification result of mild, moderate, or severe fatigue. The second module is the driver distraction detection module, which uses a lightweight target detection neural network to identify targets such as mobile phones, water cups, and cigarettes in images in real time. It also combines the duration of head yaw angle to determine behaviors such as "making a phone call," "drinking water," "smoking," or "conversing." The outputs of both modules are sent to the collaborative decision engine. Based on a preset risk matrix, the engine comprehensively assesses the coupling risk between fatigue level and behavior type, generating a five-level comprehensive judgment: no risk, low risk, medium risk, high risk, or extremely high risk. Finally, the decision result is sent as a text command to the TTS speech synthesis module via the UART interface, triggering the corresponding semantic dialogue to be broadcast through the vehicle's speakers, completing the closed-loop feedback mechanism of "perception-analysis-decision-intervention" to achieve accurate, real-time, and humanized early warning of complex driving risks.

[0097] Figure 4This is a schematic diagram of another optional vehicle safety warning method according to an embodiment of this application. Its core lies in constructing a three-level fatigue grading system with occlusion robustness and multi-feature collaborative judgment capability, and clearly distinguishing the parallel processing logic of fatigue monitoring and distraction detection. Specifically, the driver's facial video stream captured by the camera, after image preprocessing, first enters the driver's head detection module. The system obtains the coordinates of multiple key points on the face through a facial key point localization algorithm, and then calculates the pitch angle, yaw angle, and roll angle of the head based on the PnP algorithm to achieve continuous tracking of the head posture. Subsequently, the system determines in real time whether there is visual occlusion such as the driver wearing a mask or having their eyes obstructed (e.g., sunglasses, hand occlusion, strong light reflection). If the system detects that the driver's eyes are not obstructed, it performs facial key point detection and calculates the eye closure state and eye opening degree EA. The system assesses driver fatigue by detecting the driver's gaze and eye position. If the system detects that the driver is not wearing a mask, it performs mouth opening and closing detection to calculate the degree of mouth opening. If the system detects that the driver is wearing a mask or that their eyes are obstructed (e.g., by wearing sunglasses, using their hands, or glare), the traditional criteria relying on eye and mouth opening and closing may become ineffective. In this case, the system does not interrupt operation but automatically switches to a posture-driven fatigue assessment mode: by monitoring the frequency of low-frequency head nodding, the duration of abnormal yaw angle deviation, and the presence of composite posture patterns such as "drooping head + closed eyes," the fatigue state can still be determined. Simultaneously, the system continues to attempt to extract usable visual information: if the mouth is visible, it performs mouth opening and closing detection, counting the number of yawns and their duration per unit time as an auxiliary fatigue indicator; if the eyes are completely invisible, it relies entirely on head posture and historical behavioral patterns for inference. The calculated head posture, eye closure, mouth closure, and gaze state are compared with preset thresholds to determine the current driver fatigue state. In addition, the driver distraction detection function is an independent channel running in parallel with this system. Its detection targets are objects such as mobile phones, water cups, and cigarettes, as well as conversation behavior. It does not participate in the direct calculation of fatigue level, but is integrated with fatigue results in the subsequent collaborative decision-making module to determine the final risk level. Finally, the warning information corresponding to the risk level is broadcast in a script via TTS.

[0098] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0099] According to the embodiments of this application, Figure 5A structural block diagram of an optional vehicle safety warning device according to an embodiment of this application is provided. It should be noted that this device can be used to execute the aforementioned vehicle safety warning method. The device includes:

[0100] The acquisition module 501 is used to acquire a driving area image sequence, wherein the driving area image sequence is obtained by preprocessing the driving area video stream, and the driving area video stream is used to represent the real-time image sequence corresponding to the driving area.

[0101] Analysis module 502 is used to analyze and process the image sequence of the driving area to obtain multidimensional feature parameters and target driving behavior category. The multidimensional feature parameters are used to determine the driver's facial state and head posture, and the target driving behavior category is used to represent the behavior category of the driver's attention deviating from the driving task.

[0102] The determination module 503 is used to determine the current risk level based on multi-dimensional feature parameters, target driving behavior category and preset risk matrix lookup table, wherein the current risk level is used to represent the degree of risk corresponding to the current driving state;

[0103] The early warning module 504 is used to determine target early warning information based on the current risk level and a preset mapping table. The target early warning information is used to prompt the driver to perform safe driving operations, and the preset mapping table is used to represent the mapping relationship between the current risk level and the target early warning information.

[0104] Furthermore, the acquisition module 501 is also used to: acquire a driving area video stream, wherein the driving area video stream is obtained by image acquisition through an on-board image acquisition component, the on-board image acquisition component being located above the front structural pillar of the vehicle; and preprocess the driving area video stream to obtain a driving area image sequence.

[0105] Furthermore, the acquisition module 501 is also used to: perform grayscale processing on the video stream of the driving area to obtain a grayscale image sequence; perform histogram equalization processing on the grayscale images to obtain an enhanced image sequence; and perform size normalization processing on the enhanced image sequence to obtain a driving area image sequence.

[0106] Furthermore, the analysis module 502 is also used to: extract features from the image sequence of the driving area based on the target detection algorithm to obtain feature extraction results, wherein the feature extraction results include the head pitch angle and the coordinate values ​​of key facial feature points; and analyze and process the feature extraction results to obtain multidimensional feature parameters.

[0107] Furthermore, the key facial feature points include: multiple key eye feature points and multiple key mouth feature points. The multiple key mouth feature points include multiple key points on the outer edge of the upper lip and multiple key points on the outer edge of the lower lip. Multidimensional feature parameters include: eye closure rate, blink frequency, preset number of actions, and nodding frequency. The analysis module 502 is also used to: analyze and process the coordinate values ​​of the multiple key eye feature points to obtain the eye opening degree, where the eye opening degree includes the left eye opening degree and the right eye opening degree, and the eye opening degree is the average of the left eye opening degree and the right eye opening degree; and analyze and process the coordinate values ​​of the multiple key mouth feature points to obtain the mouth opening degree. The parameters are as follows: the degree of mouth opening and closing is determined based on the coordinate values ​​of multiple key points on the outer edge of the upper lip and multiple key points on the outer edge of the lower lip; the eye closure rate is determined based on the duration and a preset time when the eye opening and closing is less than a first preset threshold within a preset time; the blinking frequency is determined based on the number of times the eye opening and closing is greater than the first preset threshold and less than or equal to a second preset threshold within a preset time; the preset number of actions is determined based on the number of times the mouth opening and closing is greater than or equal to a third preset threshold and the duration is greater than or equal to a preset time threshold within a preset time; and the head pitch angle is analyzed and processed to obtain the nodding frequency.

[0108] Furthermore, the analysis module 502 is also used to: perform reasoning processing on the driving area image sequence based on the target reasoning model to obtain the target driving behavior category, wherein the target reasoning model is obtained by training an initial reasoning model based on sample data and the labels corresponding to the sample data, and the sample data includes historical driving area image sequences.

[0109] Furthermore, the determination module 503 is also used to: analyze and process the multidimensional feature parameters and preset thresholds to obtain analysis results, wherein the analysis results are used to determine the fatigue level corresponding to the multidimensional feature parameters; and determine the current risk level based on the fatigue level corresponding to the multidimensional feature parameters, the target driving behavior category, and the preset risk matrix lookup table.

[0110] Furthermore, the multidimensional feature parameters include: eye closure rate, blink frequency, preset number of actions, and nodding frequency. The determining module 503 is also used to: determine the fatigue level as mild fatigue in response to an eye closure rate greater than or equal to a first preset threshold and an eye closure rate less than a second preset threshold, and / or, a blink frequency less than or equal to a third preset threshold, and / or, a preset number of actions equal to a preset first value; determine the fatigue level as moderate fatigue in response to an eye closure rate greater than or equal to a second preset threshold and an eye closure rate less than a fourth preset threshold, and / or, a blink frequency less than or equal to a sixth preset threshold, and / or, a preset number of actions equal to a preset second value, and / or, a nodding frequency greater than or equal to a seventh preset threshold, wherein the sixth preset threshold is less than the third preset threshold; and determine the fatigue level as severe fatigue in response to an eye closure rate greater than or equal to an eighth preset threshold, and / or, a preset number of actions greater than a preset second value, wherein the eighth preset threshold is greater than the sixth preset threshold.

[0111] Embodiments of this application also provide a vehicle, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods described in various embodiments of this application when it runs.

[0112] Embodiments of this application also provide a computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0113] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0114] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium for storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0115] Embodiments of this application also provide a computer program that, when executed by a processor, implements the methods described in the various embodiments of this application.

[0116] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0117] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0118] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0119] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0120] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0121] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A vehicle safety early warning method, characterized in that, include: A driving area image sequence is obtained, wherein the driving area image sequence is obtained by preprocessing a driving area video stream, and the driving area video stream is used to represent the real-time image sequence corresponding to the driving area; The image sequence of the driving area is analyzed and processed to obtain multidimensional feature parameters and target driving behavior categories. The multidimensional feature parameters are used to determine the driver's facial state and head posture, and the target driving behavior categories are used to represent the behavior categories in which the driver's attention deviates from the driving task. Based on the multidimensional feature parameters, the target driving behavior category, and the preset risk matrix lookup table, the current risk level is determined, wherein the current risk level is used to represent the degree of risk corresponding to the current driving state; Based on the current risk level and a preset mapping table, target warning information is determined, wherein the target warning information is used to prompt the driver to perform safe driving operations, and the preset mapping table is used to represent the mapping relationship between the current risk level and the target warning information.

2. The method according to claim 1, characterized in that, The acquisition of the driving area image sequence includes: The driving area video stream is acquired, wherein the driving area video stream is obtained by image acquisition through an on-board image acquisition component, which is located above the front structural pillar of the vehicle; The video stream of the driving area is preprocessed to obtain the image sequence of the driving area.

3. The method according to claim 2, characterized in that, The preprocessing of the driving area video stream to obtain the driving area image sequence includes: The video stream of the driving area is converted to grayscale to obtain a grayscale image sequence; The grayscale image is subjected to histogram equalization to obtain an enhanced image sequence; The enhanced image sequence is subjected to size normalization processing to obtain the driving area image sequence.

4. The method according to claim 1, characterized in that, The analysis and processing of the driving area image sequence yields multidimensional feature parameters including: Based on the target detection algorithm, feature extraction is performed on the image sequence of the driving area to obtain feature extraction results, wherein the feature extraction results include the head pitch angle and the coordinate values ​​of key facial feature points; The feature extraction results are analyzed and processed to obtain the multidimensional feature parameters.

5. The method according to claim 4, characterized in that, The facial key feature points include: multiple eye key feature points and multiple mouth key feature points. The multiple mouth key feature points include multiple upper lip outer edge key points and multiple lower lip outer edge key points. The multidimensional feature parameters include: eye closure rate, blinking frequency, preset number of movements, and nodding frequency. The feature extraction results are analyzed and processed to obtain the multidimensional feature parameters, including: The coordinate values ​​of the multiple key eye feature points are analyzed and processed to obtain the eye opening degree, wherein the eye opening degree includes the left eye opening degree and the right eye opening degree, and the eye opening degree is the average value of the left eye opening degree and the right eye opening degree; The coordinate values ​​of the multiple key feature points of the mouth are analyzed and processed to obtain the mouth opening degree, wherein the mouth opening degree is determined based on the coordinate values ​​of the multiple key points of the outer edge of the upper lip and the multiple key points of the outer edge of the lower lip. The eye closure rate is determined based on the duration during which the eye opening degree is less than a first preset threshold within a preset time period and the preset time period itself. The blinking frequency is determined based on the number of times the eye opening degree is greater than the first preset threshold and the eye opening degree is less than or equal to a second preset threshold within the preset time period. Based on the number of times the mouth opening degree is greater than or equal to a third preset threshold and the duration is greater than or equal to a preset time threshold within the preset time period, the preset number of actions is determined, and the head pitch angle is analyzed and processed to obtain the nodding frequency.

6. The method according to claim 1, characterized in that, The analysis and processing of the driving area image sequence yields target driving behavior categories including: The target driving behavior category is obtained by reasoning the image sequence of the driving area based on the target reasoning model. The target reasoning model is obtained by training an initial reasoning model based on sample data and the labels corresponding to the sample data. The sample data includes historical driving area image sequences.

7. The method according to claim 1, characterized in that, The determination of the current risk level based on the multidimensional feature parameters, the target driving behavior category, and the preset risk matrix lookup table includes: The multidimensional feature parameters and preset thresholds are analyzed and processed to obtain analysis results, wherein the analysis results are used to determine the fatigue level corresponding to the multidimensional feature parameters; The current risk level is determined based on the fatigue level corresponding to the multidimensional feature parameters, the target driving behavior category, and the preset risk matrix lookup table.

8. The method according to claim 7, characterized in that, The multidimensional feature parameters include: eye closure rate, blink frequency, preset number of movements, and nodding frequency. The analysis and processing of these multidimensional feature parameters and preset thresholds yield the following results: In response to the eye closure rate being greater than or equal to a first preset threshold and the eye closure rate being less than a second preset threshold, and / or the blinking frequency being less than or equal to a third preset threshold, and / or the preset number of actions being equal to a preset first value, the fatigue level is determined to be mild fatigue. In response to the eye closure rate being greater than or equal to a second preset threshold and the eye closure rate being less than a fourth preset threshold, and / or the blinking frequency being less than or equal to a fifth preset threshold, and / or the preset number of actions being equal to a preset second value, and / or the nodding frequency being greater than or equal to a sixth preset threshold, the fatigue level is determined to be moderate fatigue, wherein the fifth preset threshold is less than the third preset threshold; In response to the eye closure rate being greater than or equal to a seventh preset threshold, and / or the preset number of actions being greater than the preset second value, the fatigue level is determined to be severe fatigue, wherein the seventh preset threshold is greater than the fifth preset threshold.

9. A vehicle, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the storage medium is located to perform the method according to any one of claims 1 to 8.

Images