Fatigue driving detection and accident early warning system based on YOLOv12 and DeepSeek large model

Abstract

The invention discloses a fatigue driving detection and accident early warning system based on a YOLOv12 and DeepSeek large model, and the system comprises a data collection module which is used for collecting a face image and a video stream of a driver, and constructing a data set; and the model training module is used for outputting a high-precision weight for the fatigue detection module to call based on the YOLOv12 framework training data set. The fatigue detection module is used for analyzing facial features (such as eye closing duration, yawn frequency and head posture) of the driver in real time; the early warning prompt module is used for triggering voice early warning according to the fatigue index threshold value; the AI large model analysis module is used for inputting the continuous behavior feature sequence into a DeepSeek large model for fatigue state evaluation and risk prediction and generating an evaluation report; and the analysis report pushing module is used for encrypting and pushing the fatigue state evaluation report. According to the invention, closed-loop management of monitoring-analysis-early warning-feedback can be realized, fatigue state identification, risk early warning and driving behavior analysis can be automatically completed, and the occurrence rate of traffic accidents caused by fatigue driving can be effectively reduced.

Description

Technical Field

[0001] This invention belongs to the field of automotive safe driving technology, specifically relating to a fatigue driving detection and accident warning system based on YOLOv12 and DeepSeek large models. Background Technology

[0002] With surging traffic volume and extended driving hours, fatigued driving has become a leading global road safety hazard. While traditional single-point behavior detection systems have been initially implemented, their false alarm rates remain high in complex scenarios such as bright sunlight, rain, and fog, and they lack long-term trend prediction, making it difficult to meet the "zero accident" requirement. Secondly, traditional fatigued driving detection models often lack real-time communication with third-party monitoring platforms and cannot automatically push warning information to multi-level interconnected networks such as transportation companies, traffic management departments, or family members, failing to provide timely and effective alerts to drivers.

[0003] Current mainstream solutions rely on multiple sensors or centralized cloud processing, which involves complex hardware deployment, high power consumption, and large latency, limiting the deployment of vehicle-mounted systems. Manual monitoring also suffers from differences in experience and decreased attention, leading to frequent missed and over-alarms, and failing to provide real-time and accurate early warnings.

[0004] Therefore, there is an urgent need to build a new generation of fatigue driving early warning architecture that integrates high-precision visual perception, lightweight edge computing and long-sequence risk modeling, so as to achieve real-time, accurate, low-power detection and early intervention in complex environments. Summary of the Invention

[0005] The purpose of this invention is to provide a fatigue driving detection and accident warning system based on YOLOv12 and DeepSeek large models, in order to solve the problems of poor adaptability to complex scenarios, insufficient prediction of long-term fatigue trends, low real-time power consumption ratio and "islanding" in the existing technology.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] A fatigue driving detection and accident warning system based on YOLOv12 and DeepSeek large models includes:

[0008] Data acquisition module: used to acquire driver facial videos, and after noise reduction and lighting, annotate the state of the eyes and mouth and key points frame by frame, and construct and divide a multimodal training set;

[0009] Model training module: used to train the detection model in the YOLOv12 framework with region attention and R-ELAN backbone to enhance eye and mouth features, and FlashAttention to compress memory;

[0010] Fatigue detection module: used to capture facial data in real time at 30fps and then output the state and confidence level of the trained model in real time;

[0011] The early warning module is used to trigger directional voice prompts when the fatigue state lasts for more than a set time and the confidence level is higher than a threshold value.

[0012] AI Large Model Analysis Module: Used to input the currently collected 2-minute sequence of driver micro-behaviors into the DeepSeek large model to predict risks and generate assessment reports;

[0013] Analysis report push module: The report is encrypted and pushed to the designated email address and fleet management platform.

[0014] Further improvements include manual labeling using the Label lmg tool during the labeling phase of the data acquisition module; and secondary sampling of difficult samples such as those with backlighting, sidelighting, or those wearing glasses / masks to ensure balanced data distribution.

[0015] Further improvements include a tiered voice strategy for the warning and alert module: a gentle alert for mild fatigue, increased volume and shorter playback intervals for moderate fatigue, and a strong alarm for severe fatigue, thereby enhancing driver compliance.

[0016] In a further improvement, the AI ​​large model analysis module utilizes a sliding window mechanism during the inference phase to update the feature sequence every 2 minutes, enabling real-time risk prediction.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] This invention optimizes the detection model through the collaborative use of region attention and FlashAttention, achieving smooth real-time facial state recognition under extremely low power consumption and significantly suppressing false alarms in complex scenes. It also enables deep modeling of continuous driving behavior using a DeepSeek time-series large model, achieving early risk warning and improving active safety margin. At the same time, it establishes a closed loop of "collection-training-detection-prediction-early warning-push", supporting use by multiple vehicle models and regulatory platforms, and providing a highly reliable and low-cost fatigue driving prevention and control solution for intelligent transportation. Attached Figure Description

[0019] Figure 1 The flowchart of the fatigue driving detection system of the present invention is as follows.

[0020] Figure 2 This is a schematic diagram of the overall technical module flow of the present invention.

[0021] Figure 3 This is a schematic diagram of the YOLOv12 network structure described in this invention.

[0022] Figure 4This is a schematic diagram of the R-ELAN structure described in this invention. Detailed Implementation

[0023] This invention proposes a fatigue driving detection and accident early warning system based on YOLOv12 and a DeepSeek large-scale model. It achieves closed-loop management of driver fatigue through a collaborative architecture of spatial feature optimization and long-term behavioral analysis. The system hardware deployment includes an in-vehicle terminal, an edge computing unit, and a locally deployed DeepSeek large-scale model: an infrared camera is installed 15-20cm behind the steering wheel to capture driver facial data at 30fps; the edge terminal uses a module connected to the camera via a USB 3.0 interface for local real-time inference; the locally deployed DeepSeek large-scale model performs deep analysis on the data collected per unit time.

[0024] In this embodiment, as Figure 1 and Figure 2 As shown, the core workflow begins with data acquisition and model training: tens of thousands of driver facial samples are collected and labeled with eye status (open / closed), mouth status (yawning / normal), and coordinates of dozens of facial key points. When training the YOLOv12 model based on the PyTorch framework, a region attention mechanism is embedded (eye region weight coefficient 0.85). An R-ELAN residual structure is used to reduce computational redundancy, and robustness is improved by adding synthetic interference such as raindrop occlusion and strong backlighting. In actual operation, after the driver activates the real-time video detection function, the camera continuously inputs data into the YOLOv12 model, which outputs the state classification (e.g., 80.5% confidence level for closed eyes) and confidence values ​​in real time. The results are simultaneously displayed on the QT interface.

[0025] The system continuously collects driver micro-behavioral sequences within a fixed 2-minute time window. These sequences include, but are not limited to, the number of eyelid closures per unit time and their corresponding confidence levels, and the frequency of yawning events and their corresponding confidence levels. The data is then transmitted to a server, which has a DeepSeek-R1 large-scale model deployed offline for encoding and inference of the behavioral sequences.

[0026] The initial fatigue estimates generated by the large model, along with confidence information, are fed into the Bayesian probability model. This model uses historical fatigue distribution data previously analyzed by the driver as a prior, updates the model function, and calculates the posterior fatigue index in real time. The calculation can be expressed as follows:

[0027]

[0028] Where Fatigue_post is the posterior fatigue index, α is the number of historically observed fatigue occurrences, β is the number of historically observed non-fatigue occurrences, k is the number of samples marked as fatigued within the sliding window, and n is the total number of samples within the window. Subsequently, the system maps Fatigue_post to risk levels 1–5 and calculates the optimal intervention time based on the expected loss function, generating a four-dimensional report containing personalized instructions such as "enter the nearest service area within 20 minutes" or "mandatory rest after 3 km". The formula for the expected loss function described above is:

[0029] E[L]=λ1·P false_alarm +λ2·P miss

[0030] Where E[L] is the expected loss, P false_alarm Let P be the false alarm probability. miss Let λ1 be the false negative probability, λ2 be the unit loss weight of a single false positive, and λ3 be the unit loss weight of a single false negative.

[0031] If the posterior fatigue index Fatigue_post remains above 0.7 for 2 minutes (i.e., 1 time window) and the confidence level is always greater than 0.85, the system will trigger a targeted voice prompt optimized by a reinforcement learning network: "A decline in attention has been detected. It is recommended to rest immediately!" The system's extended application scenarios include logistics fleet management (fatigue reports are automatically pushed to the regulatory platform) and ride-hailing safety supervision.

[0032] Taking long-haul truck drivers as an example: After the vehicle starts, the system automatically monitors the vehicle. When the confidence level of the driver with eyes closed continues to exceed the standard, the voice assistant will issue a warning. The driver can generate a DeepSeek report with one click and obtain the nearest service area recommendation. At the same time, the report is encrypted and sent to the logistics company's platform.

[0033] In this embodiment, as Figure 3 As shown, the fatigue driving detection system based on YOLOv12 further incorporates the latest YOLOv12 architecture to improve detection accuracy, reduce latency, and enhance scenario adaptability. YOLOv12 continues the "backbone-neck-head" three-segment framework in its overall design, with targeted optimizations for each segment, as detailed below.

[0034] Backbone

[0035] YOLOv12's backbone network uses a residual high-efficiency layer aggregation network (R-ELAN), such as... Figure 4 As shown, by deepening the convolutional layers and embedding residual connections, the gradient vanishing problem is alleviated, and the feature reuse rate is improved. The basic convolutional units in the backbone are based on lightweight parallel small convolutional kernels, and their forward computation can be represented as:

[0036]

[0037] Among them, F out To output the feature map, F in For the input feature map, W i b i , where represents the weight and bias of the i-th small convolutional kernel, and * denotes the convolution operation. By splitting the traditional large convolution into multiple small convolutions for parallel computation, the model significantly reduces latency while maintaining feature representation capabilities. Furthermore, the backbone network introduces 7×7 separable convolutions to replace large kernel convolutions or positional encoding, maintaining spatial awareness with fewer parameters; combined with a multi-scale feature pyramid, it ensures that targets of different sizes are fully represented.

[0038] Neck

[0039] The neck structure undertakes multi-scale feature fusion tasks. This embodiment embeds a region attention mechanism accelerated by FlashAttention into the YOLOv12 neck region to achieve real-time inference in high-resolution scenes. The region attention operation can be formally described as:

[0040]

[0041] Where Q, K, and V are the query, key, and value matrices, respectively, and d k The key vector dimension is used. The system divides the feature map into several regions according to the spatial dimension and applies FlashAttention independently to each region, reducing memory movement and computation, thus enabling real-time detection of more than 30fps even on in-vehicle edge devices.

[0042] Head

[0043] The detection head receives the refined feature map from the neck and outputs bounding box coordinates and class confidence. YOLOv12 simplifies the detection path and introduces attention weighting and confidence correction terms into the loss function, with a unified loss description:

[0044] L total =L coord +L conf +L cls +λ·L attn

[0045] Where L total L coord L conf L cls These are the uniform loss, coordinate loss, confidence loss, and classification loss, respectively. attn The loss is an attention-weighted loss, and λ is the balance coefficient. This design further improves the consistency between localization and classification while maintaining end-to-end differentiable training.

[0046] Through the coordinated optimization of the backbone, neck, and head, YOLOv12 improves the detection accuracy in fatigue driving detection tasks, providing more reliable short-term state feature input for subsequent long-term behavior analysis modules. Furthermore, the AI ​​large-scale model analysis module inputs continuously spaced fatigue driving behavior feature sequences into the DeepSeek large-scale model for fatigue state assessment and risk prediction, generating assessment reports that are promptly sent to family members and regulatory authorities, effectively ensuring the driver's safety.

[0047] Matters not covered in this invention shall be governed by publicly known technologies, and all equivalent improvements based on this architecture shall be protected.

Claims

1. A fatigue driving detection and accident warning system based on YOLOv12 and DeepSeek large models, characterized in that The method comprises the following steps: First, the data acquisition module collects the driver's facial image and video stream, performs preprocessing operations such as noise reduction and illumination compensation on the image, uses a labeling tool to label the features of the preprocessed data, constructs a multi-modal data set containing eye state (open / closed), mouth state (yawn / normal) and facial key point coordinates, and divides the data set into a training set, a validation set and a test set according to a proportion; Second, the model training module trains a detection model based on the YOLOv12 framework, enhances the target detection capability by using the AreaAttention method and the Residual Efficient Layer Aggregation Networks (R-ELAN), and adds the FlashAttention to focus on the eye and mouth regions; Third, the fatigue detection module calls the vehicle-mounted camera to capture the driver's facial data, inputs the trained YOLOv12 model at a rate of 30 frames per second, and outputs the current facial state of the driver and the real-time confidence of 0%-100%; Fourth, the early warning module triggers a response according to the fatigue index threshold: when the driver is in a fatigue state for a long time and the confidence is high, the voice assistant is started to play the prompt "attention drop detected, suggest taking a break"; Fifth, the AI large model analysis module inputs the fatigue driving behavior feature sequence of the driver within 2 minutes into the DeepSeek large model module, generates a report on fatigue state evaluation, risk level judgment, recommended measures and trend prediction, and automatically encrypts and pushes the report to a specified mailbox (including the driver's family and the supervisory department).

2. The system of claim 1, wherein: The model training module of the second step comprises: First, a backbone architecture integrating the AreaAttention mechanism and the Residual Efficient Layer Aggregation Networks (R-ELAN) is constructed, the eye and mouth region feature extraction is enhanced through spatial weight distribution, and the cross-layer residual connection is used to reduce the calculation redundancy; Second, the FlashAttention algorithm is deployed in the classification head layer, the memory usage is optimized by using the tiling technology and online soft maximum value calculation, and an eye-mouth region feature priority scheduling mechanism is established; Third, the data enhancement technology is used to perform image flipping, brightness adjustment and blurring on the data set to improve the adaptability to complex environments.

3. The system of claim 1, wherein: The early warning module and the AI large model analysis module of the fourth and fifth steps comprise: First, when the fatigue index continuously exceeds the set threshold for a preset time length and the model confidence is higher than a threshold value, a directional voice prompt is triggered; Second, a time series encoder is constructed, and the input features include the number of closed eyes and confidence per unit time, the number of yawns and confidence per unit time, the number of normal states and confidence per unit time, and the time period in different states; Third, a Bayesian probability model is used to predict the accident risk, and the output results include risk level judgment and trend prediction.

4. The system of any one of claims 1 to 3, wherein: The system is suitable for private cars, commercial vehicles, buses, online car-hailing vehicles and other transportation vehicles, and is compatible with the data docking requirements of logistics vehicle fleet management platforms, online car-hailing supervision platforms and traffic law enforcement systems.

Images