A Distracted Driving Behavior Recognition Method and System Based on Dual-Time-Scale Federated Learning

Abstract

This invention discloses a method and system for recognizing distracted driving behavior based on dual-timescale federated learning, belonging to the field of artificial intelligence and intelligent connected vehicle technology. The method constructs a three-level collaborative architecture: First, at the edge-side group data center, a diffusion model incorporating multimodal conditions is used to enhance the basic visual data, addressing the long-tail distribution problem. Second, a visual Mamba teacher model with adaptive time windows is trained using the enhanced data to extract behavioral features across both short and long timescales. A lightweight student model is then generated in the cloud based on a joint distillation mechanism of features and output distribution. Finally, the vehicle terminal uses local real-time data, combined with global consistency constraints and an experience replay mechanism, to perform federated incremental fine-tuning of the student model. This invention effectively overcomes the computational bottleneck of vehicle-side recognition model deployment while ensuring military-grade privacy, significantly improving the recognition accuracy and real-time performance of instantaneous and continuous distracted behavior in complex scenarios.

Description

Technical Field

[0001] This invention belongs to the field of intelligent connected vehicles and artificial intelligence technology, specifically relating to a method, system, device, and storage medium for distracted driving behavior recognition based on dual-time-scale federated learning, addressing the needs of limited onboard computing resources and privacy protection. This solution is particularly suitable for advanced driver assistance systems (ADAS) and traffic management platforms. Background Technology

[0002] Distracted driving is one of the leading causes of traffic accidents. With the widespread use of in-vehicle entertainment systems, drivers are increasingly diverting their attention from safe driving. Currently, visual information-based distracted driving detection methods are mainstream, but existing technologies face bottlenecks that urgently need to be addressed:

[0003] (1) The contradiction between model performance and vehicle computing power: Deep networks (such as the visual Transformer) have high accuracy but a huge number of parameters, making it difficult to run in real time on vehicle edge devices with limited computing resources; while lightweight models (such as MobileNet) often sacrifice recognition accuracy in complex scenarios. Although the Mamba architecture, which has emerged in recent years, has great advantages in sequence modeling, its direct deployment on the vehicle still faces a huge computing power barrier.

[0004] (2) Data silos and long-tail distribution problems: Distracted driving behavior data involves the high privacy of drivers, and traditional centralized training has a serious risk of leakage. At the same time, in real driving scenarios, extreme lighting, partial occlusion and some rare distracted behaviors (such as reaching for the back seat) show a serious long-tail distribution, resulting in insufficient generalization ability of the model.

[0005] (3) Limitations of traditional federated learning architecture: The existing "cloud-edge (vehicle)" two-layer federated learning architecture faces multiple technical challenges in the training and deployment of large models. On the one hand, the computing resources on the vehicle side are limited, making it difficult to undertake the local training task of large-scale models, which limits the efficiency of model updates and may reduce recognition accuracy. On the other hand, the frequent transmission of large model parameters between the cloud and the vehicle side can easily cause significant communication delays and network congestion, reducing the overall system response speed and real-time performance. At the same time, a single global model is difficult to adapt to the personalized behavioral habits of different drivers, resulting in insufficient accuracy and robustness of the model in cross-individual and cross-scenario applications.

[0006] In summary, the traditional two-layer federated learning architecture has significant shortcomings in meeting the requirements of large-scale model computation, communication efficiency, and personalized adaptation, and urgently needs to be addressed through architectural innovation and algorithm optimization. Summary of the Invention

[0007] To overcome the aforementioned shortcomings in existing technologies, this invention proposes a method and system for recognizing distracted driving behavior based on dual-timescale federated learning. This invention innovatively constructs a three-level collaborative architecture of "cloud-group data center (edge ​​side)-vehicle terminal," solves the long-tail problem of data through a conditional diffusion model, overcomes the vehicle-side computing power bottleneck through federated distillation, and achieves accurate recognition of instantaneous and continuous distracted behaviors through a dual-timescale mechanism.

[0008] The technical solution of the present invention to solve the above-mentioned technical problems is as follows:

[0009] A distracted driving behavior recognition method based on dual-time-scale federated learning includes the following steps:

[0010] Step 1: Construct an enhanced training set; The group data center obtains basic driving vision data uploaded by vehicles in anonymized form, expands the basic data based on the conditional diffusion model with a multimodal conditional encoder, generates diverse distracted driving synthetic data, and merges it with real data to form an enhanced training set;

[0011] Step 2: Cloud-based federated distillation generates a lightweight student model; each group data center uses the enhanced training set to train a visual Mamba teacher model containing a dynamic attention module; the central cloud server receives model update information from each group data center after differential privacy processing, trains a global model based on the cross-node feature and output distribution joint distillation mechanism in the federated distillation learning framework, and distills the global model into a lightweight student model before sending it to the vehicle.

[0012] Step 3: Vehicle-side multi-timescale federated fine-tuning; The vehicle acquires real-time local personalized visual sequence data, and extracts instantaneous distraction features at short timescales and continuous behavioral change features at long timescales through the lightweight student model for recognition and inference; At the same time, under the online federated learning mechanism, the lightweight student model is incrementally fine-tuned in combination with global model consistency constraints.

[0013] Step 4: Global Secure Aggregation Update; Under the federated learning framework, edge nodes or central cloud servers perform weighted aggregation of the finely tuned model update information uploaded by multiple vehicle terminals to generate an updated global lightweight model and synchronize it to each vehicle terminal, forming a continuous optimization closed loop of cloud-edge-device collaboration.

[0014] Furthermore, the data augmentation process of the conditional diffusion model described in step 1 specifically includes: using the Denoising Diffusion Probability Model (DDPM) as the basic architecture, and using driver pose key points, illumination intensity values, and occlusion masks as multimodal conditional inputs; mapping the multimodal conditions into embedding vectors, and then performing cross-attention fusion with the noise prediction network of the diffusion model; during the training process, a joint loss function including diffusion loss, diversity loss, and feature consistency loss is used, wherein the feature consistency loss is used to constrain the cosine similarity between the generated image and the real image in the multimodal feature space.

[0015] Furthermore, the network structure of the visual Mamba teacher model described in step 2 specifically includes: a visual encoder; a Mamba block containing state space projection and bidirectional selective scanning units; a dynamic attention module containing an adaptive time window selection network and a multi-head self-attention computation unit; and a multi-scale feature fusion module.

[0016] Furthermore, the objective loss function of the cross-node feature and output distribution joint distillation mechanism described in step 2 is:

[0017]

[0018] Furthermore, step 3, which incorporates incremental fine-tuning based on global model consistency constraints, specifically includes: freezing the feature extraction layers at the front of the lightweight student model and fine-tuning only the rear network; and introducing a global constraint regularization term into the loss function of the fine-tuning:

[0019]

[0020] The present invention also provides a system, electronic device, and computer-readable storage medium for performing the above-described methods.

[0021] The beneficial effects of this invention include:

[0022] (1) Breaking through the computing power bottleneck of large models on the vehicle: The first "cloud-edge-device" three-layer federated collaborative architecture is used to place the training of the resource-intensive Mamba teacher model in the group data center, and only the distilled lightweight student model is sent to the vehicle. This perfectly solves the contradiction between insufficient computing power and limited communication bandwidth on the vehicle.

[0023] (2) Significantly improve the recognition rate of complex scenes and long-tail data: Data augmentation is performed using a diffusion model based on multimodal conditions, which effectively fits the data distribution under different lighting, occlusion and rare poses, and reduces the false negative rate in real environment.

[0024] (3) Refined dual time scale recognition mechanism: The innovative adaptive time window mechanism is proposed, which enables the model to capture behavioral mutations in a very short time (such as a sudden head drop) and behavioral gradual changes in a long time (such as fatigue accumulation) at the same time, and the recognition accuracy is greatly improved.

[0025] (4) Balancing privacy protection and personalized adaptation: The original images are not left on the vehicle end, and differential privacy noise is introduced by cloud aggregation, achieving military-grade data protection; at the same time, the vehicle end combines the experience playback pool to fine-tune the consistency constraint, so that the model becomes more and more in line with the current driver's habits the more it is used. Attached Figure Description

[0026] Figure 1 is a schematic diagram of the overall "cloud-edge-device" architecture and data flow of the method of the present invention;

[0027] Figure 2 is a flowchart of the data augmentation module based on the conditional diffusion model;

[0028] Figure 3 is a flowchart of the cloud-based federated distillation learning framework;

[0029] Figure 4 is a flowchart of the vehicle-side multi-timescale federated fine-tuning and personalized adaptation module;

[0030] Figure 5 is a schematic diagram of the visual Mamba model network structure;

[0031] Figure 6 is a flowchart of system integration, quantitative deployment and real-time identification. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0033] S101: Overall System "Cloud-Edge-Device" Three-Tier Architecture and Data Flow

[0034] Existing federated learning models that directly facilitate communication between vehicles and the central cloud can lead to severe communication congestion. For example... Figure 1 As shown, this invention innovatively constructs a three-tier physical architecture: a central cloud server, a group data center (as an edge computing node, such as a regional edge server), and vehicle terminals.

[0035] The data flow is as follows: Group data centers collect basic low-resolution data uploaded by intelligent connected vehicles within the region after anonymization, and perform diffusion model enhancement locally. Subsequently, each group data center utilizes its powerful computing capabilities to train a visual Mamba teacher model and performs federated distillation based on differential privacy. The central cloud server aggregates and generates a global teacher model and distills a lightweight student model, which is then distributed to the vehicles. The vehicles rely on onboard cameras for local real-time inference and use local data to perform federated fine-tuning of the student model, uploading only gradient parameters to achieve closed-loop optimization.

[0036] S102: Data Augmentation Module Based on Conditional Diffusion Model

[0037] like Figure 2 As shown, this module aims to solve the problems of long-tail distribution and data scarcity.

[0038] A conditional diffusion model is constructed using the DDPM architecture, incorporating a Transformer-based multimodal conditional encoder. The system extracts driver pose keypoints via OpenPose, and combines these with brightness values ​​and occlusion masks extracted by an illumination estimation network, serving as conditional inputs. These conditional vectors are then fused with the U-Net noise prediction network of the diffusion model through cross-attention.

[0039] Loss function design: To ensure synthesis quality and realism, the total loss consists of three parts: diffusion loss (MSE of predicted noise and real noise), diversity loss (maximum mean difference between generated sample features and cluster centers), and feature consistency loss (the generated image and the real image are input into the pre-trained CLIP model, and their cosine similarity loss in the semantic feature space is calculated, forcing the generated image to be not only similar in appearance but also in spirit).

[0040] S103: Visual Mamba Teacher Model Design and Dual Time Scale Extraction Mechanism

[0041] like Figure 5 As shown, this invention employs an advanced visual Mamba architecture. To address the varying temporal spans of distracted driving behavior, this invention designs an adaptive time window selection network (one of its core innovations). A dynamic attention module is inserted during the Mamba block stacking process. This module dynamically allocates receptive fields based on the intensity of optical flow or feature motion between consecutive frames of the input sequence: for instantaneous behaviors such as "momentarily looking down at a phone," the network automatically allocates a short time window (e.g., 5-10 frames) to extract high-frequency local instantaneous features; for behaviors such as "continuous conversation with a passenger" or "fatigue accumulation," the network allocates a long time window (e.g., 20-30 frames) to extract low-frequency global contextual features. Finally, a multi-scale feature fusion module concatenates the features from both time scales to output the behavior probability.

[0042] S104: Cloud-based Federated Knowledge Distillation and Differential Privacy Aggregation

[0043] like Figure 3 As shown, in order to reduce communication overhead, the data centers of each group do not directly upload the hundreds of megabytes of model parameters, but instead extract intermediate layer features. With output layer logits Perform distillation communication.

[0044] Constructing combined distillation losses:

[0045] Differential privacy protection: Before uploading, the system automatically calculates the gradient or feature clipping threshold C, and adds a value with a mean of 0 and a covariance of 0 to the information. Gaussian noise distribution, ensuring It satisfies strict mathematical constraints on differential privacy.

[0046] S105: Vehicle-side Federated Fine-tuning and Incremental Personalized Learning

[0047] like Figure 4 As shown, in order to adapt the distributed lightweight student model to a specific driver, incremental fine-tuning is performed locally on the vehicle side.

[0048] The strategy employs a freeze-tuning approach: freezing the first 80% of the feature extraction layers to preserve global generalization knowledge, and only adjusting the learning rate of the last 20% of the network layers. Fine-tuning.

[0049] Anti-forgetting mechanism: Forcefully introduce a global constraint regularization term into the fine-tuning loss function. ( (Take 0.01) At the same time, a sample cache pool with a capacity of 5000 frames is established in the vehicle memory. The experience replay mechanism is used to combine with new data for joint training to prevent the model from catastrophic forgetting.

[0050] S106: Model Quantization Compression and Accelerated Deployment of Vehicle Hardware

[0051] To enable millisecond-level inference for the visual Mamba student model on in-vehicle platforms with limited computing power (such as NVIDIA Jetson), this embodiment employs 8-bit integer quantization (INT8) technology. Specifically, a calibration set is constructed using historical normal driving data from the vehicle, and the dynamic range distribution of activation values ​​at each layer of the network in FP32 format is statistically analyzed. The optimal truncation threshold is found using the Kullback-Leibler (KL) divergence minimization principle, thereby quantizing all model weights and activation values ​​into INT8 format with an accuracy loss of less than 1%. After quantization, redundant channel structures are removed (channel pruning), compressing the single forward inference time to less than 50ms.

[0052] S107: Verification of Experimental Results

[0053] Tests were conducted on a public dataset (AUC Distracted Driver Dataset) and a self-built complex multi-illumination dataset. Results show that compared to the traditional FedAvg federated learning method (84.7% accuracy), the student model deployed on the vehicle side has only 6.2M parameters, inference latency is reduced to 42ms, and recognition accuracy soars to 91.8%. Ablation experiments fully demonstrate that conditional diffusion data augmentation (+3.3%), dual-timescale adaptive mechanism (+2.1%), federated distillation (+1.6%), and vehicle-side consistency fine-tuning (+2.6%) all make significant positive contributions to the final metrics, achieving a perfect balance between performance, privacy, and real-time performance.

[0054] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for identifying distracted driving behavior based on double-time-scale federated learning, characterized in that, Includes the following steps: Step 1: Construct an enhanced training set; The group data center acquires basic driving vision data uploaded by vehicles in anonymized form, expands the basic data based on a conditional diffusion model that introduces a multimodal conditional encoder, generates diverse distracted driving synthetic data, and merges it with real data to form an enhanced training set. Step 2: Cloud-based federated distillation generates a lightweight student model; Each group data center uses the enhanced training set to train a visual Mamba teacher model containing a dynamic attention module; The central cloud server receives the model update information from each group data center after differential privacy processing, trains a global model based on the cross-node feature and output distribution joint distillation mechanism in the federated distillation learning framework, and distills the global model into a lightweight student model before sending it to the vehicle. Step 3: Vehicle-side multi-timescale federated fine-tuning; The vehicle-side acquires real-time local personalized visual sequence data, and extracts instantaneous distraction features at short timescales and continuous behavioral change features at long timescales through the lightweight student model for recognition and inference; At the same time, under the online federated learning mechanism, the lightweight student model is incrementally fine-tuned in combination with global model consistency constraints. Step 4: Global security aggregation update; Under the federated learning framework, edge nodes or central cloud servers perform weighted aggregation of the finely tuned model update information uploaded by multiple vehicle terminals, generate an updated global lightweight model, and synchronize it to each vehicle terminal, forming a continuous optimization closed loop of cloud-edge-device collaboration.

2. The method of claim 1, wherein, The data augmentation process for the conditional diffusion model described in step 1 specifically includes: Based on the Denoising Diffusion Probability Model (DDPM), the driver's attitude key points, illumination intensity values, and occlusion masks are used as multimodal conditional inputs. After mapping the multimodal conditions into embedding vectors, cross-attention fusion is performed with the noise prediction network of the diffusion model; During training, a joint loss function is used, which includes diffusion loss, diversity loss and feature consistency loss, wherein the feature consistency loss is used to constrain the cosine similarity between the generated image and the real image in the multimodal feature space.

3. The method according to claim 1, characterized in that, The network structure of the visual Mamba teacher model described in step 2 specifically includes: The visual encoder divides the input visual image into image blocks and linearly maps them into feature vectors; the Mamba block contains state space projection and bidirectional selective scanning units to extract temporally dependent features; the dynamic attention module contains an adaptive temporal window selection network and a multi-head self-attention computation unit to dynamically capture behavioral features at different time scales; and the multi-scale feature fusion module extracts the outputs of Mamba blocks at different depths, performs unified sampling and channel concatenation, and outputs the probability distribution of behavioral categories after pooling and fully connected layers.

4. The method of claim 1, wherein, The objective loss function of the cross-node feature and output distribution joint distillation mechanism in the federated distillation learning framework described in step 2 is: in, The intermediate layer features uploaded to the data centers of various groups These are intermediate layer features of the global model. Let the mean squared error loss function be used. Output layer logical values ​​(logits) uploaded to each group's data center. This is the output layer logic value of the global model. For Kullback-Leibler divergence, and This is the weighting adjustment coefficient.

5. The method of claim 1, wherein, Step 3, which involves incrementally fine-tuning the lightweight student model in conjunction with global model consistency constraints, specifically includes: The feature extraction layer at the front of the lightweight student model is frozen, and only the network layer and classification layer at the back are fine-tuned. A global constraint regularization term is introduced into the fine-tuned loss function, and its formula is as follows: Among them, L ce For cross-entropy loss, θ local For local model parameters, θ global Here are the global model parameters, and η is the regularization coefficient to prevent local overfitting.

6. The method of claim 1, wherein, The incremental fine-tuning described in step 3 adopts an incremental update algorithm based on a sample cache pool and experience replay: a sample cache pool with a fixed capacity is established on the vehicle side, and local training is triggered when the number of new samples reaches a preset threshold; during local training, historical data in the sample cache pool is combined for experience replay to alleviate catastrophic forgetting; the learning rate step size of the model update is dynamically adjusted according to the distribution characteristics of the data on the vehicle side.

7. The method of claim 1, wherein, The differential privacy processing specifically involves adding Gaussian distributed noise that meets differential privacy constraints to the intermediate layer feature and logical value information before uploading it to the group data center. The variance of the noise is positively correlated with the gradient or feature pruning threshold and a preset noise intensity parameter.

8. A distraction driving behavior recognition system based on double-time-scale federated learning, characterized in that, include: Enhanced training set construction module: Deployed in the group data center, it is used to acquire basic driving vision data uploaded by vehicles in anonymized form, expand the basic data based on the conditional diffusion model with multimodal conditional encoder, generate synthetic data of distracted driving with diversity, and fuse it with real data to form an enhanced training set; Cloud-based federated distillation generation module: Deployed on a central cloud server and group data centers, each group data center uses the enhanced training set to train a visual Mamba teacher model containing a dynamic attention module; the central cloud server is used to receive model update information from each group data center after differential privacy processing, train a global model based on the cross-node feature and output distribution joint distillation mechanism in the federated distillation learning framework, and distill the global model into a lightweight student model before sending it to the vehicle terminal. Vehicle-side multi-timescale fine-tuning module: Deployed on the vehicle, it is used to acquire real-time local personalized visual sequence data, extract instantaneous distraction features at short timescales and continuous behavioral change features at long timescales through the lightweight student model for recognition and inference; at the same time, under the online federated learning mechanism, the lightweight student model is incrementally fine-tuned in combination with global model consistency constraints. Global Security Aggregation Update Module: Deployed on edge nodes or central cloud servers, under the federated learning framework, it is used to perform weighted aggregation of fine-tuned model update information uploaded by multiple vehicle terminals, generate an updated global lightweight model, and synchronize it to each vehicle terminal.

9. The system of claim 8, wherein, The data augmentation process of the conditional diffusion model in the augmented training set construction module specifically includes: using the denoised diffusion probability model as the basic architecture, and using driver posture key points, illumination intensity values, and occlusion masks as multimodal condition inputs; mapping the multimodal conditions into embedding vectors, and then performing cross-attention fusion with the noise prediction network of the diffusion model; and using a joint loss function that includes diffusion loss, diversity loss, and feature consistency loss during the training process.

10. The system of claim 8, wherein, The network structure of the visual Mamba teacher model in the cloud-based federated distillation generation module specifically includes: The system includes a visual encoder, a Mamba block containing state space projection and bidirectional selective scanning units, a dynamic attention module containing an adaptive time window selection network and a multi-head self-attention computation unit, and a multi-scale feature fusion module.

11. The system of claim 8, wherein, The incremental fine-tuning in the vehicle-side multi-timescale fine-tuning module adopts an incremental update algorithm based on a sample cache pool and experience replay: A fixed-capacity sample cache pool is established, and local training is triggered when the number of new samples reaches a preset threshold. During local training, historical data in the sample cache pool is used for experience replay. The learning rate step size of the model update is dynamically adjusted according to the distribution characteristics of the vehicle data.

12. The system according to any one of claims 8-11, characterized in that, When the modules work together, they perform the method described in any one of claims 1-7.

13. An electronic device, comprising: include: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1 to 7. 14.A vehicle edge computing terminal, characterized in that, This includes in-vehicle AI chips, camera interfaces, and communication components; The vehicle edge computing terminal is loaded with a lightweight student model generated by the method of any one of claims 1 to 7. The lightweight student model is processed by channel pruning and 8-bit integer quantization and is used to perform low-latency inference on the driving vision data acquired by the camera interface and participate in vehicle-side federated fine-tuning via the communication component.

15. A computer readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method as described in any one of claims 1 to 7.

Images