A dangerous driving behavior detection method based on heterogeneous federated ensemble learning

Abstract

The application relates to the technical field of driver monitoring, in particular to a dangerous driving behavior detection method based on heterogeneous federal integrated learning; a client extracts a feature vector of driving behavior data, and maps the feature vector into a low-dimensional category prototype by using a prototype adapter network; local dynamic differential privacy technology is used to perform noise injection encryption processing on prototype adapter parameters, and the parameters are uploaded to a server after encryption; cross-client prototype alignment is performed on the server side, and global prototypes are generated by aggregation; the client downloads global prototype parameters and updates a local model, and performs incremental learning based on a dynamic prototype library; when a new dangerous driving behavior is detected, a new prototype is stored separately from historical prototypes; the client performs similarity matching on real-time collected driving behavior features and the global prototypes to obtain a detection result. The application provides an extensible and highly reliable federal learning framework for the landing of an intelligent automobile safety system from three dimensions of heterogeneous compatibility, privacy-efficiency balance and personalized dynamic updating.

Description

Technical Field

[0001] This invention relates to the field of driver monitoring technology, specifically to a method for detecting dangerous driving behaviors based on heterogeneous federated ensemble learning. Background Technology

[0002] With the rapid development of the automotive industry, intelligent vehicles have become an important development direction. As the most crucial participant in the driving process, the driver's condition directly affects traffic safety. Statistics show that the vast majority of traffic accidents are caused by driver error. Therefore, real-time monitoring of the driver's condition and timely detection and warning of dangerous driving behaviors are of great significance for reducing traffic accident rates and protecting people's lives and property. Driver Monitoring Systems (DMS) have emerged to address this need. They utilize various sensors and algorithms to monitor the driver's physiological and behavioral characteristics in real time, determine whether the driver is fatigued, distracted, or engaging in dangerous driving behaviors such as violations, and promptly issue warnings or take intervention measures to ensure driving safety.

[0003] Meanwhile, advancements in artificial intelligence and big data technologies have provided strong technical support for the development of DMS. Machine learning algorithms, such as deep learning, have achieved remarkable results in areas such as image recognition and behavior analysis, making intelligentization of DMS possible. The development of technologies such as cloud computing and edge computing has provided powerful computing resources and flexible deployment methods for DMS data storage, processing, and model training.

[0004] Among numerous DMS (Driver Monitoring System) solutions, driver behavior recognition technology based on video image analysis has received widespread attention due to its advantages such as being contactless, low-cost, and easy to deploy. This technology typically uses in-vehicle cameras to capture facial images or videos of the driver, and then utilizes computer vision and machine learning algorithms to extract information such as the driver's eye movement features, head posture, and facial expressions, thereby determining the driver's state. Common dangerous driving behaviors include:

[0005] Fatigue driving: Symptoms include frequent blinking, prolonged eye closure, and yawning.

[0006] Distracted driving: This manifests as having one's eyes off the road, using a mobile phone, or talking to passengers.

[0007] Traffic violations include not wearing a seatbelt, smoking, and not using both hands to control the steering wheel.

[0008] In light of the above background, existing technical solutions mainly focus on the following aspects:

[0009] Centralized data processing-based DMS (Data Management System): These systems upload all user data to the cloud for storage and training. While they can utilize the powerful computing resources of cloud computing, they pose a serious risk of data privacy breaches. For example, patent CN106611169A, "A Real-time Detection Method for Dangerous Driving Behavior Based on Deep Learning," involves building a forward vehicle image acquisition system, collecting training samples, and manually labeling them to create a dangerous driving behavior dataset; then, a deep learning-based model is used to train the model. This method is a centralized training method, which carries the risk of data privacy breaches.

[0010] DMS based on traditional machine learning: These systems typically use traditional machine learning algorithms (such as support vector machines, decision trees, etc.) for driving behavior recognition. Although the computational complexity is low, the model performance is limited by the expressive power of the algorithm itself, making it difficult to handle complex driving scenarios.

[0011] DMS Based on Simple Federated Learning: Some research has begun to explore applying federated learning to DMS to protect user data privacy. However, these methods typically only consider simple model fusion methods such as model averaging or weighted averaging, making it difficult to effectively handle the problems caused by heterogeneous data and devices. For example, some literature proposes a basic federated averaging algorithm (FedAvg), which trains the model locally on each client and then uploads the model parameters to the server for averaging. This method is a classic approach to federated learning, but it does not consider the heterogeneity of data and devices. Another patent, CN115661934A, "A Method and Device for Detecting Hazardous Driver Behavior Based on Federated Learning," uses a deep learning object detection algorithm to solve the problems of high cost and uncomfortable wearing of traditional vehicle sensors. Simultaneously, it combines object detection with federated learning to break down data silos, achieve data sharing, and reduce latency. Model inference is completed locally without requesting the server, avoiding network transmission latency. Furthermore, this method solves the problem of large-scale data processing, reduces server and storage pressure, and saves computing resources. Most importantly, local clients do not upload data, protecting user privacy and data security.

[0012] In summary, the shortcomings of existing technologies are as follows:

[0013] (1) Insufficient heterogeneous compatibility;

[0014] Existing federated learning frameworks (such as FedAvg and FedProx) rely on the assumptions of isomorphic client-side model architecture and identically distributed (IID) data. However, in DMS scenarios, the sensor types, computing capabilities, and model structures of in-vehicle devices vary significantly, leading to the following problems:

[0015] Strong coupling of model architecture: Traditional solutions require that the model structure of the client and server be completely consistent (such as FedAvg which requires global model parameter alignment). However, due to hardware resource limitations (such as low-end devices only supporting lightweight CNNs), actual in-vehicle devices cannot deploy complex models (such as Transformer), leading to model splitting and performance degradation.

[0016] Non-IID data adaptation failure: Differences in client data distribution (such as differences in driver behavior patterns across different regions) lead to a decrease in generalization ability after global model aggregation. Experiments show that FedAvg's accuracy on the CIFAR-100 dataset decreases by 21.3% in non-IID scenarios compared to IID scenarios.

[0017] (2) Privacy-efficacy imbalance;

[0018] Existing privacy protection technologies struggle to balance security and model performance:

[0019] Noise injection leads to accuracy loss: Traditional differential privacy (DP) adds noise (such as Gaussian noise) to the gradient or parameters. Although it can mask individual data features, it will reduce the convergence speed of the model (experiments show that in EMNLP2022, DP increases the number of convergence rounds of federated learning by 2.3 times) and the final accuracy will drop by 8%-15%.

[0020] The computational overhead of encryption is too high: Although homomorphic encryption (HE) can achieve ciphertext aggregation, its computational complexity far exceeds the real-time requirements of vehicle-mounted devices (requiring a response within 100ms), making it unsuitable for high-throughput scenarios such as DMS.

[0021] (3) Delayed dynamic updates and lack of personalization;

[0022] Existing model update schemes suffer from response latency and adaptation rigidity issues:

[0023] Incremental learning relies on centralized data: common incremental learning algorithms require centralized storage of historical data in the cloud to calculate parameter importance, which violates privacy protection principles (requiring the uploading of original video streams) and cannot cope with newly added categories on the device (such as new dangerous actions).

[0024] Federated continuous learning is inefficient: Solutions such as Federated Weighted Inter-client Transfer (FedWeIT) achieve task increment through parameter masking, but their model update cycle is long and they cannot dynamically adjust the model for user-specific features (such as special driving postures), resulting in an increased false alarm rate. Summary of the Invention

[0025] To address the aforementioned problems, the present invention aims to provide a dangerous driving behavior detection method based on heterogeneous federated ensemble learning. By addressing the key bottlenecks of existing DMS (Driving Management System) technologies through three dimensions—heterogeneous compatibility, privacy-performance balance, and personalized dynamic updates—this method provides a scalable and highly reliable federated learning framework for the implementation of intelligent vehicle safety systems. The technical solution is as follows:

[0026] A method for detecting dangerous driving behavior based on heterogeneous federated ensemble learning includes the following steps:

[0027] Step 1: Client-side local prototype learning: Each in-vehicle device extracts features of driver behavior data through a pre-trained model, and uses a lightweight prototype adapter module to learn the category prototype of local data from the features; the prototype adapter includes multiple convolutional layers, a parameterless dynamic frequency domain attention module, and a fully connected layer, and its output is a low-dimensional category prototype vector.

[0028] Step 2: Privacy-preserving transmission: Apply local dynamic differential privacy encryption to the prototype adapter parameters, dynamically adjust the privacy budget calculation noise ratio and add Gaussian noise to generate encrypted adapter parameters;

[0029] Step 3: Server-side global prototype aggregation: The federated server receives the encrypted parameters uploaded by each client, aligns the feature spaces of heterogeneous clients using Wasserstein distance, and aggregates the aligned prototype feature spaces using a contrastive learning strategy to generate a global prototype adapter.

[0030] Step 4: Client Model Update: Each client downloads the global prototype adapter, performs comparative loss optimization based on local data, updates the local prototype adapter model, and completes the real-time detection of dangerous driving behavior;

[0031] Step 5: Prototype Incremental Learning: The server maintains a global prototype library. Through a dynamic hybrid training strategy and knowledge distillation loss, it incrementally learns to add new driving behavior categories and updates the global prototype library.

[0032] Step 6: The client performs similarity matching between the real-time collected driving behavior features and the global prototype, and outputs the dangerous driving behavior detection results.

[0033] The beneficial effects of this invention are:

[0034] 1) This invention employs a class prototype-based federated ensemble learning framework:

[0035] Client-side local prototype learning: The client learns category prototypes on local data, effectively extracting and representing category features of the local data.

[0036] Server-side global prototype aggregation and prototype alignment: The server performs global prototype aggregation and introduces a prototype alignment mechanism to solve the feature space misalignment problem caused by model heterogeneity, ensuring that the global prototype can effectively integrate knowledge from different clients.

[0037] Client-side global prototype optimization of local model: The client uses global prototypes to optimize the local model, integrating global knowledge into the local model training to improve the model's generalization ability and robustness.

[0038] 2) This invention adopts a client-side dynamically adapted prototype adapter structure:

[0039] Lightweight Prototype Adapter Module: A lightweight prototype adapter module is inserted on the client pre-trained model to decouple feature extraction and prototype learning, and introduces a parameterless attention module (DSA) to reduce computation and communication overhead.

[0040] The prototype adapter is responsible for category prototype mining: The prototype adapter focuses on mining category prototypes from the features extracted from the pre-trained model, improving prototype learning efficiency and model accuracy.

[0041] 3) This invention employs local dynamic differential privacy encryption technology:

[0042] Client-side parameter encryption: Before uploading prototype adapter parameters to the client, local dynamic differential privacy technology is used to encrypt the parameters to protect user data privacy.

[0043] Dynamic privacy budget adjustment: The privacy budget can be dynamically adjusted according to the actual application scenario and privacy needs to balance model performance and privacy protection.

[0044] 4) This invention employs a prototype incremental learning strategy:

[0045] Global prototype library maintenance: The server maintains the global prototype library, storing global prototype information at historical moments to enable the model's continuous learning capability.

[0046] Prototype incremental learning: The model can incrementally learn prototype information in new data, adapt to the evolution of data distribution, and avoid catastrophic forgetting.

[0047] 5) This invention employs contrastive learning to accelerate prototype adaptation:

[0048] Introduction of contrastive learning method: Introducing contrastive learning method into the prototype adapter training process accelerates the global prototype and local prototype adaptation process and improves model training efficiency.

[0049] Prototype similarity and difference learning: Through comparative learning, the model can learn the similarities and differences between prototypes, improve the quality of feature representation, and accelerate prototype adaptation. Attached Figure Description

[0050] Figure 1 This is a training framework diagram for a dangerous driving behavior detection method based on heterogeneous federated ensemble learning.

[0051] Figure 2 This describes the training steps for a dangerous driving behavior detection method based on heterogeneous federated ensemble learning. Detailed Implementation

[0052] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0053] This invention proposes a class prototype-based federated ensemble learning framework. For heterogeneous data, clients learn the class prototypes of their local data, i.e., class feature representations. The server aggregates the prototypes from each client to obtain a global prototype. Clients then use the global prototype to optimize their local models, adapting them to the overall data distribution. For device heterogeneity, prototype learning does not require identical model architectures, only consistent feature space dimensions. Each device can perform feature extraction and model updates based on its own resources, achieving distributed feature extraction and learning. Regarding model heterogeneity, prototype learning achieves knowledge sharing by passing class prototypes. Clients train personalized models while also interacting with the global model, eliminating the need for a unified model architecture. Different models achieve collaborative optimization through prototype interaction and alignment, thereby solving the heterogeneity problem and improving federated learning performance.

[0054] This invention proposes a dynamically adaptable federated learning framework for the client. This scheme inserts a lightweight prototype adapter onto the existing model on each terminal device. The original pre-trained model is primarily responsible for extracting data features, while the prototype adapter mines class prototypes based on the extracted features, ultimately achieving localized data storage and processing. The prototype adapter contains three convolutional layers, one average pooling layer, and one fully connected layer. Each convolutional layer includes a parameter-free attention module (Dynamic Spectral Attention). The prototype adapter's parameters are theoretically less than 1M, making it particularly suitable for resource-constrained devices (such as smartphones and IoT devices). During training, local dynamic differential privacy technology is used to encrypt the adapter parameters. Only the encrypted adapter parameters are transmitted, rather than the original data, for aggregation on the server side. This effectively protects the privacy of users' driving behavior data while ensuring model training effectiveness, solving the data security risks faced by traditional centralized systems.

[0055] This invention proposes a distributed, dynamically updated training scheme based on federated learning. This scheme introduces a prototype incremental learning strategy, enabling the model to continuously learn from new data. Simultaneously, it accelerates the global and local prototype adaptation process using a contrastive learning method, constructing personalized models for different driving scenarios and user characteristics. This achieves rapid iterative updates and accurate adaptation of the model, effectively addressing the limitations of traditional centralized training models and improving the system's practicality and scalability.

[0056] 1. Overall Framework:

[0057] This invention proposes a dangerous driving behavior detection method based on heterogeneous federated ensemble learning. Its core architecture consists of a client (in-vehicle device) and a federated server. Distributed training and dynamic updates are achieved through a prototype-based federated ensemble learning framework, constructing an efficient, secure, and scalable intelligent driver monitoring system. The system workflow includes four stages: local prototype learning, privacy-preserving transmission, global prototype aggregation, and incremental prototype expansion. The specific architecture is as follows: Figure 1 As shown.

[0058] The framework mainly consists of two parts: a client and a server.

[0059] (1) Client (Vehicle Terminal Side): This typically refers to the vehicle computing unit installed on the terminal equipment of a smart car, such as an in-vehicle infotainment system or domain controller. The main functions of the client include:

[0060] 1) Data collection and preprocessing: The vehicle-mounted camera collects driver behavior data and performs preliminary preprocessing locally, such as data cleaning and format conversion.

[0061] 2) Feature Extraction (Pre-trained Model): High-dimensional feature representations are extracted from the pre-processed data using a pre-trained deep learning model (such as MobileNet, ResNet, etc.). This pre-trained model can be a general image feature extraction model or a model pre-trained for driving behavior data.

[0062] 3) Prototype Adapter Learning (Local Prototype Learning): Based on the features extracted by the pre-trained model, a prototype adapter module is inserted. This adapter is responsible for learning the class prototype of the local data, that is, abstracting and representing the features of each class of driving behavior (normal driving, fatigued driving, distracted driving, illegal driving, etc.).

[0063] 4) Local Dynamic Differential Privacy Encryption: To protect user privacy, Local Dynamic Differential Privacy (LDDP) technology is used to encrypt the adapter parameters before uploading the prototype adapter parameters, and an appropriate amount of noise is added.

[0064] 5) Parameter Upload (to Server): Upload the encrypted prototype adapter parameters to the federated learning server.

[0065] 6) Global model parameter download (from server): Download the aggregated global model parameters (including the global prototype) from the server to update the local model and improve its performance.

[0066] 7) Local model update: Optimize the local prototype adapter model using the downloaded global model parameters and apply the dangerous driving behavior detection application locally.

[0067] Based on their computing power, clients can be categorized into strong clients, medium clients, and weak clients, for example:

[0068] 1) Strong client: Usually equipped with a high-performance computing platform, capable of supporting the training of large models and complex computing tasks.

[0069] 2) Medium-sized client: with moderate computing power, capable of training medium-sized models.

[0070] 3) Weak clients: Limited computing resources, only able to train small models.

[0071] (2) Server (Federated Learning Server): This usually refers to a cloud server or edge server, responsible for the global coordination and model aggregation of federated learning. The main functions of the server include:

[0072] 1) Prototype Alignment (for model heterogeneity): Receives prototype parameters uploaded from different clients and performs prototype alignment. Prototype alignment aims to resolve potential differences in model structures between different clients, enabling prototype representations from different models to be aggregated in a unified feature space.

[0073] 2) Prototype Aggregation (Global Prototype Generation): The aligned prototypes are federated to obtain the global prototype. This invention uses contrastive incremental learning for prototype aggregation, and the global prototype represents the global and representative knowledge of category features in all client data.

[0074] 3) Prototype Adapter Aggregation (Global Prototype Adapter Generation): Receives prototype adapter parameters uploaded from different clients, performs secure aggregation, and obtains a global prototype adapter.

[0075] 4) Global model parameter distribution: Based on the aggregated global prototype and global adapter model, distribute them to each client.

[0076] 2. Technical implementation schemes for each functional module;

[0077] The following will elaborate on several core technical solutions of the present invention, including their specific technical implementation and principles.

[0078] 2.1 A prototype-based federated ensemble learning framework (addressing the heterogeneity issues of data, devices, and models):

[0079] To address the issue of heterogeneity in data, devices, and models, this invention proposes a class prototype federated ensemble learning framework. Its core idea is to use class prototypes as a bridge for knowledge transfer. During the federated learning process, the client and server do not directly exchange raw data or model parameters (pre-trained model parts), but instead exchange class prototypes extracted from the data, thereby achieving collaborative training and performance improvement of heterogeneous models.

[0080] (1) Client-side local prototype learning: Each client learns the category prototype of local data using a prototype adapter. Specifically, the prototype adapter receives features extracted by the pre-trained model as input and learns the category prototype representation through a prototype adapter based on the SimAM attention mechanism. The category prototype can be understood as a generalized description of the features of each category in the local dataset. It can capture the key feature information of the local data, while reducing the data dimensionality and reducing the risk of privacy leakage.

[0081] Technical Implementation: The client-side local prototype learning solution is implemented based on a prototype adapter network. This network consists of three convolutional layers, three DSA parameterless attention modules, a global average pooling layer, and a fully connected layer. The first convolutional layer receives features extracted from a pre-trained model (such as ResNet50 or MobileNet), with the number of input channels matching the output dimension of the pre-trained model (e.g., 2048 for ResNet50), and the number of output channels is 512, using a 3×3 convolutional kernel, a stride of 1, and padding of 1. The second convolutional layer converts the 512 channels to 256 channels, and the third convolutional layer converts the 256 channels to 128 channels, both using the same convolutional parameter configuration. Each convolutional layer is followed by a batch normalization layer (momentum 0.9), a DSA parameterless attention module, and a ReLU activation function. After being processed by three convolutional layers, the features are reduced to a 128-dimensional vector through global average pooling, and then converted into a 64-dimensional category prototype through the fully connected layer. Training employs the Adam optimizer (learning rate 0.001, weight decay 5e-4), with a default batch size of 32 (adjustable to 16-64 depending on computational capabilities). Training consists of 100 epochs, with the learning rate decaying to 0.1 times its original value every 15 epochs. The loss function combines contrastive loss (temperature parameter 0.07) and cross-entropy loss (weight 0.5). The former clusters similar samples and separates dissimilar samples, while the latter enhances classification performance. During inference, a feature fusion strategy is used, concatenating the original features with the prototype vectors and performing final classification through a two-layer fully connected network (128 → 64 → number of categories).

[0082] The DSA module generates attention weights and applies them to the input feature map through the following steps:

[0083] 1) Use fixed Gaussian and Laplace filters to extract low-frequency and high-frequency components.

[0084] 2) Calculate the spectral energy ratio to reflect the smoothness and detail of the features.

[0085] 3) Calculate spatial attention based on mean and variance: ;

[0086] 4) Integrate spectral information to generate final attention weights: ;

[0087] Formula explanation and hyperparameters:

[0088] : Represents the value of the feature map at channel c and spatial location (i, j).

[0089] The mean of each channel, .

[0090] Variance of each channel, .

[0091] Spectral energy ratio .

[0092] Mean spectral ratio.

[0093] β=2.0: The contribution of the control spectrum ratio.

[0094] =1×10 ﹣5 : To prevent the divisor from being zero.

[0095] λ=1×10 ﹣4 Regularization coefficient.

[0096] Principle Explanation: The client-side local prototype learning technology is based on the principle of category prototype representation. It compresses high-dimensional features into low-dimensional, efficient category prototype representations through dimensionality reduction transformation. The core idea of ​​this method is to learn a compact and information-rich prototype representation for each predefined category (e.g., "fatigued driving," "distracted driving," etc.), which captures the key features and common patterns of that category's data. The prototype adapter uses multi-layer convolutional transformations and parameterless attention mechanisms to filter and enhance information relevant to category discrimination in the features, while suppressing irrelevant information, thereby reducing feature dimensionality while retaining key information. The contrastive learning strategy clusters similar samples and separates dissimilar samples in the prototype space, enhancing category separability. Category prototype representations offer several advantages over original high-dimensional features: First, their low-dimensionality (64 dimensions compared to the original 2048 dimensions) significantly reduces computational and storage requirements; second, prototype vectors, by extracting common features, reduce interference from individual differences and enhance the model's generalization ability to new samples; third, different clients can learn the category prototypes of their local data, and due to dimensionality reduction and the extraction of common features, the prototype spaces are easier to align and aggregate even if the original data distributions differ; most importantly, prototype vectors, through dimensionality reduction and extraction of common category features, effectively reduce information containing individual characteristics, thereby enhancing privacy protection and reducing the risk of data leakage. This design makes this technical solution particularly suitable for application in privacy-conscious distributed environments.

[0097] (2) Server prototype alignment and global prototype aggregation:

[0098] Technical Implementation: Due to potential differences in model structures across different clients (model heterogeneity), directly aggregating prototype vectors from different models may lead to feature space misalignment. To address this issue, this invention introduces a prototype alignment mechanism on the server side. When the feature spaces of different clients differ, a feature mapping can be learned using Wasserstein distance to map the features from different clients to a unified feature space. Specifically, the Wasserstein distance between the source and target domains on the mapped features can be minimized, thereby achieving cross-domain feature alignment. After prototype alignment, the server employs a contrastive learning algorithm to aggregate the aligned prototypes from all clients to obtain a global prototype.

[0099] Specific steps:

[0100] 1) Server-side prototype alignment: First, represent the prototype extracted by each client as a discrete probability distribution; second, define the Wasserstein distance to measure the difference between the mapped prototype and the target distribution (global prototype); then, efficiently calculate the Wasserstein distance using the Sinkhorn iterative algorithm, iteratively updating the bivariate until convergence (when the distance change is less than 10). ﹣5 (Or when the maximum number of rounds reaches 200); then, a feature mapping function is learned by minimizing this distance; finally, the mapping function is iteratively optimized until the Wasserstein distance converges (when the distance change is less than 10). ﹣5 Or when the maximum number of rounds reaches 200).

[0101] 2) Global Prototype Aggregation: After feature alignment, the server further aggregates the aligned prototype vectors uploaded by all clients, using the InfoNCE loss function from contrastive learning to enhance consistency among prototypes of the same class. Specifically, prototype vectors belonging to the same category from different clients are considered positive sample pairs, while prototypes from different categories are considered negative samples. In each training round, the server samples prototype vectors from several categories from multiple clients, forming a prototype batch containing multiple classes and samples. For each prototype vector, another prototype of the same category is selected as its positive sample, and the remaining prototypes are used as negative samples. Its similarity to all other prototypes is calculated (using normalized cosine similarity), and training is performed using the following InfoNCE loss function: ;

[0102] Among them, z i Indicates the anchor point prototype. Let τ be the temperature coefficient (usually set to 0.07), Nt be the total number of prototypes in the batch, and S(•,•) represent the similarity function (such as cosine similarity). This loss encourages prototypes of the same category to be close together in the feature space, while prototypes of different categories are kept far apart, thereby further strengthening the clustering of category features in the aligned space. The server-side updates the parameters of the prototype mapping network through the Adam optimizer, so that the learned unified feature space not only achieves cross-client alignment but also retains clear inter-class distinguishability. The final output global prototype aggregation result can effectively represent the globally shared features of each category.

[0103] Explanation: Prototype alignment ensures the consistency of feature space outputs from models across different clients, making subsequent aggregation operations more efficient. The federated averaging algorithm effectively integrates model knowledge from various clients to obtain a global prototype representing the global data distribution. This global prototype can be viewed as a summary of common knowledge regarding categorical features across all client data.

[0104] 2.2 Local Dynamic Differential Privacy Encryption (Solving Data Security and Privacy Issues):

[0105] Technical Implementation: This invention employs a Gaussian mechanism to achieve local dynamic differential privacy. The core implementation can be summarized as follows: before the client uploads the prototype adapter parameters, carefully designed Gaussian noise is added to the parameters to obscure the user's sensitive information, thereby achieving privacy protection.

[0106] Implementation steps:

[0107] 1) Local model training: Each in-vehicle device trains a local model using locally collected driving behavior data.

[0108] 2) Local Dynamic Differential Privacy Encryption: Local dynamic differential privacy encryption technology is implemented through a Gaussian mechanism. Its core processing flow is as follows: First, a basic privacy budget (ε) is set. base =0.5), privacy relaxation parameter (δ0=10) ﹣5 The parameters are importance factor (γ=0.5) and gradient pruning threshold (C=3.0); in each iteration, the privacy budget is dynamically calculated:

[0109] ε t =ε base ·(1+α·k / K ) φ ;

[0110] Where α is the adjustment magnitude factor (0.5-2.0), φ is the nonlinear adjustment exponent (0.8-1.2), and k is the current training iteration number. K This represents the total number of iterations; subsequently, gradient calculations are performed and gradient clipping is applied.

[0111] For different parameter sets based on importance w i Calculate your personalized privacy budget ε i :

[0112] ε i =ε t ·(w i ·γ+(1-γ));

[0113] Among them, w i Determined by the proportion of gradient L2 norm.

[0114] Noise ratio calculation: ;

[0115] Δf = 2C / N represents the sensitivity, where C is the clipping threshold (1.0-5.0) and N is the batch size. This is a privacy relaxation parameter, typically set to 10. ﹣5 .

[0116] Finally, noise is added to the parameters. Encryption is implemented, and the final encryption is differentiated based on the importance of the parameter hierarchy before the model is uploaded. At the same time, a convergence-aware fine-tuning strategy is implemented to protect user privacy while maintaining model performance.

[0117] 3) Secure Parameter Aggregation: After receiving the encrypted model parameters, the federated learning server uses a secure aggregation algorithm to aggregate them and obtain an update of the global model. This process ensures that the server cannot obtain any individual user's raw data or unencrypted model parameters.

[0118] Explanation of the principle: Differential privacy protects individual privacy by adding noise to data or model parameters, making it difficult for attackers to distinguish whether an individual's data is in the dataset. Local differential privacy performs privacy protection operations on the client side, further enhancing privacy protection.

[0119] 2.3 Prototype Incremental Learning Based on Contrastive Learning (Solving the Model Training and Update Problem):

[0120] Technical Implementation: The prototype incremental learning technique based on contrastive learning aims to address the issues of model training efficiency, update speed, and continuous learning capability in federated learning environments. Its core technical solution can be summarized as follows: within an asynchronous federated learning framework, a contrastive learning mechanism is introduced to accelerate the prototype adaptation process, and a prototype incremental learning strategy is employed to support the continuous evolution of the model.

[0121] Implementation steps:

[0122] 1) Asynchronous model update: After the vehicle-mounted device completes model training locally, it asynchronously uploads the model parameters to the server without waiting for all devices to complete training.

[0123] 2) Prototype Incremental Learning: This invention achieves continuous learning ability through diversified memory enhancement and anti-forgetting mechanisms. Specifically, an experience replay buffer B with a capacity of K=200 is constructed, and representative samples are selected for storage according to the principle of uniform distribution of intra-class distance to ensure the coverage and integrity of historical knowledge.

[0124] During model updates, a dynamic fusion strategy is employed to organize training batches. The fusion ratio β = max(0.1, 0.7 - 0.005t) is gradually adjusted with the training progress t to balance the acquisition of new and old knowledge. Simultaneously, previous model parameters θ are saved. old With prototype set P old Constructing knowledge of distillation losses at temperature T=2.0: L dist =KL(softmax(F old (x;θ old ) / T), softmax(F(x;θ) / T));

[0125] Transferring knowledge from historical models to the current model; introducing L2 regularization constraints to prevent excessive parameter deviation:

[0126] L reg =λ·||θ-θ old ||²;

[0127] λ=0.01;

[0128] Among them, L reg This is the regularization loss; is the regularization coefficient, used to control the strength of the regularization term, and is set to 0.01 in this invention.

[0129] All anti-forgetting components are integrated into the total loss using dynamic weights:

[0130] L=λ1·L contrast +λ2·L cls +λ3·L dist +λ4·L reg ;

[0131] Among them, λ1=0.5 (contrastive learning loss weight), λ2=1.0 (classification loss weight), λ3=min(0.8, 0.1+0.005t) (knowledge distillation loss weight) increases with the training progress to ensure that key historical knowledge is retained during the continuous updating of the model and effectively alleviate the problem of catastrophic forgetting, and λ4=0.01 (regularization loss weight).

[0132] An adaptive new category discovery and rapid adaptation mechanism continuously monitors the distance relationship between sample features and the nearest prototype during system operation. When the minimum distance d is reached... min =min g ||F(x;θ)-p g ||Exceeds the statistical distribution threshold μ d +2σ d (μ) d and σ d When the mean and standard deviation of the historical minimum distance are respectively, the sample is marked as a potential candidate for a new category.

[0133] Subsequently, the density-based spatial clustering algorithm DBSCAN (Density-Based Spatial Clustering of Applications with Noise) (parameters ε=0.2·mean intra-cluster radius, MinPts=20) was used to verify the consistency and clustering of potential new category samples. When the clustering results met the stability condition (consistent sample collection cycles for 3 consecutive times), the discovery of new categories was confirmed.

[0134] After confirmation, the system automatically expands the prototype space P'=P∪{ p new}, and initialize the prototype vector using the new category sample set S_new:

[0135] p new =1 / |S new |·Σ x ∈S new F(x;θ);

[0136] To accelerate the learning of new categories, a priority learning strategy is implemented. During the sampling phase, the probability of new category samples γ is increased to 0.3, and a linear decay mechanism with 5000 iterations is designed to gradually reduce γ to 0.1, thereby achieving rapid acquisition and stable integration of new category knowledge. At the same time, the update rate of new category prototypes is dynamically adjusted to ensure that the newly added driving behavior patterns are effectively integrated while maintaining the stability of the feature space, thus improving the open-world adaptability of the system.

[0137] Explanation of the principle: The prototype incremental learning strategy allows the model to learn new knowledge incrementally while retaining memory of old knowledge. The global prototype library stores the essence of historical knowledge. By learning from the prototype library, the client can avoid catastrophic forgetting and gradually accumulate and integrate data knowledge from different times and scenarios, enabling the model to continuously adapt to changes in the driving environment and maintain long-term effectiveness.

[0138] In summary, the key technical points of this invention are as follows:

[0139] (1) Overcoming heterogeneity constraints;

[0140] Achieving efficient collaboration across heterogeneous devices through a prototype-based federated ensemble learning framework:

[0141] Decoupling of model architecture: The client only needs to ensure that the feature space dimension is consistent and can choose the model structure that is adapted to local resources (such as using MobileNet for low-end devices and ResNet for high-end devices). Knowledge sharing is achieved through prototype alignment rather than parameter aggregation, eliminating strong dependencies on model isomorphism.

[0142] Non-IID data adaptation: The category prototype is used to represent the global data distribution characteristics. The client aligns the local prototype with the global prototype through comparative learning, which alleviates the distribution offset problem.

[0143] (2) Achieving a balance between privacy and performance;

[0144] By using a lightweight prototype adapter and local dynamic differential privacy technology, model performance is maintained while protecting privacy.

[0145] Data localization processing: Raw data is processed only on the terminal device. Category prototypes are extracted through the prototype adapter, and dynamic differential privacy is applied to their parameters (adaptively adjusting the noise intensity according to the training rounds) to avoid transmitting raw features or gradients, thus blocking the privacy leakage path from the source.

[0146] Low-overhead encryption: Lightweight encryption (sparse noise injection) is applied only to adapter parameters. On the JetsonXavier NX automotive platform, the additional computational latency introduced by privacy protection is less than 5ms, and the loss of model accuracy is controlled within 3%.

[0147] (3) Supports real-time dynamic personalized updates;

[0148] Efficient and personalized model iteration is achieved through prototype incremental learning and comparative adaptation mechanisms.

[0149] Incremental learning without centralized storage: The client distinguishes the new category prototype from the historical prototype by comparing the loss function and uploads it to the server for incremental aggregation. The model can be dynamically expanded without caching historical data, which can adapt to the needs of new dangerous behavior detection (such as fatigue recognition under mask wearing).

[0150] Personalized prototype optimization: The server generates a multimodal global prototype cluster based on driving scenario tags (such as nighttime, rainy days) and user profiles (such as wearing glasses, wearing masks) provided by the client. The terminal device selects and matches the prototypes as needed for local optimization to improve the model detection accuracy (such as distracted driving scenarios at night).

Claims

1. A method for detecting dangerous driving behavior based on heterogeneous federated ensemble learning, characterized in that, Includes the following steps: Step 1: Client-side local prototype learning: Each in-vehicle device extracts features of driver behavior data through a pre-trained model, and uses a lightweight prototype adapter module to learn the category prototype of local data from the features; the prototype adapter includes multiple convolutional layers, a parameterless dynamic frequency domain attention module, and a fully connected layer, and its output is a low-dimensional category prototype vector. Step 2: Privacy-preserving transmission: Apply local dynamic differential privacy encryption to the prototype adapter parameters, dynamically adjust the privacy budget, calculate the noise ratio and add Gaussian noise to generate encrypted adapter parameters; Step 3: Server-side global prototype aggregation: The federated server receives the encrypted parameters uploaded by each client, aligns the feature spaces of heterogeneous clients using Wasserstein distance, and aggregates the aligned prototype feature spaces using a contrastive learning strategy to generate a global prototype adapter. Step 4: Client Model Update: Each client downloads the global prototype adapter, performs comparative loss optimization based on local data, updates the local prototype adapter model, and completes the real-time detection of dangerous driving behavior; Step 5: Prototype Incremental Learning: The server maintains a global prototype library. Through a dynamic hybrid training strategy and knowledge distillation loss, it incrementally learns to add new driving behavior categories and updates the global prototype library. Step 6: The client performs similarity matching between the real-time collected driving behavior features and the global prototype, and outputs the dangerous driving behavior detection results; The prototype incremental learning strategy in step 5 includes: Step 5.1: Construct experience replay buffer B, and select representative samples for storage based on the principle of uniform distribution of intra-class distance; Step 5.2: During the model update process, a dynamic fusion strategy is used to organize training batches. The fusion ratio is gradually adjusted as the training progresses (t) to balance the acquisition of new and old knowledge. Simultaneously, previous model parameters θ are saved. old With prototype set P old Constructing knowledge of temperature T regulation and distillation loss L dist : L dist =KL(softmax(F old (x;θ old ) / T), softmax(F(x;θ) / T)); Among them, F old F(·) represents the feature extraction result of the old model on the input sample x; F(·) represents the feature extraction result of the current model on the input sample x; x is the input sample; θ represents the parameters of the model currently being trained; softmax is the activation function; KL represents the KL divergence; Step 5.3: Transfer historical model knowledge to the current model and introduce L2 regularization constraints: L reg =λ·||θ-θ old ||²; Among them, L reg λ is the regularization loss; λ is the regularization coefficient. Step 5.4: Integrate all anti-forgetting components into a total loss L using dynamic weights: L=λ1·L contrast +λ2·L cls +λ3·L dist +λ4·L reg ; Among them, L contras The contrastive learning loss is denoted by λ1, where λ1 is the weight of the contrastive learning loss; L cls For classification loss, λ2 is the classification loss weight; λ3 is the knowledge distillation loss weight; λ4 is the regularization loss weight. Step 5.5: During system operation, continuously monitor the distance relationship between sample features and the nearest prototype. When the minimum distance d... min =min g ||F(x;θ)-p g ||Exceeds the statistical distribution threshold μ d +2σ d When this happens, the sample is labeled as a potential candidate for a new category; where μ d and σ d p represents the mean and standard deviation of the historical minimum distance, respectively. g Represents the prototype vector of the g-th category; Step 5.6: Use the density clustering algorithm DBSCAN to verify the consistency and clustering of potential new category samples. When the clustering results meet the stability condition, the discovery of a new category is confirmed. Step 5.7: After confirmation, the system automatically expands the prototype space P'=P∪{p new }, where P is the existing prototype space, p new The prototype vector of the newly discovered category; and using the new category sample set S new Initialize prototype vector: p new =1 / |S new |·S x ∈S new F(x;θ); At the same time, the update rate of new category prototypes is dynamically adjusted.

2. The dangerous driving behavior detection method based on heterogeneous federated ensemble learning according to claim 1, characterized in that, The prototype adapter in step 1 includes a three-level convolutional structure, specifically implemented as follows: First convolutional layer: The number of input channels matches the output dimension of the pre-trained model, with 512 output channels, 3×3 convolutional kernels, stride 1, and padding 1; Second convolutional layer: 512 input channels, 256 output channels, 3×3 kernel, stride 1, padding 1; Third convolutional layer: 256 input channels, 128 output channels, 3×3 kernel, stride 1, padding 1; Each convolutional layer is followed by a batch normalization layer, a dynamic spectral attention module, and a ReLU activation function, and finally outputs a 64-dimensional category prototype after global average pooling and a fully connected layer.

3. The dangerous driving behavior detection method based on heterogeneous federated ensemble learning according to claim 1, characterized in that, The calculation of attention weights for the dynamic frequency domain attention module includes the following steps: Step a: Perform Gaussian filtering and Laplace filtering on the input feature map to extract low-frequency and high-frequency components; Step b: Calculate the spectral energy ratio to reflect the smoothness and detail of the features; Step c: Calculate spatial attention based on mean and variance: ； In the formula, For spatial attention, This represents the value of the feature map at channel c and spatial location (i, j); The mean for each channel; Let be the variance for each channel; λ be the regularization coefficient; and: ； ； In the formula, and These represent the height and width of the feature map, respectively. Step d: Integrate spectral information to generate final attention weights: ； In the formula, Attention weights; β Contribution factor for controlling the spectral ratio; The mean of the spectral ratio; Let be the spectral energy ratio, and: ； In the formula, This represents the total energy in the low-frequency region of the image spectrum. This represents the total energy of the high-frequency region in the image spectrum. To prevent constants from being divided by zero.

4. The dangerous driving behavior detection method based on heterogeneous federated ensemble learning according to claim 1, characterized in that, The local dynamic differential privacy encryption in step 2 specifically includes: Step 2.1: Set a basic privacy budget ε base Privacy relaxation parameter δ0, parameter importance factor and the gradient clipping threshold C; Step 2.2: Dynamically calculate the privacy budget in each iteration. : ； Where α is the adjustment amplitude factor. The exponent is a non-linear adjustment exponent, where k is the current training iteration number and K is the total number of iterations. Step 2.3: Perform gradient calculation and apply gradient clipping, then apply gradient clipping to different parameter groups based on weights. Calculate a personalized privacy budget : ； Wherein, weight w i Determined by the proportion of the gradient L2 norm; Step 2.4: Calculate the noise ratio : ； Where Δf=2C / N is the sensitivity, C is the gradient clipping threshold, and N is the batch size; For privacy relaxation parameters; Step 2.5: Add noise to the parameters Encryption is implemented, and a differential encryption is performed based on the hierarchical importance of parameters before model upload, while a convergence-aware fine-tuning strategy is implemented; the encryption formula is expressed as: ； in, For encrypted parameters, These are the prototype adapter parameters; N(·) represents a mean of 0 and a variance of σ. 2 Gaussian noise; I is the identity matrix.

5. The dangerous driving behavior detection method based on heterogeneous federated ensemble learning according to claim 1, characterized in that, Step 3 specifically includes: Step 3.1: Server-side prototype feature alignment First, the prototype vector extracted from each client is represented as a discrete probability distribution; Secondly, the Wasserstein distance is defined to measure the difference between the mapped prototype vector and the global prototype vector; Then, the Wasserstein distance is calculated using the Sinkhorn iterative algorithm, and the bivariate is iteratively updated until convergence. Next, a feature mapping function is learned by minimizing the Wasserstein distance; Finally, the feature mapping function is iteratively optimized until the Wasserstein distance converges. Step 3.2: Global Prototype Aggregation After feature alignment is completed, the server further aggregates the aligned prototype vectors uploaded by all clients and uses the InfoNCE loss function in contrastive learning to enhance the consistency among similar prototype vectors.

6. The dangerous driving behavior detection method based on heterogeneous federated ensemble learning according to claim 5, characterized in that, Step 3.2 specifically includes: Step 3.2.1: Treat prototype vectors belonging to the same category in different clients as positive sample pairs, and prototype vectors of different categories as negative samples; in each round of training, the server samples prototype vectors of several categories from multiple clients to form a prototype batch containing multiple categories and multiple samples. Step 3.2.2: For each prototype vector, select another prototype vector of the same category as its positive sample, and the remaining prototype vectors as negative samples. Calculate its similarity with all other prototype vectors, and train using the following InfoNCE loss function: ； Among them, z i Indicates the anchor point prototype. For the positive samples of the anchor prototype, τ is the temperature coefficient, Nt is the total number of prototypes in the batch, and S(•,•) represents the similarity function; For anchor point prototype z i The loss function; z j This is the prototype in the batch; Step 3.2.3: The server updates the parameters of the prototype mapping network through the Adam optimizer, so that the learned unified feature space not only achieves cross-client alignment, but also retains clear inter-class distinguishability.

Images