Driver distraction detection method based on multi-layer alignment federated learning

Abstract

The application relates to the technical field of intelligent driving assistance, in particular to a driver distraction behavior detection method based on multi-layer alignment federated learning. The method comprises the following steps: a federated framework is built, a client and a server are configured with a same-structure deep learning model, global model parameters, a category average representation set and hyperparameters are initialized; the server selects a client subset according to participation rates, and the global parameters and the representation set are issued. The client initializes a local model based on the global parameters and the representation set as a regularization basis, calls a private data set for training, constructs a total loss function, updates the local model parameters, calculates an updated local category average representation set and uploads the set to the server. The server updates the global model parameters and the category average representation set by weighted aggregation, repeats the training until the total communication iteration number is reached, and uses the final global model as a detection model to realize driver distraction behavior detection. The method can improve the driver distraction behavior detection accuracy.

Description

Technical Field

[0001] This application relates to the field of intelligent driving assistance technology, and in particular to a method for detecting driver distraction behavior based on multi-layer aligned federated learning. Background Technology

[0002] With the development of intelligent driving technology, driver distraction behavior detection technology has become a key core technology for improving driving safety. Deep learning-based driver distraction behavior detection methods have been widely studied and applied due to their high detection accuracy. This type of technology relies on massive amounts of driving behavior data for centralized training of models, which can learn the characteristic patterns of various driver distraction behaviors, thereby achieving accurate identification of distraction behaviors.

[0003] However, current centralized training and detection methods based on deep learning face two major technical challenges: First, driving behavior data involves driver privacy and is subject to privacy protection regulations such as GDPR, posing a risk of data privacy leakage, making it difficult to achieve unrestricted centralized aggregation of massive amounts of driving data; second, driving behavior data from different vehicles and different drivers are stored in a scattered manner, forming a serious data silo problem, making data sharing and integration difficult.

[0004] To address the aforementioned issues, federated learning has been introduced into the training of driver distraction detection models. As a distributed machine learning paradigm, federated learning supports multiple clients to jointly train a global model by communicating model parameters without sharing the original data. However, in actual driver distraction detection scenarios, federated learning faces significant challenges due to data heterogeneity. Data heterogeneity mainly includes four categories: label skew, feature skew, quality skew, and quantity skew. Among them, label skew (significant differences in the distribution of distraction behavior categories among different clients) and quantity skew (uneven sample sizes of distraction behavior data among clients) are particularly prominent. This type of data heterogeneity contradicts the assumption of "independent and identically distributed data" in federated learning algorithms, easily leading to a "client drift" phenomenon where client models deviate significantly from the optimal global solution and other client models after multiple updates. The simple parameter averaging step of standard federated learning cannot specifically counteract this drift, ultimately causing the model convergence speed to slow down, the training process to oscillate, or even diverge, severely reducing the distraction detection performance of the global model. Summary of the Invention

[0005] Therefore, it is necessary to provide a driver distraction behavior detection method based on multi-layer aligned federated learning that can improve the accuracy of driver distraction behavior detection, in order to address the above-mentioned technical problems.

[0006] A method for detecting driver distraction behavior based on multi-layer aligned federated learning, the method comprising: Step 1: Build a federated learning architecture, set up multiple clients and 1 server. The clients are the driver end. The clients and the server are configured with the same deep learning model structure. Initialize the server's global model parameters and global class average representation set, and set the hyperparameters of federated learning. Step 2: The server randomly selects a subset of clients from multiple clients based on the preset client participation rate, and distributes the global model parameters and global category average representation set for the current round to all clients in the client subset. Step 3: Each client receives the global model parameters and global class average representation set sent by the server, uses the global model parameters and global class average representation set as local model initialization parameters and global regularization basis, and calls the local private driver distraction behavior dataset to train the local model. Step 4: During local training on the client side, perform data alignment processing on the local dataset to generate fused training samples and fused labels; assign class weights to each distraction behavior category based on the number of effective samples, perform output alignment processing to obtain the output alignment loss; construct a feature alignment regularization term by combining the global class average representation set, and perform feature alignment processing; fuse the processing results of data alignment, output alignment, and feature alignment to construct the client-side local total loss function, and update the local model parameters according to the client-side local total loss function; Step 5: After each client completes local training, it calculates and updates the local class average representation set, and uploads the trained local model parameters and the local class average representation set to the server. Step 6: The server collects the local model parameters and local class average representation sets uploaded by a subset of clients, and performs weighted aggregation updates on the global model parameters and global class average representation sets respectively to obtain new global model parameters and new global class average representation sets. Step 7: Determine whether the number of communication iterations on the server has reached the preset total number. If not, return to step 2 to continue the next round of federated learning training. If it has, terminate the training and use the final global model as the driver distraction behavior detection model to realize driver distraction behavior detection.

[0007] The aforementioned driver distraction behavior detection method based on multi-layer aligned federated learning achieves privacy protection of driver distraction behavior data by building a federated learning architecture, avoiding centralized transmission and sharing of raw data. At the same time, by performing three-layer alignment processing of data, output, and features during local training on the client, it systematically alleviates the client drift problem caused by data heterogeneity, effectively improving the convergence stability and detection accuracy of the global model. It solves the privacy leakage and data silo problems of traditional centralized training, as well as the poor detection performance of conventional federated learning in heterogeneous data scenarios, and greatly improves the accuracy of driver distraction behavior detection. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating a driver distraction behavior detection method based on multi-layer aligned federated learning in one embodiment. Figure 2 This is a schematic diagram of the process of a multi-layer federated alignment (FedMA) algorithm in one embodiment; Figure 3 The figure shows the results of a systematic comparison experiment between the method of this application and nine advanced federated learning algorithms in one embodiment. Detailed Implementation

[0009] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0010] In one embodiment, such as Figure 1 As shown, a driver distraction behavior detection method based on multi-layer aligned federated learning is provided, including the following steps: Step 1: Build a federated learning architecture, set up multiple clients and 1 server. The clients are the driver end. The clients and the server are configured with the same deep learning model structure. Initialize the server's global model parameters and global class average representation set, and set the hyperparameters of federated learning.

[0011] The federated learning architecture is a distributed architecture of "multi-client-single server". Each client corresponds to a driver and stores a private dataset of driver distraction behaviors locally. The server is the central server, responsible for the aggregation and updating of the global model. Clients and servers are configured with deep learning models of the same structure to ensure that model parameters can be communicated and aggregated. The global category average representation set is the set of feature average representations of each distraction behavior category initialized by the server, serving as the global basis for client feature alignment. The hyperparameters of federated learning include learning rate, client participation rate, total number of communication iterations, number of local training rounds, feature alignment regularization hyperparameter, and output alignment hyperparameter, where the output alignment hyperparameter controls the strength of the sample overlap effect. , Between 0 and 1.

[0012] Step 2: The server randomly selects a subset of clients from multiple clients based on the preset client participation rate, and distributes the global model parameters and global category average representation set for the current round to all clients in the client subset.

[0013] The client participation rate is the proportion of clients selected by the server in each training round. It is a preset fixed value and a subset of clients is selected by random sampling to ensure the randomness and representativeness of the clients participating in the training. The server sends the global model parameters and global class average representation set of the current round to the selected clients, providing the initialization basis and global regularization basis for the local model training of the clients. This process only transmits model parameters and feature representations and does not involve any original driver distraction behavior data.

[0014] Step 3: Each client receives the global model parameters and global class average representation set sent by the server, uses the global model parameters and global class average representation set as the local model initialization parameters and global regularization basis, and calls the local private driver distraction behavior dataset to train the local model.

[0015] The client directly uses the global model parameters issued by the server as the initialization parameters of the local model, ensuring initial consistency between the local and global models. The global class average representation set is used as the basis for global regularization of feature alignment during local model training, constraining the direction of local feature extraction. The local private driver distraction behavior dataset is a preprocessed dataset. Preprocessing operations include adjusting the images in the dataset to a size of 224×224×3 using bilinear interpolation, normalizing image pixel values ​​to [0,1], and performing data augmentation operations such as random horizontal flipping, random vertical flipping, random translation, and random scaling and aspect ratio cropping. In actual federated learning training, client data can be partitioned according to driver ID. Different numbers of clients can be set for different driver distraction behavior datasets; for example, 20 clients are trained for the SFDDD dataset, 40 for the 3MDAD dataset, and 38 for the AUCDDD dataset.

[0016] Step 4: During local training on the client side, perform data alignment processing on the local dataset to generate fused training samples and fused labels; assign class weights to each distraction behavior category based on the number of effective samples, perform output alignment processing to obtain the output alignment loss; construct a feature alignment regularization term by combining the global class average representation set, and perform feature alignment processing; fuse the processing results of data alignment, output alignment, and feature alignment to construct the client-side local total loss function, and update the local model parameters according to the client-side local total loss function.

[0017] Data alignment, output alignment, and feature alignment form a three-layer collaborative alignment mechanism, mitigating the negative impact of data heterogeneity from three dimensions: data distribution, loss weights, and feature representation. Client-side local training employs a stochastic gradient descent optimizer, setting weight decay coefficients, batch processing sample size, and local training epochs as preset hyperparameters. Each epoch processes the local dataset in batches and updates the local model parameters through backpropagation based on the local total loss function. Figure 2 The diagram illustrates the process of the FedMA (FedMulti-Layer Federated Alignment) algorithm, clarifying the specific procedures for parameter updates and feature calculations during local training.

[0018] Step 5: After each client completes local training, it calculates and updates the local class average representation set, and uploads the trained local model parameters and the local class average representation set to the server.

[0019] The local category average representation set is the set of feature average representations calculated by the client based on the updated local model, after extracting features from all samples in the local private driver distraction behavior dataset and classifying them by distraction behavior category. After completing a preset number of local training rounds, the client only uploads the trained local model parameters and the local category average representation set to the server, without transmitting any original driver distraction behavior data, effectively protecting data privacy.

[0020] Step 6: The server collects the local model parameters and local class average representation sets uploaded by a subset of clients, and performs weighted aggregation updates on the global model parameters and global class average representation sets respectively to obtain new global model parameters and new global class average representation sets.

[0021] The server updates both the global model parameters and the global class average representation set using a weighted average method. The global model parameters are weighted according to the total number of local training samples on the client, while the global class average representation set is weighted according to the number of training samples for each class on each client. This weighting method allows clients with larger sample sizes to have more reasonable weights in the global aggregation, thereby improving the representativeness of the global model and the global class average representation set.

[0022] Step 7: Determine whether the number of communication iterations on the server has reached the preset total number. If not, return to step 2 to continue the next round of federated learning training. If it has, terminate the training and use the final global model as the driver distraction behavior detection model to realize driver distraction behavior detection.

[0023] The number of communication iterations for the server is the number of rounds in which the server sends global parameters to the client and receives parameters uploaded by the client to complete the aggregation update. Different total communication iterations can be set for different driver distraction behavior datasets. For example, the total number of communication iterations for the SFDDD and AUCDDD datasets is 200 rounds, while the total number of communication iterations for the 3MDAD dataset is set to 500 rounds due to its larger data volume. After each round of communication iterations, the server can use the test set to test the performance of the current global model. Once the preset total number of communication iterations is reached, training is terminated, and the final global model is the trained driver distraction behavior detection model. When the driver behavior image to be detected is input into this global model, the model will output the corresponding distraction behavior category prediction result, thus achieving the classification and detection of driver distraction behavior.

[0024] The aforementioned driver distraction behavior detection method based on multi-layer aligned federated learning first establishes a multi-client-single-server federated learning architecture. Throughout the process, only model parameters and class average representations are transmitted, without sharing raw driving data. This architectural approach avoids data privacy risks and solves the data silo-induced sharing and integration challenges. Simultaneously, the client and server are configured with the same deep learning model structure, laying the foundation for parameter aggregation and global model training. Secondly, the server randomly selects clients based on their participation rate and distributes global parameters and class average representations. The clients use these as the basis for local model initialization and global regularization, ensuring that local training is always anchored to the global direction and avoiding initial bias. The key lies in the three-layer alignment processing during local client training: data alignment smooths data distribution, and output alignment balances class loss weights. These two processes work together to alleviate the local data class imbalance problem and ensure local classification performance. Finally, a feature alignment regularization term is constructed using the global class average representation, forcing local feature embeddings to align with the global model, reducing differences in feature representations between clients, and fundamentally mitigating client drift caused by data heterogeneity. Meanwhile, the client integrates the three-layer alignment results to construct a total loss function. By minimizing this function, it achieves collaborative optimization of local tasks and global knowledge, adapting to local private data features while strengthening global consistency. Finally, the server aggregates the parameters and representation sets uploaded by the client in a weighted manner according to sample size, continuously updating the global model and the average representation of the categories. Through multiple iterations, the global model is continuously optimized, fully learning the effective features of each client, reducing the generalization gap. The final trained global model has stronger robustness and accuracy in driver distraction behavior detection, effectively solving various technical pain points of traditional methods.

[0025] In one embodiment, data alignment processing is performed on the local dataset to generate fused training samples and fused labels, including: Select two samples and their corresponding one-hot encoded labels from the same mini-batch of the local dataset, and randomly sample the fusion coefficients. By performing weighted fusion of the two samples and their corresponding labels using linear interpolation, the fused training samples and fused labels are obtained as follows: ; in, To integrate training samples, , For the two selected original samples, The fusion coefficients are randomly sampled from the Beta distribution. To integrate tags, , These are the one-hot encoded labels corresponding to the two selected original samples.

[0026] Specifically, the above proposes a data alignment strategy to further improve the robustness of the output alignment loss to data distribution. This strategy obtains new fused training samples and labels by linearly interpolating and combining two samples and their corresponding labels within the same mini-batch. This aligns the distribution characteristics of the local training data towards a smoother and more continuous state, thereby mitigating the negative impact of uneven data distribution and enhancing the model's ability to distinguish samples close to the decision boundary.

[0027] In one embodiment, class weights are assigned to each distraction behavior category based on the number of valid samples, and output alignment processing is performed to obtain the output alignment loss, including: For a containing The categories of each sample y Corresponding category weights for: ; in, It is a hyperparameter. For category y The actual number of samples; Output alignment loss is ; in, It is the original cross-entropy loss.

[0028] Based on the number of effective samples, this application designs a class balancing strategy based on output alignment. This strategy sets the weight of each sample in the loss function to the reciprocal of the number of effective samples for that class. This means that even if a class has a large number of original samples, it will still receive a high weight during training if its number of effective samples is low (indicating significant overlap between samples). Conversely, a class with a small number of samples but a number of effective samples close to the original number (indicating significant differences between samples) will have its weight reduced accordingly.

[0029] Using the reciprocal of the effective number of samples as the class weights allows for a reassessment of sample value, taking into account the redundancy within the data and avoiding the limitations of traditional quantity weighting methods. Applying the class weights to the original cross-entropy loss to obtain the output alignment loss allows the model to focus more on the classes with truly scarce information during training, reasonably balancing the contribution of different classes to the total loss and mitigating the bias caused by class imbalance.

[0030] In one embodiment, the process of aligning the feature regularization term includes: ; in, For feature alignment regularization hyperparameters, For the client k Feature alignment regularization term, For the client k Local model parameters, For the global category average representation set, For the client k The number of local samples, For the client k The l One sample, For the sample Local feature embedding, For the sample xl The global category average representation of the corresponding category label, where d is the feature embedding dimension.

[0031] Specifically, feature alignment addresses the issues of overfitting in local models and excessive parameter diversity among clients. It uses the global average class representation set distributed by the server as the regularization basis for local training, constructing a feature alignment regularization term. This regularization term benefits each client by leveraging global semantic feature information. Intuitively, it enables each client to learn task-invariant representations through explicit feature distribution alignment. Therefore, while minimizing local classification error, the diversity of local feature extractors is also normalized. This regularization term can be inspired by explicitly reducing generalization error, thereby contributing to improved test accuracy.

[0032] In one embodiment, the processing results of data alignment, output alignment, and feature alignment are fused to construct a client-local total loss function, including: To achieve the local training objective of the multi-layer federated alignment (FedMA) algorithm, a total loss function for the client is constructed by integrating three modules: data alignment, output alignment, and feature alignment. This function, while ensuring the effectiveness of local supervised learning, leverages global knowledge to guide cross-client feature and output distribution alignment, effectively mitigating local overfitting and data heterogeneity issues caused by non-independent and identically distributed data. Its overall form is as follows: ; in, For the client-side local total loss function, The output alignment supervision loss is the result of data alignment. For feature alignment regularization hyperparameters, For the client k Feature alignment regularization term.

[0033] Specifically, the data alignment strategy is combined with the output alignment loss. The training data distribution is expanded through linear sample fusion, while retaining a weight redistribution mechanism for the effective number of samples. For fused samples and fused labels, this loss is a weighted fusion of the output alignment losses of the two original samples. The supervised output alignment loss after data alignment is... ; in, The fusion coefficient is... To output alignment loss, To merge the true labels of the first two original samples, For the model to fuse training samples The output is the predicted probability vector.

[0034] Specifically, the client-side local total loss function integrates the output alignment supervision loss after data alignment and the feature alignment regularization loss, achieving joint optimization of local supervised learning and global knowledge alignment. The output alignment supervision loss after data alignment combines the data alignment strategy with the output alignment loss to correct the class imbalance in the output layer of the prediction results of the fused samples. This allows the model to learn a smoother decision boundary through sample fusion and ensures the supervision contribution of minority class samples through effective sample number weights, significantly improving the adaptability of local training to class imbalanced data. The feature alignment regularization loss, as a global knowledge regularization term, constrains the learning process of the local feature extractor and alleviates the problems of local overfitting and data heterogeneity caused by non-independent and identically distributed data.

[0035] In one embodiment, a weighted aggregation update is performed on the global model parameters and the global class average representation set, including: The global model parameters are updated by weighted aggregation. ; in, These are the aggregated global model parameters. This represents the number of clients participating in this aggregation round. For the first k Aggregate weights of parameters from each client. For the first k The client number t Parameters of the model after local training For the first k Total number of local training samples for each client.

[0036] Specifically, the server assigns aggregation weights to the weighted aggregation update of global model parameters based on the total number of local training samples on the client. The larger the sample size of the client, the higher the aggregation weight of its parameters. This method allows the global model parameters to better match the feature patterns of clients with larger data volumes, thereby improving the representativeness and generalization ability of the global model. The aggregation process only calculates the local model parameters uploaded by the client and does not involve any original data, thus ensuring the privacy protection characteristics of federated learning.

[0037] In one embodiment, a weighted aggregation update is performed on the global category average representation set, and a weighted average is performed for each distraction behavior category. ; in, For the first Wheel category y The global category average representation, This represents the number of clients participating in this aggregation round. For the first k Category of clients y The number of training samples, For the first k The client number t Round calculation category y Local feature representation.

[0038] The server updates the global category average representation set by category. For each distraction behavior category, the local category feature representation of each client is weighted and averaged according to the number of training samples of that category from each client to obtain the global category average representation for that category. This weighting method allows the global category average representation to more accurately reflect the average feature level of all clients for the corresponding category, providing a more accurate global basis for the next round of client feature alignment and further reducing the difference in feature representation between clients.

[0039] In one embodiment, the hyperparameters include the learning rate, client participation rate, total number of communication iterations, number of local training rounds, feature alignment regularization hyperparameter, and output alignment hyperparameter, where the output alignment hyperparameter controls the strength of the sample overlap effect. , Between 0 and 1.

[0040] In one embodiment, when the final global model is used for driver distraction behavior detection, the driver behavior image to be detected is input into the global model, and the model outputs the distraction behavior category prediction result corresponding to the image, thereby realizing the classification and detection of driver distraction behavior.

[0041] In specific embodiments, during actual model training and testing, model performance can be verified for different data distribution scenarios, mainly including independent and identically distributed (IID) scenarios and non-independent and identically distributed (Non-IID) scenarios. The Non-IID scenario is further subdivided into label heterogeneous scenarios and quantity heterogeneous scenarios. For example... Figure 3 The image shows the distribution of the AUCDDD dataset. This dataset is generated from data collected by two cameras. The local data volumes of Camera1 and Camera2 differ significantly, and some clients have missing classes. This is a naturally non-independent and identically distributed scenario, eliminating the need to construct a heterogeneous distribution. The specific process includes: For the IID scenario, the SFDDD dataset itself has a relatively uniform distribution, so we only need to maintain its own dataset allocation. The 3MDAD dataset, on the other hand, randomly assigns 1250 images to each of the 40 clients, while the test set is fixed at 10,000 randomly selected images. To more realistically simulate the two non-IID scenarios, we use two methods to generate different non-IID settings, with no data overlap between clients in either scenario: (1) Label Heterogeneity (LabelSkew): The offset of the dataset label distribution owned by the client. If it is the SFDDD dataset, for each driver, 5 distraction behaviors are randomly removed from their local dataset; if it is the TriMDAD dataset, 8 distraction behaviors are randomly removed from their local dataset.

[0042] (2) Quantity Skew: The amount of data possessed by each client varies significantly. For each driver, a random discard ratio is generated, and samples of each category within the client are discarded according to this ratio. It is worth noting that the AUCDDD dataset contains data from cameras at two angles; the local data volume of Camera1 and Camera2 differs significantly, and some clients have missing categories, making it a naturally non-independent and identically distributed scenario. Therefore, this application does not perform any additional scene construction, and the dataset distribution is as follows: Figure 3 .

[0043] The proposed method in this application is compared with nine state-of-the-art federated learning methods, including FedAvg (classical method), Scaffold (based on correction-update), Fedprox and FedDyn (based on regularization), MOON and FedLC (based on model splitting), FedGen and FedNTD (based on knowledge distillation), and FedCross (based on heuristic search). Specifically: FedAvg, as a pioneer in federated learning algorithms, uses averaging as the aggregation method for the client-side local learning model. Scaffold employs controlled variable techniques to estimate and correct client drift, effectively mitigating the gradient discrepancy problem caused by Non-IID data by adjusting the local update direction to be closer to the global update direction. FedProx introduces a proximal term based on FedAvg, addressing the non-independent and identically distributed problem by controlling the optimization degree of the local model. Note that FedProx contains a hyperparameter. µ Used to adjust the weights of proximal terms. This application will use [this method] in experiments. µ Starting with {0.001, 0.01, 0.1, 1}, FedDyn designed a dynamic regularization term consisting of a linear term based on first-order conditions and an Euclidean distance term, ensuring that the local optimum remains consistent with the global stationary point, exhibiting good convergence under both convex and non-convex optimization settings. MOON is a model-level contrastive federated learning method that addresses the Non-IID problem by leveraging the similarity between model representations to correct local training on a single client. This method also introduces an additional loss term and a hyperparameter. µ This is used to control its weights during local training. Similarly, following the original suggestion, this application modulates the parameters from {0.001,0.01,0.1,1}. µ FedLC addresses the label distribution skew problem by calibrating the model's logits before softmax calculation to prevent overconfidence in the local majority class. FedGen trains a lightweight generator on the server side and broadcasts it to clients, who then use the generator to synthesize data to provide inductive bias, achieving knowledge distillation without sharing the original data. FedNTD uses a global model to guide local training to solve the local bias training problem. This method only passes class probabilities as knowledge to the local model. Weighting factors β The initial search grid is {0.2, 0.4, 0.6, 0.8}. FedCross uses a heuristic search strategy to optimize the federated learning process.

[0044] Federated learning is performed by simulating a corresponding number of nodes based on the number of drivers in the dataset (SFDDD dataset training client: test client = 20:6, AUCV2DDD dataset training client: test client = 38:6, 3MDAD dataset training client: test client = 40:10). The default communication rounds for the SFDDD and AUCV2 datasets are 200 rounds. For the 3MDAD dataset, which has more data, the default communication rounds are 500 rounds. In each round, a fixed proportion of 40% (ρ=0.4) of nodes is selected to participate in model aggregation. After aggregation, the performance of the global model in that round is tested on the test set. The local epoch for each node is 5, and a stochastic gradient descent (SGD) optimizer is used with a weight decay of 1e-4. Since the number of samples stored by each client is usually small, local training uses batch processing of 32 samples. For hyperparameter tuning, this application uses grid search to find the optimal hyperparameters.

[0045] To evaluate the performance of the FedMA algorithm, a comprehensive comparative experiment was conducted on three distracted driving behavior datasets. Specifically, the experiment covered two data partitioning scenarios: (1) Independent and Identically Distributed (IID) scenario; and (2) Non-Independent and Identically Distributed (Non-IID) scenario. The Non-IID scenario was further subdivided into label skew and quantity skew settings. The proposed method was systematically compared with nine state-of-the-art federated learning algorithms.

[0046] To evaluate the performance of the FedMA algorithm, comprehensive experiments were conducted on two public datasets, SFDDD and 3MDAD. The experiments used independently identically distributed (IID) data partitioning, meaning that the data class distribution of each client was consistent with the overall distribution. The proposed method was systematically compared with nine state-of-the-art federated learning algorithms, and the experimental results are shown in Table 1. On the SFDDD dataset, FedMA demonstrated a significant advantage. In terms of accuracy (Acc), FedMA achieved 87.13%, a 6.33 percentage point improvement over the classic baseline FedAvg (80.80%). Simultaneously, FedMA achieved recall (0.8728), F1 score (0.8739), precision (Pre), and AUC (0.9819), significantly outperforming all compared methods. This indicates that in the IID scenario, FedMA effectively improves model performance through a multi-layered federated alignment mechanism. This analysis suggests that FedMA's data alignment strategy enhances the diversity of training data through linear interpolation of samples, output alignment avoids overfitting of highly redundant samples by reweighting based on the number of effective samples, and feature alignment promotes the stability of local feature learning through global class average representation constraints. The synergistic effect of these three factors enables FedMA to achieve performance gains under conditions of balanced data distribution. On the more challenging 3MDAD dataset, FedMA's advantages become even more apparent, showing significant improvements compared to suboptimal methods and FedAvg. This significant improvement stems from the complexity of the 3MDAD dataset itself—it contains acquisition differences from multiple vehicle models and multiple camera perspectives, and even with IID partitioning, local data still exhibits inherent distributional differences. FedMA, through its feature alignment regularization term, explicitly constrains the alignment of local feature embeddings towards the global class average representation, effectively reducing feature representation differences between clients, thus demonstrating stronger robustness on inherently heterogeneous data. Meanwhile, FedMA achieved the best performance in recall (0.6329), F1 score (0.6278), and Pre (0.6827), while its AUC was on par with FedNTD, indicating that its overall performance is leading. A comprehensive analysis of the results from both datasets leads to the following conclusions: Compared to baseline methods, FedMA demonstrates significant advantages in IID scenarios, with even more pronounced advantages in the more complex 3MDAD dataset. In addition to these advantages, this application further analyzes the comparison between FedMA and various baseline methods. Compared to regularization-based methods such as FedAvg and FedProx, FedMA not only constrains model parameters but also directly aligns the feature space, making the optimization objective more explicit; compared to contrastive learning-based methods such as MOON and FedLC, FedMA does not require constructing a large number of negative sample pairs; compared to knowledge distillation-based methods such as FedGen and FedNTD, FedMA does not rely on additional generative models or global model soft labels, resulting in a simpler implementation.These characteristics enable FedMA to maintain high accuracy while offering good practicality and scalability. However, some limitations are worth noting. FedMA requires maintaining a global class average representation, which may increase storage and communication overhead slightly. Nevertheless, FedMA's significant improvements in core metrics such as accuracy, recall, and F1 score demonstrate the important value of multi-layered federated alignment mechanisms in federated learning, providing an effective solution for privacy-preserving learning in vision tasks.

[0047] Table 1

[0048] To further evaluate the performance of the FedMA algorithm, comprehensive experiments were conducted on three public datasets: SFDDD, 3MDAD, and AUCDDD. The experiments employed non-independent identically distributed (Non-IID) data partitioning, further subdivided into label skew and quantity skew settings. The proposed method was systematically compared with nine state-of-the-art federated learning algorithms. The experimental results are shown in Table 2. Figure 3 As shown. Since AUCDDD itself is a dataset that mixes two heterogeneous scenarios, this application does not perform additional data partitioning.

[0049] Table 2

[0050] Analysis of Heterogeneous Label Scenarios: Under the heterogeneous label setting of the SFDDD dataset, the performance of various methods generally showed a significant decline. The accuracy of FedAvg dropped to 65.25%, a decrease of 15.58 percentage points compared to its IID scenario, highlighting the serious challenge that imbalanced label distribution poses to federated learning. In this scenario, FedMA achieved an accuracy of 74.26%. Although FedAvg showed a significant improvement, its performance was still close to that of some methods such as FedGen and FedNTD. This analysis suggests that this is because the heterogeneous label distribution leads to significant differences in class distribution among clients. While FedMA's feature alignment mechanism can constrain local features to converge towards the global average representation, the estimation of the global average representation itself is biased when some classes are missing locally, limiting the effectiveness of feature alignment. Furthermore, the weighting of the effective number of output alignment samples may excessively suppress the learning of the majority class when the class imbalance is extreme, affecting the overall convergence of the model. On the 3MDAD dataset, the impact of label heterogeneity is even more severe, with FedMA achieving an accuracy of 46.02%. This indicates that, given the combined effects of data heterogeneity and task complexity, FedMA requires stronger global information to assist in local training.

[0051] Analysis of Heterogeneous Datasets: FedMA demonstrates a significant advantage in heterogeneous datasets. On the SFDDD dataset, FedMA achieves an accuracy of 86.33%, and it also outperforms other methods across multiple metrics. This indicates that FedMA's multi-level alignment mechanism effectively addresses the challenges posed by varying sample sizes in heterogeneous datasets. FedMA achieves performance gains even with uneven client-side sample sizes. On the 3MDAD dataset, FedMA achieves an accuracy of 54.93%, approaching the performance of FedGen and FedNTD. This demonstrates that FedMA remains highly competitive even on highly complex datasets. Analysis of the AUCDDD Dataset: The AUCDDD dataset itself combines both label and dataset heterogeneity, and includes data collection differences across multiple vehicle models and drivers, making it a more challenging testing scenario. On this dataset, FedMA achieves an accuracy of 71.25%, significantly outperforming all compared methods. To visually verify the model's feature learning effect on distracted driving behavior, this application uses the Grad-CAM (Gradient-weighted Class Activation Mapping) method to visualize the model's attention distribution. The visualization results show that the model can accurately locate the core feature regions of different distracted driving behaviors: for behaviors like texting with the left or right hand, attention is focused on the area where the driver's hands interact with the phone; in the hair-styling scenario, the model focuses on the head and hand combing movements; for adjusting the radio, attention is focused on the center console and the operating hands; for talking to someone, attention covers the interaction areas of the driver's face and the passenger side; for drinking and reaching back, attention is focused on the water / drink container and the hand movements of retrieving items from the back seat, respectively; and in safe driving scenarios, the model's attention is evenly distributed in the driver's normal driving posture (hands on the steering wheel, eyes forward), with no obvious abnormal focus areas.

[0052] The visualization results above demonstrate that the model does not rely on background or irrelevant regions for classification, but effectively captures the key action features of various distracted driving behaviors, verifying the effectiveness and interpretability of the model's feature extraction. The high degree of overlap between the attention region and the core behavioral regions of human visual judgment indicates that the features learned by the model have clear physical meaning, rather than being overfitted or dependent on noisy features, providing visual evidence for the reliability of subsequent distracted driving behavior recognition.

[0053] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0054] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0055] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for detecting driver distraction behavior based on multi-layer aligned federated learning, characterized in that, The method includes: Step 1: Build a federated learning architecture, set up multiple clients and 1 server. The client is the driver end. The client and the server are configured with the same deep learning model structure. Initialize the server's global model parameters and global class average representation set, and set the hyperparameters of federated learning. Step 2: The server randomly selects a subset of clients from multiple clients based on the preset client participation rate, and distributes the global model parameters and global category average representation set for the current round to all clients in the client subset. Step 3: Each client receives the global model parameters and global class average representation set sent by the server, uses the global model parameters and global class average representation set as local model initialization parameters and global regularization basis, and calls the local private driver distraction behavior dataset to train the local model. Step 4: During local training on the client side, perform data alignment processing on the local dataset to generate fused training samples and fused labels; assign class weights to each distraction behavior category based on the number of effective samples, perform output alignment processing to obtain the output alignment loss; construct a feature alignment regularization term by combining the global class average representation set, and perform feature alignment processing; fuse the processing results of data alignment, output alignment, and feature alignment to construct the client-side local total loss function, and update the local model parameters according to the client-side local total loss function; Step 5: After each client completes local training, it calculates and updates the local class average representation set, and uploads the trained local model parameters and the local class average representation set to the server. Step 6: The server collects the local model parameters and local class average representation sets uploaded by a subset of clients, and performs weighted aggregation updates on the global model parameters and global class average representation sets respectively to obtain new global model parameters and new global class average representation sets. Step 7: Determine whether the number of communication iterations on the server has reached the preset total number. If not, return to step 2 to continue the next round of federated learning training. If it has, terminate the training and use the final global model as the driver distraction behavior detection model to realize driver distraction behavior detection.

2. The method according to claim 1, characterized in that, Perform data alignment processing on the local dataset to generate fused training samples and fused labels, including: Select two samples and their corresponding one-hot encoded labels from the same mini-batch of the local dataset, and randomly sample the fusion coefficients. By performing weighted fusion of the two samples and their corresponding labels using linear interpolation, the fused training samples and fused labels are obtained as follows: in, To integrate training samples, , For the two selected original samples, The fusion coefficients are randomly sampled from the Beta distribution. To integrate tags, , These are the one-hot encoded labels corresponding to the two selected original samples.

3. The method according to claim 1, characterized in that, Based on the number of valid samples, class weights are assigned to each distraction behavior category, and output alignment processing is performed to obtain the output alignment loss, including: For a containing The categories of each sample y Corresponding category weights for: in, It is a hyperparameter. For category y The actual number of samples; The output alignment loss is in, It is the original cross-entropy loss.

4. The method according to claim 1, characterized in that, The process of aligning the feature with the regularization term includes: in, For feature alignment regularization hyperparameters, For the client k Feature alignment regularization term, For the client k Local model parameters, For the global category average representation set, For the client k The number of local samples, For the client k The l One sample, For the sample Local feature embedding, For the sample xl The global category average representation of the corresponding category label, where d is the feature embedding dimension.

5. The method according to claim 1, characterized in that, By fusing the results of data alignment, output alignment, and feature alignment, a local total loss function is constructed on the client side, including: By integrating the processing results of data alignment, output alignment, and feature alignment, a local total loss function is constructed on the client side. in, For the client's local total loss function, The output alignment supervision loss is the result of data alignment. For feature alignment regularization hyperparameters, For the client k Feature alignment regularization term.

6. The method according to claim 5, characterized in that, The output alignment supervision loss after data alignment is: in, The fusion coefficient is... To output alignment loss, To merge the true labels of the first two original samples, For the model to fuse training samples The output is the predicted probability vector.

7. The method according to claim 1, characterized in that, The global model parameters and the global class average representation set are updated using weighted aggregation, including: The global model parameters are updated by weighted aggregation. in, These are the aggregated global model parameters. This represents the number of clients participating in this aggregation round. For the first k Aggregate weights of parameters from each client. For the first k The first client t Parameters of the model after local training For the first k Total number of local training samples for each client.

8. The method according to claim 7, characterized in that, The method further includes: The global category average representation set is updated with weighted aggregation, and a weighted average is performed for each distraction behavior category. in, For the first Wheel category y The global category average representation, This represents the number of clients participating in this aggregation round. For the first k Category of clients y The number of training samples, For the first k The first client t Round calculation category y Local feature representation.

9. The method according to claim 3, characterized in that, The hyperparameters include learning rate, client participation rate, total number of communication iterations, number of local training rounds, feature alignment regularization hyperparameter, and output alignment hyperparameter, wherein the output alignment hyperparameter is a parameter that controls the intensity of the sample overlap effect. , Between 0 and 1.

10. The method according to claim 1, characterized in that, When the final global model is used for driver distraction behavior detection, the driver behavior image to be detected is input into the global model, and the model outputs the distraction behavior category prediction result corresponding to the image, thereby realizing the classification and detection of driver distraction behavior.

Images