Attention mechanism aggregation optimization system based on federated learning

Abstract

The present invention relates to an attention mechanism aggregation optimization system based on federated learning, and belongs to the field of computer science. In order to solve the problem of data heterogeneity and imbalance that is prevalent in federated learning, two key difficulties need to be solved: (1) maintaining the distribution structure of data subsets and important sample information; (2) solving the overfitting of data subset boundaries. The present invention proposes an aggregation optimization algorithm AMAOP based on the attention mechanism. In order to effectively solve the problem of data heterogeneity, AMAOP designs a hybrid domain attention module HDAM and a density-based grouping clustering module DBSCAN respectively for the two difficulties. DBSCAN uses a multidimensional algorithm to process the multidimensional weights of HDAM to complete the grouping operation, solving the boundary overfitting problem during the training process. The present invention can effectively reduce the impact of data imbalance on the federated learning network.

Description

Technical Field

[0001] The present invention belongs to the field of computers and relates to an attention mechanism aggregation optimization system based on federated learning. Background Art

[0002] In the communications sector, big data technology collects, analyzes, and processes massive amounts of user data, playing an increasingly important role in customer demand analysis and user behavior prediction. However, communications "big data" is formed by the aggregation of data from numerous individual users, which inevitably leads to localization and discretization of individual user data. Using only local, discrete data for training and learning can easily lead to overfitting results. Integrating discrete data across the network to form "big data" training and learning inevitably raises privacy concerns. To protect their privacy, users on the network are often reluctant to share their data with others. This creates data silos and privacy bottlenecks for the further application of big data technology in the communications sector.

[0003] In 2016, Google scientist Brendan McMahan proposed a new machine learning paradigm: federated learning (FL). Compared to traditional centralized learning, FL promotes a decentralized approach. Private data can only be communicated locally on each client, without inter-client data sharing. It is essentially a distributed training approach that enables individual users to collaboratively train machine learning by sharing model parameters, such as gradient parameters, without sharing datasets. This approach effectively addresses the privacy issues currently associated with centralized training.

[0004] Research on federated learning is still in its infancy, focusing on distributed data processing solutions. In 2016, Brendan McMahan et al. first proposed an iterative federated averaging algorithm (FedAvg). This algorithm uses weighted averaging of training data using stochastic gradient descent to train and update model parameters locally on the client, then centrally transmits the shared model to the server for aggregated updates. In 2020, H. Wang et al. proposed the Federated Matched Averaging Algorithm (FedMA). The core of the algorithm is that the client's network may contain similar neurons, which can be clustered together in a non-parametric manner. The average of all neurons in the same cluster is used to generate a global neuron. This matched averaging grouping of similar subjects for training can effectively reduce communication costs during training and improve overall model accuracy. However, FedAvg does not account for the varying feature distributions of client data, resulting in suboptimal solutions. FedMA's matched averaging approach exacerbates data imbalance during communication. In 2021, S. EK et al. proposed the Federated Distance Algorithm (FedDist). FedDist draws on the grouping idea in FedMA. It performs similarity grouping by calculating the Euclidean distance between similar units in the client. It can normalize and preprocess the unbalanced data in the training environment, reduce the overall communication cost, and improve the model accuracy.

[0005] Federated learning's distributed training approach results in highly inconsistent data generation and collection between devices. Clients and servers don't directly communicate specific data, but rather model parameters such as gradients. This often results in a large amount of heterogeneous data in the network, complicating the training process.

[0006] In the field of data science, two representative methods are mainly used to solve the problem of data imbalance: (1) Resampling methods: sampling methods and hybrid strategies, etc. (2) Multi-classifier ensemble methods: ensemble learning, cost-sensitive learning, decision threshold adjustment, etc. Existing solutions have two major difficulties in solving the problem of data imbalance: (1) How to maintain the data distribution structure and important sample information. (2) How to solve the problem of boundary overfitting and thus improve the generalization ability of the model. Therefore, whether these existing solutions are applicable to FL networks needs further verification. Summary of the Invention

[0007] In view of this, the purpose of the present invention is to provide an attention mechanism aggregation optimization system based on federated learning.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] The attention mechanism aggregation optimization system based on federated learning consists of a central server module and a client module;

[0010] The central server module includes three parts: LW-CNN, HDAM, and DBSCAN;

[0011] The client module and the central server module constitute the AMAOP module;

[0012] In AMAOP, first, the data model parameters of k users participating in the training are input to the client module. The client module updates the model by using the client update function Client update(k,W t ) Perform local training on the model to generate a shared model; the client then uploads the trained model to the optimization function ServerOptimization on the central server The shared model is trained in a unified aggregation mode at the central server; the parameters of the shared model are iteratively updated through the LW-CNN, HDAM, and DBSCAN modules on the central server; the updated model parameters are sent back to the client as the initial parameters for the next round of federated learning training.

[0013] In LW-CNN: First, a one-dimensional convolutional layer is used to perform a convolution operation on the input one-dimensional data. The input is 196 convolution kernels and the output is 16 convolution kernels. The activation function used in the one-dimensional convolutional layer is the relu function. Then, a one-dimensional maximum pooling layer is used, where the pooling window size pool_size is 4. Then, a flattening layer is used to convert the multidimensional input into one dimension, realizing the transition from the convolutional layer to the fully connected layer, and the flattening layer does not affect the size of the experimental batch. Finally, two fully connected layers and dropout layers are used, where the first fully connected layer uses relu as the activation function and the second fully connected layer uses softmax as the activation function. The fully connected layer plays a role in feature extraction in the entire network and can map the input feature representation to the sample label space. Two fully connected layers are used, one as the input layer and the other as the output layer, where the input layer inputs 1024 neurons. Adding a dropout layer between the two fully connected layers can prevent the network from overfitting and improve the generalization ability of the model.

[0014] HDAM includes a depth-wise separable convolution module and an attention mechanism module; the depth-wise separable convolution module consists of downsampling and depth-wise separable convolution, and the depth-wise separable convolution includes point-by-point convolution and depth-wise convolution; the attention mechanism module consists of a channel attention module, a spatial attention module, and an upsampling module; the input module first undergoes a downsampling operation and then extracts features through depth-wise separable convolution; downsampling operation is a method commonly used in deep neural networks to increase the receptive field and reduce the amount of computation; the depth-wise separable convolution is divided into point-by-point convolution and depth-wise convolution, the depth-wise convolution is used to process spatial information in the length and width directions, and the point-by-point convolution is used to process information in the channel direction;

[0015] The channel attention module obtains the weights of different channels in the overall model and enhances the expression between specific features. First, the maximum pooling and average pooling operations are used to extract the weights of the input model parameter features W, and then the weight values are sent to the shared MLP multi-layer perception to obtain two different descriptors: W max and W avg ; Add these two eigenvalues and input them into the sigmoid activation function to generate the channel attention feature W c ; Finally, the initial model parameter feature W is combined with the obtained channel attention feature W c Multiply the features to obtain the model parameter feature W' output by the channel attention module; it can be expressed as follows:

[0016] W max =MLP(MaxPool(W)) (1)

[0017] W avg =MLP(AvgPool(W)) (2)

[0018] W c =W max +W avg (3)

[0019] W′=σ(W c )*W (4)

[0020] Where W max It is represented as the feature after the maximum pooling and MLP, W avg represents the feature after MLP after average pooling, Wc represents the feature obtained by weighting the two output features, W represents the model parameter feature initially input, and W' represents the model parameter feature finally output after passing through the channel attention module;

[0021] The spatial attention module is used to obtain multi-dimensional spatial position information, highlight the position of key features of the model, and enhance the expressiveness of model features. In the spatial attention module, the model parameter feature W' after training in the channel attention module is first input, and the weight information is extracted by maximum pooling and average pooling to obtain two new features. The two feature values are concatenated using a 3*3 depthwise separable convolution, and the spatial attention feature is obtained using a sigmoid activation function. Finally, the initial model parameter feature W' is multiplied by the obtained spatial attention feature to obtain the model parameter feature output by the spatial attention module. The formula is as follows:

[0022] W′ max =MaxPool(W) (5)

[0023] W′ avg =AvgPool(W′) (6)

[0024] W s =f[W max , W avg ] (7)

[0025] W″=σ(W s )*W (8)

[0026] where W′ max Represented as the model parameter feature after maximum pooling, W′ avg It is expressed as the model parameter feature after average pooling, f[W′ max ,W′ avg ] represents the concatenation of these two features by deep convolution operation, W s It represents the model parameter characteristics after the activation function, and W' represents the model parameter characteristics output after passing through the spatial attention module.

[0027] The beneficial effect of the present invention is that the method of grouping and aggregating data models with different weights based on similarity can improve the data imbalance problem in federated learning and improve the overall federated training accuracy.

[0028] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0030] Figure 1 This is the complete iterative structure diagram of AMAOP;

[0031] Figure 2 This is the structural diagram of LW-CNN;

[0032] Figure 3 It is the overall architecture of HDAM;

[0033] Figure 4 This is the overall schematic diagram of the channel attention module;

[0034] Figure 5 This is the overall schematic diagram of the spatial attention module;

[0035] Figure 6 The model accuracy of the FedDist algorithm and the AMAOP algorithm; (a) is the model accuracy of FedDist; (b) is the model accuracy of AMAOP;

[0036] Figure 7 The model losses of the FedDist algorithm and the AMAOP algorithm; (a) is the model loss of FedDist; (b) is the model loss of AMAOP;

[0037] Figure 8 The convergence speeds of the FedDist and AMAOP algorithms for achieving optimal accuracy; (a) is the convergence speed of FedDist; (b) is the convergence speed of AMAOP. DETAILED DESCRIPTION

[0038] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0039] Among them, the accompanying drawings are only for illustrative purposes and represent only schematic diagrams rather than actual pictures, and should not be understood as limiting the present invention. In order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, enlarged or reduced, and do not represent the dimensions of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the accompanying drawings.

[0040] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "back", etc. indicating directions or positional relationships, they are based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0041] Figure 1 The complete iterative structure of the proposed AMAOP algorithm. AMAOP is mainly composed of a central server module and a client module. The central server module mainly includes three parts: LW-CNN, HDAM, and DBSCAN. The client module and the central server module constitute the AMAOP module. The general process of AMAOP is as follows: First, the data model parameters of k users participating in the training are input to the client module. The client module uses the client update function Client update(k,W t ) Perform local training on the model to generate a shared model. The client then uploads the trained model to the optimization function ServerOptimization on the central server. The shared model undergoes iterative parameter updates via the LW-CNN, HDAM, and DBSCAN modules on the central server. The updated model parameters are then sent back to the client as the initial parameters for the next round of federated learning training.

[0042] Figure 2The following is a block diagram of the LW-CNN architecture. First, a one-dimensional convolutional layer performs a convolution operation on the one-dimensional input data, with 196 convolution kernels as input and 16 convolution kernels as output. The activation function used in the one-dimensional convolutional layer is the ReLU function. This layer then passes through a one-dimensional max pooling layer with a pooling window size of 4. A flattening layer then converts the multidimensional input into one dimension, transitioning from the convolutional layer to the fully connected layer. This flattening does not affect the batch size of the experiment. Finally, two fully connected layers and a dropout layer are used. The first fully connected layer uses the ReLU activation function, and the second uses the softmax activation function. The fully connected layers perform feature extraction in the entire network, mapping the input feature representation to the sample label space. Two fully connected layers are used: one as the input layer and one as the output layer, with 1024 neurons in the input layer. Adding a dropout layer between the two fully connected layers prevents overfitting and improves the model's generalization.

[0043] Figure 3 The HDAM architecture proposed in the present invention mainly includes a depth-separable convolution module and an attention mechanism module. The depth-separable convolution module consists of downsampling and depth-separable convolution, and the depth-separable convolution includes point-by-point convolution and depth-wise convolution. The attention mechanism module consists of a channel attention module, a spatial attention module, and an upsampling module. The input module first undergoes a downsampling operation and then extracts features through depth-separable convolution. The downsampling operation is a method commonly used in deep neural networks to increase the receptive field and reduce the amount of calculation. The depth-separable convolution is divided into point-by-point convolution and depth-wise convolution. The depth-wise convolution is used to process spatial information in the length and width directions, and the point-by-point convolution is used to process information in the channel direction.

[0044] Figure 4 It is the channel attention submodule of HDAM. The channel attention module can obtain the weights of different channels in the overall model and enhance the expression between specific features. First, the input model parameter feature W is extracted using the maximum pooling and average pooling operations, and then the weight value is sent to the shared MLP multi-layer perception to obtain two different descriptors: W max and W avg The two eigenvalues are added together and input into the sigmoid activation function to generate the channel attention feature W c Finally, the initial model parameter feature W is combined with the obtained channel attention feature W c Multiply the features to get the model parameter feature W' output by the channel attention module. It can be expressed as follows:

[0045] W max =MLP(MaxPool(W)) (1)

[0046] W avg =MLP(AvgPool(W)) (2)

[0047] W c =W max +W avg (3)

[0048] W′=σ(W c )*W (4)

[0049] Where W max It is represented as the feature after the maximum pooling and MLP, W avg represents the features after MLP after average pooling, Wc represents the features obtained by weighting the two output features, W represents the model parameter features initially input, and W' represents the model parameter features finally output after passing through the channel attention module.

[0050] Figure 5 It is the spatial attention submodule of HDAM. The spatial attention module is mainly used to obtain multi-dimensional spatial position information, highlight the position of the key features of the model, and enhance the expressive ability of the model features. In the spatial attention module, the model parameter feature W' after training the channel attention module is first input, and the weight information is extracted by maximum pooling and average pooling to obtain two new features. The two feature values are spliced using a 3*3 depth-separable convolution, and the spatial attention feature is obtained using the sigmoid activation function. Finally, the initial model parameter feature W' is multiplied by the obtained spatial attention feature to obtain the model parameter feature output by the spatial attention module. It can be expressed as follows:

[0051] W′ max =MaxPool(W′) (5)

[0052] W′ avg =AvgPool(W) (6)

[0053] W s =f[W max , W avg ] (7)

[0054] W″=σ(W s )*W (8)

[0055] where W′ max Represented as the model parameter feature after maximum pooling, W′ avg It is expressed as the model parameter feature after average pooling, f[W′ max , W′ avg ] represents the concatenation of these two features by deep convolution operation, Ws It represents the model parameter characteristics after the activation function, and W' represents the model parameter characteristics output after passing through the spatial attention module.

[0056] Figure 6 Figure 1 shows a comparison based on model accuracy. (a) FedDist model accuracy; (b) AMAOP model accuracy. The model accuracy of both algorithms approaches optimal convergence with increasing training rounds, reaching optimal accuracy. The proposed AMAOP algorithm achieves optimal model accuracy at a faster rate than the FedDist algorithm, and ultimately achieves better optimal accuracy than the FedDist algorithm.

[0057] Figure 7 Figure 1 shows a comparison based on model loss. (a) shows the model loss of FedDist; (b) shows the model loss of AMAOP. The model loss of both algorithms converges with the number of training rounds, reaching the minimum model loss. The proposed AMAOP achieves the minimum model loss faster than the FedDist algorithm.

[0058] The Euclidean distance obtained from the training of the client and server is used as the convergence speed for the best accuracy. Figure 8 The comparison charts show that the client training convergence speed is proportional to the model accuracy. (a) shows the convergence speed of FedDist; (b) shows the convergence speed of AMAOP. Compared to FedDist, the proposed AMAOP can achieve optimal convergence in a shorter communication round.

[0059] This article uses F-Score and FLOPs to measure model accuracy and computational complexity. Table 1 shows the comparison of the best client accuracy and time complexity.

[0060] Table 1 Best accuracy and computational complexity of the client

[0061]

[0062] The FedDist client, used as a reference, achieved an optimal accuracy of 96.31% in the 97th communication round, a 0.19% performance improvement over Fedavg. The AMAOP client achieved an optimal accuracy of 97.06% in the 99th communication round, a 0.75% improvement over FedDist. The proposed AMAOP model has a FLOP count of 17.73M, while the computational complexity of the reference baseline algorithms, FedDist and FedAvg, is 17.61M and 17.54M, respectively. In summary, AMAOP effectively improves model performance at the expense of some time complexity.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. An attention mechanism aggregation optimization system based on federated learning, characterized by: It consists of a central server module and a client module; The central server module includes three parts: LW-CNN, HDAM, and DBSCAN; The client module and the central server module constitute the AMAOP module; In AMAOP: First, the data model parameters of k users participating in the training are input to the client module. The client module updates the model by using the client update function Clientpdate(k, W t ) Perform local training on the model to generate a shared model; the client then uploads the trained model to the optimization function ServerOptimization on the central server The shared model is trained in a unified aggregation mode at the central server; the parameters of the shared model are iteratively updated through the LW-CNN, HDAM, and DBSCAN modules on the central server; the updated model parameters are sent back to the client as the initial parameters for the next round of federated learning training. In LW-CNN: First, a one-dimensional convolutional layer is used to perform a convolution operation on the input one-dimensional data. The input is 196 convolution kernels and the output is 16 convolution kernels. The activation function used in the one-dimensional convolutional layer is the relu function. Then, a one-dimensional maximum pooling layer is used, where the pooling window size pool_size is 4. Then, a flattening layer is used to convert the multidimensional input into one dimension, realizing the transition from the convolutional layer to the fully connected layer, and the flattening layer does not affect the size of the experimental batch. Finally, two fully connected layers and dropout layers are used, where the first fully connected layer uses relu as the activation function and the second fully connected layer uses softmax as the activation function. The fully connected layer plays a role in feature extraction in the entire network and can map the input feature representation to the sample label space. Two fully connected layers are used, one as the input layer and the other as the output layer, where the input layer inputs 1024 neurons. Adding a dropout layer between the two fully connected layers can prevent the network from overfitting and improve the generalization ability of the model. HDAM includes a depth-wise separable convolution module and an attention mechanism module; The depthwise separable convolution module consists of downsampling and depthwise separable convolution. The depthwise separable convolution includes pointwise convolution and depthwise convolution. The attention mechanism module consists of channel attention module, spatial attention module, and upsampling. The input module first undergoes a downsampling operation and then extracts features through depthwise separable convolution. Downsampling is a method commonly used in deep neural networks to increase the receptive field and reduce the amount of computation. The depthwise separable convolution is divided into pointwise convolution and depthwise convolution. Depthwise convolution is used to process spatial information in the length and width directions, while pointwise convolution is used to process information in the channel direction. The channel attention module obtains the weights of different channels in the overall model and enhances the expression between specific features. First, the maximum pooling and average pooling operations are used to extract the weights of the input model parameter features W, and then the weight values are sent to the shared MLP multi-layer perception to obtain two different descriptors: W max and W avg ; Add these two eigenvalues and input them into the sigmoid activation function to generate the channel attention feature W c ; Finally, the initial model parameter feature W is combined with the obtained channel attention feature W c Multiply the features to obtain the model parameter feature W' output by the channel attention module; it can be expressed as follows: IN max =MLP(MaxPool(W)) (1) W avg =MLP(AvgPool(W)) (2) IN c =In max +W avg (3) W′=σ(W c )*W (4) Where W max It is represented as the feature after the maximum pooling and MLP, W avg represents the features after average pooling and MLP, W c It represents the feature obtained by weighting the two output features, W represents the model parameter feature initially input, and W' represents the model parameter feature finally output after passing through the channel attention module; The spatial attention module is used to obtain multi-dimensional spatial position information, highlight the position of key features of the model, and enhance the expressiveness of model features. In the spatial attention module, the model parameter feature W' after training in the channel attention module is first input, and the weight information is extracted by maximum pooling and average pooling to obtain two new features. The two feature values are concatenated using a 3*3 depthwise separable convolution, and the spatial attention feature is obtained using a sigmoid activation function. Finally, the initial model parameter feature W' is multiplied by the obtained spatial attention feature to obtain the model parameter feature output by the spatial attention module. The formula is as follows: IN' max =MaxPool(W′) (5) W′ avg =AvgPool(W′) (6) W s =f[W′ max ,W′ avg ] (7) W″=σ(W s )*W′ (8) where W′ max Represented as the model parameter feature after maximum pooling, W′ avg It is expressed as the model parameter feature after average pooling, f[W′ max , W′ avg ] represents the concatenation of these two features by deep convolution operation, W s It represents the model parameter characteristics after the activation function, and W' represents the model parameter characteristics output after passing through the spatial attention module.

Images