Single-round federated learning multi-modal model training method and device based on hybrid aggregation

Abstract

The invention discloses a single-round federated learning multi-modal model training method and device based on hybrid aggregation, and the method comprises the steps: carrying out a local training stage, training a local model on a client, uploading the updating information of the local model and the probability distribution outputted by the final training to a server, and carrying out the local training stage; then, the server obtains the weight corresponding to the local model of each client through the shallow neural network, then performs weighted aggregation to obtain a global model, takes the global model as a teacher model, inputs data features corresponding to probability distribution output by the client through final training into the teacher model to obtain probability distribution output by the teacher model, and finally obtains the probability distribution output by the teacher model; and then calculating distillation loss, and updating parameters of a local model of the client. According to the method, the model precision is improved in a single-round learning scene, the communication cost is reduced, local data can be fully utilized in a multi-client environment, and the accuracy and robustness of a global model can be effectively improved.

Description

Technical Field

[0001] This application belongs to the field of machine learning technology, and in particular relates to a method and apparatus for training multimodal models in a single round of federated learning based on hybrid aggregation. Background Technology

[0002] Federated Learning (FL), a distributed learning method, allows clients to train models locally and send only model parameters or updates to a central server, avoiding direct sharing of raw data. Federated Learning not only enables collaborative training while ensuring data privacy but also significantly reduces data transfer costs. Through model aggregation, a globally shared model is generated, allowing clients to collaboratively improve model performance without exposing their local data.

[0003] However, federated learning also faces some challenges, especially in dealing with data heterogeneity. Data heterogeneity refers to the uneven distribution of training data across different clients, leading to significant differences in the local models trained by each client. When the computing power, data volume, and distribution of clients vary greatly, traditional federated learning methods (such as FedAvg) may be affected, failing to effectively aggregate the models from various clients and thus reducing the performance of the global model.

[0004] While federated learning effectively avoids data privacy breaches, its efficiency in practical applications remains a concern. Federated learning typically requires multiple rounds of communication, with the client and server exchanging model updates or parameters in each round. This process consumes significant communication bandwidth and computational resources, especially in scenarios with a large number of devices, where communication costs become a system bottleneck. To address this issue, one-shot federated learning (OFL) emerged, significantly reducing communication overhead by limiting all model updates to a single round of communication. However, a major drawback of one-shot learning is the decrease in accuracy, particularly in environments with high data heterogeneity, where single-shot methods often struggle to achieve accuracy comparable to traditional federated learning (multi-round communication).

[0005] To improve the performance of single-round federated learning, many studies have introduced knowledge distillation (KD) techniques. Knowledge distillation is a model compression method that transfers knowledge from a complex model (the teacher model) to a smaller model (the student model), thereby reducing computational resource consumption while maintaining high accuracy. In federated learning, knowledge distillation helps the client train a smaller student model locally and utilize the teacher model on the server for knowledge transfer, mitigating the accuracy degradation caused by data heterogeneity. In this way, the client can accelerate the learning process of its local model under the guidance of the teacher model, while ensuring data privacy, thus improving the performance of federated learning.

[0006] While some single-round federated learning methods based on knowledge distillation (such as DENSE and Co-Boosting) have achieved improvements in communication costs and model accuracy, several issues remain. For example, these methods often rely on complex generative models (such as GANs or VAEs) to synthesize data or models. However, these generative models are prone to generating misleading data or inconsistent knowledge when data heterogeneity is high, leading to unsatisfactory model accuracy. Furthermore, existing knowledge distillation methods largely depend on public datasets to help generate synthetic data, which may be infeasible or unsuitable in certain scenarios. Summary of the Invention

[0007] The purpose of this application is to propose a method and apparatus for training multimodal models in single-round federated learning based on hybrid aggregation, which can improve the accuracy of single-round federated learning while ensuring data privacy and reducing communication costs.

[0008] To achieve the above objectives, the technical solution of this application is as follows: A single-round federated learning method for training multimodal models based on hybrid aggregation includes: The client collects and preprocesses data, extracts features from the preprocessed data, and generates training samples. The local model is trained on the client side, and the local model update information and the probability distribution of the final training output are uploaded to the server. After receiving all the client's local model update information, the server obtains the weights of each client's local model through a shallow neural network, and then performs weighted aggregation to obtain the global model. The global model is used as the teacher model. The data features corresponding to the probability distribution of the client's final training output are input into the teacher model to obtain the probability distribution output by the teacher model. Then, the distillation loss is calculated and the parameters of the client's local model are updated. The updated parameters of the client-side local model are returned to the client, completing the training of the client-side local model.

[0009] Furthermore, the local model update information is the local model parameters or gradient.

[0010] Furthermore, the shallow neural network employs a multilayer perceptron.

[0011] Furthermore, the distillation loss function is as follows: in, Indicates distillation loss, It is the number of data feature samples. Denotes KL divergence, The output probability distribution of the teacher model. This represents the output probability distribution of the student model.

[0012] Furthermore, the weighted aggregation yields the global model, as shown in the following formula: in, Represents global model parameters. Indicates the first k The weights corresponding to each client's local model. Indicates the first k Each client-side local model parameter.

[0013] This application also proposes a single-round federated learning multimodal model training device based on hybrid aggregation, including a processor and a memory storing a number of computer instructions, which, when executed by the processor, implement the steps of the above method.

[0014] This application proposes a single-round federated learning multimodal model training method and apparatus based on hybrid aggregation. By optimizing the aggregation strategy and knowledge distillation process, it can effectively address the problem of data heterogeneity and achieve efficient model training in resource-constrained environments. It improves model accuracy and reduces communication costs in single-round learning scenarios. It not only fully utilizes local data in multi-client environments but also effectively improves the accuracy and robustness of the global model. Attached Figure Description

[0015] Figure 1 This is a flowchart of the single-round federated learning multimodal model training method based on hybrid aggregation proposed in this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0017] This application proposes a single-round federated learning multimodal model training method based on hybrid aggregation, employing a two-stage processing flow: first, a local training phase, followed by a server-side collaborative lightweight aggregation model. This aggregation method combines knowledge distillation and ensemble techniques, effectively addressing the data heterogeneity problem through this innovative hybrid approach, ensuring the stability of model performance. This application is applicable to large-scale distributed application scenarios and can effectively improve model accuracy while ensuring privacy protection.

[0018] One embodiment of this application, such as Figure 1 As shown, a single-round federated learning multimodal model training method based on hybrid aggregation is proposed, including: Step S1: The client collects data and preprocesses it. Features are extracted from the preprocessed data to generate training samples.

[0019] In this embodiment, image data is used as input. The client device acquires image data of a specific scene through a camera or other image acquisition device. The goal of data acquisition is to obtain image information related to the target task. The data source can come from different clients, which may be located in different geographical locations and have different shooting environments. In the static image acquisition stage, the client device captures static images of the specific scene using a high-definition camera. The image resolution should meet the requirements, typically set to 256×256 pixels or higher, to ensure the preservation of image details. For tasks that require capturing time-series information, the client can use video or dynamic images to capture changes in dynamic scenes. The data from each frame of the image will be used to extract feature information such as motion patterns, which is particularly suitable for behavior recognition or dynamic environment monitoring tasks. The speech signal is filtered by a Butterworth bandpass filter (50-8000Hz), and the image data is uniformly adjusted to 256×256 resolution using bilinear interpolation. The frequency domain enhancement process is as follows: the image is subjected to a fast Fourier transform to obtain the amplitude spectrum A and the phase spectrum P; a random mask matrix M∈[0.8,1.2] is generated, and the perturbation amplitude spectrum A'=A⊙M is calculated; the image is reconstructed by inverse Fourier transform: x'=IFFT(A',P).

[0020] Data preprocessing: To ensure data quality during model training and to make the data comparable across different clients, the acquired image data undergoes preprocessing. The specific steps are as follows: Image denoising: First, remove noise caused by the shooting device, environment, or other factors. For image noise, classic denoising algorithms such as median filtering and Gaussian blur can be used to ensure image sharpness.

[0021] Image cropping and scaling: Cropping the image to a fixed size and then scaling it. To extract the most representative area, the cropping operation should be tailored to the specific scene, selecting key areas for processing.

[0022] Color standardization: To avoid inconsistencies in image color distribution across different devices, all images will undergo color standardization. Specific methods include RGB value normalization or conversion to grayscale, and adjusting image brightness and contrast to ensure that data uploaded by all clients has similar visual characteristics.

[0023] Data augmentation: To improve the robustness of the model, we employ data augmentation techniques to augment the image data. Data augmentation includes rotation, mirroring, translation, scaling, and other methods to ensure the diversity of training data and prevent overfitting.

[0024] Feature extraction: In the data preprocessing stage, in addition to cleaning and standardizing the images, feature extraction is performed using convolutional neural networks (CNNs). High-dimensional features of the images are extracted using pre-trained CNN models (such as VGG16, ResNet, etc.) and used as model input.

[0025] Through these preprocessing steps, the raw image data will be transformed into a uniform format and high-quality dataset, ensuring that the training data in federated learning can fully represent the task requirements.

[0026] Step S2: Train the local model on the client and upload the local model update information and the probability distribution of the final training output to the server.

[0027] Each client trains its local model based on its local data. During training, the optimization objective is to minimize the loss function. Stochastic gradient descent (SGD) is typically used as the optimization method to optimize the model's parameters. Each client independently completes local model training, and after training, the updated information of the local model (such as parameters or gradients) is uploaded to the server.

[0028] Cross-entropy loss is commonly used for classification tasks, and its formula is as follows: Where yi is the true label and pi is the model's prediction for the i-th class.

[0029] The loss function used in training can also include other types of loss, which will not be elaborated here.

[0030] After training locally, each client uploads its model parameters or gradients to the server. To reduce communication costs, only one communication is required, and all locally uploaded model update information from each client is received by the server. In the single-round federated learning method of this application, "single-round" means that the client only performs the update information upload operation once during the entire training process, rather than uploading after each training batch. Specifically, after completing multiple batches of training locally, the client uploads the final local model update information to the server. This single-upload method is to reduce communication costs and avoid frequent communication interactions.

[0031] It's important to note that the information uploaded by the client can be model parameters or gradients. When uploading gradients, the server can use this gradient information to update the global model. After receiving all the gradient information uploaded by the clients, the server can perform a weighted average or other aggregation strategies on these gradients to obtain the global gradient. The server then uses the aggregated global gradient to update the parameters of the global model.

[0032] This approach allows the server to optimize the global model using gradient information without directly receiving model parameters, thereby reducing communication overhead to some extent while maintaining the effectiveness of model updates.

[0033] In addition, the client will also send the probability distribution of the last batch of training and the local model output to the server along with the update information.

[0034] Step S3: After receiving all client local model update information, the server obtains the weights of each client local model through a shallow neural network, and then performs weighted aggregation to obtain the global model.

[0035] After receiving update information from all local models uploaded by clients, the server performs model aggregation. Traditional aggregation methods such as weighted averaging or voting may perform poorly in cases of data heterogeneity. Therefore, this application introduces a hybrid aggregation strategy. In this strategy, the server uses a shallow neural network as an aggregator, inputting the client's local model update information into the neural network to calculate the weight of each model, and then performing weighted aggregation based on these weights.

[0036] In one specific embodiment, the shallow neural network employs a multilayer perceptron (MLP). After receiving the update information from the client-side local model, the output is the weight corresponding to each client-side local model.

[0037] Specifically, the server receives K Model parameters for each client Shallow neural networks are f Then the weight of each client It can be calculated in the following ways: ; in, It is a shallow neural network for the first k Output of local model parameters for each client.

[0038] Final global model parameters Aggregation by weighting: .

[0039] Step S4: Use the global model as the teacher model, input the data features corresponding to the probability distribution of the client's last training output into the teacher model, obtain the probability distribution output by the teacher model, then calculate the distillation loss and update the parameters of the client's local model.

[0040] This step performs knowledge distillation, using the global model as the teacher model, with the following parameters: Each client-side local model acts as a student model, and its parameters are updated by calculating the distillation loss between itself and the teacher model. By optimizing the distillation loss, the client-side local model is tuned to more closely resemble the teacher model. In this way, the student model can inherit the performance and generalization ability of the teacher model as much as possible while maintaining low computational cost.

[0041] Specifically, the corresponding data features can be obtained by inverse mapping using the probability distribution output by the client's final training and the parameters of the client's local model. These data features are then input into the teacher model to obtain the probability distribution output by the teacher model. Inverse mapping is a relatively mature technique in this field and will not be elaborated upon here.

[0042] In a specific embodiment, the designed distillation loss function as follows: in, It is the number of data feature samples. It is the Kullback-Leibler divergence, which measures the difference between the teacher and student models. The goal is to make the student model as close as possible to the teacher model. The output probability distribution of the teacher model. This represents the output probability distribution of the student model.

[0043] After calculating the distillation loss, the student model for each client is updated using an optimization algorithm (such as SGD or Adam) to minimize the distillation loss. .

[0044] The parameter update formula for the student model can be expressed as: in It's the learning rate. It is the gradient of the distillation loss with respect to the parameters of the student model. To update the parameters of the student model before, These are the parameters for the updated student model.

[0045] Step S5: Return the updated parameters of the client-side local model to the client to complete the training of the client-side local model.

[0046] The server will use the student model parameters optimized through knowledge distillation. The results are returned to each client, and the client's local model uses the student model parameters optimized by knowledge distillation. The parameters are used to complete the training of the local model.

[0047] In this technical solution, the server calculates the weights of each client's local model using a shallow neural network and then aggregates the model parameters through weighted summation to obtain a global model. The server uses this global model as the teacher model to perform knowledge distillation optimization on each client's local model, updating the local model parameters and returning the updated model to the client. Through these two steps, the server not only completes model aggregation but also further improves the performance of the client's model through knowledge distillation, while ensuring data privacy and communication efficiency.

[0048] In the client-side local model training phase of this application, each client independently trains its model based on local data. During this phase, the client uses its own data and computing resources to train the model using common optimization algorithms such as stochastic gradient descent (SGD). The client only updates the model locally and does not engage in any communication. The goal of this phase is to ensure that each client's model is fully trained based on local data, maximizing the performance of the local data. Because there is no communication with the server, all computations are performed locally on the client, significantly reducing communication bandwidth consumption. The client only needs to transmit the trained model parameters or gradients, avoiding large amounts of data transfer during each training iteration.

[0049] In the subsequent model aggregation and global update phase, the client completes local training and sends the locally updated model (e.g., model parameters or gradients) to the server. Since each client only uploads the updated model parameters or gradients, not the original data, data privacy is guaranteed. After receiving the model updates from each client, the server uses an aggregation method (such as weighted averaging or voting mechanisms) to combine these updates and generate a global model. This aggregation method typically adjusts the weights based on each client's contribution (e.g., data volume, training performance) to ensure the global model fully reflects the data characteristics of each client. The server returns the aggregated global model to each client. Updating the global model is usually a simplified process; the client only needs to obtain the updated global model parameters without needing to retransmit large amounts of training data.

[0050] In another embodiment, this application also provides a single-round federated learning multimodal model training device based on hybrid aggregation, including a processor and a memory storing a plurality of computer instructions, which, when executed by the processor, implement the steps of the above method.

[0051] Specific limitations regarding the training device for a single-round federated learning multimodal model based on hybrid aggregation can be found in the limitations of the training method for a single-round federated learning multimodal model based on hybrid aggregation mentioned above, and will not be repeated here. The aforementioned training device for a single-round federated learning multimodal model based on hybrid aggregation can be implemented entirely or partially through software, hardware, or a combination thereof. It can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations described above.

[0052] The memory and processor are electrically connected directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory stores a computer program that can run on the processor, which implements the single-round federated learning multimodal model training method based on hybrid aggregation in this embodiment of the invention by running the computer program stored in the memory.

[0053] The memory may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory stores the program, and the processor executes the program upon receiving an execution instruction.

[0054] The processor may be an integrated circuit chip with data processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0055] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A single-round federated learning multimodal model training method based on hybrid aggregation, characterized in that, The single-round federated learning multimodal model training method based on hybrid aggregation includes: The client collects and preprocesses data, extracts features from the preprocessed data, and generates training samples. The local model is trained on the client side, and the local model update information and the probability distribution of the final training output are uploaded to the server. After receiving all the client's local model update information, the server obtains the weights of each client's local model through a shallow neural network, and then performs weighted aggregation to obtain the global model. The global model is used as the teacher model. The data features corresponding to the probability distribution of the client's final training output are input into the teacher model to obtain the probability distribution output by the teacher model. Then, the distillation loss is calculated and the parameters of the client's local model are updated. The updated parameters of the client-side local model are returned to the client, completing the training of the client-side local model.

2. The single-round federated learning multimodal model training method based on hybrid aggregation according to claim 1, characterized in that, The local model update information is the local model parameters or gradients.

3. The single-round federated learning multimodal model training method based on hybrid aggregation according to claim 1, characterized in that, The shallow neural network employs a multilayer perceptron.

4. The single-round federated learning multimodal model training method based on hybrid aggregation according to claim 1, characterized in that, The distillation loss function is as follows: in, Indicates distillation loss, It is the number of data feature samples. Denotes KL divergence, The output probability distribution of the teacher model. This represents the output probability distribution of the student model.

5. The single-round federated learning multimodal model training method based on hybrid aggregation according to claim 1, characterized in that, The weighted aggregation yields the global model, as shown in the following formula: in, Represents global model parameters. Indicates the first k The weights corresponding to each client's local model. Indicates the first k Each client-side local model parameter.

6. A training device for a single-round federated learning multimodal model based on hybrid aggregation, comprising a processor and a memory storing a plurality of computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 5.

Images