Personalized federal learning aggregation method based on multiple modes

Abstract

The invention discloses a personalized federated learning aggregation method based on multiple modes, which relates to the field of machine learning and comprises the following steps of: classifying a model into a personalized part and a public part according to a mode type; the client performs local training on the personalized part and the public part to obtain a personalized model and an announcement model, and the public model of the client is uploaded to the server after the local training is completed; aggregating the common models of different client models to obtain a global model, broadcasting the aggregated model back to the client, and updating the common models of the client models by using the updated global model; the personalized federated learning aggregation method based on the multiple modes aims at solving the problem of multi-mode data under non-independent identical distribution in federated learning, processing the multi-mode data through feature extraction and a fusion strategy, relieving the problem of non-independent identical distribution through a personalized scheme, and enhancing the generalization ability of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a personalized federated learning aggregation method based on multimodality. Background Art

[0002] With the development of mobile technology, edge devices generate a large amount of data, which often exists in isolated data silos. However, due to privacy and security considerations, data cannot be exchanged between different devices. This situation makes traditional machine learning methods impractical, because traditional machine learning methods concentrate data on servers for training. Therefore, the concept of federated learning has attracted much attention in recent years. Federated learning allows personalized model training and parameter updates on edge devices. By iteratively averaging local model parameters, the federated aggregation algorithm FedAVG achieves cooperative learning and efficient communication between distributed devices. However, most existing emotion recognition technologies rely on single-modal data, such as emotion analysis through images or audio only. This method often has difficulty in fully capturing the emotional state of students, because emotional expressions are usually multimodal, including facial expressions, voice intonation, body movements, etc.

[0003] Another problem with federated learning systems is the huge heterogeneity of data between clients, which leads to non-IID. Non-IID means that there are large differences in the distribution of data and there is a certain correlation between the data. This non-IID poses a fundamental challenge to federated learning systems. Non-IID causes the local data distribution to be different from the global data distribution, the average local model parameters to drift away from the global model parameters, and the data quality of different modalities is different, and the drift is also different. The difference in data distribution will lead to differences in model parameters learned from different clients, which in turn affects the convergence and accuracy of the global model.

[0004] To address these issues, we propose a multi-modal based personalized aggregation method in Federated Learning (PMMDFL), which can handle multi-modal models under non-IID. Experiments have shown that this method can adapt to the differences in multi-modal models under non-IID and improve the generalization and accuracy of the model. Summary of the invention

[0005] The purpose of the present invention is to provide a personalized federated learning aggregation method based on multimodality to solve the problems of multimodal data processing and non-independent and identically distributed in federated learning.

[0006] To achieve the above object, the present invention provides a personalized federated learning aggregation method based on multimodality, comprising the following steps:

[0007] S1. Classify the models into personalized type and public type according to modality type;

[0008] S2. The client performs local training on the personalized type and the public type to obtain a personalized type model and a public type model. After the local training is completed, the public type model of the client is uploaded to the server.

[0009] S3. The server aggregates the common type models of different client models to obtain a global model, and broadcasts the aggregated global model back to the client for public part training.

[0010] Preferably, the personalized type in S1 is trained locally to have more personalized features, and the personalized features of the public part are not obvious, and uploaded to the server.

[0011] Preferably, the specific steps of S2 are as follows:

[0012] The server randomly selects m devices from a total of n devices. In a total of T rounds of global training, the server uses the global model u of the current round t. (t) Broadcast to each selected device. For each selected device i, the personalization type uses the device's current personalization parameters l i Initialization i,0 , calculate the loss through the loss function F, and use the optimizer to perform E on the gradient of l l Local iteration update l i , F uses MSE mean square error, and sets the learning rate of the personalized type model to η l , in E l After local iterations, the updated personalized parameters Set to

[0013]

[0014]

[0015] Preferably, the public type receives the current global model u and initializes u i,0 , calculate the loss through the loss function F, and use the optimizer to perform E on the gradient of u u Update u in local iterations i , set the learning rate of the public type model to η u , set at E u After local iterations, the updated common parameters Set to

[0016]

[0017] Preferably, S3 collects all parameters uploaded by the selected devices. Afterwards, the server updates the global model by averaging these uploaded parameters:

[0018]

[0019] Therefore, the present invention adopts the above-mentioned personalized federated learning aggregation method based on multimodality, which has the following beneficial effects:

[0020] (1) The present invention processes multimodal learning tasks for federated learning, uses multimodal learning models for data fusion, and adapts to diverse federated learning tasks.

[0021] (2) The method proposed in the present invention takes into account that the local data distribution is different from the global data distribution due to non-independent and identically distributed data, and the average local model parameters drift away from the global model parameters. In addition, the data quality of different modalities is different, and the drift situation is also different. Models with obvious personalized features are trained locally, while models with fewer personalized features are uploaded to the global server for training. This method allows customers to retain their unique personalized features while integrating public models from different customers by aggregating the common parts.

[0022] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flowchart of an embodiment of a personalized federated learning aggregation method based on multimodality of the present invention;

[0024] Figure 2 To establish ten clients under the CMU-MOSEI dataset, the model parameters of different modalities are selected as the test accuracy after the personalization part;

[0025] Figure 3 To establish ten clients under the IEMOCAP dataset, the model parameters of different modalities are selected as the test accuracy after the personalization part;

[0026] Figure 4 This is the average test accuracy of ten clients during the 100th round of training on the CMU-MOSEI dataset.

[0027] Figure 5 This is the average test accuracy of ten clients during the 100th round of training on the IEMOCAP dataset. DETAILED DESCRIPTION

[0028] Unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0029] Example

[0030] See also Figure 1 , this embodiment provides a personalized federated learning aggregation method based on multimodality, including the following steps:

[0031] S1. Classify the model into personalized type and public type according to the modality type. The specific steps are as follows:

[0032] Non-IID will cause the local data distribution to be different from the global data distribution, and the average local model parameters will drift away from the global model parameters. In addition, the data quality of different modalities is different, and the drift is also different. Selecting the part of the model with more personalized features to be trained locally allows the personalized parts of different customers to be trained separately, so as to better reflect the personalized features and not be affected by other customers. Then, the public model with unclear personalized features will be uploaded to the server for aggregation to form a global model to improve performance. In this way, the final model not only has distinct personalized features, but also has more accurate public features through aggregation.

[0033] S2. The client performs local training on the personalized type and the public type to obtain a personalized type model and a public type model. After the local training is completed, the public model of the client is uploaded to the server. The specific steps are as follows:

[0034] The server randomly selects m devices from a total of n devices. In a total of T rounds of global training, the server uses the global model u of the current round t. (t) Broadcast to each selected device. For each selected device i, the personalization part uses the device's current personalization parameter l i Initialization i,0 , calculate the loss through the loss function F, and use the optimizer to perform E on the gradient of l lLocal iteration update l i , F uses MSE mean square error, and sets the learning rate of the personalized model to η l , in E l After local iterations, the updated personalized parameters Set to

[0035]

[0036] The public part receives the current global model u and initializes u i,0 , calculate the loss through the loss function F, and use the optimizer to perform E on the gradient of u u Update u in local iterations i , set the public model learning rate to η u , set at E u After local iterations, the updated common parameters Set to

[0037]

[0038]

[0039] The updated public parameters Set to And Upload to the server.

[0040] S3. The server aggregates the public models of different client models and broadcasts them to the client for public part training. The specific steps are as follows:

[0041] After collecting all parameters uploaded by the selected devices Afterwards, the server updates the global model by averaging these uploaded parameters:

[0042]

[0043] The aggregated global model u (t+1) Broadcast back to the client for public model training.

[0044] This embodiment provides a personalized federated learning aggregation method based on multimodality, and the specific application steps are as follows:

[0045] (1) The IEMOCAP and CMU-MOSEI databases were selected. The IEMOCAP dataset features dialogues between ten actors in scripts and improvisational scenes, and contains text and audio modal data. Text features are extracted using the language pre-training model BERT, and audio features are processed using the speech signal processing package Librosa. After denoising, the audio features are segmented into 283-dimensional vectors at a sampling rate of 22,050 Hz. The CMU-MOSEI dataset was developed by Carnegie Mellon University and focuses on sentiment and emotion analysis. The dataset includes 3,228 videos, 23,453 sentences, and more than 65 hours of data from 1,000 speakers. Text features are extracted using the language pre-training model BERT to obtain a 768-dimensional vector. Audio features are extracted using the speech processing algorithm COVAREP to obtain a 74-dimensional vector; video features are processed using the facial expression analysis tool Emotient FACET to also obtain a 74-dimensional vector.

[0046] (2) In the CMU-MOSEI dataset, ten clients are established and model parameters of different modalities are selected as the personalized part. Figure 2 As shown in the figure, the test results show that as long as the text modality is available locally, the average test accuracy of the client will be very high, while when the text modality is not available, the average test accuracy of the client will be very low. Therefore, we choose to keep the text modality model as the personalized part locally for separate training, and upload the models of other modalities as the public part.

[0047] In the IEMOCAP dataset, there are only text modality and audio modality data, and the same operation is performed. Figure 3 As shown, the average test results of the client show that the training results of locally retained audio modality data are the best, so the audio modality model is selected as the personalized part and the text modality model is aggregated as the common part.

[0048] (3) In local training, after receiving the global model initialization, the personalized part training is performed first, and then the public part training is performed after the training is completed.

[0049] (4) We select the traditional federated aggregation algorithm FedAVG and the personalized federated learning algorithms Adaptive Personalized Federated Learning (APFL) and Adaptive Local Aggregation for Personalized Federated Learning (FedALA) for comparison. Figure 4 and Figure 5As shown in the figure, the average test accuracy of the proposed multimodal personalized federated learning aggregation method (PMMDFL) in both sets of experiments is higher than that of the other algorithms. This method is superior to FedAVG and other personalized federated learning algorithms in terms of generalization ability and accuracy, and effectively alleviates the performance degradation caused by non-independent and identically distributed data.

[0050] Therefore, this embodiment proposes a personalized federated learning aggregation method based on multimodality to alleviate the non-independent and identically distributed problem in multimodal federated learning, and at the same time completes the multimodal federated learning task. This method takes into account that the local data distribution is different from the global data distribution due to non-independent and identically distributed data, and the average local model parameters may also be far away from the global model parameters. Models with obvious personalized features are trained locally, and models with fewer personalized features are uploaded to the global server for training. This method allows customers to retain their unique personalized features while integrating common features from different customers through aggregation of common parts. Experimental results on IEMOCAP and CMU-MOSEI datasets show that this method is superior to the federated aggregation algorithm FedAVG and other personalized federated learning algorithms in both learning ability and accuracy, effectively alleviating the performance degradation caused by non-independent and identically distributed data and enhancing the generalization ability of the model. Experimental results on multimodal datasets show that this embodiment is superior to traditional federated aggregation algorithms and other personalized federated algorithms in terms of test accuracy.

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A personalized federated learning aggregation method based on multimodality, characterized in that: The following steps are involved: S1. Classify the models into personalized type and public type according to modality type; S2. The client performs local training on the personalized type and the public type to obtain a personalized model and a public model. After the local training is completed, the public model of the client is uploaded to the server. S3. The server aggregates the common models of different client models to obtain a global model, and broadcasts the aggregated global model back to the client for public type training.

2. The multimodal personalized federated learning aggregation method according to claim 1, characterized in that: The personalized type in S1 is trained locally, and the public type model is uploaded to the server.

3. According to the multimodal personalized federated learning aggregation method of claim 1, it is characterized in that: The specific steps of S2 are as follows: The server randomly selects m devices from a total of n devices. In a total of T rounds of global training, the server uses the global model u of the current round t. (t) Broadcast to each selected device. For each selected device i, the personalization type uses the device's current personalization parameter l i Initialization i,0 , calculate the loss through the loss function F, and use the optimizer to perform E on the gradient of l l Local iteration update l i , F uses MSE mean square error, and sets the learning rate of the personalized type model to η l , in E l After local iterations, the updated personalized parameters Set to 4. The multimodal personalized federated learning aggregation method according to claim 3, characterized in that: The public type receives the current global model u and initializes u i,0 , calculate the loss through the loss function F, and use the optimizer to perform E on the gradient of u u Update u in local iterations i , set the learning rate of the public type model to η u , set at E u After local iterations, the updated common parameters Set to 5. The multimodal personalized federated learning aggregation method according to claim 1, characterized in that: S3 collects all parameters uploaded by the selected devices Afterwards, the server updates the global model by averaging all uploaded parameters: Among them, S (t) express.

Images