Audio modality data heterogeneous federated learning method, system, device and medium

Abstract

The application discloses an audio modal data heterogeneous federated learning method and application, and the method comprises the following steps: selecting multiple clients, sending global model parameters and global knowledge to the selected clients, and enabling the selected clients to perform local model training; receiving and aggregating the local model parameters and local knowledge uploaded by the selected clients; updating the global parameters and global knowledge based on the aggregated local model parameters and local knowledge and in combination with historical global knowledge; reselecting multiple clients, sending the updated global parameters and global knowledge to the reselected multiple clients for next round of training until the global model meets the set condition. Compared with the prior art, the audio modal data heterogeneous federated learning method provided by the application has higher precision of the optimized model parameters, and the more optimal local model enables the precision of the global model to be further improved, thereby overcoming the problem of stability decline caused by client data heterogeneity.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence deep learning, and particularly relates to an audio modal data heterogeneous federated learning method, system, device and readable storage medium. BACKGROUND

[0002] Federated learning is a new machine learning paradigm, which is essentially a distributed machine learning technology or machine learning framework. It does not require users to upload local data sets to the server, but uploads local model parameters to the server for aggregation, which not only protects the privacy of users, but also enables distributed training of models. By designing encrypted parameter transmission in the training process instead of the original remote data transmission, the security and privacy of the data of each party are ensured, the requirements of existing laws and regulations for data security are met, and the problem of users being in a data silo is solved.

[0003] Although federated learning avoids uploading local data sets, due to differences in devices, environments, data quantity and quality, etc. in different regions, not all user data is of high quality and homogeneously distributed, i.e. not independent and identically distributed. Due to the highly unbalanced distribution of client data, the convergence speed of the aggregated global model is slow and the performance is degraded, so that the local models trained have large differences and the global model aggregated has unstable performance.

[0004] Therefore, in view of the above technical problems, it is necessary to provide a federated learning method that can maintain stable performance of a global model in a data heterogeneous scenario.

[0005] The information disclosed in this BACKGROUND section is only intended to increase an understanding of the general background of the application and is not intended to be a recognition or any form of suggestion that this information forms prior art that is already known to those of ordinary skill in the art. SUMMARY

[0006] The present application aims to provide an audio modal data heterogeneous federated learning method, system, device and readable storage medium, which can improve the performance of a global model in a learning scenario with heterogeneous client data and alleviate the problem of unstable performance of a global model caused by data heterogeneity.

[0007] To achieve the above-mentioned purpose, the technical solution provided by an embodiment of the present application is as follows:

[0008] In a first aspect, the present application provides an audio modal data heterogeneous federated learning method, which comprises:

[0009] receiving global model parameters and global knowledge issued by a server, and performing local model training based on the received global model parameters and global knowledge;

[0010] obtain local knowledge corresponding to the local data set based on the trained local model, the local data set comprising local audio data and labels corresponding to the local audio data;

[0011] upload the trained local model parameters and the local knowledge to the server, so that the server updates a global model and global knowledge;

[0012] receive the global model parameters and the global knowledge issued by the server, and start the next round of training until the global model meets a set condition.

[0013] In one or more embodiments of the present application, the obtaining of the local knowledge corresponding to the local data set based on the trained local model and the uploading of the trained local knowledge to the server comprise:

[0014] convert the local audio data into a mel-spectrogram, input the mel-spectrogram into a corresponding local model, and obtain and record the output of an intermediate layer as time-frequency features corresponding to the local audio data;

[0015] input the time-frequency features corresponding to the local audio data into a linear layer of the local model, and obtain the output of the linear layer as logits corresponding to the local audio data;

[0016] obtain the mean of the time-frequency features corresponding to local audio data with the same label and the mean of the logits corresponding to local audio data with the same label;

[0017] upload the mean of the time-frequency features and the mean of the logits corresponding to each label to the server.

[0018] In one or more embodiments of the present application, the performing of the local model training based on the received global model parameters and global knowledge comprises:

[0019] updating the local model based on the global model parameters, and training the updated local model based on a preset local data set;

[0020] generating a first loss function based on the probability distribution of each received global knowledge;

[0021] optimizing the local model parameters in the training based on the first loss function.

[0022] In one or more embodiments of the present application, the first loss function is: Loss=KLDivLoss(logsoftmax(Φ(feature global ) / T), softmax(logit global / T))×T×T

[0023] wherein, T is a coefficient of aggregation; feature global is the mean of the global time-frequency features of multiple rounds; logit global is the mean of the global logit of multiple rounds; Loss is the first loss function.

[0024] In one or more embodiments of the present application, the first loss function is:

[0025] loss CE =CEloss(W(input), target)

[0026] loss KLD =KLDivLoss(logsoftmax(Φ(feature global ) / T), softmax(logit global / T)) x T x T

[0027] Loss = β x loss CE +(1-β)loss KLD

[0028] wherein, W(input) is the predicted label obtained by inputting the local audio data into the local model, target is the real label corresponding to the input local audio data; T is a coefficient of aggregation; feature global is the mean of the global time-frequency features of multiple rounds; logit global is the mean of the global logit of multiple rounds; Loss is the first loss function; and β is a number greater than 0 and less than 1.

[0029] In a second aspect, the present application provides an audio modal data heterogeneous federated learning method, which comprises:

[0030] selecting a plurality of clients, sending global model parameters and global knowledge to the selected clients, and making the selected clients perform local model training;

[0031] receiving and aggregating the local model parameters and local knowledge uploaded by the selected clients;

[0032] updating the global model parameters based on the aggregated local model parameters, and updating the new global knowledge by combining the aggregated local knowledge with the historical global knowledge;

[0033] reselecting a plurality of clients, sending the updated global parameters and global knowledge to the reselected plurality of clients for the next round of training until the global model meets the set conditions.

[0034] In one or more embodiments of the present application, the formula for aggregating the local knowledge uploaded by the selected client is:

[0035]

[0036]

[0037] wherein, is the sum of the number of local audio data with label c j in all clients participating in the training, is the number of local audio data with label c j in client b participating in the training; is the logit corresponding to the local audio data with label c j in client b participating in the training; is the feature corresponding to the local audio data with label c j in client b participating in the training; is the global logit of the class label c j obtained by aggregation in the t-th round; is the global time-frequency feature of the class label c j obtained by aggregation in the t-th round.

[0038] In one or more embodiments of the present application, the updating of the new global knowledge based on the aggregated local knowledge and the historical global knowledge comprises:

[0039] collecting the global logit in multiple rounds and the global time-frequency feature in multiple rounds;

[0040] calculating the mean of the global logit in multiple rounds as the new global logit;

[0041] calculating the mean of the global time-frequency feature in multiple rounds as the new global time-frequency feature.

[0042] In a third aspect, the present application provides a computer device, comprising a memory and a processor, which are communicatively connected with each other, and the memory stores computer instructions, and the processor executes the audio modal data heterogeneous federated learning method by executing the computer instructions.

[0043] In a fourth aspect, the present application provides a computer readable storage medium, which stores computer instructions for making a computer execute the audio modal data heterogeneous federated learning method.

[0044] Compared to existing technologies, the heterogeneous federated learning method for audio modal data provided in this invention, after local model training on the client side, not only uploads the local model parameters but also the intermediate layer outputs and linear layer outputs based on the local dataset obtained from the local model. These outputs originate from the Mel spectrogram obtained by converting local audio data. By mapping audio signals to Mel-scale frequencies, it can better simulate human auditory characteristics, reduce feature dimensionality complexity, and enhance feature representation capabilities. The above outputs, after adaptive aggregation on the server, can serve as the basis for optimizing the local model. After updating the local model based on the aggregated global model, to avoid performance degradation of the global model due to heterogeneous data from various clients participating in training, further optimization of the local model parameters is needed based on the above outputs distributed to each client after aggregation. The optimized model parameters will have higher accuracy, and the superior local model further improves the accuracy of the global model, overcoming the stability degradation problem caused by heterogeneous client data. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a schematic diagram of a heterogeneous federated learning scenario for audio modal data according to one embodiment of the present invention;

[0047] Figure 2 This is a flowchart illustrating a heterogeneous federated learning method for audio modal data according to one embodiment of the present invention.

[0048] Figure 3 This is a structural block diagram of a heterogeneous federated learning system for audio modality data according to one embodiment of the present invention;

[0049] Figure 4 This is a structural block diagram of an electronic device according to one embodiment of the present invention;

[0050] Figure 5 This is a Mel-spectral graph of the audio segment labeled "yes" in a specific embodiment of the present invention;

[0051] Figure 6 This is a Mel-spectral graph of the audio segment labeled "no" in a specific embodiment of the present invention;

[0052] Figure 7 This is a line graph showing the relationship between the change in the number of clients participating in training and the global model accuracy in a specific embodiment of the present invention;

[0053] Figure 8 This is a flowchart illustrating a heterogeneous federated learning method for audio modality data in another embodiment of the present invention.

[0054] Figure 9 This is a structural block diagram of a heterogeneous federated learning system for audio modality data in another embodiment of the present invention. Detailed Implementation

[0055] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0056] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprises" shall be understood to include the stated elements or components without excluding other elements or other components.

[0057] Existing technical solutions include federated learning methods, which involve: before formal training begins, a central server distributes an initial model to each participant, and each participant trains its model using its local dataset. Next, each participant uploads its locally trained model parameters to the central server. The central server aggregates all model parameters and then sends the aggregated global model parameters back to each participant. Based on the aggregated global model, the client retrains using its local dataset and then uploads the new model parameters. During this iterative training process, as the global model converges, training ends when the accuracy reaches a preset threshold.

[0058] However, since the core of federated learning lies in enabling participating parties to collaboratively train machine learning models without disclosing private data to other participants, each client involved in the training will influence the generation of the global model. Therefore, if a malicious attacker among the participating clients intentionally uses heterogeneous data for local model training and then uploads incorrect model parameters in an attempt to disrupt the stability of the global model; or in certain special application scenarios where each client's data distribution is highly unbalanced due to its own nature and environment, this technical solution will inevitably suffer from low model aggregation accuracy, high requirements for client data structures, and poor adaptability.

[0059] The inventor of the present application found the main shortcomings of the prior art and proposed a new technical implementation idea based on the shortcomings of the prior art: randomly selecting clients participating in training in each learning round, and uploading not only the local model parameters but also the output results of the intermediate layers / linear layers of the model corresponding to the local audio data Mel spectrogram in the training process. Subsequently, based on the aggregated global model, the local model is updated and the local parameters are optimized based on the aggregated output results. Through the optimization of the local model, the accuracy of the global model is improved.

[0060] Please refer to Figure 1 , which shows an application scenario of the audio modal data heterogeneous federated learning method provided by the present application in an embodiment. The scenario specifically includes a server and multiple clients.

[0061] It should be noted that the multiple clients are all connected with the server through communication connections, but whether the clients are connected with each other can be flexibly set according to the use scenario. The communication network derived from the above communication connection can include various connection types, including but not limited to wired connection, wireless connection or optical cable connection, etc. At the same time, the communication network can be a local area network, a metropolitan area network, a wide area network or any combination of the three.

[0062] Each client can transmit data with the server based on the above communication connection. At the same time, each client is provided with a corresponding local data set, and the local model can be trained based on the local data set and the global model. In the present application, the local model has multiple model layers. By converting the local audio data into a Mel spectrogram and then inputting it into the corresponding model, the time-frequency features corresponding to the local audio data can be obtained, and by inputting the time-frequency features of the local audio data into the linear layer of the corresponding model, the logit corresponding to the local audio data can be obtained.

[0063] The server is pre-set with an aggregation model, which can receive the local model parameters and local knowledge uploaded from the clients based on the communication connection, and aggregate the model parameters, logit and time-frequency features of each client respectively to generate global model parameters, global time-frequency features and global logit, and then return the aggregated data to the clients participating in the training.

[0064] It should be noted that the audio modal data heterogeneous federated learning method provided by the present application corresponds to the application scenario, and a plurality of user terminals are also provided. The user terminal is used to provide the user with the necessary artificial settings for implementing the present solution, such as the aggregation parameters of the server, the data set settings of the local model, the configuration of the model aggregation function, etc. At the same time, it can be understood that the user terminal is installed with a computer software program matched with the audio modal data heterogeneous federated learning method provided by the present method; the user terminal can include but is not limited to a desktop computer (PC terminal), a desktop computer, a smart phone, a handheld computer, a tablet computer, a personal digital assistant (PDA), and the like portable electronic devices or wearable electronic devices, and the present application embodiments do not limit the above content.

[0065] It should be noted that the audio modal data heterogeneous federated learning method of the present application embodiment can be applied to the audio modal data heterogeneous federated learning system of the present application embodiment. The audio modal data heterogeneous federated learning system can be configured in a terminal. The terminal can include but is not limited to a PC (Personal Computer), a PDA (tablet computer), a smart phone, a smart wearable device, and the like.

[0066] On the other hand, the server can be a stand-alone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and big data and artificial intelligence platforms, etc. Basic cloud computing services, and the present application embodiments do not limit this.

[0067] Please refer to Figure 2 Fig. 1 shows a flowchart of the audio modal data heterogeneous federated learning method in an embodiment of the present application. The audio modal data heterogeneous federated learning method specifically includes the following steps:

[0068] S201: receiving the global model parameters and global knowledge issued by the server, and performing local model training based on the received global model parameters and global knowledge;

[0069] It can be understood that in the first round of training, since the local knowledge has not been uploaded, the global knowledge received at this time is empty, and the global model parameters received are the model initialized on the server side. At the same time, in the embodiments of the present application, the local model is a multi-layer structure, specifically including an input layer, an intermediate layer and an output layer. Generally, the intermediate layer can include but is not limited to convolutional layers, pooling layers, recurrent layers, etc.; the output layer, i.e., the last layer, is usually a linear layer, which is used for the integration of global features. In particular, the embodiments of the present application do not limit the types of global models and local models. However, considering that the present application focuses on the application scenario of the audio modality. Unlike image recognition, audio is strongly associated with time, while pictures are only related to pixels, colors, etc. Therefore, in the embodiments of the present application, the preferred local model and / or global model is a time convolutional neural network model (TC-ResNet), so that the model can learn the time relationship features in the audio.

[0070] In an exemplary embodiment, performing local model training based on the received global model parameters and global knowledge includes: updating the local model based on the global model parameters, and training the updated local model based on a preset local data set; generating a first loss function based on the probability distribution of each of the received global knowledge; and optimizing the local model parameters in the training based on the first loss function.

[0071] wherein, in the tthround, the formula for updating the local model based on the global model parameters and training the updated local model based on a preset local data set is:

[0072]

[0073] wherein, w t-1 is the global model parameter of the (t-1) thround, D i is the local data set of the client i, is the model parameter generated by the client i after training based on the global model in the tthround.

[0074] It should be noted that in the embodiments of the present application, the types of knowledge are not limited. Due to the correlation of audio data with time, the intermediate layer features (time-frequency features) that can reflect the time-frequency characteristics of audio are used, which can capture the dynamic changes and frequency characteristics on the time sequence, and pay more attention to the time sequence and frequency distribution. Further, in order to ensure that another type of knowledge can be closely related to the above-mentioned time-frequency features, the time-frequency features based on the output logit of the linear layer can be selected. That is, in the embodiments of the present application, the time-frequency features corresponding to the local data set and the logit are preferably used as the local knowledge; and the aggregated global logit and global time-frequency features are used as the global knowledge.

[0075] In an exemplary embodiment, a new local model is trained based on the global model and global knowledge returned by the server. The global time-frequency features are input to the linear layer of the new local model of the client to form a new logit representation. A first loss function is formed by measuring the difference between the probability distribution of the new logit representation and the global logit probability distribution. The method of measuring the difference can include, but is not limited to, cross-entropy calculation, KL divergence calculation, etc., and the embodiments of the present application do not limit this. Based on minimizing the value of the above-mentioned first loss function, the parameters of the new local model can be optimized.

[0076] It can be understood that the sum of each probability of the directly obtained logit probability distribution and / or the probability distribution of the time-frequency features is not 1, which is not conducive to subsequent quantitative analysis. Therefore, when measuring the difference between the probability distribution of the new logit representation and the global logit probability distribution, the above-mentioned probability distribution can be arranged based on a preset activation function. The activation function can include, but is not limited to, a softmax function, a logsftmax function, etc., and the embodiments of the present application do not limit this. In particular, when the relative entropy is used to measure the above-mentioned difference, in order to avoid the final loss function value being negative, resulting in the need for additional calculation steps for data adjustment, it is preferred that one of the two inputs uses the softmax function and the other uses the logsoftmax function.

[0077] Further, in order to enable the client to learn more information about the prediction distribution, the aggregation coefficient T is used to make the distribution of the logit smoother, and the smoothed probability distribution contains more relationship information between categories. Therefore, the probability distribution logit is divided by the preset aggregation coefficient to make the probability distribution relatively smooth and carry more information. On the other hand, the aggregation coefficient is squared outside the function to ensure the stability of the loss scale. Because in the actual loss, both corresponding output quantities are processed by softmax and logsoftmax, and are scaled by the aggregation coefficient T, in order to keep the scale of the loss unchanged, the aggregation coefficient T 2 .

[0078] Specifically, the first loss function can be set as:

[0079] Loss=KLDivLoss(logsoftmax(Φ(feature global ) / T), softmax(logit global / T))×T×T

[0080] wherein T is an aggregation coefficient; feature global is the mean of the global time-frequency features of multiple rounds of multiple data categories; and logitglobal is the mean of the global logit of the multiple rounds of multiple data categories; and Loss is the first loss function. It can be understood that the data of different categories represents data with different labels.

[0081] In another embodiment, the loss function of training the local model on the local data set is considered simultaneously, the loss function caused by training is fused with the aforementioned logit and the loss function of the time-frequency feature, and a better first loss function is generated. The formula is as follows:

[0082] loss CE =CEloss(W(input), target)

[0083] loss KLD =KLDivLoss(logsoftmax(Φ(feature global ) / T), softmax(logit global / T))xT xT

[0084] Loss=βxloss CE +(1-β)loss KLD

[0085] Wherein, W(input) is the predicted label obtained by inputting the local audio data into the local model, target is the real label corresponding to the input local audio data; T is an aggregation coefficient; feature global is the mean of the global time-frequency feature of the multiple rounds of multiple data categories; logit global is the mean of the global logit of the multiple rounds of multiple data categories; Loss is the first loss function; and β is a number greater than 0 and less than 1.

[0086] S202: Obtain local knowledge corresponding to the local data set based on the trained local model;

[0087] In an exemplary embodiment of the present application, obtaining local knowledge corresponding to the local data set based on the trained local model comprises: converting the local audio data into a mel spectrum graph, and inputting the mel spectrum graph into the corresponding local model to obtain and record the output of the intermediate layer as the time-frequency feature corresponding to the local audio data; inputting the time-frequency feature corresponding to the local audio data into the linear layer of the local model to obtain the output of the linear layer as the logit corresponding to the local audio data; obtaining the mean of the time-frequency features corresponding to the local audio data with the same label and the mean of the logit corresponding to the local audio data with the same label; and uploading the mean of the time-frequency features and the mean of the logit corresponding to each label to the server.

[0088] It should be noted that the Mel spectrogram is a two-dimensional time-frequency graph, with time on the horizontal axis and Mel frequency on the vertical axis. Color or intensity represents the frequency energy at each moment. By mapping audio signals to Mel-scale frequencies, the Mel spectrogram can better simulate human auditory characteristics, reduce feature dimensionality complexity, and enhance feature representation. Optionally, before inputting the Mel spectrogram into the local model, a filtering operation can be added to the acquired Mel spectrogram. The filtering operation can remove some unnecessary noise from the Mel spectrogram, thereby emphasizing the main components and features of the corresponding local audio data, thus improving the performance of subsequent tasks. It is understood that the embodiments of the present invention do not limit the definition of noise frequencies or the filtering methods.

[0089] It should be noted that in the context of deep learning, logit is a vector, which can be viewed as the unnormalized probability output of the neural network. In this embodiment of the invention, the extracted time-frequency features and their corresponding logit reflect the relationship between the time-frequency features and their predicted distribution, allowing the local client model to learn global knowledge about the predicted distribution of the time-frequency features. In one embodiment of the invention, assume that the local dataset D of client i... i C is present i = {c1, c2, ...} labels. Let one of the labels c... j ∈C i For example, the specific operation is as follows: Assume it is tag c. j Local audio dataset in Total The label is c j The audio data. All data, in audio data For example, convert it into a Mel spectrogram. Then, by inputting it into the trained model, you will get... in Representing data Input to local model The result obtained from the output of the last linear layer. Representing data Input to local model The time-frequency characteristics obtained from the intermediate layer.

[0090] S203: Upload the trained local model parameters and local knowledge to the server, so that the server updates the global model and global knowledge;

[0091] It should be noted that, in order to reduce the uploading of redundant data and ensure the reliability of the uploaded data, and to avoid the leakage of user privacy by directly uploading all the time-frequency features and logit corresponding to the data, what should be uploaded to the server is the mean of the time-frequency features and the mean of the logit corresponding to each tag, rather than the time-frequency features and logit corresponding to the directly obtained local audio data.

[0092] Furthermore, it should be noted that in deep learning tasks within the audio modality, even with a small number of samples for certain categories, their time-frequency features remain highly valuable. This is because different clients may possess similar images, but due to differences in individual voiceprint information, clients are unlikely to have identical audio data. The core features of audio data include temporal variations, and this time-frequency information is often closely related to the semantic expression of the audio. By retaining data knowledge across all categories, we can more comprehensively cover the temporal characteristics of audio, avoiding the omission of rare but important feature information that may contribute to the model. Therefore, in this embodiment of the invention, the mean of the time-frequency features corresponding to all labels in the client and the mean of the logit need to be uploaded.

[0093] S204: Receive global model parameters and global knowledge from the server, start the next round of training, until the global model meets the set conditions.

[0094] It should be noted that the conditions for ending training can be either reaching a preset threshold for the number of training rounds or reaching a preset threshold for the accuracy of the local model / global model. This embodiment of the invention does not impose any restrictions on this.

[0095] Please refer to Figure 8 The diagram shown is a flowchart of a heterogeneous federated learning method for audio modality data according to an embodiment of the present invention. This heterogeneous federated learning method for audio modality data specifically includes the following steps:

[0096] S801: Select multiple clients, send global model parameters and global knowledge to the selected clients, and enable the selected clients to perform local model training;

[0097] It should be noted that the number of clients selected to participate in the training should be less than or equal to the total number of clients. For example... Figure 7 The figure shown is a line graph illustrating the relationship between the number of clients participating in training in each learning round and the global model accuracy in a specific embodiment of the present invention. It is evident that the more clients participating in training in each learning round, the fewer rounds are required for the global model accuracy to improve.

[0098] On the other hand, in order to improve the accuracy of the global model, the server needs to perform additional aggregation tasks, and as the number of clients participating in training increases, the GPU computing power consumed by the server also increases, causing load pressure on the system. At the same time, the local training of the clients almost all installed on the same device during the test phase will also cause a large consumption of system GPU resources. Therefore, in order to reduce the pressure on the system, the number of clients participating in learning in each round can be dynamically adjusted according to the GPU consumption and the size of the system GPU. The number of clients participating in learning in each round and the selection method of the clients participating in training are not limited in the embodiments of the present application.

[0099] S802: receiving and aggregating the local model parameters and local knowledge uploaded by the selected clients;

[0100] In an embodiment of the present application, aggregating the local model parameters and local knowledge uploaded by the selected clients includes: aggregating the uploaded local model parameters to generate global model parameters; aggregating the uploaded logit to generate global logit; and aggregating the uploaded time-frequency features to generate global time-frequency features.

[0101] Further, the formula for aggregating the uploaded logit and time-frequency features is:

[0102]

[0103]

[0104] wherein, is the sum of the number of local audio data with label c j of all clients participating in training, is the number of local audio data with label c j of the client b participating in training; is the logit corresponding to the local audio data with label c j of the client b participating in training; is the feature corresponding to the local audio data with label c j of the client b participating in training; is the global logit of class c j obtained by aggregation in the tthround; is the global time-frequency feature of class c j obtained by aggregation in the tthround.

[0105] In the embodiments of the present application, the aggregation of the logit uploaded by each client and the aggregation of the time-frequency features uploaded by each client do not use the direct average operation method. Considering the data heterogeneity of each client in the application scenario of the present application. Therefore, if there is a client with very little data corresponding to a certain label, the uploaded logit and time-frequency features corresponding thereto will have a large deviation, and direct averaging will cause the final global logit and global time-frequency features to be inaccurate. Therefore, the present application uses the data quantity ratio as a weight coefficient, which more accurately reflects the influence of the logit and the time-frequency features.

[0106] S803: updating the global model parameters based on the aggregated local model parameters; updating the new global knowledge by combining the historical global knowledge based on the aggregated local knowledge;

[0107] In an exemplary embodiment of the present application, the combination of the historical global knowledge and the updating of the global knowledge comprises: collecting the global logit of multiple rounds and the global time-frequency features of multiple rounds; calculating the mean of the global logit of multiple rounds as the new global logit; and calculating the mean of the global time-frequency features of multiple rounds as the new global time-frequency features.

[0108] In a specific embodiment of the present application, the formula for updating the global knowledge is as follows:

[0109]

[0110]

[0111] wherein, is the global logit of the class c j uploaded by the client in the Tth round; is the global time-frequency feature of the class c j uploaded by the client in the Tth round. is the mean of the global logit of the class c j in the last N rounds, is the mean of the global time-frequency feature of the class c j in the last N rounds.

[0112] Based on this, the final global knowledge is obtained as follows:

[0113]

[0114]

[0115] It should be noted that, since the data of the client is not independent and identically distributed, even if all the category knowledge uploaded by the local client is aggregated together, the global knowledge may lack a certain category, or the knowledge of a certain category may be unstable due to a small number. In order to avoid the above problems, in an embodiment of the present application, the server collects the global knowledge of the last r rounds and averages it, so as to avoid the problem of unstable knowledge of a certain category or missing knowledge of a certain category. In the next round of training, the server parameters and the averaged global knowledge are sent to the clients participating in the training

[0116] S804: Re-select a plurality of clients, and send the updated global parameters and global knowledge to the re-selected plurality of clients for the next round of training until the global model meets the set condition.

[0117] By re-selecting a plurality of clients, the updated global parameters and global knowledge are sent to the re-selected plurality of clients for the next round of training, and a new training cycle is started.

[0118] It should be noted that the condition for ending the training can be that the number of training rounds reaches a preset threshold, or that the accuracy of the local model / global model reaches a preset threshold, and the embodiments of the present application do not limit this.

[0119] In order to further illustrate the technical solutions of the present application and the beneficial effects possessed by the solutions, a specific embodiment is given as follows:

[0120] The data set used in the experiment is the Google open voice command data set Speech Commands V1 and V2. Each voice in the data set contains a single word, and the voice contains 12 class labels, which are yes, no, up, down, left, right, on, off, stop, go, silence and unknown. These voices need to extract audio features, i.e. mel-spectrogram, before being input into the model, Figure 5 and Figure 6 respectively show the mel-spectrogram of the labels yes and no. The data distribution adopts Dirichlet distribution, in which α = 0.1, the data distribution heterogeneity is serious, we think it is not independent and identically distributed, α = 5, the data distribution is uniform, very close to independent and identically distributed.

[0121] There are 100 clients and a server, but only 10 clients are selected to participate in training in each round, and a total of 50 rounds are trained. The server stores the global knowledge of the last 3 rounds and averages it, the aggregation coefficient T is 2, and the weight β of the loss function is 0.6.

[0122] As can be seen from Table 1, when a is 0.1 and 5, the global model performance can be improved compared with the prior art federated learning framework on the data sets Speech Commands V1 and V2, especially when a is 0.1, the global model accuracy is improved by more than 10%.

[0123] Table 1

[0124]

[0125] As shown in Figure 3 , based on the same inventive concept as the aforementioned heterogeneous federated learning method for audio modal data, an embodiment of the present application provides an audio modal data heterogeneous federated learning system 300, which comprises a receiving module 301, an obtaining module 302, an uploading module 303 and a circulating module 304.

[0126] Specifically, the receiving module 301 is configured to receive the global model parameters and global knowledge issued by the server, and perform local model training based on the received global model parameters and global knowledge; the obtaining module 302 is configured to obtain the local knowledge corresponding to the local data set based on the trained local model; the uploading module 303 is configured to upload the trained local model parameters and local knowledge to the server, so that the server updates the global model and global knowledge; the circulating module 304 is configured to receive the global model parameters and global knowledge issued by the server, and start the next round of training until the global model meets the set condition.

[0127] As shown in Figure 9 , based on the same inventive concept as the aforementioned another embodiment of the method for heterogeneous federated learning of audio modal data, an embodiment of the present application provides an audio modal data heterogeneous federated learning system 900, which comprises a selecting module 901, an aggregating module 902, an updating module 903 and a sending module 904.

[0128] Specifically, the selecting module 901 is configured to select a plurality of clients, and send the global model parameters and global knowledge to the selected clients, so that the selected clients perform local model training; the aggregating module 902 is configured to receive and aggregate the local model parameters and local knowledge uploaded by the selected clients; the updating module 903 is configured to update the global model parameters based on the aggregated local model parameters; aggregate the new knowledge, and update the new global knowledge in combination with the historical global knowledge; the sending module 904 is configured to reselect a plurality of clients, and send the updated global parameters and global knowledge to the reselected plurality of clients for the next round of training until the global model meets the set condition.

[0129] As shown in Figure 4As shown, the embodiment of the present application further provides an electronic device 400, which comprises at least one processor 401, a memory 402 (for example, a non-volatile memory), a storage 403 and a communication interface 404, and the at least one processor 401, the memory 402, the storage 403 and the communication interface 404 are connected together via a bus 405. The at least one processor 401 is configured to invoke at least one program instruction stored or encoded in the memory 402, so as to enable the at least one processor 401 to perform various operations and functions of the audio modal data heterogeneous federated learning method described in various embodiments of the present specification.

[0130] In the embodiments of the present specification, the electronic device 400 can include, but is not limited to, a personal computer, a server computer, a workstation, a desktop computer, a laptop computer, a notebook computer, a mobile electronic device, a smart phone, a tablet computer, a cellular phone, a personal digital assistant (PDA), a handheld device, a messaging device, a wearable electronic device, a consumer electronic device, and the like.

[0131] The embodiment of the present application further provides a computer readable medium, which carries computer execution instructions, and the computer execution instructions, when executed by a processor, can be used to implement various operations and functions of the audio modal data heterogeneous federated learning method described in various embodiments of the present specification.

[0132] The computer readable medium in the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus.

[0133] In this case, the computer readable program code can be implemented in a high level processor or object oriented programming language to communicate with a computer system. However, the computer readable program code can be implemented in assembly or machine language, if desired. The computer readable program code can be stored in any appropriate computer readable storage medium including, but not limited to, semiconductor memory, optical or magnetic disk, or the like. The computer readable program code can be downloaded from an external source via a computer readable signal. A computer readable signal includes, but is not limited to, data signals, carrier waves, and the like. The computer readable signal can be transmitted via a communication media, such as a metal wire, twisted pair, coaxial cable, optical fiber, or the like.

[0134] Those skilled in the art will appreciate that embodiments of the present application can be devised for a variety of computer readable media that can store computer readable program code. The computer readable program code can also be embodied in a computer readable signal medium that can communicate the program code to a processor for execution. Examples of a computer readable signal medium include, but are not limited to, a data transmission that can employ a carrier wave or other propagated signal to convey the program code.

[0135] The present application is described in reference to the drawings and flowchart illustrations and / or block diagrams of methods, apparatuses, systems, and computer program products according to embodiments of the application, in which it is assumed that the directions within the block diagrams of the various embodiments are taken in the direction indicated by the arrows. Figure 1 The means for carrying out any one of the functions or a plurality of the functions and / or blocks specified in any one of the flowcharts or a plurality of the flowcharts. Figure 1 The means for carrying out any one of the functions or a plurality of the functions and / or blocks specified in any one of the flowcharts or a plurality of the flowcharts.

[0136] The foregoing description of specific exemplary embodiments of the application has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the application to the precise form disclosed, and various modifications and variations are possible in light of the above teachings. It is intended that the application encompass all such modifications and variations as fall within the scope of the claims and their equivalents. It is intended that the scope of the application extend to all alternative

[0137] It will be obvious to a person skilled in the art that the application is not limited to the details of the foregoing exemplary embodiments and can be implemented in other concrete forms without departing from the spirit or essential characteristics of the application. The embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. No reference signs in the claims should be considered as limiting the scope of the claims to the identity of the reference signs therein.

[0138] Furthermore, it should be understood that although the description is made on the basis of the embodiments, not every embodiment contains only one independent technical solution, and the description of the specification is only for the sake of clarity, and those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that those skilled in the art can understand.

Claims

1. An audio modal data heterogeneous federated learning method, characterized in that, The method comprises the following steps: receiving global model parameters and global knowledge issued by a server, and performing local model training based on the received global model parameters and global knowledge; obtaining local knowledge corresponding to a local data set based on the trained local model, wherein the local data set comprises local audio data and labels corresponding to the local audio data; uploading the trained local model parameters and local knowledge to the server to enable the server to update the global model and global knowledge; receiving global model parameters and global knowledge issued by the server, and starting the next round of training until the global model meets the set conditions; The local knowledge corresponding to the local data set is obtained based on the trained local model, and the trained local knowledge is uploaded to a server, including: converting the local audio data into a mel spectrum graph, and inputting the mel spectrum graph into a corresponding local model to obtain and record the output of an intermediate layer as time-frequency features corresponding to the local audio data; inputting the time-frequency features corresponding to the local audio data into a linear layer of the local model to obtain the output of the linear layer as ; obtaining the mean of the time-frequency features corresponding to the local audio data with the same label and the mean of the corresponding to the local audio data with the same label; uploading the mean of the time-frequency features corresponding to each label and the mean of the to the server.

2. The audio modality data heterogeneous federated learning method according to claim 1, characterized in that, the step of performing local model training based on the received global model parameters and global knowledge comprises the following steps: updating the local model based on the global model parameters, and training the updated local model based on a preset local data set; generating a first loss function based on the probability distribution of each received global knowledge; optimizing the local model parameters in the training based on the first loss function.

3. The audio modality data heterogeneous federated learning method according to claim 2, characterized in that, The first loss function is: where T is a polymerization coefficient; is the mean of the global time-frequency features over multiple rounds; is the mean of the global over multiple rounds; is the first loss function.

4. The audio modality data heterogeneous federated learning method according to claim 2, characterized in that, The first loss function is: wherein, is a predicted label obtained by inputting the local audio data into the local model, is a true label corresponding to the input local audio data; T is an aggregation coefficient; is a mean value of the global time-frequency features of the multiple rounds; is a mean value of the global of the multiple rounds; is a first loss function; is a number greater than 0 and less than 1.

5. An audio modal data heterogeneous federated learning method, characterized in that, The method comprises the following steps: selecting a plurality of clients, and sending global model parameters and global knowledge to the selected clients to enable the selected clients to perform local model training; receiving and aggregating the local model parameters and local knowledge uploaded by the selected clients; updating the global model parameters based on the aggregated local model parameters; updating the global knowledge based on the aggregated local knowledge and historical global knowledge to obtain new global knowledge; reselecting a plurality of clients, and sending the updated global parameters and global knowledge to the reselected plurality of clients for the next round of training until the global model meets the set conditions; The updating of the new global knowledge based on the aggregated local knowledge and the historical global knowledge comprises: collecting the global time-frequency features of multiple rounds and the global time-frequency features of multiple rounds computing the mean of the global of the multiple rounds as a new global ; computing the mean of the global time-frequency features of the multiple rounds as a new global time-frequency feature.

6. The audio modality data heterogeneous federated learning method according to claim 5, characterized in that, the formula for aggregating the local knowledge uploaded by the selected clients is: wherein, is the sum of the number of local audio data labeled of all clients participating in the training, is the number of local audio data labeled of the client participating in the training; is the corresponding of the local audio data labeled of the client participating in the training; is the corresponding of the local audio data labeled of the client participating in the training; is the global labeled obtained by the aggregation of the t-th round; is the global time-frequency feature labeled obtained by the aggregation of the t-th round.

7. A computer device, comprising: The method comprises the following steps: a memory and a processor, which are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the audio modal data heterogeneous federated learning method of any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for enabling a computer to perform the audio modal data heterogeneous federated learning method of any one of claims 1-6.

Images