Heterogeneous federated learning method, system and equipment for audio modal data and medium

Abstract

The invention discloses an audio modal data heterogeneous federal learning method and application, and the method comprises the steps: selecting a plurality of clients, sending global model parameters and global knowledge to the selected clients, and enabling the selected clients to execute local model training; receiving and aggregating local model parameters and local knowledge uploaded by the selected client; updating global parameters and global knowledge based on the aggregated local model parameters and local knowledge in combination with historical global knowledge; and reselecting a plurality of clients, and sending the updated global parameters and global knowledge to the plurality of reselected clients for next round of training until the global model meets a set condition. Compared with the prior art, the audio modal data heterogeneous federated learning method provided by the invention has the advantages that the optimized model parameters have higher precision, the precision of the global model is further improved by the better local model, and the problem of stability reduction caused by client data heterogeneous is solved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence deep learning technology, and specifically relates to a heterogeneous federated learning method, system, device and readable storage medium for audio modal data. Background Art

[0002] Federated learning is a new machine learning paradigm, which is essentially a distributed machine learning technology or machine learning framework. It does not require users to upload local data sets to the server, but upload local model parameters to the server for aggregation, which not only protects the privacy of users, but also realizes distributed training models. By designing encrypted parameter transmission instead of the original remote data transmission during the training process, the security and privacy of data of all parties are guaranteed, while meeting the requirements of the laws and regulations on data security, and solving the problem of users being in data silos.

[0003] Although federated learning avoids users uploading local data sets, due to differences in equipment, environment, data quantity and quality in different regions, not all user data is of high quality and identically distributed, that is, non-independent and identically distributed. Due to the highly unbalanced distribution of client data, the aggregated global model converges slowly and performs poorly. As a result, the local models trained are very different, and the performance of the aggregated global model is unstable.

[0004] Therefore, in response to the above technical problems, it is necessary to provide a federated learning method that can maintain stable global model performance even in data heterogeneous scenarios.

[0005] The information disclosed in this background technology section is only intended to enhance the understanding of the overall background of the invention and should not be regarded as an acknowledgment or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art. Summary of the invention

[0006] The purpose of the present invention is to provide a method, system, device and readable storage medium for heterogeneous federated learning of audio modal data, which can improve the performance of the global model in a learning scenario with heterogeneous client data and alleviate the problem of unstable global model performance caused by data heterogeneity.

[0007] In order to achieve the above object, a technical solution provided by a specific embodiment of the present invention is as follows:

[0008] In a first aspect, the present invention provides a method for heterogeneous federated learning of audio modality data, which comprises:

[0009] Receiving global model parameters and global knowledge sent by the server, and performing local model training based on the received global model parameters and global knowledge;

[0010] Acquire local knowledge corresponding to a local data set based on the trained local model, where the local data set includes local audio data and labels corresponding to the local audio data;

[0011] Uploading the trained local model parameters and local knowledge to the server so that the server updates the global model and global knowledge;

[0012] Receive the global model parameters and global knowledge sent by the server and start the next round of training until the global model meets the set conditions.

[0013] In one or more embodiments of the present invention, the acquiring local knowledge corresponding to the local data set based on the trained local model and uploading the trained local knowledge to the server includes:

[0014] Convert the local audio data into a Mel-spectrogram, input the Mel-spectrogram into the corresponding local model, and obtain and record the output of the intermediate layer as the time-frequency feature corresponding to the local audio data;

[0015] Input the time-frequency features corresponding to the local audio data into the linear layer of the local model, and obtain the output of the linear layer as the logit corresponding to the local audio data;

[0016] Get the mean of the time-frequency features corresponding to the local audio data with the same label and the mean of the logit corresponding to the local audio data with the same label;

[0017] The mean of the time-frequency features and the mean of the logit corresponding to each label are uploaded to the server.

[0018] In one or more embodiments of the present invention, performing local model training based on the received global model parameters and global knowledge includes:

[0019] Updating the local model based on the global model parameters, and training the updated local model based on a preset local data set;

[0020] Generate a first loss function based on the received probability distribution of each of the global knowledge;

[0021] Optimize local model parameters in training based on the first loss function.

[0022] In one or more embodiments of the present invention, the first loss function is: Loss = KLDivLoss(logsoftmax(Φ(feature global ) / T), softmax(logit global / T))×T×T

[0023] Among them, T is the aggregation coefficient; feature global is the mean of the global time-frequency features of multiple rounds; logit global is the mean of the global logit of multiple rounds; Loss is the first loss function.

[0024] In one or more embodiments of the present invention, the first loss function is:

[0025] loss CE =CEloss(W(input),target)

[0026] loss KLD =KLDivLoss(logsoftmax(Φ(feature global ) / T), softmax(logit global / T))×T×T

[0027] Loss = β × loss CE +(1-β)loss KLD

[0028] Among them, W(input) refers to the predicted label obtained by inputting local audio data into the local model, target is the real label corresponding to the input local audio data; T is the aggregation coefficient; featureglobal is the mean of the global time-frequency features of multiple rounds; logit global is the mean of the global logit of multiple rounds; Loss is the first loss function; β is a number greater than 0 and less than 1.

[0029] In a second aspect, the present invention provides a method for heterogeneous federated learning of audio modality data, which includes:

[0030] Select multiple clients, send global model parameters and global knowledge to the selected clients, and enable the selected clients to perform local model training;

[0031] Receive and aggregate local model parameters and local knowledge uploaded by the selected client;

[0032] Based on the aggregated local model parameters, the global model parameters are updated; based on the aggregated local knowledge, combined with the historical global knowledge, new global knowledge is updated;

[0033] Reselect multiple clients and send the updated global parameters and global knowledge to the reselected multiple clients for the next round of training until the global model meets the set conditions.

[0034] In one or more embodiments of the present invention, the formula for aggregating the local knowledge uploaded by the selected client is:

[0035]

[0036]

[0037] in, is the number of clients with label c among all the clients participating in the training j The sum of the number of local audio data, The client b participating in the training has a label of c j The amount of local audio data; The label of client b participating in the training is c j The logit corresponding to the local audio data; The label of client b participating in the training is c j The feature corresponding to the local audio data; The category label obtained by the aggregation in the tth round is c j The global logit of The category label obtained by the aggregation in the tth round is c j The global time-frequency characteristics.

[0038] In one or more embodiments of the present invention, the updating of new global knowledge based on the aggregated local knowledge and combining with the historical global knowledge includes:

[0039] Collect global logit and global time-frequency features of multiple rounds;

[0040] Calculate the mean of the global logit of the multiple rounds as a new global logit;

[0041] The average of the global time-frequency features of the multiple rounds is calculated as a new global time-frequency feature.

[0042] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the heterogeneous federated learning method for audio modality data by executing the computer instructions.

[0043] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the heterogeneous federated learning method for audio modality data.

[0044] Compared with the prior art, the heterogeneous federated learning method for audio modal data provided by the present invention not only uploads the local model parameters after the client performs local model training, but also uploads the intermediate layer output results and linear layer output results of the local data set obtained based on the local model. The output result is derived from the Mel spectrum obtained by converting the local audio data. By mapping the audio signal to the Mel scale frequency, it can better simulate the auditory characteristics of humans, reduce the complexity of feature dimensions, and enhance the feature expression ability. The above output results after adaptive aggregation by the server can be used as the basis for optimizing the local model. After updating the local model based on the aggregated global model, in order to avoid the degradation of the global model performance due to the heterogeneity of the data of each client participating in the training, it is also necessary to further optimize the local model parameters based on the above output results sent to each client after aggregation. The optimized model parameters will have higher accuracy, and the better local model will further improve the accuracy of the global model, overcoming the problem of decreased stability caused by the heterogeneity of client data. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 It is a schematic diagram of a heterogeneous federated learning scenario of audio modality data in one embodiment of the present invention;

[0047] Figure 2 is a flowchart of a heterogeneous federated learning method for audio modality data in one embodiment of the present invention;

[0048] Figure 3 is a structural block diagram of a heterogeneous federated learning system for audio modality data in one embodiment of the present invention;

[0049] Figure 4 is a structural block diagram of an electronic device in one embodiment of the present invention;

[0050] Figure 5 is a mel-spectrogram of an audio segment labeled yes in a specific embodiment of the present invention;

[0051] Figure 6 is a mel-spectrogram of an audio segment labeled as no in a specific embodiment of the present invention;

[0052] Figure 7 It is a line graph showing the relationship between the change in the number of clients participating in training and the accuracy of the global model in a specific embodiment of the present invention;

[0053] Figure 8 is a flow chart of a heterogeneous federated learning method for audio modality data in another embodiment of the present invention;

[0054] Fig. 9 It is a structural block diagram of a heterogeneous federated learning system for audio modality data in another embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0056] Unless explicitly stated otherwise, throughout the specification and claims, the term “comprise” or variations such as “include” or “comprising”, etc., will be understood to include the stated elements or components but not to exclude other elements or components.

[0057] In the existing technical solutions, there is a method of federated learning, which specifically includes: before the formal training begins, the central server first distributes the initial model to each participant, and then each participant trains the obtained model according to the local data set. Next, each participant uploads the model parameters obtained from the local training to the central server. The central server aggregates all model parameters and then transmits the aggregated global model parameters back to each participant. Based on the aggregated global model, the client retrains through the local data set and then uploads the new model parameters. In the cyclic training process, as the global model continues to converge, the training ends when the accuracy reaches a preset threshold.

[0058] However, the core of federated learning is to allow the dispersed participants to collaborate in machine learning model training without disclosing private data to other participants, that is, each client participating in the training will have an impact on the generation of the global model. Then, when there are malicious attackers among the clients participating in the training, they deliberately use heterogeneous data for local model training, and then upload wrong model parameters in an attempt to destroy the stability of the global model; or in some special application scenarios, each client is limited by its own nature, environment and other factors, and its own data distribution is highly unbalanced. In this case, the technical solution will inevitably have problems such as low model aggregation accuracy, high requirements on the client's data structure, and poor adaptability of the solution itself.

[0059] The inventor of the present invention discovered the main shortcomings of the prior art and proposed a new technical implementation idea based on the shortcomings of the prior art: randomly select clients to participate in the training in each learning round. During the training process, not only the local model parameters are uploaded, but also the output results of the intermediate layer / linear layer of the Mel spectrum graph corresponding to the local audio data are uploaded. Subsequently, the local model is updated based on the aggregated global model and the local parameters are optimized based on the above-mentioned output results after aggregation. By optimizing the local model, the accuracy of the global model is improved.

[0060] Please refer to Figure 1 , which is a schematic diagram of an application scenario of the audio modality data heterogeneous federated learning method provided by the present invention under one implementation mode, and the scenario specifically includes: a server and multiple clients.

[0061] It should be noted that multiple clients are all connected to the server through communication connections, but whether there is a communication connection between the clients can be flexibly set according to the usage scenario. The communication network derived from the above communication connection may include various connection types, including but not limited to: wired connection, wireless connection or optical fiber cable connection, etc. At the same time, the communication network may be a local area network, a metropolitan area network, a wide area network or any combination of the three.

[0062] Each client can perform data transmission with the server based on the above communication connection. At the same time, each client is provided with a corresponding local data set, and a local model can be trained based on the local data set and the global model. In the present invention, the local model has multiple model layers. By converting the local audio data into a Mel-spectrogram and then inputting it into the corresponding model, the time-frequency features corresponding to the local audio data can be obtained, and by inputting the time-frequency features of the local audio data into the linear layer of the corresponding model, the logit corresponding to the local audio data can be obtained.

[0063] The server is preset with an aggregation model, which can receive local model parameters and local knowledge uploaded by the client based on the communication connection, and aggregate the model parameters, logit, and time-frequency features of each client respectively to generate global model parameters, global time-frequency features and global logit, and then transmit the aggregated data back to the clients participating in the training.

[0064] It should be noted that, in the application scenario corresponding to the heterogeneous federated learning method for audio modal data provided by the present invention, multiple user terminals are also provided. The user terminal is used to provide the user with the necessary manual settings for implementing this solution, such as the aggregation parameters of the server, the data set settings of the local model, the configuration of the model aggregation function, etc. At the same time, it can be understood that the user terminal is installed with a computer software program that matches the heterogeneous federated learning method for audio modal data provided by this method; the user terminal may include but is not limited to a desktop computer (PC terminal), a desktop computer, a smart phone, a handheld computer, a tablet computer, a personal digital assistant (PDA), and other portable electronic devices or wearable electronic devices, and the embodiments of the present invention do not limit the above content.

[0065] It should also be noted that the audio modality data heterogeneous federated learning method of the embodiment of the present invention can be applied to the audio modality data heterogeneous federated learning system of the embodiment of the present invention. The audio modality data heterogeneous federated learning system can be configured in a terminal. The terminal may include but is not limited to a PC (Personal Computer), a PDA (tablet computer), a smart phone, a smart wearable device, and the like.

[0066] On the other hand, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms, but the embodiments of the present invention do not limit this.

[0067] Please refer to Figure 2 FIG. 2 is a flow chart of a method for heterogeneous federated learning of audio modal data in one embodiment of the present invention. The method for heterogeneous federated learning of audio modal data specifically includes the following steps:

[0068] S201: receiving global model parameters and global knowledge sent by a server, and performing local model training based on the received global model parameters and global knowledge;

[0069] It is understandable that in the first round of training, since the local knowledge has not been uploaded, the global knowledge received at this time is empty, and the global model parameters received are the model initialized by the server side. At the same time, in an embodiment of the present invention, the local model is a multi-layer structure, specifically including an input layer, an intermediate layer and an output layer. Usually the intermediate layer may include but is not limited to a convolutional layer, a pooling layer, a loop layer, etc.; the output layer, which is also the last layer, is usually a linear layer for the integration of global features. In particular, the embodiments of the present invention do not limit the types of global models and local models. However, considering that the present invention focuses on the application scenario of the audio modality. Different from image recognition, audio is strongly correlated with time, while pictures are only related to pixels, colors, etc. Therefore, in an embodiment of the present invention, the local model and / or global model preferably used is a temporal convolutional neural network model (TC-ResNet) so that the model can learn the temporal relationship characteristics in the audio.

[0070] In an exemplary embodiment, local model training is performed based on the received global model parameters and global knowledge, including: updating the local model based on the global model parameters, and training the updated local model based on a preset local data set; generating a first loss function based on the probability distribution of each received global knowledge; and optimizing the local model parameters in training based on the first loss function.

[0071] Among them, in the tth round, the local model is updated based on the global model parameters, and the formula for training the updated local model based on the preset local data set is:

[0072]

[0073] Among them, w t-1 is the global model parameter of the t-1th round, D i is the local dataset of client i, are the model parameters generated by client i in the tth round based on the global model training.

[0074] It should be noted that, in the embodiments of the present invention, the types of knowledge are not limited. Also due to the correlation between audio data and time, the use of intermediate layer features (time-frequency features) that can reflect the time-frequency characteristics of audio can capture the dynamic changes and frequency characteristics in the time series, and pay more attention to the timing and frequency distribution. Furthermore, in order to ensure that another type of knowledge can be closely associated with the above-mentioned time-frequency features, it is possible to choose to base the time-frequency features on the output logit of the linear layer. That is, preferably, in an embodiment of the present invention, the time-frequency features and logit corresponding to the local data set are used as local knowledge; the aggregated global logit and global time-frequency features are used as global knowledge.

[0075] In an exemplary embodiment, a new local model is trained based on the global model and global knowledge transmitted back by the server. The global time-frequency features are used as input, and a new logit representation is formed through the linear layer of the new local model of the client. The first loss function is formed by measuring the difference between the probability distribution of the new logit representation and the global logit probability distribution. Among them, the method of measuring the difference value may include but is not limited to: cross entropy calculation, KL divergence calculation, etc., which is not limited in the embodiment of the present invention. Based on minimizing the value of the above-mentioned first loss function, the parameters of the new local model can be optimized.

[0076] It is understandable that the sum of the probabilities of the directly obtained logit probability distribution and / or the probability distribution of the time-frequency features is not 1, which is not conducive to subsequent quantitative analysis. Therefore, when measuring the difference between the probability distribution represented by the new logit and the global logit probability distribution, the above probability distribution can be sorted based on a preset activation function. The activation function may include but is not limited to a softmax function, a logsftmax function, etc., which is not limited in the embodiment of the present invention. In particular, when measuring the above difference value with relative entropy, in order to avoid the final loss function value being negative, which requires additional calculation steps for data adjustment, it is preferred to use a softmax function for one of the two inputs and a logsoftmax function for the other.

[0077] Furthermore, in order to enable the client to learn more about the predicted distribution, the aggregation coefficient T is used to make the distribution of logit smoother. The smoothed probability distribution contains more relationship information between categories. Therefore, the probability distribution logit is divided by the preset aggregation coefficient to make the probability distribution relatively smooth and carry more information. On the other hand, multiplying by the square of the aggregation coefficient outside the function is to ensure the stability of the loss scale. Because in the actual loss, the two corresponding output quantities are processed by softmax and logsoftmax, and scaled by the aggregation coefficient T. In order to keep the scale of the loss unchanged, it is necessary to multiply the square of the aggregation coefficient T in the final loss calculation. 2 .

[0078] Specifically, the first loss function can be set as:

[0079] Loss=KLDivLoss(logsoftmax(Φ(feature global ) / T), softmax(logit global / T))×T×T

[0080] Among them, T is the aggregation coefficient; feature global is the mean of the global time-frequency features of multiple rounds containing multiple data categories; logitglobal is the mean of the global logit of multiple rounds containing multiple data categories; Loss is the first loss function. It can be understood that data of different categories represent data with different labels.

[0081] In another embodiment, the loss function of training the local model on the local data set is considered at the same time, and the loss function caused by the training is fused with the loss function of the aforementioned logit and time-frequency features to generate a better first loss function formula as follows:

[0082] loss CE =CEloss(W(input),target)

[0083] loss KLD =KLDivLoss(logsoftmax(Φ(feature global ) / T), softmax(logit global / T))×T×T

[0084] Loss = β × loss CE +(1-β)loss KLD

[0085] Among them, W (input) refers to the predicted label obtained by inputting local audio data into the local model, target is the real label corresponding to the input local audio data; T is the aggregation coefficient; feature global is the mean of the global time-frequency features of multiple rounds containing multiple data categories; logit global is the mean of the global logit of multiple rounds containing multiple data categories; Loss is the first loss function; β is a number greater than 0 and less than 1.

[0086] S202: Acquire local knowledge corresponding to the local data set based on the trained local model;

[0087] In an exemplary embodiment of the present invention, local knowledge corresponding to a local data set is obtained based on a trained local model, including: converting the local audio data into a Mel-spectrogram, inputting the Mel-spectrogram into the corresponding local model, obtaining and recording the output of the intermediate layer as the time-frequency features corresponding to the local audio data; inputting the time-frequency features corresponding to the local audio data into the linear layer of the local model, and obtaining the output of the linear layer as the logit corresponding to the local audio data; obtaining the mean of the time-frequency features corresponding to the local audio data with the same label and the mean of the logit corresponding to the local audio data with the same label; uploading the mean of the time-frequency features and the mean of the logit corresponding to each label to a server.

[0088] It should be noted that the Mel-spectrogram is a two-dimensional time-frequency diagram, with the horizontal axis being time and the vertical axis being the Mel-frequency, and the color or intensity representing the frequency energy at each moment. The Mel-spectrogram can better simulate the auditory characteristics of humans, reduce the complexity of feature dimensions, and enhance the ability to express features by mapping the audio signal to the Mel-scale frequency. Optionally, before the Mel-spectrogram is input into the local model, a filtering operation may be added to the acquired Mel-spectrogram. The filtering operation may remove some unnecessary noise in the Mel-spectrogram, thereby emphasizing the main components and features of the corresponding local audio data, thereby improving the performance of subsequent tasks. It is understandable that the embodiments of the present invention do not limit the definition of noise frequency and the means of filtering.

[0089] It should be noted that in the context of deep learning, logit is a vector that can be regarded as the unnormalized probability of the output of the neural network. In the embodiment of the present invention, the extracted time-frequency features and their corresponding logit reflect the relationship between the time-frequency features and their predicted distribution, allowing the local client model to learn the global knowledge about the predicted distribution of the time-frequency features. In one embodiment of the present invention, assuming that the local data set D of client i is i There is C i = {c1, c2...} labels. Take one of the labels c j ∈C i For example, the specific operation is as follows: Assume that the label is c j Local audio dataset in Total The label is c j The audio data. All data in the form of audio data For example, convert it into a Mel spectrum Then input it into the trained model and you will get in Representation data Import to local model The result of the last linear layer output obtained in Representation data Import to local model The time-frequency features obtained in the middle layer.

[0090] S203: Uploading the trained local model parameters and local knowledge to the server, so that the server updates the global model and global knowledge;

[0091] It should be noted that in order to reduce the uploading of redundant data and ensure the reliability of uploaded data, and at the same time, avoid directly uploading the time-frequency features and logit corresponding to all data, which will leak the privacy of users, what should be uploaded to the server is the mean of the time-frequency features and the mean of the logit corresponding to each label rather than the time-frequency features and logit corresponding to the local audio data directly obtained.

[0092] At the same time, it should be noted that in deep learning tasks under the audio modality, even if the number of samples in certain categories is small, their time-frequency features are still of great value. Because different clients may have similar pictures, but because each person's voiceprint information is different, it is very unlikely that the client will have the same audio data. The core features of audio data include information about changes in the time dimension, and this time-frequency information is often closely related to the semantic expression of the audio. By retaining data knowledge of all categories, the temporal characteristics of the audio can be more comprehensively covered, avoiding the omission of rare but important feature information that may contribute to the model. Therefore, in an embodiment of the present invention, the mean of the time-frequency features corresponding to all labels in the client and the mean of the logit need to be uploaded.

[0093] S204: Receive the global model parameters and global knowledge sent by the server, and start the next round of training until the global model meets the set conditions.

[0094] It should be noted that the condition for the end of training may be that the number of training rounds reaches a preset threshold, or that the accuracy of the local model / global model reaches a preset threshold, and the embodiment of the present invention does not impose any limitation on this.

[0095] Please refer to Figure 8 FIG. 2 is a flow chart of a method for heterogeneous federated learning of audio modal data in one embodiment of the present invention. The method for heterogeneous federated learning of audio modal data specifically includes the following steps:

[0096] S801: Select multiple clients, send global model parameters and global knowledge to the selected clients, and enable the selected clients to perform local model training;

[0097] It should be noted that the number of clients selected for training should be less than or equal to the total number of clients. Figure 7 As shown, it is a line graph showing the relationship between the number of clients participating in training in each learning round and the accuracy of the global model under a specific embodiment of the present invention. It can be seen that the more clients participate in learning in each learning round, the fewer rounds are required to improve the accuracy of the global model.

[0098] On the other hand, in order to improve the accuracy of the global model, the server needs to perform additional aggregation tasks. As the number of clients participating in the training increases, the GPU computing power consumed by the server will also increase, causing load pressure on the system. At the same time, almost all of the clients simulated in the test phase are placed on the same device, and the local training of the client will also cause a large consumption of the system's GPU resources. Therefore, in order to reduce the pressure on the system, the number of clients participating in each round of learning can be dynamically adjusted according to the GPU consumption and the GPU size of the system. The embodiment of the present invention does not limit the number of clients participating in each round of learning and the method of selecting the clients participating in the training.

[0099] S802: receiving and aggregating local model parameters and local knowledge uploaded by the selected client;

[0100] Among them, in one embodiment of the present invention, the local model parameters and local knowledge uploaded by the selected client are aggregated, including: aggregating the uploaded local model parameters to generate global model parameters; aggregating the uploaded logit to generate global logit; aggregating the uploaded time-frequency features to generate global time-frequency features.

[0101] Furthermore, the formula for aggregating the uploaded logit and time-frequency features is:

[0102]

[0103]

[0104] in, is the number of clients with label c among all the clients participating in the training j The sum of the number of local audio data, The client b participating in the training has a label of c j The amount of local audio data; The label of client b participating in the training is c j The logit corresponding to the local audio data; The label of client b participating in the training is c j The feature corresponding to the local audio data; The category obtained by the aggregation in the tth round is c j The global logit of The category obtained by the aggregation in the tth round is c j The global time-frequency characteristics.

[0105] In an embodiment of the present invention, the logit uploaded by each client and the time-frequency features uploaded by each client are aggregated and the operation method of direct averaging is not adopted. Considering the data heterogeneity of each client in the application scenario of the present invention. Therefore, if there is a client with very little data corresponding to a certain label, then the corresponding uploaded logit and time-frequency features will have a large deviation, and direct averaging will cause the final global logit and global time-frequency features to be inaccurate. Therefore, the present invention adopts the data volume ratio as the weight coefficient to more accurately reflect the impact of logit and time-frequency features.

[0106] S803: Based on the aggregated local model parameters, update the global model parameters; based on the aggregated local knowledge, combine the historical global knowledge, and update to obtain new global knowledge;

[0107] In an exemplary embodiment of the present invention, the combination of historical global knowledge and updating of global knowledge includes: collecting multiple rounds of global logits and multiple rounds of global time-frequency features; calculating the mean of the multiple rounds of global logits as a new global logit; and calculating the mean of the multiple rounds of global time-frequency features as a new global time-frequency feature.

[0108] In a specific implementation of the present invention, the formula for updating global knowledge is as follows:

[0109]

[0110]

[0111] in, The category obtained by aggregating local knowledge in round T is c j The global logit of The category obtained by aggregating local knowledge in round T is c j The global time-frequency characteristics. The category of the last N rounds is c j The mean of the global logit, is the category c in the last N rounds j The mean of the global time-frequency features.

[0112] Based on this, the final global knowledge is:

[0113]

[0114]

[0115] It should be noted that, since the client data is not independent and identically distributed, even if all the category knowledge uploaded by the local client is aggregated together, the global knowledge may lack a certain category, or the knowledge of a certain category may be unstable due to the small number. In order to avoid the above problems, in one embodiment of the present invention, the server collects the global knowledge of the last r rounds and averages it, thus avoiding the problem of unstable knowledge of a certain category or missing knowledge of a certain category. In the next round of training, the server parameters and the averaged global knowledge are sent to the clients participating in the training.

[0116] S804: Reselect multiple clients, and send the updated global parameters and global knowledge to the reselected multiple clients for the next round of training until the global model meets the set conditions.

[0117] By reselecting multiple clients, the updated global parameters and global knowledge are sent to the reselected multiple clients for the next round of training, and a new training cycle begins.

[0118] It should be noted that the condition for the end of training may be that the number of training rounds reaches a preset threshold, or that the accuracy of the local model / global model reaches a preset threshold, and the embodiment of the present invention does not impose any limitation on this.

[0119] In order to further illustrate the technical solution of the present invention and the beneficial effects of the solution, a group of specific embodiments are first given as follows:

[0120] The experimental data set uses Google's open speech command data set Speech Commands V1 and V2. Each speech in the data set contains one word, and the speech contains 12 class labels, namely yes, no, up, down, left, right, on, off, stop, go, silence, and unknown. These speech need to extract audio features, namely Mel spectrograms, before being input into the model. Figure 5 as well as Figure 6 The mel spectrograms labeled yes and no are shown respectively. The data distribution adopts Dirichlet distribution, where the data distribution heterogeneity of α = 0.1 is serious, which we think is non-IID, and the data distribution of α = 5 is uniform, which is very close to IID.

[0121] There are 100 clients and one server, but only 10 clients are selected to participate in training in each round, and a total of 50 rounds of training. The server stores the global knowledge of the last 3 rounds for averaging, the aggregation coefficient T is 2, and the weight β of the loss function is 0.6.

[0122] It can be seen from Table 1 that when α is 0.1 and 5, on the datasets Speech Commands V1 and V2, compared with the federated learning framework of the prior art, the present invention can improve the global model performance, especially when α is 0.1, the global model accuracy is improved by more than 10%.

[0123] Table 1

[0124]

[0125] Please refer to Figure 3 As shown, based on the same inventive concept as the aforementioned audio modality data heterogeneous federated learning method, an embodiment of the present invention provides an audio modality data heterogeneous federated learning system 300, which includes: a receiving module 301, an acquisition module 302, an upload module 303 and a loop module 304.

[0126] Specifically, the receiving module 301 is used to receive the global model parameters and global knowledge sent by the server, and perform local model training based on the received global model parameters and global knowledge; the acquisition module 302 is used to obtain the local knowledge corresponding to the local data set based on the trained local model; the upload module 303 is used to upload the trained local model parameters and local knowledge to the server, so that the server updates the global model and global knowledge; the loop module 304 is used to receive the global model parameters and global knowledge sent by the server, and start the next round of training until the global model meets the set conditions.

[0127] Please refer to Fig. 9 As shown, based on the same inventive concept as the heterogeneous federated learning method for audio modal data in another embodiment described above, an embodiment of the present invention provides a heterogeneous federated learning system 900 for audio modal data, which includes: a selection module 901, an aggregation module 902, an update module 903 and a sending module 904.

[0128] Specifically, the selection module 901 is used to select multiple clients, send the global model parameters and global knowledge to the selected clients, and enable the selected clients to perform local model training; the aggregation module 902 is used to receive and aggregate the local model parameters and local knowledge uploaded by the selected clients; the update module 903 is used to update the global model parameters based on the aggregated local model parameters; the new knowledge obtained by aggregation is combined with the historical global knowledge to update the new global knowledge; the sending module 904 is used to re-select multiple clients, and send the updated global parameters and global knowledge to the re-selected multiple clients for the next round of training until the global model meets the set conditions.

[0129] Please refer to Figure 4As shown, an embodiment of the present invention further provides an electronic device 400, which includes at least one processor 401, a memory 402 (for example, a non-volatile memory), a memory 403, and a communication interface 404, and the at least one processor 401, the memory 402, the memory 403, and the communication interface 404 are connected together via a bus 405. At least one processor 401 is used to call at least one program instruction stored or encoded in the memory 402, so that the at least one processor 401 performs various operations and functions of the heterogeneous federated learning method for audio modality data described in various embodiments of this specification.

[0130] In an embodiment of the present specification, the electronic device 400 may include, but is not limited to, a personal computer, a server computer, a workstation, a desktop computer, a laptop computer, a notebook computer, a mobile electronic device, a smart phone, a tablet computer, a cellular phone, a personal digital assistant (PDA), a handheld device, a messaging device, a wearable electronic device, a consumer electronic device, and the like.

[0131] An embodiment of the present invention also provides a computer-readable medium, which carries computer execution instructions. When the computer execution instructions are executed by a processor, they can be used to implement various operations and functions of the heterogeneous federated learning method for audio modal data described in the various embodiments of this specification.

[0132] The computer-readable medium in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0133] In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0134] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatuses, systems, and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as a combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0136] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.

[0137] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0138] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A method for heterogeneous federated learning of audio modal data, characterized in that: include: Receiving global model parameters and global knowledge sent by the server, and performing local model training based on the received global model parameters and global knowledge; Acquire local knowledge corresponding to a local data set based on the trained local model, where the local data set includes local audio data and labels corresponding to the local audio data; Uploading the trained local model parameters and local knowledge to the server so that the server updates the global model and global knowledge; Receive the global model parameters and global knowledge sent by the server and start the next round of training until the global model meets the set conditions.

2. The method for heterogeneous federated learning of audio modality data according to claim 1, characterized in that: The acquiring local knowledge corresponding to the local data set based on the trained local model and uploading the trained local knowledge to the server includes: Convert the local audio data into a Mel-spectrogram, input the Mel-spectrogram into the corresponding local model, and obtain and record the output of the intermediate layer as the time-frequency feature corresponding to the local audio data; Input the time-frequency features corresponding to the local audio data into the linear layer of the local model, and obtain the output of the linear layer as the logit corresponding to the local audio data; Get the mean of the time-frequency features corresponding to the local audio data with the same label and the mean of the logit corresponding to the local audio data with the same label; The mean of the time-frequency features and the mean of the logit corresponding to each label are uploaded to the server.

3. The method for heterogeneous federated learning of audio modality data according to claim 1, characterized in that: The performing local model training based on the received global model parameters and global knowledge includes: Updating the local model based on the global model parameters, and training the updated local model based on a preset local data set; Generate a first loss function based on the received probability distribution of each of the global knowledge; Optimize local model parameters in training based on the first loss function.

4. The method for heterogeneous federated learning of audio modality data according to claim 3, characterized in that: The first loss function is: Loss=KLDivLoss(logsoftmax(Φ(feature global ) / T),softmax(logit global / T))×T×T Among them, T is the aggregation coefficient; feature global is the mean of the global time-frequency features of multiple rounds; logit global is the mean of the global logit of multiple rounds; Loss is the first loss function.

5. The method for heterogeneous federated learning of audio modality data according to claim 3, characterized in that: The first loss function is: loss CE =CEloss(W(input),target) loss KLD =KLDivLoss(logsoftmax(Φ(feature global ) / T),softmax(logit global / T))×T×T Loss=β×loss CE +(1-β)loss KLD Among them, W (input) refers to the predicted label obtained by inputting local audio data into the local model, target is the real label corresponding to the input local audio data; T is the aggregation coefficient; feature global is the mean of the global time-frequency features of multiple rounds; logit global is the mean of the global logit of multiple rounds; Loss is the first loss function; β is a number greater than 0 and less than 1.

6. A method for heterogeneous federated learning of audio modal data, characterized in that: include: Select multiple clients, send global model parameters and global knowledge to the selected clients, and enable the selected clients to perform local model training; Receive and aggregate local model parameters and local knowledge uploaded by the selected client; Based on the aggregated local model parameters, updating the global model parameters; Based on the aggregated local knowledge and combined with the historical global knowledge, new global knowledge is updated; Reselect multiple clients and send the updated global parameters and global knowledge to the reselected multiple clients for the next round of training until the global model meets the set conditions.

7. The method for heterogeneous federated learning of audio modality data according to claim 6, characterized in that: The formula for aggregating the local knowledge uploaded by the selected client is: in, is the number of clients with label c among all the clients participating in the training j The sum of the number of local audio data, The client b participating in the training has a label of c j The amount of local audio data; The label of client b participating in the training is c j The logit corresponding to the local audio data; The label of client b participating in the training is c j The feature corresponding to the local audio data; The label obtained by the aggregation in the tth round is c j The global logit of The label obtained by the aggregation in the tth round is c j The global time-frequency characteristics.

8. The method for heterogeneous federated learning of audio modality data according to claim 6, characterized in that: The local knowledge after aggregation is combined with the historical global knowledge to update the new global knowledge, including: Collect global logit and global time-frequency features of multiple rounds; Calculate the mean of the global logit of the multiple rounds as a new global logit; The average of the global time-frequency features of the multiple rounds is calculated as a new global time-frequency feature.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the audio modality data heterogeneous federated learning method according to any one of claims 1 to 8 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the audio modality data heterogeneous federated learning method described in any one of claims 1-8.

Images