Voiceprint recognition method based on federal learning

Abstract

The invention relates to a voiceprint recognition method based on federated learning, and the method comprises the steps: enabling a to-be-processed client to generate a local personalized initial parameter group based on an initial parameter group issued by a server through the distributed characteristics of federated learning, and connecting model memory and momentum increment, a voiceprint recognition model is obtained through a first sample subset and a second sample subset which are independently sampled on the basis of a local audio data sample set, iteration training updating of a parameter set is achieved, and finally the voiceprint recognition model is obtained in a global aggregation mode of a server and is used for achieving recognition of each preset audio sender tag for audio; a voiceprint recognition method is designed, the problems of data privacy protection and distribution heterogeneity of audio data among different devices are effectively solved, and the robustness and generalization ability of a voiceprint recognition model in a system in practical application are improved.

Description

Technical Field

[0001] The present invention relates to a voiceprint recognition method based on federated learning, and belongs to the technical field of distributed computing architecture and voiceprint recognition. Background Art

[0002] As a biometric recognition technology, voiceprint recognition has broad application prospects. However, due to the privacy protection and data silo problems of voiceprint data, traditional centralized learning methods are difficult to meet actual needs. Federated Learning effectively solves the problems of data privacy protection and data silos by training models locally on multiple clients and aggregating global models. However, due to the diversity and distribution heterogeneity of audio data, traditional federated learning methods face many challenges in handling voiceprint recognition tasks. The characteristics of audio data include language or dialect differences, recording content diversity, environmental noise, device performance, etc. The heterogeneity of these features will have a negative impact on traditional federated learning models, making it difficult for the model to effectively process and learn these complex audio features during local training and global aggregation. Therefore, voiceprint recognition systems based on traditional federated learning often perform poorly when facing audio data with high statistical heterogeneity, and it is difficult to achieve ideal recognition performance. In addition, in traditional federated learning, each client usually adopts a fixed task sampling strategy, ignoring the differences between different tasks. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a voiceprint recognition method based on federated learning, which utilizes the distributed characteristics of federated learning to effectively solve the problems of data privacy protection and distribution heterogeneity of audio data among different devices, and improve the robustness and generalization ability of the voiceprint recognition model in the system in practical applications.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions: the present invention designs a voiceprint recognition method based on federated learning, based on a server and a client set, the server initializes r=1, and initializes the model to be trained to form the model to be trained under the rth global iteration, and then performs the following steps to obtain a voiceprint recognition model for realizing the recognition of preset labels of each audio sender for audio;

[0005] Step A. The server randomly selects at least two clients from the client set as the clients to be processed in the rth global iteration, and sends the model to be trained in the rth global iteration to the clients to be processed in the rth global iteration, and then proceeds to step B;

[0006] Step B. In the rth global iteration, each client to be processed is based on its local audio data sample set, connects the model memory and momentum increment, takes the audio sample as input and the audio sender corresponding to the audio sample as output, trains the model to be trained, obtains the local trained model, and uploads it to the server in combination with the corresponding prediction accuracy; then proceeds to step C;

[0007] Step C. The server aggregates the parameter groups of the local trained models uploaded by each client to be processed under the r-th global iteration to obtain the global parameter group under the r-th global iteration, and then obtains the average accuracy of the prediction accuracy of the local trained models uploaded by each client to be processed under the r-th global iteration, and determines whether the average accuracy is greater than the preset global model accuracy threshold or whether the number of global iterations reaches the preset global maximum number of iterations. If so, the model to be trained is combined with the global parameter group under the r-th global iteration to form a voiceprint recognition model; otherwise, the model to be trained is combined with the global parameter group under the r-th global iteration to form a model to be trained under the r+1-th global iteration, and the value of r is updated by adding 1, and then returns to step A.

[0008] As a preferred technical solution of the present invention: in step B, each client to be processed in the rth global iteration initializes h=1 respectively, and performs the following steps:

[0009] Step B1. The client to be processed randomly obtains a preset first number of samples covering all audio sender labels in the audio data sample set from its local audio data sample set including each audio sample and each audio sample corresponding to the audio sender label, and forms an initial sample subset. And based on the starting parameter set of the model to be trained The loss function corresponding to the client to be processed and the model to be trained take the audio sample as input and the audio sender corresponding to the audio sample as output to generate a local personalized starting parameter set for the model to be trained. Will As As Then proceed to step B2;

[0010] Among them, 1≤i r ≤I r , I r represents the number of clients to be processed in the rth global iteration, Indicates the i-th r The initial sample subset corresponding to the clients to be processed, Indicates the i-th rThe starting parameter set of the model to be trained in the pending clients, Indicates the i-th r The local personalized starting parameter set of the model to be trained in the client to be processed, Indicates the i-th r The parameter group of the model to be trained in the client to be processed corresponding to the hth local iteration; Indicates the i-th r The local personalized parameter group of the model to be trained in the client to be processed corresponding to the h-th local iteration;

[0011] Step B2: The client to be processed randomly obtains a preset second number of samples covering all audio sender labels in the audio data sample set from its local audio data sample set to form a first sample subset. And according to the parameter group Contact the loss function corresponding to the client to be processed, and the model to be trained corresponding to the audio sample as input and the audio sender corresponding to the audio sample as output to obtain the parameter set Then proceed to step B3; wherein, Indicates the i-th r The first sample subset of the hth local iteration corresponds to the clients to be processed. Indicates the i-th r The parameter group of the model to be trained in the client to be processed corresponding to the h+1th local iteration;

[0012] Step B3. The client to be processed is based on the parameter group Local Personalization Parameter Group The first sample subset The model memory and the preset momentum increment are linked, the loss function corresponding to the client to be processed, and the model to be trained takes the audio sample as input and the audio sender corresponding to the audio sample as output to obtain the local personalized parameter set. Then proceed to step B4; wherein, Indicates the i-th r The local personalized parameter group of the to-be-trained model in the to-be-processed client corresponding to the h+1th local iteration;

[0013] Step B4: The client to be processed randomly obtains a preset third number of samples covering all audio sender labels in the audio data sample set from its local audio data sample set to form a second sample subset. At the same time, a preset fourth number of samples covering all audio sender labels in the audio data sample set are randomly obtained to form a third sample subset Then proceed to step B5; wherein, Indicates the i-th r The second sample subset of the hth local iteration corresponds to the clients to be processed. Indicates the i-th r The third sample subset of the hth local iteration corresponding to the clients to be processed;

[0014] Step B5. The client to be processed receives the second sample subset. The third sample subset Parameter Group Local Personalization Parameter Group Apply the Hessian matrix to link the loss function corresponding to the client to be processed, and the model to be trained to take the audio sample as input and the audio sender corresponding to the audio sample as output, and for the parameter group Update and then go to step B6;

[0015] Step B6. The client to be processed determines the parameter group Is the prediction accuracy of the model to be trained greater than the preset local prediction accuracy threshold? If yes, the parameter group The trained model is used as the local trained model of the client to be processed, and is uploaded to the server in combination with the corresponding prediction accuracy; otherwise, the value of h is updated by adding 1, and the process returns to step B2.

[0016] As a preferred technical solution of the present invention: in step B1, the client to be processed is based on the initial sample subset Based on the starting parameter set of the model to be trained The model to be trained takes the audio sample as input and the audio source corresponding to the audio sample as output, according to the following formula:

[0017]

[0018] Get the local personalized starting parameter set of the model to be trained Among them, α represents the preset first learning rate, represents the derivative calculation, Indicates the i-th r The loss function corresponding to the clients to be processed is: Indicates the i-th r Pending clients based on parameter group The model to be trained is about the initial sample subset The loss function corresponding to the sample x in The loss result is Represents the initial sample subset The number of samples in .

[0019] As a preferred technical solution of the present invention: in step B2, the client to be processed is configured according to the parameter group The first sample subset The model to be trained takes the audio sample as input and the audio source corresponding to the audio sample as output, according to the following formula:

[0020]

[0021] Get parameter group Among them, α represents the preset first learning rate, represents the derivative calculation, Indicates the i-th r The loss function corresponding to the clients to be processed is: Indicates the i-th r Pending clients based on parameter group The model to be trained is about the first sample subset The loss function corresponding to sample y in The loss result is Represents the first sample subset The number of samples in .

[0022] As a preferred technical solution of the present invention: in step B3, the client to be processed is configured according to the parameter group Local Personalization Parameter Group The first sample subset The model memory, the preset momentum increment β, and the model to be trained correspond to the audio sample as input and the audio emitter corresponding to the audio sample as output, according to the following formula:

[0023]

[0024] Get local personalized parameter group Among them, α represents the preset first learning rate, represents the derivative calculation, Indicates the i-th r The loss function corresponding to the clients to be processed is: Indicates the i-th r Pending clients based on parameter group The model to be trained is about the first sample subset The loss function corresponding to sample y in The loss result is Represents the first sample subset The number of samples in .

[0025] As a preferred technical solution of the present invention: in step B5, the client to be processed is based on the second sample subset The third sample subset Parameter Group Local Personalization Parameter Group The model to be trained takes the audio sample as input and the audio source corresponding to the audio sample as output, according to the following formula:

[0026]

[0027] For parameter groups Update; wherein α represents the preset first learning rate, η represents the preset second learning rate, I represents the unit matrix, represents derivative calculation, T represents transposition, Indicates the i-th r The loss function corresponding to the clients to be processed is: Indicates the i-th r Pending clients based on parameter group The model to be trained is about the second sample subset The loss function corresponding to sample z in The loss result is Represents the second sample subset The number of samples in Indicates the i-th r Pending clients based on local personalized parameter groups The model to be trained is about the third sample subset The loss function corresponding to sample t in The loss result is Represents the third sample subset The number of samples in .

[0028] In view of the above-mentioned design scheme, the present invention further designs a computer device, which is characterized by comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements a voiceprint recognition method based on federated learning when executing the computer program.

[0029] The voiceprint recognition method based on federated learning described in the present invention adopts the above technical solution and has the following technical effects compared with the prior art:

[0030] (1) The present invention designs a voiceprint recognition method based on federated learning. By utilizing the distributed characteristics of federated learning, the client to be processed generates a local personalized starting parameter group based on the starting parameter group sent by the server, and then connects the model memory and momentum increment to implement iterative training and updating of the parameter group based on the first sample subset and the second sample subset independently sampled from the local audio data sample set, and finally connects the global aggregation method of the server to obtain a voiceprint recognition model for realizing the recognition of the preset labels of each audio sender for the audio; the voiceprint recognition method is designed to effectively solve the data privacy protection and distribution heterogeneity problems of audio data between different devices, and improve the robustness and generalization ability of the voiceprint recognition model in the system in practical applications;

[0031] (2) In the voiceprint recognition method based on federated learning designed by the present invention, a model memory mechanism is introduced to track and estimate the personalized model gradient of each task, thereby enhancing the adaptability of the model to local data features, making the design highly adaptable when dealing with heterogeneous data distribution; and in combination with the momentum increment strategy, the momentum factor is used to smooth the model update to avoid the model falling into the local optimal solution, thereby improving the stability and generalization ability of the model, and converging towards the optimal solution more quickly, while maintaining the high stability of the internal model of the federated system;

[0032] (3) In the voiceprint recognition method based on federated learning designed by the present invention, three independent sampling data sets are used to calculate the gradient estimation, which effectively reduces the estimation variance and improves the accuracy of the gradient estimation. Model agnostic meta-learning (MAML) is introduced for local training of the client, focusing on learning deep feature knowledge, especially processing complex voiceprint features. This combination makes the local model in the system have an advantage in the initial convergence speed, and can quickly achieve a high accuracy rate, saving time and resources; and gradient incremental balance is introduced. By balancing the mixed gradient of sample initialization gradient, random meta-learning gradient and federated aggregation global model, overfitting and catastrophic forgetting are effectively prevented, so that the designed voiceprint recognition method shows strong robustness, and can still maintain high performance and robustness in a highly heterogeneous environment, significantly improving the adaptability and generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1-1 It is a schematic diagram of the client process in the voiceprint recognition method based on federated learning designed by the present invention;

[0034] Figure 1-2 It is a flow chart of the server in the voiceprint recognition method based on federated learning designed by the present invention;

[0035] Figure 2 The following is a flowchart showing a simple application of the system according to the present invention. DETAILED DESCRIPTION

[0036] The specific implementation modes of the present invention will be further described in detail below in conjunction with the accompanying drawings.

[0037] The present invention designs a voiceprint recognition method based on federated learning, such as Figure 2 As shown, based on a set of servers and clients (such as mobile phones, microphones, computers and other devices), the server initializes r=1, and initializes the model to be trained to form the model to be trained under the rth global iteration, and then performs the following steps to obtain a voiceprint recognition model for realizing the recognition of preset labels of each audio sender for audio, solving the problem that each device in the client set has significant heterogeneity in sound quality, noise, and audio signal lights due to performance and brand, and training a voiceprint recognition model of a time delay neural network (TDNN) combined with a channel attention mechanism in an environment with distribution heterogeneity.

[0038] Step A. Figure 1-2 As shown, the server randomly selects at least two clients from the client set as the clients to be processed under the r-th global iteration, and the server sends the model to be trained under the r-th global iteration to each client to be processed under the r-th global iteration, and then enters step B.

[0039] Step B. Under the rth global iteration, each client to be processed is based on its local audio data sample set, connects the model memory and momentum increment, takes the audio sample as input and the audio sender corresponding to the audio sample as output, trains the model to be trained, obtains the local trained model, and uploads it to the server in combination with the corresponding prediction accuracy; then enters step C.

[0040] In practical application, the above step B is Figure 1-1 As shown, in the rth global iteration, each client to be processed initializes h=1 and executes the following steps B1 to B6.

[0041] Step B1. The client to be processed randomly obtains a preset first number of samples covering all audio sender labels in the audio data sample set from its local audio data sample set including each audio sample and each audio sample corresponding to the audio sender label, and forms an initial sample subset. And based on the starting parameter set of the model to be trained The loss function corresponding to the client to be processed, such as the cross entropy loss function SGD, and the model to be trained corresponding to the audio sample as input and the audio sender corresponding to the audio sample as output, are as follows:

[0042]

[0043] Get the local personalized starting parameter set of the model to be trained Will As As Then proceed to step B2. Initial sample subset after random sampling For model initialization and gradient calculation, through the initial sample subset Calculate the gradient of the loss function to make preliminary adjustments when the model is initialized, providing a solid foundation for the subsequent formal local training; where 1≤i r ≤I r , I r represents the number of clients to be processed in the rth global iteration, Indicates the i-th r The initial sample subset corresponding to the clients to be processed, Indicates the i-th r The starting parameter set of the model to be trained in the pending clients, Indicates the i-th r The local personalized starting parameter set of the model to be trained in the client to be processed, Indicates the i-th r The parameter group of the model to be trained in the client to be processed corresponding to the hth local iteration; Indicates the i-th r The model to be trained in the client to be processed corresponds to the local personalized parameter group of the hth local iteration; it determines the size of each update step. An appropriate learning rate can ensure that the model converges to the optimal solution quickly, avoiding instability caused by too large a step size or slow convergence caused by too small a step size. represents the derivative calculation, Indicates the i-th r The loss function corresponding to the clients to be processed is: Indicates the i-th r Pending clients based on parameter group The model to be trained is about the initial sample subset The loss function corresponding to the sample x in The loss result is Represents the initial sample subset The number of samples in .

[0044] Step B2: The client to be processed randomly obtains a preset second number of samples covering all audio sender labels in the audio data sample set from its local audio data sample set to form a first sample subset. And according to the parameter group The loss function corresponding to the client to be processed and the model to be trained corresponding to the audio sample as input and the audio sender corresponding to the audio sample as output are as follows:

[0045]

[0046] Get parameter group Then proceed to step B3; wherein, Indicates the i-th r The first sample subset of the hth local iteration corresponds to the clients to be processed. Indicates the i-th r The parameter group of the model to be trained in the client to be processed corresponding to the h+1th local iteration; Indicates the i-th r Pending clients based on parameter group The model to be trained is about the first sample subset The loss function corresponding to sample y in The loss result is Represents the first sample subset The number of samples in .

[0047] The above initial sample subset The first sample subset In practical applications, a sample subset is selected by uniform random sampling. The sampling size ratio can be set according to the client computing power and data scale.

[0048]

[0049] in, Indicates the i-th r The audio data sample set of the client to be processed, K is the number or proportion of samples, and this sample subset covers all the audio sender labels in the audio data sample set, fully reflecting the diversity of local data. According to statistical theory, small samples can reflect the characteristics of the overall data set, and if the number of samples is sufficient, statistical indicators such as mean and variance will tend to the true value of the overall data. When the sample is large enough, the mean of the randomly selected sample is close to the overall mean.

[0050] Step B3. In order to improve the adaptability of the model to each client's specific data, the traditional gradient descent and momentum update strategies are combined to greatly enhance the stability and robustness of the model while ensuring rapid convergence. Local Personalization Parameter Group The first sample subset The model memory and the preset momentum increment β are linked, as well as the loss function corresponding to the client to be processed, and the model to be trained corresponds to the audio sample as input and the audio sender corresponding to the audio sample as output, according to the following formula:

[0051]

[0052] Get local personalized parameter group Then proceed to step B4; wherein the momentum increment β is used to control the influence of the current iteration momentum on the next iteration update. Indicates the i-th r The local personalized parameter group of the model to be trained in the client to be processed corresponding to the h+1th local iteration, Indicates the i-th r Pending clients based on parameter group The model to be trained is about the first sample subset The loss function corresponding to sample y in The loss result is Represents the first sample subset The number of samples in .

[0053] It is an auxiliary vector used to update the model parameters. It can "remember" the previous gradient direction during the update, so that it can maintain a certain update speed when encountering saddle points or flat areas. Beta binding via momentum increment Information and current gradient information By introducing the β value, the update path can be smoothed during the gradient descent process, violent shocks can be avoided, and the convergence speed can be accelerated. A larger β value indicates that the previous generation of momentum has a greater impact on the current momentum, and the model is more inclined to move along the previous gradient direction when updating; on the contrary, the current gradient has a greater impact, and the update direction is more dependent on the current gradient information.

[0054] Step B4: The client to be processed randomly obtains a preset third number of samples covering all audio sender labels in the audio data sample set from its local audio data sample set to form a second sample subset. At the same time, a preset fourth number of samples covering all audio sender labels in the audio data sample set are randomly obtained to form a third sample subset Then proceed to step B5; wherein, represents the second sample subset of the h-th local iteration corresponding to the ir-th client to be processed in the r-th global iteration, Indicates the i-th rThe clients to be processed correspond to the third sample subset of the hth local iteration.

[0055] In the actual implementation of the above design, the second sample subset With the third sample subset Will be used to calculate the Hessian matrix and gradient estimation respectively, where the second sample subset The sample set used to calculate the Hessian matrix is ​​randomly sampled from the client local data set To ensure that the sample set can reflect the second-order statistical characteristics of the local data set.

[0056]

[0057] Among them, K2 is the number or proportion of samples. When selecting, the number of samples required to calculate the Hessian matrix should be considered. According to the different environments and data deployed in this patent, K2 is adjusted so that this sample set should contain enough data points to accurately estimate the curvature of the loss function.

[0058] The third sample subset The sample set used to calculate the gradient and the second sample subset The same, both are obtained by random sampling from the client local data set D i It is selected from to reflect the first-order statistical characteristics of the local data set.

[0059]

[0060] Where K3 is the sample size or proportion, The size of can be determined according to the number of samples and data environment required to calculate the gradient, ensuring the accuracy of the gradient estimation.

[0061] Second sample subset Will be used to calculate the Hessian moment The Hessian matrix is ​​the loss function For model parameters The second-order derivative matrix of , which describes the curvature information of the loss function in the parameter space. Specifically, the calculation process of the Hessian matrix is ​​as follows:

[0062]

[0063] In order to take into account the diversity and statistical characteristics of the data, we The second-order derivative of each sample z in is calculated and the average of these second-order derivatives is taken:

[0064]

[0065] The above formula is based on the second-order derivatives of the loss function with respect to the model parameters, which are calculated from the Jacobian of the gradient (i.e., the derivative matrix of the first-order derivative vector). Transpose the gradient and derive it again to get the second-order derivative of the loss function with respect to the model parameters The formula is as follows:

[0066]

[0067] Substituting this into the definition of the original Hessian:

[0068]

[0069] After the above derivation, the Hessian matrix In fact, the sample set The average value of the second-order derivative of each sample in is:

[0070]

[0071] Finally, the above formula shows how to The Hessian matrix is ​​calculated for each sample in and they are averaged to get the overall second-order derivative of the loss function.

[0072] The third sample subset Used to calculate the loss function Relative to model parameters The first-order derivative (i.e. gradient) of It is one of the core parts of the optimization process. Through this gradient, the model can know in which direction it should be adjusted and better fit the data. The gradient of the first-order derivative of the model parameters to guide the correction matrix.

[0073] According to the above and Calculate the final gradient update direction The specific formula is as follows:

[0074]

[0075] in, represents the i-th global iteration r The gradient update direction of the model to be trained in the client to be processed corresponds to the h+1th local iteration, I is the identity matrix, which is usually the same as the Hessian matrix The diagonal matrix of the same dimension plays the role of maintaining the original gradient direction in matrix multiplication. α represents the preset first learning rate, which is used to control the influence of the Hessian matrix on the gradient correction. The learning rate determines the extent to which the Hessian matrix changes the gradient update direction. Correction matrix The purpose is to adjust the direction and size of the original gradient. The role of this matrix is ​​to correct the gradient direction by introducing the Hessian matrix, making the gradient update more robust and accurate, especially when the loss function surface has a complex geometric structure. In summary, the new gradient update direction is obtained by the correction matrix of the Hessian matrix and the gradient update.

[0076] Step B5. The client to be processed receives the second sample subset. The third sample subset Parameter Group Local Personalization Parameter Group Apply the Hessian matrix, link the loss function corresponding to the client to be processed, and the model to be trained corresponding to the audio sample as input and the audio sender corresponding to the audio sample as output, according to the following formula:

[0077]

[0078] For parameter groups Update, and then enter step B6, where η represents the preset second learning rate, I represents the identity matrix, T represents the transposition, Indicates the i-th r Pending clients based on parameter group The model to be trained is about the second sample subset The loss function corresponding to sample z in The loss result is Represents the second sample subset The number of samples in Indicates the i-th r Pending clients based on local personalized parameter groups The model to be trained is about the third sample subset The loss function corresponding to sample t in The loss result is Represents the third sample subset The number of samples in .

[0079] Step B6. The client to be processed determines the parameter group Is the prediction accuracy of the model to be trained greater than the preset local prediction accuracy threshold? If yes, the parameter group The trained model is used as the local trained model of the client to be processed, and is uploaded to the server in combination with the corresponding prediction accuracy; otherwise, the value of h is updated by adding 1, and the process returns to step B2.

[0080] Step C. Figure 1-2 As shown, the server aggregates the parameter groups of the local trained models uploaded by each client to be processed under the r-th global iteration to obtain the global parameter group under the r-th global iteration, and then obtains the average accuracy of the prediction accuracy of the local trained models uploaded by each client to be processed under the r-th global iteration, and determines whether the average accuracy is greater than the preset global model accuracy threshold or whether the number of global iterations reaches the preset global maximum number of iterations. If so, the model to be trained is combined with the global parameter group under the r-th global iteration to form a voiceprint recognition model; otherwise, the model to be trained is combined with the global parameter group under the r-th global iteration to form a model to be trained under the r+1-th global iteration, and the value of r is updated by adding 1, and then returns to step A.

[0081] The practical application of the above technical solution utilizes the distributed characteristics of federated learning. Based on the starting parameter group sent by the server, the client to be processed generates a local personalized starting parameter group, and then links the model memory and momentum increment to achieve iterative training and update of the parameter group based on the first sample subset and the second sample subset independently sampled from the local audio data sample set, and finally links the global aggregation method of the server to obtain a voiceprint recognition model for audio recognition of preset labels of each audio sender; designs a voiceprint recognition method to effectively solve the problem of data privacy protection of audio data between different devices, and improves the robustness and generalization ability of the voiceprint recognition model in the system in practical applications.

[0082] Here, based on the design of sampling sample subsets in proportion to the local audio data sample set, the training is compared with the direct training with the entire local audio data sample set. When the local audio data is highly heterogeneous, if the training is performed with all local audio data samples (sampling ratio is 1), the performance of the trained model is lower than the performance of the model trained with the sample subset under small proportion sampling. The comparison of experimental data shows a decrease of about 8%. Among them, the accuracy of the model trained with all local audio data samples is 65.09%, and the accuracy of the model trained with the sample subset with a small proportion of 0.3 is 71.0 7%; in addition, in the implementation of the design of the present invention, momentum increment is also introduced to participate in training. Under actual sample verification, the accuracy of the model after training without momentum participation is 67.55%, and the accuracy of the model after training with momentum participation is about 71.11%. Moreover, when the data heterogeneity is not strong (the data is evenly distributed), training without momentum may fall into a local optimal solution, which is not as stable as the effect of training with momentum increment participation. Therefore, the application of a small proportion of sample subsets and the design of momentum participation in the design of the present invention effectively solve the problem of distribution heterogeneity of audio data among different devices.

[0083] And by introducing a model memory mechanism, tracking and estimating the personalized model gradient of each task, the model's adaptability to local data features is enhanced, making the design highly adaptable when dealing with heterogeneous data distributions. And combined with the momentum increment strategy, the momentum factor is used to smooth the model update to avoid the model falling into the local optimal solution, thereby improving the model's stability and generalization ability, and converging towards the optimal solution faster, while maintaining the high stability of the model within the federated system.

[0084] Moreover, three independent sampling data sets are used to calculate the gradient estimate, which effectively reduces the estimation variance and improves the accuracy of the gradient estimate. Model agnostic meta-learning (MAML) is introduced for local training on the client side, focusing on learning deep feature knowledge, especially processing complex voiceprint features. This combination makes the local model in the system have an advantage in the initial convergence speed, and can quickly achieve a high accuracy rate, saving time and resources. In addition, gradient incremental balance is introduced to effectively prevent overfitting and catastrophic forgetting by balancing the mixed gradient of sample initialization gradient, random meta-learning gradient and federated aggregated global model, so that the designed voiceprint recognition method shows strong robustness, and can still maintain high performance and robustness in highly heterogeneous environments, significantly improving the adaptability of the model.

[0085] The application of a small proportion of sample subsets and the design of momentum participation in the design of the present invention effectively solve the problem of distribution heterogeneity of audio data among different devices. In practical applications, compared with other algorithms, under the design of the present invention, the lower the data heterogeneity, the lower the decline in model performance after training. The specific experimental data are shown in Table 1 below. The top layer (1, 0.5, 0.1, 0.01, etc.) represents data heterogeneity. The lower the value, the higher the degree of data heterogeneity. Compared with other prior art algorithms, the design method of the present invention is superior to other methods when the degree of data heterogeneity is high, and can significantly improve the generalization ability of the model. The experimental verification idea based on the design of the present invention is as follows. The audio data used is LibriSpeech, which is a public data set widely used in automatic speech recognition research. It is based on the LibriVox audio book project and contains about 1,000 hours of high-quality English voice recordings with a sampling rate of 16kHz. The data set is divided into training, verification and test sets of different difficulty levels, which are often used to train and evaluate speech recognition models and develop language models. In addition, the distributed environment set in the experiment is 1 central server and 10 clients, 500 global rounds and 20 local training rounds, etc. The degree of heterogeneity of data distribution in the client is controlled by Dirichlet distribution, from 1 to 0.001, indicating a higher degree of heterogeneity. By conducting experiments under different degrees of data heterogeneity, the final model performance (i.e., the accuracy of voiceprint recognition) is obtained for comparison and verification.

[0086] Table 1

[0087]

[0088] Based on the same inventive concept, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a voiceprint recognition method based on federated learning is implemented.

[0089] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] The present invention is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0091] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0092] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0093] The above embodiments are only for illustrating the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A voiceprint recognition method based on federated learning, characterized in that: Based on the server and client set, the server initializes r=1, and initializes the model to be trained to form the model to be trained under the rth global iteration, and then performs the following steps to obtain a voiceprint recognition model for realizing the recognition of preset labels of each audio sender for the audio; Step A. The server randomly selects at least two clients from the client set as the clients to be processed in the rth global iteration, and sends the model to be trained in the rth global iteration to the clients to be processed in the rth global iteration, and then proceeds to step B; Step B. In the rth global iteration, each client to be processed is based on its local audio data sample set, connects the model memory and momentum increment, takes the audio sample as input and the audio sender corresponding to the audio sample as output, trains the model to be trained, obtains the local trained model, and uploads it to the server in combination with the corresponding prediction accuracy; then proceeds to step C; Step C. The server aggregates the parameter groups of the local trained models uploaded by each client to be processed under the r-th global iteration to obtain the global parameter group under the r-th global iteration, and then obtains the average accuracy of the prediction accuracy of the local trained models uploaded by each client to be processed under the r-th global iteration, and determines whether the average accuracy is greater than the preset global model accuracy threshold or whether the number of global iterations reaches the preset global maximum number of iterations. If so, the model to be trained is combined with the global parameter group under the r-th global iteration to form a voiceprint recognition model; otherwise, the model to be trained is combined with the global parameter group under the r-th global iteration to form a model to be trained under the r+1-th global iteration, and the value of r is updated by adding 1, and then returns to step A.

2. The voiceprint recognition method based on federated learning according to claim 1, characterized in that: In step B, each client to be processed in the rth global iteration initializes h=1 and executes the following steps: Step B1. The client to be processed randomly obtains a preset first number of samples covering all audio sender labels in the audio data sample set from its local audio data sample set including each audio sample and each audio sample corresponding to the audio sender label, and forms an initial sample subset. And based on the starting parameter set of the model to be trained The loss function corresponding to the client to be processed and the model to be trained take the audio sample as input and the audio sender corresponding to the audio sample as output to generate a local personalized starting parameter set for the model to be trained. Will As As Then proceed to step B2; Among them, 1≤i r ≤I r , I r represents the number of clients to be processed in the rth global iteration, Indicates the i-th r The initial sample subset corresponding to the clients to be processed, Indicates the i-th r The starting parameter set of the model to be trained in the pending clients, Indicates the i-th r The local personalized starting parameter set of the model to be trained in the client to be processed, Indicates the i-th r The parameter group of the model to be trained in the client to be processed corresponding to the hth local iteration; Indicates the i-th r The local personalized parameter group of the model to be trained in the client to be processed corresponding to the h-th local iteration; Step B2: The client to be processed randomly obtains a preset second number of samples covering all audio sender labels in the audio data sample set from its local audio data sample set to form a first sample subset. And according to the parameter group Contact the loss function corresponding to the client to be processed, and the model to be trained corresponding to the audio sample as input and the audio sender corresponding to the audio sample as output to obtain the parameter set Then proceed to step B3; wherein, Indicates the i-th r The first sample subset of the hth local iteration corresponds to the clients to be processed. Indicates the i-th r The parameter group of the model to be trained in the client to be processed corresponding to the h+1th local iteration; Step B3. The client to be processed is based on the parameter group Local Personalization Parameter Group The first sample subset The model memory and the preset momentum increment are linked, the loss function corresponding to the client to be processed, and the model to be trained takes the audio sample as input and the audio sender corresponding to the audio sample as output to obtain the local personalized parameter set. Then proceed to step B4; wherein, Indicates the i-th r The local personalized parameter group of the to-be-trained model in the to-be-processed client corresponding to the h+1th local iteration; Step B4: The client to be processed randomly obtains a preset third number of samples covering all audio sender labels in the audio data sample set from its local audio data sample set to form a second sample subset. At the same time, a preset fourth number of samples covering all audio sender labels in the audio data sample set are randomly obtained to form a third sample subset Then proceed to step B5; wherein, Indicates the i-th r The second sample subset of the hth local iteration corresponds to the clients to be processed. Indicates the i-th r The third sample subset of the hth local iteration corresponding to the clients to be processed; Step B5. The client to be processed receives the second sample subset. The third sample subset Parameter Group Local Personalization Parameter Group Apply the Hessian matrix to link the loss function corresponding to the client to be processed, and the model to be trained to take the audio sample as input and the audio sender corresponding to the audio sample as output, and for the parameter group Update and then go to step B6; Step B6. The client to be processed determines the parameter group Is the prediction accuracy of the model to be trained greater than the preset local prediction accuracy threshold? If yes, the parameter group The trained model is used as the local trained model of the client to be processed, and is uploaded to the server in combination with the corresponding prediction accuracy; otherwise, the value of h is updated by adding 1, and the process returns to step B2.

3. A voiceprint recognition method based on federated learning according to claim 2, characterized in that: In step B1, the client to be processed is based on the initial sample subset Based on the starting parameter set of the model to be trained The model to be trained takes the audio sample as input and the audio source corresponding to the audio sample as output, according to the following formula: Get the local personalized starting parameter set of the model to be trained Among them, α represents the preset first learning rate, represents the derivative calculation, Indicates the i-th r The loss function corresponding to the clients to be processed, Indicates the i-th r Pending clients based on parameter group The model to be trained is about the initial sample subset The loss function corresponding to the sample x in The loss result is Represents the initial sample subset The number of samples in .

4. The voiceprint recognition method based on federated learning according to claim 2, characterized in that: In step B2, the client to be processed is configured according to the parameter group. The first sample subset The model to be trained takes the audio sample as input and the audio source corresponding to the audio sample as output, according to the following formula: Get parameter group Among them, α represents the preset first learning rate, represents the derivative calculation, Indicates the i-th r The loss function corresponding to the clients to be processed, Indicates the i-th r Pending clients based on parameter group The model to be trained is about the first sample subset The loss function corresponding to sample y in The loss result is Represents the first sample subset The number of samples in .

5. The voiceprint recognition method based on federated learning according to claim 2, characterized in that: In step B3, the client to be processed is configured according to the parameter group. Local Personalization Parameter Group The first sample subset The model memory, the preset momentum increment β, and the model to be trained correspond to the audio sample as input and the audio emitter corresponding to the audio sample as output, according to the following formula: Get local personalized parameter group Among them, α represents the preset first learning rate, represents the derivative calculation, Indicates the i-th r The loss function corresponding to the clients to be processed, Indicates the i-th r Pending clients based on parameter group The model to be trained is about the first sample subset The loss function corresponding to sample y in The loss result is Represents the first sample subset The number of samples in .

6. The voiceprint recognition method based on federated learning according to claim 2, characterized in that: In step B5, the client to be processed The third sample subset Parameter Group Local Personalization Parameter Group The model to be trained takes the audio sample as input and the audio source corresponding to the audio sample as output, according to the following formula: For parameter groups Update; wherein α represents the preset first learning rate, η represents the preset second learning rate, I represents the unit matrix, represents derivative calculation, T represents transposition, Indicates the i-th r The loss function corresponding to the clients to be processed, Indicates the i-th r Pending clients based on parameter group The model to be trained is about the second sample subset The loss function corresponding to sample z in The loss result is Represents the second sample subset The number of samples in Indicates the i-th r Pending clients based on local personalized parameter groups The model to be trained is about the third sample subset The loss function corresponding to sample t in The loss result is Represents the third sample subset The number of samples in .

7. A computer device, characterized in that: The invention comprises a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a voiceprint recognition method based on federated learning is implemented.

Images