A voiceprint recognition method based on federated learning

Abstract

The present application relates to a kind of voiceprint identification methods based on federal learning, utilize the distributed characteristics of federal learning, based on the starting parameter group issued by server, local individual starting parameter group is generated by the client to be processed, again contact model memory and momentum increment, to realize the iterative training update of parameter group based on the first sample subset, second sample subset of local audio data sample set independent sampling each other, and finally contact the global aggregation mode of server, obtain voiceprint identification model, for audio realization about the identification of the label of each audio emitter of preset, design voiceprint identification method, effectively solve the data privacy protection and distribution heterogeneity problem of audio data between different devices, improve the robustness and generalization ability of voiceprint identification model in the actual application in system.

Description

TECHNICAL FIELD

[0001] The present application relates to a voiceprint recognition method based on federated learning, belonging to the field of distributed computing architecture and voiceprint recognition. BACKGROUND

[0002] Voiceprint recognition, as a biometric identification technology, has a wide application prospect. However, due to the privacy protection and data island problems of voiceprint data, traditional centralized learning methods are difficult to meet the actual needs. Federated learning effectively solves the problems of data privacy protection and data island by training the model locally on multiple clients and aggregating the global model. However, due to the diversity and distribution heterogeneity of audio data, traditional federated learning methods face many challenges in processing voiceprint recognition tasks. The characteristics of audio data include language or dialect differences, recording content diversity, environmental noise, device performance, etc. The heterogeneity of these characteristics will have a negative impact on traditional federated learning models, making it difficult for the model to effectively process and learn these complex audio features during local training and global aggregation. Therefore, the voiceprint recognition system based on traditional federated learning often performs poorly when facing highly statistically heterogeneous audio data, and it is difficult to achieve ideal recognition performance. In traditional federated learning, each client usually uses a fixed task sampling strategy, ignoring the differences between different tasks. SUMMARY

[0003] The technical problem to be solved by the present application is to provide a voiceprint recognition method based on federated learning, which utilizes the distributed characteristics of federated learning to effectively solve the problems of data privacy protection and distribution heterogeneity of audio data between different devices, and improve the robustness and generalization ability of the voiceprint recognition model in the system in practical applications.

[0004] The present application adopts the following technical solution to solve the above technical problems: The present application designs a voiceprint recognition method based on federated learning, based on a server and a client set, the server initializes r=1 and initializes a to-be-trained model to form a to-be-trained model under the rth global iteration, and then executes the following steps to obtain a voiceprint recognition model for identifying the pre-set labels of each audio emitter for audio.

[0005] Step A. The server randomly selects at least two clients from the client set as each to-be-processed client under the rth global iteration, and the server downloads the to-be-trained model under the rth global iteration to each to-be-processed client under the rth global iteration, and then enters step B.

[0006] Step B. In the rth global iteration, each client to be processed uses its local audio data sample set, connects the model memory and momentum increment, uses the audio sample as input and the audio source corresponding to the audio sample as output, trains the model to be trained, obtains the local trained model, and uploads it to the server based on the corresponding prediction accuracy; then proceeds to Step C;

[0007] Step C. The server aggregates the parameter groups of the local trained models uploaded by each client to be processed under the r-th global iteration to obtain the global parameter group under the r-th global iteration, and then obtains the average accuracy of the prediction accuracy of the local trained models uploaded by each client to be processed under the r-th global iteration, and determines whether the average accuracy is greater than the preset global model accuracy threshold or whether the number of global iterations reaches the preset global maximum number of iterations. If so, the model to be trained is combined with the global parameter group under the r-th global iteration to form a voiceprint recognition model; otherwise, the model to be trained is combined with the global parameter group under the r-th global iteration to form a model to be trained under the r+1-th global iteration, and the value of r is updated by adding 1, and then returns to step A.

[0008] As a preferred technical solution of the present invention: in step B, each client to be processed in the rth global iteration initializes h=1 and performs the following steps:

[0009] Step B1. The client to be processed randomly obtains a preset first number of samples covering all audio sender labels in the audio data sample set from its local audio data sample set including each audio sample and each audio sample corresponding to the audio sender label, and forms an initial sample subset. And based on the starting parameter set of the model to be trained Contact the loss function corresponding to the client to be processed, and the model to be trained corresponding to the audio sample as input and the audio sender corresponding to the audio sample as output, to generate the local personalized starting parameter set of the model to be trained Will As As Then proceed to step B2;

[0010] Where 1≤i r ≤I r , I r represents the number of clients to be processed in the rth global iteration, Indicates the i-th global iteration r The initial sample subset corresponding to the clients to be processed, Indicates the i-th global iteration rThe starting parameter set of the model to be trained in the pending clients, Indicates the i-th global iteration r The local personalized starting parameter set of the model to be trained in the client to be processed, Indicates the i-th global iteration r The parameter group of the model to be trained in the client to be processed corresponding to the hth local iteration; Indicates the i-th global iteration r The local personalized parameter group of the to-be-trained model in the to-be-processed client corresponding to the hth local iteration;

[0011] Step B2: The client to be processed randomly obtains a preset second number of samples from its local audio data sample set, and covers all audio sender labels in the audio data sample set, to form a first sample subset. And according to the parameter group Contact the loss function corresponding to the client to be processed, and the model to be trained corresponding to the audio sample as input and the audio emitter corresponding to the audio sample as output to obtain the parameter set Then proceed to step B3; wherein, Indicates the i-th global iteration r The clients to be processed correspond to the first sample subset of the hth local iteration, Indicates the i-th global iteration r The parameter group of the model to be trained in the client to be processed corresponding to the h+1th local iteration;

[0012] Step B3. The client to be processed is based on the parameter group Local personalized parameter group The first sample subset The model memory and the preset momentum increment are linked, as well as the loss function corresponding to the client to be processed, and the model to be trained takes the audio sample as input and the audio sender corresponding to the audio sample as output to obtain the local personalized parameter set. Then proceed to step B4; wherein, Indicates the i-th global iteration r The local personalized parameter group of the to-be-trained model in the to-be-processed client corresponding to the h+1th local iteration;

[0013] Step B4. The client to be processed randomly obtains a preset third number of samples from its local audio data sample set, and covers all audio sender labels in the audio data sample set, to form a second sample subset. At the same time, a preset fourth number of samples covering all audio sender labels in the audio data sample set are randomly obtained to form a third sample subset. Then proceed to step B5; wherein, Indicates the i-th global iteration r The clients to be processed correspond to the second sample subset of the hth local iteration, Indicates the i-th global iteration r The third sample subset of the hth local iteration corresponding to the clients to be processed;

[0014] Step B5. The client to be processed receives the second sample subset. The third sample subset Parameter Group Local personalized parameter group Apply the Hessian matrix to connect the loss function corresponding to the client to be processed, and the model to be trained to take the audio sample as input and the audio sender corresponding to the audio sample as output, for the parameter group Update and then go to step B6;

[0015] Step B6. The client to be processed determines the parameter group Is the prediction accuracy of the model to be trained greater than the preset local prediction accuracy threshold? If so, the parameter group The trained model is used as the local trained model of the client to be processed, and is uploaded to the server in combination with the corresponding prediction accuracy; otherwise, the value of h is updated by adding 1, and the process returns to step B2.

[0016] As a preferred technical solution of the present invention: in step B1, the client to be processed is based on the initial sample subset Based on the starting parameter set of the model to be trained The model to be trained takes the audio sample as input and the audio source corresponding to the audio sample as output, according to the following formula:

[0017]

[0018] Get the local personalized starting parameter set of the model to be trained Among them, α represents the preset first learning rate, represents the derivative calculation, Indicates the i-th global iteration r The loss function corresponding to the clients to be processed, Indicates the i-th global iteration r Pending clients based on parameter groups The model to be trained is about the initial sample subset The loss function corresponding to the sample x in The loss result is Represents the initial sample subset The number of samples in .

[0019] As a preferred technical solution of the present invention: in step B2, the client to be processed is configured according to the parameter group The first sample subset The model to be trained takes the audio sample as input and the audio source corresponding to the audio sample as output, according to the following formula:

[0020]

[0021] Get parameter group Among them, α represents the preset first learning rate, represents the derivative calculation, Indicates the i-th global iteration r The loss function corresponding to the clients to be processed, Indicates the i-th global iteration r Pending clients based on parameter groups The model to be trained is about the first sample subset The loss function corresponding to the sample y in The loss result is Represents the first sample subset The number of samples in .

[0022] As a preferred technical solution of the present invention: in step B3, the client to be processed is configured according to the parameter group Local personalized parameter group The first sample subset The connection model memory, the preset momentum increment β, and the corresponding audio sample to be trained are input and the audio source corresponding to the audio sample are output, according to the following formula:

[0023]

[0024] Get local personalized parameter group Among them, α represents the preset first learning rate, represents the derivative calculation, Indicates the i-th global iteration r The loss function corresponding to the clients to be processed, Indicates the i-th global iteration r Pending clients based on parameter groups The model to be trained is about the first sample subset The loss function corresponding to the sample y in The loss result is Represents the first sample subset The number of samples in .

[0025] As a preferred technical solution of the present invention: in step B5, the client to be processed is based on the second sample subset The third sample subset Parameter Group Local personalized parameter group The model to be trained takes the audio sample as input and the audio source corresponding to the audio sample as output, according to the following formula:

[0026]

[0027] For parameter groups Update; where α represents the preset first learning rate, η represents the preset second learning rate, and I represents the unit matrix. Indicates derivative calculation, T indicates transposition, Indicates the i-th global iteration r The loss function corresponding to the clients to be processed, Indicates the i-th global iteration r Pending clients based on parameter groups The model to be trained is about the second sample subset The loss function corresponding to sample z in The loss result is Represents the second sample subset The number of samples in Indicates the i-th global iteration r Pending clients based on local personalized parameter groups The model to be trained is about the third sample subset The loss function corresponding to sample t in The loss result is Represents the third sample subset The number of samples in .

[0028] In response to the above design scheme, the present invention further designs a computer device, which is characterized by comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and implementing a voiceprint recognition method based on federated learning when the processor executes the computer program.

[0029] The voiceprint recognition method based on federated learning described in the present invention, which adopts the above technical solution, has the following technical effects compared with the existing technology:

[0030] (1) The present invention designs a voiceprint recognition method based on federated learning, which utilizes the distributed characteristics of federated learning. Based on the starting parameter group sent by the server, the client to be processed generates a local personalized starting parameter group, and then links the model memory and momentum increment to realize iterative training and updating of the parameter group based on the first sample subset and the second sample subset sampled independently from each other based on the local audio data sample set, and finally links the global aggregation method of the server to obtain a voiceprint recognition model for realizing the recognition of the preset labels of each audio sender for audio; the voiceprint recognition method is designed to effectively solve the data privacy protection and distribution heterogeneity problems of audio data between different devices, and improve the robustness and generalization ability of the voiceprint recognition model in the system in practical applications;

[0031] (2) In the voiceprint recognition method based on federated learning designed by the present invention, by introducing a model memory mechanism, the personalized model gradient of each task is tracked and estimated, thereby enhancing the adaptability of the model to local data features, making the design highly adaptable when dealing with heterogeneous data distribution; and in combination with the momentum increment strategy, the momentum factor is used to smooth the model update, avoiding the model from falling into the local optimal solution, improving the stability and generalization ability of the model, and converging towards the optimal solution faster, while maintaining the high stability of the internal model of the federated system;

[0032] (3) In the voiceprint recognition method based on federated learning designed by the present invention, three independent sampling data sets are used to calculate the gradient estimation, which effectively reduces the estimation variance and improves the accuracy of the gradient estimation. Model agnostic meta-learning (MAML) is introduced for local training of the client, focusing on learning deep feature knowledge, especially processing complex voiceprint features. This combination makes the local model in the system have an advantage in the initial convergence speed, and can quickly achieve a high accuracy rate, saving time and resources; and gradient incremental balance is introduced. By balancing the mixed gradient of sample initialization gradient, random meta-learning gradient and federated aggregation global model, overfitting and catastrophic forgetting are effectively prevented, so that the designed voiceprint recognition method shows strong robustness, and can still maintain high performance and robustness in highly heterogeneous environments, significantly improving the adaptability and generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1-1 This is a schematic diagram of the client process in the voiceprint recognition method based on federated learning designed by the present invention;

[0034] Figure 1-2 This is a flow chart of the server in the voiceprint recognition method based on federated learning designed by the present invention;

[0035] Figure 2 The following is a flowchart illustrating a simple application of the system of the present invention. DETAILED DESCRIPTION

[0036] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0037] The present invention designs a voiceprint recognition method based on federated learning, such as Figure 2 As shown, based on a set of servers and clients (such as mobile phones, microphones, computers and other devices), the server initializes r=1, and initializes the model to be trained to form the model to be trained under the rth global iteration, and then performs the following steps to obtain a voiceprint recognition model, which is used to realize the recognition of preset labels of each audio sender for audio, solve the problem that each device in the client set has significant heterogeneity in sound quality, noise, and audio signal lights due to performance and brand reasons, and train a voiceprint recognition model of a time delay neural network (TDNN) combined with a channel attention mechanism in an environment with distribution heterogeneity.

[0038] Step A. Figure 1-2 As shown, the server randomly selects at least two clients from the client set as the clients to be processed under the r-th global iteration, and sends the to-be-trained model under the r-th global iteration to the clients to be processed under the r-th global iteration, and then enters step B.

[0039] Step B. Under the rth global iteration, each client to be processed is based on its local audio data sample set, connects the model memory and momentum increment, takes the audio sample as input and the audio emitter corresponding to the audio sample as output, trains the model to be trained, obtains the local trained model, and uploads it to the server in combination with the corresponding prediction accuracy; then proceeds to step C.

[0040] In practical application, the above step B is as follows: Figure 1-1 As shown, in the rth global iteration, each client to be processed initializes h=1 and executes the following steps B1 to B6.

[0041] Step B1. The client to be processed randomly obtains a preset first number of samples covering all audio sender labels in the audio data sample set from its local audio data sample set including each audio sample and each audio sample corresponding to the audio sender label, and forms an initial sample subset. And based on the starting parameter set of the model to be trained The loss function corresponding to the client to be processed, such as the cross entropy loss function SGD, and the model to be trained corresponding to the audio sample as input and the audio source corresponding to the audio sample as output, are as follows:

[0042]

[0043] Get the local personalized starting parameter set of the model to be trained Will As As Then go to step B2. Initial sample subset after random sampling For model initialization and gradient calculation, through the initial sample subset Calculate the gradient of the loss function to make preliminary adjustments when the model is initialized, providing a solid foundation for the subsequent formal local training; where 1≤i r ≤I r , I r represents the number of clients to be processed in the rth global iteration, Indicates the i-th global iteration r The initial sample subset corresponding to the clients to be processed, Indicates the i-th global iteration r The starting parameter set of the model to be trained in the pending clients, Indicates the i-th global iteration r The local personalized starting parameter set of the model to be trained in the client to be processed, Indicates the i-th global iteration r The parameter group of the model to be trained in the client to be processed corresponding to the hth local iteration; Indicates the i-th global iteration r The model to be trained in each client to be processed corresponds to the local personalized parameter group of the hth local iteration; it determines the size of each update step. An appropriate learning rate can ensure that the model converges quickly to the optimal solution, avoiding instability caused by too large a step size or slow convergence caused by too small a step size. represents the derivative calculation, Indicates the i-th global iteration r The loss function corresponding to the clients to be processed, Indicates the i-th global iteration r Pending clients based on parameter groups The model to be trained is about the initial sample subset The loss function corresponding to the sample x in The loss result is Represents the initial sample subset The number of samples in .

[0044] Step B2: The client to be processed randomly obtains a preset second number of samples from its local audio data sample set, and covers all audio sender labels in the audio data sample set, to form a first sample subset. And according to the parameter group The loss function corresponding to the client to be processed and the model to be trained corresponding to the audio sample as input and the audio source corresponding to the audio sample as output are as follows:

[0045]

[0046] Get parameter group Then proceed to step B3; wherein, Indicates the i-th global iteration r The clients to be processed correspond to the first sample subset of the hth local iteration, Indicates the i-th global iteration r The parameter group of the model to be trained in the client to be processed corresponding to the h+1th local iteration; Indicates the i-th global iteration r Pending clients based on parameter groups The model to be trained is about the first sample subset The loss function corresponding to the sample y in The loss result is Represents the first sample subset The number of samples in .

[0047] The above initial sample subset The first sample subset In practical applications, a sample subset is selected by uniform random sampling. The sampling size ratio can be set according to the client computing power and data scale.

[0048]

[0049] in, Indicates the i-th global iteration r The audio data sample set of the client to be processed is K, which is the number or proportion of samples. This sample subset covers all the audio sender labels in the audio data sample set, fully reflecting the diversity of the local data. According to statistical theory, small samples can reflect the characteristics of the entire dataset. If the sample size is sufficient, statistical indicators such as mean and variance will tend to the true value of the overall data. When the sample size is large enough, the mean of the randomly selected sample is close to the overall mean.

[0050] Step B3. In order to improve the adaptability of the model to each client's specific data, the traditional gradient descent and momentum update strategies are combined to ensure rapid convergence while greatly enhancing the stability and robustness of the model. The client to be processed is based on the parameter group. Local personalized parameter group The first sample subset Relating the model memory and the preset momentum increment β, the loss function corresponding to the client to be processed, and the model to be trained corresponding to the audio sample as input and the audio source corresponding to the audio sample as output, the following formula is used:

[0051]

[0052] Get local personalized parameter group Then proceed to step B4; wherein the momentum increment β is used to control the influence of the current iteration momentum on the next iteration update, Indicates the i-th global iteration r The local personalized parameter group of the model to be trained in the client to be processed corresponds to the h+1th local iteration, Indicates the i-th global iteration r Pending clients based on parameter groups The model to be trained is about the first sample subset The loss function corresponding to the sample y in The loss result is Represents the first sample subset The number of samples in .

[0053] It is an auxiliary vector used to update the model parameters. It can "remember" the previous gradient direction during the update, so that it can maintain a certain update speed when encountering saddle points or flat areas. Beta binding via momentum increment Information and current gradient information The introduction of the β value can smooth the update path during gradient descent, avoid violent fluctuations, and accelerate convergence. A larger β value indicates that the previous generation's momentum has a greater influence on the current momentum, and the model is more likely to follow the previous gradient direction when updating; conversely, the current gradient has a greater influence, and the update direction is more dependent on the current gradient information.

[0054] Step B4. The client to be processed randomly obtains a preset third number of samples from its local audio data sample set, and covers all audio sender labels in the audio data sample set, to form a second sample subset. At the same time, a preset fourth number of samples covering all audio sender labels in the audio data sample set are randomly obtained to form a third sample subset. Then proceed to step B5; wherein, represents the second sample subset of the h-th local iteration corresponding to the ir-th client to be processed in the r-th global iteration, Indicates the i-th global iteration rThe clients to be processed correspond to the third sample subset of the hth local iteration.

[0055] In the actual implementation of the above design, the second sample subset With the third sample subset Will be used to calculate the Hessian matrix and gradient estimation respectively, where the second sample subset The sample set used to calculate the Hessian matrix is ​​randomly sampled from the client local data set To ensure that the sample set can reflect the second-order statistical characteristics of the local data set.

[0056]

[0057] Among them, K2 is the number or proportion of samples. When selecting, the number of samples required to calculate the Hessian matrix should be considered. According to the different environments and data deployed in this patent, K2 is adjusted so that this sample set should contain enough data points to accurately estimate the curvature of the loss function.

[0058] The third sample subset The sample set used to calculate the gradient and the second sample subset The same, both are obtained by random sampling from the client local dataset D i It is selected from , which is used to reflect the first-order statistical characteristics of the local data set.

[0059]

[0060] Where K3 is the sample size or proportion, The size of can be determined according to the number of samples and data environment required to calculate the gradient, ensuring the accuracy of the gradient estimation.

[0061] Second sample subset Will be used to calculate the Hessian moment The Hessian matrix is ​​the loss function Model parameters The second-order derivative matrix of , which describes the curvature information of the loss function in the parameter space. Specifically, the calculation process of the Hessian matrix is ​​as follows:

[0062]

[0063] In order to take into account the diversity and statistical characteristics of the data, we The second-order derivative of each sample z in is calculated and the average of these second-order derivatives is taken:

[0064]

[0065] The above formula is based on the second-order derivative of the loss function with respect to the model parameters, which are calculated from the Jacobian of the gradient (that is, the derivative matrix of the first-order derivative vector). Perform gradient transposition and derive again to obtain the second-order derivative of the loss function with respect to the model parameters The formula is as follows:

[0066]

[0067] Substitute this into the definition of the original Hessian:

[0068]

[0069] After the above derivation, the Hessian matrix In fact, the sample set The average value of the second derivative of each sample in , that is:

[0070]

[0071] Finally, the above formula shows how to Calculate the Hessian matrix for each sample in and average them to get the overall second-order derivative of the loss function.

[0072] The third sample subset Used to calculate the loss function Relative to model parameters The first-order derivative (i.e. gradient) of It is one of the core parts of the optimization process. Through this gradient, the model can know in which direction to adjust and better fit the data. The gradient of the first-order derivative of the model parameters to guide the correction matrix.

[0073] According to the above and Calculate the final gradient update direction The specific formula is as follows:

[0074]

[0075] in, Represents the i-th global iteration r The gradient update direction of the model to be trained in the client to be processed corresponds to the h+1th local iteration, I is the identity matrix, which is usually the same as the Hessian matrix The diagonal matrix of the same dimension plays the role of maintaining the original gradient direction in matrix multiplication. α represents the preset first learning rate, which is used to control the influence of the Hessian matrix on the gradient correction. The learning rate determines the extent to which the Hessian matrix changes the gradient update direction. Correction matrix The purpose is to adjust the direction and size of the original gradient. The role of this matrix is ​​to correct the gradient direction by introducing the Hessian matrix, making the gradient update more robust and accurate, especially when the loss function surface has a complex geometric structure. In summary, the new gradient update direction is obtained by the correction matrix of the Hessian matrix and the gradient update.

[0076] Step B5. The client to be processed receives the second sample subset. The third sample subset Parameter Group Local personalized parameter group Apply the Hessian matrix to connect the loss function corresponding to the client to be processed, and the model to be trained corresponding to the audio sample as input and the audio source corresponding to the audio sample as output, according to the following formula:

[0077]

[0078] For parameter groups Update, and then enter step B6, where η represents the preset second learning rate, I represents the identity matrix, and T represents transposition. Indicates the i-th global iteration r Pending clients based on parameter groups The model to be trained is about the second sample subset The loss function corresponding to sample z in The loss result is Represents the second sample subset The number of samples in Indicates the i-th global iteration r Pending clients based on local personalized parameter groups The model to be trained is about the third sample subset The loss function corresponding to sample t in The loss result is Represents the third sample subset The number of samples in .

[0079] Step B6. The client to be processed determines the parameter group Is the prediction accuracy of the model to be trained greater than the preset local prediction accuracy threshold? If so, the parameter group The trained model is used as the local trained model of the client to be processed, and is uploaded to the server in combination with the corresponding prediction accuracy; otherwise, the value of h is updated by adding 1, and the process returns to step B2.

[0080] Step C. Figure 1-2 As shown, the server aggregates the parameter groups of the local trained models uploaded by each client to be processed under the r-th global iteration to obtain the global parameter group under the r-th global iteration, and then obtains the average accuracy of the prediction accuracy of the local trained models uploaded by each client to be processed under the r-th global iteration, and determines whether the average accuracy is greater than the preset global model accuracy threshold or whether the number of global iterations reaches the preset global maximum number of iterations. If so, the model to be trained is combined with the global parameter group under the r-th global iteration to form a voiceprint recognition model; otherwise, the model to be trained is combined with the global parameter group under the r-th global iteration to form a model to be trained under the r+1-th global iteration, and the value of r is updated by adding 1, and then returns to step A.

[0081] The practical application of the above technical solution utilizes the distributed characteristics of federated learning. Based on the starting parameter group sent by the server, the client to be processed generates a local personalized starting parameter group, and then links the model memory and momentum increment to realize iterative training and update of the parameter group based on the first sample subset and the second sample subset independently sampled from the local audio data sample set, and finally links the global aggregation method of the server to obtain a voiceprint recognition model for realizing the recognition of the preset labels of each audio sender for audio; designs a voiceprint recognition method to effectively solve the data privacy protection problem of audio data between different devices, and improve the robustness and generalization ability of the voiceprint recognition model in the system in practical applications.

[0082] Here, based on the design of sampling sample subsets proportionally from the local audio data sample set, we compare the direct training with the entire local audio data sample set. When the local audio data is highly heterogeneous, if the training is performed with the entire local audio data sample (sampling ratio is 1), the performance of the trained model is lower than the performance of the model trained with the sample subset under small sampling ratio. The comparison of experimental data shows a decrease of about 8%. Among them, the accuracy of the model trained with the entire local audio data sample is 65.09%, and the accuracy of the model trained with the sample subset with a small ratio of 0.3 is 71.0 7%; in addition, in the implementation of the design scheme of the present invention, momentum increment is also introduced to participate in training. Under actual sample verification, the accuracy of the model after training without momentum participation is 67.55%, and the accuracy of the model after training with momentum participation is about 71.11%. Moreover, when the data heterogeneity is not strong (the data is evenly distributed), the training without momentum may fall into the local optimal solution, which is not as stable as the effect of training with momentum increment participation. Therefore, the application of a small proportion of sample subsets and the design of momentum participation in the design scheme of the present invention effectively solve the problem of distribution heterogeneity of audio data among different devices.

[0083] By introducing a model memory mechanism, the personalized model gradient of each task is tracked and estimated, thereby enhancing the model's adaptability to local data features, making the design highly adaptable when dealing with heterogeneous data distributions. In combination with a momentum increment strategy, the momentum factor is used to smooth model updates, preventing the model from falling into a local optimal solution. This improves the model's stability and generalization ability, allowing it to converge toward the optimal solution more quickly while maintaining the high stability of the model within the federated system.

[0084] Moreover, three independent sampling data sets are used to calculate the gradient estimation, which effectively reduces the estimation variance and improves the accuracy of the gradient estimation. Model agnostic meta-learning (MAML) is introduced for local training of the client, focusing on learning deep feature knowledge, especially processing complex voiceprint features. This combination makes the local model in the system have an advantage in initial convergence speed, and can quickly achieve a high accuracy rate, saving time and resources; and gradient incremental balance is introduced. By balancing the mixed gradient of sample initialization gradient, random meta-learning gradient and federated aggregation global model, overfitting and catastrophic forgetting are effectively prevented, so that the designed voiceprint recognition method shows strong robustness, and can still maintain high performance and robustness in highly heterogeneous environments, significantly improving the adaptability of the model.

[0085] The use of a small sample subset and momentum-based design in the design of the present invention effectively address the problem of heterogeneous distribution of audio data across different devices. In practical applications, compared with other algorithms, the lower the data heterogeneity, the smaller the decline in model performance after training. Specific experimental data are shown in Table 1 below. The top layer (1, 0.5, 0.1, 0.01, etc.) represents data heterogeneity, with lower values ​​indicating higher data heterogeneity. Compared with other existing algorithms, the design method of the present invention outperforms other methods when data heterogeneity is high, significantly improving the model's generalization ability. The experimental verification strategy based on the design of the present invention is as follows. The audio data used is LibriSpeech, a public dataset widely used in automatic speech recognition research. Based on the LibriVox audiobook project, it contains approximately 1000 hours of high-quality English speech recordings with a sampling rate of 16kHz. This dataset is divided into training, validation, and test sets of varying difficulty and is commonly used for training and evaluating speech recognition models and developing language models. Furthermore, the distributed environment used in the experiment consisted of one central server and ten clients, with 500 global and 20 local training rounds. Dirichlet distribution was used to control the heterogeneity of data distribution among the clients, with values ​​ranging from 1 to 0.001 indicating higher heterogeneity. By conducting experiments under varying degrees of data heterogeneity, the final model performance (i.e., voiceprint recognition accuracy) was derived for comparison and verification.

[0086] Table 1

[0087]

[0088] Based on the same inventive concept, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a voiceprint recognition method based on federated learning is implemented.

[0089] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0091] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0092] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0093] The above embodiments are only for illustrating the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A voiceprint recognition method based on federated learning, characterized by: Based on the server and client set, the server initializes r = 1 and initializes the model to be trained to form the model to be trained under the rth global iteration, and then performs the following steps to obtain a voiceprint recognition model for realizing the recognition of the preset labels of each audio sender for the audio; Step A. The server randomly selects at least two clients from the client set as the clients to be processed in the rth global iteration, and sends the to-be-trained model in the rth global iteration to each client to be processed in the rth global iteration, and then proceeds to step B; Step B. In the rth global iteration, each client to be processed uses its local audio data sample set, connects the model memory and momentum increment, uses the audio sample as input and the audio source corresponding to the audio sample as output, trains the model to be trained, obtains the local trained model, and uploads it to the server based on the corresponding prediction accuracy; then proceeds to Step C; In step B, each client to be processed in the rth global iteration initializes h = 1 and performs the following steps: Step B1. The client to be processed randomly obtains a preset first number of samples covering all audio sender labels in the audio data sample set from its local audio data sample set including each audio sample and each audio sample corresponding to the audio sender label, and forms an initial sample subset. And based on the starting parameter set of the model to be trained Contact the loss function corresponding to the client to be processed, and the model to be trained corresponding to the audio sample as input and the audio sender corresponding to the audio sample as output, to generate the local personalized starting parameter set of the model to be trained Will As As Then proceed to step B2; Where 1≤i r ≤I r , I r represents the number of clients to be processed in the rth global iteration, Indicates the i-th global iteration r The initial sample subset corresponding to the clients to be processed, Indicates the i-th global iteration r The starting parameter set of the model to be trained in the pending clients, Indicates the i-th global iteration r The local personalized starting parameter set of the model to be trained in the client to be processed, Indicates the i-th global iteration r The parameter group of the model to be trained in the client to be processed corresponding to the hth local iteration; Indicates the i-th global iteration r The local personalized parameter group of the to-be-trained model in the to-be-processed client corresponding to the hth local iteration; Step B2: The client to be processed randomly obtains a preset second number of samples from its local audio data sample set, and covers all audio sender labels in the audio data sample set, to form a first sample subset. And according to the parameter group Contact the loss function corresponding to the client to be processed, and the model to be trained corresponding to the audio sample as input and the audio emitter corresponding to the audio sample as output to obtain the parameter set Then proceed to step B3; wherein, Indicates the i-th global iteration r The clients to be processed correspond to the first sample subset of the hth local iteration, Indicates the i-th global iteration r The parameter group of the model to be trained in the client to be processed corresponding to the h+1th local iteration; Step B3. The client to be processed is based on the parameter group Local personalized parameter group First sample subset The model memory and the preset momentum increment are linked, as well as the loss function corresponding to the client to be processed, and the model to be trained takes the audio sample as input and the audio sender corresponding to the audio sample as output to obtain the local personalized parameter set. Then proceed to step B4; wherein, Indicates the i-th global iteration r The local personalized parameter group of the to-be-trained model in the to-be-processed client corresponding to the h+1th local iteration; Step B4. The client to be processed randomly obtains a preset third number of samples from its local audio data sample set, and covers all audio sender labels in the audio data sample set, to form a second sample subset. At the same time, a preset fourth number of samples covering all audio sender labels in the audio data sample set are randomly obtained to form a third sample subset. Then proceed to step B5; wherein, Indicates the i-th global iteration r The clients to be processed correspond to the second sample subset of the hth local iteration, Indicates the i-th global iteration r The third sample subset of the hth local iteration corresponding to the clients to be processed; Step B5. The client to be processed receives the second sample subset. The third sample subset Parameter Group Local personalized parameter group Apply the Hessian matrix to connect the loss function corresponding to the client to be processed, and the model to be trained to take the audio sample as input and the audio sender corresponding to the audio sample as output, for the parameter group Update and then go to step B6; Step B6. The client to be processed determines the parameter group Is the prediction accuracy of the model to be trained greater than the preset local prediction accuracy threshold? If so, the parameter group The trained model is used as the local trained model of the client to be processed, and is uploaded to the server in combination with the corresponding prediction accuracy; otherwise, the value of h is updated by adding 1, and the process returns to step B2; Step C. The server aggregates the parameter groups of the local trained models uploaded by each client to be processed under the r-th global iteration to obtain the global parameter group under the r-th global iteration, and then obtains the average accuracy of the prediction accuracy of the local trained models uploaded by each client to be processed under the r-th global iteration, and determines whether the average accuracy is greater than the preset global model accuracy threshold or whether the number of global iterations reaches the preset global maximum number of iterations. If so, the model to be trained is combined with the global parameter group under the r-th global iteration to form a voiceprint recognition model; otherwise, the model to be trained is combined with the global parameter group under the r-th global iteration to form a model to be trained under the r+1-th global iteration, and the value of r is updated by adding 1, and then returns to step A.

2. The voiceprint recognition method based on federated learning according to claim 1, characterized in that: In step B1, the client to be processed is based on the initial sample subset Based on the starting parameter set of the model to be trained The model to be trained takes the audio sample as input and the audio source corresponding to the audio sample as output, according to the following formula: Get the local personalized starting parameter set of the model to be trained Among them, α represents the preset first learning rate, Δ represents the derivative calculation, Indicates the i-th global iteration r The loss function corresponding to the client to be processed, Indicates the i-th global iteration r Pending clients based on parameter groups The model to be trained is about the initial sample subset The loss function corresponding to the sample x in The loss result is Represents the initial sample subset The number of samples in .

3. The voiceprint recognition method based on federated learning according to claim 1, characterized in that: In step B2, the client to be processed is configured according to the parameter group. First sample subset The model to be trained takes the audio sample as input and the audio source corresponding to the audio sample as output, according to the following formula: Get parameter group Among them, α represents the preset first learning rate, represents the derivative calculation, Indicates the i-th global iteration r The loss function corresponding to the client to be processed, Indicates the i-th global iteration r Pending clients based on parameter groups The model to be trained is about the first sample subset The loss function corresponding to the sample y in The loss result is Represents the first sample subset The number of samples in .

4. The voiceprint recognition method based on federated learning according to claim 1, characterized in that: In step B3, the client to be processed is configured according to the parameter group. Local personalized parameter group First sample subset The connection model memory, the preset momentum increment β, and the corresponding audio sample to be trained are input and the audio source corresponding to the audio sample are output, according to the following formula: Get local personalized parameter group Among them, α represents the preset first learning rate, represents the derivative calculation, Indicates the i-th global iteration r The loss function corresponding to the client to be processed, Indicates the i-th global iteration r Pending clients based on parameter groups The model to be trained is about the first sample subset The loss function corresponding to the sample y in The loss result is Represents the first sample subset The number of samples in .

5. The voiceprint recognition method based on federated learning according to claim 1, characterized in that: In step B5, the client to be processed is based on the second sample subset The third sample subset Parameter Group Local personalized parameter group The model to be trained takes the audio sample as input and the audio source corresponding to the audio sample as output, according to the following formula: For parameter groups Update; where α represents the preset first learning rate, η represents the preset second learning rate, and I represents the unit matrix. Indicates derivative calculation, T indicates transposition, Indicates the i-th global iteration r The loss function corresponding to the client to be processed, Indicates the i-th global iteration r Pending clients based on parameter groups The model to be trained is about the second sample subset The loss function corresponding to sample z in The loss result is Represents the second sample subset The number of samples in Indicates the i-th global iteration r Pending clients based on local personalized parameter groups The model to be trained is about the third sample subset The loss function corresponding to sample t in The loss result is Represents the third sample subset The number of samples in .

6. A computer device, characterized in that: The invention comprises a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a voiceprint recognition method based on federated learning is implemented.

Images