A personalized federated learning method and system for speech recognition models

By conducting supervised and semi-supervised training in speech recognition federated learning, combining model decoupling optimization of feature extractors and classifiers, a personalized speech recognition model is generated, which solves the problem of speech data distribution differences, improves model performance and reduces Error recognition rate.

CN114783443BActive Publication Date: 2024-11-01INST OF ACOUSTICS CHINESE ACAD OF SCI
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202210325453.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-30
Publication Date
2024-11-01
Estimated Expiration
2042-03-30

AI Technical Summary

Technical Problem

The existing federated learning method for speech recognition fails to effectively consider the differences in speech data distribution of each client, resulting in poor performance of speech recognition models and high error recognition rates.

Method used

By passing non-raw data between the center and the client, using local annotated speech samples for supervised training, the seed model of the speech recognition model is obtained, and semi-supervised training is performed on the client, combining feature extractors and classifiers to perform models Decoupling optimization to generate a personalized speech recognition model for multiple clients.

Benefits of technology

It effectively improves the performance of the speech recognition model, reduces the error recognition rate, solves the problem of voice data distribution differences, and ensures the security of data information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114783443B_ABST
    Figure CN114783443B_ABST
Patent Text Reader

Abstract

The present application proposes a personalized federated learning method and system for a speech recognition model, which is applied to a central end and multiple clients. At the central end, the method includes: using local annotated speech samples to perform supervised training on the speech recognition model to obtain a seed model of the speech recognition model, and the seed model includes a feature extractor and a classifier; the feature extractor is used to process the speech information input by multiple clients to determine the personalized features of the speech information of each of the multiple clients; using the personalized features of the speech information of each of the multiple clients to perform semi-supervised training on the classifier to obtain a trained classifier; and transmitting the trained classifier to each of the multiple clients. The present application realizes personalized federated learning by decoupling and optimizing the feature extractor and classifier of the speech recognition model, effectively improving the performance of the speech recognition model and reducing the error recognition rate.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of speech recognition, and specifically to a personalized federated learning method and system for a speech recognition model. Background Art

[0002] The speech recognition model can realize the function of converting speech into text. The traditional centralized intelligent speech system needs to aggregate the speech samples of all clients to the central end for centralized training. However, in the centralized training paradigm, since the voice samples of customers need to be transmitted to the central end, this process may cause data leakage and lead to threats to data information security.

[0003] Federated learning methods use all client voice samples for learning by transferring non-original data between the client and the central end without transmitting the voice samples to the central end. In real-world scenarios, the distribution of voice data of each client is usually different, but existing federated learning methods for speech recognition usually do not consider the distribution differences of each client. Summary of the invention

[0004] The purpose of the embodiments of the present application is to effectively improve the performance of the speech recognition model to reduce the error recognition rate for the personalized speech of multiple clients.

[0005] To achieve the above-mentioned objectives, in a first aspect, an embodiment of the present application provides a personalized federated learning method for a speech recognition model, which is applied to a central end and is characterized in that it includes: using local annotated speech samples to perform supervised training on a speech recognition model to obtain a seed model of the speech recognition model, and the seed model includes a feature extractor and a classifier; the feature extractor is used to process speech information input by multiple clients, and determine the personalized features of the speech information of each of the multiple clients; the classifier is used to convert the speech information into text according to the personalized features; using the personalized features of the speech information of each of the multiple clients to perform semi-supervised training on the classifier to obtain a trained classifier; and transmitting the trained classifier to each of the multiple clients.

[0006] As a feasible implementation method, the supervised training of the speech recognition model using the local annotated speech samples includes: taking the annotated speech samples as input and taking the first loss function as the learning target, and performing supervised training on the speech recognition model, wherein the first loss function is:

[0007]

[0008] Where x is the annotated speech sample, y is the text annotation corresponding to the annotated speech sample, p(x) is the distribution function of the annotated speech sample, θ t is the model parameter at the tth training iteration, and a(·) is the spectral data enhancement function.

[0009] As a feasible implementation manner, the semi-supervised training of the classifier using the personalized features of the voice information of each of the multiple clients includes: taking the personalized features of each of the multiple clients as input and taking a third loss function as a learning target, and performing semi-supervised training on the classifier, wherein the third loss function is:

[0010]

[0011] Among them, x2 is the personalized feature, is the pseudo label of the personalized feature.

[0012] As a feasible implementation, the pseudo-label for:

[0013]

[0014] in, are the teacher model parameters of the classifier at the tth training iteration.

[0015] As a feasible implementation, the teacher model parameters of the classifier are The value of is the exponential sliding average of the model parameters of the tth iteration and the previous teacher model parameters:

[0016]

[0017] Among them, α is the attenuation coefficient.

[0018] The embodiment of the present application provides a personalized federated learning method for speech recognition, which is applied to a client, and includes: obtaining a seed model of a speech recognition model; using a local speech sample to perform semi-supervised training on a feature extractor of the seed model to obtain a trained feature extractor; inputting speech information in a training set into the personalized feature extractor to obtain personalized features of the speech information, and sending the personalized features to a central end; combining the trained feature extractor with a trained classifier to obtain a trained speech recognition model; and the trained classifier is trained by the central end.

[0019] As a feasible implementation, the use of local speech samples to perform semi-supervised training on the feature extractor of the seed model includes: taking the local speech samples as input and a second loss function as a learning target, and performing semi-supervised training on the speech recognition model, wherein the second loss function is:

[0020]

[0021] Among them, x1 is the speech sample, is a pseudo label.

[0022] As a feasible implementation, the pseudo-label for:

[0023]

[0024] in, are the teacher model parameters at the tth training iteration.

[0025] As a feasible implementation, the teacher model parameters The value of is the exponential sliding average of the student model parameters of the tth iteration and the previous teacher model parameters:

[0026]

[0027] Among them, the model parameter θ t is the student model parameter at the tth training iteration, and α is the decay coefficient.

[0028] In a second aspect, an embodiment of the present application provides a personalized federated learning system for a speech recognition model, the system comprising: a central end, for using local annotated speech samples to perform supervised training on a speech recognition model to obtain a seed model of the speech recognition model, the seed model comprising a feature extractor and a classifier; the feature extractor is used to process speech information input by multiple clients, and determine the personalized features of the speech information of each of the multiple clients; the classifier is used to convert the speech information into text according to the personalized features; the classifier is semi-supervised trained using the personalized features of the speech information of each of the multiple clients to obtain a trained classifier; the trained classifier is transmitted to each of the multiple clients; multiple clients, each of the multiple clients is used to obtain a seed model of the speech recognition model; the feature extractor of the seed model is semi-supervised trained using local speech samples to obtain a trained feature extractor; the speech information in the training set is input into the personalized feature extractor to obtain the personalized features of the speech information, and the personalized features are sent to the central end; the trained feature extractor is combined with the trained classifier to obtain a trained speech recognition model; the trained classifier is trained by the central end.

[0029] A personalized federated learning method and system for a speech recognition model proposed in an embodiment of the present application can achieve personalized federated learning by decoupling and optimizing the feature extractor and classifier of the speech recognition model, thereby generating personalized speech recognition models for multiple clients through joint training of the center and the client, effectively improving the performance of the speech recognition model trained by federated learning and reducing the error recognition rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the multiple embodiments disclosed in this specification, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only the multiple embodiments disclosed in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 A schematic diagram of a speech recognition model proposed in an embodiment of the present application;

[0032] Figure 2 A system architecture diagram of a personalized federated learning method for a speech recognition model according to an embodiment of the present application;

[0033] Figure 3 A schematic diagram of a learning scenario of a personalized federated learning method for a speech recognition model proposed in an embodiment of the present application;

[0034] Figure 4 Flowchart of the personalized federated learning method for the speech recognition model proposed in the embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0036] In the description of the embodiments of the present application, words such as "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a concrete way.

[0037] In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, B exists alone, and A and B exist at the same time. In addition, unless otherwise specified, the term "multiple" means two or more. For example, multiple systems refers to two or more systems, and multiple terminals refers to two or more terminals.

[0038] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprises", "has" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0039] In the following description, the numbers representing the steps, such as S110, S120, etc., do not necessarily mean that the steps must be executed in this manner. If permitted, the order of the previous and next steps can be interchanged, or they can be executed simultaneously.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0041] Figure 1 A schematic diagram of a speech recognition model proposed in the embodiment of the present application. Figure 1As shown, the speech recognition model 10 includes a feature extractor 11 and a classifier 12. The feature extractor 11 is used to process speech information input by multiple clients to obtain personalized features of the speech information of each client; the classifier 12 is used to convert the speech information into text according to the personalized features.

[0042] The speech recognition model proposed in the embodiment of the present application can be applied to a client, which can be an intelligent device in the fields of smart terminals, smart homes, smart security, and autonomous driving. It provides a voice user interface (VUI) on clients such as smart phones, speakers, and smart vehicle terminals (electronic control units, ECUs), and converts the voice information input by the user into text.

[0043] Exemplarily, smart devices include smart TVs, smart speakers, robots, smart air conditioners, smart smoke alarms, smart fire extinguishers, smart car terminals, mobile phones, tablets, laptops, desktop computers, all-in-one computers, etc.

[0044] Figure 2 This is a system architecture diagram of a personalized federated learning method for a speech recognition model according to an embodiment of the present application. Figure 2 As shown, the system includes a central terminal 21 and multiple clients 22. Through joint training of a central terminal and multiple clients, a personalized speech recognition model for each of the multiple clients is generated, effectively improving the speech recognition performance of the speech recognition model obtained through federated learning training.

[0045] Based on this, the embodiment of the present application provides a personalized federated learning system for speech recognition, which includes a central end, the central end uses local annotated speech samples to perform supervised training on the speech recognition model to obtain a seed model of the speech recognition model, and the seed model includes a feature extractor and a classifier; the feature extractor is used to process the speech information input by multiple clients, and determine the personalized features of the speech information of each client in the multiple clients; the classifier is semi-supervised trained using the personalized features of the speech information of each client in the multiple clients to obtain a trained classifier; the trained classifier is transmitted to each client in the multiple clients. The system also includes multiple clients, each of the multiple clients obtains a seed model of the speech recognition model; the feature extractor of the seed model is semi-supervised trained using local speech samples to obtain a trained feature extractor; the speech information in the training set is input into the personalized feature extractor to obtain the personalized features of the speech information, and the personalized features are sent to the central end; the trained feature extractor is combined with the trained classifier to obtain a trained speech recognition model; the trained classifier is trained by the central end.

[0046] Figure 3Schematic diagram of the learning scenario of the personalized federated learning method of the speech recognition model proposed in the embodiment of the present application. Figure 3 As shown, in this learning scenario, a central terminal 21 and a client terminal 22 are included.

[0047] A seed model of a speech recognition model 10 is trained at the central end 21 using labeled speech samples. The seed model includes a feature extractor and a classifier. The seed model of the speech recognition model 10 is transmitted to each client 22 among multiple clients.

[0048] Taking into account the distribution differences of each client 22, the client 22 uses its own personalized voice data to train the feature extractor 11, uses the trained feature extractor 11 to process the voice sample of the client, obtains the personalized features of the client, and aggregates the obtained personalized features to the central end 21.

[0049] The central end 21 performs semi-supervised training on the classifier 12 of the speech recognition model 10 using the personalized features of all the clients among the multiple clients to obtain a trained classifier; and transmits the trained classifier 12 to the client 22 .

[0050] The client 22 combines its own trained feature extractor 11 with the trained classifier 12 to obtain a personalized speech recognition model 10 of the client.

[0051] In the personalized federated learning scenario of the speech recognition model proposed in the embodiment of the present application, the central end 21 and the client end 22 transmit non-original data to achieve learning using the personalized voice samples of each client without transmitting the voice samples to the central end, which can solve the problem of differences in the distribution of voice data of each client.

[0052] Based on the above learning scenario, an embodiment of the present application proposes a personalized federated learning method for a speech recognition model, which realizes personalized federated learning by decoupling and optimizing the feature extractor and classifier, thereby generating a personalized speech recognition model for multiple clients, effectively improving the performance of the speech recognition model and reducing the error recognition rate.

[0053] Figure 4 This is a flow chart of the personalized federated learning method of the speech recognition model proposed in the embodiment of the present application. Figure 4 As shown, the process of the federated learning method is applied to a central end and multiple clients, including the following steps 101-107.

[0054] Step 101, the central end uses the labeled speech samples of the central end to supervise and train a speech recognition model 10, where the speech recognition model includes a feature extractor and a classifier.

[0055] In one practicable implementation, any annotated speech sample in the training set at the central end can be input into a speech recognition model 10 for supervised training, and the connectionist temporal classification loss can be calculated. Values:

[0056]

[0057] Among them, x is the speech sample, y is the corresponding text annotation, p(x) is the distribution function of the speech sample, θ t is the model parameter at the tth training iteration, and a(·) is the spectrum data enhancement method. Recorded as the first loss function value.

[0058] When connectionist temporal classification loss When the value of reaches or is less than the set threshold, or reaches the set number of iterations t, the training is terminated to obtain the seed model of the speech recognition model 10, and the model parameters of the seed model are the current model parameters θ t .

[0059] Step 102 : The central end transmits the seed model of the speech recognition model 10 to each client 22 .

[0060] Step 103: On each client 22 , speech samples of the local personalized training set are used to perform semi-supervised training on the feature extractor 11 of the seed model of the speech recognition model 10 , and each client obtains a trained feature extractor 11 .

[0061] In an achievable implementation, the client 22 can input any speech sample x1 in the local personalized training set into the feature extractor 11 of the seed model and output a pseudo label The connectionist temporal classification loss is computed during this process Values:

[0062]

[0063] Different from the training criteria in step 101, in formula (2), the text annotation y corresponding to the speech sample x1 is not available, so a pseudo label is used Instead, the pseudo-label The generation method is:

[0064]

[0065] In formula (3), is the teacher model parameter at the tth training iteration, which is the exponential sliding average of the student model parameter at the tth iteration and the previous teacher model parameter:

[0066]

[0067] In formula (4), the model parameter θ t is the student model parameter at the tth training iteration, α is the attenuation coefficient, which is taken as a constant of 0.99 in this patent. Recorded as the second loss function value.

[0068] When connectionist temporal classification loss When the value reaches or is less than the set threshold, or the number of iterations reaches the set number t, the training ends. The model parameters of the feature extractor 11 are updated to the current teacher model parameters

[0069] During the training of step 103 , the parameters of the classifier 12 are kept fixed, and only the parameters of the feature extractor 11 are updated.

[0070] Step 104 , each client uses the trained feature extractor 11 to process the voice sample of the client to obtain the personalized features of the client, and aggregates the obtained personalized features to the central end 21 .

[0071] In a feasible implementation, the client 22 may input any speech sample in the local personalized training set into the trained feature extractor 11 , extract the personalized features of the speech sample, and send the obtained personalized features to the central end 21 .

[0072] Among them, personalized features are feature vectors extracted based on the differences between different clients. The differences between different clients include regional differences in spoken language or accent, such as reading style and natural conversation style, and channel differences in voice transmission, such as telephone channels and mobile phone microphone channels.

[0073] Step 105 , the central end 21 performs semi-supervised training on the classifier 12 of the speech recognition model using the received personalized features of all the clients 22 to obtain a trained classifier 12 .

[0074] In one practicable implementation, the central end 21 can input the personalized feature x2 into the classifier 12 and output the pseudo label y2, during which the connectionist temporal classification loss is calculated. Values:

[0075]

[0076] In formula (5), the text annotation y corresponding to the personalized feature x2 is not available, so the pseudo label of the personalized feature x2 is used Instead, the pseudo-label The generation method is:

[0077]

[0078] In formula (6), is the teacher model parameter at the tth training iteration, which is the exponential sliding average of the student model parameter at the tth iteration and the previous teacher model parameter:

[0079]

[0080] In formula (7), the model parameter θ t is the student model parameter at the tth training iteration, α is the attenuation coefficient, which is taken as a constant of 0.99 in this patent. Recorded as the third loss function value.

[0081] when When the value reaches or is less than the set threshold, or the number of iterations reaches the set number t, the training ends. The model parameters of the classifier 12 are updated to the current teacher model parameters

[0082] In the training of step 105 , the feature extractor 11 does not participate in the model training, and only the parameters of the backend classifier 12 are updated.

[0083] Step 106 , the central end 21 sends the trained classifier 12 to each client 22 .

[0084] Step 107, on any client, the trained feature extractor 11 is combined with the trained classifier 12 to obtain a personalized speech recognition model 10 for the client.

[0085] The speech recognition model 10 is used to process the speech information input by the client, and the speech information can be converted into corresponding text.

[0086] The rationality and effectiveness of the speech recognition system based on the embodiment of the present application have been verified in the actual system. In the experimental verification, the center has annotated speech samples including 6919 hours of English speech in different styles. There are three clients, each of which contains 100 hours of additional accented English speech. The accents of the three clients are Australian accent, British accent and Indian accent respectively. The experimental platform uses a graphics card V100 with a core frequency of 5012MHZ, a video memory of 3*32GB, a host equipped with 440GB memory, and a CPU of 2.70GHz.

[0087] Table 1

[0088]

[0089] The experimental results are shown in Table 1, and it can be seen that the method of the embodiment of the present application can effectively reduce the error recognition rate.

[0090] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0091] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented by hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0092] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above description is only the specific implementation method of the embodiments of the present application and is not intended to limit the protection scope of the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.

Claims

1. A personalized federated learning method for speech recognition models, applied to the central end, characterized by: The method comprises: Using locally annotated speech samples to perform supervised training on a speech recognition model, a seed model of the speech recognition model is obtained, wherein the seed model includes a feature extractor and a classifier; the feature extractor is used to process speech information input by multiple clients and determine personalized features of the speech information of each of the multiple clients; the classifier is used to convert the speech information into text based on the personalized features; Performing semi-supervised training on the classifier using personalized features of the voice information of each client among the multiple clients to obtain a trained classifier; The trained classifier is transferred to each of the plurality of clients; the supervised training of the speech recognition model using the local annotated speech samples includes: The speech recognition model is trained in a supervised manner using the labeled speech sample as input and the first loss function as the learning target. The first loss function is: Where x is the annotated speech sample, y is the text annotation corresponding to the annotated speech sample, p(x) is the distribution function of the annotated speech sample, θ t is the model parameter at the t-th training iteration, a(·) is the spectrum data enhancement function; and performing semi-supervised training on the classifier using the personalized features of the speech information of each client among the multiple clients includes: The classifier is semi-supervisedly trained using the personalized features of each of the multiple clients as input and a third loss function as a learning target, where the third loss function is: Among them, x2 is the personalized feature, is a pseudo label of the personalized feature; Pseudo labels of the personalized features for: in, is the teacher model parameter of the classifier at the tth training iteration; the teacher model parameter of the classifier The value of is the exponential sliding average of the model parameters of the tth iteration and the previous teacher model parameters: Where α is the attenuation coefficient.

2. A personalized federated learning method for speech recognition models, applied to multiple clients, characterized in that: include: Obtain the seed model of the speech recognition model; Using local speech samples to perform semi-supervised training on the feature extractor of the seed model to obtain a trained feature extractor; Inputting the speech information in the training set into the feature extractor to obtain personalized features of the speech information, and sending the personalized features to the central end; Combining the trained feature extractor with the trained classifier to obtain a trained speech recognition model; the trained classifier is trained by the central end; The semi-supervised training of the feature extractor of the seed model using the local speech sample includes: The local speech sample is used as input and the second loss function is used as the learning target to perform semi-supervised training on the speech recognition model. The second loss function is: Among them, x1 is the speech sample, is the pseudo label of the speech sample; the pseudo label of the speech sample for: in, is the teacher model parameter at the tth training iteration; the teacher model parameter The value of is the exponential sliding average of the student model parameters of the tth iteration and the previous teacher model parameters: Among them, the model parameter θ t is the student model parameter at the tth training iteration, and α is the decay coefficient.

3. A personalized federated learning system for speech recognition models, characterized by: The system comprises: A central end is used to use local annotated voice samples to perform supervised training on a speech recognition model to obtain a seed model of the speech recognition model, wherein the seed model includes a feature extractor and a classifier; the feature extractor is used to process voice information input by multiple clients and determine the personalized features of the voice information of each of the multiple clients; the classifier is used to convert the voice information into text based on the personalized features; the classifier is semi-supervised trained using the personalized features of the voice information of each of the multiple clients to obtain a trained classifier; the trained classifier is transmitted to each of the multiple clients; the central end performs personalized federated learning of the speech recognition model based on the method according to claim 1; Multiple clients, each of the multiple clients is used to obtain a seed model of the speech recognition model; use local speech samples to perform semi-supervised training on the feature extractor of the seed model to obtain a trained feature extractor; input speech information in a training set into the feature extractor to obtain personalized features of the speech information, and send the personalized features to a central end; combine the trained feature extractor with a trained classifier to obtain a trained speech recognition model; the trained classifier is trained by a central end; the multiple clients perform personalized federated learning of the speech recognition model based on the method as claimed in claim 2.

Citation Information

Cited By

  • Safe and efficient vehicle voice recognition method and system based on federal learning

    CN116564339A

  • Safe and efficient vehicle-machine voice recognition method and system based on federated learning

    CN116564339B