Federated learning-based speech recognition methods, systems, and computer devices

Abstract

This application relates to the field of natural language processing, providing a speech recognition method, system, and computer device based on federated learning. The method acquires a local speech sample set through a learning endpoint to obtain an initial local model, and uses an autoencoder network for feature extraction to obtain output values. Based on the output values ​​and sample labels, the initial local model is trained to obtain first fusion adaptive parameters. First global fusion parameters are obtained through a server. Based on the first global fusion parameters, the initial local model is trained to obtain second fusion adaptive parameters. When a preset termination condition is met, a target local model is obtained, and the target speech recognition model is determined by the server. The application then uses the target speech recognition model to perform speech recognition on the speech data. This application embodiment, by transmitting only the modified parameters of the model's feature fusion structure to the server, can increase the feature representation capability while reducing the number of model parameters, resulting in higher speech recognition accuracy.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a speech recognition method, system and computer device based on federated learning. Background Technology

[0002] Federated learning is an emerging foundational technology in artificial intelligence, involving multiple terminals and servers. In the training process of a federated learning-based speech recognition model, each terminal's speech samples are not shared with other terminals or the server. The server and terminals only communicate model parameters, ensuring that each terminal can benefit from the data of other terminals while protecting its own speech samples, thus guaranteeing model accuracy and high speech recognition accuracy. With technological advancements, speech samples have increasingly more labels, or the speech sequences input to the model become too long, leading to an increase in the number of model parameters. Related technologies address this by adding adapter modules at appropriate locations within a large model structure. By freezing the original model and updating only the adapter module's parameters, the number of model parameter modifications is significantly reduced. However, the adapter module is a relatively simple structure with limited correction and learning capabilities, resulting in a decrease in speech recognition accuracy. Summary of the Invention

[0003] The purpose of this application is to at least partially solve the problems of the prior art and provide a speech recognition method, system and computer device based on federated learning. By transmitting only the parameters of the model's feature fusion structure modification to the server, it can increase the representation ability of features while reducing the number of model parameters, and has a high speech recognition accuracy.

[0004] The technical solution of this application embodiment is as follows:

[0005] Firstly, this application provides a federated learning-based speech recognition method applied to a speech recognition system, the speech recognition system including an application terminal, a server, and multiple learning terminals, the method comprising:

[0006] The learning terminal acquires a local speech sample set, which includes multiple speech samples and sample labels corresponding to each speech sample.

[0007] The learning end acquires an initial local model, and the autoencoder network of the initial local model performs the following processing: performing a first feature extraction process on each of the speech samples to obtain a first speech feature set; performing a second feature extraction process on the first speech feature set to obtain a second speech feature set; performing a first fusion process on the first speech feature set and the second speech feature set to obtain a third speech feature set; and performing a second fusion process on the first speech feature set, the second speech feature set, and the third speech feature set to obtain an output value.

[0008] The learning end trains the initial local model based on the output value and the sample label to obtain the first fusion adaptive parameters;

[0009] The learning end sends the first fusion adaptive parameter to the server and receives the first global fusion parameter obtained by aggregating the first fusion adaptive parameter from the server.

[0010] The learning end trains the initial local model based on the first global fusion parameters to obtain the second fusion adaptive parameters. Under the preset termination conditions, the learning end obtains the target local model corresponding to the second fusion adaptive parameters and sends the target local model to the server.

[0011] The server determines the target speech recognition model based on the target local model sent by each learning terminal, and sends the target speech recognition model to the application terminal;

[0012] The application uses the target speech recognition model to perform speech recognition on the input speech data.

[0013] According to some embodiments of this application, the second fusion process of the first speech feature set, the second speech feature set, and the third speech feature set to obtain an output value includes:

[0014] The first speech feature set is used as the first matrix of the attention mechanism;

[0015] The second speech feature set is used as the second matrix of the attention mechanism;

[0016] The third speech feature set is used as the third matrix of the attention mechanism;

[0017] The learning end performs a first matrix fusion process on the first matrix, the second matrix, and the third matrix according to a preset fusion algorithm to obtain the output value.

[0018] According to some embodiments of this application, the learning end performs a first matrix fusion process on the first matrix, the second matrix, and the third matrix according to a preset fusion algorithm to obtain the output value, including:

[0019] The learning end performs a second matrix fusion process on the first matrix and the second matrix to obtain a first fusion value;

[0020] The learning end uses a normalization layer to normalize the first fusion value to obtain a second fusion value;

[0021] The learning end performs a third matrix fusion process on the second fusion value and the third matrix to obtain the output value.

[0022] According to some embodiments of this application, after the initial local model is trained on the learning end based on the first global fusion parameters to obtain the second fusion adaptive parameters, the method further includes:

[0023] If the preset termination condition is not met, the learning end sends the second fusion adaptive parameter to the server;

[0024] The learning end receives a second global fusion parameter obtained by aggregating the second fusion adaptive parameter from the server;

[0025] The learning end trains the initial local model based on the second global fusion parameters.

[0026] According to some embodiments of this application, the learning end trains the initial local model based on the output value and the sample label to obtain first fusion adaptive parameters, including:

[0027] The learning end obtains the value of the loss function based on the output value and the sample label corresponding to the output value;

[0028] The learning end trains the initial local model based on the value of the loss function to obtain the first fusion adaptive parameters.

[0029] According to some embodiments of this application, the step of performing a second feature extraction process on the first speech feature set to obtain a second speech feature set includes:

[0030] The learning end performs a downsampling operation on the first speech feature set to obtain sampled features;

[0031] The learning end performs an upsampling operation on the sampled features to obtain the second speech feature set.

[0032] According to some embodiments of this application, the learning end obtains the value of the loss function based on the output value and the sample label corresponding to the output value, including:

[0033] The learning end uses KL divergence to calculate the output value and the value of the sample label corresponding to the output value, and obtains the value of the loss function.

[0034] Secondly, this application provides a federated learning-based speech recognition system, which includes an application terminal, a server, and multiple learning terminals, wherein the learning terminals include:

[0035] The data acquisition module is used to acquire a local speech sample set, which includes multiple speech samples and sample labels corresponding to each speech sample;

[0036] The model acquisition module is used to acquire an initial local model. The autoencoder network of the initial local model performs the following processing: performing a first feature extraction process on each of the speech samples to obtain a first speech feature set; performing a second feature extraction process on the first speech feature set to obtain a second speech feature set; performing a first fusion process on the first speech feature set and the second speech feature set to obtain a third speech feature set; and performing a second fusion process on the first speech feature set, the second speech feature set, and the third speech feature set to obtain an output value.

[0037] The first processing module is used to train the initial local model based on the output value and the sample label to obtain the first fusion adaptive parameters;

[0038] The sending module is used to send the first fusion adaptive parameters to the server, and the receiving module is used to receive the first global fusion parameters obtained by aggregating the first fusion adaptive parameters sent by the server.

[0039] The second processing module is used to train the initial local model based on the first global fusion parameters to obtain the second fusion adaptive parameters, and to obtain the target local model corresponding to the second fusion adaptive parameters under the preset termination conditions. The sending module is also used to send the target local model to the server.

[0040] The server determines the target speech recognition model based on the target local model sent by each learning terminal, and sends the target speech recognition model to the application terminal;

[0041] The application uses the target speech recognition model to perform speech recognition on the input speech data.

[0042] Thirdly, this application provides a computer device including a memory and a processor, wherein the memory stores computer-readable instructions that, when executed by one or more of the processors, cause the one or more processors to perform the steps of any of the methods described in the first aspect above.

[0043] Fourthly, this application also provides a computer-readable storage medium that can be read and written by a processor, the storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of any of the methods described in the first aspect above.

[0044] The technical solution provided in this application has the following beneficial effects:

[0045] This application proposes a speech recognition method, system, and computer device based on federated learning. The federated learning-based speech recognition method includes: a learning end acquiring a local speech sample set, which includes multiple speech samples and corresponding sample labels for each speech sample; the learning end acquiring an initial local model, the autoencoder network of which performs the following processing: performing a first feature extraction process on each speech sample to obtain a first speech feature set; performing a second feature extraction process on the first speech feature set to obtain a second speech feature set; performing a first fusion process on the first and second speech feature sets to obtain a third speech feature set; and performing a second fusion process on the first, second, and third speech feature sets to obtain an output value. The fusion process increases the representational power of the features; the learning end trains the initial local model based on the output value and sample labels to obtain first fusion adaptive parameters; and the learning end feeds data to a server... The server sends only the first fusion adaptive parameters, instead of all model parameters, reducing parameter transmission and improving communication speed. It also receives the first global fusion parameters from the server, which are obtained by aggregating the first fusion adaptive parameters. This facilitates subsequent fine-tuning of the initial local model using the first global fusion parameters. The learning end trains the initial local model based on the first global fusion parameters to obtain the second fusion adaptive parameters. Under preset termination conditions, it obtains the target local model corresponding to the second fusion adaptive parameters and sends the target local model to the server. The server determines the target speech recognition model based on the target local models sent by each learning end, obtains the target speech recognition model through federated learning, which increases the accuracy of model speech recognition, and sends the target speech recognition model to the application end. The application end performs speech recognition on the input speech data using the target speech recognition model. This embodiment of the application, by performing feature fusion on the model and transmitting only the parameters modifying the model's feature fusion structure to the server, can increase the feature representation capability while reducing the number of model parameters, resulting in higher speech recognition accuracy. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating a federated learning-based speech recognition method provided in one embodiment of this application;

[0047] Figure 2 yes Figure 1 A flowchart illustrating a sub-step of step S200;

[0048] Figure 3 yes Figure 2 A flowchart illustrating a sub-step of step S240;

[0049] Figure 4This is a flowchart illustrating a federated learning-based speech recognition method provided in another embodiment of this application;

[0050] Figure 5 yes Figure 1 A flowchart illustrating a sub-step of step S300;

[0051] Figure 6 yes Figure 1 A flowchart illustrating another sub-step of step S200;

[0052] Figure 7 This is a schematic diagram of the structure of a federated learning-based speech recognition system provided in one embodiment of this application;

[0053] Figure 8 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0056] First, let's analyze some of the terms used in this application:

[0057] Federated learning is an emerging foundational technology in artificial intelligence. Its design goal is to ensure information security during big data exchange, protect the privacy of endpoint and personal data, and guarantee efficient machine learning across multiple participants or computing nodes within legal and compliant frameworks. In federated learning, model training is distributed across a master server and multiple endpoints. The data held by each endpoint is not shared with other endpoints or the master server. The master server and the endpoints only communicate the planned changes to model parameters (or model parameters) during training. This approach allows each endpoint to benefit from the data of other endpoints while ensuring its own data security.

[0058] The Transformer model consists of an encoder and a decoder. Both the encoder and decoder contain blocks for feature extraction. The encoder mainly consists of feedforward neural network layers and multi-head self-attention layers. Each sub-layer (self-attention and feedforward network) in the encoder has a residual connection around it, and each is followed by a "layer normalization" step. The decoder's internal components are largely similar to the encoder. The decoder's first attention layer is called a masked multi-head attention mechanism, which, by adding a masking operation, allows only the earlier positions in the output sequence to be processed.

[0059] This application proposes a speech recognition method, system, and computer device based on federated learning. The federated learning-based speech recognition method includes: a learning end acquiring a local speech sample set, which includes multiple speech samples and corresponding sample labels for each speech sample; the learning end acquiring an initial local model, the autoencoder network of which performs the following processing: performing a first feature extraction process on each speech sample to obtain a first speech feature set; performing a second feature extraction process on the first speech feature set to obtain a second speech feature set; performing a first fusion process on the first and second speech feature sets to obtain a third speech feature set; and performing a second fusion process on the first, second, and third speech feature sets to obtain an output value. The fusion process increases the representational power of the features; the learning end trains the initial local model based on the output value and sample labels to obtain first fusion adaptive parameters; and the learning end feeds data to a server... The server sends only the first fusion adaptive parameters, instead of all model parameters, reducing parameter transmission and improving communication speed. It also receives the first global fusion parameters from the server, which are obtained by aggregating the first fusion adaptive parameters. This facilitates subsequent fine-tuning of the initial local model using the first global fusion parameters. The learning end trains the initial local model based on the first global fusion parameters to obtain the second fusion adaptive parameters. Under preset termination conditions, it obtains the target local model corresponding to the second fusion adaptive parameters and sends it to the server. The server determines the target speech recognition model based on the target local models sent by each learning end, obtains the target speech recognition model through federated learning, which increases the accuracy of the model's speech recognition, and sends the target speech recognition model to the application end. The application end performs speech recognition on the input speech data using the target speech recognition model. This embodiment of the application, by transmitting only the parameters modifying the model's feature fusion structure to the server, can increase the feature representation capability while reducing the number of model parameters, resulting in higher speech recognition accuracy.

[0060] It should be noted that this federated learning-based speech recognition method optimizes the entire federated system by reducing parameter transmission between the learning end and the server. The initial local model in the learning end is a transformer model or a variant of the transformer model, which can recognize natural language or processed speech and has a wide range of applications. The transformer model is applicable to various application scenarios. By using the transformer model at each learning end of the federated system and selecting the appropriate transformer model or its variant based on different input datasets, different tasks can be accomplished, demonstrating its broad applicability.

[0061] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0062] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0063] The embodiments of this application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0064] The federated learning-based speech recognition method, system, and computer device provided in the embodiments of this application will now be described with reference to the accompanying drawings.

[0065] See Figure 1 , Figure 1 The diagram illustrates a flowchart of a federated learning-based speech recognition method provided in an embodiment of this application. The federated learning-based speech recognition method is applied to a speech recognition system, which includes an application terminal, a server, and multiple learning terminals. The federated learning-based speech recognition method includes, but is not limited to, steps S100, S200, S300, S400, S500, S600, and S700.

[0066] Step S100: The learning device obtains a local speech sample set, which includes multiple speech samples and sample labels corresponding to each speech sample.

[0067] In one embodiment, acquiring a local speech sample set at the learning end is beneficial for subsequently training the network model using this local speech sample set. The speech sample set includes multiple speech samples and corresponding sample labels for each speech sample. Each speech sample can be natural language, or it can be natural language or processed speech; the sample labels are the correct text information corresponding to the speech. The speech sample sets for each learning end can be the same or different.

[0068] Step S200: The learning end obtains the initial local model. The autoencoder network of the initial local model performs the following processing: a first feature extraction process is performed on each speech sample to obtain a first speech feature set; a second feature extraction process is performed on the first speech feature set to obtain a second speech feature set; a first fusion process is performed on the first speech feature set and the second speech feature set to obtain a third speech feature set; and a second fusion process is performed on the first speech feature set, the second speech feature set, and the third speech feature set to obtain the output value.

[0069] In one embodiment, the acquired initial local model can be a transformer model or a variant of the transformer model. The initial local model is a transformer model. In the encoder network of the initial local model, a first feature extraction process can be performed on each speech sample using an encoder network or an encoder network with an adaptive module to obtain a first speech feature set. This is beneficial for subsequent feature extraction from the first speech feature set. A second feature extraction process is then performed on the first speech feature set using the adaptive module in the encoder network with the adaptive module to obtain a second speech feature set. This is beneficial for subsequent feature fusion operations. The first and second speech feature sets are then fused in a first process to obtain a third speech feature set. Finally, the first, second, and third speech feature sets are fused in a second process to obtain the output value. By fusing the output features, the local model can achieve better performance.

[0070] For example, the initial local model is a transformer model, which includes a feedforward module, an adaptive module, and an adaptive fusion module. The feedforward module performs a first feature extraction process on each speech sample to obtain a first speech feature set. The adaptive module performs a second feature extraction process on the first speech feature set to obtain a second speech feature set. The adaptive fusion module performs a first fusion process on the first and second speech feature sets to obtain a third speech feature set. Finally, the first, second, and third speech feature sets are fused together to obtain the output value. By fusing the output features of the feedforward and adaptive modules through the adaptive fusion module, the local model can achieve better performance.

[0071] In one embodiment, the first speech feature set includes multiple first speech features, which are representations of features extracted from speech samples; the second speech feature set includes multiple second speech features, which are representations of features extracted from the first speech feature set; and the third speech feature set includes multiple third speech features, which are representations of fused features. The first fusion process and the second fusion process can be either a dot product of features or a concatenation of features.

[0072] like Figure 2 As shown, the first speech feature set, the second speech feature set, and the third speech feature set are subjected to a second fusion process to obtain the output value, including but not limited to the following steps:

[0073] Step S210: Use the first speech feature set as the first matrix of the attention mechanism.

[0074] In one embodiment, an attention mechanism is used for processing. This attention mechanism can be either self-attention or multi-head attention, as long as it can construct a QKV matrix; details are omitted here. When the attention mechanism is self-attention, the first extracted feature set is used as the first matrix of the attention mechanism, where the first matrix is ​​the Q-matrix of the self-attention mechanism. Obtaining the first matrix facilitates subsequent calculation of the output value.

[0075] Step S220: Use the second speech feature set as the second matrix of the attention mechanism.

[0076] In one embodiment, according to step S210, when the attention mechanism is a self-attention mechanism, the second extracted feature set is used as the second matrix of the attention mechanism, wherein the second matrix is ​​the K matrix of the self-attention mechanism. Obtaining the second matrix is ​​beneficial for subsequent calculation of the output value based on the second matrix.

[0077] Step S230: Use the third speech feature set as the third matrix of the attention mechanism.

[0078] In one embodiment, according to steps S210 and S220, when the attention mechanism is a self-attention mechanism, the third extracted feature set is used as the third matrix of the attention mechanism, wherein the third matrix is ​​the V matrix of the self-attention mechanism. Obtaining the third matrix is ​​beneficial for subsequent calculation of the output value based on the third matrix.

[0079] In step S240, the learning end performs first matrix fusion processing on the first matrix, the second matrix, and the third matrix according to the preset fusion algorithm to obtain the output value.

[0080] like Figure 3 As shown, the learning end performs matrix fusion processing on the first matrix, the second matrix, and the third matrix according to a preset fusion algorithm to obtain the output value, including but not limited to the following steps:

[0081] Step S241: The learning end performs a second matrix fusion process on the first matrix and the second matrix to obtain the first fusion value;

[0082] In one embodiment, the learning end performs a second matrix fusion process on the first matrix and the second matrix to obtain a first fusion value. By performing the fusion process, the extracted features can have better performance capabilities.

[0083] In step S242, the learning end uses a normalization layer to normalize the first fusion value to obtain the second fusion value.

[0084] In one embodiment, the learning end uses a normalization layer to normalize the first fused value to obtain a second fused value. The normalization layer can be a softmax layer or a sigmoid layer. By performing normalization, the features can be standardized to a certain range, avoiding feature values ​​that are too large or too small, thereby affecting the accuracy of the model.

[0085] Step S243: The learning end performs third matrix fusion processing on the second fusion value and the third matrix to obtain the output value.

[0086] In one embodiment, the learning end performs a third matrix fusion process on the second fusion value and the third matrix to obtain an output value. This fusion process enhances the performance of the extracted features. The first matrix fusion process, the second matrix fusion process, and the third matrix fusion process involve fusing two or more matrices; these can be matrix dot products or matrix concatenation. Both the first fusion value and the second fusion value are representations of matrix fusion.

[0087] like Figure 6 As shown, the second speech feature set is obtained by performing a second feature extraction process on the first speech feature set, including but not limited to the following steps:

[0088] In step S250, the learning end performs a downsampling operation on the first speech feature set to obtain sampled features.

[0089] In one embodiment, the learning end can either average the values ​​of each first speech feature in the first speech feature set to obtain sampled features, or it can take the maximum value of each first speech feature in the first speech feature set to obtain sampled features. Downsampling reduces computation and avoids model overfitting.

[0090] In step S260, the learning end performs an upsampling operation on the sampled features to obtain the second speech feature set.

[0091] In one embodiment, the learning end can use dilated convolution to sample the features to obtain a second speech feature set, or it can use deconvolution to sample the features to obtain a second speech feature set. Upsampling can expand the receptive field and enhance the feature representation capability.

[0092] In step S300, the learning end trains the initial local model based on the output value and sample labels to obtain the first fusion adaptive parameters.

[0093] like Figure 5 As shown, the learning end trains the initial local model based on the output value and sample labels to obtain the first fusion adaptive parameters, including but not limited to the following steps:

[0094] In step S310, the learning end obtains the value of the loss function based on the output value and the sample label corresponding to the output value.

[0095] In one embodiment, the learning end can use Kullback-Leible divergence (KL divergence) to calculate the output value and the corresponding sample label value to obtain the loss function value; alternatively, it can use Jenson's Shannon divergence (JS divergence) to calculate the output value and the corresponding sample label value to obtain the loss function value. Obtaining the loss function value is beneficial for subsequent training of the initial local model.

[0096] Step S320: The learning end trains the initial local model based on the value of the loss function to obtain the first fusion adaptive parameters.

[0097] In one embodiment, the learning end performs backpropagation on the initial local model based on the value of the loss function, and updates the weights and biases of the initial local model using the gradient descent algorithm, causing the initial local model to converge. The updated weights and biases are the weights and biases used for the first fusion processing, while the parameters of other feature extraction processes of the initial local model are frozen and not updated, thus obtaining the first fusion adaptive parameters. The first fusion adaptive parameters are the weights and biases used for the first fusion processing. Obtaining the first fusion adaptive parameters facilitates the subsequent determination of the first global fusion parameters.

[0098] In step S400, the learning end sends the first fusion adaptive parameters to the server and receives the first global fusion parameters obtained by aggregating the first fusion adaptive parameters from the server.

[0099] In one embodiment, the first fusion adaptive parameters are obtained according to steps S100-S400. Each learning end sends the first fusion adaptive parameters to the server. Sending only the first fusion adaptive parameters to the server, instead of all model parameters, reduces parameter transmission and improves communication speed. It also receives the first global fusion parameters obtained from the server by aggregating the first fusion adaptive parameters. Obtaining the first global fusion parameters is beneficial for subsequent fine-tuning of the initial local model parameters. The first global fusion parameters are the model parameters obtained from the server.

[0100] In one embodiment, the server receives the first fusion adaptive parameters sent by each learning end. It can calculate the average value of each first fusion adaptive parameter to obtain the first global fusion parameter; or it can select the first fusion adaptive parameter corresponding to the best performance of the initial local model from each first fusion adaptive parameter to obtain the first global fusion parameter; or it can use other calculation methods to obtain the first global fusion parameter, which will not be elaborated here.

[0101] In step S500, the learning end trains an initial local model based on the first global fusion parameters to obtain the second fusion adaptive parameters. Under the preset termination conditions, the target local model corresponding to the second fusion adaptive parameters is obtained and sent to the server.

[0102] In one embodiment, according to step S400, the first global fusion parameters are obtained, and the first fusion adaptive parameters are replaced with the first global fusion parameters. Then, the initial local model is trained based on the first global fusion parameters. During the training process, when the model backpropagates, only the weights and biases that have undergone the first fusion processing are updated to obtain the second fusion adaptive parameters. Under the preset termination condition, the target local model is obtained and sent to the server, which is beneficial for obtaining the target speech recognition model through the server later. The preset termination condition can be that the loss function meets a preset value, the initial local model converges, and the loss function no longer fluctuates significantly; or it can be that the training number of the initial local model reaches a preset maximum training number, and the training ends to obtain the target local model. For example, the preset maximum training number can be 100,000 times. The second fusion adaptive parameters are obtained by updating the first global fusion parameters during training.

[0103] like Figure 4 As shown, after training the initial local model based on the first global fusion parameters at the learning end and obtaining the second fusion adaptive parameters, the federated learning-based speech recognition method also includes, but is not limited to, the following steps:

[0104] In step S810, if the preset termination condition is not met, the learning end sends the second fusion adaptive parameter to the server.

[0105] In one embodiment, if the preset termination condition is not met (i.e., the initial local model has not yet converged), multiple rounds of training are required. The learning end sends the second fusion adaptive parameters to the server, which is beneficial for obtaining the second global fusion parameters from the server later. The second global fusion parameters are the model parameters obtained through the server.

[0106] In step S820, the learning end receives the second global fusion parameters obtained by aggregating the second fusion adaptive parameters from the server.

[0107] In one embodiment, the server receives the second fusion adaptive parameters sent by each learning end. It can calculate the average of these second fusion adaptive parameters to obtain the second global fusion parameter; alternatively, it can select the second fusion adaptive parameter that corresponds to the best initial local model performance from among the various second fusion adaptive parameters to obtain the second global fusion parameter; or it can use other calculation methods to obtain the second global fusion parameter, which will not be elaborated here. The server sends the calculated second global fusion parameter to each learning end, and each learning end receiving the second global fusion parameter from the server facilitates subsequent model training based on the second global fusion parameter.

[0108] Step S830: The learning end trains the initial local model based on the second global fusion parameters.

[0109] In one embodiment, the second global fusion parameter is obtained according to step S820, the second fusion adaptive parameter is replaced by the second global fusion parameter, and then the initial local model is trained based on the second global fusion parameter. During the training process, when the model backpropagates, only the weights and biases that have undergone the first fusion processing are updated.

[0110] In step S600, the server determines the target speech recognition model based on the target local model sent by each learning end, and sends the target speech recognition model to the application end.

[0111] In one embodiment, based on the target local models received by the server from each learning end in step S500, the server selects the model with the best performance from among the target local models as the target speech recognition model. This target speech recognition model is obtained through federated learning, which can increase the accuracy of speech recognition. The target speech recognition model is then sent to the application end. Obtaining the target speech recognition model helps improve speech recognition accuracy.

[0112] In step S700, the application performs speech recognition on the input speech data using the target speech recognition model.

[0113] In one embodiment, according to step S600, the application receives the target speech recognition model sent by the server, and performs speech recognition on the input speech data through the target speech recognition model to achieve intelligent conversion, saving time, manpower, material resources and financial resources.

[0114] refer to Figure 7This application provides a federated learning-based speech recognition system 100, which includes multiple learning terminals 110, a server 120, and an application terminal 130. Each learning terminal 110 includes: a data acquisition module 111 for acquiring a local speech sample set, which includes multiple speech samples and corresponding sample labels; a model acquisition module 112 for acquiring an initial local model, wherein the autoencoder network of the initial local model performs the following processing: performing a first feature extraction process on each speech sample to obtain a first speech feature set; performing a second feature extraction process on the first speech feature set to obtain a second speech feature set; performing a first fusion process on the first and second speech feature sets to obtain a third speech feature set; and performing a second fusion process on the first, second, and third speech feature sets to obtain an output value. The fusion process increases the representational power of the features; a first processing module 113 for training the initial local model based on the output value and sample labels to obtain first fusion adaptive parameters; and a sending module 114. The receiving module 115 is used to send the first fusion adaptive parameters to the server 120. Only the first fusion adaptive parameters are sent to the server, instead of all model parameters, which reduces parameter transmission and improves communication speed. The receiving module 115 is used to receive the first global fusion parameters obtained by aggregating the first fusion adaptive parameters sent by the server 120, which is beneficial for subsequent fine-tuning of the parameters of the initial local model using the first global fusion parameters. The second processing module 116 is used to train the initial local model based on the first global fusion parameters to obtain the second fusion adaptive parameters. Under the preset termination conditions, the target local model corresponding to the second fusion adaptive parameters is obtained. The sending module 114 is also used to send the target local model to the server 120. The server 120 determines the target speech recognition model based on the target local model sent by each learning end 110. The target speech recognition model is obtained through federated learning, which can increase the accuracy of model speech recognition. The target speech recognition model is then sent to the application end 130. The application end 130 performs speech recognition on the input speech data through the target speech recognition model.

[0115] It should be noted that the data acquisition module 111 is connected to the model acquisition module 112, the model acquisition module 112 is connected to the first processing module 113, the first processing module 113 is connected to the sending module 114, the sending module 114 is connected to the receiving module 115, and the receiving module 115 is connected to the second processing module 116. The above-described federated learning-based speech recognition method is applied to the federated learning-based speech recognition system 100. This federated learning-based speech recognition system 100, by transmitting only the parameters modifying the model's feature fusion structure to the server, can increase the feature representation capability while reducing the number of model parameters, resulting in higher speech recognition accuracy. Both the first processing module 113 and the second processing module 116 are central processing units (CPUs). A CPU typically consists of a logic unit, a control unit, and a storage unit. Utilizing a CPU for computation saves significant human resources.

[0116] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0117] Figure 8 This application illustrates a computer device 500 provided in an embodiment of the present application. The computer device 500 may be a server or a terminal, and its internal structure includes, but is not limited to:

[0118] Memory 510 is used to store programs;

[0119] The processor 520 is used to execute the program stored in the memory 510. When the processor 520 executes the program stored in the memory 510, the processor 520 is used to execute the above-mentioned federated learning-based speech recognition method.

[0120] The processor 520 and memory 510 can be connected via a bus or other means.

[0121] The memory 510, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs, such as the federated learning-based speech recognition method described in any embodiment of this application. The processor 520 implements the above-described federated learning-based speech recognition method by running the non-transitory software program and instructions stored in the memory 510.

[0122] The memory 510 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store the federated learning-based speech recognition method described above. Furthermore, the memory 510 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 510 may optionally include memory remotely located relative to the processor 520, and these remote memories may be connected to the processor 520 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0123] The non-transient software program and instructions required to implement the above-described federated learning-based speech recognition method are stored in the memory 510. When executed by one or more processors 520, the federated learning-based speech recognition method provided in any embodiment of this application is executed.

[0124] This application also provides a computer-readable storage medium storing computer-executable instructions for executing the above-described federated learning-based speech recognition method.

[0125] In one embodiment, the storage medium stores computer-executable instructions that are executed by one or more control processors 520, such as one of the processors 520 in the computer device 500, to enable the one or more processors 520 to perform the federated learning-based speech recognition method provided in any embodiment of this application.

[0126] The embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0127] The terms “first,” “second,” “third,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0128] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0129] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0130] The foregoing has provided a detailed description of the preferred embodiments of this application. However, this application is not limited to the above-described embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A speech recognition method based on federated learning, characterized in that, Applied to a speech recognition system, the speech recognition system including an application terminal, a server, and multiple learning terminals, the method includes: The learning terminal acquires a local speech sample set, which includes multiple speech samples and sample labels corresponding to each speech sample. The learning end acquires an initial local model, and the autoencoder network of the initial local model performs the following processing: performing a first feature extraction process on each of the speech samples to obtain a first speech feature set; performing a second feature extraction process on the first speech feature set to obtain a second speech feature set; performing a first fusion process on the first speech feature set and the second speech feature set to obtain a third speech feature set; and performing a second fusion process on the first speech feature set, the second speech feature set, and the third speech feature set to obtain an output value. The learning end trains the initial local model based on the output value and the sample label to obtain the first fusion adaptive parameters; wherein, the first fusion adaptive parameters are the weights and biases for the first fusion processing; The learning end sends the first fusion adaptive parameter to the server and receives the first global fusion parameter obtained by aggregating the first fusion adaptive parameter from the server. The learning end trains the initial local model based on the first global fusion parameters to obtain the second fusion adaptive parameters. Under the preset termination conditions, the learning end obtains the target local model corresponding to the second fusion adaptive parameters and sends the target local model to the server. If the preset termination condition is not met, the learning end sends the second fusion adaptive parameter to the server; The learning end receives a second global fusion parameter obtained by aggregating the second fusion adaptive parameter from the server; The learning end trains the initial local model based on the second global fusion parameters; The server determines the target speech recognition model based on the target local model sent by each learning terminal, and sends the target speech recognition model to the application terminal; The application uses the target speech recognition model to perform speech recognition on the input speech data.

2. The method according to claim 1, characterized in that, The step of performing a second fusion process on the first speech feature set, the second speech feature set, and the third speech feature set to obtain an output value includes: The first speech feature set is used as the first matrix of the attention mechanism; The second speech feature set is used as the second matrix of the attention mechanism; The third speech feature set is used as the third matrix of the attention mechanism; The learning end performs a first matrix fusion process on the first matrix, the second matrix, and the third matrix according to a preset fusion algorithm to obtain the output value.

3. The method according to claim 2, characterized in that, The learning end performs a first matrix fusion process on the first matrix, the second matrix, and the third matrix according to a preset fusion algorithm to obtain the output value, including: The learning end performs a second matrix fusion process on the first matrix and the second matrix to obtain a first fusion value; The learning end uses a normalization layer to normalize the first fusion value to obtain a second fusion value; The learning end performs a third matrix fusion process on the second fusion value and the third matrix to obtain the output value.

4. The method according to claim 1, characterized in that, The learning end trains the initial local model based on the output value and the sample label to obtain the first fusion adaptive parameters, including: The learning end obtains the value of the loss function based on the output value and the sample label corresponding to the output value; The learning end trains the initial local model based on the value of the loss function to obtain the first fusion adaptive parameters.

5. The method according to claim 1, characterized in that, The step of performing a second feature extraction process on the first speech feature set to obtain a second speech feature set includes: The learning end performs a downsampling operation on the first speech feature set to obtain sampled features; The learning end performs an upsampling operation on the sampled features to obtain the second speech feature set.

6. The method according to claim 4, characterized in that, The learning end obtains the value of the loss function based on the output value and the sample label corresponding to the output value, including: The learning end uses KL divergence to calculate the output value and the value of the sample label corresponding to the output value, and obtains the value of the loss function.

7. A speech recognition system based on federated learning, characterized in that, The speech recognition system includes an application terminal, a server, and multiple learning terminals, wherein the learning terminals include: The data acquisition module is used to acquire a local speech sample set, which includes multiple speech samples and sample labels corresponding to each speech sample; The model acquisition module is used to acquire an initial local model. The autoencoder network of the initial local model performs the following processing: performing a first feature extraction process on each of the speech samples to obtain a first speech feature set; performing a second feature extraction process on the first speech feature set to obtain a second speech feature set; performing a first fusion process on the first speech feature set and the second speech feature set to obtain a third speech feature set; and performing a second fusion process on the first speech feature set, the second speech feature set, and the third speech feature set to obtain an output value. The first processing module is used to train the initial local model based on the output value and the sample label to obtain the first fusion adaptive parameters; The sending module is used to send the first fusion adaptive parameters to the server, and the receiving module is used to receive the first global fusion parameters obtained by aggregating the first fusion adaptive parameters sent by the server. The second processing module is used to train the initial local model based on the first global fusion parameters to obtain the second fusion adaptive parameters, and to obtain the target local model corresponding to the second fusion adaptive parameters under the preset termination conditions. The sending module is also used to send the target local model to the server. The learning end is also used to: send the second fusion adaptive parameters to the server if the preset termination condition is not met; receive the second global fusion parameters obtained by aggregating the second fusion adaptive parameters from the server; and train the initial local model based on the second global fusion parameters. The server determines the target speech recognition model based on the target local model sent by each learning terminal, and sends the target speech recognition model to the application terminal; The application uses the target speech recognition model to perform speech recognition on the input speech data.

8. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing computer-readable instructions that, when executed by one or more of the processors, cause the one or more processors to perform the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium is readable and writable by a processor, and the storage medium stores computer-readable instructions that, when executed by one or more processors, cause one or more processors to perform the steps of the method as described in any one of claims 1 to 6.

Images