A voice processing method and related apparatus

Abstract

A speech processing method is applied to speech recognition. In the method, during the speech processing, a trained speech enhancement network is used to perform enhancement processing on the speech, and then speech recognition is performed on the enhanced speech. Moreover, the trained speech enhancement network is trained based on multiple parallel speech recognition networks, and the outputs of the multiple speech recognition networks are used to jointly constitute a loss function for training the speech enhancement network, which can effectively improve the generalization and universality of the trained speech enhancement network. In this way, performing speech processing based on the trained speech enhancement network can effectively improve the effect of speech recognition, and the trained speech enhancement network can be applied to speech recognition networks that have not been jointly trained, thereby meeting the needs in various scenarios.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a speech processing method and related apparatus. Background Technology

[0002] Automatic speech recognition (ASR) technology is a technique that automatically converts a speaker's speech into text information. For example... Figure 1 As shown, the voice signal is usually received by the speaker by the microphone first, and then the automatic speech recognition system deployed outside the microphone recognizes the voice signal and finally outputs the recognized text.

[0003] With the maturity of automatic speech recognition technology, various products using voice as the interaction interface have flooded the market and appeared in daily life. However, natural voice interaction in real-world scenarios is affected by many factors, such as environmental noise and room reverberation. These factors reduce the system's recognition rate of the target speech, resulting in a poor user experience. How to improve the accuracy of speech recognition systems in real-world scenarios is a long-standing problem facing the industry.

[0004] To improve the accuracy of speech recognition, a speech enhancement network is typically added before the speech recognition network. This enhancement network first reduces noise in the speech data before it is input into the speech recognition network for processing. However, the speech enhancement networks used in these technologies are usually used in conjunction with specific speech recognition networks, resulting in poor applicability of the speech enhancement networks. Summary of the Invention

[0005] This application provides a model training and speech processing method that can effectively improve the generalization and versatility of the trained speech enhancement network, enabling the speech enhancement network to be applied to speech recognition networks that have not been jointly trained.

[0006] Firstly, a model training method is provided for training a speech enhancement network. This method includes: firstly, acquiring training samples, which include speech samples and the corresponding real text. The speech sample refers to a segment of speech data with noise, such as a user's speech captured by a microphone in a noisy environment. Furthermore, the real text is the actual text content corresponding to the speech content in the speech sample.

[0007] Then, the speech sample is input into the speech enhancement network, which performs speech enhancement processing on the speech sample to obtain the enhanced speech sample. The speech enhancement network is a neural network, such as a convolutional neural network, a recurrent neural network, or a long short-term memory (LSTM) artificial neural network.

[0008] Secondly, the enhanced speech samples are input into multiple different speech recognition networks to obtain the speech recognition results output by each network. Each speech recognition network outputs a recognized text based on the enhanced speech sample. Furthermore, the multiple speech recognition networks each have a different network structure.

[0009] Finally, the speech enhancement network is trained based on the total loss function value to obtain the speech enhancement network with updated parameters. The total loss function value is obtained based on the loss function value of each speech recognition network, and the loss function value of each speech recognition network is obtained based on the speech recognition result output by each network and the actual text.

[0010] In other words, after processing the enhanced speech sample through each speech recognition network, the speech recognition result output by each network can be obtained. Based on the real text corresponding to the speech sample and the speech recognition result output by each network, a loss function for each network can be constructed, and the loss function value for each network specific to that speech sample can be calculated. Finally, by combining the multiple loss function values ​​from multiple speech recognition networks, the total loss function value used to train the speech enhancement network can be obtained, thus enabling the updating of the speech enhancement network's parameters based on the total loss function value.

[0011] In this scheme, during the training of the speech enhancement network, multiple parallel speech recognition networks are connected after the speech enhancement network. Each of these different speech recognition networks processes the speech data output by the speech enhancement network, and then a loss function is constructed for each speech recognition network based on its output. Finally, by combining the loss functions of multiple speech recognition networks to train the speech enhancement network, the generalization and versatility of the speech enhancement network can be effectively improved, enabling it to be applied to speech recognition networks that have not undergone joint training.

[0012] In one possible implementation, the parameters of multiple speech recognition networks remain unchanged during the training of the speech enhancement network. That is, the multiple speech recognition networks are not trained together with the speech enhancement network. By freezing the parameters of the multiple speech recognition networks, changes in the parameters of the speech recognition networks can be prevented from affecting the parameters of the speech enhancement network. This effectively simulates the application scenario where the speech recognition network is a black-box system (i.e., the parameters of the speech recognition network are unknown and cannot be adjusted), improving the generalization ability of the speech enhancement network. This allows the finally trained speech enhancement network to be applied to other speech recognition networks that have not undergone joint training.

[0013] In another possible implementation, multiple speech recognition networks are pre-trained networks. That is, multiple speech recognition networks are all pre-trained. For example, multiple speech recognition networks can be pre-trained based on training samples, and multiple speech recognition networks have good speech recognition performance after training.

[0014] In this solution, a pre-trained speech recognition network is used in conjunction with a speech enhancement network. During training, only the speech enhancement network is trained. This ensures that the speech recognition network outputs high-accuracy speech recognition results during training, preventing low-accuracy results from negatively impacting the training performance of the speech enhancement network. Furthermore, after the speech enhancement network is trained, it is typically used in conjunction with a trained speech recognition network. Therefore, combining the training process with the pre-trained speech recognition network effectively simulates subsequent application scenarios for the speech enhancement network, improving its speech enhancement performance in real-world applications.

[0015] In another possible implementation, multiple speech recognition networks correspond to different types of network structures. For example, the different speech recognition networks could be of the following types: Listen-Attend Spell (LAS) network, Connectionist Temporal Classification (CTC) network, Attention-based Encoder-Decoder (AED) network, Recurrent Neural Network Transducer (RNN-T) network, and Transformer Transducer network.

[0016] In this scheme, by setting the multiple speech recognition networks participating in the training of the speech enhancement network to correspond to different types of network structures, the structural differences between the multiple speech recognition networks can be maximized, thereby improving the versatility of the trained speech enhancement network and preventing the speech enhancement network from being trained to only improve the speech recognition performance of a certain part of the speech recognition network with similar structures.

[0017] In another possible implementation, the structural similarity between different speech recognition networks is less than a first threshold. That is, for the multiple speech recognition networks connected to the speech enhancement network, the structural similarity between any two speech recognition networks is less than the first threshold.

[0018] In this scheme, by comparing the structural similarity between speech recognition networks and selecting multiple speech recognition networks with low structural similarity to assist in training the speech enhancement network, the structural differences between multiple speech recognition networks can be maximized, thereby improving the versatility of the trained speech enhancement network and preventing the speech enhancement network from being trained to only improve the speech recognition performance of a certain part of the structurally similar speech recognition network.

[0019] In another possible implementation, the total loss function value is obtained by performing a weighted sum of the loss function values ​​corresponding to each speech recognition network. That is, each speech recognition network can be assigned a weight, and then the weight corresponding to each speech recognition network is multiplied by the loss function value to obtain the weighted loss function value for each speech recognition network; finally, the weighted loss function values ​​corresponding to each speech recognition network are summed to obtain the total loss function value.

[0020] In this scheme, the total loss function value is calculated by weighted summation, which can effectively integrate the output results of multiple speech recognition networks. This facilitates the training of speech enhancement networks based on multiple speech recognition networks, thereby improving the training effect of speech enhancement networks.

[0021] In another possible implementation, the weight value of each speech recognition network is positively correlated with the loss function value of that network. That is, the larger the loss function value, the larger the weight value; conversely, the smaller the loss function value, the smaller the weight value.

[0022] In this scheme, by setting a positive correlation between the weight values ​​of the speech recognition network and the loss function value, greater weights can be assigned to speech recognition networks with larger loss function values. This makes the speech enhancement network more inclined to improve speech recognition networks with relatively poor recognition performance during the training process, thereby improving the adaptability of the speech enhancement network to various speech recognition networks.

[0023] In another possible implementation, the weight values ​​corresponding to each speech recognition network are also related to the weight values ​​of each speech recognition network during historical training iterations. That is, when determining the weight values ​​corresponding to the speech recognition network, it is necessary to base them on the weight values ​​of the speech recognition network during historical training iterations. In other words, the weight values ​​corresponding to the speech recognition network change based on the weight values ​​during historical training iterations, and will not differ too much from the weight values ​​during historical training iterations.

[0024] In this scheme, by introducing the weight values ​​of the speech recognition network during the historical iteration training process, the weight values ​​of the speech recognition network can be ensured to change relatively smoothly during the training process, thereby avoiding excessive changes in the total loss function value that would affect the training effect of the speech enhancement network.

[0025] In another possible implementation, among multiple speech recognition networks, the weight value corresponding to the speech recognition network whose loss function value is greater than the second threshold is 0.

[0026] In this scheme, a threshold is set for the loss function values ​​of multiple speech recognition networks. When the loss function value of a speech recognition network exceeds the threshold, the loss function value is discarded. That is, the total loss function value is no longer determined based on the loss function value, thereby avoiding the negative impact of bad learning samples on the training of the speech enhancement network.

[0027] In another possible implementation, the second threshold is N times the target loss function value, where the target loss function value is the smallest loss function value among multiple loss function values ​​corresponding to multiple speech recognition networks, and N is greater than 1.

[0028] Secondly, a speech processing method is provided for speech recognition. In this method, an electronic device first obtains a speech recognition request, which includes a first speech and is used to request the recognition of the first speech. For example, in scenarios such as caption generation, meeting minutes, speech translation, and human-computer dialogue, the electronic device responsible for speech recognition processing (such as a server or smartphone) can obtain a speech recognition request sent by a microphone, which includes the first speech collected by the microphone. The electronic device responsible for speech recognition processing and the microphone can be integrated into the same device (e.g., integrated into the same smartphone) or they can be separate devices (e.g., the electronic device is a server, and the microphone is a voice recorder).

[0029] Then, the electronic device processes the first speech through the trained speech enhancement network to obtain the enhanced first speech. The trained speech enhancement network is obtained by training a speech enhancement network, which is connected to multiple parallel speech recognition networks during training. The inputs of these multiple speech recognition networks are the outputs of the speech enhancement network, and the outputs of these multiple speech recognition networks are used to construct the loss function for training the speech enhancement network.

[0030] Finally, the electronic device performs speech recognition on the enhanced first speech (e.g., by using a speech recognition network) to obtain the recognized text corresponding to the first speech. In other words, the recognized text is the text obtained after the electronic device performs speech recognition on the first speech. This recognized text can then be used to perform subsequent tasks, such as generating subtitles, meeting minutes, or speech translation results.

[0031] In this scheme, during speech processing, a trained speech enhancement network is first used to enhance the speech, followed by speech recognition. Furthermore, the trained speech enhancement network is based on multiple parallel speech recognition networks, and the outputs of these networks are jointly used to construct the loss function for training the speech enhancement network. This effectively improves the generalization and versatility of the trained network. Thus, performing speech processing based on the trained speech enhancement network significantly improves speech recognition performance, and the network can be extended to speech recognition networks that have not undergone joint training, thereby meeting the needs of various scenarios.

[0032] In one possible implementation, among the multiple speech recognition networks used to train the speech enhancement network, different speech recognition networks correspond to different types of network structures.

[0033] In another possible implementation, the structural similarity between different speech recognition networks used to train the speech enhancement network is less than a first threshold.

[0034] In another possible implementation, the trained speech enhancement network is trained based on the total loss function value, which is obtained by weighted summation of multiple loss function values ​​corresponding to multiple speech recognition networks. These multiple loss function values ​​are obtained based on the outputs of multiple speech recognition networks.

[0035] In another possible implementation, the weight value of each speech recognition network in the multiple speech recognition networks is positively correlated with the loss function value of each speech recognition network.

[0036] In another possible implementation, the weight value corresponding to each speech recognition network is also related to the weight value of each speech recognition network in the historical iteration training process.

[0037] In another possible implementation, among multiple speech recognition networks, the weight value corresponding to the speech recognition network whose loss function value is greater than the second threshold is 0.

[0038] In another possible implementation, the second threshold is N times the target loss function value, where the target loss function value is the smallest loss function value among multiple loss function values ​​corresponding to multiple speech recognition networks, and N is greater than 1.

[0039] In another possible implementation, the parameters of multiple speech recognition networks remain unchanged during the training of the speech enhancement network.

[0040] In another possible implementation, multiple speech recognition networks are pre-trained networks.

[0041] Thirdly, a model training device is provided, comprising: an acquisition module for acquiring training samples, the training samples including speech samples and corresponding real text; a processing module for inputting the speech samples into a speech enhancement network to obtain enhanced speech samples; the processing module is further configured to input the enhanced speech samples into multiple different speech recognition networks to obtain the speech recognition results output by each speech recognition network; and a training module for training the speech enhancement network based on the total loss function value to obtain a parameter-updated speech enhancement network; wherein the total loss function value is obtained based on the loss function value corresponding to each of the multiple speech recognition networks, and the loss function value corresponding to each speech recognition network is obtained based on the speech recognition results output by each speech recognition network and the real text.

[0042] In one possible implementation, the parameters of multiple speech recognition networks remain unchanged during the training of the speech enhancement network.

[0043] In another possible implementation, multiple speech recognition networks are pre-trained networks.

[0044] In another possible implementation, multiple speech recognition networks are used, with different speech recognition networks corresponding to different types of network structures.

[0045] In another possible implementation, the structural similarity between different speech recognition networks is less than a first threshold.

[0046] In another possible implementation, the total loss function value is obtained by performing a weighted summation of the loss function values ​​corresponding to each speech recognition network.

[0047] In another possible implementation, the weight values ​​of each speech recognition network are positively correlated with the loss function values ​​of each speech recognition network.

[0048] In another possible implementation, the weight value corresponding to each speech recognition network is also related to the weight value of each speech recognition network in the historical iteration training process.

[0049] In another possible implementation, among multiple speech recognition networks, the weight value corresponding to the speech recognition network whose loss function value is greater than the second threshold is 0.

[0050] In another possible implementation, the second threshold is N times the target loss function value, where the target loss function value is the smallest loss function value among multiple loss function values ​​corresponding to multiple speech recognition networks, and N is greater than 1.

[0051] Fourthly, a speech processing apparatus is provided, comprising: an acquisition module for acquiring a speech recognition request, the speech recognition request including a first speech; a processing module for processing the first speech through a trained speech enhancement network to obtain enhanced speech, wherein the trained speech enhancement network is obtained by training the speech enhancement network, and the speech enhancement network is connected to multiple parallel speech recognition networks during the training process, the inputs of the multiple speech recognition networks are the outputs of the speech enhancement network, and the outputs of the multiple speech recognition networks are used to construct a loss function for training the speech enhancement network; the processing module is further configured to perform speech recognition on the enhanced speech to obtain recognized text corresponding to the first speech.

[0052] In one possible implementation, among the multiple speech recognition networks used to train the speech enhancement network, different speech recognition networks correspond to different types of network structures.

[0053] In another possible implementation, the structural similarity between different speech recognition networks used to train the speech enhancement network is less than a first threshold.

[0054] In another possible implementation, the trained speech enhancement network is trained based on the total loss function value, which is obtained by weighted summation of multiple loss function values ​​corresponding to multiple speech recognition networks. These multiple loss function values ​​are obtained based on the outputs of multiple speech recognition networks.

[0055] In another possible implementation, the weight value of each speech recognition network in the multiple speech recognition networks is positively correlated with the loss function value of each speech recognition network.

[0056] In another possible implementation, the weight value corresponding to each speech recognition network is also related to the weight value of each speech recognition network in the historical iteration training process.

[0057] In another possible implementation, among multiple speech recognition networks, the weight value corresponding to the speech recognition network whose loss function value is greater than the second threshold is 0.

[0058] In another possible implementation, the second threshold is N times the target loss function value, where the target loss function value is the smallest loss function value among multiple loss function values ​​corresponding to multiple speech recognition networks, and N is greater than 1.

[0059] In another possible implementation, the parameters of multiple speech recognition networks remain unchanged during the training of the speech enhancement network.

[0060] In another possible implementation, multiple speech recognition networks are pre-trained networks.

[0061] Fifthly, an electronic device is provided, which may include a processor and a memory coupled together. The memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method described in any implementation of the first or second aspect is implemented. For details regarding the steps in each possible implementation of the first or second aspect executed by the processor, please refer to the first or second aspect; further details will not be elaborated here.

[0062] A sixth aspect provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the method described in either the first or second aspect.

[0063] A seventh aspect provides a circuit system including a processing circuit configured to perform the method described in any implementation of the first or second aspect.

[0064] Eighthly, a computer program product is provided that, when run on a computer, causes the computer to perform the method described in either the first or second aspect.

[0065] A ninth aspect provides a chip system including a processor for supporting a server or threshold acquisition device in implementing the functions involved in any implementation of the first or second aspect described above, such as transmitting or processing data and / or information involved in the methods described above. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the server or communication device. The chip system may be composed of chips or may include chips and other discrete devices.

[0066] The beneficial effects of aspects two through nine mentioned above can be referred to the introduction of aspect one above, and will not be repeated here. Attached Figure Description

[0067] Figure 1 A schematic diagram illustrating the working principle of an automatic speech recognition system provided in this application;

[0068] Figure 2 A schematic diagram illustrating the process of independently training a speech enhancement network and a speech recognition network, as provided in this application;

[0069] Figure 3 A schematic diagram illustrating one application scenario provided in this application;

[0070] Figure 4 A flowchart illustrating a model training method provided in this application;

[0071] Figure 5 A schematic diagram illustrating the process of training a speech enhancement network provided in this application;

[0072] Figure 7 A schematic diagram illustrating the application of a trained speech enhancement network provided in this application;

[0073] Figure 8 A schematic diagram of the structure of a speech enhancement network provided in this application;

[0074] Figure 9 A schematic diagram illustrating the operation comparison of a depth filter provided in this application;

[0075] Figure 10A A schematic diagram of the structure of a model training device provided in this application;

[0076] Figure 10B A schematic diagram of the structure of a voice processing device provided in this application;

[0077] Figure 11 A schematic diagram of an electronic device 1100 provided in this application;

[0078] Figure 12 A schematic diagram of the structure of the chip provided in this application;

[0079] Figure 13 This is a schematic diagram of the structure of a computer-readable storage medium provided in this application. Detailed Implementation

[0080] This application involves a great deal of knowledge related to speech and neural networks. To better understand the solution of this application, the relevant terms and concepts of speech and neural networks that may be involved in this application will be introduced below. It should be understood that the related concept explanations may be limited due to the specific circumstances of this application, but it does not mean that this application is limited to only that specific situation. The specific circumstances of different embodiments may also differ, and no specific limitation is made here.

[0081] (1) Speech recognition

[0082] Speech recognition refers to converting the lexical content of human speech into computer-readable input, such as character sequences or binary codes.

[0083] (2) Speech recognition results

[0084] Speech recognition results refer to the text obtained by recognizing speech through speech recognition technology.

[0085] (3) Speech enhancement

[0086] Speech enhancement refers to the technique of extracting useful speech signals from a noisy background and suppressing or reducing noise interference when a speech signal is interfered with or even submerged by various types of noise. In this embodiment, speech enhancement refers to performing enhancement processing on the speech signal to improve its quality and increase its signal-to-noise ratio, thereby enabling the speech recognition network to output more accurate recognized text based on the enhanced speech signal.

[0087] (4) Neural Network

[0088] A neural network can be composed of neural units, specifically understood as a neural network with input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. Neural networks with many hidden layers are called deep neural networks (DNNs). The function of each layer in a neural network can be expressed mathematically. To describe it physically, each layer in a neural network can be understood as transforming the input space (the set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space. These five operations are: 1. Dimensionality increase / decrease; 2. Magnification / scaling; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are... Operation 4 is completed using "+b", and operation 5 is implemented using "a()". The term "space" is used here because the objects being classified are not individual things, but a class of things; space refers to the set of all individuals within this class of things. Here, W is the weight matrix of each layer of the neural network, where each value represents the weight of a neuron in that layer. This matrix W determines the spatial transformation from the input space to the output space, as described above; that is, the W of each layer of the neural network controls how the space is transformed. The purpose of training the neural network is to ultimately obtain the weight matrices of all layers of the trained neural network. Therefore, the training process of a neural network is essentially learning how to control spatial transformation, more specifically, learning the weight matrix.

[0089] (5) Convolutional Neural Network (CNN)

[0090] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as performing convolution between a trainable filter and an input feature map. A convolutional layer is a layer of neural units in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neural unit may be connected to only a portion of the neighboring neural units. A convolutional layer typically contains several feature planes, each of which can be composed of a series of rectangularly arranged neural units. Neural units on the same feature plane share weights; these shared weights are the convolutional kernel.

[0091] Convolutional kernels can be initialized as matrices of random size, and during the training of a convolutional neural network, they can learn appropriate weights. Furthermore, sharing weights directly reduces the number of connections between layers in the convolutional neural network, while also lowering the risk of overfitting.

[0092] (6) Recurrent Neural Network (RNN)

[0093] Recurrent neural networks (RNNs) are a type of recursive neural network that takes sequence data as input, recurses in the direction of sequence evolution, and connects all nodes (recurrent units) in a chain-like manner.

[0094] Recurrent neural networks (RNNs) possess memory, parameter sharing, and Turing completeness, giving them an advantage in learning the nonlinear features of sequences. RNNs have applications in Natural Language Processing (NLP), such as speech recognition, language modeling, and machine translation, and are also used in various time series forecasting applications.

[0095] (7) Long Short-Term Memory (LSTM) artificial neural network

[0096] LSTM is a type of time-recurrent neural network specifically designed to address the long-term dependency problem inherent in general recurrent neural networks. It is typically used to process and predict important events in time series with very long intervals and delays.

[0097] (8) Loss Function

[0098] During neural network training, to ensure the network's output closely approximates the desired predicted value, we compare the network's current prediction with the target value. Based on the difference, we update the weight matrix of each layer (usually pre-configuring parameters before the initial update). For example, if the network's prediction is too high, we adjust the weight matrix to lower it, continuing this adjustment process until the network accurately predicts the target value. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the basis of the loss function or objective function. These are crucial equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, making neural network training a process of minimizing this loss as much as possible.

[0099] (9) Backpropagation algorithm

[0100] During the training of a neural network, the back propagation (BP) algorithm can be used to correct the parameters in the initial neural network model, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss. By backpropagating this error loss information, the parameters in the initial neural network model are updated, thus bringing the error loss to a convergence. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0101] (10) Black-box system

[0102] A black-box system is a system whose input and output ports are accessible, but whose internal parameters are unknown and cannot be adjusted. For example, a neural network is a black-box system; one can only input data into the neural network and obtain the output data after the neural network processes the input data, but cannot know or adjust the parameters of the neural network.

[0103] (11) White-box system

[0104] A white-box system is the opposite of a black-box system. A white-box system refers to a system whose internal parameters are known and can be adjusted. For example, a neural network is a white-box system because its structure and parameters are known, and the parameters can be adjusted.

[0105] (12) Attention Network

[0106] Attention networks are network models that utilize attention mechanisms to improve model training speed. Currently, typical attention networks include the Transformer model. Models applying attention mechanisms can assign different weights to each part of the input sequence, thereby extracting more important feature information from the input sequence and resulting in a more accurate output.

[0107] Specifically, there are currently two ways to use speech enhancement networks in conjunction with speech recognition networks in related technologies.

[0108] In the first approach, speech enhancement and speech recognition are treated as two independent tasks. A separate speech enhancement network is used to improve the perceived quality and intelligibility of speech using speech enhancement metrics, making the processed audio sound clearer to the human ear. Then, the audio processed by the speech enhancement network is input into the speech recognition network to obtain the recognition result output by the speech recognition network. In other words, the speech enhancement network and the speech recognition network are trained independently, and the training objective of the speech enhancement network is to output speech that is more easily recognized by the human ear.

[0109] For example, please refer to Figure 2 , Figure 2 This application provides a schematic diagram illustrating the process of independently training a speech enhancement network and a speech recognition network. (See attached diagram.) Figure 2 As shown, during the training of the speech enhancement network and the speech recognition network, noisy speech (e.g., noisy speech obtained by artificially adding noise to clean speech) is first input into the speech enhancement network. The speech enhancement network enhances the noisy speech and outputs denoised speech. Then, the denoised speech output by the speech enhancement network is input into the speech recognition network, which performs speech recognition and outputs the recognized text. Thus, based on the denoised speech output by the speech enhancement network and the clean speech corresponding to the noisy speech, a speech enhancement loss function can be constructed, and the speech enhancement network can be trained based on this loss function. Furthermore, based on the recognized text output by the speech recognition network and the real text corresponding to the noisy speech, a speech recognition loss function can be constructed, and the speech recognition network can be trained based on this loss function. It can be seen that although the denoised speech output by the speech enhancement network is used as input to the speech recognition network, the loss functions used by the speech enhancement network and the speech recognition network are different, meaning that the training objectives of the two networks are different.

[0110] In the second approach, speech enhancement and speech recognition are treated as a whole. After initial pre-training of the speech enhancement network and the speech recognition network separately, the speech enhancement network is connected before the speech recognition network, thereby jointly training the two networks and simultaneously adjusting their parameters. In other words, both the speech enhancement network and the speech recognition network are trained simultaneously based on the same loss function, and their parameters are updated together during training.

[0111] In the first approach described above, the inventors discovered that human hearing and machine cognition differ, and speech enhancement that is more pleasing to the human ear often fails to improve machine cognition. Because the speech enhancement network is trained solely to produce speech that is more conducive to human hearing, without considering the loss function of speech recognition, it is ultimately found that adding the speech enhancement network does not significantly improve the accuracy of speech recognition, and in some cases may even impair its performance.

[0112] In the second approach described above, while joint training of the speech enhancement network and the speech recognition network can help improve the accuracy of speech recognition, it alters the parameters of the speech recognition network, reducing its robustness when used independently. Furthermore, it decreases the robustness of the entire system (i.e., the combination of the speech enhancement and speech recognition networks) when facing data from a wider range of real-world scenarios, thus impairing the accuracy of speech recognition outside the training context. Moreover, in this second approach, joint training of the speech enhancement and speech recognition networks requires acquiring the parameters of the speech recognition network and performing iterative learning, making it unsuitable for scenarios where the speech recognition network is a black-box system (i.e., the specific parameters of the speech recognition network cannot be obtained).

[0113] In general, the speech enhancement networks added in related technologies are usually used in conjunction with specific speech recognition networks. The applicability of speech enhancement networks is poor, and there are few scenarios in which they can be applied.

[0114] Based on this, this application provides a model training method. During the training of the speech enhancement network, multiple parallel speech recognition networks are connected after the speech enhancement network. Each of these different speech recognition networks performs recognition processing on the speech data output by the speech enhancement network (i.e., the input to each speech recognition network is the speech data output by the speech enhancement network). Then, a loss function is constructed for each speech recognition network based on its output speech recognition result. Finally, by training the speech enhancement network using the combined loss functions of multiple speech recognition networks, the generalization and versatility of the speech enhancement network can be effectively improved, enabling it to be applied to speech recognition networks that have not undergone joint training.

[0115] The model training method provided in this application is described in detail below with reference to the accompanying drawings.

[0116] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating one application scenario provided by this application. For example... Figure 3 As shown, after acquiring speech data including the user's original speech and noise through a sound pickup device, the speech data is input into a speech enhancement network trained using the model training method provided in this embodiment. The speech enhancement network enhances the speech data to obtain enhanced speech data. Then, the enhanced speech data is input into a speech recognition network, which outputs recognized text. This recognized text can be applied in various scenarios, such as subtitle generation, meeting minutes, speech translation, and human-computer dialogue.

[0117] For example, in fields such as news broadcasting, live streaming, or film production, speech enhancement networks and speech recognition networks are used to recognize the user's spoken lines in real time, obtaining recognized text. This recognized text is then used to generate subtitles, enabling synchronized audio and video playback.

[0118] For example, in video or audio conferencing scenarios, speech enhancement and speech recognition networks can be used in real time to recognize the content spoken by participants, resulting in recognized text. After the meeting, this recognized text can be used to generate meeting minutes, allowing participants to summarize the meeting content.

[0119] For example, in scenarios involving speech translation or human-computer dialogue, speech in the current environment is recognized in real time using speech enhancement networks and speech recognition networks to obtain recognized text. This recognized text can then be further applied to perform translations in the relevant language, or the machine can perform corresponding actions based on instructions from the recognized text.

[0120] In summary, the speech enhancement network trained by the model training method provided in this application can be applied to enhance speech data in order to improve the accuracy of language data recognition, so that the recognized text can be applied to various downstream natural speech processing tasks.

[0121] The model training method provided in this application can be applied to electronic devices, especially those that need to perform speech recognition tasks. Examples of such electronic devices include servers, personal computers, smartphones, PCs, laptops, etc.

[0122] As one possible implementation, the model training method provided in this application can also be applied to a cluster system consisting of multiple servers, where the servers in the cluster system are used together to execute the model training method, for example, multiple electronic devices respectively execute one or more steps in the model training process.

[0123] The above describes the scenarios and devices used in the model training method provided in this application. The specific process of the model training method provided in this application will be described next.

[0124] Please see Figure 4 , Figure 4 This is a flowchart illustrating a model training method provided in this application. Figure 4 As shown, the process of this model training method includes the following steps 401-404.

[0125] Step 401: Obtain training samples, which include speech samples and the corresponding real text.

[0126] In this embodiment, before training the speech enhancement network, training samples for training can be obtained. A training sample includes a speech sample and its corresponding real text. The speech sample refers to a segment of speech data with noise, such as a user's speech captured by a microphone in a noisy environment. The real text is the actual text content corresponding to the speech content in the speech sample. For example, assuming the speech sample is a segment of speech data about a user saying "Please play my favorite music," the corresponding real text would be "Please play my favorite music." In other words, the real text corresponding to the speech sample is actually the recognized text obtained by the speech recognition network after recognizing the speech sample.

[0127] Step 402: Input the speech sample into the speech enhancement network to obtain the enhanced speech sample.

[0128] In this embodiment, the speech enhancement network is the network to be trained, and it can enhance the input speech samples and output the enhanced speech samples. The speech enhancement network is mainly used to enhance the input speech samples, improve the quality of the speech samples, and increase the signal-to-noise ratio of the speech samples, thereby enabling the speech recognition network to output more accurate recognized text based on the enhanced speech samples.

[0129] Specifically, the speech enhancement network is a neural network, such as a convolutional neural network, a recurrent neural network, or an LSTM neural network. This embodiment does not limit the specific structure of the speech enhancement network.

[0130] Step 403: Input the enhanced speech samples into different speech recognition networks to obtain the speech recognition results output by each speech recognition network.

[0131] In this embodiment, multiple different speech recognition networks are connected after the speech enhancement network, and the input of each speech recognition network is the enhanced speech sample output by the speech enhancement network, that is, the input of multiple speech recognition networks is the same.

[0132] Among them, multiple speech recognition networks are multiple speech recognition networks with different network structures. Therefore, each speech recognition network may output different speech recognition results after performing speech recognition on the enhanced speech samples.

[0133] For example, please refer to Figure 5 , Figure 5 This is a schematic diagram illustrating a training process for a speech enhancement network provided in this application. Figure 5 In this diagram, the speech enhancement network is followed by three speech recognition networks: speech recognition network 1, speech recognition network 2, and speech recognition network 3. The input to these three speech recognition networks is the enhanced speech sample output by the speech enhancement network. Specifically, speech recognition network 1 performs speech recognition processing on the enhanced speech sample and outputs speech recognition result 1; speech recognition network 2 performs speech recognition processing on the enhanced speech sample and outputs speech recognition result 2; and speech recognition network 3 performs speech recognition processing on the enhanced speech sample and outputs speech recognition result 3.

[0134] It should be noted that, Figure 5 This example uses three speech recognition networks. This embodiment does not limit the number of speech recognition networks connected after the speech enhancement network. The specific number can be determined based on the available number of speech recognition networks and computing resources in the actual application. Generally, the more speech recognition networks connected after the speech enhancement network, the more computing resources are required during training, and the better the final performance of the trained speech enhancement network.

[0135] Step 404: Train the speech enhancement network based on the total loss function value to obtain the speech enhancement network with updated parameters.

[0136] In this embodiment, the total loss function value is obtained based on the loss function value corresponding to each of the multiple speech recognition networks. The loss function value for each speech recognition network is derived from the speech recognition result output by each network and the actual text. In other words, after processing the enhanced speech sample through each speech recognition network, the speech recognition result output by each network can be obtained. Thus, based on the actual text corresponding to the speech sample and the speech recognition result output by each network, a loss function for each network can be constructed, and the loss function value for each network for that speech sample can be calculated. Finally, by combining the multiple loss function values ​​from the multiple speech recognition networks, the total loss function value used to train the speech enhancement network can be obtained, thereby updating the parameters of the speech enhancement network based on the total loss function value. The method of updating the parameters of the speech enhancement network based on the total loss function value can be implemented using the backpropagation algorithm; this embodiment does not specifically limit this method.

[0137] In summary, the output of the speech enhancement network is not directly used to construct the loss function for training the speech enhancement network. Instead, it is input into multiple speech recognition networks, and the final loss function for training the speech enhancement network is based on the outputs of these multiple speech recognition networks. Therefore, since the loss function for training the speech enhancement network is constructed based on the outputs of multiple speech recognition networks, the training objective of the speech enhancement network is actually to improve the recognition accuracy of these multiple speech recognition networks. In other words, during training, the speech enhancement network learns how to output appropriate enhanced speech so that when the multiple speech recognition networks perform speech recognition based on the enhanced speech, they can obtain more accurate speech recognition results.

[0138] In practical applications, a loss function can be constructed for each speech recognition network, and the total loss function for the speech enhancement network used for training can be obtained by combining the loss functions of each speech recognition network. Thus, after determining the total loss function for the speech enhancement network based on the loss functions of each speech recognition network, the total loss function value for each iteration of training can be calculated.

[0139] In other words, the goal of training a speech enhancement network based on the total loss function value is to enable the enhanced speech data output by the speech enhancement network after performing enhancement processing on the speech data to effectively improve the recognition performance of multiple speech recognition networks, so that the speech enhancement network has good adaptability to multiple speech recognition networks.

[0140] For example, such as Figure 5As shown, for speech recognition result 1 output by speech recognition network 1, the loss function value 1 corresponding to speech recognition network 1 can be determined by calculating the difference between speech recognition result 1 and the real text; for speech recognition result 2 output by speech recognition network 2, the loss function value 2 corresponding to speech recognition network 2 can be determined by calculating the difference between speech recognition result 2 and the real text; for speech recognition result 3 output by speech recognition network 3, the loss function value 3 corresponding to speech recognition network 3 can be determined by calculating the difference between speech recognition result 3 and the real text. Finally, by combining loss function value 1, loss function value 2, and loss function value 3, the total loss function value can be obtained.

[0141] In this scheme, during the training of the speech enhancement network, multiple parallel speech recognition networks are connected after the speech enhancement network. Each of these different speech recognition networks processes the speech data output by the speech enhancement network, and then a loss function is constructed for each speech recognition network based on its output. Finally, by combining the loss functions of multiple speech recognition networks to train the speech enhancement network, the generalization and versatility of the speech enhancement network can be effectively improved, enabling it to be applied to speech recognition networks that have not undergone joint training.

[0142] Furthermore, since this embodiment uses multiple speech recognition networks to jointly train the speech enhancement network, the resulting speech enhancement network has strong versatility. Thus, in practical applications, even if the target speech recognition network is a black-box system and cannot be trained one-to-one with the speech enhancement network, the effectiveness of directly using the speech enhancement network with the target speech recognition network can still be guaranteed when the speech enhancement network is used in conjunction with the target speech recognition network, thereby expanding the scenarios in which the speech enhancement network can be applied.

[0143] Essentially, the model training method provided in this embodiment is actually a dynamic multi-teacher training method, which uses multiple speech recognition networks as teacher networks and speech enhancement networks as student networks, and trains the student networks based on the teacher networks. See Table 1 below, which shows the differences between existing distillation methods and the dynamic multi-teacher training method provided in this embodiment.

[0144] Table 1

[0145]

[0146] It should be noted that steps 401-404 above actually describe one round of training iterations for the speech enhancement network. In the actual training process of the speech enhancement network, multiple rounds of training iterations will be performed, and the process of each round can refer to steps 401-404 above. Finally, the speech enhancement network is trained using the total loss function. There are several ways to determine the extent of training; some termination conditions for speech enhancement network training are provided below.

[0147] (1) The total loss function value reaches the preset threshold.

[0148] Since the total loss function value used to train the speech enhancement network is obtained based on the loss functions corresponding to multiple speech recognition networks, a threshold (e.g., 0.03) can be set in advance for the total loss function value. During the iterative training of the speech enhancement network, after each training session, it is determined whether the total loss function value obtained in the current training round has reached the threshold. If it has not reached the threshold, training continues. If it reaches the preset threshold, training is terminated. Then, the network parameter values ​​of the speech enhancement network determined in the current training round are used as the network parameter values ​​of the finally trained speech enhancement network.

[0149] (2) The total loss function begins to converge.

[0150] During the training of the speech enhancement network, if the difference between the total loss function value obtained in the current training round and the total loss function value obtained in the previous training round is within a preset range (e.g., within 0.01), then the total loss function used to train the speech enhancement network is considered to have converged, and the training can be terminated. Then the network parameter values ​​of the speech enhancement network determined in the current training round will be used as the network parameter values ​​of the finally trained speech enhancement network.

[0151] (3) The training has reached the preset number of times.

[0152] In this approach, the number of iterations for training the speech enhancement network can be pre-configured (e.g., 1000 times). During the iterative training of the speech enhancement network, after each training round, the values ​​of the network parameters for that round are stored until the preset number of iterations is reached. Then, the speech enhancement networks obtained from each round are validated using test data, and the network parameter values ​​with the best performance are selected as the final network parameter values ​​for that speech enhancement network.

[0153] Optionally, during the training of the speech enhancement network, the parameters of multiple speech recognition networks remain unchanged; that is, the multiple speech recognition networks are not trained together with the speech enhancement network. By freezing the parameters of multiple speech recognition networks, changes in the parameters of the speech recognition networks can be prevented from affecting the parameters of the speech enhancement network. This effectively simulates the application scenario of the speech recognition network as a black-box system (i.e., the parameters of the speech recognition network are unknown and cannot be adjusted), improving the generalization ability of the speech enhancement network. This allows the finally trained speech enhancement network to be applied to other speech recognition networks that have not undergone joint training.

[0154] For example, multiple speech recognition networks are pre-trained networks, meaning they are all pre-trained. For instance, each speech recognition network may have been pre-trained based on training samples, and all of them exhibit good speech recognition performance after training. In other words, by using pre-trained speech recognition networks in conjunction with a speech enhancement network, and training only the speech enhancement network during the training process, the speech recognition network can output high-accuracy speech recognition results during the training phase, avoiding the impact of low-accuracy speech recognition results on the training performance of the speech enhancement network. Furthermore, after the speech enhancement network is trained, it is usually used in conjunction with a trained speech recognition network. Therefore, combining the training of the speech enhancement network with the pre-trained speech recognition network during the training process can effectively simulate the subsequent application scenarios of the speech enhancement network, improving its speech enhancement effect in practical applications.

[0155] The above describes the process of determining the total loss function value based on the output results of multiple speech recognition networks and training the speech enhancement network. The following will describe how the total loss function is determined based on the output results of multiple speech recognition networks in step 404 above.

[0156] In one possible example, the total loss function value used to train the speech enhancement network is obtained by performing a weighted sum of the loss function values ​​corresponding to each speech recognition network. That is, each speech recognition network can be assigned a weight, and then the weight of each network is multiplied by its corresponding loss function value to obtain the weighted loss function value for that network. Finally, the weighted loss function values ​​for each network are summed to obtain the total loss function value. Since the loss function value for each speech recognition network is known, what actually needs to be determined in each training iteration is the weight of each network.

[0157] Optionally, for multiple speech recognition networks, the weight value of each speech recognition network is positively correlated with the loss function value of each speech recognition network. That is, the larger the loss function value of a speech recognition network, the larger its weight value; the smaller the loss function value of a speech recognition network, the smaller its weight value.

[0158] In this scheme, by setting a positive correlation between the weight values ​​of the speech recognition network and the loss function value, greater weights can be assigned to speech recognition networks with larger loss function values. This makes the speech enhancement network more inclined to improve speech recognition networks with relatively poor recognition performance during the training process, thereby improving the adaptability of the speech enhancement network to various speech recognition networks.

[0159] Optionally, the weight values ​​corresponding to each speech recognition network are also related to the weight values ​​of each speech recognition network in the historical training iterations (i.e., historical weight values). For example, for any speech recognition network, the weight value corresponding to that network in the current training iteration is related to the weight value corresponding to that network in the previous training iteration. In other words, when determining the weight values ​​corresponding to a speech recognition network, it is necessary to base them on the weight values ​​of the speech recognition network in the historical training iterations. That is, the weight values ​​corresponding to the speech recognition network change based on the weight values ​​in the historical training iterations, and will not differ too much from the weight values ​​in the historical training iterations.

[0160] In this scheme, by introducing the weight values ​​of the speech recognition network during the historical iteration training process, the weight values ​​of the speech recognition network can be ensured to change relatively smoothly during the training process, thereby avoiding excessive changes in the total loss function value that would affect the training effect of the speech enhancement network.

[0161] Furthermore, since this embodiment combines the loss function values ​​of multiple speech recognition networks to obtain the total loss function value used to train the speech enhancement network, if the loss function value of a certain speech recognition network is particularly large (i.e., the loss function value of this speech recognition network is much larger than the loss function values ​​of other speech recognition networks), it will generate a large gradient backpropagation during the training process of the speech enhancement network, affecting the training effect of the speech enhancement network.

[0162] Based on this, in this embodiment, a threshold is set for the loss function values ​​of multiple speech recognition networks. When the loss function value of a speech recognition network exceeds the threshold, the loss function value is discarded, that is, the total loss function value is no longer determined based on the loss function value, thereby avoiding the negative impact of bad learning samples on the training of the speech enhancement network.

[0163] For example, in multiple speech recognition networks, the weights of speech recognition networks whose loss function values ​​are greater than the second threshold are 0. Thus, during the calculation of the total loss function value, since the weights of speech recognition networks whose loss function values ​​are greater than the second threshold are 0, the loss function values ​​of those networks no longer affect the magnitude of the total loss function value.

[0164] The second threshold is N times the target loss function value, where the target loss function value is the smallest loss function value among multiple loss function values ​​corresponding to multiple speech recognition networks, and N is greater than 1. In other words, after obtaining the loss function value corresponding to each of the multiple speech recognition networks, the smallest loss function value among these values ​​can be determined as the target loss function value. This determined target loss function value is then multiplied by N to obtain the second threshold. Specifically, the value of N can be, for example, an integer in the range of 5-10; this embodiment does not impose a specific limitation on the value of N.

[0165] To facilitate understanding, the following will explain in detail, with specific examples, the process of determining the total loss function value based on the loss function values ​​of multiple speech recognition networks.

[0166] In one possible example, the process of determining the total loss function value can be divided into 6 parts, and the specific operations of each part are described below.

[0167] 1. Assume that in this embodiment, n speech recognition networks are connected after the speech enhancement network, and these n speech recognition networks are represented as follows: Before training begins, the weights for each speech recognition network are first initialized. In this embodiment, the mean weight 'a' is used. i = (1 / n) is used as the initial weight value for each speech recognition network, meaning that in the first training iteration, the weight value for each speech recognition network is 1 / n. Where, a i Represents the weight value corresponding to the i-th speech recognition network, 0 <i≤n。

[0168] 2. After training begins, the original speech x (i.e., speech with noise) is first input into the speech enhancement network f. se (·), resulting in the enhanced speech x′=f se (x). Then, the enhanced speech x′ is input into the above n speech recognition networks respectively to obtain the speech recognition results. Then, combined with the speech recognition results y i With real text y ref The loss function value L corresponding to each speech recognition network is calculated. i =h(y i ,y ref ).

[0169] 3. Next, the adjustment threshold b is set based on the loss function value corresponding to each speech recognition network. Specifically, assuming β is a manually set loss threshold adjustment coefficient, β is used to determine the adjustment threshold b. In this embodiment, b is designed based on β times the minimum loss function value, that is, b = β * min([L i In this embodiment, the value of β is generally 5 to 10.

[0170] 4. Update the weights of the loss function value for each speech recognition network using the global loss function value: Where α is a manually set weight attenuation coefficient, which can be set by analogy to the integral term in the proportional-integral-derivative (PID) control method commonly used in automatic control systems; a i Let a be the weight value of the i-th speech recognition network in the current training iteration; i Let ' be the weight value of the i-th speech recognition network in the previous training iteration. As can be seen from the formula above, during training, the weight value of the loss function corresponding to each speech recognition network is dynamically adjusted based on the historical weight values ​​and the current loss function value. The weight decay coefficient α is used to incorporate historical weight values, thus making the change in the weight value of the current loss function value relatively stable.

[0171] 5. Based on the loss function value L of each speech recognition network i To calculate Boolean quantity m i Here, m is used to dynamically control the impact of the loss function value of the speech recognition network. If the loss function value of the speech recognition network is less than the threshold b, then the m value corresponding to that speech recognition network is set. i =1, so that the loss function value of the speech recognition network can participate normally in training; if the loss function value of the speech recognition network is less than the threshold b, then set the m value corresponding to the speech recognition network to 1. i =0, so that the loss function value of the speech recognition network does not participate in the backpropagation of the front-end speech enhancement network. This is achieved by designing a Boolean quantity m. i This can prevent a large gradient backpropagation from an abnormal loss function value, thereby eliminating the influence of bad samples and ensuring the training effect of the speech enhancement network.

[0172] 6. At the end of each training iteration, the total loss function value used to train the speech enhancement network is calculated by combining the loss function values, weight values, and Boolean values ​​of all the speech recognition networks mentioned above. The total loss function value L is used for backpropagation to update the parameters of the speech enhancement network.

[0173] The above describes the process of determining the total loss function value used to train the speech enhancement network based on the loss function values ​​of multiple speech recognition networks. In actual training, the selection of multiple speech recognition networks also affects the training effect of the speech enhancement network; therefore, the specific method for selecting multiple speech recognition networks in this embodiment will be described below.

[0174] Specifically, during training, a speech enhancement network learns how to improve the speech recognition performance of various speech recognition networks by outputting enhanced speech data. If the structures of the multiple speech recognition networks connected to the speech enhancement network during training are more similar, the network is more likely to be trained to improve the performance of these structurally similar networks, while having limited impact on other network structures. Therefore, to ensure the versatility of the speech enhancement network, the structural differences between the connected speech recognition networks should be set as large as possible during training.

[0175] Furthermore, each of the multiple speech recognition networks needs to have a high speech recognition capability, and the speech recognition capabilities of the multiple speech recognition networks should be similar to avoid speech recognition networks with lower speech recognition capabilities outputting significantly different speech recognition results, which would affect the training effect of the speech enhancement network.

[0176] In one possible implementation, the speech recognition network is selected based on its structural type, thereby obtaining speech recognition networks with different structural types.

[0177] Specifically, existing speech recognition networks can be categorized into several types based on their structural types. Then, a typical speech recognition network is selected from each of these categories, resulting in multiple speech recognition networks with different structural types. In other words, among the multiple speech recognition networks connected by the speech enhancement network, different speech recognition networks correspond to different types of network structures.

[0178] For example, based on the structural type of speech recognition networks, speech recognition networks can be divided into the following five types: Listen Attend Spell (LAS) networks, Connectionist Temporal Classification (CTC) networks, Attention-based Encoder-Decoder (AED) networks, Recurrent Neural Network Transducer (RNN-T) networks, and Transformer Transducer networks.

[0179] The LAS network typically consists of two components: a listener and a speller. The listener is a pyramid recursive network encoder that receives a filter bank spectrum as input; the speller is a periodic network decoder based on an attention mechanism that emits characters as output.

[0180] The CTC network is an improved version of the RNN, capable of being trained directly using sequences. Specifically, the CTC network introduces a new loss function that allows the RNN to be trained directly using unchopped sequences.

[0181] The AED network is a network structure based on an Encoder and a Decoder. In an AED network, the Encoder extracts information from the input sequence (speech data), while the Decoder is an autoregressive model on the target sequence (text) (i.e., predicting the next text unit based on the previous text unit). Simultaneously, during autoregressive computation, an attention mechanism is used to obtain the output encoded information of the Encoder, thus utilizing information from the input sequence.

[0182] RNN-T networks are improvements on the CTC network. Essentially, they combine the hidden layer outputs of the Encoder part of the CTC network with another output, where the combined output is the result of LSTM computation on one of the final network's outputs. In essence, RNN-T networks add relationships between sequences to the CTC network.

[0183] The Transformer Transducer network is actually an improvement on the RNN-T network, mainly by replacing the Encoder in the RNN-T network from LSTM with a Transformer.

[0184] During the training process of the speech enhancement network, a speech recognition network can be selected from each of the five types mentioned above to participate in the training. Specifically, the multiple speech recognition networks connected to the speech enhancement network are LAS, CTC, AED, RNN-T, and Transformer Transducer networks. In this way, since the speech enhancement network uses various types of speech recognition networks for training, during the inference phase, when paired with the target speech recognition network to perform speech recognition, even if the target speech recognition network is a black-box system, the speech enhancement network can effectively improve the speech recognition performance of the target speech recognition network.

[0185] In another possible implementation, a speech recognition network is selected based on the structural similarity between the speech recognition networks, thereby selecting a speech recognition network with a large structural difference.

[0186] Specifically, before training the speech enhancement network, a large number of speech recognition networks with different structures can be obtained as candidate networks. Then, for each candidate speech recognition network, the structural similarity between any two speech recognition networks is determined, thus obtaining the structural similarity between any two speech recognition networks. In this way, based on the structural similarity between speech recognition networks, multiple speech recognition networks with relatively low structural similarity can be selected as networks to cooperate with the training of the speech enhancement network.

[0187] For example, among the multiple speech recognition networks connected to the speech enhancement network, the structural similarity between different speech recognition networks is less than a first threshold. That is, for any two speech recognition networks connected to the speech enhancement network, the structural similarity between them is less than the first threshold. The first threshold can be determined based on the method for determining the structural similarity of the speech recognition networks, or based on the needs of the actual application (e.g., a limited number of selected speech recognition networks). This embodiment does not limit the specific value of the first threshold.

[0188] In one possible example, this embodiment could involve loading the candidate speech recognition networks into a unified deep learning framework, then outputting the structure of each speech recognition network as text (e.g., printing the structure of each speech recognition network to a log document), resulting in structural text for each speech recognition network (i.e., text describing the structure of the speech recognition network). Then, based on a text comparison algorithm, the differences between different structural texts are compared to obtain a similarity score between each pair of structural texts. This similarity score can then be used as the structural similarity between speech recognition networks.

[0189] For example, please refer to Tables 2 and 3 below, which both show the similarity scores between structured texts corresponding to multiple different speech recognition networks.

[0190] Table 2

[0191] Speech Recognition Network 1 4.64 3.13 3.12 3.12 Speech Recognition Network 2 4.46 3.67 3.67 Speech Recognition Network 3 4.50 4.50 Speech Recognition Network 4 4.50

[0192] In Table 2, this embodiment selected four different speech recognition networks and determined the structural similarity between each pair of speech recognition networks by comparing the similarity scores between the structured texts corresponding to these four speech recognition networks. Table 2 shows the similarity scores between speech recognition networks; a higher similarity score indicates a higher structural similarity between the two speech recognition networks, while a lower similarity score indicates a lower structural similarity between the two speech recognition networks.

[0193] Table 3

[0194]

[0195] In Table 3, this embodiment is based on the four speech recognition networks selected in Table 2. After training these four speech recognition networks with the same batch of training data, the similarity scores between the structured texts corresponding to these four trained speech recognition networks are compared.

[0196] Table 2 shows that similar speech recognition networks generally have high similarity scores, around 4.5; while the differences between different speech recognition networks are significant, generally below 4.0. Table 3 shows that after training, under the combined effect of parameters and structure in the speech recognition network, the differences in similarity scores between different speech recognition networks persist, and the similarity scores of the same speech recognition network are also preserved to some extent, proving the effectiveness of determining the structural similarity of speech recognition networks based on structured text.

[0197] After obtaining the trained speech enhancement network based on the model training method provided in this embodiment, the trained speech enhancement network can be used in conjunction with the target speech recognition network to complete the speech recognition task in a real-world application scenario. The target speech recognition network refers to the network used to perform the speech recognition task in a real-world application scenario. This target speech recognition network can be, for example, a black-box system or a white-box system; this embodiment does not limit this.

[0198] For example, please refer to Figure 6 , Figure 6 This is a flowchart illustrating a speech processing method provided in an embodiment of this application. Figure 6 As shown, the speech processing method includes the following steps 601-603.

[0199] Step 601: Obtain a speech recognition request. The speech recognition request includes a first speech and is used to request the recognition of the first speech.

[0200] For example, in scenarios such as caption generation, meeting minutes, voice translation, and human-computer dialogue, the electronic device responsible for voice recognition processing (such as a server, personal computer, smartphone, or tablet) can receive a voice recognition request sent by the microphone, which includes the first voice collected by the microphone. The electronic device responsible for voice recognition processing and the microphone can be integrated into the same device (e.g., integrated into the same smartphone) or they can be separate devices (e.g., the electronic device is a server, and the microphone is a voice recorder).

[0201] Step 602: Process the first speech through the trained speech enhancement network to obtain the enhanced first speech. The trained speech enhancement network is obtained by training the speech enhancement network. During the training process, the speech enhancement network is connected to multiple speech recognition networks in parallel. The inputs of the multiple speech recognition networks are the outputs of the speech enhancement network. The outputs of the multiple speech recognition networks are used to construct the loss function for training the speech enhancement network.

[0202] Specifically, the trained speech enhancement network used in step 602 is trained based on the model training method described in the above embodiments. Therefore, the training process of the trained speech enhancement network can be referred to the description in the above embodiments, and will not be repeated here.

[0203] Step 603: Perform speech recognition on the enhanced first speech to obtain the recognized text corresponding to the first speech.

[0204] The electronic device can employ a speech recognition network to perform speech recognition on the enhanced first speech, obtaining the recognized text. The speech recognition network is a neural network used to perform speech recognition; specific details can be found in related technologies and will not be elaborated upon here. Furthermore, the speech recognition network can be a black-box system, meaning its parameters are unavailable, and it does not participate in the training of the subsequently trained speech enhancement network. In this case, since the trained speech enhancement network is pre-trained based on multiple parallel speech recognition networks, it possesses high generalization and versatility. Therefore, even if the speech recognition network (which is a black-box system) cannot be jointly trained with the speech enhancement network, good speech recognition results can still be obtained.

[0205] Furthermore, the recognized text can be used by electronic devices to perform subsequent tasks, such as generating captions, generating meeting minutes, generating speech translation results, or outputting human-computer dialogue results.

[0206] In one possible implementation, among the multiple speech recognition networks used to train the speech enhancement network, different speech recognition networks correspond to different types of network structures.

[0207] In another possible implementation, the structural similarity between different speech recognition networks used to train the speech enhancement network is less than a first threshold.

[0208] In another possible implementation, the trained speech enhancement network is trained based on the total loss function value, which is obtained by weighted summation of multiple loss function values ​​corresponding to multiple speech recognition networks. These multiple loss function values ​​are obtained based on the outputs of multiple speech recognition networks.

[0209] In another possible implementation, the weight value of each speech recognition network in the multiple speech recognition networks is positively correlated with the loss function value of each speech recognition network.

[0210] In another possible implementation, the weight value corresponding to each speech recognition network is also related to the weight value of each speech recognition network in the historical iteration training process.

[0211] In another possible implementation, among multiple speech recognition networks, the weight value corresponding to the speech recognition network whose loss function value is greater than the second threshold is 0.

[0212] In another possible implementation, the second threshold is N times the target loss function value, where the target loss function value is the smallest loss function value among multiple loss function values ​​corresponding to multiple speech recognition networks, and N is greater than 1.

[0213] In another possible implementation, the parameters of multiple speech recognition networks remain unchanged during the training of the speech enhancement network.

[0214] In another possible implementation, multiple speech recognition networks are pre-trained networks.

[0215] For example, please refer to Figure 7 , Figure 7 This is a schematic diagram illustrating the application of a trained speech enhancement network provided in this application. For example... Figure 7 As shown, the trained speech enhancement network is applied to the speech recognition system to cooperate with the target speech recognition network to complete the speech recognition task. The speech recognition system consists of the following four parts.

[0216] The first part of the structure is the input device, which can include various types of audio pickup devices, such as headphone microphones and computer microphones. The input device is primarily used to acquire the speech signal to be recognized, and the speech signal is not limited to mono or multi-channel. In addition, the input device can also be a speech signal receiving device, used to receive speech signals transmitted over a network, such as speech signals from a conference system or other sources. After acquiring the speech signal, the input device converts it into a format that the computer can store and temporarily stores it in the computer system, for example, converting the speech signal into storage formats such as WAV and MP3 for storage.

[0217] The second part of the structure is the speech enhancement network trained using the model training method provided in this embodiment. In this embodiment, the speech enhancement network takes the speech waveform as input, transforms the speech waveform from the time domain to the frequency domain through Fourier transform, obtains the enhanced speech signal after nonlinear calculation mapping by the speech enhancement network, and then transforms the enhanced speech signal back to the time domain waveform through inverse Fourier transform.

[0218] The third part of the structure is a speech recognition network, which can be a black-box system or a white-box system. The input of the speech recognition network is the speech waveform output by the speech enhancement network, and the output is the text predicted by the speech recognition network.

[0219] The fourth part is the output device, which is used to output or transmit the recognized text obtained in the previous step over a network. For example, the output device is a local terminal device (such as a personal computer, vehicle terminal, smartphone, etc.), which can output the recognized text as real-time subtitles for local software, or encode the recognized text for instant messaging over a network, etc. This embodiment does not limit the use of the recognized text.

[0220] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a speech enhancement network provided in this application. Specifically, an example of the structure of a speech enhancement network trained using the model training method provided in the above embodiments is shown below. Figure 8 As shown. Figure 8 As shown, the speech enhancement network includes the following structural modules connected in sequence: molecular strip module, complex convolution module, complex time-frequency domain recurrent neural network, complex convolution module, depth filter, and offspring merging module.

[0221] exist Figure 8 In the speech enhancement network shown, the input to the molecular banding module is the spectrum of the speech signal. The molecular banding module multiplies the input spectrum by independent learnable parameters for each frequency band, performing molecular banding operations. Parallel processing improves the model's running speed. The speech enhancement network concludes with a sub-generation merging module, corresponding to the molecular banding module.

[0222] The molecular band module is followed by a complex convolution module. The complex convolution module uses a two-layer encoding structure (real part convolution layer and imaginary part convolution layer). The real part convolution layer processes the real part of the input complex number, and the imaginary part convolution layer processes the imaginary part of the input complex number, thus simulating complex multiplication.

[0223] The complex convolutional module is followed by a complex time-frequency domain recurrent neural network (RTN). The structure of the RNN follows the same design principle as the encoding layer in the complex convolutional module; that is, the RNN uses two LSTM layers to simulate complex multiplication. The RNN first performs complex multiplication on the frequency axis (F-LSTM), using a bidirectional LSTM; then it performs complex multiplication on the time axis (T-LSTM), using a unidirectional LSTM to ensure the network model is causal.

[0224] The complex time-frequency domain recurrent neural network is followed by a complex convolution module, and the two complex convolution modules before and after the complex time-frequency domain recurrent neural network are connected through a convolution channel.

[0225] The complex convolutional module following the complex time-frequency domain recurrent neural network is also connected to a depth filter. (See also...) Figure 9 , Figure 9 A schematic diagram illustrating the operation comparison of a depth filter provided in this application. For example... Figure 9 As shown in (a), typically, after estimating the mask, a speech enhancement network multiplies the mask by the original input spectrum to obtain the output spectrum. In this case, there is a one-to-one correspondence between the input and output spectra; for example... Figure 9 As shown in (b), the deep filter has a many-to-one correspondence, and the output spectrum point is inferred from the input spectrum point and its neighboring points. By introducing contextual information, it is beneficial to improve the output results of the speech enhancement network.

[0226] Based on the model training method and the application of the trained speech enhancement network proposed in the above embodiments, this embodiment conducts comparative experiments and effect verification. Specifically, this embodiment selects three speech recognition networks as demonstrations, and after freezing the parameters of these three speech recognition networks, simulates a black-box system to jointly train with the speech enhancement network (N-to-1 training). The black-box system used for testing does not participate in the training. Simultaneously, as a comparison, another set of experiments is designed, using a pre-parameterized and adjustable speech recognition network (i.e., a white-box system) to jointly train with the speech enhancement network. The training and test data used in each set of experiments are consistent, and the experimental results are shown in Table 4 below.

[0227] Table 4

[0228]

[0229]

[0230] Generally, the evaluation metric for speech recognition systems is the Word Error Rate (WER), with a lower WER indicating better recognition results. As shown in the table above, based on rows 1 and 4, both white-box and black-box speech recognition networks exhibit similar performance on the target test set, meaning that without any external speech enhancement system, both have an error rate approaching 30%, significantly impacting speech recognition results. When combined with one-to-one joint training, the speech enhanced by the speech enhancement network, when then passed through the white-box speech recognition network, achieves an absolute improvement of 4.55% and a relative improvement of 16%. This demonstrates that joint training effectively improves the accuracy of speech recognition networks when noise interferes with their performance. However, when the speech enhancement network is coupled with the black-box speech recognition network, a side effect occurs: the speech recognition accuracy decreases, highlighting the limitations of traditional methods. After applying the method provided in this embodiment to train a speech enhancement network by combining multiple speech recognition networks, the trained speech enhancement network can effectively couple with the speech recognition network as a black box system, achieving a WER of 24.13%, which is close to the effect of 1-to-1 joint training in a white box system.

[0231] The above describes the model training method provided in this application. Next, we will introduce the device used to perform the above model training method and speech processing method.

[0232] Please refer to Figure 10A , Figure 10A This is a schematic diagram of the structure of a model training device provided in this application. Figure 10A As shown, the model training apparatus provided in this application includes: an acquisition module 1001 for acquiring training samples, the training samples including speech samples and corresponding real text; a processing module 1002 for inputting the speech samples into a speech enhancement network to obtain enhanced speech samples; the processing module 1002 is further used to input the enhanced speech samples into multiple different speech recognition networks to obtain the speech recognition results output by each speech recognition network; and a training module 1003 for training the speech enhancement network based on the total loss function value to obtain a speech enhancement network with updated parameters; wherein, the total loss function value is obtained based on the loss function value corresponding to each of the multiple speech recognition networks, and the loss function value corresponding to each speech recognition network is obtained based on the speech recognition results output by each speech recognition network and the real text.

[0233] In one possible implementation, the parameters of multiple speech recognition networks remain unchanged during the training of the speech enhancement network.

[0234] In one possible implementation, multiple speech recognition networks are pre-trained networks.

[0235] In one possible implementation, multiple speech recognition networks exist, with different networks corresponding to different types of network structures.

[0236] In one possible implementation, the structural similarity between different speech recognition networks is less than a first threshold.

[0237] In one possible implementation, the total loss function value is obtained by performing a weighted summation of the loss function values ​​corresponding to each speech recognition network.

[0238] In one possible implementation, the weight value of each speech recognition network is positively correlated with the loss function value of each speech recognition network.

[0239] In one possible implementation, the weight value corresponding to each speech recognition network is also related to the weight value of each speech recognition network during the historical iteration training process.

[0240] In one possible implementation, among multiple speech recognition networks, the weight value corresponding to the speech recognition network whose loss function value is greater than a second threshold is 0.

[0241] In one possible implementation, the second threshold is N times the target loss function value, where the target loss function value is the smallest loss function value among multiple loss function values ​​corresponding to multiple speech recognition networks, and N is greater than 1.

[0242] because Figure 10A The model training device shown corresponds to Figure 4 The method shown, therefore Figure 10A The specific implementation of the model training device shown and its technical effects can be found in the relevant descriptions in the foregoing embodiments, and will not be repeated here.

[0243] Please refer to Figure 10B , Figure 10B This is a schematic diagram of the structure of a speech processing device provided in this application. Figure 10BAs shown, the speech processing apparatus provided in this application includes: an acquisition module 1004 for acquiring a speech recognition request, the speech recognition request including a first speech; a processing module 1005 for processing the first speech through a trained speech enhancement network to obtain enhanced speech, wherein the trained speech enhancement network is obtained by training the speech enhancement network, and the speech enhancement network is connected to multiple parallel speech recognition networks during the training process, the inputs of the multiple speech recognition networks are the outputs of the speech enhancement network, and the outputs of the multiple speech recognition networks are used to construct the loss function for training the speech enhancement network; the processing module 1005 is also used to perform speech recognition on the enhanced speech to obtain the recognized text corresponding to the first speech.

[0244] In one possible implementation, among the multiple speech recognition networks used to train the speech enhancement network, different speech recognition networks correspond to different types of network structures.

[0245] In another possible implementation, the structural similarity between different speech recognition networks used to train the speech enhancement network is less than a first threshold.

[0246] In another possible implementation, the trained speech enhancement network is trained based on the total loss function value, which is obtained by weighted summation of multiple loss function values ​​corresponding to multiple speech recognition networks. These multiple loss function values ​​are obtained based on the outputs of multiple speech recognition networks.

[0247] In another possible implementation, the weight value of each speech recognition network in the multiple speech recognition networks is positively correlated with the loss function value of each speech recognition network.

[0248] In another possible implementation, the weight value corresponding to each speech recognition network is also related to the weight value of each speech recognition network in the historical iteration training process.

[0249] In another possible implementation, among multiple speech recognition networks, the weight value corresponding to the speech recognition network whose loss function value is greater than the second threshold is 0.

[0250] In another possible implementation, the second threshold is N times the target loss function value, where the target loss function value is the smallest loss function value among multiple loss function values ​​corresponding to multiple speech recognition networks, and N is greater than 1.

[0251] In another possible implementation, the parameters of multiple speech recognition networks remain unchanged during the training of the speech enhancement network.

[0252] In another possible implementation, multiple speech recognition networks are pre-trained networks.

[0253] Figure 11 This is a schematic diagram of an electronic device 1100 provided in this application. The electronic device 1100 may be, for example, used for independently performing... Figure 4 The model training method shown.

[0254] In some possible implementations, Figure 4 When the model training method shown is executed jointly by multiple electronic devices in a cluster system, each electronic device in the cluster system can, as... Figure 11 The electronic device shown is 1100.

[0255] like Figure 11 As shown, the electronic device 1100 includes a processor 1101, a memory 1102, and a communication interface 1103. The processor 1101, memory 1102, and communication interface 1103 communicate via a bus 1104, or via wireless transmission or other means. The memory 1102 stores instructions, and the processor 1101 executes the instructions stored in the memory 1102. Further, the electronic device 1100 may also include a memory unit 1105, which is connected to the processor 1101, the storage medium 1102, and the communication interface 1103 via the bus 1104. The memory 1102 stores program code, and the processor 1101 can call and execute the program code stored in the memory 1102. Figure 4 The model training method shown.

[0256] It should be understood that in this application, the processor 1101 can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete device assemblies, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0257] The memory 1102 may include read-only memory and random access memory, and provides instructions and data to the processor 1101. The memory 1102 may also include non-volatile random access memory. For example, the memory 1102 may also store device type information.

[0258] The memory 1102 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0259] The communication interface 1103 is used to communicate with other devices connected to the electronic device 1100. The bus 1104 may include a data bus, as well as a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus 1104 in the figure.

[0260] It should be understood that the electronic device 1100 according to this application may correspond to the model training device in this application, and the above and other operations and / or functions implemented by the electronic device 1100 are respectively for the purpose of implementing Figure 4 The corresponding processes of each method in the code will not be elaborated here for the sake of brevity.

[0261] Optionally, the electronic device provided in this application can be implemented using a chip. The chip includes a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the model hyperparameter selection method described in the above embodiments, or to cause the chip within the training device to execute the model hyperparameter selection method described in the above embodiments. Optionally, the storage unit is a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0262] For details, please refer to Figure 12 , Figure 12 This is a schematic diagram of a chip provided in this application. The chip can be represented as a neural network processor (NPU) 1200. The NPU 1200 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core of the NPU is the arithmetic circuit 1203, which is controlled by a controller 1204 to retrieve matrix data from the memory and perform multiplication operations.

[0263] In some implementations, the arithmetic circuit 1203 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1203 is a two-dimensional pulsating array. The arithmetic circuit 1203 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1203 is a general-purpose matrix processor.

[0264] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1202 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1201 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1208.

[0265] Unified memory 1206 is used to store input and output data. Weight data is directly transferred to weight memory 1202 via Direct Memory Access Controller (DMAC) 1205. Input data is also transferred to unified memory 1206 via DMAC.

[0266] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1209.

[0267] The Bus Interface Unit (BIU) 1210 is used by the instruction fetch memory 1209 to fetch instructions from external memory, and also by the memory access controller 1205 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0268] The DMAC is mainly used to move input data from external memory DDR to unified memory 1206, or to weight data to weight memory 1202, or to input data to input memory 1201.

[0269] The vector computation unit 1207 includes multiple arithmetic processing units that, when needed, further process the output of the computation circuit 1203, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization (BN), pixel-level summation, and upsampling of feature planes.

[0270] In some implementations, the vector computation unit 1207 can store the processed output vector in the unified memory 1206. For example, the vector computation unit 1207 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1203, such as linear interpolation of feature planes extracted by a convolutional layer, or, for example, a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 1207 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 1203, for example, for use in subsequent layers of the neural network.

[0271] The instruction fetch buffer 1209 connected to the controller 1204 is used to store the instructions used by the controller 1204;

[0272] Unified memory 1206, input memory 1201, weight memory 1202, and instruction fetch memory 1209 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0273] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0274] Please refer to Figure 13 , Figure 13 This is a schematic diagram of a computer-readable storage medium provided in this application. This application also provides a computer-readable storage medium in some embodiments, wherein the above-described... Figure 4 The disclosed method can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0275] Figure 13 A conceptual partial view of an example computer-readable storage medium arranged according to at least some of the embodiments shown herein is illustrated schematically. The example computer-readable storage medium includes a computer program for executing computer processes on a computing device.

[0276] In one embodiment, the computer-readable storage medium 1300 is provided using a signal bearer medium 1301. The signal bearer medium 1301 may include one or more program instructions 1302, which, when executed by one or more processors, can provide the above-mentioned... Figure 4 The described function or part of the function. In addition... Figure 13 The program instruction 1302 in the document also describes example instructions.

[0277] In some examples, signal carrying medium 1301 may include computer-readable medium 1303, such as, but not limited to, hard disk drive, compact disc (CD), digital video disc (DVD), digital magnetic tape, memory, ROM or RAM, etc.

[0278] In some embodiments, the signal-bearing medium 1301 may include a computer-recordable medium 1304, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, etc. In some embodiments, the signal-bearing medium 1301 may include a communication medium 1305, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, etc.). Therefore, for example, the signal-bearing medium 1301 may be transmitted by a wireless communication medium 1305 (e.g., a wireless communication medium conforming to the IEEE 802.13 standard or other transmission protocols).

[0279] One or more program instructions 1302 may be, for example, computer-executable instructions or logical implementation instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to one or more program instructions 1302 conveyed to the computing device via a computer-readable medium 1303, a computer-recordable medium 1304, and / or a communication medium 1305.

[0280] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0281] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0282] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0283] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A speech processing method, characterized in that, include: A speech recognition request is obtained, the speech recognition request including a first speech, the speech recognition request being used to request the recognition of the first speech; The first speech is processed by a trained speech enhancement network to obtain enhanced first speech. The trained speech enhancement network is obtained by training a speech enhancement network, and the speech enhancement network is connected to multiple parallel speech recognition networks during the training process. The inputs of the multiple speech recognition networks are the outputs of the speech enhancement network, and the outputs of the multiple speech recognition networks are used to construct the loss function for training the speech enhancement network. Among the multiple speech recognition networks, different speech recognition networks correspond to different types of network structures, and the structural similarity between different speech recognition networks is less than a first threshold. Speech recognition is performed on the enhanced first speech to obtain the recognized text corresponding to the first speech.

2. The method according to claim 1, characterized in that, The trained speech enhancement network is obtained by training the speech enhancement network based on the total loss function value, which is obtained by weighted summation of multiple loss function values ​​corresponding to the multiple speech recognition networks, and the multiple loss function values ​​are obtained based on the output of the multiple speech recognition networks.

3. The method according to claim 2, characterized in that, The weight value of each speech recognition network in the plurality of speech recognition networks is positively correlated with the loss function value of each speech recognition network.

4. The method according to claim 3, characterized in that, The weight value corresponding to each speech recognition network is also related to the weight value of each speech recognition network in the historical iterative training process.

5. The method according to any one of claims 2-4, characterized in that, Among the multiple speech recognition networks, the weight value corresponding to the speech recognition network whose loss function value is greater than the second threshold is 0.

6. The method according to claim 5, characterized in that, The second threshold is N times the target loss function value, where the target loss function value is the smallest loss function value among the multiple loss function values ​​corresponding to the multiple speech recognition networks, and N is greater than 1.

7. The method according to any one of claims 1-4, characterized in that, During the training of the speech enhancement network, the parameters of the plurality of speech recognition networks remain unchanged.

8. The method according to any one of claims 1-4, characterized in that, All of the aforementioned speech recognition networks are pre-trained networks.

9. A voice processing device, characterized in that, include: The acquisition module is used to acquire a speech recognition request, wherein the speech recognition request includes a first speech. The processing module is used to process the first speech through a trained speech enhancement network to obtain enhanced speech. The trained speech enhancement network is obtained by training a speech enhancement network, and the speech enhancement network is connected to multiple speech recognition networks in parallel during the training process. The inputs of the multiple speech recognition networks are the outputs of the speech enhancement network, and the outputs of the multiple speech recognition networks are used to construct the loss function for training the speech enhancement network. Among the multiple speech recognition networks, different speech recognition networks correspond to different types of network structures, and the structural similarity between different speech recognition networks is less than a first threshold. The processing module is further configured to perform speech recognition on the enhanced speech to obtain the recognized text corresponding to the first speech.

10. An electronic device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the electronic device performs the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, It includes computer-readable instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 8.

Images