Speech processing method and related device

Abstract

A speech processing method is applied to speech recognition. According to the method, in the process of executing voice processing, a trained voice enhancement network is adopted to execute enhancement processing on voice, and then voice recognition is executed on the enhanced voice. Moreover, the trained speech enhancement network is obtained based on the training of a plurality of parallel speech recognition networks, and the outputs of the plurality of speech recognition networks are used for jointly forming a loss function for training the speech enhancement network, so that the generalization and universality of the trained speech enhancement network can be effectively improved. Thus, voice processing is executed based on the trained voice enhancement network, the voice recognition effect can be effectively improved, and the trained voice enhancement network can be popularized and applied to a voice recognition network which is not subjected to joint training, so that the requirements in various scenes are met.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a speech processing method and related devices. Background Art

[0002] Automatic Speech Recognition (ASR) technology is a technology that automatically converts a speaker's speech into text information. Figure 1 As shown, usually, a sound pickup device first receives a voice signal from a speaker, and then an automatic speech recognition system deployed outside the sound pickup device recognizes the voice signal, and finally outputs a recognized text.

[0003] As automatic speech recognition technology matures, various products that use voice as an interactive portal have flooded the market and appeared in daily life. However, natural voice interaction in real scenarios will be interfered by many factors, such as environmental noise, room reverberation, etc. The presence of these factors will reduce the system's recognition rate of the target voice, making the user experience worse. How to improve the accuracy of speech recognition systems in real scenarios is a long-standing problem facing the industry.

[0004] In order to improve the accuracy of speech recognition, a speech enhancement network is usually added before the speech recognition network, and the speech enhancement network first performs noise reduction processing on the speech data before inputting it into the speech recognition network for speech recognition. However, the speech enhancement network added in the related art is usually used in conjunction with a specific speech recognition network, resulting in poor applicability of the speech enhancement network. Summary of the invention

[0005] The present application provides a model training and speech processing method, which can effectively improve the generalization and versatility of the trained speech enhancement network, so that the speech enhancement network can be extended and applied to speech recognition networks that have not been jointly trained.

[0006] In a first aspect, a model training method is provided for training a speech enhancement network. The method comprises: first, obtaining a training sample, the training sample comprising a speech sample and a real text corresponding to the speech sample. The speech sample refers to a piece of speech data with noise, such as the speech of a user captured by a sound pickup device in a noisy scene. In addition, the real text refers to the text content actually corresponding to the speech content in the speech sample.

[0007] Then, the speech sample is input into the speech enhancement network, and the speech enhancement network performs speech enhancement processing on the speech sample to obtain an enhanced speech sample. The speech enhancement network is a neural network, such as a convolutional neural network, a recurrent neural network, or a long short-term memory artificial neural network (Long Short-Term Memory, LSTM) or other neural networks.

[0008] Secondly, the enhanced speech samples are respectively input into different multiple speech recognition networks to obtain the speech recognition results output by each speech recognition network. The speech recognition results output by each speech recognition network are recognition texts for the enhanced speech samples. In addition, the multiple speech recognition networks have different network structures.

[0009] Finally, the speech enhancement network is trained based on the total loss function value to obtain a speech enhancement network with updated parameters. The total loss function value is obtained based on the loss function value corresponding to each speech recognition network in the multiple speech recognition networks, and the loss function value corresponding to each speech recognition network is obtained based on the speech recognition results output by each speech recognition network and the real text.

[0010] That is to say, after the enhanced speech sample is processed by each speech recognition network, the speech recognition result output by each speech recognition network can be obtained. In this way, based on the real text corresponding to the speech sample and the speech recognition result output by each speech recognition network, the loss function corresponding to each speech recognition network can be constructed, and the loss function value of each speech recognition network for the speech sample can be calculated. Finally, by combining multiple loss function values ​​corresponding to multiple speech recognition networks, the total loss function value for training the speech enhancement network can be obtained, thereby updating the parameters of the speech enhancement network based on the total loss function value.

[0011] In this solution, during the training process of the speech enhancement network, multiple parallel speech recognition networks are connected after the speech enhancement network, and multiple different speech recognition networks respectively perform recognition processing on the speech data output by the speech enhancement network, and then construct the loss function corresponding to each speech recognition network based on the speech recognition result output by each speech recognition network. Finally, by training the speech enhancement network with the loss functions of multiple speech recognition networks, the generalization and versatility of the speech enhancement network can be effectively improved, so that the speech enhancement network can be extended and applied to speech recognition networks that have not been jointly trained.

[0012] In one possible implementation, during the training of the speech enhancement network, the parameters of the multiple speech recognition networks do not change. That is, the multiple speech recognition networks are not trained together with the speech enhancement network. In this way, by freezing the parameters of the multiple speech recognition networks, it is possible to avoid the parameter changes of the speech recognition network from affecting the parameters of the speech enhancement network. This can well simulate the application scenario where the speech recognition network is a black box system (i.e., the parameters of the speech recognition network are unknown and the parameters cannot be adjusted), improve the generalization of the speech enhancement network, and enable the speech enhancement network finally trained to be applied to other speech recognition networks that have not undergone joint training.

[0013] In another possible implementation, the multiple speech recognition networks are all pre-trained networks. That is, the multiple speech recognition networks are all pre-trained. For example, the multiple speech recognition networks may be pre-trained based on training samples, and the multiple speech recognition networks have good speech recognition effects after training.

[0014] In this solution, by using a pre-trained speech recognition network to combine with the speech enhancement network, and only training the speech enhancement network during the training process, the speech recognition network can output a speech recognition result with higher accuracy during the training phase, and avoid the speech recognition result with lower accuracy affecting the training effect of the speech enhancement network. In addition, after the speech enhancement network is trained, it is usually used in conjunction with the trained speech recognition network. Therefore, in the training process, the speech enhancement network is trained in conjunction with the pre-trained speech recognition network, which can well simulate the subsequent application scenarios of the speech enhancement network and improve the speech enhancement effect of the speech enhancement network in actual application.

[0015] In another possible implementation, in multiple speech recognition networks, different speech recognition networks correspond to different types of network structures. For example, the types corresponding to different speech recognition networks in multiple speech recognition networks are: Listen Attend Spell (LAS) network, Connectionist Temporal Classification (CTC) network, Attention-based Encoder-Decoder (AED) network, Recurrent Neural Network Transducer (RNN-T) network, Transformer Transducer network.

[0016] In this scheme, by setting the multiple speech recognition networks participating in the speech enhancement network training to correspond to different types of network structures, the structural differences between the multiple speech recognition networks can be made as large as possible, thereby improving the versatility of the trained speech enhancement network and avoiding the speech enhancement network being trained to tend to only improve the speech recognition effect of a certain part of the speech recognition network with a similar structure.

[0017] In another possible implementation, among the multiple speech recognition networks, the structural similarity between different speech recognition networks is less than the first threshold. That is, for the multiple speech recognition networks connected to the speech enhancement network, the structural similarity between any two speech recognition networks is less than the first threshold.

[0018] In this scheme, by comparing the structural similarities between speech recognition networks and selecting multiple speech recognition networks with lower structural similarities to assist in training the speech enhancement network, the structural differences between the multiple speech recognition networks can be made as large as possible, thereby improving the versatility of the trained speech enhancement network and avoiding the speech enhancement network being trained to tend to only improve the speech recognition effect of a certain part of the speech recognition network with similar structure.

[0019] In another possible implementation, the total loss function value is obtained by performing a weighted summation of the loss function values ​​corresponding to each speech recognition network. That is, a weight can be assigned to each speech recognition network, and then the weight corresponding to each speech recognition network and the loss function value are multiplied to obtain the weighted loss function value corresponding to each speech recognition network; finally, the weighted loss function values ​​corresponding to each speech recognition network are added together to obtain the total loss function value.

[0020] In this scheme, the total loss function value is calculated by weighted summation, which can well integrate the output results of multiple speech recognition networks, facilitate the auxiliary training of speech enhancement networks based on multiple speech recognition networks, and improve the training effect of speech enhancement networks.

[0021] In another possible implementation, the weight value corresponding to each speech recognition network is positively correlated with the loss function value corresponding to each speech recognition network. That is, the larger the loss function value corresponding to the speech recognition network, the larger the weight value corresponding to the speech recognition network; the smaller the loss function value corresponding to the speech recognition network, the smaller the weight value corresponding to the speech recognition network.

[0022] In this scheme, by setting the weight value corresponding to the speech recognition network to have a positive correlation with the loss function value, a larger weight can be given to the speech recognition network with a large loss function value, so that the speech enhancement network is more inclined to improve the speech recognition network with relatively poor recognition effect during the training process, thereby improving the adaptability of the speech enhancement network to various speech recognition networks.

[0023] In another possible implementation, the weight value corresponding to each speech recognition network is also related to the weight value of each speech recognition network in the historical iterative training process. That is to say, when determining the weight value corresponding to the speech recognition network, it is necessary to determine it based on the weight value of the speech recognition network in the historical iterative training process, that is, the weight value corresponding to the speech recognition network is changed based on the weight value in the historical iterative training process, and will not differ too much from the weight value in the historical iterative training process.

[0024] In this scheme, in the process of determining the weight value of the speech recognition network, by introducing the weight value of the speech recognition network in the historical iterative training process, it can be ensured that the weight value of the speech recognition network changes relatively smoothly during the training process, thereby avoiding excessive changes in the total loss function value and affecting the training effect of the speech enhancement network.

[0025] In another possible implementation, among the multiple speech recognition networks, the weight value corresponding to the speech recognition network whose loss function value is greater than the second threshold is 0.

[0026] In this scheme, a threshold is set for the loss function values ​​of multiple speech recognition networks. When the loss function value of the speech recognition network exceeds the threshold, the loss function value is discarded, that is, the total loss function value is no longer determined based on the loss function value, thereby avoiding bad learning samples from having a negative impact on the training of the speech enhancement network.

[0027] In another possible implementation, the second threshold is N times the target loss function value, the target loss function value is the smallest loss function value among multiple loss function values ​​corresponding to multiple speech recognition networks, and N is greater than 1.

[0028] In the second aspect, a speech processing method is provided, which is applied to recognize speech. In the speech processing method, the electronic device first obtains a speech recognition request, which includes a first speech, and the speech recognition request is used to request recognition of the first speech. For example, in scenarios such as subtitle generation, meeting minutes, speech translation, and human-computer dialogue, the electronic device responsible for speech recognition processing (such as a server or a smart phone, etc.) can obtain a speech recognition request sent by a sound pickup device, and the speech recognition request includes the first speech collected by the sound pickup device. Among them, the electronic device responsible for speech recognition processing and the sound pickup device can be integrated on the same device (such as integrated on the same smart phone), or they can be separate different devices (such as the electronic device is a server and the sound test device is a voice recorder).

[0029] Then, the electronic device processes the first speech through the trained speech enhancement network to obtain an enhanced first speech. The trained speech enhancement network is obtained after training the speech enhancement network, and the speech enhancement network is connected to multiple parallel speech recognition networks during the training process, the inputs of the multiple speech recognition networks are all outputs of the speech enhancement network, and the outputs of the multiple speech recognition networks are used to construct a loss function for training the speech enhancement network.

[0030] Finally, the electronic device performs speech recognition on the enhanced first voice (for example, performs speech recognition on the enhanced first voice through a speech recognition network) to obtain a recognition text corresponding to the first voice, that is, the recognition text is the text obtained after the electronic device performs speech recognition on the first voice. The recognition text can be used to perform subsequent tasks, such as generating subtitles, generating meeting minutes, or generating speech translation results.

[0031] In this solution, during the speech processing, the trained speech enhancement network is used to first perform enhancement processing on the speech, and then speech recognition is performed on the enhanced speech. In addition, the trained speech enhancement network used is obtained based on the training of multiple parallel speech recognition networks, and the outputs of the multiple speech recognition networks are used to jointly constitute the loss function for training the speech enhancement network, which can effectively improve the generalization and versatility of the trained speech enhancement network. In this way, performing speech processing based on the trained speech enhancement network can effectively improve the effect of speech recognition, and the trained speech enhancement network can be extended to speech recognition networks that have not been jointly trained, thereby meeting the needs of various scenarios.

[0032] In a possible implementation, among the multiple speech recognition networks used to train the speech enhancement network, different speech recognition networks correspond to different types of network structures.

[0033] In another possible implementation, among the multiple speech recognition networks used to train the speech enhancement network, the structural similarity between different speech recognition networks is less than a first threshold.

[0034] In another possible implementation, the trained speech enhancement network is obtained by training the speech enhancement network based on a total loss function value, where the total loss function value is obtained by weighted summing up the corresponding multiple loss function values ​​in multiple speech recognition networks, and the multiple loss function values ​​are obtained based on the outputs of multiple speech recognition networks.

[0035] In another possible implementation, the weight value corresponding to each speech recognition network in the multiple speech recognition networks is positively correlated with the loss function value corresponding to each speech recognition network.

[0036] In another possible implementation, the weight value corresponding to each speech recognition network is also related to the weight value of each speech recognition network in the historical iterative training process.

[0037] In another possible implementation, among the multiple speech recognition networks, the weight value corresponding to the speech recognition network whose loss function value is greater than the second threshold is 0.

[0038] In another possible implementation, the second threshold is N times the target loss function value, the target loss function value is the smallest loss function value among multiple loss function values ​​corresponding to multiple speech recognition networks, and N is greater than 1.

[0039] In another possible implementation, during the training of the speech enhancement network, the parameters of the multiple speech recognition networks are not changed.

[0040] In another possible implementation, the multiple speech recognition networks are all pre-trained networks.

[0041] In a third aspect, a model training device is provided, comprising: an acquisition module for acquiring training samples, the training samples comprising speech samples and real texts corresponding to the speech samples; a processing module for inputting the speech samples into a speech enhancement network to obtain enhanced speech samples; the processing module is also used to input the enhanced speech samples into different multiple speech recognition networks respectively to obtain speech recognition results output by each speech recognition network; a training module is used to train the speech enhancement network based on a total loss function value to obtain a speech enhancement network after parameter update; wherein the total loss function value is obtained based on the loss function value corresponding to each speech recognition network in the multiple speech recognition networks, and the loss function value corresponding to each speech recognition network is obtained based on the speech recognition results output by each speech recognition network and the real text.

[0042] In one possible implementation, during the training of the speech enhancement network, the parameters of the multiple speech recognition networks are not changed.

[0043] In another possible implementation, the multiple speech recognition networks are all pre-trained networks.

[0044] In another possible implementation, among the multiple speech recognition networks, different speech recognition networks correspond to different types of network structures.

[0045] In another possible implementation, among the multiple speech recognition networks, the structural similarity between different speech recognition networks is less than a first threshold.

[0046] In another possible implementation, the total loss function value is obtained by performing a weighted summation on the loss function value corresponding to each speech recognition network.

[0047] In another possible implementation, the weight value corresponding to each speech recognition network is positively correlated with the loss function value corresponding to each speech recognition network.

[0048] In another possible implementation, the weight value corresponding to each speech recognition network is also related to the weight value of each speech recognition network in the historical iterative training process.

[0049] In another possible implementation, among the multiple speech recognition networks, the weight value corresponding to the speech recognition network whose loss function value is greater than the second threshold is 0.

[0050] In another possible implementation, the second threshold is N times the target loss function value, the target loss function value is the smallest loss function value among multiple loss function values ​​corresponding to multiple speech recognition networks, and N is greater than 1.

[0051] In a fourth aspect, a speech processing device is provided, comprising: an acquisition module, used to obtain a speech recognition request, the speech recognition request comprising a first speech; a processing module, used to process the first speech through a trained speech enhancement network to obtain an enhanced speech, the trained speech enhancement network is obtained after training the speech enhancement network, and the speech enhancement network is connected to a plurality of parallel speech recognition networks during the training process, the inputs of the plurality of speech recognition networks are all outputs of the speech enhancement network, and the outputs of the plurality of speech recognition networks are used to construct a loss function for training the speech enhancement network; the processing module is also used to perform speech recognition on the enhanced speech to obtain a recognition text corresponding to the first speech.

[0052] In a possible implementation, among the multiple speech recognition networks used to train the speech enhancement network, different speech recognition networks correspond to different types of network structures.

[0053] In another possible implementation, among the multiple speech recognition networks used to train the speech enhancement network, the structural similarity between different speech recognition networks is less than a first threshold.

[0054] In another possible implementation, the trained speech enhancement network is obtained by training the speech enhancement network based on a total loss function value, where the total loss function value is obtained by weighted summing up the corresponding multiple loss function values ​​in multiple speech recognition networks, and the multiple loss function values ​​are obtained based on the outputs of multiple speech recognition networks.

[0055] In another possible implementation, the weight value corresponding to each speech recognition network in the multiple speech recognition networks is positively correlated with the loss function value corresponding to each speech recognition network.

[0056] In another possible implementation, the weight value corresponding to each speech recognition network is also related to the weight value of each speech recognition network in the historical iterative training process.

[0057] In another possible implementation, among the multiple speech recognition networks, the weight value corresponding to the speech recognition network whose loss function value is greater than the second threshold is 0.

[0058] In another possible implementation, the second threshold is N times the target loss function value, the target loss function value is the smallest loss function value among multiple loss function values ​​corresponding to multiple speech recognition networks, and N is greater than 1.

[0059] In another possible implementation, during the training of the speech enhancement network, the parameters of the multiple speech recognition networks are not changed.

[0060] In another possible implementation, the multiple speech recognition networks are all pre-trained networks.

[0061] In a fifth aspect, an electronic device is provided, which may include a processor, the processor and a memory are coupled, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method described in any implementation of the first aspect or the second aspect is implemented. For the processor to execute the steps in each possible implementation of the first aspect or the second aspect, the details can be referred to the first aspect or the second aspect, and will not be repeated here.

[0062] In a sixth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes the method described in any implementation of the first aspect or the second aspect.

[0063] In a seventh aspect, a circuit system is provided, the circuit system comprising a processing circuit, the processing circuit being configured to execute the method described in any implementation of the first aspect or the second aspect.

[0064] In an eighth aspect, a computer program product is provided, which, when executed on a computer, enables the computer to execute the method described in any implementation of the first or second aspect.

[0065] In a ninth aspect, a chip system is provided, the chip system including a processor for supporting a server or a threshold value acquisition device to implement the functions involved in any implementation of the first aspect or the second aspect, for example, sending or processing the data and / or information involved in the above method. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the server or communication device. The chip system can be composed of a chip, or it can include a chip and other discrete devices.

[0066] The beneficial effects of the second to ninth aspects mentioned above can be referred to the introduction of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A schematic diagram of the working principle of an automatic speech recognition system provided for this application;

[0068] Figure 2 A schematic diagram of a process for independent training of a speech enhancement network and a speech recognition network provided in this application;

[0069] Figure 3 A schematic diagram of an application scenario provided for this application;

[0070] Figure 4 A flowchart of a model training method provided in this application;

[0071] Figure 5 A flowchart of a training speech enhancement network provided in this application;

[0072] Figure 7 A schematic diagram of an application of a trained speech enhancement network provided in this application;

[0073] Figure 8 A schematic diagram of the structure of a speech enhancement network provided in this application;

[0074] Fig. 9 A schematic diagram for comparing the operation of a depth filter provided in this application;

[0075] Fig. 10A A schematic diagram of the structure of a model training device provided in this application;

[0076] Fig. 10B A schematic diagram of the structure of a speech processing device provided in this application;

[0077] Fig.11 A schematic diagram of an electronic device 1100 provided in the present application;

[0078] Fig.12 A schematic diagram of the structure of the chip provided in this application;

[0079] Fig.13 A schematic diagram of the structure of a computer-readable storage medium provided for this application. DETAILED DESCRIPTION

[0080] This application involves a lot of knowledge about speech and neural networks. In order to better understand the solution of this application, the following is an introduction to the relevant terms and concepts of speech and neural networks that may be involved in this application. It should be understood that the interpretation of related concepts may be limited due to the specific circumstances of this application, but it does not mean that this application is limited to this specific situation. The specific circumstances in different embodiments may also be different, which is not specifically limited here.

[0081] (1) Speech Recognition

[0082] Speech recognition refers to the conversion of the vocabulary content in human speech into computer-readable input, such as character sequences or binary codes.

[0083] (2) Speech recognition results

[0084] The speech recognition result refers to the text obtained by recognizing speech through speech recognition technology.

[0085] (3) Voice Enhancement

[0086] Speech enhancement refers to the technology of extracting useful speech signals from the noise background and suppressing and reducing noise interference when the speech signal is interfered with or even drowned by various noises. In this embodiment, speech enhancement refers to performing enhancement processing on the speech signal to improve the quality of the speech signal and the signal-to-noise ratio of the speech signal, so that the speech recognition network can output more accurate recognition text based on the enhanced speech signal.

[0087] (4) Neural Network

[0088] A neural network can be composed of neural units. Specifically, it can be understood as a neural network with an input layer, a hidden layer, and an output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. Among them, a neural network with many hidden layers is called a deep neural network (DNN). The work of each layer in the neural network can be expressed mathematically. To describe, from a physical level, the work of each layer in the neural network can be understood as completing the transformation from input space to output space (i.e., from the row space to the column space of the matrix) through five operations on the input space (the set of input vectors). These five operations include: 1. Dimension increase / reduction; 2. Enlargement / reduction; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are represented by Completed, operation 4 is completed by "+b", and operation 5 is implemented by "a()". The word "space" is used here because the classified object is not a single thing, but a class of things. Space refers to the collection of all individuals of this class of things, where W is the weight matrix of each layer of the neural network, and each value in the matrix represents the weight value of a neuron in the layer. The matrix W determines the spatial transformation from the input space to the output space described above, that is, the W of each layer of the neural network controls how to transform the space. The purpose of training a neural network is to eventually obtain the weight matrices of all layers of the trained neural network. Therefore, the training process of a neural network is essentially about learning how to control spatial transformation, or more specifically, learning the weight matrix.

[0089] (5) Convolutional Neural Network (CNN)

[0090] A convolutional neural network is a deep neural network with a convolutional structure. A convolutional neural network contains a feature extractor consisting of a convolutional layer and a subsampling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input feature map. A convolutional layer refers to a neural unit layer in a convolutional neural network that performs convolution processing on the input signal. In a convolutional layer of a convolutional neural network, a neural unit can only be connected to some neural units in adjacent layers. A convolutional layer usually contains several feature planes, and each feature plane can be composed of some neural units arranged in a rectangular shape. The neural units in the same feature plane share weights, and the shared weights here are the convolution kernels.

[0091] The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0092] (6) Recurrent Neural Network (RNN)

[0093] A recurrent neural network is a type of recursive neural network that takes sequence data as input, performs recursion in the direction of sequence evolution, and all nodes (recurrent units) are connected in a chain.

[0094] Recurrent neural networks have memory, parameter sharing, and Turing completeness, so they have certain advantages in learning the nonlinear features of sequences. Recurrent neural networks are used in natural language processing (NLP), such as speech recognition, language modeling, machine translation, and other fields, and are also used for various time series forecasting.

[0095] (7) Long Short-Term Memory (LSTM)

[0096] LSTM is a time recurrent neural network that is specially designed to solve the long-term dependency problem of general recurrent neural networks. It is usually used to process and predict important events with very long intervals and delays in time series.

[0097] (8) Loss Function

[0098] In the process of training a neural network, because we hope that the output of the neural network is as close as possible to the value we really want to predict, we can compare the current network's predicted value with the target value we really want, and then update the weight matrix of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the neural network). For example, if the network's predicted value is high, adjust the weight matrix to make it predict lower, and keep adjusting until the neural network can predict the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network becomes a process of minimizing this loss as much as possible.

[0099] (9) Back propagation algorithm

[0100] In the training process of the neural network, the error back propagation (BP) algorithm can be used to correct the size of the parameters in the initial neural network model, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0101] (10) Black Box System

[0102] A black box system is a system that can only access input ports and output ports, and the parameters inside the system are unknown and cannot be adjusted. For example, if a neural network is a black box system, you can only input data into the neural network and obtain the output data after the neural network processes the input data, but you cannot know and adjust the parameters in the neural network.

[0103] (11) White Box System

[0104] A white-box system is the opposite of a black-box system. A white-box system is a system whose internal parameters are known and adjustable. For example, a neural network is a white-box system, in which the structure and parameters of the neural network are known and adjustable.

[0105] (12) Attention Network

[0106] Attention network is a network model that uses attention mechanism to improve the model training speed. At present, the typical attention network includes Transformer model. The model using attention mechanism can give different weights to each part of the input sequence, so as to extract more important feature information from the input sequence, so that the model can finally get more accurate output.

[0107] Specifically, there are currently two ways of using a speech enhancement network in conjunction with a speech recognition network in the related art.

[0108] In the first method, speech enhancement and speech recognition are treated as two independent tasks. For the independent speech enhancement network, the speech enhancement index is used to improve the perceived quality and intelligibility of speech, so that the audio processed by the speech enhancement network sounds clearer to the human ear. Then, the audio processed by the speech enhancement network is input into the speech recognition network to obtain the recognition result output by the speech recognition network. In other words, the speech enhancement network and the speech recognition network are trained separately, and the training goal of the speech enhancement network is to make the speech enhancement network output speech that is easier for the human ear to recognize.

[0109] For example, see Figure 2 , Figure 2 A flow chart of independent training of a speech enhancement network and a speech recognition network provided in this application. Figure 2 As shown, in the training process of the speech enhancement network and the speech recognition network, the noisy speech (for example, the noisy speech obtained by artificially adding noise to the clean speech) is first input into the speech enhancement network, and the speech enhancement network performs enhancement processing on the noisy speech and outputs the denoised speech. Then, the denoised speech output by the speech enhancement network is input into the speech recognition network, and the speech recognition network performs speech recognition and outputs the recognized text. In this way, based on the denoised speech output by the speech enhancement network and the clean speech corresponding to the noisy speech, a speech enhancement loss function can be constructed, and the speech enhancement network is trained based on the speech enhancement loss function. In addition, based on the recognized text output by the speech recognition network and the real text corresponding to the noisy speech, a speech recognition loss function can be constructed, and then the speech recognition network is trained based on the speech recognition loss function. It can be seen that although the denoising network output by the speech enhancement network is used as the input of the speech recognition network, the loss functions used by the speech enhancement network and the speech recognition network are different, that is, the training objectives of the two networks are different.

[0110] In the second method, speech enhancement and speech recognition are treated as a whole. After the speech enhancement network and the speech recognition network are preliminarily pre-trained, the speech enhancement network is connected before the speech recognition network, so that the speech enhancement network and the speech recognition network are jointly trained, so as to adjust the parameters of the two networks at the same time. In other words, the speech enhancement network and the speech recognition network are trained simultaneously based on the same loss function, and the parameters of the speech enhancement network and the speech recognition network are updated together during the training process.

[0111] In the first method mentioned above, the inventors found that human ear perception and machine cognition are different, and human ear-friendly speech enhancement often cannot improve machine cognition. Since the speech enhancement network is only considered to make the speech enhancement network output more conducive to human hearing when training, without considering the loss function of speech recognition, it is ultimately found that adding a speech enhancement network does not significantly improve the accuracy of speech recognition, and in some cases may even damage the performance of speech recognition.

[0112] In the second method mentioned above, although the joint training of the speech enhancement network and the speech recognition network can help the speech enhancement network improve the accuracy of speech recognition, it will change the parameters of the speech recognition network, reduce the robustness of independent use, and make the entire system (i.e., the combination of the speech enhancement network and the speech recognition network) less robust in the face of data from more actual scenarios, thereby damaging the accuracy of speech recognition outside the training scenario. Moreover, in the second method, the joint training of the speech enhancement network and the speech recognition network means that the parameters of the speech recognition network need to be obtained and learning iterations need to be performed, which cannot be applied in scenarios where the speech recognition network is a black box system (i.e., the specific parameters in the speech recognition network cannot be obtained).

[0113] In general, the speech enhancement network added in the related technology is usually used in combination with a specific speech recognition network. The applicability of the speech enhancement network is poor and there are fewer scenarios in which it can be applied.

[0114] Based on this, the present application provides a model training method. During the training process of the speech enhancement network, multiple parallel speech recognition networks are connected after the speech enhancement network, and multiple different speech recognition networks respectively perform recognition processing on the speech data output by the speech enhancement network (that is, the input of each speech recognition network is the speech data output by the speech enhancement network), and then the loss function corresponding to each speech recognition network is constructed based on the speech recognition result output by each speech recognition network. Finally, by training the speech enhancement network with the loss functions of multiple speech recognition networks, the generalization and versatility of the speech enhancement network can be effectively improved, so that the speech enhancement network can be extended to speech recognition networks that have not been jointly trained.

[0115] The model training method provided in this application is described in detail below with reference to the accompanying drawings.

[0116] See also Figure 3 , Figure 3 A schematic diagram of an application scenario provided by this application. Figure 3 As shown, after the voice data including the user's original voice and noise are obtained through the sound pickup device, the voice data is input into the voice enhancement network trained by the model training method provided in this embodiment, and the voice enhancement network performs enhancement processing on the voice data to obtain enhanced voice data. Then, the enhanced voice data is input into the voice recognition network, and the voice recognition network outputs the recognition text, which can be applied in various scenarios, such as subtitle generation, meeting minutes, voice translation, and human-computer dialogue.

[0117] For example, in the fields of news broadcasting, live broadcasting or film production, the user's spoken lines are recognized in real time through the speech enhancement network and the speech recognition network to obtain the recognized text. Among them, the recognized text is used to generate subtitles to form a synchronized picture with sound and picture.

[0118] For example, in a video conference or voice conference scenario, the speech enhancement network and the speech recognition network can recognize the oral content of the participants in real time to obtain recognized text. In this way, after the meeting, the recognized text can be used to generate meeting minutes so that the participants can summarize the meeting content.

[0119] For example, in the scenario of speech translation or human-computer dialogue, the speech in the current environment is recognized in real time based on the speech enhancement network and the speech recognition network to obtain the recognized text. Then, the recognized text can be further used to perform translation in related languages, or the machine can perform corresponding actions according to the instructions of the recognized text.

[0120] In general, the speech enhancement network trained by the model training method provided in this application can be used to enhance speech data in order to improve the accuracy of language data recognition, so that the recognized text can be applied to various natural speech processing tasks downstream.

[0121] The model training method provided in the present application can be applied to electronic devices, especially electronic devices that need to perform speech recognition tasks. For example, the electronic device can be a server, a personal computer, a mobile phone, a personal computer (PC), a laptop, etc.

[0122] As a possible implementation method, the model training method provided in the present application can also be applied to a cluster system composed of multiple servers, in which the servers are used together to execute the model training method. For example, one or more steps in the model training process are respectively executed by multiple electronic devices.

[0123] The above introduces the scenarios and equipment used in the model training method provided by this application. Next, the specific process of the model training method provided by this application will be introduced.

[0124] See also Figure 4 , Figure 4 A flow chart of a model training method provided in this application. Figure 4 As shown, the process of the model training method includes the following steps 401-404.

[0125] Step 401: Obtain training samples, where the training samples include speech samples and real text corresponding to the speech samples.

[0126] In this embodiment, before training the speech enhancement network, a training sample for performing the training may be obtained first. In a training sample, a speech sample and a real text corresponding to the speech sample are included. The speech sample refers to a piece of speech data with noise, such as the voice of a user captured by a sound pickup device in a noisy scene. In addition, the real text is the text content that actually corresponds to the speech content in the speech sample. For example, assuming that the speech sample is a piece of speech data about the user's voice "Please play my favorite music", the real text corresponding to the speech sample is "Please play my favorite music". In other words, the real text corresponding to the speech sample is actually the recognition text obtained after the speech recognition network is expected to recognize the speech sample.

[0127] Step 402: Input the speech sample into the speech enhancement network to obtain an enhanced speech sample.

[0128] In this embodiment, the speech enhancement network is a network to be trained, and the speech enhancement network can perform speech enhancement on the input speech sample and output the enhanced speech sample. The speech enhancement network is mainly used to enhance the input speech sample, improve the quality of the speech sample, and improve the signal-to-noise ratio of the speech sample, so that the speech recognition network can output a more accurate recognition text based on the enhanced speech sample.

[0129] Specifically, the speech enhancement network is a neural network, such as a convolutional neural network, a recurrent neural network, or a LSTM neural network. This embodiment does not limit the specific structure of the speech enhancement network.

[0130] Step 403: Input the enhanced speech samples into different multiple speech recognition networks respectively to obtain the speech recognition results output by each speech recognition network.

[0131] In this embodiment, a plurality of different speech recognition networks are connected after the speech enhancement network, and the input of each speech recognition network is the enhanced speech sample output by the speech enhancement network, that is, the inputs of the plurality of speech recognition networks are the same.

[0132] Among them, the multiple speech recognition networks are multiple speech recognition networks with different network structures. Therefore, after each speech recognition network performs speech recognition on the enhanced speech sample, it may output different speech recognition results.

[0133] For example, see Figure 5 , Figure 5 A flow chart of a training speech enhancement network provided in this application. Figure 5 In the example, three speech recognition networks are connected after the speech enhancement network, namely speech recognition network 1, speech recognition network 2 and speech recognition network 3, and the inputs of the three speech recognition networks are all enhanced speech samples output by the speech enhancement network. Among them, after speech recognition network 1 performs speech recognition processing on the enhanced speech sample, it outputs speech recognition result 1; after speech recognition network 2 performs speech recognition processing on the enhanced speech sample, it outputs speech recognition result 2; after speech recognition network 3 performs speech recognition processing on the enhanced speech sample, it outputs speech recognition result 3.

[0134] It should be noted that Figure 5 In the figure, three speech recognition networks are used as an example for explanation. This embodiment does not limit the number of speech recognition networks connected after the speech enhancement network. Specifically, the number of speech recognition networks connected after the speech enhancement network can be determined based on the number of speech recognition networks that can be obtained in actual applications and factors such as computing power resources. Generally speaking, the more speech recognition networks connected after the speech enhancement network, the more computing power resources are required during the training process, and the better the effect of the speech enhancement network finally trained.

[0135] Step 404: Train the speech enhancement network based on the total loss function value to obtain a speech enhancement network with updated parameters.

[0136] In this embodiment, the total loss function value is obtained based on the loss function value corresponding to each speech recognition network in the multiple speech recognition networks, and the loss function value corresponding to each speech recognition network is obtained based on the speech recognition results and the real text output by each speech recognition network. That is to say, after the enhanced speech sample is processed by each speech recognition network, the speech recognition result output by each speech recognition network can be obtained. In this way, based on the real text corresponding to the speech sample and the speech recognition result output by each speech recognition network, the loss function corresponding to each speech recognition network can be constructed, and the loss function value of each speech recognition network for the speech sample can be calculated. Finally, by combining the multiple loss function values ​​corresponding to the multiple speech recognition networks, the total loss function value for training the speech enhancement network can be obtained, thereby realizing the updating of the parameters of the speech enhancement network based on the total loss function value. Among them, the method of updating the parameters of the speech enhancement network based on the total loss function value can be implemented by adopting the back propagation algorithm, and this embodiment does not specifically limit this.

[0137] In general, the output of the speech enhancement network is not directly used to construct the loss function for training the speech enhancement network, but is input into multiple speech recognition networks, and finally the loss function for training the speech enhancement network is constructed based on the outputs of multiple speech recognition networks. In this way, since the loss function for training the speech enhancement network is constructed based on the outputs of multiple speech recognition networks, the training goal of the speech enhancement network is actually to improve the recognition accuracy of multiple speech recognition networks, that is, the speech enhancement network learns how to output appropriate enhanced speech during the training process, so that multiple speech recognition networks can obtain more accurate speech recognition results when performing speech recognition based on the enhanced speech.

[0138] In practical applications, for each speech recognition network, the loss function corresponding to each speech recognition network can be constructed, and the total loss function of the speech enhancement network used for training can be obtained by combining the loss functions corresponding to each speech recognition network. In this way, after determining the total loss function corresponding to the speech enhancement network based on the loss function corresponding to each speech recognition network, the total loss function value in each round of iterative training can be calculated.

[0139] In other words, the goal of training the speech enhancement network based on the total loss function value is actually to make the enhanced speech data output by the speech enhancement network after performing enhancement processing on the speech data effectively improve the recognition effect of multiple speech recognition networks, that is, to make the speech enhancement network have better adaptability to multiple speech recognition networks.

[0140] For example, Figure 5As shown, for speech recognition result 1 output by speech recognition network 1, by calculating the difference between speech recognition result 1 and the real text, the loss function value 1 corresponding to speech recognition network 1 can be determined; for speech recognition result 2 output by speech recognition network 2, by calculating the difference between speech recognition result 2 and the real text, the loss function value 2 corresponding to speech recognition network 2 can be determined; for speech recognition result 3 output by speech recognition network 3, by calculating the difference between speech recognition result 3 and the real text, the loss function value 3 corresponding to speech recognition network 3 can be determined. Finally, by combining loss function value 1, loss function value 2 and loss function value 3, the total loss function value can be obtained.

[0141] In this solution, during the training process of the speech enhancement network, multiple parallel speech recognition networks are connected after the speech enhancement network, and multiple different speech recognition networks respectively perform recognition processing on the speech data output by the speech enhancement network, and then construct the loss function corresponding to each speech recognition network based on the speech recognition result output by each speech recognition network. Finally, by training the speech enhancement network with the loss functions of multiple speech recognition networks, the generalization and versatility of the speech enhancement network can be effectively improved, so that the speech enhancement network can be extended and applied to speech recognition networks that have not been jointly trained.

[0142] In addition, since the speech enhancement network is jointly trained based on multiple speech recognition networks in this embodiment, the trained speech enhancement network will have strong versatility. In this way, in actual application scenarios, when the speech enhancement network needs to be used with the target speech recognition network, even if the target speech recognition network is a black box system and the target speech recognition network and the speech enhancement network cannot be jointly trained one-to-one, the effect of the speech enhancement network when it is directly used with the target speech recognition network can be guaranteed, thereby improving the scenarios in which the speech enhancement network can be applied.

[0143] In essence, the model training method provided in this embodiment is actually a dynamic multi-teacher training method, that is, multiple speech recognition networks are used as teacher networks, and the speech enhancement network is used as the student network, and the student network is trained based on the teacher network. Please refer to the following Table 1, which shows the difference between the existing distillation method and the dynamic multi-teacher training method provided in this embodiment.

[0144] Table 1

[0145]

[0146] It should be noted that the above steps 401-404 actually introduce a round of training iterations performed on the speech enhancement network. In the actual training process of the speech enhancement network, multiple rounds of training iterations will be performed on the speech enhancement network, and the process of each round of training iterations can refer to the above steps 401-404. Finally, there are many ways to judge the degree of training of the speech enhancement network using the total loss function. The following provides some training termination conditions for the speech enhancement network.

[0147] (1) The total loss function value reaches the preset threshold.

[0148] Since the total loss function value used to train the speech enhancement network is obtained based on the loss functions corresponding to multiple speech recognition networks, a threshold value (e.g., 0.03) can be set in advance for the total loss function value. During the iterative training of the speech enhancement network, after each training, it is determined whether the total loss function value obtained in the current round of training reaches the threshold value. If not, the training continues. If it reaches the preset threshold value, the training is terminated. In this way, the network parameter values ​​of the speech enhancement network determined by the current round of training are used as the network parameter values ​​of the finally trained speech enhancement network.

[0149] (2) The total loss function begins to converge.

[0150] During the training of the speech enhancement network, if the difference between the total loss function value obtained in the current round of training and the total loss function value obtained in the previous round of training is within a preset range (e.g., 0.01), it is considered that the total loss function used to train the speech enhancement network has converged, and the training can be terminated. The values ​​of the network parameters of the speech enhancement network determined by the current round of training are used as the network parameter values ​​of the final trained speech enhancement network.

[0151] (3) The training reaches the preset number of times.

[0152] In this way, the number of iterations for training the speech enhancement network can be pre-configured (e.g., 1000 times). In the process of iterative training of the speech enhancement network, after each round of training, the values ​​of the network parameters of the speech enhancement network of that round are stored until the number of training iterations reaches the preset number. Afterwards, the speech enhancement network obtained in each round is verified using test data, and the value of the network parameter with the best performance is selected as the final value of the network parameter of the speech enhancement network.

[0153] Optionally, during the training of the speech enhancement network, the parameters of the multiple speech recognition networks do not change, that is, the multiple speech recognition networks are not trained together with the speech enhancement network. In this way, by freezing the parameters of the multiple speech recognition networks, it is possible to avoid the parameter changes of the speech recognition networks affecting the parameters of the speech enhancement network, and can well simulate the application scenario where the speech recognition network is a black box system (that is, the parameters of the speech recognition network are unknown and the parameters cannot be adjusted), improve the generalization of the speech enhancement network, and enable the speech enhancement network finally trained to be applied to other speech recognition networks that have not undergone joint training.

[0154] Exemplarily, multiple speech recognition networks are all pre-trained networks, that is, multiple speech recognition networks are all pre-trained. For example, multiple speech recognition networks can be pre-trained based on training samples, and multiple speech recognition networks have good speech recognition effects after training. In other words, by using a pre-trained speech recognition network to combine with a speech enhancement network, and only training the speech enhancement network during the training process, the speech recognition network can output a speech recognition result with higher accuracy during the training phase, and avoid the speech recognition result with lower accuracy affecting the training effect of the speech enhancement network. In addition, after the speech enhancement network is trained, it is usually used in conjunction with a trained speech recognition network. Therefore, in the training process, the speech enhancement network is trained in conjunction with the pre-trained speech recognition network, which can well simulate the subsequent application scenarios of the speech enhancement network and improve the speech enhancement effect of the speech enhancement network in actual application.

[0155] The above describes the process of determining the total loss function value based on the output results of multiple speech recognition networks and training the speech enhancement network. The following describes how to determine the total loss function based on the output results of multiple speech recognition networks in the above step 404.

[0156] In one possible example, the total loss function value used to train the speech enhancement network is obtained by performing a weighted summation of the loss function values ​​corresponding to each speech recognition network. In other words, a weight can be assigned to each speech recognition network, and then the weight and loss function value corresponding to each speech recognition network are multiplied to obtain the weighted loss function value corresponding to each speech recognition network; finally, the weighted loss function values ​​corresponding to each speech recognition network are added together to obtain the total loss function value. Since the loss function value corresponding to each speech recognition network is known, what actually needs to be determined during each round of training iterations is the weight corresponding to each speech recognition network.

[0157] Optionally, for multiple speech recognition networks, the weight value corresponding to each speech recognition network has a positive correlation with the loss function value corresponding to each speech recognition network. In other words, the larger the loss function value corresponding to the speech recognition network, the larger the weight value corresponding to the speech recognition network; the smaller the loss function value corresponding to the speech recognition network, the smaller the weight value corresponding to the speech recognition network.

[0158] In this scheme, by setting the weight value corresponding to the speech recognition network to have a positive correlation with the loss function value, a larger weight can be given to the speech recognition network with a large loss function value, so that the speech enhancement network is more inclined to improve the speech recognition network with relatively poor recognition effect during the training process, thereby improving the adaptability of the speech enhancement network to various speech recognition networks.

[0159] Optionally, the weight value corresponding to each speech recognition network is also related to the weight value of each speech recognition network in the historical iterative training process (i.e., the historical weight value). For example, for any speech recognition network, the weight value corresponding to the speech recognition network in the current training iteration round is related to the weight value corresponding to the speech recognition network in the previous training iteration process. In other words, when determining the weight value corresponding to the speech recognition network, it is necessary to determine it based on the weight value of the speech recognition network in the historical iterative training process, that is, the weight value corresponding to the speech recognition network is changed based on the weight value in the historical iterative training process, and will not differ too much from the weight value in the historical iterative training process.

[0160] In this scheme, in the process of determining the weight value of the speech recognition network, by introducing the weight value of the speech recognition network in the historical iterative training process, it can be ensured that the weight value of the speech recognition network changes relatively smoothly during the training process, thereby avoiding excessive changes in the total loss function value and affecting the training effect of the speech enhancement network.

[0161] In addition, since the loss function values ​​of multiple speech recognition networks are combined to obtain the total loss function value used to train the speech enhancement network in this embodiment, if the loss function value of a certain speech recognition network is particularly large (that is, the loss function value of the speech recognition network is much larger than the loss function values ​​of other speech recognition networks), a large gradient feedback will be generated during the training process of the speech enhancement network, affecting the training effect of the speech enhancement network.

[0162] Based on this, in this embodiment, a threshold is set for the loss function values ​​of multiple speech recognition networks. When the loss function value of the speech recognition network exceeds the threshold, the loss function value is discarded, that is, the total loss function value is no longer determined based on the loss function value, thereby avoiding bad learning samples from having a negative impact on the training of the speech enhancement network.

[0163] Exemplarily, among the multiple speech recognition networks, the weight value corresponding to the speech recognition network whose loss function value is greater than the second threshold is 0. In this way, in the process of calculating the total loss function value, since the weight value corresponding to the speech recognition network whose loss function value is greater than the second threshold is 0, the loss function value corresponding to the speech recognition network no longer affects the size of the total loss function value.

[0164] Among them, the second threshold is N times the target loss function value, the target loss function value is the smallest loss function value among the multiple loss function values ​​corresponding to the multiple speech recognition networks, and N is greater than 1. That is to say, after obtaining the loss function value corresponding to each speech recognition network in the multiple speech recognition networks, the smallest loss function value among the multiple loss function values ​​can be determined as the target loss function value, and the determined target loss function value is multiplied by N to obtain the second threshold. Specifically, the value of N can be, for example, an integer in the range of 5-10, and this embodiment does not specifically limit the value of N.

[0165] For ease of understanding, the following will introduce in detail the process of determining the total loss function value based on the loss function values ​​of multiple speech recognition networks with specific examples.

[0166] In one possible example, the process of determining the total loss function value can be divided into 6 parts, and the specific operation of each part is described as follows.

[0167] 1. Assume that in this embodiment, n speech recognition networks are connected after the speech enhancement network, and these n speech recognition networks are respectively represented as: Before starting training, the weight values ​​of each speech recognition network are initialized. In this embodiment, the average value a is used. i =(1 / n) as the initial weight value of each speech recognition network, that is, in the first training iteration, the weight value of each speech recognition network is 1 / n. i Represents the weight value corresponding to the i-th speech recognition network, 0 <i≤n。

[0168] 2. After the training starts, the original speech x (i.e. speech with noise) is first input into the speech enhancement network f se (·), and the enhanced speech x′=f se (x). Then, the enhanced speech x′ is input into the n speech recognition networks mentioned above to obtain the speech recognition results Then, combined with the speech recognition result y i With the real text y ref , calculate the loss function value L corresponding to each speech recognition network i =h(y i ,y ref ).

[0169] 3. Next, the adjustment threshold b is set in combination with the loss function value corresponding to each speech recognition network. Specifically, assuming that β is the manually set loss threshold adjustment coefficient, β is used to determine the adjustment threshold b. In this embodiment, b is designed based on β times the minimum loss function value, that is, b = β*min([L i ,i=1,2,…,n]), the value of β in this embodiment is generally 5 to 10.

[0170] 4. Use the global loss function value to update the weight of the loss function value corresponding to each speech recognition network: Among them, α is the manually set weight attenuation coefficient, which can be set by imitating the integral term in the proportional integral derivative (PID) control method commonly used in automatic control systems; a i is the weight value of the i-th speech recognition network in the current round of training iteration; a i ' is the weight value of the i-th speech recognition network in the last round of training iteration. It can be seen from the above formula that during the training process, the weight value of the loss function value corresponding to each speech recognition network will be dynamically adjusted according to the historical weight value and the current loss function value. The weight decay coefficient α is to introduce the historical weight value so that the change of the weight value of the current loss function value is relatively stable.

[0171] 5. According to the loss function value L of each speech recognition network i To calculate the Boolean quantity m i . Among them, m is used to dynamically control the impact of the loss function value of the speech recognition network. If the loss function value of the speech recognition network is less than the threshold b, then set the m corresponding to the speech recognition network i =1, so that the loss function value of the speech recognition network can participate in the training normally; if the loss function value of the speech recognition network is less than the threshold b, then set the m corresponding to the speech recognition network i = 0, so that the loss function value of the speech recognition network does not participate in the back propagation of the front-end speech enhancement network. By designing the Boolean quantity m i , which can avoid a certain abnormal loss function value from generating a large gradient return, thereby eliminating the influence of bad samples and ensuring the training effect of the speech enhancement network.

[0172] 6. At the end of each round of training iteration, the loss function values, weight values, and Boolean values ​​corresponding to all the above speech recognition networks are combined to calculate the total loss function value used to train the speech enhancement network. Among them, the total loss function value L is used for back propagation to update the parameters of the speech enhancement network.

[0173] The above describes the process of determining the total loss function value for training the speech enhancement network based on the loss function values ​​of multiple speech recognition networks. In the actual training process, the selection of multiple speech recognition networks will also affect the training effect of the speech enhancement network, so the specific method of selecting multiple speech recognition networks in this embodiment will be described below.

[0174] Specifically, during the training process, the speech enhancement network actually learns how to make the output enhanced speech data improve the speech recognition effect of each speech recognition network. If the structures of the multiple speech recognition networks connected to the speech enhancement network during the training process are closer, the speech enhancement network will be more easily trained to improve the speech recognition effect of these speech recognition networks with similar structures, while the improvement effect on speech recognition networks with other structures will be limited. Therefore, in order to ensure the versatility of the speech enhancement network, during the training process of the speech enhancement network, the structural differences between the multiple speech recognition networks connected to the speech enhancement network can be set as large as possible.

[0175] In addition, multiple speech recognition networks themselves need to have high speech recognition capabilities, and the speech recognition capabilities of multiple speech recognition networks should be close to each other, so as to avoid the situation where speech recognition networks with low speech recognition capabilities output speech recognition results with large differences, thereby affecting the training effect of the speech enhancement network.

[0176] In a possible implementation, the speech recognition network is selected based on the structural type of the speech recognition network, so that speech recognition networks with different structural types are selected.

[0177] Specifically, for the existing speech recognition network, the speech recognition network can be divided into several types according to the structural type of the speech recognition network. Then, a typical speech recognition network is selected from the divided types, thereby obtaining multiple speech recognition networks with different structural types. That is to say, among the multiple speech recognition networks connected to the speech enhancement network, different speech recognition networks correspond to different types of network structures.

[0178] Exemplarily, based on the structural type of the speech recognition network, the speech recognition network can be divided into the following five types: Listen Attend Spell (LAS) network, Connectionist Temporal Classification (CTC) network, Attention-based Encoder-Decoder (AED) network, Recurrent Neural Network Transducer (RNN-T) network, and Transformer Transducer network.

[0179] Among them, the LAS network usually includes two components: a listener and a speller. The listener is a pyramid recursive network encoder that receives the filter bank spectrum as input; the speller is a periodic network decoder based on the attention mechanism that can emit characters as output.

[0180] The CTC network is an improved RNN network that can directly use sequences for training. Specifically, the CTC network introduces a new loss function that allows the RNN to directly use unsegmented sequences for training.

[0181] The AED network is a network structure based on Encoder and Decoder. In the AED network, the Encoder extracts information from the input sequence (speech data), and the Decoder is an autoregressive model on the target sequence (text) (i.e., the previous text unit is input to predict the next text unit). At the same time, during the autoregressive calculation, the output encoding information of the Encoder is obtained based on the attention mechanism, so that the information of the input sequence can be utilized.

[0182] The RNN-T network is an improved network based on the CTC network. It mainly combines the hidden layer output of the Encoder part of the network with another output based on the CTC network, and this combined output is the result of a certain output of the final network after LSTM calculation. In essence, the RNN-T network adds the relationship between sequences to the CTC network.

[0183] The Transformer Transducer network is actually an improvement on the RNN-T network, mainly by replacing the Encoder in the RNN-T network from LSTM to Transformer.

[0184] During the training process of the speech enhancement network, one speech recognition network can be selected from the five types mentioned above to participate in the training of the speech enhancement network. That is, the multiple speech recognition networks connected to the speech enhancement network are LAS network, CTC network, AED network, RNN-T network and Transformer Transducer network. In this way, since the speech enhancement network uses various types of current speech recognition networks to cooperate with the training during the training stage, in the inference stage, when the speech enhancement network is matched with the target speech recognition network to perform speech recognition, even if the target speech recognition network is a black box system, the speech enhancement network can effectively improve the speech recognition effect of the target speech recognition network.

[0185] In another possible implementation, the speech recognition network is selected based on the structural similarity between the speech recognition networks, so as to select a speech recognition network with a relatively large structural difference.

[0186] Specifically, before executing the training of the speech enhancement network, a large number of speech recognition networks with different structures can be obtained as candidate networks. Then, for the speech recognition networks used as candidate networks, the structural similarity between the speech recognition networks is first determined, thereby obtaining the structural similarity between any two speech recognition networks. In this way, based on the structural similarity between the speech recognition networks, multiple speech recognition networks with relatively small structural similarities can be selected as networks to cooperate with the speech enhancement network training.

[0187] Exemplarily, among the multiple speech recognition networks connected to the speech enhancement network, the structural similarity between different speech recognition networks is less than the first threshold. That is to say, for the multiple speech recognition networks connected to the speech enhancement network, the structural similarity between any two speech recognition networks is less than the first threshold. Among them, the first threshold can be determined according to the method of determining the structural similarity of the speech recognition network, and according to the needs in the actual application process (such as the number of selected speech recognition networks is limited). This embodiment does not limit the specific value of the first threshold.

[0188] In one possible example, in this embodiment, the speech recognition network as a candidate network may be loaded using a unified deep learning framework, and then the structure of each speech recognition network is output in the form of text (for example, the structure of each speech recognition network is printed in a log document), and the structural text corresponding to each speech recognition network is obtained (i.e., the text used to describe the structure of the speech recognition network). Then, based on the text comparison algorithm, the differences between different structural texts are compared to obtain the similarity scores between the two structural texts, and the similarity scores can be used as the structural similarity between the speech recognition networks.

[0189] For example, please refer to Table 2 and Table 3 below, both of which show similarity scores between structured texts corresponding to multiple different speech recognition networks.

[0190] Table 2

[0191] Speech Recognition Network 1 Speech Recognition Network 2 Speech Recognition Network 3 Speech Recognition Network 4 Speech Recognition Network 1 4.64 3.13 3.12 3.12 Speech Recognition Network 2 4.46 3.67 3.67 Speech Recognition Network 3 4.50 4.50 Speech Recognition Network 4 4.50

[0192] In Table 2, this embodiment selects four different speech recognition networks, and determines the structural similarity between the two speech recognition networks by comparing the similarity scores between the structural texts corresponding to the four speech recognition networks. Table 2 shows the similarity scores between the speech recognition networks. The higher the similarity score, the higher the structural similarity between the two speech recognition networks; the smaller the similarity score, the lower the structural similarity between the two speech recognition networks.

[0193] Table 3

[0194]

[0195] In Table 3, this embodiment is based on the four speech recognition networks selected in Table 2, after training these four speech recognition networks with the same batch of training data, and then compares the similarity scores between the structured texts corresponding to the four trained speech recognition networks.

[0196] As can be seen from Table 2, the similarity scores of the same speech recognition network are relatively high, generally around 4.5; while the gap between different speech recognition networks is relatively large, generally below 4.0. As can be seen from Table 3, after training, under the joint effect of parameters and structure in the speech recognition network, the similarity score differences of different speech recognition networks continue, and the similarity scores of the same speech recognition network can also be retained, proving the effectiveness of determining the structural similarity of speech recognition networks based on structured text.

[0197] After obtaining the trained speech enhancement network based on the model training method provided in this embodiment, the trained speech enhancement network can be used with the target speech recognition network to complete the speech recognition task in the actual application scenario. The target speech recognition network refers to a network used to perform speech recognition tasks in the actual application scenario. The target speech recognition network is, for example, a black box system or a white box system, which is not limited in this embodiment.

[0198] For example, see Figure 6 , Figure 6 A flow chart of a speech processing method provided in an embodiment of the present application. Figure 6 As shown, the speech processing method includes the following steps 601-603.

[0199] Step 601: Obtain a speech recognition request, where the speech recognition request includes a first speech, and the speech recognition request is used to request recognition of the first speech.

[0200] For example, in scenarios such as subtitle generation, meeting minutes, voice translation, and human-computer dialogue, the electronic device responsible for voice recognition processing (such as a server, personal computer, smart phone, tablet computer, etc.) can obtain a voice recognition request sent by the sound pickup device, and the voice recognition request includes the first voice collected by the sound pickup device. The electronic device responsible for voice recognition processing and the sound pickup device can be integrated into the same device (such as integrated into the same smart phone), or they can be separate different devices (such as the electronic device is a server and the sound test device is a voice recorder).

[0201] Step 602, the first speech is processed by the trained speech enhancement network to obtain an enhanced first speech, the trained speech enhancement network is obtained after training the speech enhancement network, and the speech enhancement network is connected to multiple parallel speech recognition networks during the training process, the inputs of the multiple speech recognition networks are all outputs of the speech enhancement network, and the outputs of the multiple speech recognition networks are used to construct a loss function for training the speech enhancement network.

[0202] Specifically, the trained speech enhancement network used in step 602 is obtained by training based on the model training method introduced in the above embodiment. Therefore, the training process of the trained speech enhancement network can refer to the introduction of the above embodiment, which will not be repeated here.

[0203] Step 603: Perform speech recognition on the enhanced first speech to obtain a recognition text corresponding to the first speech.

[0204] Among them, the electronic device can use a speech recognition network to perform speech recognition on the enhanced first speech to obtain a recognized text. The speech recognition network is a neural network used to perform speech recognition. For details, reference can be made to the network provided in the relevant technology, which will not be repeated here. In addition, the speech recognition network can be a black box system, that is, the parameters of the speech recognition network are not available, and the speech recognition network does not participate in the training of the trained speech enhancement network. In this case, since the trained speech enhancement network is pre-trained based on multiple parallel speech recognition networks, it has high generalization and versatility. Therefore, even if the speech recognition network as a black box system cannot be jointly trained with the speech enhancement network, better speech recognition results can be obtained.

[0205] In addition, the recognized text can be used by electronic devices to perform subsequent tasks, such as generating subtitles, generating meeting minutes, generating speech translation results, or outputting human-computer dialogue results.

[0206] In a possible implementation, among the multiple speech recognition networks used to train the speech enhancement network, different speech recognition networks correspond to different types of network structures.

[0207] In another possible implementation, among the multiple speech recognition networks used to train the speech enhancement network, the structural similarity between different speech recognition networks is less than a first threshold.

[0208] In another possible implementation, the trained speech enhancement network is obtained by training the speech enhancement network based on a total loss function value, where the total loss function value is obtained by weighted summing up the corresponding multiple loss function values ​​in multiple speech recognition networks, and the multiple loss function values ​​are obtained based on the outputs of multiple speech recognition networks.

[0209] In another possible implementation, the weight value corresponding to each speech recognition network in the multiple speech recognition networks is positively correlated with the loss function value corresponding to each speech recognition network.

[0210] In another possible implementation, the weight value corresponding to each speech recognition network is also related to the weight value of each speech recognition network in the historical iterative training process.

[0211] In another possible implementation, among the multiple speech recognition networks, the weight value corresponding to the speech recognition network whose loss function value is greater than the second threshold is 0.

[0212] In another possible implementation, the second threshold is N times the target loss function value, the target loss function value is the smallest loss function value among multiple loss function values ​​corresponding to multiple speech recognition networks, and N is greater than 1.

[0213] In another possible implementation, during the training of the speech enhancement network, the parameters of the multiple speech recognition networks are not changed.

[0214] In another possible implementation, the multiple speech recognition networks are all pre-trained networks.

[0215] For example, see Figure 7 , Figure 7 This is a schematic diagram of an application of a trained speech enhancement network provided in this application. Figure 7 As shown, the trained speech enhancement network is applied to the speech recognition system to cooperate with the target speech recognition network to complete the speech recognition task. Among them, the speech recognition system includes the following four parts.

[0216] The first part of the structure is the input device, which can include various types of sound pickup devices, such as headphone receivers, computer microphones, etc. The input device is mainly used to obtain the voice signal to be recognized, and the voice signal is not limited to mono or multi-channel. In addition, the input device can also be a language signal receiving device, which is used to receive voice signals transmitted from the network, such as voice signals from conference systems, voice signals from other sources, etc. After acquiring the voice signal, the input device converts the voice signal into a form that can be stored by the computer and temporarily stores it in the computer system, for example, converting the voice signal into a storage format such as wav, mp3, etc. for storage.

[0217] The second part of the structure is a speech enhancement network trained by the model training method provided in this embodiment. In this embodiment, the speech enhancement network takes the speech waveform as input, converts the speech waveform from the time domain to the frequency domain through Fourier transform, and then obtains the enhanced speech signal after nonlinear calculation mapping of the speech enhancement network, and then converts the enhanced speech signal back to the time domain waveform through inverse Fourier transform.

[0218] The third part of the structure is a speech recognition network, and the speech recognition network can be a black box system or a white box system. The input of the speech recognition network is the speech waveform output by the speech enhancement network, and the recognition text is predicted by the speech recognition network as the output.

[0219] The fourth part is an output device, which is used to output or transmit the recognized text obtained in the previous step to a network. For example, the output device is a local terminal device (such as a personal computer, a vehicle-mounted terminal, a smart phone, etc.), which can output the recognized text as a real-time subtitle display of a local software, or encode the recognized text for network instant messaging, etc. This embodiment does not limit the use of the recognized text.

[0220] See also Figure 8 , Figure 8 Schematic diagram of the structure of a speech enhancement network provided in the present application. Specifically, an example of the structure of a speech enhancement network trained by the model training method provided in the above embodiment is as follows: Figure 8 As shown. Figure 8 As shown, the speech enhancement network includes the following structural modules connected in sequence: molecular band module, complex convolution module, complex time-frequency domain recurrent neural network, complex convolution module, deep filter, and child merging module.

[0221] exist Figure 8 In the speech enhancement network shown, the input of the molecular band module is the spectrum of the speech signal. The molecular band module is used to multiply the input spectrum by the independent learnable parameters of each frequency band to perform molecular band operations. Through parallel processing, the running speed of the model is improved. The speech enhancement network finally has a child merging module, which corresponds to the molecular band module.

[0222] The complex convolution module is connected to the molecular band module. The complex convolution module uses two convolution layers (real convolution layer and imaginary convolution layer) to design the encoding structure. Among them, the real convolution layer is used to process the real part of the input complex number, and the imaginary convolution layer is used to process the imaginary part of the input complex number to realize simulated complex multiplication.

[0223] The complex convolution module is followed by a complex time-frequency domain recurrent neural network. The structure of the complex time-frequency domain recurrent neural network is the same as the design idea of ​​the encoding layer in the complex convolution module, that is, the complex time-frequency domain recurrent neural network uses two LSTM layers to simulate complex multiplication. The complex time-frequency domain recurrent neural network first performs complex multiplication on the frequency axis (F-LSTM), and a bidirectional LSTM is used at this time; then, complex multiplication is performed on the time axis (T-LSTM), and a unidirectional LSTM is used at this time to ensure that the network model is a causal model.

[0224] A complex convolution module is connected after the complex time-frequency domain recurrent neural network, and the two complex convolution modules before and after the complex time-frequency domain recurrent neural network are connected through a convolution channel.

[0225] The complex convolutional module after the complex time-frequency domain recurrent neural network is also connected with a deep filter. Fig. 9 , Fig. 9 A schematic diagram of the working comparison of a depth filter provided in this application. Fig. 9 As shown in (a) in FIG, after estimating the mask, the speech enhancement network usually multiplies the mask with the original input spectrum to obtain the output spectrum. At this time, there is a one-to-one correspondence between the input spectrum and the output spectrum. Fig. 9 As shown in (b), there is a many-to-one correspondence in the deep filter. The output spectrum point is inferred from the input spectrum point and its adjacent points. The introduction of contextual information is beneficial to improve the output result of the speech enhancement network.

[0226] Based on the model training method proposed in the above embodiment and the application method of the speech enhancement network obtained by training, this embodiment conducts comparative experiments and effect verification. Specifically, in this embodiment, three speech recognition networks are selected as demonstrations, and after freezing the parameters of the three speech recognition networks, the black box system is simulated to conduct joint training with the speech enhancement network (N to 1 training), and the black box system used for testing does not participate in the training. At the same time, as a comparison, another set of experiments is designed, which uses a speech recognition network with parameters that are already adjustable (i.e., a white box system) to cooperate with the speech enhancement network for joint training. The training data and test data used in each group of experiments are consistent, and the experimental results are shown in Table 4 below.

[0227] Table 4

[0228]

[0229]

[0230] Generally speaking, the evaluation index of speech recognition system is word error rate (WER), and the smaller the WER, the better the recognition result. As can be seen from the above table, first of all, according to the first and fourth rows, when the speech recognition network is a white box system or a black box system, there are similar effects on the target test set, that is, when no speech enhancement system is connected externally, there is an error rate of nearly 30%, which has a significant impact on the cognition of speech recognition results. When combined with 1-to-1 joint training, the speech enhanced by the speech enhancement network and then passed through the speech recognition network as a white box system can obtain an absolute improvement of 4.55%, and a relative improvement of 16%, which proves that joint training can effectively improve the situation where noise interferes with the accuracy of the speech recognition network. However, when the speech enhancement network is coupled with the speech recognition network as a black box system, side effects occur, and the speech recognition accuracy decreases instead, which also illustrates the limitations of traditional methods. After applying the method provided in this embodiment for training a speech enhancement network by combining multiple speech recognition networks, the trained speech enhancement network can effectively couple with the speech recognition network as a black box system, achieving a WER of 24.13%, which is close to the effect of one-to-one joint training under a white box system.

[0231] The above introduces the model training method provided by this application. Next, the device used to execute the above-mentioned model training method and speech processing method will be introduced.

[0232] See also Fig. 10A , Fig. 10A This is a schematic diagram of the structure of a model training device provided in this application. Fig. 10A As shown, the model training device provided by the present application includes: an acquisition module 1001, which is used to acquire training samples, and the training samples include speech samples and real texts corresponding to the speech samples; a processing module 1002, which is used to input the speech samples into a speech enhancement network to obtain enhanced speech samples; the processing module 1002 is also used to input the enhanced speech samples into different multiple speech recognition networks respectively to obtain the speech recognition results output by each speech recognition network; a training module 1003, which is used to train the speech enhancement network based on the total loss function value to obtain the speech enhancement network after parameter update; wherein the total loss function value is obtained based on the loss function value corresponding to each speech recognition network in the multiple speech recognition networks, and the loss function value corresponding to each speech recognition network is obtained based on the speech recognition results output by each speech recognition network and the real text.

[0233] In one possible implementation, during the training of the speech enhancement network, the parameters of the multiple speech recognition networks are not changed.

[0234] In a possible implementation, the multiple speech recognition networks are all pre-trained networks.

[0235] In a possible implementation, among the multiple speech recognition networks, different speech recognition networks correspond to different types of network structures.

[0236] In a possible implementation, among the multiple speech recognition networks, the structural similarity between different speech recognition networks is less than a first threshold.

[0237] In a possible implementation, the total loss function value is obtained by performing a weighted summation on the loss function value corresponding to each speech recognition network.

[0238] In a possible implementation, the weight value corresponding to each speech recognition network is positively correlated with the loss function value corresponding to each speech recognition network.

[0239] In a possible implementation, the weight value corresponding to each speech recognition network is also related to the weight value of each speech recognition network in the historical iterative training process.

[0240] In a possible implementation, among multiple speech recognition networks, the weight value corresponding to the speech recognition network whose loss function value is greater than the second threshold is 0.

[0241] In one possible implementation, the second threshold is N times the target loss function value, the target loss function value is the smallest loss function value among multiple loss function values ​​corresponding to multiple speech recognition networks, and N is greater than 1.

[0242] because Fig. 10A The model training setup shown corresponds to Figure 4 The method shown, therefore Fig. 10A The specific implementation method of the model training device shown and its technical effects can be found in the relevant description in the aforementioned embodiments, which will not be repeated here.

[0243] See also Fig. 10B , Fig. 10B This is a schematic diagram of the structure of a speech processing device provided in this application. Fig. 10BAs shown, the speech processing device provided by the present application includes: an acquisition module 1004, which is used to obtain a speech recognition request, and the speech recognition request includes a first speech; a processing module 1005, which is used to process the first speech through a trained speech enhancement network to obtain an enhanced speech, the trained speech enhancement network is obtained after training the speech enhancement network, and the speech enhancement network is connected to a plurality of parallel speech recognition networks during the training process, the inputs of the plurality of speech recognition networks are all outputs of the speech enhancement network, and the outputs of the plurality of speech recognition networks are used to construct a loss function for training the speech enhancement network; the processing module 1005 is also used to perform speech recognition on the enhanced speech to obtain a recognition text corresponding to the first speech.

[0244] In a possible implementation, among the multiple speech recognition networks used to train the speech enhancement network, different speech recognition networks correspond to different types of network structures.

[0245] In another possible implementation, among the multiple speech recognition networks used to train the speech enhancement network, the structural similarity between different speech recognition networks is less than a first threshold.

[0246] In another possible implementation, the trained speech enhancement network is obtained by training the speech enhancement network based on a total loss function value, where the total loss function value is obtained by weighted summing up the corresponding multiple loss function values ​​in multiple speech recognition networks, and the multiple loss function values ​​are obtained based on the outputs of multiple speech recognition networks.

[0247] In another possible implementation, the weight value corresponding to each speech recognition network in the multiple speech recognition networks is positively correlated with the loss function value corresponding to each speech recognition network.

[0248] In another possible implementation, the weight value corresponding to each speech recognition network is also related to the weight value of each speech recognition network in the historical iterative training process.

[0249] In another possible implementation, among the multiple speech recognition networks, the weight value corresponding to the speech recognition network whose loss function value is greater than the second threshold is 0.

[0250] In another possible implementation, the second threshold is N times the target loss function value, the target loss function value is the smallest loss function value among multiple loss function values ​​corresponding to multiple speech recognition networks, and N is greater than 1.

[0251] In another possible implementation, during the training of the speech enhancement network, the parameters of the multiple speech recognition networks are not changed.

[0252] In another possible implementation, the multiple speech recognition networks are all pre-trained networks.

[0253] Fig.11 A schematic diagram of an electronic device 1100 provided in the present application, the electronic device 1100 may be used to independently execute Figure 4 The model training method shown.

[0254] In some possible implementations, Figure 4 When the model training method shown is performed by multiple electronic devices in a cluster system, each electronic device in the cluster system can Fig.11 An electronic device 1100 is shown.

[0255] like Fig.11 As shown, the electronic device 1100 includes a processor 1101, a memory 1102, and a communication interface 1103. The processor 1101, the memory 1102, and the communication interface 1103 communicate via a bus 1104, and may also communicate via other means such as wireless transmission. The memory 1102 is used to store instructions, and the processor 1101 is used to execute instructions stored in the memory 1102. Furthermore, the electronic device 1100 may also include a memory unit 1105, and the memory unit 1105 may be connected to the processor 1101, the storage medium 1102, and the communication interface 1103 via the bus 1104. The memory 1102 stores program code, and the processor 1101 may call the program code stored in the memory 1102 to execute. Figure 4 The model training method shown.

[0256] It should be understood that in the present application, the processor 1101 may be a CPU, and the processor 1101 may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete device components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0257] The memory 1102 may include a read-only memory and a random access memory, and provide instructions and data to the processor 1101. The memory 1102 may also include a non-volatile random access memory. For example, the memory 1102 may also store information about the device type.

[0258] The memory 1102 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0259] The communication interface 1103 is used to communicate with other devices connected to the electronic device 1100. The bus 1104 may include a power bus, a control bus, a status signal bus, etc. in addition to the data bus. However, for the sake of clarity, various buses are labeled as bus 1104 in the figure.

[0260] It should be understood that the electronic device 1100 according to the present application may correspond to the model training device in the present application, and the above and other operations and / or functions implemented by the electronic device 1100 are respectively for implementing Figure 4 For the sake of brevity, the corresponding processes of each method in are not repeated here.

[0261] Optionally, the electronic device provided in the present application can be specifically implemented by a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit, etc. The processing unit can execute the computer execution instructions stored in the storage unit, so that the chip in the execution device executes the method for selecting the model hyperparameters described in the above embodiment, or so that the chip in the training device executes the method for selecting the model hyperparameters described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc., and the storage unit can also be a storage unit located outside the chip in the wireless access device end, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0262] For details, please refer to Fig.12 , Fig.12 A schematic diagram of the structure of the chip provided in this application, the chip can be expressed as a neural network processor NPU 1200, NPU 1200 is mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 1203, which is controlled by the controller 1204 to extract matrix data from the memory and perform multiplication operations.

[0263] In some implementations, the operation circuit 1203 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 1203 is a two-dimensional systolic array. The operation circuit 1203 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1203 is a general-purpose matrix processor.

[0264] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 1202 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1201 and performs matrix operation with matrix B, and the partial result or final result of the matrix is ​​stored in the accumulator 1208.

[0265] The unified memory 1206 is used to store input data and output data. The weight data is directly transferred to the weight memory 1202 through the direct memory access controller (DMAC) 1205. The input data is also transferred to the unified memory 1206 through the DMAC.

[0266] BIU stands for Bus Interface Unit, i.e., bus interface unit 1210 , which is used for the interaction between AXI bus, DMAC and instruction fetch buffer (IFB) 1209 .

[0267] The bus interface unit 1210 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 1209 to obtain instructions from the external memory, and is also used for the storage unit access controller 1205 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0268] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1206 or to transfer weight data to the weight memory 1202 or to transfer input data to the input memory 1201.

[0269] The vector calculation unit 1207 includes multiple operation processing units, and further processes the output of the operation circuit 1203 when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization (Batch Normalization, BN), pixel-level summation, upsampling of feature planes, etc.

[0270] In some implementations, the vector calculation unit 1207 can store the processed output vector to the unified memory 1206. For example, the vector calculation unit 1207 can apply a linear function; or a nonlinear function to the output of the operation circuit 1203, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 1207 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1203, for example, for use in a subsequent layer in a neural network.

[0271] An instruction fetch buffer 1209 connected to the controller 1204 is used to store instructions used by the controller 1204;

[0272] Unified memory 1206, input memory 1201, weight memory 1202 and instruction fetch memory 1209 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0273] The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0274] See also Fig.13 , Fig.13 This is a schematic diagram of the structure of a computer-readable storage medium provided by the present application. The present application also provides a computer-readable storage medium. In some embodiments, the above Figure 4 The disclosed methods may be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of manufacture.

[0275] Fig.13 Schematically illustrates a conceptual partial view of an example computer-readable storage medium including a computer program for executing a computer process on a computing device, arranged in accordance with at least some embodiments presented herein.

[0276] In one embodiment, the computer readable storage medium 1300 is provided using a signal bearing medium 1301. The signal bearing medium 1301 may include one or more program instructions 1302, which when executed by one or more processors may provide the above-mentioned Figure 4 The functions or parts of the functions described. In addition, Fig.13 Program instructions 1302 in also describe example instructions.

[0277] In some examples, signal bearing medium 1301 may include computer readable medium 1303 such as, but not limited to, a hard drive, a compact disk (CD), a digital video disk (DVD), a digital tape, memory, ROM or RAM, and the like.

[0278] In some embodiments, the signal bearing medium 1301 may include a computer recordable medium 1304, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, etc. In some embodiments, the signal bearing medium 1301 may include a communication medium 1305, such as, but not limited to, a digital and / or analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communication link, a wireless communication link, etc.). Thus, for example, the signal bearing medium 1301 may be communicated by a wireless form of the communication medium 1305 (e.g., a wireless communication medium that complies with the IEEE 802.13 standard or other transmission protocol).

[0279] The one or more program instructions 1302 may be, for example, computer executable instructions or logic implementation instructions. In some examples, the computing device of the computing device may be configured to provide various operations, functions, or actions in response to the program instructions 1302 communicated to the computing device via one or more of the computer readable medium 1303, the computer recordable medium 1304, and / or the communication medium 1305.

[0280] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0281] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0282] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0283] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, training equipment, or data center to another website, computer, training equipment, or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, data center, etc. that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

Claims

1. A speech processing method, characterized in that: include: Obtaining a speech recognition request, the speech recognition request including a first speech, the speech recognition request being used to request recognition of the first speech; Processing the first speech through a trained speech enhancement network to obtain an enhanced first speech, wherein the trained speech enhancement network is obtained after training the speech enhancement network, and the speech enhancement network is connected to a plurality of parallel speech recognition networks during the training process, the inputs of the plurality of speech recognition networks are all outputs of the speech enhancement network, and the outputs of the plurality of speech recognition networks are used to construct a loss function for training the speech enhancement network; Speech recognition is performed on the enhanced first speech to obtain a recognition text corresponding to the first speech.

2. The method according to claim 1, characterized in that Among the multiple speech recognition networks, different speech recognition networks correspond to different types of network structures.

3. The method according to claim 1 or 2, characterized in that: Among the multiple speech recognition networks, the structural similarity between different speech recognition networks is less than a first threshold.

4. The method according to any one of claims 1 to 3, characterized in that: The trained speech enhancement network is obtained by training the speech enhancement network based on a total loss function value, wherein the total loss function value is obtained by weighted summing up the corresponding multiple loss function values ​​in the multiple speech recognition networks, and the multiple loss function values ​​are obtained based on the outputs of the multiple speech recognition networks.

5. The method according to claim 4, characterized in that The weight value corresponding to each speech recognition network in the multiple speech recognition networks has a positive correlation with the loss function value corresponding to each speech recognition network.

6. The method according to claim 4 or 5, characterized in that: The weight value corresponding to each speech recognition network is also related to the weight value of each speech recognition network in the historical iterative training process.

7. The method according to any one of claims 4 to 6, characterized in that: Among the multiple speech recognition networks, the weight value corresponding to the speech recognition network whose loss function value is greater than the second threshold is 0.

8. The method according to claim 7, characterized in that The second threshold is N times the target loss function value, the target loss function value is the smallest loss function value among the multiple loss function values ​​corresponding to the multiple speech recognition networks, and N is greater than 1.

9. The method according to any one of claims 1 to 8, characterized in that: During the training of the speech enhancement network, the parameters of the plurality of speech recognition networks are not changed.

10. The method according to any one of claims 1 to 9, characterized in that: The multiple speech recognition networks are all pre-trained networks.

11. A speech processing device, characterized in that: include: An acquisition module, configured to acquire a speech recognition request, wherein the speech recognition request includes a first speech; A processing module, configured to process the first speech through a trained speech enhancement network to obtain an enhanced speech, wherein the trained speech enhancement network is obtained after training a speech enhancement network, and the speech enhancement network is connected to a plurality of parallel speech recognition networks during the training process, the inputs of the plurality of speech recognition networks are all outputs of the speech enhancement network, and the outputs of the plurality of speech recognition networks are used to construct a loss function for training the speech enhancement network; The processing module is further used to perform speech recognition on the enhanced speech to obtain a recognition text corresponding to the first speech.

12. An electronic device, characterized in that: The electronic device comprises a memory and a processor; the memory stores codes, the processor is configured to execute the codes, and when the codes are executed, the electronic device executes the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The method comprises computer-readable instructions, which, when executed on a computer, cause the computer to perform the method according to any one of claims 1 to 10.

Images