Speech recognition model training, speech recognition method and device

Abstract

The present invention provides a speech recognition model training, speech recognition method, and apparatus, belonging to the field of artificial intelligence technology. The method comprises: inputting a speech sample into a speech recognition model to be trained; obtaining the total loss during the recognition process to train the speech recognition model. The speech recognition model comprises a speech encoder, an information decoupling module, and a speech decoder; the information decoupling module comprises a speech information encoding unit, a speech information reconstruction unit, a gradient inversion unit, a content information reconstruction unit, and a paralanguage information reconstruction unit; the content information reconstruction module reconstructs paralanguage information into content information; and the paralanguage information reconstruction module reconstructs content information into paralanguage information. The present invention combines the task of identifying content information with the task of removing paralanguage information to form an end-to-end method. During the training process, content information and paralanguage information are clearly defined, facilitating the model's ability to clearly remove paralanguage information, thereby improving the robustness of the speech recognition system.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a speech recognition model training, a speech recognition method and a speech recognition device. Background Art

[0002] Automatic speech recognition, or Automated Speech Recognition (ASR), involves a computer processor receiving speech signals and converting them into human-readable text. This technology is widely used in mobile voice assistants, input method software, in-car navigation systems, and various AI wearable devices, and possesses significant application value. Paralinguistic information in speech, which refers to information unrelated to the content of the speech, such as the speaker's voiceprint and accent, often negatively impacts speech recognition tasks.

[0003] While some research indicates that paralinguistic information negatively impacts speech recognition, relatively little research has focused on optimizing paralinguistic information in speech to improve it. A representative example is the pre-training framework ContentVec proposed by the MIT-IBM Watson AI Lab. While this study doesn't explicitly identify paralinguistic information as a key factor in speech recognition, it does note that speaker characteristics (a type of paralinguistic information) in speech signals can reduce speech recognition performance. The study proposes a Hubert-based pre-training framework that removes speaker information from acoustics during speech representation learning, thereby improving speech recognition. A study published by the University of California, Berkeley, and Amazon proposes separating content-relevant information from speech signals (serving content-relevant tasks, such as speech recognition) from content-irrelevant information (serving content-independent tasks, such as voiceprint recognition). This research then uses the information decomposition scheme from DiCyR to decouple the single mixed information into two discriminative pieces.

[0004] Existing solutions, such as ContentVec, require pre-training with large amounts of speech data to generate a speech feature representation model with a small amount of speaker characteristics. This model is then fine-tuned using speech recognition tasks. These solutions only remove speaker information during pre-training, but lack a corresponding method during fine-tuning. This can lead to increased speaker information in the speech representations obtained during fine-tuning.

[0005] As for the algorithm for decomposing content-related information from content-irrelevant information, first, the definition of these two types of information in the algorithm is not clear enough, resulting in the decomposed content-related information still containing a lot of content-irrelevant information, which cannot achieve the ideal effect; second, the algorithm decouples the deep features of the speech encoder, and the deep features of the model often contain more content information, while the shallow features contain more paralinguistic information. The content information of the deep features has been interfered with by the paralinguistic information of the shallow features, which makes it difficult for the model to separate the paralinguistic information.

[0006] In view of this, there is an urgent need to improve the existing speech recognition model to clearly define the content information and paralinguistic information used for speech recognition, so that the model can clearly remove the paralinguistic information from the speech information and use the extracted content information for speech recognition. Summary of the Invention

[0007] The present invention provides a speech recognition model training, a speech recognition method and a device, which are used to solve the defect in the prior art that content information cannot be accurately extracted from speech information.

[0008] In a first aspect, the present invention provides a speech recognition model training method, comprising:

[0009] Input any speech sample into the speech recognition model to be trained;

[0010] Obtaining a total loss in a process in which the speech recognition model recognizes a speech sample, and training the speech recognition model with the goal of minimizing the total loss;

[0011] The total loss is determined based on at least the first loss, the second loss and the third loss;

[0012] The speech recognition model includes a speech encoder, an information decoupling module and a speech decoder, wherein the information decoupling module includes a speech information encoding unit, a speech information reconstruction unit, a first gradient reversal unit, a second gradient reversal unit, a content information reconstruction unit and a paralanguage information reconstruction unit;

[0013] The speech encoder is used to extract features from the input speech sample and input the extracted first speech information into the information decoupling module;

[0014] The speech information encoding unit is configured to encode the first speech information into first content information and first paralanguage information; the first content information is inversely propagated through the first gradient inversion unit to the paralanguage information reconstruction unit to generate second paralanguage information; the first paralanguage information is inversely propagated through the second gradient inversion unit to the content information reconstruction unit to generate second content information; a loss incurred by the paralanguage information reconstruction unit when generating the second paralanguage information is obtained as the first loss, and a loss incurred by the content information reconstruction unit when generating the second content information is obtained as the second loss;

[0015] A loss when the voice information reconstruction unit reconstructs the first content information and the first paralanguage information into second voice information is obtained as the third loss.

[0016] According to a speech recognition model training method provided by the present invention, there are multiple speech encoders and information decoupling modules, and a content information extraction layer is formed by one speech encoder and one information decoupling module;

[0017] The first layer of the plurality of content information extraction layers connected in series is connected as an input layer, and the last layer is connected as an output layer;

[0018] The speech encoder of the input layer receives the first speech information as input;

[0019] The first content information output by the voice information encoding unit of the output layer is input to the voice decoder.

[0020] According to a speech recognition model training method provided by the present invention, the total loss is comprehensively determined based on the first loss, the second loss and the third loss of all the content information extraction layers.

[0021] According to a speech recognition model training method provided by the present invention, the total loss also includes a fourth loss;

[0022] The fourth loss is determined by the following steps:

[0023] Using a speech encoder to perform feature extraction on any two independent speech samples to obtain third speech information and fourth speech information;

[0024] using the voice information encoding unit to encode the third voice information and the fourth voice information respectively, to obtain third content information and third paralanguage information related to the third voice information, and fourth content information and fourth paralanguage information related to the fourth voice information;

[0025] After reconstructing the third content information and the fourth paralanguage information into fifth voice information using the voice information reconstruction unit, encoding the fifth voice information using the voice information encoding unit to obtain fifth content information and fifth paralanguage information;

[0026] The fourth loss is determined with the goal of minimizing the gap between the fifth content information and the third content information, maximizing the gap between the fifth paralanguage information and the third paralanguage information, and minimizing the gap between the fifth paralanguage information and the fourth paralanguage information.

[0027] According to a speech recognition model training method provided by the present invention, the total loss also includes a fifth loss and a sixth loss;

[0028] The fifth loss is determined based on speech recognition task constraints;

[0029] The sixth loss is determined based on at least one of a speaker recognition task constraint, an emotion recognition task constraint, and an accent recognition task constraint.

[0030] According to a speech recognition model training method provided by the present invention, the speech decoder is used to decode the first content information output by the speech information encoding unit and output a text recognition result corresponding to the speech sample.

[0031] In a second aspect, the present invention further provides a speech recognition method, comprising:

[0032] receiving speech to be recognized;

[0033] Inputting the speech into a speech recognition model to obtain a recognition text output by the speech recognition model;

[0034] The speech recognition model is obtained by training an initial speech recognition model based on the speech recognition model training method described in any one of the first aspects.

[0035] In a third aspect, the present invention further provides a speech recognition model training device, comprising:

[0036] A sample input unit, used to input any speech sample into the speech recognition model to be trained;

[0037] A training control unit, configured to obtain a total loss in a process in which the speech recognition model recognizes the speech sample, and train the speech recognition model with the goal of minimizing the total loss;

[0038] The total loss is determined based on at least the first loss, the second loss and the third loss;

[0039] The speech recognition model includes a speech encoder, an information decoupling module and a speech decoder, wherein the information decoupling module includes a speech information encoding unit, a speech information reconstruction unit, a first gradient reversal unit, a second gradient reversal unit, a content information reconstruction unit and a paralanguage information reconstruction unit;

[0040] The speech encoder is used to extract features from the input speech sample and input the extracted first speech information into the information decoupling module;

[0041] The speech information encoding unit is configured to encode the first speech information into first content information and first paralanguage information; the first content information is inversely propagated through the first gradient inversion unit to the paralanguage information reconstruction unit to generate second paralanguage information; the first paralanguage information is inversely propagated through the second gradient inversion unit to the content information reconstruction unit to generate second content information; a loss incurred by the paralanguage information reconstruction unit when generating the second paralanguage information is obtained as the first loss, and a loss incurred by the content information reconstruction unit when generating the second content information is obtained as the second loss;

[0042] A loss when the voice information reconstruction unit reconstructs the first content information and the first paralanguage information into second voice information is obtained as the third loss.

[0043] In a fourth aspect, the present invention further provides a speech recognition device, comprising:

[0044] A speech receiving unit, configured to receive speech to be recognized;

[0045] a speech recognition unit, configured to input the speech into a speech recognition model and obtain a recognition text output by the speech recognition model;

[0046] The speech recognition model is obtained by training an initial speech recognition model based on the speech recognition model training method described in any one of the first aspects.

[0047] In a fifth aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the speech recognition model training method or speech recognition method as described above is implemented.

[0048] In a sixth aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the speech recognition model training methods or speech recognition methods described above.

[0049] In a seventh aspect, the present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described speech recognition model training methods or speech recognition methods.

[0050] The speech recognition model training, speech recognition method, and apparatus provided by the present invention combine the task of identifying content information with the task of removing paralinguistic information to form an end-to-end method. During the training process, content information and paralinguistic information are clearly defined, allowing the model to clearly remove paralinguistic information, thereby improving the robustness of the speech recognition system. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 Schematic diagram of the flow of the speech recognition model training method provided by the present invention;

[0053] Figure 2 It is a structural diagram of the information decoupling module provided by the present invention;

[0054] Figure 3 It is a structural diagram of the speech recognition model provided by the present invention;

[0055] Figure 4 1 is a flow chart of the speech recognition method provided by the present invention;

[0056] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0057] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0058] It should be noted that, in the description of the present invention, the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, the phrase "comprises a..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising the elements. Terms such as "upper" and "lower" indicate positions or relationships based on those shown in the accompanying drawings and are intended solely to facilitate the description of the present invention and simplify the description. They are not intended to indicate or imply that the devices or elements referred to must have a specific orientation, be constructed, or operate in a specific orientation, and are therefore not to be construed as limitations on the present invention. Unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be broadly construed, for example, to mean fixed, removable, or integral; mechanical or electrical; direct or indirect through an intermediary; or internal communication between two elements. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0059] The terms "first," "second," and so forth, used herein are used to distinguish similar objects, not to describe a specific order or precedence. It should be understood that such terms are interchangeable where appropriate, allowing embodiments of the present invention to be implemented in an order other than that illustrated or described herein. Furthermore, the terms "first," "second," and so forth generally distinguish objects of a single type, and do not limit the number of objects. For example, the first object may be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates an "or" relationship between the connected objects.

[0060] The following combination Figure 1-Figure 5 The speech recognition model training, speech recognition method and device provided by the present invention are described.

[0061] Existing technologies often fail to effectively separate content information closely related to speech recognition from negative language information that negatively impacts speech recognition. This results in the extracted content information still containing some negative language information, which can affect the speech recognition results to a certain extent. The present invention provides a speech recognition model training method by focusing on model training of the speech recognition model, which can fundamentally address this technical shortcoming.

[0062] In summary, the speech recognition model training method provided by this invention is an end-to-end training method that completes the training of the Automatic Speech Recognition (ASR) task during the entire training process while simultaneously removing paralinguistic information from the encoder's encoding features. This is primarily accomplished by using an information decoupling network during the encoder encoding process.

[0063] The speech recognition model training method provided by the present invention is described in detail below with reference to specific embodiments.

[0064] Figure 1 It is a flow chart of the speech recognition model training method provided by the present invention, such as Figure 1 As shown, including but not limited to the following steps:

[0065] Step 101: Input any speech sample into the speech recognition model to be trained.

[0066] Step 102: Obtain the total loss of the speech recognition model in the process of recognizing the speech sample, and train the speech recognition model with the goal of minimizing the total loss.

[0067] The total loss is determined based on at least the first loss, the second loss and the third loss.

[0068] The speech recognition model used in the present invention mainly includes: a speech encoder, an information decoupling module and a speech decoder.

[0069] Figure 2 This is a schematic diagram of the structure of the information decoupling module provided by the present invention. Figure 2 As shown in the dotted box, the information decoupling module mainly includes a voice information encoding unit F and a voice information reconstruction unit. The first gradient reversal unit (Gradient Reversal Unit, GRL1), the second gradient reversal unit GRL2, the content information reconstruction unit r τ and paralinguistic information reconstruction unit r σ .

[0070] Among them, the speech encoder of the speech recognition model is mainly used to extract features from each input speech sample and input the extracted first speech information into the information decoupling module.

[0071] The speech sample is a speech signal, which is represented by a continuous time series, and the speech encoder provided by the present application can convert it into a fixed-length feature representation. This can be achieved by dividing the speech signal into short time frames (such as 20-30 milliseconds) and applying a preset feature extraction method to each frame. The feature extraction method mainly includes one or more of Mel-frequency Cepstral Coefficients (MFCC), filter bank features (Filter-BankFeatures), etc. These feature extraction methods can capture the spectral information and other relevant features of the speech signal.

[0072] Of course, in order to capture the contextual information in the speech sample, a certain context window can be introduced on the frame-level features. This can be achieved by sliding the window on the time axis and summarizing the features within the window (such as averaging or splicing). This can capture the temporal relationship between the frame-level features and their surrounding frames.

[0073] After obtaining the feature representation of each speech sample, these features are represented using a vector matrix, such as a two-dimensional vector matrix, where each row represents a feature vector for a time step.

[0074] Furthermore, the above feature representation can be converted into a higher-level representation, such as using convolutional neural network (CNN), recurrent neural network (RNN) or self-attention mechanism (Self-Attention), multi-layer feedforward neural network (Feedforward Neural Network, FNN) and other methods for feature extraction and encoding, so as to obtain the first speech information corresponding to the speech sample.

[0075] like Figure 2 As shown, the voice information encoding unit F is mainly used to encode the first voice information into first content information, denoted as τ, and first paralanguage information, denoted as σ.

[0076] The first content information τ is reversely propagated to the paralinguistic information reconstruction unit r through the first gradient reversal unit GRL1. σ , generate second language information The first paralinguistic information σ is reversely propagated to the content information reconstruction unit r through the second gradient reversal unit GRL2 τ , generating the second content information

[0077] Further, obtain the paralinguistic information reconstruction unit r σ Generate the second secondary language information The loss at this time is regarded as the first loss, denoted as Lossσ , and obtain the content information reconstruction unit r τ Generate the second content information The loss at this time is the second loss, recorded as Loss τ ;

[0078] In addition, it is also necessary to obtain the voice information reconstruction unit Reconstruct the first content information τ and the first paralinguistic information σ into second voice information The loss at that time shall be regarded as the third loss.

[0079] It should be noted that in addition to the first gradient reversal unit GRL1 and the second gradient reversal unit GRL2, the information decoupling module provided by the present invention includes a speech encoder, a speech decoder, etc., and the others can be implemented by learning a network and using a multi-layer feedforward neural network (FNN). The forward propagation of the first gradient reversal unit GRL1 and the second gradient reversal unit GRL2 is the identity function, and the back propagation reverses the gradient.

[0080] The core improvement of the speech recognition model training method provided by the present invention lies in extracting content information τ related only to the speech recognition task and irrelevant paralinguistic information σ from the speech information X related to the input speech sample through the speech information encoding unit F. Furthermore, the model needs to be trained so that when the trained model processes the speech information X, the information between the paralinguistic information σ encoded in the speech information X and the content information τ is as independent as possible, that is, the mutual information between the two is as small as possible.

[0081] In order to minimize the mutual information between the paralinguistic information σ and the content information τ, in the process of training the speech recognition model, the first content information τ is reversely propagated by the first gradient reversal unit GRL1, and then the paralinguistic information reconstruction unit r is used to reconstruct the content information τ. σ Reconstruct the contained paralinguistic information to obtain the second paralinguistic information At the same time, the second gradient reversal unit GRL2 reverses the propagation of the first paralinguistic information σ and uses the content information to reconstruct the unit r τ Reconstruct the included content information to obtain the second content information

[0082] In this way, the paralinguistic information is constrained by the loss function and the unit r is reconstructed. σ Reconstructing second language information The loss is used as the first loss, and the content information is reconstructed into unit r τ Generate the second content information The loss at that time is regarded as the second loss.

[0083] As an optional embodiment, the above-mentioned loss function can adopt the mean squared error function (MSE), or the cross-entropy loss function (Cross-Entropy Loss), the mean absolute error loss function (MAE) or the Huber loss function (Huber Loss), etc., which is not specifically limited in the present invention.

[0084] The first loss and the second loss ensure that the content information reconstruction unit r τ and paralinguistic information reconstruction unit r σ The second paralanguage information can be restored as much as possible from the first paralanguage information σ and the first content information τ and second content information

[0085] Due to the presence of the first and second gradient reversal units GRL1 and GRL2, the speech information encoding unit F attempts to maximize the first and second losses during prior training of the speech recognition model. This makes it difficult to recover the content information and paralinguistic information from the first paralinguistic information σ and first content information τ encoded by the speech information encoding unit F. Through this adversarial training, the first paralinguistic information σ and first content information τ encoded by the speech information encoding unit F are made as unrelated as possible, minimizing the mutual information between them.

[0086] Mutual information measures the correlation between two random variables. Minimizing the mutual information between the first paralinguistic information σ and the first content information τ aims to minimize the mutual information between the first paralinguistic information σ and the first content information τ. By minimizing the mutual information, the content information and paralinguistic information are made as unrelated as possible in the encoded representation, thus achieving separation of the content information and paralinguistic information.

[0087] Taking the mean square error loss function as an example, in order to achieve the goal of minimizing mutual information, the mean square error loss function is used to constrain the content information reconstruction unit. r τ and paralinguistic information reconstruction unit r σ Output.

[0088] Specifically, by comparing the content information reconstruction unit r τ, paralinguistic information reconstruction unit r σ The reconstruction result of the first content information τ and the first paralinguistic information σ, namely the second paralinguistic information and second content information The difference between them and their true values ​​is used to calculate the first loss Loss τ , Second Loss σ , clearly distinguishing the content information used for speech recognition from the irrelevant paralinguistic information, so that the model can clearly remove the paralinguistic information, and thus effectively guide the training process of the speech recognition model.

[0089] Loss τ =MSEE(r τ (GRL(σ)),τ);

[0090] Loss σ =MSE(r σ (GRL(τ)),σ).

[0091] Among them, r τ (GRL(σ)) refers to the second content information r σ (GRL(τ)) refers to the second paralinguistic information

[0092] Due to the presence of the first gradient reversal unit GRL1 and the second gradient reversal unit GRL2, the speech information encoding unit F hopes to maximize the first loss Loss τ , Second Loss σ , which means that the speech information encoding unit F hopes to improve the content information reconstruction unit r τ , paralinguistic information reconstruction unit r σ , respectively, the reconstruction capability of the first content information τ and the first paralinguistic information σ, so that the second paralinguistic information encoded by the speech information encoding unit F Second content information It is more difficult to restore the original first content information τ and first paralinguistic information σ.

[0093] Through adversarial training, specifically minimizing the reconstruction loss, the speech information encoding unit F learns an encoding method that makes it more difficult to recover the original content and paralanguage information from the first paralanguage information σ and the first content information τ encoded by the speech information encoding unit F. In other words, the encoded representation learned by the speech information encoding unit F does not contain specific information about the content and paralanguage, thus achieving information separation.

[0094] During speech recognition model training, the present invention uses a loss function to impose constraints and employs adversarial training to minimize the unrelatedness between content information and paralinguistic information in the encoded representation, thereby achieving the goal of minimizing mutual information. This operation can help the speech information encoding unit F learn a more robust and effective representation, thereby improving the performance of the speech recognition model.

[0095] Furthermore, in order to ensure that the first paralinguistic information σ and the first content information τ decoupled by the voice information encoding unit F do not lose the voice feature information of the original input first voice information X, the present invention uses the voice information reconstruction unit The first paralinguistic information σ and the first content information τ are restored to the input features, that is, reconstructed into the second speech information Then, the second speech information is analyzed by the loss function The third loss Loss between the first voice information X rec To constrain:

[0096]

[0097] in, Speech Information Reconstruction Unit Output second voice information

[0098] In summary, the present invention provides an end-to-end speech recognition model training method, which uses multiple speech samples to iteratively train the speech recognition model to be trained. During each iterative training process, by calculating the above-mentioned first loss, second loss and third loss, and minimizing the total loss of the above-mentioned three losses as the goal, the relevant parameters in the speech recognition model (such as the relevant parameters of the speech encoder, information decoupling module and speech decoder) are corrected until the training results converge and the total loss is reduced to a preset threshold.

[0099] In short, the first and second gradient reversal units GRL1 and GRL2 work together with the total loss function to minimize the amount of paralinguistic information contained in the decoupled content. The parameters after the two GRL layers aim to minimize the loss, while the parameters before the GRL layers aim to maximize the loss, thus minimizing the amount of paralinguistic information contained in the decoupled content.

[0100] The speech recognition model training method provided by the present invention combines the task of identifying content information with the task of removing paralinguistic information to form an end-to-end method. During the training process, content information and paralinguistic information are clearly defined, which facilitates the model to clearly remove paralinguistic information, thereby improving the robustness of the speech recognition system.

[0101] Based on the content of the above embodiment, as an optional embodiment, the speech recognition model provided by the present invention includes multiple speech encoders and information decoupling modules, and a content information extraction layer is formed by one speech encoder and one information decoupling module;

[0102] The first layer of the plurality of content information extraction layers connected in series is connected as an input layer, and the last layer is connected as an output layer;

[0103] The speech encoder of the input layer receives the first speech information as input;

[0104] The first content information output by the voice information encoding unit of the output layer is input to the voice decoder.

[0105] Figure 3 This is a schematic diagram of the structure of the speech recognition model provided by the present invention. Figure 3 As shown in the example, in the speech recognition model provided by the present invention, the speech encoder (Speech Encoder) uses a conformer structural block to obtain a representation of speech information.

[0106] The present invention adopts this setting to perform information decoupling at different depths of the speech coding model, further enhancing the ability to separate paralanguage information.

[0107] Optionally, in the present invention, a 16-layer conformer structure block is used, and the i-th conformer structure block can be represented as Block i Each layer of the Conformer structure is a model structure based on the self-attention mechanism and convolutional neural network, which can effectively learn the time and frequency domain properties of the speech signal. Its input is the original speech signal, such as a sound waveform represented in the form of a time domain waveform. Then, through a series of processing steps, the speech signal is converted into a fixed-dimensional feature vector or feature matrix. This vector or matrix is ​​the representation of the first speech information X.

[0108] Among them, the speech encoder usually includes a series of processing steps, including: preprocessing, feature extraction (converting the input speech signal into a feature representation vector or matrix. Common feature extraction methods include Mel-frequency cepstral coefficients (MFCC), linear prediction coefficients (LPC), filter bank features (filterbank)), etc.), frame-level feature processing (processing the features of each speech frame, such as performing differentiation in the time domain to extract the dynamic information of the speech signal), sequence-level feature processing (processing the features of the entire speech signal sequence, such as performing convolution, pooling, self-attention mechanism and other operations in the time dimension to extract richer speech features), dimensionality reduction (reducing the dimension of the feature vector or matrix to reduce the amount of calculation and improve the generalization ability of the model), normalization (standardizing the feature vector or matrix to eliminate the scaling factor between different features and improve the robustness of the model), etc.

[0109] This case proposes an end-to-end speech recognition model with a 16-layer conformer structure, that is, a total of 16 blocks i The speech encoder is connected in series, in each block i Add an information decoupling module to the output end of i In the actual speech recognition process, disentanglement i The first encoded content information τ is used as the i+1th information decoupling module, namely disentanglement i+1 input.

[0110] In this way, the data flow is consistent during the training and actual use of the speech recognition model, and the first conformer block before the last conformer block (i.e. Block 15 ) output as the acoustic features for speech recognition. In the speech recognition model provided by the present invention, the acoustic features ultimately used for speech recognition undergo layer-by-layer decoupling, i.e., information decoupling at varying depths. Compared to existing technologies, this significantly attenuates paralinguistic information, enabling better performance in downstream speech recognition tasks.

[0111] Based on the contents of the above embodiment, as an optional embodiment, during the training of a speech recognition model, the present invention can first determine the first loss and second loss of each layer of the conformer structure block when separating content information from paralinguistic information, and simultaneously calculate the third loss of the speech information reconstruction unit in each conformer structure block when reconstructing the first content information and the first paralinguistic information into the second speech information. Then, the losses of all conformer structure blocks are accumulated as the total loss of the entire speech recognition model. In this way, an end-to-end training method can be implemented, effectively improving the efficiency and robustness of training.

[0112] Based on the content of the above embodiment, as an optional embodiment, the speech recognition model training method provided by the present invention can further focus on the fourth loss during the actual training process.

[0113] The fourth loss is determined by the following steps:

[0114] Using a speech encoder to perform feature extraction on any two independent speech samples to obtain third speech information and fourth speech information;

[0115] using the voice information encoding unit to encode the third voice information and the fourth voice information respectively, to obtain third content information and third paralanguage information related to the third voice information, and fourth content information and fourth paralanguage information related to the fourth voice information;

[0116] After reconstructing the third content information and the fourth paralanguage information into fifth voice information using the voice information reconstruction unit, encoding the fifth voice information using the voice information encoding unit to obtain fifth content information and fifth paralanguage information;

[0117] The fourth loss is determined with the goal of minimizing the gap between the fifth content information and the third content information, maximizing the gap between the fifth paralanguage information and the third paralanguage information, and minimizing the gap between the fifth paralanguage information and the fourth paralanguage information.

[0118] Specifically, two independent voice samples are taken and feature extraction is performed on each of them to obtain the third voice information X and the fourth voice information X'.

[0119] The third voice information X and the fourth voice information X' are encoded by the information encoding module F, thereby obtaining third content information and third paralanguage information related to the third voice information X, denoted as (τ, σ), and fourth content information and fourth paralanguage information related to the fourth voice information, denoted as (τ', σ').

[0120] Furthermore, the unit is reconstructed through voice information Reconstruct the third content information τ and the fourth paralanguage information σ′ to obtain the fifth voice information Then the voice information encoding unit F Encoding to obtain the fifth content information and the fifth paralinguistic information In order to make the specific content information in any paralinguistic information scenario, the voice information encoding unit F can encode the independent and accurate content information, the fifth content information τ and the third content information are required. The distance between them needs to be close enough. In order to make the paralinguistic information extracted from different samples different, the paralinguistic information between similar samples should be as close as possible. and the third paralinguistic information σ as far as possible, the fifth paralinguistic information σ ～ As close as possible to the fourth paralinguistic information σ′.

[0121] From this, the fourth loss can be defined, and the expression of its loss function is:

[0122]

[0123] In this way, when training the model, when considering the total loss, in combination with the first loss Loss τ , Second Loss σ and the third loss rec Based on the integration of the fourth loss Loss cyc .

[0124] As an optional embodiment, the speech recognition model training method provided by the present invention also considers the fifth loss and the sixth loss when considering the loss constraints during the training process.

[0125] The fifth loss is determined based on the speech recognition task constraint in order to ensure that the content information and the paralinguistic information have their own independent distribution spaces; the sixth loss is determined based on at least one of the speaker recognition task constraint, the emotion recognition task constraint, and the accent recognition task constraint.

[0126] In summary, the loss of any information decoupling module is recorded as Loss disent , then the calculation formula is:

[0127] LosSdisentLossτ+LosS. +LosSrec+LosScyc+LLosSasr+LosSpara

[0128] The total loss of the entire speech recognition model is:

[0129]

[0130] Where k is the number of information decoupling modules.

[0131] It should be noted that Figure 3 Taking the example shown in the figure, when a multi-layer conformer structure block is used, the speech recognition loss and paralanguage-related task loss output by each information decoupling module need to be added together to form part of the loss of the entire model training.

[0132] This layer-by-layer loss calculation method makes it easier to decouple content information and paralanguage information from shallow features that contain more paralanguage information, so that the interference of paralanguage information on content information gradually decreases as the number of layers increases, further enhancing the ability to separate paralanguage information.

[0133] Based on the content of the above embodiment, as an optional embodiment, the speech decoder is used to decode the first content information output by the speech information encoding unit, and output a text recognition result corresponding to the speech sample.

[0134] It is particularly important to note that when the speech recognition model has multiple layers of conformer structural blocks, the data flow method is consistent during its training and actual use, and the first content information τ output by the conformer structural block before the last conformer structural block is used as the acoustic feature of speech recognition.

[0135] In the speech recognition model provided by the present invention, the acoustic features used for final speech recognition have undergone layer-by-layer decoupling, that is, information decoupling at different depths. Compared with the existing technology, this significantly attenuates the paralinguistic information therein, which can achieve better results in downstream speech recognition tasks.

[0136] It should be noted that the structure of the information decoupling module in the present invention is mainly used to distinguish the downstream tasks of content information and paralinguistic information. For the downstream tasks of content information, the present invention uses speech recognition tasks, that is, it is implemented by using a speech decoder, including but not limited to commonly used Seq2Seq, CTC, RNN-T and other network models. That is, the present invention decodes the content information extracted by the information decoupling module through the speech decoder and outputs the text recognition result of the speech.

[0137] Figure 4 It is a flow chart of the speech recognition method provided by the present invention, such as Figure 4 As shown, the present invention further provides a speech recognition method based on the speech recognition model training method provided in any of the above embodiments, which mainly includes but is not limited to the following steps:

[0138] Step 401: receiving speech to be recognized;

[0139] Step 402: Input the speech into a speech recognition model to obtain a recognition text output by the speech recognition model.

[0140] The above-mentioned speech recognition model is obtained by training the initial speech recognition model based on the speech recognition model training method provided in any embodiment.

[0141] The speech recognition model mainly includes a speech encoder, an information decoupling module and a speech decoder. The information decoupling module includes a speech information encoding unit, a speech information reconstruction unit, a first gradient reversal unit, a second gradient reversal unit, a content information reconstruction unit and a paralanguage information reconstruction unit.

[0142] Among them, the speech encoder is mainly used to extract features from the input speech sample and input the extracted first speech information into the information decoupling module.

[0143] The speech information encoding unit is mainly used to encode the first speech information into first content information and first paralanguage information; the first content information is inversely propagated to the paralanguage information reconstruction unit through the first gradient reversal unit to generate second paralanguage information; the first paralanguage information is inversely propagated to the content information reconstruction unit through the second gradient reversal unit to generate second content information.

[0144] The speech decoder is used to decode the first content information output by the speech information encoding unit and output a text recognition result corresponding to the speech sample.

[0145] Furthermore, there may be multiple speech encoders and information decoupling modules, and a content information extraction layer is formed by a speech encoder and one of the information decoupling modules;

[0146] The first layer of the plurality of content information extraction layers connected in series is connected as an input layer, and the last layer is connected as an output layer;

[0147] The speech encoder of the input layer receives the first speech information as input;

[0148] The first content information output by the voice information encoding unit of the output layer is input to the voice decoder.

[0149] The speech recognition method provided by the present invention combines the task of identifying content information with the task of removing paralinguistic information to form an end-to-end method. During the training process, content information and paralinguistic information are clearly defined, which facilitates the model to clearly remove paralinguistic information, thereby improving the robustness of the speech recognition system.

[0150] The present invention also provides a speech recognition model training device, which mainly includes but is not limited to:

[0151] The sample input unit is mainly used to input any speech sample into the speech recognition model to be trained;

[0152] The training control unit is mainly used to obtain the total loss of the speech recognition model in the process of recognizing the speech sample, and train the speech recognition model with the goal of minimizing the total loss.

[0153] The above-mentioned total loss is determined based on at least the first loss, the second loss and the third loss.

[0154] The speech recognition model includes a speech encoder, an information decoupling module and a speech decoder. The information decoupling module includes a speech information encoding unit, a speech information reconstruction unit, a first gradient reversal unit, a second gradient reversal unit, a content information reconstruction unit and a paralanguage information reconstruction unit.

[0155] The speech encoder is used to extract features from the input speech sample and input the extracted first speech information into the information decoupling module.

[0156] The speech information encoding unit is configured to encode the first speech information into first content information and first paralanguage information; the first content information is inversely propagated through the first gradient reversal unit to the paralanguage information reconstruction unit to generate second paralanguage information; the first paralanguage information is inversely propagated through the second gradient reversal unit to the content information reconstruction unit to generate second content information; a loss incurred by the paralanguage information reconstruction unit when generating the second paralanguage information is obtained as the first loss, and a loss incurred by the content information reconstruction unit when generating the second content information is obtained as the second loss.

[0157] A loss when the voice information reconstruction unit reconstructs the first content information and the first paralanguage information into second voice information is obtained as the third loss.

[0158] It should be noted that the speech recognition model training device provided by the present invention can execute the speech recognition model training method described in any of the above embodiments during specific operation, which will not be described in detail in this embodiment.

[0159] In addition, the present invention also provides a speech recognition device, which mainly includes but is not limited to:

[0160] The speech receiving unit is mainly used to receive the speech to be recognized;

[0161] The speech recognition unit is mainly used to input the speech into the speech recognition model and obtain the recognition text output by the speech recognition model.

[0162] The speech recognition model is obtained by training an initial speech recognition model based on the speech recognition model training method described in any of the above embodiments.

[0163] The speech recognition model training device and speech recognition device provided by the present invention combine the task of recognizing content information with the task of removing paralinguistic information to form an end-to-end method. During the training process, content information and paralinguistic information are clearly defined, which facilitates the model to clearly remove paralinguistic information, thereby improving the robustness of the speech recognition system.

[0164] Figure 5 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute a speech recognition model training method or a speech recognition method, which includes: inputting any speech sample into the speech recognition model to be trained; obtaining the total loss of the speech recognition model in the process of recognizing the speech sample, and training the speech recognition model with the goal of minimizing the total loss.

[0165] The total loss is determined based on at least a first loss, a second loss, and a third loss. The speech recognition model includes a speech encoder, an information decoupling module, and a speech decoder. The information decoupling module includes a speech information encoding unit, a speech information reconstruction unit, a first gradient reversal unit, a second gradient reversal unit, a content information reconstruction unit, and a paralinguistic information reconstruction unit. The speech encoder is configured to extract features from the input speech sample and input the extracted first speech information into the information decoupling module. The speech information encoding unit is configured to encode the first speech information into first content information and first paralinguistic information. The first content information is inversely propagated to the paralinguistic information reconstruction unit via the first gradient reversal unit to generate second paralinguistic information. The first paralinguistic information is inversely propagated to the content information reconstruction unit via the second gradient reversal unit to generate second content information. The loss of the paralinguistic information reconstruction unit when generating the second paralinguistic information is obtained as the first loss, and the loss of the content information reconstruction unit when generating the second content information is obtained as the second loss. The loss of the speech information reconstruction unit when reconstructing the first content information and the first paralinguistic information into the second speech information is obtained as the third loss.

[0166] The speech recognition method includes: receiving speech to be recognized; inputting the speech into a speech recognition model, and obtaining recognition text output by the speech recognition model; the speech recognition model is obtained by training an initial speech recognition model based on the above-mentioned speech recognition model training method.

[0167] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0168] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute a speech recognition model training method or a speech recognition method. The speech recognition model training method includes: inputting any speech sample into the speech recognition model to be trained; obtaining the total loss in the process of the speech recognition model recognizing the speech sample, and training the speech recognition model with the goal of minimizing the total loss.

[0169] The total loss is determined based on at least a first loss, a second loss, and a third loss. The speech recognition model includes a speech encoder, an information decoupling module, and a speech decoder. The information decoupling module includes a speech information encoding unit, a speech information reconstruction unit, a first gradient reversal unit, a second gradient reversal unit, a content information reconstruction unit, and a paralinguistic information reconstruction unit. The speech encoder is configured to extract features from the input speech sample and input the extracted first speech information into the information decoupling module. The speech information encoding unit is configured to encode the first speech information into first content information and first paralinguistic information. The first content information is inversely propagated to the paralinguistic information reconstruction unit via the first gradient reversal unit to generate second paralinguistic information. The first paralinguistic information is inversely propagated to the content information reconstruction unit via the second gradient reversal unit to generate second content information. The loss of the paralinguistic information reconstruction unit when generating the second paralinguistic information is obtained as the first loss, and the loss of the content information reconstruction unit when generating the second content information is obtained as the second loss. The loss of the speech information reconstruction unit when reconstructing the first content information and the first paralinguistic information into the second speech information is obtained as the third loss.

[0170] The speech recognition method includes: receiving speech to be recognized; inputting the speech into a speech recognition model, and obtaining recognition text output by the speech recognition model; the speech recognition model is obtained by training an initial speech recognition model based on the above-mentioned speech recognition model training method.

[0171] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to perform a speech recognition model training method or a speech recognition method. The speech recognition model training method includes: inputting any speech sample into the speech recognition model to be trained; obtaining the total loss of the speech recognition model in the process of recognizing the speech sample, and training the speech recognition model with the goal of minimizing the total loss.

[0172] The total loss is determined based on at least a first loss, a second loss, and a third loss. The speech recognition model includes a speech encoder, an information decoupling module, and a speech decoder. The information decoupling module includes a speech information encoding unit, a speech information reconstruction unit, a first gradient reversal unit, a second gradient reversal unit, a content information reconstruction unit, and a paralinguistic information reconstruction unit. The speech encoder is configured to extract features from the input speech sample and input the extracted first speech information into the information decoupling module. The speech information encoding unit is configured to encode the first speech information into first content information and first paralinguistic information. The first content information is inversely propagated to the paralinguistic information reconstruction unit via the first gradient reversal unit to generate second paralinguistic information. The first paralinguistic information is inversely propagated to the content information reconstruction unit via the second gradient reversal unit to generate second content information. The loss of the paralinguistic information reconstruction unit when generating the second paralinguistic information is obtained as the first loss, and the loss of the content information reconstruction unit when generating the second content information is obtained as the second loss. The loss of the speech information reconstruction unit when reconstructing the first content information and the first paralinguistic information into the second speech information is obtained as the third loss.

[0173] The speech recognition method includes: receiving speech to be recognized; inputting the speech into a speech recognition model, and obtaining recognition text output by the speech recognition model; the speech recognition model is obtained by training an initial speech recognition model based on the above-mentioned speech recognition model training method.

[0174] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0175] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A speech recognition model training method, characterized in that: include: Input any speech sample into the speech recognition model to be trained; Obtaining a total loss in a process in which the speech recognition model recognizes a speech sample, and training the speech recognition model with the goal of minimizing the total loss; The total loss is determined based on at least the first loss, the second loss and the third loss; The speech recognition model includes a speech encoder, an information decoupling module and a speech decoder, wherein the information decoupling module includes a speech information encoding unit, a speech information reconstruction unit, a first gradient reversal unit, a second gradient reversal unit, a content information reconstruction unit and a paralanguage information reconstruction unit; The speech encoder is used to extract features from the input speech sample and input the extracted first speech information into the information decoupling module; The voice information encoding unit is configured to encode the first voice information into first content information and first paralanguage information; The first content information is inversely propagated through the first gradient inversion unit to the paralanguage information reconstruction unit to generate second paralanguage information; The first paralanguage information is inverted and propagated to the content information reconstruction unit through the second gradient reversal unit to generate second content information; a loss when the paralanguage information reconstruction unit generates the second paralanguage information is obtained as the first loss, and a loss when the content information reconstruction unit generates the second content information is obtained as the second loss; A loss when the voice information reconstruction unit reconstructs the first content information and the first paralanguage information into second voice information is obtained as the third loss.

2. The speech recognition model training method according to claim 1, characterized in that There are multiple speech encoders and information decoupling modules, and one speech encoder and one information decoupling module constitute a content information extraction layer; The first layer of the plurality of content information extraction layers connected in series is connected as an input layer, and the last layer is connected as an output layer; The speech encoder of the input layer receives the first speech information as input; The first content information output by the voice information encoding unit of the output layer is input to the voice decoder.

3. The speech recognition model training method according to claim 2, characterized in that: The total loss is determined based on the first loss, the second loss, and the third loss of all the content information extraction layers.

4. The speech recognition model training method according to any one of claims 1 to 3, characterized in that: Said total loss also includes the fourth loss; The fourth loss is determined by the following steps: Using a speech encoder to perform feature extraction on any two independent speech samples to obtain third speech information and fourth speech information; using the voice information encoding unit to encode the third voice information and the fourth voice information respectively, to obtain third content information and third paralanguage information related to the third voice information, and fourth content information and fourth paralanguage information related to the fourth voice information; After reconstructing the third content information and the fourth paralanguage information into fifth voice information using the voice information reconstruction unit, encoding the fifth voice information using the voice information encoding unit to obtain fifth content information and fifth paralanguage information; The fourth loss is determined with the goal of minimizing the gap between the fifth content information and the third content information, maximizing the gap between the fifth paralanguage information and the third paralanguage information, and minimizing the gap between the fifth paralanguage information and the fourth paralanguage information.

5. The speech recognition model training method according to claim 4, characterized in that: Said total loss also includes the fifth loss and the sixth loss; The fifth loss is determined based on speech recognition task constraints; The sixth loss is determined based on at least one of a speaker recognition task constraint, an emotion recognition task constraint, and an accent recognition task constraint.

6. The speech recognition model training method according to claim 1, characterized in that: The speech decoder is used to decode the first content information output by the speech information encoding unit, and output a text recognition result corresponding to the speech sample.

7. A speech recognition method, characterized in that: include: receiving speech to be recognized; Inputting the speech into a speech recognition model to obtain a recognition text output by the speech recognition model; The speech recognition model is obtained by training an initial speech recognition model based on the speech recognition model training method according to any one of claims 1 to 6.

8. A speech recognition model training device, characterized in that: include: A sample input unit, used to input any speech sample into the speech recognition model to be trained; A training control unit, configured to obtain a total loss in a process in which the speech recognition model recognizes the speech sample, and train the speech recognition model with the goal of minimizing the total loss; The total loss is determined based on at least the first loss, the second loss and the third loss; The speech recognition model includes a speech encoder, an information decoupling module and a speech decoder, wherein the information decoupling module includes a speech information encoding unit, a speech information reconstruction unit, a first gradient reversal unit, a second gradient reversal unit, a content information reconstruction unit and a paralanguage information reconstruction unit; The speech encoder is used to extract features from the input speech sample and input the extracted first speech information into the information decoupling module; The voice information encoding unit is configured to encode the first voice information into first content information and first paralanguage information; The first content information is inversely propagated through the first gradient inversion unit to the paralanguage information reconstruction unit to generate second paralanguage information; The first paralanguage information is inverted and propagated to the content information reconstruction unit through the second gradient reversal unit to generate second content information; a loss when the paralanguage information reconstruction unit generates the second paralanguage information is obtained as the first loss, and a loss when the content information reconstruction unit generates the second content information is obtained as the second loss; A loss when the voice information reconstruction unit reconstructs the first content information and the first paralanguage information into second voice information is obtained as the third loss.

9. A speech recognition device, characterized in that: include: A speech receiving unit, configured to receive speech to be recognized; a speech recognition unit, configured to input the speech into a speech recognition model and obtain a recognition text output by the speech recognition model; The speech recognition model is obtained by training an initial speech recognition model based on the speech recognition model training method according to any one of claims 1 to 6.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the speech recognition model training method according to any one of claims 1 to 6 or the speech recognition method according to claim 7.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the speech recognition model training method according to any one of claims 1 to 6 or the speech recognition method according to claim 7.

12. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the speech recognition model training method according to any one of claims 1 to 6 or the speech recognition method according to claim 7.

Images