A speech recognition method, apparatus, storage medium and device

Abstract

This application discloses a speech recognition method, apparatus, storage medium, and device. The method includes: firstly acquiring target speech data to be recognized; then inputting the target speech data into a pre-constructed speech recognition model to predict the speech recognition result of the target speech data. The speech recognition model is obtained by pre-training using unsupervised speech data and speaker-related information, followed by fine-tuning using supervised speech data. Therefore, because this application constructs a speech recognition model by first pre-training using unsupervised speech data and speaker-related information, and then fine-tuning using supervised speech data, it effectively improves the robustness of the model for speech recognition of different speakers. Thus, when using this speech recognition model to recognize target speech data, it can effectively improve the accuracy of the recognition result, thereby improving the recognition effect.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to a speech recognition method, apparatus, storage medium and device. Background Technology

[0002] With the continuous breakthroughs in artificial intelligence technology and the increasing popularity of various smart terminal devices, human-computer interaction is occurring more and more frequently in people's daily work and life. Voice interaction, as the next generation of human-computer interaction, can bring great convenience to people's lives, and among its key aspects is speech recognition technology.

[0003] Traditional speech recognition methods typically employ a pre-training approach, first using massive amounts of inexpensive unsupervised data to pre-train the model, and then fine-tuning the pre-trained model using a small amount of supervised speech recognition data. This addresses the problem of insufficient training data for various speech recognition scenarios. However, the recognition model trained in this way only performs well on small datasets. This is because the model training process does not consider the influence of different speaker-related information, leading to inaccuracies in speech recognition results due to speaker differences. Summary of the Invention

[0004] The main objective of this application is to provide a speech recognition method, apparatus, storage medium, and device that can effectively improve the accuracy of recognition results during speech recognition.

[0005] This application provides a speech recognition method, including:

[0006] Acquire the target speech data to be recognized;

[0007] The target speech data is input into a pre-built speech recognition model to predict the speech recognition result of the target speech data;

[0008] The speech recognition model is obtained by pre-training using unsupervised speech data and speaker-related information, and then fine-tuning using supervised speech data.

[0009] In one possible implementation, before inputting the target speech data into a pre-built speech recognition model, the method further includes:

[0010] The target speech data is cleaned to obtain preprocessed target speech data;

[0011] The step of inputting the target speech data into a pre-built speech recognition model to predict the speech recognition result of the target speech data includes:

[0012] The preprocessed target speech data is input into a pre-built speech recognition model to predict the speech recognition result of the target speech data.

[0013] In one possible implementation, the speech recognition model includes a convolutional layer, a mask prediction layer, a fully connected layer, and a decoding layer; the step of inputting the target speech data into the pre-constructed speech recognition model to predict the speech recognition result of the target speech data includes:

[0014] The target speech data is input into the convolutional layer of a pre-built speech recognition model to extract frame-level features of the target speech data;

[0015] The frame-level features of the target speech data are masked, and the processed frame-level features are input into the mask prediction layer of the speech recognition model for feature prediction to obtain the speech features of the target speech data.

[0016] The speech features of the target speech data are input into the fully connected layer of the speech recognition model to predict the frame-level speech features of the target speech data.

[0017] The frame-level speech features of the target speech data are input into the decoding layer of the speech recognition model for decoding, and the speech recognition result corresponding to the target speech data is determined based on the decoding result.

[0018] In one possible implementation, the method further includes:

[0019] Using unsupervised speech data and a preset target loss function, the initial speech recognition model is pre-trained by removing speaker-related information layer by layer, resulting in a pre-trained speech recognition model.

[0020] The pre-trained speech recognition model is fine-tuned using supervised speech data to obtain the speech recognition model.

[0021] In one possible implementation, the step of pre-training the initial speech recognition model by progressively removing speaker-related information layer by layer using unsupervised speech data and a preset target loss function to obtain a pre-trained speech recognition model includes:

[0022] Obtain unsupervised sample speech data;

[0023] Using the unsupervised sample speech data, the first objective loss function, the second objective loss function, and the third objective loss function, the initial speech recognition model is pre-trained to obtain the pre-trained speech recognition model.

[0024] The first target loss function is used to narrow the distance between the frame-level mask features extracted by the mask prediction layer in the initial speech recognition model and the corresponding frame-level quantization features extracted by the quantizer, and to widen the distance between the target loss function and other frame-level quantization features extracted by the quantizer.

[0025] The second objective loss function is used to improve the accuracy of the speaker features extracted by the initial speech recognition model;

[0026] The third objective loss function is used to constrain the weights of the frame-level features extracted by the N transform network layers contained in the mask prediction layer of the initial speech recognition model; where N is a positive integer greater than 0.

[0027] In one possible implementation, the step of pre-training the initial speech recognition model using the unsupervised sample speech data, a first objective loss function, a second objective loss function, and a third objective loss function to obtain a pre-trained speech recognition model includes:

[0028] The unsupervised sample speech data is input into a time-delay neural network to extract the speaker features of the sample speech data;

[0029] The unsupervised sample speech data is input into the convolutional layer of the initial speech recognition model to extract the frame-level features of the sample speech data.

[0030] The frame-level features are input into a quantizer for clustering to obtain frame-level quantized features; and the frame-level features are masked and the processed frame-level features are input into the N-layer transform network layer contained in the mask prediction layer of the initial speech recognition model for feature extraction to obtain frame-level mask features and the frame-level features extracted by each of the N-layer transform network layers.

[0031] The initial speech recognition model is pre-trained based on the voiceprint features, the frame-level quantization features, the frame-level mask features, the frame-level features extracted by each of the N-layer transform network layers, the first target loss function, the second target loss function, and the third target loss function to generate a pre-trained speech recognition model.

[0032] In one possible implementation, the first objective loss function is a contrastive loss function; the second objective loss function is a mean squared loss function; and the third objective loss function is an optimized entropy loss function.

[0033] In one possible implementation, the step of fine-tuning the pre-trained speech recognition model using supervised speech data to obtain the speech recognition model includes:

[0034] Obtain supervised sample speech data;

[0035] The pre-trained speech recognition model is fine-tuned using the supervised sample speech data and the CTC loss constraint function to obtain the speech recognition model.

[0036] In one possible implementation, the step of fine-tuning the pre-trained speech recognition model using the supervised sample speech data and the CTC loss constraint function to obtain the speech recognition model includes:

[0037] Extract the convolutional layer and mask prediction layer from the pre-trained speech recognition model as the initial encoding layer of the speech recognition model;

[0038] After connecting the initial coding layer to the fully connected layer, the CTC loss constraint function, and the decoding layer, supervised fine-tuning training is performed using the supervised sample speech data. The resulting convolutional layer, mask prediction layer, fully connected layer, and decoding layer constitute the speech recognition model.

[0039] This application also provides a voice recognition device, including:

[0040] The acquisition unit is used to acquire the target speech data to be recognized.

[0041] The prediction unit is used to input the target speech data into a pre-built speech recognition model and predict the speech recognition result of the target speech data.

[0042] The speech recognition model is obtained by pre-training using unsupervised speech data and speaker-related information, and then fine-tuning using supervised speech data.

[0043] In one possible implementation, the device further includes:

[0044] The cleaning unit is used to clean the target speech data to obtain preprocessed target speech data.

[0045] The prediction unit is specifically used for:

[0046] The preprocessed target speech data is input into a pre-built speech recognition model to predict the speech recognition result of the target speech data.

[0047] In one possible implementation, the speech recognition model includes a convolutional layer, a mask prediction layer, a fully connected layer, and a decoding layer; the prediction unit includes:

[0048] The first extraction subunit is used to input the target speech data into the convolutional layer of a pre-built speech recognition model and extract the frame-level features of the target speech data.

[0049] The first prediction subunit is used to perform masking processing on the frame-level features of the target speech data, and input the processed frame-level features into the masking prediction layer of the speech recognition model for feature prediction to obtain the speech features of the target speech data.

[0050] The second prediction subunit is used to input the speech features of the target speech data into the fully connected layer of the speech recognition model to predict the frame-level speech features of the target speech data.

[0051] A determination subunit is used to input the frame-level speech features of the target speech data into the decoding layer of the speech recognition model for decoding, and determine the speech recognition result corresponding to the target speech data based on the decoding result.

[0052] In one possible implementation, the device further includes:

[0053] The pre-training unit is used to perform pre-training processing on the initial speech recognition model by removing speaker-related information layer by layer using unsupervised speech data and a preset target loss function, so as to obtain the pre-trained speech recognition model.

[0054] The fine-tuning unit is used to fine-tune the pre-trained speech recognition model using supervised speech data to obtain the speech recognition model.

[0055] In one possible implementation, the pre-training unit includes:

[0056] The first acquisition subunit is used to acquire unsupervised sample speech data;

[0057] The pre-training subunit is used to pre-train the initial speech recognition model using the unsupervised sample speech data, the first target loss function, the second target loss function, and the third target loss function to obtain the pre-trained speech recognition model.

[0058] The first target loss function is used to narrow the distance between the frame-level mask features extracted by the mask prediction layer in the initial speech recognition model and the corresponding frame-level quantization features extracted by the quantizer, and to widen the distance between the target loss function and other frame-level quantization features extracted by the quantizer.

[0059] The second objective loss function is used to improve the accuracy of the speaker features extracted by the initial speech recognition model;

[0060] The third objective loss function is used to constrain the weights of the frame-level features extracted by the N transform network layers contained in the mask prediction layer of the initial speech recognition model; where N is a positive integer greater than 0.

[0061] In one possible implementation, the pre-trained subunit includes:

[0062] The second extraction subunit is used to input the unsupervised sample speech data into a time-delay neural network to extract the voiceprint features of the sample speech data.

[0063] The third extraction subunit is used to input the unsupervised sample speech data into the convolutional layer of the initial speech recognition model and extract the frame-level features of the sample speech data.

[0064] The input subunit is used to input the frame-level features into a quantizer for clustering processing to obtain frame-level quantized features; and to perform masking processing on the frame-level features, and input the processed frame-level features into the N-layer transform network layer contained in the mask prediction layer of the initial speech recognition model for feature extraction to obtain frame-level mask features and the frame-level features extracted by each of the N-layer transform network layers.

[0065] A generation subunit is used to pre-train the initial speech recognition model based on the voiceprint features, the frame-level quantization features, the frame-level mask features, the frame-level features extracted by each of the N-layer transform network layers, the first target loss function, the second target loss function, and the third target loss function, to generate a pre-trained speech recognition model.

[0066] In one possible implementation, the first objective loss function is a contrastive loss function; the second objective loss function is a mean squared loss function; and the third objective loss function is an optimized entropy loss function.

[0067] In one possible implementation, the fine-tuning unit includes:

[0068] The second acquisition subunit is used to acquire supervised sample speech data;

[0069] The fine-tuning subunit is used to fine-tune the pre-trained speech recognition model using the supervised sample speech data and the CTC loss constraint function to obtain the speech recognition model.

[0070] In one possible implementation, the fine-tuning subunit includes:

[0071] The fourth extraction subunit is used to extract the convolutional layer and mask prediction layer from the pre-trained speech recognition model as the initial encoding layer of the speech recognition model.

[0072] The sub-unit is used to connect the initial coding layer to the fully connected layer, the CTC loss constraint function, and the decoding layer, and then perform supervised fine-tuning training using the supervised sample speech data. The resulting convolutional layer, mask prediction layer, fully connected layer, and decoding layer constitute the speech recognition model.

[0073] This application also provides a voice recognition device, including: a processor, a memory, and a system bus;

[0074] The processor and the memory are connected via the system bus;

[0075] The memory is used to store one or more programs, the one or more programs including instructions, which, when executed by the processor, cause the processor to perform any of the above-described implementations of the speech recognition method.

[0076] This application also provides a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform any of the above-described implementations of the speech recognition method.

[0077] This application also provides a computer program product, which, when run on a terminal device, causes the terminal device to execute any of the above-described implementations of the speech recognition method.

[0078] This application provides a speech recognition method, apparatus, storage medium, and device. First, target speech data to be recognized is acquired. Then, the target speech data is input into a pre-built speech recognition model to predict the speech recognition result. The speech recognition model is obtained by pre-training using unsupervised speech data and speaker-related information, followed by fine-tuning using supervised speech data. Therefore, by constructing the speech recognition model using unsupervised speech data and speaker-related information, and then fine-tuning it using supervised speech data, the robustness of the model to speech recognition of different speakers is effectively improved. This results in higher accuracy of the recognition result when using the speech recognition model to recognize target speech data, thereby improving the recognition performance. Attached Figure Description

[0079] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0080] Figure 1 A flowchart illustrating a speech recognition method provided in an embodiment of this application;

[0081] Figure 2 A schematic diagram illustrating the pre-training process of the speech recognition model provided in this application embodiment;

[0082] Figure 3 A schematic diagram illustrating the fine-tuning process of the speech recognition model provided in this application embodiment;

[0083] Figure 4 This is a schematic diagram illustrating the composition of a speech recognition device provided in an embodiment of this application. Detailed Implementation

[0084] Speech recognition typically refers to enabling machines to understand a speaker's voice and convert it into corresponding text using signal processing and other methods. Current deep learning-based speech recognition systems perform well when the training dataset is large enough and unbiased. However, speech recognition has diverse applications, and due to data collection and annotation costs, obtaining datasets with sufficiently large amounts of data and a wide coverage of speakers is quite challenging.

[0085] To address this, commonly used speech recognition methods such as wav2vec2, HuBERT, and Data2vec generally utilize convolutional modules to convert speech into frame-level features. These frame-level features are then used to construct masked units based on a transformer structure for prediction tasks. Deep network features of the speech are learned through comparative learning loss and other methods. For specific downstream speech recognition tasks, the pre-trained model can be treated as an encoder layer, followed by a wiring layer. The CTC loss can then be used as a supervisory signal for fine-tuning, achieving good speech recognition results with a small amount of supervised speech recognition data. This solves the problem of insufficient training data for various speech recognition scenarios.

[0086] However, although the above method can utilize a large amount of unsupervised data to address the lack of supervised data and train a recognition model, it only performs well for speech recognition with small amounts of data. This is because the influence of different speaker information is not considered during the model training process, which leads to the inaccuracy of speech recognition results based on speech pre-training due to the differences between different speakers.

[0087] To address the aforementioned shortcomings, this application provides a speech recognition method. First, target speech data to be recognized is acquired. Then, the target speech data is input into a pre-constructed speech recognition model to predict the speech recognition result. The speech recognition model is obtained by pre-training using unsupervised speech data and speaker-related information, followed by fine-tuning using supervised speech data. Therefore, because this application constructs a speech recognition model by first pre-training with unsupervised speech data and speaker-related information, and then fine-tuning with supervised speech data, it effectively improves the robustness of the model in recognizing speech from different speakers. Consequently, when using this speech recognition model to recognize target speech data, it can effectively improve the accuracy of the recognition results, thereby enhancing the recognition performance.

[0088] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0089] First Embodiment

[0090] See Figure 1 This is a flowchart illustrating a speech recognition method provided in this embodiment. The method includes the following steps:

[0091] S101: Acquire the target speech data to be recognized.

[0092] In this embodiment, any speech data that needs to be recognized is defined as the target speech data to be recognized. It should be noted that this embodiment does not limit the language type of the target speech data. For example, the target speech data can be speech data composed of Chinese or speech data composed of English, etc. At the same time, this embodiment does not limit the length of the target speech data. For example, the target speech data can be a sentence or a paragraph, etc.

[0093] It is understood that the target speech data can be obtained through recording or other means as needed. For example, telephone conversations in people's daily lives or recordings from a conference system can be used as target speech data. While obtaining the target speech data, the solution provided in this embodiment is used to process the target speech data in order to identify the text information corresponding to the target speech data.

[0094] S102: Input the target speech data into a pre-built speech recognition model to predict the speech recognition result of the target speech data; wherein, the speech recognition model is obtained by pre-training using unsupervised speech data and related speaker information, and then fine-tuning using supervised speech data.

[0095] In this embodiment, after obtaining the target speech data to be recognized through step S101, in order to accurately identify the text information corresponding to the target speech data, the target speech data can be further input into a pre-built speech recognition model to predict the speech recognition result of the target speech data.

[0096] It should be noted that, in order to accurately identify the text information corresponding to the target speech data, after obtaining the target speech data to be identified through step S101, an optional implementation method is to first use existing or future data cleaning methods to clean the target speech data to obtain preprocessed target speech data. For example, the target speech data can be processed into 16000Hz single-channel audio, etc., as preprocessed target speech data. Then, the preprocessed target speech data is input into the pre-built speech recognition model to predict a more accurate speech recognition result corresponding to the target speech data.

[0097] It should also be noted that, in order to improve the accuracy of speech recognition results, this application pre-trains the model using a large amount of unsupervised speech data and speaker-related information, and then fine-tunes the pre-trained model using a small amount of supervised speech data, thereby constructing a speech recognition model with better recognition performance. Furthermore, this application does not limit the specific network structure of the speech recognition model and can select and set it according to actual conditions. A preferred implementation is that the pre-built speech recognition model may include, but is not limited to, convolutional layers, mask prediction layers, fully connected layers, and decoder layers. The mask prediction layer may contain N transformer layers, where N is a positive integer greater than 0. Thus, by first pre-training the model using unsupervised speech data and speaker-related information, and then fine-tuning it using supervised speech data, the robustness of the model to speech recognition from different speakers can be effectively improved. Therefore, when using this speech recognition model to recognize target speech data, the accuracy of the recognition results can be effectively improved, thereby enhancing the recognition performance.

[0098] Specifically, one optional implementation is that when the pre-built speech recognition model includes convolutional layers, mask prediction layers, fully connected layers, and a decoder, the specific implementation process of "inputting the target speech data into the pre-built speech recognition model and predicting the speech recognition result of the target speech data" in step S102 may include the following steps S1021-S1024:

[0099] Step S1021: Input the target speech data into the convolutional layer of the pre-built speech recognition model to extract frame-level features of the target speech data.

[0100] In this implementation, after obtaining the target speech data to be recognized through step S101, the target speech data (or preprocessed target speech data) can be further input into the convolutional layer (such as convmodule) of the speech recognition model for feature extraction to obtain the frame-level features of the target speech data, which are then used to execute the subsequent step S1022.

[0101] This application does not limit the specific network structure of the convolutional layer, and it can be selected according to the actual situation. For example, a single convolutional network or multiple convolutional networks, or a combination of normalization network and activation network can be used as the convolutional layer.

[0102] Step S1022: Mask the frame-level features of the target speech data, and input the processed frame-level features into the mask prediction layer of the speech recognition model for feature prediction to obtain the speech features of the target speech data.

[0103] In this implementation, after the target speech data (or preprocessed target speech data) is input into the convolutional layer of the speech recognition model in step S1021 to obtain the frame-level features of the target speech data, the frame-level features of the target speech data can be further masked, and the processed frame-level features are input into the mask prediction layer of the speech recognition model for feature prediction to obtain the speech features corresponding to the target speech data, which are then used to execute the subsequent step S1023.

[0104] This application does not limit the specific network structure of the mask prediction layer, and it can be selected according to the actual situation. For example, the mask prediction layer can be constructed using an N-layer transformer network.

[0105] Step S1023: Input the speech features of the target speech data into the fully connected layer of the speech recognition model to predict the frame-level speech features of the target speech data.

[0106] In this implementation, the frame-level features of the target speech data are masked in step S1022, and the processed frame-level features are input into the mask prediction layer of the speech recognition model for feature prediction. After obtaining the speech features of the target speech data, the speech features of the target speech data can be further input into the fully connected layer (fc) of the speech recognition model to predict the frame-level speech features corresponding to the target speech data, which are then used to execute the subsequent step S1024.

[0107] Among them, frame-level speech features refer to the text information corresponding to the target speech data, which is used to emphasize the text information, so as to facilitate subsequent preparation for accurate speech recognition.

[0108] Step S1024: Input the frame-level speech features of the target speech data into the decoding layer of the speech recognition model for decoding, and determine the speech recognition result corresponding to the target speech data based on the decoding result.

[0109] In this implementation, after predicting the frame-level speech features that emphasize the text information corresponding to the target speech data in step S1023, the frame-level speech features can be further input into the decoding layer of the speech recognition model for decoding to obtain text data that can accurately represent the semantics corresponding to the target speech data, which serves as the speech recognition result corresponding to the target speech data.

[0110] In one possible implementation of this application, to improve the accuracy of speech recognition results, a speech recognition model with better recognition performance is pre-constructed. The specific construction process may include a pre-training process and a fine-tuning process. The pre-training process refers to using unsupervised speech data and a preset target loss function to perform layer-by-layer pre-training processing on an initial speech recognition model (including but not limited to convolutional layers, mask prediction layers, etc.) to remove speaker-related information, resulting in a pre-trained speech recognition model. The specific content of the aforementioned preset target loss function can be limited according to actual circumstances; this application does not limit it. For example, the preset target loss function can be the first target loss function, the second target loss function, the third target loss function, etc., mentioned later. The fine-tuning process refers to using supervised speech data to fine-tune the pre-trained speech recognition model. Specifically, it involves further fine-tuning some of the structural components extracted from the trained speech recognition model to construct a speech recognition model with better performance.

[0111] Next, this embodiment will provide a detailed description of the pre-training and fine-tuning processes of the speech recognition model. For example... Figure 2 The diagram illustrates the pre-training process of the speech recognition model provided in this application embodiment. The specific implementation process may include the following steps A1-A2:

[0112] Step A1: Obtain unsupervised sample speech data.

[0113] In this embodiment, to construct a speech recognition model, a large amount of unsupervised speech data is first required for model pre-training. This necessitates extensive preparatory work, namely, collecting a large amount of unsupervised speech data (i.e., speech data without corresponding text recognition results) in various languages ​​(such as Chinese, English, German, French, etc.). This can be done using a professional recording studio, a quiet room or conference room, or even using professional recording equipment or simple recording devices like mobile phones to record user speech data. This speech data can come from multiple different speakers. After collecting this speech data, it can be used as sample speech data to execute subsequent step A2, thereby achieving pre-training of the speech recognition model.

[0114] Step A2: Use unsupervised sample speech data, the first objective loss function, the second objective loss function, and the third objective loss function to pre-train the initial speech recognition model to obtain the pre-trained speech recognition model.

[0115] It should be noted that the initial speech recognition model used in this application may include, but is not limited to, convolutional layers and mask prediction layers, such as... Figure 2 As shown in the diagram. The convolutional layers can be Convolutional Neural Networks (CNNs) or conv modules, etc.; the mask prediction layer can include N layers of transformer networks.

[0116] In this embodiment, in order to pre-train the language recognition model, one optional approach is to further clean the unsupervised sample speech data after obtaining a large amount of unsupervised sample speech data through step A1, so as to obtain pre-processed unsupervised sample speech data. For example, the unsupervised sample speech data can be cleaned to retain only the speech with a single audio duration of 1-20 seconds and a silent segment ratio of no more than 20%, and then processed into a single-channel audio with a sampling rate of 16000Hz as pre-processed unsupervised sample speech data for pre-training the initial speech recognition model.

[0117] Based on this, one possible implementation is, as follows: Figure 2 As shown, unsupervised sample speech data (or pre-processed unsupervised sample speech data) can first be input into a pre-trained Time Delay Neural Network (TDNN) to extract the speaker characteristics (x-vector) of the sample speech data, and define it as f.xvec It should be noted that this application focuses on voiceprint features f xvec The extraction method is not limited; the aforementioned method of feature extraction using TDNN is merely a preferred example.

[0118] And, as Figure 2 As shown, unsupervised sample speech data (or preprocessed unsupervised sample speech data) can also be input into the convolutional layer (such as a conv module) in the initial speech recognition model to extract frame-level features of the sample speech data.

[0119] Then, as Figure 2 As shown, the frame-level feature is directly input into the quantizer (Qantizer) for clustering without masking, resulting in the frame-level quantized feature, which is defined as... The frame-level features are then masked, and the processed frame-level features are input into the N-layer transformer network contained in the mask prediction layer of the initial speech recognition model for feature extraction. The frame-level features extracted by the i-th (i is a positive integer not greater than N) layer transformer network are defined as... And the frame-level features output by the Nth layer transformer network (i.e. ) is defined as a frame-level mask feature.

[0120] Next, based on the obtained voiceprint features and frame-level quantization features... Frame-level mask features The frame-level features extracted by each layer of the N-layer transform network The initial speech recognition model is pre-trained using a first objective loss function, a second objective loss function, and a third objective loss function. During the training process, the model parameters of the initial speech recognition model can be continuously updated based on the changes in the function values ​​of the first objective loss function, the second objective loss function, and the third objective loss function until their function values ​​meet the requirements, such as the weighted sum of the three reaching the minimum value and the change being very small (basically unchanged), or reaching the preset maximum number of iterations (such as 10,000 times). At this point, the update of the model parameters stops, the pre-training of the initial speech recognition model is completed, and the pre-trained speech recognition model is generated.

[0121] The first objective function is used to bring the frame-level mask features extracted by the mask prediction layer in the initial speech recognition model closer to the target image. The distance between the target loss function and the corresponding frame-level quantization features extracted by the quantizer, and the distance used to widen the gap with other frame-level quantization features extracted by the quantizer, are used to achieve comparative learning between the mask prediction result output by the Nth layer transformer network in the mask prediction layer and the prediction result of the corresponding frame in the quantizer. A preferred implementation is that the first objective loss function can be a contrastive loss function, and the specific calculation formula for the first objective function is as follows:

[0122]

[0123] Among them, L contrant Represents the first objective loss function; The contrastive loss is designed to enable the model to learn the predicted mask results (i.e., frame-level mask features) through the mask prediction task. ) and the corresponding frame-level quantization features after quantization by the quantizer (i.e. The cosine distance of the feature is closer to that of the feature in the frame and farther from the feature in other frames.

[0124] The second objective function aims to improve the accuracy of speaker features extracted by the initial speech recognition model. Specifically, during pre-training, the frame-level features extracted by each of the N layers of the transformer network are... We can first perform a weighted sum, such as Figure 2 As shown, pooling and linear transformation are then performed to predict the speaker features of the unsupervised sample speech data, and this feature is defined as f. spk f is used to characterize the relevant features of the speaker who produced these unsupervised sample speech data. spk The specific calculation formula is as follows:

[0125]

[0126] Among them, w i represents the normalized weights of the frame-level features extracted by the i-th layer transformer; g(·) represents post-processing functions such as pooling and linear transformation.

[0127] Furthermore, the speaker characteristics f of the unsupervised sample speech data predicted by the model can be calculated. spk The speaker features f extracted from unsupervised sample speech data via TDNN xvec The mean squared loss is used to improve the accuracy of speaker features extracted by the initial speech recognition model. One preferred implementation is that the second objective loss function can be the mean squared loss function (MSE loss), and the specific calculation formula for the second objective function is as follows:

[0128]

[0129] The third objective function constrains the weights of frame-level features extracted by the N-layer transform network in the initial speech recognition model's mask prediction layer, ensuring that lower-level features have larger weights and higher-level features have smaller weights. Combined with the second objective loss function, this guarantees that the lower layers of the N-layer transformer network contain richer speaker information, while the higher layers contain less. This achieves the goal of removing speaker-related information layer by layer, ultimately obtaining speaker-independent speech features at the higher layers. A preferred implementation is that the third objective loss function can be an optimized entropy loss function, and the specific calculation formula for the third objective function is as follows:

[0130]

[0131] Among them, L w Let represent the third objective loss function, which is an improvement upon the entropy loss function. The general entropy loss function can be expressed as: This allows the learned weight parameters of the model to become sharper, rather than uniformly distributed across the parameters of each layer. Here, this application further increases the weights w of the higher layers (i.e., the i-th layer) in the N-layer transformer network by multiplying by i. i log(w i This reduces the influence of the function value on the weights w corresponding to the features in higher layers (i.e., the i-th layer). i The purpose of determining the value.

[0132] An alternative implementation is to also use the first objective function L contrant Second objective function L mse The third objective function L w Weighted summations are performed to construct the overall loss function, defined as L, which is used to pre-train the initial speech recognition model. During training, the model parameters of the initial speech recognition model are continuously updated based on the changes in the value of this overall loss function L, until the function value of L meets the requirements, such as reaching a minimum value with very small changes (basically unchanged), indicating model convergence. At this point, updating the model parameters stops, completing the pre-training of the initial speech recognition model and generating the pre-trained speech recognition model. The specific formula for calculating L is as follows:

[0133] L = L contrast +α*l MSE +β*L w

[0134] In this application, α and β are both preset coefficients that can be set according to actual conditions and empirical values. The specific values ​​of the two are not limited in this application. For example, α and β can be set to 0.1 and 0.5 respectively.

[0135] Through the above steps A1-A2, the speech recognition model is pre-trained. After obtaining the pre-trained speech recognition model, it can be further fine-tuned using supervised speech datasets to extract some structural components from the trained speech recognition model, thereby constructing a speech recognition model with better recognition performance. Figure 3 The diagram illustrates the fine-tuning process of the speech recognition model provided in this embodiment of the application. The specific implementation process may include the following steps B1-B2:

[0136] Step B1: Obtain supervised sample speech data.

[0137] In this embodiment, to construct a speech recognition model, not only is model pre-training necessary, but also a small amount of supervised speech data is required for model fine-tuning. Therefore, it is necessary to pre-collect a small amount of supervised speech data (i.e., speech data containing corresponding text recognition results) in various languages ​​(such as Chinese, English, German, French, etc.), and this speech data can come from multiple different speakers. After collecting this supervised speech data, it can be used as sample speech data to perform subsequent step B2, thereby achieving fine-tuning of the pre-trained speech recognition model.

[0138] It should be noted that, in order to pre-train the language recognition model, one possible approach is to further preprocess the supervised sample speech data after obtaining a small amount of supervised sample speech data to obtain preprocessed unsupervised sample speech data. For example, the supervised sample speech data can be processed into single-channel audio with a sampling rate of 16000Hz as preprocessed unsupervised sample speech data. The text annotations can then be divided into sub-words according to the modeling units. For example, Chinese text can be divided into single characters, and English words can be divided into sub-words with fewer letters, etc., to fine-tune the pre-trained speech recognition model.

[0139] Step B2: Use supervised sample speech data and the CTC loss constraint function to fine-tune the pre-trained speech recognition model to obtain the speech recognition model.

[0140] It should be noted that after pre-training the speech recognition model through steps A1-A2 above, and obtaining the pre-trained speech recognition model, the convolutional layers (such as convmodules) and mask prediction layers (including N-layer transformer networks) contained in the pre-trained speech recognition model can be extracted as the initial encoder layer for the speech recognition task. This initial encoder layer is then connected to a fully connected layer (fc), a CTC (Connectionist Temporal Classification) loss constraint function, and a decoding layer. Supervised fine-tuning training is then performed using the supervised sample speech data obtained in step B1. Figure 3 As shown, the convolutional layer, mask prediction layer, fully connected layer, and decoding layer obtained after training are then used to construct a speech recognition model.

[0141] It should also be noted that, to ensure the final speech recognition model inherits the pre-trained model's insensitivity to speaker acoustic characteristics and is thus unaffected by specific speakers, achieving better speech recognition results, the supervised training (fine-tuning) of the model for speech recognition maintains the parameters of the convolutional layers (e.g., convmodules) unchanged. The parameters of the mask prediction layer (including N layers of transformer network) remain fixed for the first M iterations (the specific value can be set according to the actual situation; this application does not limit it, for example, M can be set to 10000, etc.), adjusting only the parameters of the fully connected (fc) layer. Subsequently, the parameters of the mask prediction layer (including N layers of transformer network) are updated together with the fc parameters. This phased fine-tuning of the pre-trained model parameters ensures that the final speech recognition model can better inherit the pre-trained model's ability to extract speaker-independent speech features, thereby achieving better speech recognition results.

[0142] In summary, the speech recognition method provided in this embodiment first acquires the target speech data to be recognized, then inputs the target speech data into a pre-built speech recognition model to predict the speech recognition result of the target speech data. The speech recognition model is obtained by pre-training using unsupervised speech data and speaker-related information, followed by fine-tuning using supervised speech data. It is evident that because this application constructs a speech recognition model by first pre-training using unsupervised speech data and speaker-related information, and then fine-tuning using supervised speech data, it effectively improves the robustness of the model in recognizing speech from different speakers. Therefore, when using this speech recognition model to recognize target speech data, it can effectively improve the accuracy of the recognition result, thereby improving the recognition effect.

[0143] Second Embodiment

[0144] This embodiment will introduce a speech recognition device; please refer to the above method embodiment for related content.

[0145] See Figure 4 This is a schematic diagram of the composition of a voice recognition device provided in this embodiment. The device 400 includes:

[0146] Acquisition unit 401 is used to acquire target speech data to be recognized;

[0147] The prediction unit 402 is used to input the target speech data into a pre-built speech recognition model and predict the speech recognition result of the target speech data;

[0148] The speech recognition model is obtained by pre-training using unsupervised speech data and speaker-related information, and then fine-tuning using supervised speech data.

[0149] In one implementation of this embodiment, the apparatus further includes:

[0150] The cleaning unit is used to clean the target speech data to obtain preprocessed target speech data.

[0151] The prediction unit 402 is specifically used for:

[0152] The preprocessed target speech data is input into a pre-built speech recognition model to predict the speech recognition result of the target speech data.

[0153] In one implementation of this embodiment, the speech recognition model includes a convolutional layer, a mask prediction layer, a fully connected layer, and a decoding layer; the prediction unit 402 includes:

[0154] The first extraction subunit is used to input the target speech data into the convolutional layer of a pre-built speech recognition model and extract the frame-level features of the target speech data.

[0155] The first prediction subunit is used to perform masking processing on the frame-level features of the target speech data, and input the processed frame-level features into the masking prediction layer of the speech recognition model for feature prediction to obtain the speech features of the target speech data.

[0156] The second prediction subunit is used to input the speech features of the target speech data into the fully connected layer of the speech recognition model to predict the frame-level speech features of the target speech data.

[0157] A determination subunit is used to input the frame-level speech features of the target speech data into the decoding layer of the speech recognition model for decoding, and determine the speech recognition result corresponding to the target speech data based on the decoding result.

[0158] In one implementation of this embodiment, the apparatus further includes:

[0159] The pre-training unit is used to perform pre-training processing on the initial speech recognition model by removing speaker-related information layer by layer using unsupervised speech data and a preset target loss function, so as to obtain the pre-trained speech recognition model.

[0160] The fine-tuning unit is used to fine-tune the pre-trained speech recognition model using supervised speech data to obtain the speech recognition model.

[0161] In one implementation of this embodiment, the pre-training unit includes:

[0162] The first acquisition subunit is used to acquire unsupervised sample speech data;

[0163] The pre-training subunit is used to pre-train the initial speech recognition model using the unsupervised sample speech data, the first target loss function, the second target loss function, and the third target loss function to obtain the pre-trained speech recognition model.

[0164] The first target loss function is used to narrow the distance between the frame-level mask features extracted by the mask prediction layer in the initial speech recognition model and the corresponding frame-level quantization features extracted by the quantizer, and to widen the distance between the target loss function and other frame-level quantization features extracted by the quantizer.

[0165] The second objective loss function is used to improve the accuracy of the speaker features extracted by the initial speech recognition model;

[0166] The third objective loss function is used to constrain the weights of the frame-level features extracted by the N transform network layers contained in the mask prediction layer of the initial speech recognition model; where N is a positive integer greater than 0.

[0167] In one implementation of this embodiment, the pre-training subunit includes:

[0168] The second extraction subunit is used to input the unsupervised sample speech data into a time-delay neural network to extract the voiceprint features of the sample speech data.

[0169] The third extraction subunit is used to input the unsupervised sample speech data into the convolutional layer of the initial speech recognition model and extract the frame-level features of the sample speech data.

[0170] The input subunit is used to input the frame-level features into a quantizer for clustering processing to obtain frame-level quantized features; and to perform masking processing on the frame-level features, and input the processed frame-level features into the N-layer transform network layer contained in the mask prediction layer of the initial speech recognition model for feature extraction to obtain frame-level mask features and the frame-level features extracted by each of the N-layer transform network layers.

[0171] A generation subunit is used to pre-train the initial speech recognition model based on the voiceprint features, the frame-level quantization features, the frame-level mask features, the frame-level features extracted by each of the N-layer transform network layers, the first target loss function, the second target loss function, and the third target loss function, to generate a pre-trained speech recognition model.

[0172] In one implementation of this embodiment, the first objective loss function is a contrastive loss function; the second objective loss function is a mean squared loss function; and the third objective loss function is an optimized entropy loss function.

[0173] In one implementation of this embodiment, the fine-tuning unit includes:

[0174] The second acquisition subunit is used to acquire supervised sample speech data;

[0175] The fine-tuning subunit is used to fine-tune the pre-trained speech recognition model using the supervised sample speech data and the CTC loss constraint function to obtain the speech recognition model.

[0176] In one implementation of this embodiment, the fine-tuning subunit includes:

[0177] The fourth extraction subunit is used to extract the convolutional layer and mask prediction layer from the pre-trained speech recognition model as the initial encoding layer of the speech recognition model.

[0178] The sub-unit is used to connect the initial coding layer to the fully connected layer, the CTC loss constraint function, and the decoding layer, and then perform supervised fine-tuning training using the supervised sample speech data. The resulting convolutional layer, mask prediction layer, fully connected layer, and decoding layer constitute the speech recognition model.

[0179] Furthermore, embodiments of this application also provide a voice recognition device, including: a processor, a memory, and a system bus;

[0180] The processor and the memory are connected via the system bus;

[0181] The memory is used to store one or more programs, the one or more programs including instructions that, when executed by the processor, cause the processor to perform any of the above-described implementations of the speech recognition method.

[0182] Furthermore, embodiments of this application also provide a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform any of the above-described implementations of the speech recognition method.

[0183] Furthermore, this application embodiment also provides a computer program product, which, when run on a terminal device, causes the terminal device to execute any of the above-described implementation methods of the speech recognition method.

[0184] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0185] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0186] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0187] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech recognition method, characterized in that, include: Acquire the target speech data to be recognized; The target speech data is input into a pre-built speech recognition model to predict the speech recognition result of the target speech data; The method further includes: Using unsupervised speech data and a preset target loss function, the initial speech recognition model is pre-trained by removing speaker-related information layer by layer, resulting in a pre-trained speech recognition model. The pre-trained speech recognition model is fine-tuned using supervised speech data to obtain the speech recognition model.

2. The method according to claim 1, characterized in that, The speech recognition model includes convolutional layers, mask prediction layers, fully connected layers, and decoding layers; the step of inputting the target speech data into the pre-constructed speech recognition model to predict the speech recognition result of the target speech data includes: The target speech data is input into the convolutional layer of a pre-built speech recognition model to extract frame-level features of the target speech data; The frame-level features of the target speech data are masked, and the processed frame-level features are input into the mask prediction layer of the speech recognition model for feature prediction to obtain the speech features of the target speech data. The speech features of the target speech data are input into the fully connected layer of the speech recognition model to predict the frame-level speech features of the target speech data. The frame-level speech features of the target speech data are input into the decoding layer of the speech recognition model for decoding, and the speech recognition result corresponding to the target speech data is determined based on the decoding result.

3. The method according to claim 1, characterized in that, The process of pre-training the initial speech recognition model by removing speaker-related information layer by layer using unsupervised speech data and a preset target loss function to obtain a pre-trained speech recognition model includes: Obtain unsupervised sample speech data; Using the unsupervised sample speech data, the first objective loss function, the second objective loss function, and the third objective loss function, the initial speech recognition model is pre-trained to obtain the pre-trained speech recognition model. The first target loss function is used to narrow the distance between the frame-level mask features extracted by the mask prediction layer in the initial speech recognition model and the corresponding frame-level quantization features extracted by the quantizer, and to widen the distance between the target loss function and other frame-level quantization features extracted by the quantizer. The second objective loss function is used to improve the accuracy of the speaker features extracted by the initial speech recognition model; The third objective loss function is used to constrain the weights of the frame-level features extracted by the N transform network layers contained in the mask prediction layer of the initial speech recognition model; where N is a positive integer greater than 0.

4. The method according to claim 3, characterized in that, The step of pre-training the initial speech recognition model using the unsupervised sample speech data, the first objective loss function, the second objective loss function, and the third objective loss function to obtain the pre-trained speech recognition model includes: The unsupervised sample speech data is input into a time-delay neural network to extract the speaker features of the sample speech data; The unsupervised sample speech data is input into the convolutional layer of the initial speech recognition model to extract the frame-level features of the sample speech data. The frame-level features are input into a quantizer for clustering to obtain frame-level quantized features; and the frame-level features are masked and the processed frame-level features are input into the N-layer transform network layer contained in the mask prediction layer of the initial speech recognition model for feature extraction to obtain frame-level mask features and the frame-level features extracted by each of the N-layer transform network layers. The initial speech recognition model is pre-trained based on the voiceprint features, the frame-level quantization features, the frame-level mask features, the frame-level features extracted by each of the N-layer transform network layers, the first target loss function, the second target loss function, and the third target loss function to generate a pre-trained speech recognition model.

5. The method according to claim 3 or 4, characterized in that, The first objective loss function is the contrastive loss function; the second objective loss function is the mean squared loss function; and the third objective loss function is the optimized entropy loss function.

6. The method according to claim 1, characterized in that, The step of fine-tuning the pre-trained speech recognition model using supervised speech data to obtain the speech recognition model includes: Obtain supervised sample speech data; The pre-trained speech recognition model is fine-tuned using the supervised sample speech data and the CTC loss constraint function to obtain the speech recognition model.

7. The method according to claim 6, characterized in that, The step of fine-tuning the pre-trained speech recognition model using the supervised sample speech data and the CTC loss constraint function to obtain the speech recognition model includes: Extract the convolutional layer and mask prediction layer from the pre-trained speech recognition model as the initial encoding layer of the speech recognition model; After connecting the initial coding layer to the fully connected layer, the CTC loss constraint function, and the decoding layer, supervised fine-tuning training is performed using the supervised sample speech data. The resulting convolutional layer, mask prediction layer, fully connected layer, and decoding layer constitute the speech recognition model.

8. A voice recognition device, characterized in that, include: The acquisition unit is used to acquire the target speech data to be recognized. The prediction unit is used to input the target speech data into a pre-built speech recognition model and predict the speech recognition result of the target speech data. The pre-training unit is used to perform pre-training processing on the initial speech recognition model by removing speaker-related information layer by layer using unsupervised speech data and a preset target loss function, so as to obtain the pre-trained speech recognition model. The fine-tuning unit is used to fine-tune the pre-trained speech recognition model using supervised speech data to obtain the speech recognition model.

9. A voice recognition device, characterized in that, include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, the one or more programs including instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the method described in any one of claims 1-7.

Images