Speech recognition method, apparatus, device, and computer readable storage medium

Abstract

Embodiments of the present application provide a speech recognition method, device and equipment and a computer readable storage medium. The method comprises: obtaining audio data to be recognized; determining a phoneme sequence corresponding to the audio data according to the audio data and a trained target acoustic neural network. The target acoustic neural network is trained based on unsupervised sample data and supervised sample data carrying text labels; inputting the phoneme sequence into a trained language model for processing, and outputting a speech recognition result of the audio data. The embodiments of the present application train an acoustic neural network by using a large amount of unsupervised sample data and supervised sample data carrying text labels, which can effectively make up for the insufficient estimation of the speech feature space caused by the insufficient order of magnitude of the supervised sample data, improve the recognition accuracy of the trained acoustic neural network, and thus improve the subsequent speech recognition effect.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, specifically to a speech recognition method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] Speech recognition, as an important component of artificial intelligence, has received increasing attention in recent years and has made significant progress. Traditional speech recognition generally includes two parts: acoustics and language. Acoustics mainly focuses on learning the characteristics of audio pronunciation, while language mainly focuses on learning grammatical rules. Deep Neural Networks (DNNs), due to their excellent classification capabilities and ability to express nonlinear relationships, are increasingly being used in the modeling process of acoustic and language models.

[0003] However, the accuracy of both acoustic and language models in speech recognition relies on a large amount of labeled training data. In the actual business of telephone customer service, models are typically trained using open-source libraries and third-party companies' labeled business data. However, the actual labeled data closely reflecting the business scenario may only be a few hundred or a few thousand hours, which is not an ideal amount of data, resulting in less than satisfactory performance of the trained speech recognition model. Summary of the Invention

[0004] This application provides a speech recognition method, apparatus, device, and computer-readable storage medium, aiming to solve the problem in existing speech recognition technologies where the amount of labeled data used for training is insufficient, resulting in unsatisfactory performance of the trained speech recognition model.

[0005] On one hand, embodiments of this application provide a speech recognition method, including:

[0006] Obtain the audio data to be recognized;

[0007] Based on the audio data and the trained target acoustic neural network, the phoneme sequence corresponding to the audio data is determined; the target acoustic neural network is trained based on unsupervised sample data and supervised sample data carrying text labels.

[0008] The phoneme sequence is input into a trained language model for processing, and the speech recognition result of the audio data is output.

[0009] As a feasible embodiment of this application, before determining the phoneme sequence corresponding to the audio data based on the audio data and the trained target acoustic neural network, the method includes:

[0010] Obtain sample audio data and text labels corresponding to supervised audio data within the sample audio data;

[0011] An initial acoustic neural network and a first language model are obtained by training the supervised audio data and its corresponding text labels.

[0012] The text labels corresponding to the unsupervised audio data are determined based on the initial acoustic neural network, the first language model, and the unsupervised audio data in the sample audio data.

[0013] The target acoustic neural network is trained based on the supervised audio data and its corresponding text labels, as well as the unsupervised audio data and its corresponding text labels.

[0014] As a feasible embodiment of this application, the step of training a target acoustic neural network based on the supervised audio data and its corresponding text labels, and the unsupervised audio data and its corresponding text labels, includes:

[0015] The supervised audio data and its corresponding text tags are forcibly aligned according to a preset alignment model to obtain supervised state tags and their corresponding supervised audio segments.

[0016] The unsupervised audio data and its corresponding text labels are forcibly aligned according to the alignment model to obtain unsupervised state labels and their corresponding unsupervised audio segments.

[0017] The preset initial multi-task neural network is trained based on the supervised state labels and their corresponding supervised audio segments, as well as the unsupervised state labels and their corresponding unsupervised audio segments, to obtain the target multi-task neural network.

[0018] The target acoustic neural network is generated based on the input layer, intermediate layer and first output layer of the target multi-task neural network.

[0019] As a feasible embodiment of this application, the step of training a preset initial multi-task neural network based on the supervised state labels and their corresponding supervised audio segments, and the unsupervised state labels and their corresponding unsupervised audio segments, to obtain a target multi-task neural network, includes:

[0020] The unsupervised state labels and their corresponding unsupervised audio segments are merged according to the unsupervised state labels to obtain unsupervised phoneme labels and their corresponding unsupervised audio segments.

[0021] The target multi-task neural network is obtained by training a preset initial multi-task neural network based on the supervised state labels and their corresponding supervised audio segments, as well as the unsupervised phoneme labels and their corresponding unsupervised audio segments.

[0022] As a feasible embodiment of this application, before inputting the phoneme sequence into a trained language model for processing and outputting the speech recognition result of the audio data, the method further includes:

[0023] A second language model is trained based on the text recognition results corresponding to unsupervised audio data.

[0024] The preset first language model and the second language model are fused to obtain a trained language model; the first language model is trained using text labels corresponding to supervised audio data.

[0025] As a feasible embodiment of this application, the step of training a second language model based on unsupervised audio data and its corresponding text recognition results includes:

[0026] Based on the text length and confidence level of the text recognition results corresponding to the unsupervised audio data, target audio data is filtered out from the unsupervised audio data;

[0027] A second language model is trained based on the text recognition results corresponding to the target audio data.

[0028] As a feasible embodiment of this application, determining the phoneme sequence corresponding to the audio data based on the audio data and the trained target acoustic neural network includes:

[0029] The audio data is enhanced to obtain enhanced audio data;

[0030] Extract the audio features from the enhanced audio data;

[0031] The audio features are input into a trained target acoustic neural network to obtain the phoneme sequence corresponding to the audio data.

[0032] On the other hand, embodiments of this application also provide a voice recognition device, including:

[0033] The acquisition module is used to acquire the audio data to be recognized;

[0034] A phoneme determination module is used to determine the phoneme sequence corresponding to the audio data based on the audio data and a trained target acoustic neural network; the target acoustic neural network is trained based on unsupervised sample data and supervised sample data carrying speech labels.

[0035] The speech recognition module is used to input the phoneme sequence into a trained language model for processing and output the speech recognition result of the audio data.

[0036] On the other hand, this application embodiment also provides a voice recognition device, which includes a processor, a memory, and a voice recognition program stored in the memory and executable on the processor. The processor executes the voice recognition program to implement the steps in the above-described voice recognition method.

[0037] On the other hand, embodiments of this application also provide a computer-readable storage medium storing a speech recognition program, which is executed by a processor to implement the steps in the speech recognition device method described above.

[0038] The speech recognition method proposed in this application uses a large amount of unsupervised sample data and supervised sample data with text labels to train an acoustic neural network. This effectively compensates for the problem of insufficient estimation of speech feature space caused by insufficient supervised sample data, improves the recognition accuracy of the trained acoustic neural network, and further enhances the speech recognition effect of subsequent audio to be recognized using the acoustic neural network, thereby improving the accuracy of speech recognition. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 A schematic diagram illustrating an implementation scenario of a speech recognition method provided in this application embodiment;

[0041] Figure 2 This is a flowchart illustrating the steps of a speech recognition method provided in an embodiment of this application.

[0042] Figure 3 This application provides a schematic flowchart illustrating the steps involved in training an acoustic neural network.

[0043] Figure 4 This application provides a schematic flowchart illustrating the steps involved in obtaining an acoustic neural network through multi-task training.

[0044] Figure 5 This application provides a schematic diagram of a network model structure for a multi-task neural network.

[0045] Figure 6 A flowchart illustrating the steps for determining phoneme tags in unsupervised audio data is provided in this application embodiment.

[0046] Figure 7 This application provides a schematic diagram of the steps for training a language model according to an embodiment of the present application.

[0047] Figure 8 This application provides a schematic flowchart of steps for filtering unsupervised audio data.

[0048] Figure 9 This application provides a schematic flowchart of a process for preprocessing audio data to determine a phoneme sequence.

[0049] Figure 10 This application provides a complete implementation process for training and applying a speech recognition model.

[0050] Figure 11 This is a schematic diagram of the structure of a speech recognition device provided in an embodiment of this application;

[0051] Figure 12 This is a schematic diagram of the structure of a voice recognition device provided in an embodiment of this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of the present invention.

[0053] In this application, the term "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to implement and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be implemented without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in the embodiments of this application.

[0054] This application provides a speech recognition method, apparatus, device, and computer-readable storage medium, which will be described in detail below.

[0055] In this embodiment, the speech recognition method is deployed as a program on a speech recognition device, which is installed in the speech recognition equipment as a processor. The speech recognition device in the speech recognition equipment executes the following steps by running the program corresponding to the speech recognition method: acquiring audio data to be recognized; determining the phoneme sequence corresponding to the audio data based on the audio data and a trained target acoustic neural network; the target acoustic neural network is trained based on unsupervised sample data and supervised sample data carrying text labels; inputting the phoneme sequence into a trained language model for processing, and outputting the speech recognition result of the audio data.

[0056] like Figure 1 As shown, Figure 1 This is a schematic diagram illustrating an implementation scenario of a speech recognition method provided in this application embodiment. The implementation scenario provided in this application embodiment includes a speech acquisition device 100 and a speech recognition device 200. The speech acquisition device 100 is mainly used to acquire audio data to be recognized and transmit it to the speech recognition device 200. The speech recognition device 200 is mainly used to perform speech recognition processing on the audio data transmitted by the speech acquisition device 100 to obtain the speech recognition result of the audio data.

[0057] It should be noted that, Figure 1 The schematic diagram of the speech recognition method shown is merely an example. The scenarios of the speech recognition method described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application.

[0058] Based on the above schematic diagram of the implementation scenario of the speech recognition method, a specific embodiment of the speech recognition method is proposed.

[0059] like Figure 2 As shown, Figure 2 This is a flowchart illustrating the steps of a speech recognition method provided in an embodiment of this application. The speech recognition method in this embodiment includes steps 201 to 203:

[0060] 201, Obtain the audio data to be recognized.

[0061] In this embodiment, the audio data acquired by the speech recognition device refers to the data that needs to be processed for speech recognition, collected by the audio acquisition device. Specifically, the audio data usually exists in the form of audio signals.

[0062] 202. Based on the audio data and the trained target acoustic neural network, determine the phoneme sequence corresponding to the audio data.

[0063] In this embodiment, the target acoustic neural network is trained based on unsupervised sample data and supervised sample data carrying text labels. Supervised sample data refers to sample data that has been pre-labeled with text labels. Taking actual telephone customer service business as an example, it is usually obtained by using open-source libraries and third-party companies to label business data, with a data volume typically in the hundreds or thousands. In contrast to supervised sample data, unsupervised sample data is usually obtained from business accumulation in actual business scenarios, generally having a base of tens of thousands or even larger. However, since unsupervised sample data does not carry any labels, it cannot be directly used to train the model.

[0064] To enable the use of large-scale, unlabeled unsupervised sample data for model training, this application further proposes a speech recognition training method that combines supervised and unsupervised methods, which can effectively utilize supervised and unsupervised data to train a network model with higher accuracy.

[0065] In this embodiment of the application, as can be seen from the aforementioned background information, a model for speech recognition generally includes an acoustic model for learning audio pronunciation features and a language model for learning grammatical rules. Considering that a phoneme is the smallest unit of speech obtained by analyzing the pronunciation actions within a syllable, and that the same pronunciation actions generally produce similar audio pronunciation features, it means that audio pronunciation features and phonemes can be associated through pronunciation actions. The target acoustic neural network utilizes the learned audio pronunciation features and their associated relationships to process the pronunciation features of audio data, thereby obtaining the phoneme sequence corresponding to the audio data. Specifically, phonemes are divided into two main categories: initials and finals. To facilitate understanding of the phoneme sequence proposed in this embodiment, a phoneme sequence is shown below: “ni3hao3zhong1guo2”, where the numbers 1, 2, and 3 after the phoneme represent tones.

[0066] Furthermore, in order to better utilize the target acoustic neural network to process audio data, the audio data is usually preprocessed before processing with the target acoustic neural network to further improve the accuracy of the identified phoneme sequences.

[0067] 203. Input the phoneme sequence into the trained language model for processing, and output the speech recognition result of the audio data.

[0068] In this embodiment of the application, as can be seen from the aforementioned background information, a language model can learn grammatical rules to make the translated speech text from phoneme sequences more accurate. Specifically, different texts may correspond to the same phoneme. For example, the texts "phoneme," "factor," and "sound speed" all correspond to the same phoneme "yin1su4." By processing the phoneme sequence using a trained language model, the phonemes can be restored to the correct corresponding text information by combining the context, thus serving as the speech recognition result of the audio data.

[0069] Specifically, language models are usually trained based on statistical probabilities. For example, the N-gram language model is quite common. Of course, it is also possible to use other methods, such as deep learning, to train a language model. In this case, the language model can be a recurrent neural network language model, a feedforward neural network language model, and so on.

[0070] To further improve the training effect of the language model trained in the embodiments of this application, as another optional embodiment of this application, the language model, like the acoustic neural network provided above, is also trained based on unsupervised sample data and supervised sample data carrying text labels.

[0071] The speech recognition method proposed in this application uses a large amount of unsupervised sample data and supervised sample data with text labels to train an acoustic neural network. This effectively compensates for the problem of insufficient estimation of speech feature space caused by insufficient supervised sample data, improves the recognition accuracy of the trained acoustic neural network, and further enhances the speech recognition effect of subsequent audio to be recognized using the acoustic neural network, thereby improving the accuracy of speech recognition.

[0072] like Figure 3 As shown, Figure 3 This application provides a flowchart illustrating the steps for training an acoustic neural network, specifically including steps 301 to 304:

[0073] 301. Obtain sample audio data and text labels corresponding to supervised audio data in the sample audio data.

[0074] In this embodiment, the sample audio data refers to the audio data used to train the neural network. This includes two types: unsupervised audio data without text labels, which is typically obtained through business accumulation in specific business scenarios; and supervised audio data with text labels, which is usually obtained by using open-source libraries and third-party companies to annotate business data. Specifically, the text labels corresponding to the supervised audio data can be understood as the speech recognition results of the audio segment annotated manually.

[0075] 302. An initial acoustic neural network and a first language model are obtained by training the supervised audio data and its corresponding text labels.

[0076] In this embodiment, based on deep learning, supervised audio data and its corresponding text labels are used for deep learning training to obtain an initial acoustic neural network and language model, namely, the initial acoustic neural network and the first language model. It should be noted that due to the inherent limitations in the amount of supervised audio data, the speech recognition performance of the initial acoustic neural network and the first language model trained in this way is not ideal and cannot meet the accuracy requirements of subsequent applications.

[0077] Furthermore, during actual training, supervised audio data and their corresponding text labels within the sample audio data can be used to train an alignment model that maps each label to a frame segment in the audio data. This model is then used in the subsequent training of the target acoustic neural network to map each frame segment in the audio data to its corresponding label. Of course, in addition to training the model, existing pre-trained templates can also be used directly.

[0078] 303. Determine the text label corresponding to the unsupervised audio data based on the initial acoustic neural network, the first language model, and the unsupervised audio data in the sample audio data.

[0079] In this embodiment, after training an initial acoustic neural network and a first language model based on a small amount of supervised audio data and its corresponding text labels, the initial acoustic neural network and the first language model can be used to process unsupervised audio data to obtain the text recognition result of the unsupervised audio data, that is, the text label corresponding to the unsupervised audio data.

[0080] 304. A target acoustic neural network is trained based on the supervised audio data and its corresponding text labels, as well as the unsupervised audio data and its corresponding text labels.

[0081] In this embodiment of the application, after obtaining unsupervised audio data and its corresponding text labels, the large data volume of unsupervised audio data can effectively compensate for the insufficient estimation of speech feature space caused by the insufficient volume of supervised sample data, thereby improving the performance of the target acoustic neural network finally trained.

[0082] Furthermore, it should be noted that since the text labels corresponding to unsupervised audio data are obtained based on the initial acoustic neural network and the first language model, and the initial acoustic neural network and the first language model have performance limitations, the text labels of the unsupervised audio data determined by the initial acoustic neural network and the first language model will contain certain errors. To reduce the impact of these errors on the training process of the target acoustic neural network, as a feasible embodiment of this application, an implementation scheme based on a multi-task network and labels at different levels is proposed to train the target acoustic neural network. Details are as follows.

[0083] like Figure 4 As shown, Figure 4 A flowchart illustrating the steps for obtaining an acoustic neural network based on multi-task training, as provided in this application embodiment, is described in detail below.

[0084] To reduce the impact of text label errors in unsupervised audio data on the training process of the target acoustic neural network, this application provides an implementation scheme for training the target acoustic neural network based on a multi-task network and labels at different levels, specifically including steps 401-404:

[0085] 401. The supervised audio data and its corresponding text tags are forcibly aligned according to a preset alignment model to obtain supervised state tags and their corresponding supervised audio segments.

[0086] In this embodiment of the application, as can be seen from the foregoing description, the alignment model can be used to align tags with frame segments in audio data. Therefore, by using a preset alignment model to perform forced alignment processing on supervised audio data and its corresponding text tags, supervised audio segments corresponding to each text tag can be obtained. On this basis, further, since text tags can be mapped into phoneme sequences through a dictionary, and based on the idea of ​​HMM (Hidden Markov Model), phonemes can be further subdivided into multiple different states. For example, a phoneme can be subdivided into three states: early stage [0], middle stage [1], and late stage [2]. For example, taking the aforementioned phoneme sequence (label): “ni3hao3zhong1guo2”, this phoneme label can be broken down into finer-level state labels: “n[0]n[1]n[2]i3[0]i3[2]i3[2]h[0]h[1]h[2]ao3[0]ao3[1]ao3[2]zh[0]zh[1]zh[2]ong1[0]ong1[1]ong1[2]g[0]g[1]g[2]uo2[0]uo2[1]uo2[2]”. In other words, the state labels can be regarded as finer-level phoneme labels. Finally, by using an alignment model to force alignment of the supervised audio data and its corresponding text labels, we can further subdivide the supervised state labels and their corresponding supervised audio segments.

[0087] 402. According to the alignment model, the unsupervised audio data and its corresponding text labels are subjected to forced alignment processing to obtain unsupervised state labels and their corresponding unsupervised audio segments.

[0088] In this embodiment of the application, similar to the forced alignment process performed on supervised audio data and its corresponding text labels in step 401 above, the forced alignment process performed on unsupervised audio data and its corresponding text labels according to the alignment model can also obtain subdivided unsupervised state labels and their corresponding unsupervised audio segments.

[0089] It should be noted that since supervised state labels and their corresponding supervised audio segments are obtained by processing supervised audio data and manually annotated text labels, and manually annotated text labels have a relatively high accuracy rate, the label accuracy of supervised state labels and their corresponding supervised audio segments will be higher. On the other hand, unsupervised state labels and their corresponding supervised audio segments are obtained by processing unsupervised audio data using text labels, so the label accuracy of unsupervised state labels and their corresponding unsupervised audio segments will be relatively lower, but the amount of data is larger.

[0090] 403. The preset initial multi-task neural network is trained based on the supervised state labels and their corresponding supervised audio segments, as well as the unsupervised state labels and their corresponding unsupervised audio segments, to obtain the target multi-task neural network.

[0091] In this embodiment, after determining the state labels corresponding to each supervised / unsupervised audio segment, a neural network model can be trained using deep learning to obtain the association between the features of each audio segment and its corresponding state. The resulting trained neural network model is the target acoustic neural network. Specifically, as described above, supervised state labels and their corresponding supervised audio segments have higher label accuracy, while unsupervised state labels and their corresponding unsupervised audio segments have relatively lower accuracy. Therefore, as a feasible embodiment of this application, it is possible to use unsupervised state labels and their corresponding unsupervised audio segments, which have a larger data volume, to train the model's pronunciation feature extraction capability, while using supervised state labels and their corresponding supervised audio segments, which have higher accuracy, to train the model's classification capability. That is, to better integrate the characteristics of the two training samples, a multi-task neural network approach can be used for training.

[0092] Specifically, the multi-task neural network mainly consists of an input layer, an intermediate layer, and an output layer. The output layer is a dual-output layer structure for multi-task applications, including a first output layer and a second output layer. The first output layer uses state classification as a node to construct a classification task using supervised audio segments and their state labels, while the second output layer uses phoneme classification as a node to construct a classification task using unsupervised data and its phoneme labels. The two task output layers share the intermediate layer.

[0093] Furthermore, since unsupervised state labels and their corresponding unsupervised audio segments are primarily used to train the model's pronunciation feature extraction capabilities, they can be further converted into coarser phoneme labels to reduce frame-based labeling errors and achieve better training results. Specific implementation details can be found in subsequent sections. Figure 6 And its explanations and descriptions.

[0094] 404. Generate a target acoustic neural network based on the input layer, intermediate layer and first output layer of the target multi-task neural network.

[0095] In this embodiment, since the classification accuracy of phoneme labels in unsupervised audio data is lower than that of state labels in supervised audio segments, their main function is to train and learn the hidden layer parameters through coarse classification, thereby capturing the broader and more universal phoneme pronunciation features contained in the unsupervised audio segments. Supervised audio segments, on the other hand, share hidden layer parameters, and can be corrected using more accurate state labels based on parameters obtained from the larger statistically significant unsupervised data. Therefore, the input layer, intermediate layer, and first output layer of the final trained target multi-task neural network constitute the target acoustic neural network capable of accurately achieving phoneme recognition.

[0096] To better understand the model structure of the multi-task neural network provided in the embodiments of this application, as follows: Figure 5 As shown, Figure 5 This is a schematic diagram of a network model structure for a multi-task neural network provided in an embodiment of this application.

[0097] In this embodiment, the input layer is an audio segment, and the intermediate layer contains several hidden layers. The hidden layers can be constructed using commonly used and proven effective speech recognition structures, such as TDNN and LSTM. In actual training, firstly, the unsupervised audio segment is passed through a shared hidden layer and a second output layer to obtain a feedforward output, and the gradient is calculated to update the hidden layer parameters in reverse. Secondly, the supervised audio segment is passed through the updated shared hidden layer and the first output layer to obtain a feedforward output, and the gradient is calculated to update the hidden layer parameters in reverse. This process is repeated iteratively until both classification tasks converge, at which point training ends.

[0098] like Figure 6 As shown, Figure 6 The following is a detailed flowchart illustrating the steps of reverse mapping a status label to obtain a phoneme label, as provided in an embodiment of this application.

[0099] In this embodiment of the application, a scheme is proposed to convert unsupervised state labels and their corresponding unsupervised audio segments into coarser phoneme labels to reduce frame-based labeling errors and thus achieve better training results. Specifically, it includes steps 601 to 602:

[0100] 601. Merge the unsupervised state label and its corresponding unsupervised audio segment according to the unsupervised state label to obtain the unsupervised phoneme label and its corresponding unsupervised audio segment.

[0101] In this embodiment of the application, as can be seen from the foregoing description, the state label can be understood as a finer-level phoneme label. That is, a phoneme label can be divided into several finer-grained state labels. Therefore, by using unsupervised state labels, unsupervised state labels and their corresponding unsupervised audio segments can be reversed and merged to obtain unsupervised phoneme labels and their corresponding unsupervised audio segments.

[0102] 602. The preset initial multi-task neural network is trained based on the supervised state labels and their corresponding supervised audio segments, as well as the unsupervised phoneme labels and their corresponding unsupervised audio segments, to obtain the target multi-task neural network.

[0103] In this embodiment, similar to the process of training a multi-task neural network using supervised state labels and their corresponding supervised audio segments, and unsupervised state labels and their corresponding unsupervised audio segments, training a multi-task neural network can also be achieved using unsupervised phoneme labels and their corresponding unsupervised audio segments. Furthermore, by using coarser-grained phoneme labels instead of fine-grained state labels, mislabeling is reduced, improving the model's ability to extract pronunciation features, while not affecting the model's classification ability when trained with supervised data. In other words, the technical solution provided in this embodiment ensures the model's classification ability while improving its ability to extract pronunciation features.

[0104] like Figure 7 As shown, Figure 7 A flowchart illustrating the steps for training a language model, as provided in this embodiment of the application, is described in detail below.

[0105] To improve the performance of the trained language model, this application provides an implementation scheme for training a language model based on unsupervised sample data and supervised sample data carrying text labels, specifically including steps 701-702:

[0106] 701. A second language model is trained based on the text recognition results corresponding to the unsupervised audio data.

[0107] In this embodiment, the text recognition result corresponding to the unsupervised audio data is obtained based on the aforementioned step 601, that is, by processing the unsupervised audio data using the initial acoustic neural network and language model trained with supervised audio data and its text labels. Furthermore, after determining the unsupervised audio data and its corresponding text recognition result, a second language model can be trained using the text recognition result corresponding to the unsupervised audio data, such as an N-gram language model, a recurrent neural network language model, a feedforward neural network language model, etc., based on statistical probability or deep learning methods.

[0108] Furthermore, as an optional embodiment of this application, taking the N-gram language model as an example, during the process of processing unsupervised audio data using the initial acoustic neural network and language model to obtain text recognition results, the confidence level of the text recognition results is also output simultaneously. The confidence level represents the credibility of the text recognition result; a higher confidence level indicates higher accuracy. Therefore, to further improve the performance of the second language model trained using unsupervised audio data and its corresponding text recognition results, at the initial stage of training, reliable audio data with sufficient length and confidence level is selected from the unsupervised audio data using the confidence level and the length of the text recognition results for subsequent language model training.

[0109] 702. Perform model fusion processing on the preset first language model and the second language model to obtain the trained language model.

[0110] In this embodiment of the application, the first language model is obtained based on the aforementioned step 302, that is, it is trained using supervised audio data and its corresponding text labels.

[0111] In this embodiment, as described above, the first language model is trained based on supervised audio data and its corresponding text labels, while the second language model is trained based on unsupervised audio data and its corresponding text recognition results. Therefore, the second language model can be fine-tuned based on the first language model to fuse the features of both models. Specifically, the fusion process can involve interpolating and pruning the first and second models according to a certain ratio, or it can involve selecting word group probabilities according to certain principles and probabilities.

[0112] like Figure 8 As shown, Figure 8 A flowchart illustrating the steps for filtering unsupervised audio data, as provided in this application embodiment, is described in detail below.

[0113] In this embodiment of the application, a method is provided for filtering unsupervised audio data to select reliable audio data with sufficient length and confidence for subsequent language model training, specifically including steps 801-802:

[0114] 801. Based on the text length and confidence level of the text recognition result corresponding to the unsupervised audio data, target audio data is selected from the unsupervised audio data.

[0115] In this embodiment, considering that short text recognition results are generally modal particles and are usually not representative, and that text recognition results with low confidence also indicate low reliability and lack of representativeness, we can select representative audio data with text lengths exceeding a preset length threshold and confidence levels exceeding a preset confidence threshold from the text recognition results corresponding to unsupervised audio data. The preset length threshold and preset confidence threshold can be set according to actual needs; for example, the preset length threshold can be 8, and the preset confidence threshold can be 80%.

[0116] 802, A second language model is trained based on the text recognition results corresponding to the target audio data.

[0117] In this embodiment of the application, as can be seen from the foregoing description, the target audio data is typical and representative audio data selected based on the text length and confidence of the text recognition results. Therefore, by using the text recognition results corresponding to the target audio data, a second language model with better performance can be trained compared to using the text recognition results corresponding to all unsupervised audio data.

[0118] like Figure 9 As shown, Figure 9 A flowchart illustrating the steps for preprocessing audio data to determine a phoneme sequence, as provided in this application embodiment, specifically includes steps 901 to 903:

[0119] 901, The audio data is enhanced to obtain enhanced audio data.

[0120] In this embodiment, the enhancement processing of audio data mainly involves noise reduction processing of the audio signal, that is, removing the noise that may exist in the audio data and improving the signal-to-noise ratio. There are many specific implementation schemes for enhancing audio data, such as wavelet-based noise reduction methods, smoothing filter-based noise reduction methods, etc.

[0121] 902, Extract the audio features of the enhanced audio data.

[0122] In this embodiment, the audio features of the enhanced audio data are mainly obtained by performing a Fourier transform on the audio signal to obtain the feature information of the audio signal in the time domain and frequency domain. These feature information can all be regarded as the audio features of the enhanced audio data.

[0123] 903, The audio features are input into the trained target acoustic neural network to obtain the phoneme sequence corresponding to the audio data.

[0124] In this embodiment of the application, the audio features extracted from the audio data are input into the trained target acoustic neural network. The target acoustic neural network can then determine the phoneme sequence corresponding to the audio data based on the learned correlation between the audio pronunciation features and phonemes.

[0125] To better understand the training model provided in the embodiments of this application and the complete process of using the trained model to perform speech recognition on audio data, the following will combine... Figures 2-9 The provided embodiments illustrate a complete flowchart of model training and model application. Specifically, as shown... Figure 10 As shown, Figure 10 The complete implementation process of a speech recognition model training and application provided in this application embodiment is detailed below.

[0126] 1) Based on existing open-source data and labeled business data containing text labels, supervised audio data is used for conventional model training to obtain an alignment model and an initial acoustic neural network. This includes data augmentation, feature extraction, alignment model training, and acoustic neural network training. Based on existing open-source data and labeled business data containing text labels, a first language model based on statistical probability is trained.

[0127] 2) For unsupervised audio data such as telephone channel audio service data obtained from business accumulation, use the initial acoustic neural network and first language model obtained in step 1) to decode the data, obtain the text recognition result of the unsupervised audio data, and map it to phoneme labels according to the preset dictionary.

[0128] 3) Using the alignment model obtained in step 1), force alignment is performed on the unsupervised audio data and its corresponding phoneme labels to determine the unsupervised audio segments and their corresponding phoneme labels.

[0129] 4) Further subdivide the supervised audio data and its corresponding phoneme labels to obtain state labels, and use the alignment model obtained in step 1) to force alignment of the supervised audio data and its corresponding state labels, thereby determining the supervised audio segments and their corresponding state labels.

[0130] 5) Construct a multi-task neural network. The input layer consists of supervised / unsupervised audio segments, and the intermediate layers contain several hidden layers. The hidden layers can be constructed using proven structures commonly used in speech recognition, such as TDNN and LSTM. The output layer is a dual-output layer structure for multiple tasks. The first output layer uses state classification as nodes, constructing a classification task using supervised audio segments and their state labels. The second output layer uses phoneme classification as nodes, constructing a classification task using unsupervised audio segments and their phoneme labels. The two tasks share hidden layers. In actual training, firstly, the unsupervised audio segments are fed forward through the shared hidden layer and the second output layer to obtain feedforward output, and gradients are calculated to update the hidden layer parameters in reverse. Secondly, the supervised audio segments are fed forward through the updated shared hidden layer and the first output layer to obtain feedforward output, and gradients are calculated to update the hidden layer parameters in reverse. This process is repeated iteratively until both classification tasks converge, training ends, and the input layer, intermediate layers, and first output layer are extracted to obtain the trained target acoustic neural network.

[0131] 6) Use the text recognition results of the unsupervised audio data obtained in step 2) as the text corpus, and select the text corpus with confidence scores higher than the preset confidence threshold and text lengths greater than the preset length threshold. Train the second language model based on statistical probability, and then perform interpolation pruning on the first and second models according to a certain ratio, or select the word probability according to certain principles and probability values, thereby completing the fusion of the first and second models and obtaining the final trained language model.

[0132] Subsequently, when it is necessary to perform text recognition on the acquired audio data, the acquired audio data is enhanced and preprocessed with features extraction, and the extracted audio features are input into the target acoustic neural network trained in step 5). After processing, the resulting phoneme sequence is further input into the language model trained in step 6) to obtain the final text recognition result.

[0133] To better implement the speech recognition method provided in the embodiments of this application, a speech recognition device is also provided in the embodiments of this application, based on the speech recognition method. For example... Figure 11 As shown, Figure 11 This is a schematic diagram of the structure of a speech recognition device provided in an embodiment of this application. Specifically, the speech recognition device includes:

[0134] The acquisition module 1101 is used to acquire the audio data to be recognized;

[0135] The phoneme determination module 1102 is used to determine the phoneme sequence corresponding to the audio data based on the audio data and the trained target acoustic neural network; the target acoustic neural network is trained based on unsupervised sample data and supervised sample data carrying speech labels.

[0136] The speech recognition module 1103 is used to input the phoneme sequence into a trained language model for processing and output the speech recognition result of the audio data.

[0137] In some embodiments of this application, the speech recognition device includes an acoustic model training module, which includes:

[0138] The sample acquisition sub-module is used to acquire sample audio data and text labels corresponding to supervised audio data in the sample audio data.

[0139] The initial model training submodule is used to train an initial acoustic neural network and a first language model based on the supervised audio data and its corresponding text labels.

[0140] The text label determination submodule is used to determine the text label corresponding to the unsupervised audio data based on the initial acoustic neural network, the first language model, and the unsupervised audio data in the sample audio data.

[0141] The target model training submodule is used to train a target acoustic neural network based on the supervised audio data and its corresponding text labels, as well as the unsupervised audio data and its corresponding text labels.

[0142] In some embodiments of this application, the target model training sub-module includes:

[0143] The first alignment unit is used to perform forced alignment processing on the supervised audio data and its corresponding text tags according to a preset alignment model to obtain supervised state tags and their corresponding supervised audio segments.

[0144] The second alignment unit is used to perform forced alignment processing on the unsupervised audio data and its corresponding text labels according to the alignment model, so as to obtain unsupervised state labels and their corresponding unsupervised audio segments.

[0145] A multi-task training unit is used to train a preset initial multi-task neural network based on the supervised state labels and their corresponding supervised audio segments, as well as the unsupervised state labels and their corresponding unsupervised audio segments, to obtain a target multi-task neural network.

[0146] The model generation unit is used to generate a target acoustic neural network based on the input layer, intermediate layer and first output layer of the target multi-task neural network.

[0147] In some embodiments of this application, the multi-task training unit includes:

[0148] The reverse mapping subunit is used to merge the unsupervised state label and its corresponding unsupervised audio segment according to the unsupervised state label to obtain the unsupervised phoneme label and its corresponding unsupervised audio segment.

[0149] The multi-task training subunit is used to train a preset initial multi-task neural network based on the supervised state labels and their corresponding supervised audio segments, as well as the unsupervised phoneme labels and their corresponding unsupervised audio segments, to obtain a target multi-task neural network.

[0150] In some embodiments of this application, the speech recognition device includes a language model training module, which includes:

[0151] The second model training module is used to train a second language model based on the text recognition results corresponding to the unsupervised audio data.

[0152] The model fusion submodule performs model fusion processing on the preset first language model and the second language model to obtain a trained language model; the first language model is trained using text labels corresponding to supervised audio data.

[0153] In some embodiments of this application, the second model training sub-module includes:

[0154] The filtering unit is used to filter out target audio data from the unsupervised audio data based on the text length and confidence level of the text recognition result corresponding to the unsupervised audio data;

[0155] The second model training unit is used to train a second language model based on the text recognition results corresponding to the target audio data.

[0156] In some embodiments of this application, the phoneme determination module includes:

[0157] The enhancement module is used to enhance the audio data to obtain enhanced audio data;

[0158] The feature extraction submodule is used to extract the audio features of the enhanced audio data;

[0159] The model processing submodule is used to input the audio features into the trained target acoustic neural network to obtain the phoneme sequence corresponding to the audio data.

[0160] This application also provides a voice recognition device, such as... Figure 12 As shown, Figure 12 This is a schematic diagram of the structure of a voice recognition device provided in an embodiment of this application.

[0161] The speech recognition device includes a memory, a processor, and a speech recognition program stored in the memory and executable on the processor. When the processor executes the speech recognition program, it implements the steps of the speech recognition method provided in any embodiment of this application.

[0162] Specifically, a voice recognition device may include components such as a processor 1201 with one or more processing cores, a memory 1202 with one or more storage media, a power supply 1203, and an input unit 1204. Those skilled in the art will understand that... Figure 12 The structure of the speech recognition device shown does not constitute a limitation on the speech recognition device. It may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0163] The processor 1201 is the control center of the speech recognition device. It connects various parts of the device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 1202, and by calling data stored in the memory 1202, thereby providing overall monitoring of the speech recognition device. Optionally, the processor 1201 may include one or more processing cores; preferably, the processor 1201 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 1201.

[0164] The memory 1202 can be used to store software programs and modules. The processor 1201 executes various functional applications and data processing by running the software programs and modules stored in the memory 1202. The memory 1202 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the voice recognition device, etc. In addition, the memory 1202 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 1202 may also include a memory controller to provide the processor 1201 with access to the memory 1202.

[0165] The voice recognition device also includes a power supply 1203 that supplies power to the various components. Preferably, the power supply 1203 can be logically connected to the processor 1201 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 1203 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0166] The voice recognition device may also include an input unit 1204, which can be used to receive input numeric or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0167] Although not shown, the voice recognition device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 1201 in the voice recognition device loads the executable files corresponding to the processes of one or more applications into the memory 1202 according to the following instructions, and the processor 1201 runs the applications stored in the memory 1202, thereby implementing the steps in the voice recognition method provided in any embodiment of this application.

[0168] Therefore, embodiments of this application provide a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc. The computer-readable storage medium stores a speech recognition program, which, when executed by a processor, implements the steps of the speech recognition method provided in any embodiment of this application.

[0169] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed descriptions of other embodiments above, which will not be repeated here.

[0170] In practice, each of the above units or structures can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units or structures, please refer to the previous method embodiments, which will not be repeated here.

[0171] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0172] The above provides a detailed description of a speech recognition method provided by the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A speech recognition method, characterized in that, include: Obtain the audio data to be recognized; Obtain sample audio data and text labels corresponding to supervised audio data within the sample audio data; An initial acoustic neural network and a first language model are obtained by training the supervised audio data and its corresponding text labels. The text labels corresponding to the unsupervised audio data are determined based on the initial acoustic neural network, the first language model, and the unsupervised audio data in the sample audio data. The supervised audio data and its corresponding text tags are forcibly aligned according to a preset alignment model to obtain supervised state tags and their corresponding supervised audio segments. The unsupervised audio data and its corresponding text labels are forcibly aligned according to the alignment model to obtain unsupervised state labels and their corresponding unsupervised audio segments. The preset initial multi-task neural network is trained based on the supervised state labels and their corresponding supervised audio segments, as well as the unsupervised state labels and their corresponding unsupervised audio segments, to obtain the target multi-task neural network. A target acoustic neural network is generated based on the input layer, intermediate layer, and first output layer of the target multi-task neural network. Based on the audio data to be identified and the trained target acoustic neural network, the phoneme sequence corresponding to the audio data to be identified is determined; the target acoustic neural network is trained based on unsupervised sample data and supervised sample data carrying text labels. The phoneme sequence is input into a trained language model for processing, and the speech recognition result of the audio data to be recognized is output.

2. The speech recognition method according to claim 1, characterized in that, The step of training a preset initial multi-task neural network based on the supervised state labels and their corresponding supervised audio segments, and the unsupervised state labels and their corresponding unsupervised audio segments, to obtain a target multi-task neural network, includes: The unsupervised state labels and their corresponding unsupervised audio segments are merged according to the unsupervised state labels to obtain unsupervised phoneme labels and their corresponding unsupervised audio segments. The target multi-task neural network is obtained by training a preset initial multi-task neural network based on the supervised state labels and their corresponding supervised audio segments, as well as the unsupervised phoneme labels and their corresponding unsupervised audio segments.

3. The speech recognition method according to claim 1, characterized in that, Before inputting the phoneme sequence into a trained language model for processing and outputting the speech recognition result of the audio data to be recognized, the method further includes: A second language model is trained based on the text recognition results corresponding to unsupervised audio data. The preset first language model and the second language model are fused to obtain a trained language model; the first language model is trained using text labels corresponding to supervised audio data.

4. The speech recognition method according to claim 3, characterized in that, The process of training a second language model based on the text recognition results corresponding to unsupervised audio data includes: Based on the text length and confidence level of the text recognition results corresponding to the unsupervised audio data, target audio data is filtered out from the unsupervised audio data; A second language model is trained based on the text recognition results corresponding to the target audio data.

5. The speech recognition method according to any one of claims 1 to 4, characterized in that, The step of determining the phoneme sequence corresponding to the audio data to be identified based on the audio data to be identified and the trained target acoustic neural network includes: The audio data to be identified is enhanced to obtain enhanced audio data; Extract the audio features from the enhanced audio data; The audio features are input into a trained target acoustic neural network to obtain the phoneme sequence corresponding to the audio data to be identified.

6. A voice recognition device, characterized in that, include: The acquisition module is used to acquire the audio data to be recognized; Acoustic model training module, the acoustic model training module includes: The sample acquisition sub-module is used to acquire sample audio data and text labels corresponding to supervised audio data in the sample audio data. The initial model training submodule is used to train an initial acoustic neural network and a first language model based on the supervised audio data and its corresponding text labels. The text label determination submodule is used to determine the text label corresponding to the unsupervised audio data based on the initial acoustic neural network, the first language model, and the unsupervised audio data in the sample audio data. The first alignment unit is used to perform forced alignment processing on the supervised audio data and its corresponding text tags according to a preset alignment model to obtain supervised state tags and their corresponding supervised audio segments. The second alignment unit is used to perform forced alignment processing on the unsupervised audio data and its corresponding text labels according to the alignment model, so as to obtain unsupervised state labels and their corresponding unsupervised audio segments. A multi-task training unit is used to train a preset initial multi-task neural network based on the supervised state labels and their corresponding supervised audio segments, as well as the unsupervised state labels and their corresponding unsupervised audio segments, to obtain a target multi-task neural network. The model generation unit is used to generate a target acoustic neural network based on the input layer, intermediate layer and first output layer of the target multi-task neural network. A phoneme determination module is used to determine the phoneme sequence corresponding to the audio data to be identified based on the audio data to be identified and the trained target acoustic neural network; the target acoustic neural network is trained based on unsupervised sample data and supervised sample data carrying speech tags. The speech recognition module is used to input the phoneme sequence into a trained language model for processing and output the speech recognition result of the audio data to be recognized.

7. A voice recognition device, characterized in that, The speech recognition device includes a processor, a memory, and a speech recognition program stored in the memory and executable on the processor. The processor executes the speech recognition program to implement the steps of the speech recognition method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a speech recognition program, which is executed by a processor to implement the steps of the speech recognition method according to any one of claims 1 to 5.

Images