Training data screening method, system, device and medium

Abstract

The present invention relates to the field of speech recognition processing technology, and is particularly suitable for obtaining training data for machine learning models used in speech recognition transcription. In view of the high cost of data acquisition, high consumption, poor data quality / accuracy, and poor screening effect of existing pseudo-label data in different scenarios, a method, system, device, and medium for screening training data of the present invention are proposed, aiming to solve the technical problem of how to screen high-quality training data for models such as speech recognition, search, and transcription based on the accuracy of pseudo-labels of semi-supervised learning. To this end, the method of the present invention uses the mean number of node links of the decoding path in decoding to sort the decoding results to screen the pseudo-label speech data with the highest sorting as model training data. The screening efficiency and data quality are improved, and the cost and consumption are reduced.

Description

Technical Field

[0001] The present invention belongs to the field of speech recognition processing technology, and is particularly suitable for obtaining training data for a machine learning model used in speech recognition transcription. More specifically, it relates to a method, system, device and medium for screening training data. Background Art

[0002] In scenarios such as intelligent voice interaction and human-computer interaction, voice recognition, voice conversion, voice recognition search and other processing can be achieved through machine learning, such as various voice / audio recognition models. As voice recognition technology becomes more and more mature, voice recognition technology trained with large amounts of data can already surpass humans in certain specific scenarios. However, the training of voice models requires a large amount of manually annotated data to improve the model's predictive recognition performance (precision and accuracy), that is, supervised learning, to better achieve voice recognition and interaction; if there is not enough data, no matter how good the model structure is, it cannot learn enough "knowledge". Although the data with manual labeling is of high quality, the price is often high, and the cost is high and the efficiency is low. Therefore, it is more desirable to use unlabeled data to enhance the performance of the model. Using unlabeled data in existing data to train a model is called semi-supervised learning. For example, the existing voice model is directly used to recognize audio data to obtain the "pseudo-label" of audio data, that is, to label the audio data with some pseudo-labels, and then add relatively good quality "pseudo-labeled" data to the labeled data, and continue to train the model again.

[0003] However, since it is uncertain whether the quality of the pseudo-labeled audio data obtained meets the training data requirements, the accuracy of the pseudo-labeled data, that is, its data recognition / transcription accuracy, will be determined according to certain indicators, and then the pseudo-labeled data with higher accuracy will be screened out based on the accuracy of the judgment. The high-accuracy pseudo-labeled data will be added to the training data and provided to the speech recognition model for further model training to ensure that the model recognition efficiency and recognition effect are improved.

[0004] The existing common training data screening schemes are: one is to screen according to sentence level confidence, that is, to calculate sentence level confidence; the other is to use MBR (Minimum Bayes Risk) to calculate word level confidence.

[0005] The former solution is often used in online systems, and the difference in the total cost of the optimal and suboptimal paths of each sentence lattice is used to measure the sentence confidence.

[0006] sentence_confidence=1-exp(-(best_cost-second_best_cost))

[0007] The confidence calculated in this way is rough and inaccurate, and cannot reflect the actual quality of sentence recognition. This is especially true when encountering long sentences, as the language model does not score long sequences well enough, making this standard completely invalid.

[0008] The latter scheme uses a decoding algorithm to decode, and calculates the word-level confidence based on the MBR value of the decoding result, and uses it as a condition for filtering data. The disadvantage of filtering data based on the MBR value is that the calculation of the MBR value is based on the result obtained from the entire decoding step including the language model, which will be affected by the acoustic-scale and LM rescore. Different decoding strategies will obtain different results. Therefore, the strategy of filtering based on indicators such as MBR value is not stable. For specific ideas on MBR indicator filtering, please refer to "Xu H, Povey D, Mangu L, et al. Minimum bayes risk decoding and system combination based on a recursion for edit distance [J]. Computer Speech & Language, 2011, 25 (4): 802-828.".

[0009] Therefore, it is necessary to improve the existing method of screening pseudo-label data so that the screening is less subject to interference, the screening is stable, the efficiency is high, and the accuracy of the pseudo-label data can be determined more accurately and quickly, thereby efficiently screening out valuable and high-quality pseudo-label data and obtaining more and better training data. Summary of the invention

[0010] 1. Technical issues to be resolved

[0011] The present invention aims to solve the technical problem of how to screen training data based on the accuracy of pseudo labels in semi-supervised learning; further, to solve the technical problem of how to effectively utilize / screen existing data for semi-supervised training to improve the recognition effect of intelligent speech models in the K12 scenario AI teaching system; further, to solve the technical problem of how to screen out high-quality training data for speech models in different scenarios.

[0012] (II) Technical solution

[0013] To solve the above technical problems, the first aspect of the present invention proposes a method for screening training data, including: taking the average number of nodes that each node of each decoding path in all decoding paths during decoding of speech recognition as an indicator for judging the accuracy; labeling pseudo-labels for audio data corresponding to decoding results with high accuracy, and using the audio data with pseudo-labels as selected training data.

[0014] According to a preferred embodiment of the present invention, all decoding paths of speech recognition during decoding specifically include: using the speech recognition model of the Kaldi toolkit to perform speech recognition on input unlabeled audio data; performing beam search decoding according to a predetermined beam width beamsize during the speech recognition process; and storing all expected decoding paths that have been searched during decoding.

[0015] According to a preferred embodiment of the present invention, all expected decoding paths that have been searched during decoding are stored, specifically including: performing a decoding path search from the state network WFST during decoding to obtain a decoding path graph; using a lattice to store all decoding paths that have been searched in the decoding path graph; and outputting the optimal decoding path as a decoding result corresponding to the audio data.

[0016] According to a preferred embodiment of the present invention, the average number of nodes that each node of each decoding path in all decoding paths is expected to link to is used as an indicator for judging the accuracy, specifically including: from all decoding paths stored in lattice, obtaining the number of paths that each node in each decoding path can link backward and calculating the average; using the average as an indicator latticedepth for judging the accuracy; and sorting and comparing the corresponding multiple indicators latticedepth obtained when decoding multiple audio data in the data set through speech recognition to judge the accuracy.

[0017] According to a preferred embodiment of the present invention, the corresponding multiple indicators latticedepth obtained when decoding multiple audio data in a data set through speech recognition are sorted and compared to determine the accuracy rate, specifically including: sorting the indicator latticedepth corresponding to each of the multiple audio data and comparing them with a preset threshold; if the ranking of the indicator latticedepth is below the preset threshold, it means that the decoding result of the corresponding audio data has a higher accuracy rate.

[0018] According to a preferred embodiment of the present invention, pseudo-labeling is performed for audio data corresponding to a decoding result with a high accuracy, specifically comprising: obtaining unlabeled audio data corresponding to a decoding result with a high accuracy; and annotating the audio data with pseudo-labels according to text labels in the decoding result output by speech recognition.

[0019] According to a preferred embodiment of the present invention, audio data with pseudo labels are used as selected training data, specifically including: screening out unlabeled audio data corresponding to all decoding results with high accuracy; after the audio data are labeled with pseudo labels, adding the audio data with the pseudo labels to the model training set of speech recognition, and retraining the speech recognition model.

[0020] According to a preferred embodiment of the present invention, it also includes: based on a semi-supervised learning method, a speech recognition model trained by a small amount of manually annotated labels, or a speech recognition model initialized with text labels, directly inputting multiple unlabeled audio data in a data set into the model for speech recognition, and outputting corresponding multiple decoding results, so that when one or more decoding results with high accuracy are screened out, the unlabeled audio data corresponding to the decoding results are directly labeled with the text labels in the decoding results.

[0021] To solve the above technical problems, the second aspect of the present invention proposes a training data screening system, including: setting an indicator module, used to use the average number of nodes that each node in the decoding path of speech recognition during decoding as an indicator for judging the accuracy rate; a data screening module, used to annotate pseudo-labels for audio data corresponding to decoding results with high accuracy so as to select the audio data with pseudo-labels as training data.

[0022] To solve the above technical problems, the third aspect of the present invention proposes an electronic device, comprising a processor and a memory, wherein the memory is used to store a computer executable program, and when the computer program is executed by the processor, the processor executes the method described in any one of the first aspects.

[0023] In order to solve the above technical problems, the fourth aspect of the present invention proposes a computer-readable medium storing a computer executable program, and when the computer executable program is executed, it implements the method described in any one of the first aspects.

[0024] In order to solve the above technical problem, the fifth aspect of the present invention proposes a computer executable program, which, when executed, implements the method described in any one of the first aspects.

[0025] (III) Beneficial effects

[0026] The present invention aims to solve the above-mentioned technical problem of how to screen high-quality training data for models such as speech recognition, search, and transcription based on the accuracy of pseudo-labels of semi-supervised learning, and further, how to use (screen) existing transcription data for semi-supervised training to improve model recognition effect when the training data is insufficiently labeled in the intelligent speech recognition of the K12 scenario AI teaching system. By sorting the decoding results by the mean number of node links in the decoding path during decoding, a new screening indicator is adopted to judge the accuracy of pseudo-labels, and the pseudo-label speech data with the highest ranking is screened as model training data, thereby improving the screening efficiency and data quality, reducing costs and consumption, and performing model training based on high-quality training data, thereby improving the model training efficiency and model recognition effect.

[0027] Furthermore, by determining the lattice depth index through the nodes of the decoding graph obtained by the decoding path search process in the state network of the speech recognition model, such as the Kaldi speech recognition tool itself, the lattice depth index can be used as a screening index for the reflux data without basically increasing the computing power consumption, thereby efficiently and low-cost screening of high-quality unlabeled data (such as unlabeled audio data) to retrain the model and enhance the model effect. It does not introduce other parameters, classes, etc., and has the advantages of being simple and easy to use, high-quality screened audio, and low noise.

[0028] Furthermore, the index screening method adopted in the embodiment of the present invention can be used in any speech recognition scenario. As long as the speech recognition decoder uses FST, the decoding graph can be obtained. This is a relatively general screening solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a main flow chart of an embodiment of a method for screening training data according to the present invention;

[0030] Figure 2 is a main structural block diagram of an embodiment of a system for screening training data according to the present invention;

[0031] Figure 3 is a main structural block diagram of an embodiment of an electronic device according to the present invention;

[0032] Figure 4 is a schematic diagram of the main structure of an embodiment of a more specific electronic device according to the present invention;

[0033] Figure 5 It is a main structural diagram of an embodiment of a computer-readable medium according to the present invention. DETAILED DESCRIPTION

[0034] In the process of introducing specific embodiments, the detailed description of the structure, performance, effect or other characteristics is to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can implement the present invention with a technical solution that does not contain the above-mentioned structure, performance, effect or other characteristics under certain circumstances.

[0035] The flowchart in the accompanying drawings is only an exemplary process demonstration, and does not mean that the solution of the present invention must include all the contents, operations and steps in the flowchart, nor does it mean that it must be executed in the order shown in the figure. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. The execution order shown in the flowchart can be changed according to actual conditions without departing from the main purpose of the present invention.

[0036] Frames in the attached figure Figure 1 Generally, it refers to functional entities, which do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processing unit devices and / or microcontroller devices.

[0037] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be called the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0038] The following is an explanation of some technical terms that may be involved in the relevant content of the present invention:

[0039] ASR: Automatic Speech Recognition (ASR) Speech recognition is an interdisciplinary subfield of computer science and computational linguistics that develops methods and techniques that enable computers to recognize and translate spoken language into text.

[0040] Kaldi: refers to the speech recognition open source toolkit that originated from the summer workshop at Johns Hopkins University in 2009. Kaldi is one of the most popular speech technology toolkits in recent years.

[0041] FA: A finite automaton (FA) is a set of finite states and state transitions, each of which has at least one label. The most basic FA is a finite state acceptor (FSA). For a given input sequence, FSA returns two states: "accept" or "not accept".

[0042] FST: Finite State Transducers (FST) is an extension of FSA. Each state transition has an output label, called an input-output label pair. Through label pairs, FST can describe the conversion of a set of rules or the conversion of one set of symbol sequences to another set of symbol sequences.

[0043] Lattice: Lattice, a finite state transducer (FST), its input and output can be any FST label (usually transition id and word), weights include acoustic model, language model, conversion weight. WFST is a state network formed by the decoding graph or decoding recognition search path based on FST, and finds the best or most matching sound path (such as sentence, word, etc.) through search.

[0044] CER: Character Error Rate (CER), the recognition result of the audio, the edit distance relative to the actual annotation of the audio, divided by the number of actual annotated characters to get the percentage value.

[0045] acoustic-scale: The scaling factor of the acoustic model log probability score. In Kaldi's speech recognition, the acoustic model and language model can be used together to recognize audio to ensure that the output of the audio sound is more in line with the habits and logic of human speech. This scaling factor is a coefficient for scaling the probability output of the acoustic model in order to balance the acoustic and language models.

[0046] LM rescore: language model re-scoring. After obtaining a speech recognition model, if you want the model to better recognize data in a certain field, or want to combine the advantages of different language models (such as NGRAM+RNNLM) to enhance the model effect, you will often use language models trained with data from different fields or different model structures to re-score the recognition results, and use the re-scoring results as the final results.

[0047] One embodiment of the present invention mainly realizes training data acquisition or screening based on the pseudo-label accuracy screening method of semi-supervised learning, and its implementation process is, for example: S1, taking the average number of nodes that can be linked to each node of each decoding path in all decoding paths during speech recognition decoding as an indicator for judging the accuracy rate; S2, marking pseudo-labels for audio data corresponding to decoding results with high accuracy to use the audio data with pseudo-labels as selected training data.

[0048] [Example 1]

[0049] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the method implementation of the present invention is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0050] Here we will combine Figure 1The main steps of an embodiment of the method of the present invention are described in the flowchart shown in the figure. The method is mainly based on a semi-supervised learning method. Through a speech recognition model trained with a small amount of manually annotated labels, or through a speech recognition model with initialized text labels, multiple unlabeled audio data in the data set are directly input into the model for speech recognition, and multiple corresponding decoding results are output. When one or more decoding results with high accuracy are screened out, the unlabeled audio data corresponding to the decoding results are directly labeled with the text labels in the decoding results. The main process includes:

[0051] Step S110, performing speech recognition on the audio data based on a speech recognition model trained by semi-supervised learning.

[0052] In one implementation, the pseudo-label accuracy of the training data is used in the model training of speech recognition based on semi-supervised learning. Specifically, a speech recognition model is constructed, including an acoustic model and a language model, such as a speech recognition tool implemented by the Kaldi toolkit.

[0053] Further, the speech recognition models in different application scenarios often require training data in specific fields to achieve the best results. For example, in the tasks of bullet screen speech recognition and quality inspection speech transcription, it is necessary to use the data of the respective scene reflow to conduct targeted enhancement training on the model to adapt to the needs in different scenarios. An embodiment of the present invention based on class supervised learning to train the speech recognition model can be adopted: after inputting a very small amount of audio data with manually annotated text labels to train the speech recognition model, it can output the corresponding text label when the recognition is completed and the decoding result / recognition result is output, so that the audio data corresponding to the high-accuracy decoding result is directly marked with the text label in the result as the reflow training data to retrain the model. Alternatively, it can be adopted: pre-set text labels for the speech recognition model, such as according to the actual application scenarios: the characteristics of different scenarios such as speech bullet screen recognition, quality inspection speech transcription, and speech search recognition, directly input unlabeled audio data into the model to evaluate the accuracy and filter the data and use the corresponding text labels predicted in the decoding results for annotation.

[0054] It can be seen that in the K12 scenario, when the intelligent speech recognition of the AI ​​teaching system uses insufficient labeled data for the speech recognition model, it can effectively utilize (screen) the existing transcription data (audio data corresponding to the text label) for semi-supervised training, obtain training data to train the model, and improve the recognition performance, effect, and quality of the model. In order to avoid the influence of noise, the existing transcription data must also meet certain quality requirements before it can be used to expand the training data, that is, it is necessary to screen out high-quality data to ensure that the transcription accuracy of the screened data is high enough and the noise is small enough. Using such data for training can really help improve the performance and effect of the model.

[0055] In step S120, the average value of the number of nodes that each node of each decoding path is expected to link to in all decoding paths during speech recognition decoding is used as an indicator for determining the accuracy rate.

[0056] In one implementation, the speech recognition model of the Kaldi toolkit can be used to perform speech recognition on input unlabeled audio data; during the speech recognition process, beam search decoding is performed according to a predetermined beam width beamsize; and all expected decoding paths that have been searched during decoding are stored.

[0057] Furthermore, all expected decoding paths that have been searched during decoding are stored, specifically including: performing a decoding path search from the state network WFST during decoding to obtain a decoding path graph; using a lattice to store all decoding paths that have been searched in the decoding path graph; and outputting the optimal decoding path as a decoding result corresponding to the audio data.

[0058] Furthermore, the average number of nodes that each node of each decoding path in all decoding paths is expected to link to is used as an indicator for judging the accuracy, specifically including: obtaining the number of paths that each node in each decoding path can link backward from all decoding paths stored in the lattice and calculating the average; using the average as the lattice depth, an indicator for judging the accuracy; and sorting and comparing the corresponding multiple indicators lattice depth obtained when multiple audio data in the data set are decoded through speech recognition to judge the accuracy.

[0059] Furthermore, the corresponding multiple indicators lattice depth obtained when the multiple audio data in the data set are decoded by speech recognition are sorted and compared to determine the accuracy, specifically including: sorting the indicator lattice depth corresponding to each of the multiple audio data and comparing it with a preset threshold; if the ranking of the indicator lattice depth is below the preset threshold, it means that the accuracy of the decoding result of the corresponding audio data is higher.

[0060] Specifically, the speech recognition tools implemented by the Kaldi toolkit, such as the HMM model, construct a WFST state network, that is, a state network based on FST and lattice. Its decoding is to search for the path that best matches the sound to be recognized from the state network. For example, when decoding, a beam search (finding the global optimal path) can be performed according to the given beam width beam size to obtain the corresponding decoding graph, which has multiple possible decoding paths. The best path is found as the decoding result. At this time, the lattice can be used to store all possible decoding paths, which is equivalent to storing the decoding graph. The optimal path is the decoding result, and the decoding result of the model, that is, the audio recognition result, can be output.

[0061] The mean of the number of nodes that each node in the decoding path may link to is used as a ranking indicator or a judgment indicator lattice depth for evaluating the accuracy of pseudo labels. This calculation averages only the number of nodes that may be linked or the number of paths that may be expanded. The algorithm is simple, efficient, and consumes little computing resources. It is directly obtained during the decoding process without introducing any interference or uncertainty.

[0062] In the recognition and training of the model based on semi-supervised learning speech recognition, a data set such as multiple unlabeled audios is input into the model for decoding to obtain decoding results and multiple indicators latticedepth corresponding to the multiple audios, and the multiple indicators latticedepth are sorted from high to low or from low to high, preferably from low to high, and a suitable threshold is pre-set, such as in the bullet screen speech recognition scenario, the threshold is set to 5. Then, all audios with latticedepth values ​​of all recognition results lower than the threshold of 5 are recovered, and these audios are high-quality audios, and these indicators latticedepth lower than the threshold of 5 correspond to decoding results with high accuracy.

[0063] It can be seen that the use of lattice depth as an accuracy ranking index for an audio decoding result does not require the introduction of any other operation steps or parameter processing, such as: no introduction of acoustic-scale scaling factors, LM rescore re-scoring, no need to perform confidence score calculations, rough sentence-level confidence calculations, no interference from various decoding strategies, etc., is completely unaffected by acoustic-scale and LM rescore, and is more stable than indicators such as MBR, so that it can more accurately find out the part of the audio that is accurately decoded from all audio. In addition, in speech recognition, for audio with little noise and easy recognition, it is less likely to generate too many decoding paths during decoding. Therefore, a smaller lattice depth of the decoded recognition audio can indicate that the quality of the audio is high, that is, audio A is recognized as language A very accurately, with little noise, so it is not easy to have a negative impact on the model. When the indicator lattice depth is actually tested, it has a good discrimination for sentences that are well recognized. Further, when the indicator lattice depth is obtained, the computing power resources consumed by its calculation are very small, that is, it can effectively reduce the consumption of redundant resources and will not increase the computing power burden of computer decoding.

[0064] Step S130 , annotating pseudo labels for audio data corresponding to decoding results with high accuracy, and using the audio data with pseudo labels as selected training data.

[0065] In one implementation, unlabeled audio data corresponding to a decoding result with a high accuracy is obtained; and the audio data is pseudo-labeled according to text labels in the decoding result output by speech recognition.

[0066] In one embodiment, audio data with pseudo labels are used as selected training data, and for example: all unlabeled audio data corresponding to decoding results with high accuracy are screened out; after the audio data is labeled with pseudo labels, the audio data with the pseudo labels are added to the model training set of speech recognition, and the speech recognition model is retrained.

[0067] Specifically, after the audio is decoded by the speech recognition model, the decoding result is output, such as the recognized text data. The text label / text data in the decoding result evaluated as having a high accuracy by the indicator latticedepth is directly used as the pseudo-label of the corresponding audio, that is, the model recognition result is the pseudo-label of the high-quality audio. Then, the reflowed audio data (that is, the audio that has been labeled with the pseudo-label) is added to the labeled data as the training data in the training set, and the model is retrained. Therefore, the reflow corpus screening of ASR semi-supervised learning based on lattice depth can screen out suitable training data and continue to provide a large amount of data training to improve the recognition performance, effect and quality of the model.

[0068] It can be seen that directly providing pseudo labels is simple and easy to implement, and the selected training data can also ensure a sufficiently high transcription accuracy with low noise. In addition, the data selected by lattice depth can also be used to enhance the training of models in scenarios such as speech synthesis and noise elimination; more broadly, the lattice depth indicator represents the number of possible results of pattern recognition, that is, the degree of certainty of data classification. Similarly, in other artificial intelligence fields (such as image classification in computer vision), this idea can be used to design other similar indicators to select high-quality data to train and enhance the performance of existing models. Therefore, the scope of application of this indicator will be wider in the future.

[0069] [Example 2]

[0070] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the system implementation of the present invention is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0071] Here we will combine Figure 2 The main structural block diagram of an embodiment of the method of the present invention is shown. The system is also mainly based on a semi-supervised learning method, through a speech recognition model trained with a small amount of manually annotated labels, or through a speech recognition model with initialized text labels, directly inputs multiple unlabeled audio data in the data set into the model for speech recognition, and outputs corresponding multiple decoding results, so that when one or more decoding results with high accuracy are screened out, the unlabeled audio data corresponding to the decoding results are directly labeled with the text labels in the decoding results. This embodiment mainly includes:

[0072] The training recognition module B210 trains a speech recognition model based on semi-supervised learning and performs speech recognition on audio data.

[0073] In one implementation, the pseudo-label accuracy of the training data is used in the model training of speech recognition based on semi-supervised learning. Specifically, a speech recognition model is constructed, including an acoustic model and a language model, such as a speech recognition tool implemented by the Kaldi toolkit.

[0074] Further, the speech recognition models in different application scenarios often require training data in specific fields to achieve the best results. For example, in the tasks of bullet screen speech recognition and quality inspection speech transcription, it is necessary to use the data of the respective scene reflow to conduct targeted enhancement training on the model to adapt to the needs in different scenarios. An embodiment of the present invention based on class supervised learning to train the speech recognition model can be adopted: after inputting a very small amount of audio data with manually annotated text labels to train the speech recognition model, it can output the corresponding text label when the recognition is completed and the decoding result / recognition result is output, so that the audio data corresponding to the high-accuracy decoding result is directly marked with the text label in the result as the reflow training data to retrain the model. Alternatively, it can be adopted: pre-set text labels for the speech recognition model, such as according to the actual application scenarios: the characteristics of different scenarios such as speech bullet screen recognition, quality inspection speech transcription, and speech search recognition, directly input unlabeled audio data into the model to evaluate the accuracy and filter the data and use the corresponding text labels predicted in the decoding results for annotation.

[0075] It can be seen that in the K12 scenario, when the intelligent speech recognition of the AI ​​teaching system uses insufficient labeled data for the speech recognition model, it can effectively utilize (screen) the existing transcription data (audio data corresponding to the text label) for semi-supervised training, obtain training data to train the model, and improve the recognition performance, effect, and quality of the model. In order to avoid the influence of noise, the existing transcription data must also meet certain quality requirements before it can be used to expand the training data, that is, it is necessary to screen out high-quality data to ensure that the transcription accuracy of the screened data is high enough and the noise is small enough. Using such data for training can really help improve the performance and effect of the model.

[0076] An indicator module B220 is set to use the average number of nodes that each node of each decoding path is expected to link to in all decoding paths during speech recognition decoding as an indicator for determining the accuracy rate.

[0077] In one implementation, the speech recognition model of the Kaldi toolkit can be used to perform speech recognition on input unlabeled audio data; during the speech recognition process, beam search decoding is performed according to a predetermined beam width beamsize; and all expected decoding paths that have been searched during decoding are stored.

[0078] Furthermore, all expected decoding paths that have been searched during decoding are stored, specifically including: performing a decoding path search from the state network WFST during decoding to obtain a decoding path graph; using a lattice to store all decoding paths that have been searched in the decoding path graph; and outputting the optimal decoding path as a decoding result corresponding to the audio data.

[0079] Furthermore, the average number of nodes that each node of each decoding path in all decoding paths is expected to link to is used as an indicator for judging the accuracy, specifically including: obtaining the number of paths that each node in each decoding path can link backward from all decoding paths stored in the lattice and calculating the average; using the average as the lattice depth, an indicator for judging the accuracy; and sorting and comparing the corresponding multiple indicators lattice depth obtained when multiple audio data in the data set are decoded through speech recognition to judge the accuracy.

[0080] Furthermore, the corresponding multiple indicators lattice depth obtained when the multiple audio data in the data set are decoded by speech recognition are sorted and compared to determine the accuracy, specifically including: sorting the indicator lattice depth corresponding to each of the multiple audio data and comparing it with a preset threshold; if the ranking of the indicator lattice depth is below the preset threshold, it means that the accuracy of the decoding result of the corresponding audio data is higher.

[0081] Specifically, the speech recognition tools implemented by the Kaldi toolkit, such as the HMM model, construct a WFST state network, that is, a state network based on FST and lattice. Its decoding is to search for the path that best matches the sound to be recognized from the state network. For example, when decoding, a beam search (finding the global optimal path) can be performed according to the given beam width beam size to obtain the corresponding decoding graph, which has multiple possible decoding paths. The best path is found as the decoding result. At this time, the lattice can be used to store all possible decoding paths, which is equivalent to storing the decoding graph. The optimal path is the decoding result, and the decoding result of the model, that is, the audio recognition result, can be output.

[0082] The mean of the number of nodes that each node in the decoding path may link to is used as a ranking indicator or a judgment indicator lattice depth for evaluating the accuracy of pseudo labels. This calculation averages only the number of nodes that may be linked or the number of paths that may be expanded. The algorithm is simple, efficient, and consumes little computing resources. It is directly obtained during the decoding process without introducing any interference or uncertainty.

[0083] In the recognition and training of the model for speech recognition based on semi-supervised learning, a data set such as multiple unlabeled audios is input into the model for decoding to obtain decoding results and multiple indicators lattice depth corresponding to the multiple audios, and the multiple indicators lattice depth are sorted from high to low or from low to high, preferably from low to high, and a suitable threshold is pre-set, such as in the bullet screen speech recognition scenario, the threshold is set to 5. Then, all audios with lattice depth values ​​of all recognition results lower than the threshold of 5 are recovered, and these audios are high-quality audios, and the indicators lattice depth lower than the threshold of 5 correspond to decoding results with high accuracy.

[0084] It can be seen that the use of lattice depth as an accuracy ranking index for an audio decoding result does not require the introduction of any other operation steps or parameter processing, such as: no introduction of acoustic-scale scaling factors, LM rescore re-scoring, no need to perform confidence score calculations, rough sentence-level confidence calculations, no interference from various decoding strategies, etc., is completely unaffected by acoustic-scale and LM rescore, and is more stable than indicators such as MBR, so that it can more accurately find out the part of the audio that is accurately decoded from all audio. In addition, in speech recognition, for audio with little noise and easy recognition, it is less likely to generate too many decoding paths during decoding. Therefore, the smaller lattice depth of the decoded recognition audio can indicate that the quality of the audio is high, that is, audio A is recognized as language A very accurately, with little noise, so it is not easy to have a negative impact on the model. When the lattice depth of the indicator is actually tested, its discrimination for the recognized sentences is very good. Further, when the indicator lattice depth is obtained, the computing power resources consumed by its calculation are very small, that is, it can effectively reduce the consumption of redundant resources and will not increase the computing power burden of computer decoding.

[0085] The data screening module B230 annotates pseudo labels for the audio data corresponding to the decoding results with high accuracy, and uses the audio data with pseudo labels as selected training data.

[0086] In one implementation, unlabeled audio data corresponding to a decoding result with a high accuracy is obtained; and the audio data is pseudo-labeled according to text labels in the decoding result output by speech recognition.

[0087] In one embodiment, audio data with pseudo labels are used as selected training data, and for example: all unlabeled audio data corresponding to decoding results with high accuracy are screened out; after the audio data is labeled with pseudo labels, the audio data with the pseudo labels are added to the model training set of speech recognition, and the speech recognition model is retrained.

[0088] Specifically, after the audio is decoded by the speech recognition model, the decoding result is output, such as the recognized text data. The text label / text data in the decoding result evaluated as having a high accuracy by the lattice depth indicator is directly used as the pseudo-label of the corresponding audio, that is, the model recognition result is the pseudo-label of the high-quality audio. Then, the reflowed audio data (that is, the audio that has been labeled with the pseudo-label) is added to the labeled data as the training data in the training set, and the model is retrained. Therefore, the reflow corpus screening of ASR semi-supervised learning based on lattice depth can screen out suitable training data and continue to provide a large amount of data training to improve the recognition performance, effect and quality of the model.

[0089] It can be seen that directly providing pseudo labels is simple and easy to implement, and the selected training data can also ensure a sufficiently high transcription accuracy with low noise. In addition, the data selected by lattice depth can also be used to enhance the training of models in scenarios such as speech synthesis and noise elimination; more broadly, the lattice depth indicator represents the number of possible results of pattern recognition, that is, the degree of certainty of data classification. Similarly, in other artificial intelligence fields (such as image classification in computer vision), this idea can be used to design other similar indicators to select high-quality data to train and enhance the performance of existing models. Therefore, the scope of application of this indicator will be wider in the future.

[0090] [Example 3]

[0091] The following describes an overall application scenario and combines Examples 1 and 2 to further illustrate the implementation process of the present invention:

[0092] Use the Kaldi toolkit to implement speech recognition. A speech recognition model in Kaldi based on semi-supervised learning training configuration. The training can be directly performed using unlabeled audio. The training combines self-labeling, screening of training data (audio), and retraining to obtain labeled training data in a simple, efficient, low-cost, and low-resource manner, so as to scalably adapt to models in various scenarios and train models to improve model performance and enhance its recognition effect and quality.

[0093] The speech recognition model of the Kaldi toolkit can use sound models such as HMM combined with language models to recognize the input audio. For example, the input audio is divided into frames and then its state is recognized, the state is combined into phonemes, and the phonemes are used to synthesize words. Kaldi's HMM supports FST and lattice. Therefore, when decoding the recognition state through HMM, a state network WFST is constructed according to FST. According to the given beam width beamsize, the best path is searched in the WFST through beam search, that is, the path (decoding path) that best matches the sound is found. During the search process, all possible decoding paths are stored through lattice (see the above explanation of the relationship between lattice and FST). The number of nodes that each node in the decoding path may link to or the number of paths that may be connected is obtained, and the mean is calculated as the accuracy or accuracy indicator latticedepth of the decoding result. For example: in path 1, node A links 1 node, node B links 2 nodes, and node C links 1 node; in path 2, node A links 1 node, node C links 2 nodes, and node D links 1 node. The average is 8 / 4=2, that is, the value of the indicator is 2 (this is only a conceptual explanation and not an example of actual calculation).

[0094] The smaller the value of this indicator, the higher the accuracy, and the higher the quality of the corresponding audio used as training data, that is, its recognition accuracy can better meet the actual needs of the current application scenario.

[0095] The latticedepth index of each decoding result of multiple unlabeled input audios is sorted from small to large, and compared with a preset threshold, such as a threshold of 5 for bullet screen speech recognition. This threshold setting also meets the needs of actual application fields, so one index can be used in more different scenarios that require speech recognition.

[0096] Assume that the values ​​of the latticedepth index after the three audio segments A, B, and C are input are 6, 5.5, and 4 respectively. Then, among 4, 5.5, and 6, only the index of audio C is lower than the threshold value of 5. Furthermore, the text label in the decoding result of C can be directly used as an annotation for C, that is, C is screened out, and C annotated with the text label is added to the training data set. When training the model based on semi-supervised learning (that is, the speech recognition model in the configured Kaldi), the new training data C is continued to be used for model training.

[0097] [Example 4]

[0098] Figure 3It is a structural schematic block diagram of an electronic device according to an embodiment of the present invention, the electronic device includes a processor and a memory, the memory is used to store a computer executable program, when the computer program is executed by the processor, the processor executes the embodiment steps of the method involved in the aforementioned embodiments 1 and 3.

[0099] like Figure 3 As shown, the electronic device is presented in the form of a general-purpose computing device. The processor may be one or more and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0100] The memory stores a computer executable program, usually a machine-readable code, which can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.

[0101] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0102] Optionally, in this embodiment, the electronic device further includes an I / O interface, which is used for the electronic device to exchange data with external devices. The I / O interface can represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0103] More specifically, see Figure 4 1 is a block diagram of a more specific example of the electronic device described in the embodiment shown. The electronic device 200 of the exemplary embodiment is in the form of a general data processing device. The components of the electronic device 200 may include, but are not limited to: at least one processing unit 210, at least one storage unit 220, a bus 230 connecting different system components (including the storage unit 220 and the processing unit 210), a display unit 240, etc.

[0104] The storage unit 220 stores a computer-readable program, which may be a source program or a code of a read-only program. The program may be executed by the processing unit 210, so that the processing unit 210 performs the steps of various embodiments of the present invention. For example, the processing unit 210 may perform the various steps of the methods of the aforementioned embodiments 2 to 5.

[0105] The storage unit 220 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 2201 and / or a cache storage unit 2202, and may further include a read-only storage unit (ROM) 2203. The storage unit 220 may also include a program / utility 2204 having a set (at least one) of program modules 2205, such program modules 2205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include the implementation of a network environment.

[0106] Bus 230 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0107] The electronic device 200 may also communicate with one or more external devices 300 (e.g., keyboards, displays, network devices, Bluetooth devices, etc.) so that a user can interact with the electronic device 200 via these external devices 300, and / or so that the electronic device 200 can communicate with one or more other data processing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 250, and may also be performed through a network adapter 260 with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet). The network adapter 260 may communicate with other modules of the electronic device 200 via the bus 230. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in the electronic device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0108] It should be understood that Figure 3 , 4 The electronic device shown is only an example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.

[0109] [Example 5]

[0110] Figure 5 Schematic diagram of a computer readable recording medium according to an embodiment of the present invention. Figure 5As shown, a computer executable program is stored in a computer-readable recording medium, and when the computer executable program is executed, the above-mentioned method for providing a tutoring strategy based on a question of the present invention is implemented. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0111] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0112] Through the above description of the implementation mode, it is easy for those skilled in the art to understand that the present invention can be implemented by hardware capable of executing a specific computer program, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. contained in the system. The present invention can also be implemented by a vehicle comprising at least a part of the above system or components. The present invention can also be implemented by computer software that executes the method of the present invention, such as control software executed by a microprocessor, an electronic control unit, a client, a server, etc. on the locomotive side. However, it should be noted that the computer software that executes the method of the present invention is not limited to being executed by one or a specific hardware entity, and it can also be implemented in a distributed manner by unspecified specific hardware, such as some method steps executed by the computer program can be executed on the locomotive side, and another part can be executed in a mobile terminal or a smart helmet, etc. For computer software, the software product can be stored in a computer-readable storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), or it can be distributed and stored on the network, as long as it enables the electronic device to execute the method according to the present invention.

[0113] Through the above description of the implementation modes, those skilled in the art can easily understand that the exemplary embodiments described in the present invention can be implemented by software, or by combining software with necessary hardware.

[0114] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the present invention is not inherently related to any specific computer, virtual device or electronic device, and various general devices can also implement the present invention. The above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for screening training data, characterized in that: include: The average number of nodes that each node in each decoding path is expected to link to in all decoding paths during speech recognition decoding is used as an indicator for judging the accuracy, including: All decoding paths of speech recognition during decoding include: using the speech recognition model of the Kaldi toolkit to perform speech recognition on input unlabeled audio data; performing beam search decoding according to a predetermined beam width beamsize during the speech recognition process; storing all expected decoding paths that have been searched during decoding, specifically including: performing a decoding path search from the state network WFST during decoding to obtain a decoding path graph, using lattice to store all decoding paths that have been searched in the decoding path graph, and outputting the optimal decoding path as a decoding result corresponding to the audio data; The average number of nodes that each node of each decoding path in all decoding paths is expected to link to is used as an indicator for judging the accuracy rate, including: obtaining the number of paths that each node in each decoding path can link backward from all decoding paths stored in lattice and calculating the average; using the average as an indicator latticedepth for judging the accuracy rate; sorting and comparing the corresponding multiple indicators latticedepth obtained when multiple audio data in the data set are decoded by speech recognition to judge the accuracy rate, specifically including: sorting the indicators lattice depth corresponding to each of the multiple audio data and comparing them with a preset threshold value, if the ranking of the indicator lattice depth is below the preset threshold value, it means that the decoding result of the corresponding audio data has a higher accuracy rate; Pseudo-labels are labeled for audio data corresponding to decoding results with high accuracy, and the audio data with pseudo-labels are used as selected training data; wherein, based on a semi-supervised learning method, a speech recognition model trained with a small amount of manually labeled labels, or a speech recognition model with text labels initialized, multiple unlabeled audio data in a data set are directly input into the model for speech recognition, and multiple corresponding decoding results are output, so that when one or more decoding results with high accuracy are screened out, the unlabeled audio data corresponding to the decoding results are directly labeled with the text labels in the decoding results.

2. The method according to claim 1, characterized in that The speech recognition models using the Kaldi toolkit include: HMM model.

3. The method according to claim 2, characterized in that The HMM model also includes: constructing a state network WFST based on FST and lattice.

4. The method according to claim 3, characterized in that Calculating the mean also includes averaging only the number of nodes that will be linked or the number of paths that will be expanded.

5. The method according to claim 4, characterized in that The sorting of the indicator lattice depth also includes: sorting from low to high; and recovering audio data at positions below the preset threshold.

6. The method according to any one of claims 1 to 5, characterized in that Pseudo labels are added to the audio data corresponding to the decoding results with high accuracy, including: Obtaining unlabeled audio data corresponding to a decoding result with high accuracy; The audio data is annotated with pseudo labels according to the text labels in the decoding results output by the speech recognition.

7. The method according to claim 6, characterized in that The audio data with pseudo labels is selected as training data, including: Filter out all unlabeled audio data corresponding to the decoding results with high accuracy; After the audio data is annotated with pseudo labels, the audio data with the pseudo labels are added to a speech recognition model training set to retrain the speech recognition model.

8. The method according to any one of claims 1 to 7, characterized in that Also includes: The unlabeled audio data corresponding to the decoding result is directly labeled with the text label in the decoding result and used as the reflow training data to retrain the model.

9. An electronic device comprising a processor and a memory, wherein the memory is used to store a computer executable program, wherein: When the computer executable program is executed by the processor, the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable medium storing a computer-executable program, wherein when the computer-executable program is executed, the method according to any one of claims 1 to 7 is implemented.

Images