Audio data screening method, device, equipment and computer-readable medium

Abstract

The present application discloses an audio data screening method, apparatus, device and computer-readable medium, which relates to the field of audio processing technology, including: obtaining an audio data set, the audio data set including multiple audio data and text data corresponding to each audio data; aligning the audio data and the corresponding text data to obtain a first sequence including each first phoneme and a first pronunciation probability; performing speech recognition on the audio data to obtain a second sequence including each second phoneme and a second pronunciation probability, determining the third pronunciation probability of each text in the text data in the audio data based on the first sequence, the first pronunciation probability, the second sequence and the second pronunciation probability; based on the third pronunciation probability, screening the target audio data and the corresponding target annotated text in the audio data set to construct an audio training set. The above method can solve the technical problem that errors in artificially constructed annotated texts are prone to occur, thus affecting the training effect of the acoustic model.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to an audio data screening method, apparatus, device and computer-readable medium. Background Art

[0002] Speech recognition technology, also known as automatic speech recognition, aims to convert the vocabulary content of human speech into computer-readable input, such as keystrokes, binary codes, or text character sequences. Currently, with the development of speech recognition technology, speech recognition technology is gradually penetrating into people's daily lives.

[0003] The implementation of speech recognition technology is inseparable from acoustic models. Acoustic models are the models used to identify sounds within speech recognition models. Training an acoustic model requires a large amount of training data. Generally speaking, training data includes a large amount of speech data and corresponding annotated text, where the annotated text is the text annotation corresponding to the speech content in the speech data. Currently, the construction of annotated text is done manually, and errors and omissions are inevitable during the annotation process, resulting in human-induced ambiguity in the annotated text. In this case, during the training of the acoustic model, the presence of human-induced erroneous annotated text in the training data can cause the acoustic model to learn the incorrect annotated text, affecting the training effect of the acoustic model itself. Summary of the Invention

[0004] The present application provides an audio data screening method, apparatus, device and computer-readable medium for solving the technical problem that errors are easily generated in artificially constructed annotated texts and thus affect the training effect of acoustic models.

[0005] In a first aspect, an embodiment of the present application provides an audio data screening method, comprising:

[0006] Acquire an audio data set, the audio data set including multiple audio data and text data corresponding to each audio data;

[0007] After aligning the audio data with the corresponding text data, obtaining a first pronunciation probability of each first phoneme and a first sequence containing each first phoneme, where the first phoneme is a phoneme in the text data, and the first sequence further includes a first start and end time of each first phoneme in the audio data;

[0008] After performing speech recognition on the audio data, obtaining a second pronunciation probability of each second phoneme and a second sequence including each second phoneme, where the second phoneme is the phoneme obtained after recognizing the audio data;

[0009] Determine a third pronunciation probability of each text in the text data in the audio data according to the first sequence, the first pronunciation probability, the second sequence, and the second pronunciation probability;

[0010] According to the third pronunciation probability, target audio data and corresponding target text data are screened out from the audio data set, and the target audio data set and the target text data constitute an audio training set.

[0011] In a second aspect, an embodiment of the present application provides an audio data screening device, the device comprising:

[0012] A data set acquisition module is used to acquire an audio data set, wherein the audio data set includes multiple audio data and text data corresponding to each audio data;

[0013] an alignment module, configured to align the audio data with the corresponding text data to obtain a first pronunciation probability of each first phoneme and a first sequence containing each first phoneme, wherein the first phoneme is a phoneme in the text data, and the first sequence further includes a first start and end time of each first phoneme in the audio data;

[0014] a recognition module configured to perform speech recognition on the audio data to obtain a second pronunciation probability of each second phoneme and a second sequence containing each second phoneme, where the second phoneme is a phoneme obtained after recognizing the audio data;

[0015] a pronunciation probability determination module, configured to determine a third pronunciation probability of each text in the text data in the audio data based on the first sequence, the first pronunciation probability, the second sequence, and the second pronunciation probability;

[0016] The audio data screening module is used to screen out target audio data and corresponding target text data from the audio data set according to the third pronunciation probability, and the target audio data set and the target text data constitute an audio training set.

[0017] In a third aspect, an embodiment of the present application provides an audio data screening device, including a processor and a memory;

[0018] The memory is used to store program code and transmit the program code to the processor;

[0019] The processor is configured to execute the audio data screening method as described in the first aspect according to the instructions in the program code.

[0020] In a fourth aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the audio data screening method as described in the first aspect.

[0021] The above-mentioned audio data screening method, device, equipment and computer-readable medium first obtain an audio data set including multiple audio data and corresponding text data, and then align the audio data and the corresponding text data to obtain a first pronunciation probability of each first phoneme in the text data and a first sequence containing each first phoneme and the first start and end time of the first phoneme in the audio data; perform speech recognition on the audio data to identify the second pronunciation probability of each second phoneme in the audio data and a second sequence containing each second phoneme and the second phoneme in the audio data, and then determine the third pronunciation probability of each text in the text data in the audio data based on the first sequence, the first pronunciation probability, the second sequence and the second pronunciation probability, and then, based on the third pronunciation probability, screen out the target audio data and the corresponding target annotated text that can be used to construct an audio training set in the audio data set, which solves the technical problem that artificially constructed annotated texts are prone to errors and affect the training effect of the acoustic model. By aligning the audio data and the corresponding text data, the first pronunciation probability and the first start and end time of each first phoneme in the text data appearing in the audio data can be determined. By identifying the audio data, the second phoneme appearing in the audio data and the second pronunciation probability of the second phoneme appearing in the audio data can be identified. Then, the third pronunciation probability of each text in the text data appearing in the audio data is determined in combination with the above parameters, that is, whether the text data and the audio data are consistent. At this time, even if there is an error in the text data, it can be reflected by the third pronunciation probability, and the quality of the text data and audio data can be clarified. Then, appropriate target text data and target audio data are selected to construct an audio training set to ensure the training effect of the acoustic model when using the audio data set to train the acoustic model. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flowchart of an audio data screening method provided in an embodiment of the present application.

[0023] Figure 2 A flowchart of an audio data screening method provided in an embodiment of the present application.

[0024] Figure 3 A diagram illustrating the working principle of an alignment network model provided in an embodiment of the present application.

[0025] Figure 4 A schematic diagram of the structure of an audio data screening device provided in an embodiment of the present application.

[0026] Figure 5 A schematic diagram of the structure of an audio data screening device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following description and accompanying drawings sufficiently illustrate specific embodiments of the present application to enable those skilled in the art to practice them. The examples represent only possible variations. Unless expressly required, individual components and functions are optional, and the order of operations may vary. Portions and features of some embodiments may be included in or replace portions and features of other embodiments. The scope of the embodiments of the present application includes the entire scope of the claims, as well as all available equivalents of the claims. Herein, each embodiment may be referred to individually or collectively by the term "invention," which is merely for convenience and is not intended to automatically limit the scope of the application to any single invention or inventive concept if more than one invention is in fact disclosed. Herein, relational terms such as first and second, etc., are used solely to distinguish one entity or operation from another, and do not require or imply any actual relationship or order between these entities or operations. Furthermore, the terms "comprise," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or apparatus comprising a set of elements includes not only those elements, but also other elements not explicitly listed. The various embodiments herein are described in a progressive manner, with each embodiment focusing on the differences from the other embodiments. Reference can be made to the common and similar parts between the various embodiments. For structures, products, etc. disclosed in the embodiments, the description is relatively simple because they correspond to the parts disclosed in the embodiments. For relevant parts, refer to the method description.

[0028] The audio data screening method provided in the embodiments of the present application can be performed by an audio data screening device, which can be implemented by software and / or hardware. The audio data screening device can be composed of two or more physical entities or a single physical entity. For example, the audio data screening device can be a computer, tablet, smartphone, server, or other device that can directly process data.

[0029] Figure 1 A flowchart of an audio data screening method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the audio data screening method includes:

[0030] Step 101: Acquire an audio data set, where the audio data set includes multiple audio data and text data corresponding to each audio data.

[0031] Generally, audio is an important medium in multimedia, a form of sound signal. As a carrier of information, audio can be categorized into three types: speech, music, and other sounds. In this embodiment, audio is described as the speech type. Audio data refers to digitized sound data, which can be obtained by digitizing sound data. Digitizing sound data involves performing analog-to-digital conversion on analog audio signals from a device such as a microphone at a specific frequency to obtain audio data. Text data refers to data containing text, which can include English words, Chinese characters, numbers not used as numerical values, and other inputtable text symbols.

[0032] The text data corresponding to each piece of audio data refers to text data generated by textually annotating the audio content in the audio data. For example, a recording is made while a user says "I am Chinese," obtaining audio data containing the audio content "I am Chinese," and constructing text data containing the text content "I am Chinese." In this case, the audio data and text data correspond to each other. It will be appreciated that in this embodiment, there is no specific limitation on the audio content in each piece of audio data and the text content in the text data corresponding to each piece of audio data.

[0033] The collection of multiple audio data and the text data corresponding to each audio data constitutes an audio data set. It is understandable that the number of audio data and the text data corresponding to each audio data can be selected according to actual needs. In this embodiment, the number of audio data and text data is not specifically limited. The audio data set can be obtained by obtaining an existing audio data set through the network, or by obtaining a pre-constructed data set. In this embodiment, the method of obtaining the audio data set is not specifically limited. In one embodiment, the audio data in the audio data set is collected in advance, and then the corresponding text data is manually constructed based on the audio data. It is understandable that in the process of manually constructing text data, errors may occur in the text data, that is, the audio data and the corresponding text data may be inconsistent.

[0034] Step 102: After aligning the audio data and the corresponding text data, obtain the first pronunciation probability of each first phoneme and a first sequence containing each first phoneme, where the first phoneme is a phoneme in the text data, and the first sequence also contains the first start and end time of each first phoneme in the audio data.

[0035] In the embodiment, the audio data is composed of audio frames, and each audio frame can be considered as a playback moment.

[0036] Alignment refers to aligning the audio content in the audio data with the text content in the corresponding text data according to the time of appearance. For example, the audio content in the audio data is "I am Chinese", and the text content in the text data is also "I am Chinese". If the playback time of the "I" text in the audio data is from the 3rd audio frame to the 13th audio frame, then the "I" text in the text data is aligned with the 3rd audio frame to the 13th audio frame of the audio data. In the embodiment, alignment specifically aligns the phonemes of each text in the text data with the playback time of the phoneme in the audio data. Among them, phonemes are the smallest speech units divided according to the natural properties of speech. According to the pronunciation action in the syllable, one action constitutes a phoneme, and phonemes are divided into two categories: vowels and consonants.

[0037] Exemplarily, after aligning the audio data and the text data, the start and end time of each phoneme in the text data in the audio data can be obtained. In an embodiment, each phoneme in the text data is recorded as a first phoneme, the start and end time corresponding to the first phoneme is recorded as a first start and end time, and the sequence consisting of each first phoneme and the corresponding first start and end time is recorded as a first sequence. Among them, the first phonemes in the first sequence are arranged in the order of appearance in the text data, and each first phoneme corresponds to a first start and end time. The first start and end time may include the playback time corresponding to the audio frame where the first phoneme starts to pronounce and the playback time corresponding to the audio frame where the pronunciation ends during the playback of the audio data. For example, the text content of the text data is "one", and the audio content of the corresponding audio data is also "one". The first phoneme obtained after alignment is: yh ige. The playback time is represented by the frame number, that is, each playback time corresponds to an audio frame. At this time, assuming that the number of frames of the audio data is 30 frames, after alignment, yh appears in the audio data from the 6th to the 10th frame, i appears in the audio data from the 11th to the 13th frame, g appears in the audio data from the 14th to the 18th frame, and e appears in the audio data from the 19th to the 23rd frame. The frames without phonemes are considered to be silent frames and are represented by sil. The first sequence is: The superscripts and subscripts of the first phoneme represent the first start and end times of the first phoneme. In the first sequence, each first phoneme has a first pronunciation probability. It is understood that when repeated first phonemes appear in the text data, the repeated first phonemes are independent of each other and each has a corresponding first pronunciation probability and first start and end time.

[0038] Exemplarily, after aligning the audio data and the text data, the pronunciation probability of each phoneme in the text data in the audio data can also be obtained. The pronunciation probability can also be understood as the possibility that the audio data is pronounced as the phoneme. In the embodiment, the pronunciation probability corresponding to the first phoneme is recorded as the first pronunciation probability. Each first phoneme has a pronunciation probability under each corresponding audio frame, that is, the first pronunciation probability can be understood as the probability that the phoneme pronounced in the current audio frame in the audio data is the current first phoneme. It can be understood that the greater the first pronunciation probability, the greater the probability that the current audio frame is pronounced as the first phoneme.

[0039] Furthermore, the method of aligning audio data and text data can be set according to actual conditions. For example, in an embodiment, alignment is achieved using an acoustic model and a WFST (Weighted Finite State Transducers) network. Among them, the acoustic model is a model used to recognize sound in a speech recognition model. Optionally, the acoustic model is obtained by training with an audio data set. In one embodiment, the acoustic model adopts a traditional GMM-HMM or DNN-HMM structure; in the GMM-HMM structure, the GMM (Gaussian mixture model) is used to model the observation probability of speech, and the HMM (hidden Markov model) models the time sequence of speech; in the DNN-HMM structure, the DNN (deep neural network model) replaces the GMM to model the observation probability of the input speech. It should be noted that the Markov model is a discrete time-domain finite state automaton, and the hidden Markov model means that the internal state of this Markov model is invisible to the outside world, and the outside world can only see the output value at each moment. Using a hidden Markov model to characterize speech signals requires two assumptions: one is that the transition of the internal state is only related to the previous state, and the other is that the output value is only related to the current state (or the current state transition). These two assumptions greatly reduce the complexity of the acoustic model. The WFST network is also recorded as a finite weighted state transition machine. Each text data has a corresponding WFST network to achieve alignment of the text data with the corresponding audio data. Exemplarily, during alignment, the audio features of each audio frame are first extracted from the audio data. After that, the acoustic model is used to identify the audio features to determine the possible pronunciations of the audio frame and the probability of pronouncing each phoneme. After that, the possible pronunciations of each audio frame and the corresponding probabilities are input into the WFST network corresponding to the text data, so that the WFST network outputs the first phoneme corresponding to each audio frame in the audio data. After that, a first sequence is obtained based on the first phoneme corresponding to each audio frame and the playback time of each audio frame. After obtaining the first sequence, the first pronunciation probability of the first phoneme in the corresponding audio frame can be calculated, that is, the probability of the audio frame being pronounced as the first phoneme is used as the first pronunciation probability of the first phoneme.

[0040] Step 103 : After performing speech recognition on the audio data, a second pronunciation probability of each second phoneme and a second sequence containing each second phoneme are obtained, where the second phoneme is the phoneme obtained after the audio data is recognized.

[0041] Speech recognition refers to identifying the audio content in audio data, and then clarifying the corresponding text content based on the audio content. For example, by recognizing audio data with the audio content of "I am Chinese", it can be clarified that the audio content is the audio of "I am Chinese". In an embodiment, during the speech recognition process, it is necessary to first identify the phonemes pronounced in each audio frame in the audio data. In an embodiment, the phonemes pronounced in each audio frame in the identified audio data are recorded as second phonemes. Exemplarily, the audio data is "one", and the second phoneme obtained after speech recognition is: yh i, then the second sequence is: yhi. It can be understood that when there are repeated second phonemes in the audio data, the repeated second phonemes are independent. Optionally, the second phoneme sequence also includes a second start and end time, and the second start and end time represents the start and end time of the second phoneme in the audio data.

[0042] Furthermore, after speech recognition, a pronunciation probability corresponding to each second phoneme can be obtained. In this embodiment, this pronunciation probability is referred to as a second pronunciation probability. The second pronunciation probability can be understood as the probability that the current audio frame of the audio data is pronounced as the second phoneme. In the second sequence, each second phoneme has a corresponding second pronunciation probability for each corresponding audio frame. A greater second pronunciation probability indicates a greater probability that the current audio frame is pronounced as the recognized second phoneme.

[0043] Furthermore, the method of performing speech recognition on audio data can be set according to actual conditions. For example, in an embodiment, the speech recognition of audio data is realized by using an acoustic model and a recognition network model. The recognition network model is also a WFST network, and each audio data shares a recognition network model. Exemplarily, when performing recognition, the audio features of each audio frame are first extracted from the audio data. After that, the audio features are recognized using an acoustic model to determine the various phonemes that the audio frame may pronounce and the probability of pronouncing each phoneme. It is understandable that in actual applications, the audio features extracted during the alignment process and the various phonemes that the determined audio frame may pronounce and the probability of pronouncing each phoneme can be directly used. Afterwards, the phonemes that may be pronounced in each audio frame and the probability of pronouncing each phoneme are taken as input information and input into a pre-built recognition network model, so that the recognition network model outputs the second phoneme corresponding to each audio frame in the audio data. Afterwards, a second sequence is obtained based on the second phoneme corresponding to each audio frame and the playback time of each audio frame. After obtaining the second sequence, the second pronunciation probability of the second phoneme in the corresponding audio frame can be calculated, that is, the probability of the audio frame being pronounced as the second phoneme is taken as the first pronunciation probability of the first phoneme.

[0044] Step 104 : Determine a third pronunciation probability of each text in the text data in the audio data according to the first sequence, the first pronunciation probability, the second sequence, and the second pronunciation probability.

[0045] For example, when the text data includes Chinese, one text can be understood as one Chinese character; when the text data includes English, one text can be understood as one English word; when the text data includes numbers, one text can be understood as one number.

[0046] The third pronunciation probability is the pronunciation probability of each text in the text data in the audio data. The higher the third pronunciation probability, the greater the probability that the text is correctly labeled; the lower the third pronunciation probability, the lower the probability that the text is correctly labeled. That is, the third pronunciation probability of each text can be used to determine whether the audio data and text data are consistent, and thus determine the quality of the audio data and text data. In one embodiment, the start and end times of the text in the audio data can be determined based on the first sequence. Specifically, the start and end times of the text can be determined based on the first start and end times corresponding to the first phoneme contained in the text. Then, the first pronunciation probability of each first phoneme within the start and end time is obtained, and the second pronunciation probability of each second phoneme within the start and end time is determined in the second sequence. Then, the third pronunciation probability of the text is obtained based on the first pronunciation probability of each first phoneme and the second pronunciation probability of each second phoneme within the start and end time. It is understood that the calculation method of the third pronunciation probability can be set according to actual circumstances. For example, the ratio of the first pronunciation probability to the second pronunciation probability at the same pronunciation moment can be calculated. Then, the average of the ratios within the start and end time of the text is calculated and used as the third pronunciation probability. In this case, each text in the text data corresponds to a third pronunciation probability.

[0047] Step 105 : Filter target audio data and corresponding target text data from the audio data set according to the third pronunciation probability. The target audio data set and the target text data constitute an audio training set.

[0048] A training set refers to data samples used to train a model used for machine learning, such as a neural network model. In one embodiment, an audio training set refers to a training set used to train an acoustic model, and the audio training set consists of audio data and corresponding text data. It is understood that the acoustic model can identify audio features to obtain corresponding acoustic scores. In this embodiment, after the acoustic model training is completed, the acoustic model is retrained using the filtered audio training set to improve the training accuracy of the acoustic model.

[0049] Since the third pronunciation probability can reflect the probability of each text in the text data being pronounced (i.e., appearing) in the audio data, the target audio data and the corresponding target text data can be screened out in the audio data set according to the third pronunciation probability, wherein the target audio data and the target text data refer to the audio data and text data that meet the requirements of constructing the audio training set. When the acoustic model is trained based on the audio training set constructed based on the target audio data and the target text data, a relatively good training effect can be obtained. In one embodiment, for any text data, if the third pronunciation probability of each text therein is very high, it means that the possibility of the content of the text data being consistent with the audio data is very high, that is, the greater the probability of the text data being correctly marked, if the third pronunciation probability of some of the texts is very low, it means that the possibility of the content of the text data being consistent with the audio data is relatively low, that is, the smaller the probability of the text data being correctly marked. Therefore, text data and audio data with relatively consistent content can be selected in the audio data set as target text data and target audio data according to the third pronunciation probability of each text in the text data. Optionally, when selecting text data and audio data with relatively consistent content, text data can be found in the audio data set where the probability of the third pronunciation of each text is higher than a certain threshold. The threshold can be determined based on actual conditions. When the third pronunciation probability is higher than the threshold, it indicates that the corresponding text has a high probability of being correctly annotated. Then, some of the text data found is deleted. It can be understood that the acoustic model can achieve stable learning results by learning audio data and text data with consistent content without requiring an excessive amount of data. Therefore, in the embodiment, some text data and corresponding audio data with consistent content are deleted from the audio data set to reduce the amount of data processing. Based on the above, when selecting text data and audio data with relatively consistent content, text data that is considered to be incorrectly annotated can also be found in the audio data set, where the third pronunciation probability of some text in the text data is much lower than the above threshold. Then, the incorrectly annotated text data and corresponding audio data are deleted from the audio data set, and the audio data and text data retained in the audio data set are then used as the target audio data and target text data. It should be noted that, generally speaking, there is no text data where the third pronunciation probability of each text is very low. If there is text data where the third pronunciation probability of each text is very low, the text data and corresponding audio data are also deleted.

[0050] In the above, first, an audio data set including multiple audio data and corresponding text data is obtained, and then the audio data and the corresponding text data are aligned to obtain the first pronunciation probability of each first phoneme in the text data and a first sequence including each first phoneme and the first start and end time of the first phoneme in the audio data; speech recognition is performed on the audio data to identify the second pronunciation probability of each second phoneme in the audio data and a second sequence including each second phoneme, and then the third pronunciation probability of each text in the text data in the audio data is determined according to the first sequence, the first pronunciation probability, the second sequence and the second pronunciation probability, and then according to the third pronunciation probability, the target audio data and the corresponding target annotated text that can be used to construct an audio training set are screened out in the audio data set, which solves the technical problem that the artificially constructed annotated text is prone to errors and affects the training effect of the acoustic model. By aligning the audio data and the corresponding text data, the first pronunciation probability and the first start and end time of each first phoneme in the text data appearing in the audio data can be determined. By identifying the audio data, the second phoneme appearing in the audio data and the second pronunciation probability of the second phoneme appearing in the audio data can be identified. Then, the third pronunciation probability of each text in the text data appearing in the audio data is determined in combination with the above parameters, that is, whether the text data and the audio data are consistent. At this time, even if there is an error in the text data, it can be reflected by the third pronunciation probability, and the quality of the text data and audio data can be clarified. Then, appropriate target text data and target audio data are selected to construct an audio training set to ensure the training effect of the acoustic model when using the audio data set to train the acoustic model.

[0051] Figure 2 This is a flowchart of an audio data screening method provided in an embodiment of the present application. The audio data screening method is concretized based on the above embodiment. Figure 2 As shown, the audio data screening method includes:

[0052] Step 201: Acquire an audio data set, where the audio data set includes multiple audio data and text data corresponding to each audio data.

[0053] In one embodiment, a constructed audio data set is obtained from a database, where the audio data set includes multiple audio data and text data corresponding to each audio data.

[0054] Step 202: Extract audio features of each audio frame in the audio data.

[0055] Audio features refer to the features of audio data, and audio features may include MFCC, FBANK, and PLP. MFCC stands for Mel-frequency cepstral coefficients, which are cepstral parameters extracted in the Mel-scale frequency domain. They are features widely used in automatic speech and speaker recognition. FBANK stands for FilterBank. Since the human ear's response to the sound spectrum is nonlinear, FBANK is a front-end processing algorithm that obtains the FBANK features of a speech signal. It processes audio in a similar way to the human ear, and the FBANK features can improve the performance of speech recognition. PLP stands for linear prediction analysis feature. A speech sample can be approximated by a linear combination of several past speech samples. The linearly predicted samples approximate the actual speech samples in the sense of minimum mean square error, and a unique set of prediction coefficients can be obtained. Audio features can be obtained by extracting from audio data, and the specific extraction method is not limited in the embodiment.

[0056] Step 203: Calculate an acoustic score of the corresponding audio frame based on the audio features. The acoustic score is used to represent the probability that the audio frame belongs to each preset phoneme. The preset phonemes include a first phoneme and a second phoneme.

[0057] After obtaining the audio features of each audio frame in the audio data, the acoustic score of each audio frame can be calculated based on the audio features of each audio frame. Each audio frame corresponds to an acoustic score, and the acoustic score can be used to reflect the probability that the phonemes pronounced by the audio frame belong to each preset phoneme. Among them, the preset phonemes are phonemes that may appear in the preset audio. In one embodiment, the preset phonemes include at least a first phoneme and a second phoneme. It can be understood that the first phoneme and the second phoneme have the same phoneme. Optionally, the acoustic score can be represented in the form of a matrix, which shows each phoneme and the corresponding probability. In one embodiment, the acoustic score of each audio frame is calculated using an acoustic model. Therefore, this step can specifically include: inputting the audio features into a pre-constructed acoustic model so that the acoustic model outputs the acoustic score corresponding to the audio features. The acoustic score represents the probability that the audio frame belongs to each preset phoneme. The preset phonemes include the first phoneme and the second phoneme.

[0058] In one embodiment, an acoustic model is used to obtain an acoustic score for each audio feature. At this time, after the audio feature is input into the acoustic model, the acoustic model identifies the audio feature to determine the probability that the phonemes pronounced in the corresponding audio frame belong to each preset phoneme. At this time, the probability of belonging to each preset phoneme can be recorded as an acoustic score. It should be noted that the acoustic model used in this step is an acoustic model pre-trained using other audio data sets (not the audio training set mentioned in this embodiment), wherein the embodiment of the process of training the acoustic model is not limited. Optionally, the acoustic model adopts a traditional GMM-HMM or DNN-HMM structure, and its specific model structure and model parameter embodiments are not limited.

[0059] Step 204 : Obtain a first pronunciation probability of each first phoneme and a first sequence containing each first phoneme according to the acoustic score and the text data.

[0060] After obtaining the acoustic score for each audio frame in the audio data, a first sequence and a first pronunciation probability can be obtained based on the acoustic score and the text data. In one embodiment, an audio frame in the audio data that pronounces a first phoneme is determined based on the acoustic score, and then aligned with the first phoneme in the corresponding text data. The first start and end times of the first phoneme are obtained based on the alignment result. Subsequently, a first sequence is formed from each first phoneme and the corresponding first start and end times. Subsequently, the probability of the first phoneme in the acoustic score for each audio frame within the first start and end time is determined and used as the first pronunciation probability.

[0061] In an embodiment, this step may specifically include steps 2041 to 2043:

[0062] Step 2041: Obtain an alignment network model for the text data, where each text data corresponds to an alignment network model.

[0063] The alignment network model is used to align text data and audio data. In the embodiment, a WFST network is used as the alignment network model, and each piece of text data corresponds to an alignment network model.

[0064] Exemplarily, for each piece of text data, a corresponding alignment network model can be generated through offline compilation. In this step, the offline generated alignment network model can be directly obtained, wherein the generation rules of the alignment network model are not limited in the embodiment. It is understandable that since the alignment network model is used to align audio data and text data, the specific structure and model parameters of the alignment network model can be set according to the text data so that the audio data and text data can be aligned.

[0065] Step 2042: The acoustic score of each audio frame is used as input information and input into the corresponding alignment network model to obtain a first sequence containing each first phoneme, where the first phoneme is a phoneme in the text data, and the first sequence also includes the first start and end time of each first phoneme in the audio data.

[0066] In one embodiment, after obtaining an alignment network model for text data, the acoustic scores of each audio frame in the corresponding audio data are input into the alignment network model. In this embodiment, after obtaining an alignment network model generated by offline compilation and inputting the corresponding acoustic scores into the corresponding alignment network model, the alignment network model can determine in which audio frame of the audio data each first phoneme in the text data appears based on the acoustic scores, thereby generating a phoneme sequence. This phoneme sequence arranges the first phonemes in the text data in the order of the corresponding audio frames. During the processing, the alignment network model uses a Viterbi search method to obtain the phoneme sequence. It should be noted that the Viterbi search method can also be understood as a Viterbi algorithm. It is understood that in addition to sound, audio data may also contain silence, emptiness, etc. In this case, phonemes corresponding to silence, emptiness, etc. can be set and referenced when constructing the alignment network model. This ensures that the generated phoneme sequence includes phonemes representing silence, emptiness, etc. in addition to the first phoneme. Subsequently, based on the audio frames corresponding to each phoneme in the phoneme sequence, the first start and end times of the first phoneme are obtained, thereby forming the first sequence. For example, if the text content in the text data is "we", the first phoneme includes w o3 m en1, and the alignment network model is as follows Figure 3 As shown in , the alignment network model is a WFST network constructed based on "us", which can also clearly identify silent phonemes and empty phonemes. Figure 3 In the example, each edge of the alignment network model has three values: "input:output / weight", which represent input, output, and weight respectively. Sil represents silent phonemes and eps represents empty phonemes. It is understandable that the nodes in the alignment network model may have self-loop paths, such as Figure 3The second node in the example has a self-looping path. When the pronunciation time of w is long, the self-looping path of the second node can absorb multiple audio frames. When the alignment network model is working, each node moves to the next node according to the direction of the path. During the movement, the acoustic score of the audio frame needs to be combined. For example, the audio data and text data are both described as "we". The audio data containing "we" is feature extracted to obtain audio features, and the audio features are input into the acoustic model to calculate the acoustic score. After that, the calculated acoustic score is input as input information into the alignment network model corresponding to the text data. In the process of the first node in the alignment network model moving along the first path to the second node, the acoustic score P(o|w) of the audio frame is required, where o is the audio feature of the current audio frame and w is the corresponding phoneme. For the first path of the alignment network model, w can also be understood as the pronunciation unit on the first path, and the pronunciation unit can also be understood as a phoneme. P(o|w) represents the probability that the current audio frame is pronounced as the pronunciation unit of the current path. During the node movement process, the acoustic score P(o|w) is weighted and summed with the weight "weight" on the first path to obtain the total score to reach the second node. If the total score is within a reasonable range (the range can be set according to the actual situation), it means that the path is a feasible path, and the total score is retained. After the acoustic scores of all audio frames are used in the alignment network model, the total score of each node is calculated, and the optimal path from the first node to the last node is selected based on the total score. The phonemes input to each node in the optimal path are the pronunciation phonemes. The input phonemes of each node in the optimal path are obtained, and the input sequence is generated and aligned. For example, Figure 3 In the example, the input sequence is aligned by the alignment network model to wwwww o3 o3 o3 sil sil mmm en1 en1 en1, a total of 16 frames. The phonemes corresponding to the first to fifth frames in the audio data are w, the phonemes corresponding to the sixth to eighth frames are o3, and the ninth and tenth frames are silence. The phonemes corresponding to the eleventh to thirteenth frames are m, and the phonemes corresponding to the fourteenth to sixteenth frames are en1. The alignment process is completed, and the first sequence is generated as follows:

[0067] Step 2043: Determine the first pronunciation probability of each first phoneme at each moment within the corresponding first start and end time according to the acoustic score.

[0068] In one embodiment, after obtaining the first sequence, the probability that the audio frame belongs to the first phoneme can be determined based on the acoustic score corresponding to the audio frame. Then, the probability is used as the first pronunciation probability of the first phoneme in the audio frame. According to the above method, the first pronunciation probability of each first phoneme in each audio frame (i.e., each moment) within the corresponding first start and end time can be determined. For example, the first start and end time of a first audio is the moment corresponding to the 5th frame and the moment corresponding to the 10th frame. Then, the first audio has a corresponding first pronunciation probability at each moment from the 5th frame to the 10th frame. In this embodiment, the first pronunciation probability is expressed as P1(p t |f t ), where p t is the first phoneme corresponding to the tth moment, f t is the audio feature corresponding to the audio frame at time t, P1(p t |f t ) indicates that according to f t Determine the phoneme at time t as p t probability.

[0069] It can be understood that, in the embodiment, the network model required for the alignment process is composed of the acoustic model and the alignment network model.

[0070] Step 205: Extract audio features of each audio frame in the audio data.

[0071] Step 206 : Calculate an acoustic score of the corresponding audio frame based on the audio features. The acoustic score is used to represent the probability that the audio frame belongs to each preset phoneme. The preset phonemes include a first phoneme and a second phoneme.

[0072] It is understandable that steps 205 and 206 are the same process as steps 202 and 203. In actual applications, steps 205 and 206 may be omitted and steps 202 and 203 may be used directly. In this case, steps 204 and 207 are performed simultaneously or sequentially after steps 202 and 203 are executed.

[0073] Step 207: Obtain a second pronunciation probability of each second phoneme and a second sequence containing each second phoneme according to the acoustic score of the audio frame.

[0074] After obtaining the acoustic score for each audio frame in the audio data, a second sequence and a second pronunciation probability can be obtained based on the acoustic score. In one embodiment, a pre-built recognition network model can be used to identify the acoustic score to obtain the second phoneme with the highest pronunciation probability for each audio frame in the audio data, thereby obtaining the second sequence. After speech recognition, the pronunciation probability corresponding to each second phoneme, i.e., the second pronunciation probability, can also be obtained.

[0075] In one embodiment, this step specifically includes steps 2071 to 2072:

[0076] Step 2071: Use the acoustic scores of each audio frame as input information and input them into a pre-constructed recognition network model to obtain a second sequence containing each second phoneme, where the second phoneme is the phoneme obtained after recognizing the audio data.

[0077] The recognition network model is used to recognize the second phoneme corresponding to each audio frame in the audio data. In one embodiment, a WFST network is used as the recognition network model, and each audio data shares one recognition network model. In one embodiment, during the construction of the recognition network model, it is first necessary to train the recognition network model through a large amount of audio data and the phonemes it contains. During use, the acoustic scores of each audio frame in the audio data are input into the recognition network model to generate a phoneme sequence, which is used to sequentially describe the second phoneme corresponding to each audio frame according to the playback time. The second sequence can be obtained based on this phoneme sequence. It can be understood that both the recognition network model and the alignment network model are WFST networks, and the principles of the two network models during processing are the same, which will not be elaborated in this embodiment. It should be noted that since the recognition network model can recognize a large amount of audio data, while the alignment network model only processes the audio data corresponding to the corresponding text data, the recognition network model is much larger than the alignment network model. In one embodiment, during the recognition process of the audio data by the recognition network model, there is no text data for comparison, and it is only the result of recognizing the audio data. Therefore, the second sequence determined based on the recognition network model may be different from the first sequence determined based on the alignment network model. For example, if the audio data is "我" (I), and the corresponding text data is "我们" (we), the recognition network model does not recognize the second phoneme corresponding to "们" (men) after recognition. Therefore, the result of the second sequence it outputs is w o3, and the first sequence obtained after the audio data is processed by the alignment network model At this time, the phonemes appearing in the second sequence and the first sequence are not equal.

[0078] It can be understood that combining the recognition network model and the acoustic model results in the network model required for performing speech recognition on the audio data to obtain the second sequence.

[0079] Step 2072: Determine the second pronunciation probability of each second phoneme within the corresponding time according to the acoustic scores.

[0080] In one embodiment, after obtaining the second sequence, the probability that the audio frame belongs to the second phoneme can be determined based on the acoustic score corresponding to the audio frame. Then, the probability is used as the second pronunciation probability of the second phoneme in the audio frame. In the above manner, the second pronunciation probability of each second phoneme in each audio frame (i.e., each moment) can be determined. In this embodiment, the second pronunciation probability is expressed as P2(q t |f t ), where q t is the second phoneme corresponding to the tth moment, f t is the audio feature corresponding to the audio frame at time t, P2(q t |f t ) is passed through f t Determine the tth moment as q t probability.

[0081] Step 208: Determine the third start and end time of each text in the text data in the audio data based on the first start and end time of each first phoneme in the first sequence.

[0082] The third start and end time includes the time when the corresponding text in the text data starts to be pronounced and the time when the pronunciation ends during the playback of the audio data. In one embodiment, after obtaining the first start and end time of each first phoneme in the first sequence, the start and end time of each text in the audio data can be determined according to the first phoneme contained in each text in the text data, that is, the third start and end time. For example, if the text content in the text data is "we", the first sequence is Since the first phonemes contained in the "I" text are w and o3, the third start and end time of the "I" text can be determined based on the first start and end time of w and o3. Therefore, it can be obtained that the third start and end time of the "I" text in the audio data is the first frame to the eighth frame.

[0083] Step 209 : Determine a third pronunciation probability of the text based on a first pronunciation probability of the first phoneme within the third start and end time and a second pronunciation probability of the second phoneme within the third start and end time, wherein the second phoneme within the third start and end time is determined by the second sequence.

[0084] In one embodiment, after obtaining the third start and end time of each text in the audio data, the first pronunciation probability of each first phoneme within the third start and end time can be determined in the first sequence, and the second pronunciation probability of each second phoneme within the third start and end time can be determined in the second sequence. In one embodiment, the first sequence and the second sequence can be segmented according to the third start and end time. For example, if the third start and end time of a text is from frame 10 to frame 28, when segmenting the first sequence and the second sequence, frames 10 to 28 of the two sequences are used as a set of segmentation results and correspond to the text.

[0085] Afterwards, the third pronunciation probability of the text in the audio data is calculated based on the first pronunciation probability and the second pronunciation probability. In one embodiment, the calculation formula of the third pronunciation probability is:

[0086]

[0087] Among them, P3 is the third pronunciation probability of the text, T is the third start and end time of the text, t is the tth moment in the third start and end time, and p t The first phoneme corresponding to time t, f t is the audio feature corresponding to the audio frame at time t, P1(p t |f t ) is p t The probability of the first pronunciation at time t, q t is the second phoneme corresponding to the tth moment, P2(q t |f t ) is q t The second pronunciation probability at time t. As can be seen from the above, in this embodiment, the average of the ratios of the first pronunciation probability to the second pronunciation probability at each time within the third start and end time is used as the third pronunciation probability corresponding to the third start and end time. In this case, each text in the text data corresponds to a third pronunciation probability.

[0088] For example, if the text content in the text data is "we", the first sequence is The second sequence is w o3. Since the first phonemes contained in the "I" text are w and o3, the third start and end time in the audio data is from the first frame to the eighth frame. Therefore, the first pronunciation probability of w and o3 and the second pronunciation probability of w and o3 in the time period from the first frame to the eighth frame are obtained. Then, based on the first pronunciation probability of w and o3 and the second pronunciation probability of w and o3, the third pronunciation probability of the "I" text can be calculated using the above formula.

[0089] It can be understood that the higher the third pronunciation probability is, the more consistent the text obtained by recognizing the audio data is with the text in the corresponding text data.

[0090] Step 210: Find a first data subset in the audio data set based on the third pronunciation probability, wherein the number of text data in the first data subset whose third pronunciation probability is less than the first target probability reaches a first number.

[0091] In this embodiment, the third pronunciation probability of each text in the text data is compared with a set probability value. If the third pronunciation probability of the text is less than the set probability value, it indicates that the probability of the text appearing in the audio data is low, which in turn indicates that there is a high probability that the text is incorrectly labeled, or there is a high probability that the phoneme corresponding to the text in the audio data has not been accurately recognized. In this embodiment, the set probability value is recorded as the first target probability. The first target probability can be set according to actual needs, and this embodiment does not limit the specific value of the first target probability.

[0092] Furthermore, if the third pronunciation probability of some texts in the text data is less than the first target probability, it means that the probability of the text data accurately describing the audio data is low, that is, the consistency between the audio data and the text data is not high, which may affect the training effect of the subsequent acoustic model. Therefore, in the embodiment, it is necessary to find this type of text data in the audio data set, and form this type of text data and the corresponding audio data into a data set. In the embodiment, the data set is recorded as the first data subset. In one embodiment, this type of text data is found by setting a first quantity. For example, after setting the first quantity, the third pronunciation probability of each text in the text data is compared with the first target probability. If the number of texts lower than the first target probability reaches the first quantity (i.e., equal to or greater than the first quantity), it is determined that the text data and the corresponding audio data are added to the first data subset. Among them, the specific value of the first quantity can be set according to the actual situation. For example, a unified first quantity is set. At this time, the number of texts in each text data whose third pronunciation probability is lower than the first target probability is compared with the first quantity. For another example, the first quantity is determined in combination with the text length of the text data. At this time, each text data corresponds to a first quantity. For example, when the text data is long, a larger first quantity can be set, and when the text data is short, a smaller first quantity can be set.

[0093] Step 211: Delete the text data and the corresponding audio data contained in the first data subset from the audio data set.

[0094] Since the accuracy of text data annotation in the first data subset is too low, that is, there are errors and omissions in the annotated text data, in order to avoid the subsequent audio training set being mixed with too much data with low accuracy, which affects the training effect of the acoustic model, therefore, in the embodiment, in the audio data set, the text data and the corresponding audio data contained in the first data subset are deleted.

[0095] It is understandable that in actual applications, the first data subset may not be set. In this case, the third pronunciation probability of each text in the text data is compared with the first target probability. If the number of texts lower than the first target number reaches the first number (that is, equal to or greater than the first number), the text data and the corresponding audio data are directly deleted.

[0096] Step 212: Find a second data subset in the audio data set based on the third pronunciation probability, wherein the third pronunciation probability of each text in the text data included in the second data subset is greater than the second target probability, and the second target probability is greater than the first target probability.

[0097] In this embodiment, the third pronunciation probability of each text in the text data is compared with another set probability value. If the third pronunciation probability of the text is greater than the set probability value, it indicates that the text has a high probability of appearing in the audio data, and therefore, it indicates that the text is likely to be correctly labeled. In this embodiment, the set probability is recorded as the second target probability. The second target probability can be set according to actual needs. In this embodiment, the specific value of the second target probability is not limited. Generally speaking, the second target probability is greater than the first target probability.

[0098] Furthermore, if the third pronunciation probability of each text in the text data is greater than the second target probability, it means that the text data accurately describes the audio data, that is, the content of the text data and the audio data are consistent. It should be noted that for the pre-built acoustic model, the acoustic model can already accurately identify the correct text data and audio data. When the acoustic model is subsequently trained using the screened audio training set, this part of the data has little effect on improving the acoustic model. Therefore, when constructing the audio training set, the number of correct text data and audio data can be reduced, and there is no need to set an excessive amount of text data and audio data, thereby improving the training speed of the acoustic model. Therefore, in the embodiment, it is necessary to find such correct text data in the audio data set, and form such text data and corresponding audio data into a data set. In the embodiment, this data set is recorded as the second data subset. In one embodiment, the third pronunciation probability of each text in the text data is compared with the second target probability. If the third pronunciation probability of each text is higher than the second target probability, it is determined that the text data and the corresponding audio data are added to the second data subset. Moreover, for audio data with low accuracy in text data annotation (such as data other than the second data subset and the first data subset), the pre-built acoustic model cannot accurately identify this type of data. If the recognition effect of the acoustic model is poorly improved after adding this type of audio data and the corresponding text data to the audio training set, it means that there is an error in the annotation of this type of text data (that is, the text data is inconsistent with the audio data), and it is necessary to re-correct this type of text data (optionally, manual correction is used) so that the audio data corresponds to the corresponding corrected text data, and the corrected audio data and text data are added back to the audio training set, and the acoustic model is re-trained to ensure the recognition effect of the acoustic model.

[0099] Step 213: Delete the second amount of text data and corresponding audio data in the second data subset from the audio data set.

[0100] In one embodiment, since the probability that the annotated text in the text data of the second data subset is correct is relatively high, when constructing the audio training set, some text data and audio data can be selected as the audio training set to reduce the amount of learning data and save training time when training the acoustic model. In the embodiment, the selection of text data and audio data in the second data subset is achieved by the second quantity. The specific value of the second quantity can be set according to actual conditions, such as the second quantity is a fixed quantity, or the second quantity is determined according to the number of text data contained in the second data subset. Further, when deleting the second quantity of text data and corresponding audio data in the second data subset, the second quantity of text data and corresponding audio data can be randomly deleted, or the text data and corresponding audio data with a small third pronunciation probability can be deleted, or other rules can be used to delete the second quantity of text data and corresponding audio data.

[0101] Step 214: Use the text data retained in the audio dataset as target text data, and use the audio data corresponding to the target text data as target audio data. The target audio dataset and the target annotated text constitute an audio training set.

[0102] In one embodiment, after deleting the second amount of text data and audio data in the first data subset and the second data subset in the audio data set, the text data retained in the audio data set is used as the target text data, and the audio data corresponding to the target text data is used as the target audio data. Thereafter, an audio training set is constructed based on the target text data and the target audio data, and the audio training set is used to train the pre-constructed acoustic model again to improve the training results of the acoustic model.

[0103] The above-mentioned method extracts audio features from audio data, inputs the audio features into the acoustic model to obtain acoustic scores corresponding to the audio features, and inputs the acoustic scores into the alignment network model corresponding to the text data and the recognition network model corresponding to the audio data, respectively, to obtain a first sequence containing each first phoneme and a second sequence containing each second phoneme, and calculates the first pronunciation probability of the first phoneme and the second pronunciation probability of the second phoneme based on the acoustic scores, the first sequence and the second sequence, calculates the third pronunciation probability of each text according to the first pronunciation probability and the second pronunciation probability, and selects the target text data and the corresponding audio data from the audio data set according to the third pronunciation probability to construct an audio training set. This technical solution solves the technical problem that artificially constructed annotated texts are prone to errors and affect the training effect of the acoustic model. The third pronunciation probability can be used to clarify whether the text data is consistent with the audio data, that is, to clarify whether the text data is accurately identified. Afterwards, by deleting the first data subset with too low text data accuracy and the second data subset with too high text data accuracy from the audio dataset, the number of labeling errors in the audio training set was reduced, ensuring the training effect of the acoustic model. At the same time, only a portion of the audio training set with completely correct labeling was retained to save training time and improve the training efficiency of the acoustic model. In addition, the acoustic model and alignment network model can accurately align audio data and text data, and the acoustic model and recognition network model can accurately realize text recognition of audio data.

[0104] Figure 4 For a structural diagram of an audio data screening device provided in an embodiment of the present application, please refer to Figure 4 The device includes: a data set acquisition module 301, an alignment module 302, a recognition module 303, a pronunciation probability determination module 304, and an audio data screening module 305.

[0105] Among them, the data set acquisition module 301 is used to acquire an audio data set, which includes multiple audio data and text data corresponding to each audio data; the alignment module 302 is used to align the audio data and the corresponding text data to obtain the first pronunciation probability of each first phoneme and the first sequence containing each first phoneme, the first phoneme is the phoneme in the text data, and the first sequence also contains the first start and end time of each first phoneme in the audio data; the recognition module 303 is used to perform speech recognition on the audio data to obtain the second pronunciation probability of each second phoneme and the second sequence containing each second phoneme, the second phoneme is the phoneme obtained after recognizing the audio data; the pronunciation probability determination module 304 is used to determine the third pronunciation probability of each text in the text data in the audio data based on the first sequence, the first pronunciation probability, the second sequence and the second pronunciation probability; the audio data screening module 305 is used to screen out the target audio data and the corresponding target text data in the audio data set based on the third pronunciation probability, and the target audio data set and the target text data constitute an audio training set.

[0106] Based on the above embodiment, the alignment module 302 includes: a first audio feature extraction submodule, which is used to extract audio features of each audio frame in the audio data; a first acoustic score calculation submodule, which is used to calculate the acoustic score of the corresponding audio frame based on the audio features, and the acoustic score is used to represent the probability that the audio frame belongs to each preset phoneme, and the preset phonemes include a first phoneme and a second phoneme; a first sequence construction submodule, which is used to obtain the first pronunciation probability of each first phoneme and the first sequence containing each first phoneme based on the acoustic score and text data.

[0107] Based on the above embodiment, the first sequence construction submodule includes: an alignment network model unit, which is used to obtain an alignment network model of text data, and each text data corresponds to an alignment network model; a first sequence acquisition unit, which is used to take the acoustic score of each audio frame as input information and input it into the corresponding alignment network model to obtain a first sequence containing each first phoneme; a first pronunciation probability determination unit, which is used to determine the first pronunciation probability of each first phoneme at each moment within the corresponding first start and end time based on the acoustic score.

[0108] Based on the above embodiment, the recognition module 303 includes: a second audio feature extraction submodule, which is used to extract audio features of each audio frame in the audio data; a second acoustic score calculation submodule, which is used to calculate the acoustic score of the corresponding audio frame based on the audio features, and the acoustic score is used to represent the probability that the audio frame belongs to each preset phoneme, and the preset phonemes include a first phoneme and a second phoneme; a second sequence construction submodule, which is used to obtain a second pronunciation probability of each second phoneme and a second sequence containing each second phoneme based on the acoustic score of the audio frame.

[0109] Based on the above embodiment, the second sequence construction submodule includes: a second sequence acquisition unit, which is used to take the acoustic score of each audio frame as input information and input it into a pre-built recognition network model to obtain a second sequence containing each second phoneme; a second pronunciation probability determination unit, which is used to determine the second pronunciation probability of each second phoneme at the corresponding moment according to the acoustic score.

[0110] Based on the above embodiment, the first acoustic score calculation submodule and the second acoustic score calculation submodule are specifically configured to input the audio features into a pre-built acoustic model so that the acoustic model outputs acoustic scores corresponding to the audio features.

[0111] On the basis of the above embodiment, the pronunciation probability determination module 304 includes: a third start and end time determination submodule, which is used to determine the third start and end time of each text in the text data in the audio data based on the first start and end time of each first phoneme in the first sequence; a third pronunciation probability determination submodule, which is used to determine the third pronunciation probability of the text based on the first pronunciation probability of the first phoneme within the third start and end time and the second pronunciation probability of the second phoneme within the third start and end time, and the second phoneme within the third start and end time is determined by the second sequence.

[0112] Based on the above embodiment, the calculation formula of the third pronunciation probability is:

[0113]

[0114] Among them, P3 is the third pronunciation probability of the text, T is the third start and end time of the text, t is the tth moment in the third start and end time, p t is the first phoneme corresponding to the tth moment, f t is the audio feature corresponding to the audio frame at time t, P1(p t |f t ) is p t The probability of the first pronunciation at time t, q t is the second phoneme corresponding to the tth moment, P2(q t |f t ) is q t The probability of the second pronunciation at time t.

[0115] Based on the above embodiment, the audio data screening module 305 includes: a first data subset search submodule, which is used to find the first data subset in the audio data set according to the third pronunciation probability, and the number of texts in each text data contained in the first data subset whose third pronunciation probability is less than the first target probability reaches a first number; a first data subset deletion submodule, which is used to delete the text data and the corresponding audio data contained in the first data subset in the audio data set; a target audio confirmation submodule, which is used to use the text data retained in the audio data set as the target text data, and the audio data corresponding to the target text data as the target audio data.

[0116] Based on the above embodiment, the audio data screening device also includes: a second subset determination module, which is used to delete the text data and the corresponding audio data contained in the first data subset in the audio data set, and then find a second data subset in the audio data set according to the third pronunciation probability, and the third pronunciation probability of each text in the text data contained in the second data subset is greater than the second target probability, and the second target probability is greater than the first target probability; a second subset deletion module, which is used to delete the second amount of text data and the corresponding audio data in the second data subset in the audio data set.

[0117] The audio data screening device described above can be used to execute any audio data screening method and has corresponding functions and beneficial effects.

[0118] It is worth noting that in the embodiment of the above-mentioned audio data screening device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application.

[0119] Figure 5 A schematic diagram of the structure of the audio data screening device provided in the embodiment of the present application is shown as follows: Figure 5 As shown, the audio data screening device includes a processor 400, a memory 401, an input device 402, and an output device 403; the number of processors 400 in the audio data screening device can be one or more. Figure 5 In the audio data screening device, the processor 400, the memory 401, the input device 402, and the output device 403 can be connected via a bus or other means. Figure 5 The bus connection is taken as an example.

[0120] The memory 401, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the audio data screening method in the embodiments of the present application (for example, the data set acquisition module 301, alignment module 302, recognition module 303, pronunciation probability determination module 304, and audio data screening module 305 in the audio data screening device). The processor 400 executes the various functional applications and data processing of the audio data screening device by running the software programs, instructions, and modules stored in the memory 401, thereby implementing the above-mentioned audio data screening method.

[0121] The memory 401 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data generated based on the use of the audio data screening device, etc. Furthermore, the memory 401 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some instances, the memory 401 may further include memory remotely located relative to the processor 400, and these remote memories may be connected to the audio data screening device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0122] The input device 402 may be used to receive input digital or character information and generate key signal inputs related to user settings and function control of the audio data screening device. The output device 403 may include a display device such as a display screen. In addition, the audio data screening device may also include a communication device to enable communication with external devices.

[0123] The audio data screening device described above includes corresponding audio data screening means, which can be used to execute any audio data screening method and has corresponding functions and beneficial effects.

[0124] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and equipment can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0125] In addition, an embodiment of the present application also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform relevant operations in the audio data screening method provided in any embodiment of the present application, and have corresponding functions and beneficial effects.

[0126] Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products.

[0127] Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0128] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-permanent storage in a computer-readable medium, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0129] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0130] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0131] The specific embodiments described above further illustrate the objectives, technical solutions, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this application by those skilled in the art should be included within the scope of protection of this application.

Claims

1. A method for screening audio data, characterized in that: include: Acquire an audio data set, the audio data set including multiple audio data and text data corresponding to each audio data, wherein the text data is manually constructed based on the corresponding audio data; After aligning the audio data with the corresponding text data, obtaining a first pronunciation probability of each first phoneme and a first sequence containing each first phoneme, where the first phoneme is a phoneme in the text data, and the first sequence further includes a first start and end time of each first phoneme in the audio data; After performing speech recognition on the audio data, a second pronunciation probability of each second phoneme and a second sequence including each second phoneme are obtained, where the second phoneme is the phoneme obtained after recognizing the audio data; Determine a third pronunciation probability of each text in the text data in the audio data according to the first sequence, the first pronunciation probability, the second sequence, and the second pronunciation probability; According to the third pronunciation probability, target audio data and corresponding target text data are screened out from the audio data set, and the target audio data set and the target text data constitute an audio training set.

2. The audio data screening method according to claim 1, wherein: After aligning the audio data with the corresponding text data, obtaining a first pronunciation probability of each first phoneme and a first sequence containing each first phoneme includes: Extracting audio features of each audio frame in the audio data; Calculating an acoustic score of the corresponding audio frame according to the audio feature, the acoustic score being used for a probability that the audio frame belongs to each preset phoneme, the preset phonemes including the first phoneme and the second phoneme; A first pronunciation probability of each first phoneme and a first sequence containing each first phoneme are obtained according to the acoustic score and the text data.

3. The audio data screening method according to claim 2, characterized in that: The step of obtaining a first pronunciation probability of each first phoneme and a first sequence containing each first phoneme according to the acoustic score and the text data includes: Obtaining an alignment network model for the text data, where each text data corresponds to an alignment network model; Using the acoustic score of each of the audio frames as input information and inputting it into a corresponding alignment network model to obtain a first sequence including each first phoneme; A first pronunciation probability of each first phoneme at each moment within the corresponding first start and end time is determined according to the acoustic score.

4. The audio data screening method according to claim 1, wherein: After performing speech recognition on the audio data, obtaining a second pronunciation probability of each second phoneme and a second sequence containing each second phoneme includes: Extracting audio features of each audio frame in the audio data; Calculating an acoustic score of the corresponding audio frame according to the audio feature, the acoustic score being used to represent a probability that the audio frame belongs to each preset phoneme, the preset phonemes including the first phoneme and the second phoneme; A second pronunciation probability of each second phoneme and a second sequence containing each second phoneme are obtained according to the acoustic score of the audio frame.

5. The audio data screening method according to claim 4, characterized in that: The obtaining, according to the acoustic score of the audio frame, a second pronunciation probability of each second phoneme and a second sequence containing each second phoneme comprises: Inputting the acoustic score of each of the audio frames as input information into a pre-built recognition network model to obtain a second sequence containing each second phoneme; A second pronunciation probability of each of the second phonemes at a corresponding time is determined according to the acoustic score.

6. The audio data screening method according to claim 2 or 4, characterized in that: Calculating the acoustic score of the corresponding audio frame according to the audio feature includes: The audio features are input into a pre-built acoustic model so that the acoustic model outputs an acoustic score corresponding to the audio features.

7. The audio data screening method according to claim 1, wherein: Determining a third pronunciation probability of each text in the text data in the audio data according to the first sequence, the first pronunciation probability, the second sequence, and the second pronunciation probability includes: determining a third start and end time of each text in the text data in the audio data according to the first start and end time of each first phoneme in the first sequence; A third pronunciation probability of the text is determined according to a first pronunciation probability of the first phoneme within the third start and end time and a second pronunciation probability of the second phoneme within the third start and end time, wherein the second phoneme within the third start and end time is determined by the second sequence.

8. The audio data screening method according to claim 7, characterized in that: The calculation formula of the third pronunciation probability is: Among them, P3 is the third pronunciation probability of the text, T is the third start and end time of the text, t is the tth moment in the third start and end time, p t is the first phoneme corresponding to the tth moment, f t is the audio feature corresponding to the audio frame at time t, P1(p t |f t ) is p t The probability of the first pronunciation at time t, q t is the second phoneme corresponding to the tth moment, P2(q t |f t ) is q t The probability of the second pronunciation at time t.

9. The audio data screening method according to claim 1, wherein: The step of screening out target audio data and corresponding target text data from the audio data set according to the third pronunciation probability includes: Searching for a first data subset in the audio data set according to the third pronunciation probability, wherein the number of texts in each text data included in the first data subset whose third pronunciation probability is less than the first target probability reaches a first number; In the audio data set, deleting the text data and the corresponding audio data included in the first data subset; The text data retained in the audio data set is used as target text data, and the audio data corresponding to the target text data is used as target audio data.

10. The audio data screening method according to claim 9, characterized in that: In the audio data set, after deleting the text data and the corresponding audio data included in the first data subset, the method further includes: Searching for a second data subset in the audio data set according to the third pronunciation probability, wherein the third pronunciation probability of each text in each text data included in the second data subset is greater than a second target probability, and the second target probability is greater than the first target probability; In the audio data set, a second amount of text data and corresponding audio data in the second data subset are deleted.

11. An audio data screening device, characterized in that: The device comprises: A data set acquisition module is used to acquire an audio data set, wherein the audio data set includes multiple audio data and text data corresponding to each audio data, wherein the text data is manually constructed based on the corresponding audio data; an alignment module, configured to align the audio data with the corresponding text data to obtain a first pronunciation probability of each first phoneme and a first sequence containing each first phoneme, wherein the first phoneme is a phoneme in the text data, and the first sequence further includes a first start and end time of each first phoneme in the text data; a recognition module configured to perform speech recognition on the audio data to obtain a second pronunciation probability of each second phoneme and a second sequence containing each second phoneme, where the second phoneme is a phoneme obtained after recognizing the audio data; a pronunciation probability determination module, configured to determine a third pronunciation probability of each text in the text data in the audio data based on the first sequence, the first pronunciation probability, the second sequence, and the second pronunciation probability; The audio data screening module is used to screen out target audio data and corresponding target text data from the audio data set according to the third pronunciation probability, and the target audio data set and the target text data constitute an audio training set.

12. An audio data screening device, characterized in that: including a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the audio data screening method according to any one of claims 1 to 10 according to the instructions in the program code.

13. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the audio data screening method according to any one of claims 1 to 10 is implemented.

Images