An automated speech data verification method for speech synthesis

Abstract

A method for automatically verifying speech data for speech synthesis comprises the following steps: step 1: a professional voice actor records an audio corpus in a professional recording studio; step 2: after the voice actor has recorded a complete audio segment, the audio editing process begins; step 3: after the entire recorded audio segment is cropped, each short audio segment is automatically recognized to obtain the text corresponding to each audio file; step 4: the obtained line audio data and the corresponding line text are verified for multiple or fewer words and processed; step 5: the obtained line audio data and the corresponding line text are checked for typos and corrected to ensure that the training data and labels are free of errors and outliers. The present invention utilizes deep learning technology, and according to the characteristics of the speech synthesis data processing flow, combines natural language processing and signal processing methods to automatically verify and clean the speech synthesis data to improve the efficiency and accuracy of speech synthesis data processing.

Description

Technical Field

[0001] The invention belongs to the field of speech synthesis, in particular to an automatic speech data verification method oriented to speech synthesis. Background Art

[0002] In recent years, with the rapid development of deep learning, artificial intelligence technology has become ubiquitous in all aspects of life. Speech synthesis technology has also made unprecedented progress alongside the advancement of deep learning. Early speech synthesis systems, while able to automatically synthesize speech through recording and splicing, produced an unacceptably robotic sound. However, deep neural network learning based on big data has enabled the synthesis of relatively natural speech. However, compared to traditional splicing speech synthesis systems, speech synthesis methods based on deep neural networks require a vast amount of training data.

[0003] With the emergence of the metaverse concept, virtual digital humans have become a hot topic in artificial intelligence technology. The most significant characteristic of virtual digital humans is their high degree of anthropomorphism, which is particularly important during voice interaction. To enable digital humans in the metaverse to possess rich voice interaction capabilities, such as multiple timbres and multiple emotions, it is necessary to train robust speech synthesis models. However, training these powerful speech synthesis models requires large amounts of high-quality training data. This large-scale, high-quality training data requires voice actors to record large amounts of speech data, which increases both the recording workload and the data processing workload. Furthermore, the rapid creation of a virtual human requires speech synthesis with a specific timbre, which requires rapid acquisition of training data and, in other words, rapid speech data processing. Generating speech synthesis data typically requires significant manual processing, such as segmenting the speech, manually verifying the data, and ultimately generating audio-to-text data for training.

[0004] The main processing flow of traditional speech synthesis data is as follows:

[0005] The first step is recording. After preparing the lines, hire a professional voice actor to record in a professional studio. There are two recording methods. One is to record and cut the audio one by one during the recording, which is slower. The other is to record all the lines and then cut them together later, which makes the subsequent cutting process more difficult.

[0006] The second step is to verify the voice data. Since speech synthesis data requires high data quality and word-for-word accuracy, which is difficult to achieve with traditional speech recognition technology, manual verification is required.

[0007] The last step is data correction. After verifying the data, the erroneous data needs to be corrected, mainly by modifying the text labels of the audio to match the recorded audio. This process also requires a lot of labor costs.

[0008] Currently, the processing of speech synthesis data largely relies on manual labor, which is prohibitively expensive when processing large amounts of text quickly. This is especially true when processing long, emotionally charged text data, which significantly increases the workload. First, the audio recording stage requires extensive manual editing, and voice actors must also coordinate with the recording engineer's editing requirements, significantly reducing recording efficiency. Second, speech verification requires significant labor costs and is relatively inefficient when processing large amounts of text. Although deep learning end-to-end speech synthesis technology can directly convert text into speech, verification data remains crucial to ensure the quality and accuracy of the generated speech. The quality of the training dataset is crucial for the quality of the generated speech. While traditional speech recognition can assist with some speech verification tasks, the high data requirements of speech synthesis do not significantly reduce labor costs, and manual word-by-word verification is still required. Therefore, before training the speech synthesis model, significant labor costs are required to verify and clean the data to remove errors and outliers and ensure data accuracy and consistency. Summary of the Invention

[0009] The present invention provides an automatic speech data verification system and method for speech synthesis, which are used to solve the defects in the prior art.

[0010] The present invention is achieved through the following technical solutions:

[0011] An automated speech data verification method for speech synthesis comprises the following steps:

[0012] Step 1: Professional voice actors record audio materials in a professional recording studio;

[0013] Step 2: After the voice actor has recorded a complete audio clip, they begin to edit the audio to obtain training audio data for subsequent processing and acoustic model training.

[0014] Step 3: After trimming the entire recorded audio, we obtain short audio files one by one. We perform automatic speech recognition on each short audio file to obtain the text corresponding to each audio file. We then perform audio recall processing on the line text according to the order of the lines, and obtain the audio data of each line.

[0015] Step 4: Verify and process the obtained dialogue audio data and the corresponding dialogue text to ensure that the training data and labels are free of errors and outliers.

[0016] Step 5: Check and correct typos in the obtained audio data and the corresponding text to ensure that the training data and labels are free of errors and outliers.

[0017] In the above-mentioned automated voice data verification method for speech synthesis, the text corpus in step 1 is prepared lines, consisting of independent texts. To ensure high recording efficiency, the voice actor needs to master the lines in advance before recording, and the number of lines to be recorded in a single time is set. The recording process is continuous; after each line is recorded, there is a pause of 3 seconds or more. If the voice actor finds that the content is recorded incorrectly, he or she can say "do it again", then pause for 1 second, and then re-record the content of the line.

[0018] In the above-mentioned automatic voice data verification method for speech synthesis, the step 2 adopts a silence boundary detection algorithm, referred to as VAD (Voice Activity Detection); and uses Mel-scale Frequency Cepstral Coefficients (MFCC features) to perform silence detection.

[0019] In the above-mentioned automatic voice data verification method for speech synthesis, the silence detection comprises the following steps:

[0020] Step 1: Calculate the MFCC features corresponding to each frame of audio;

[0021] Step 2: Find the first frame whose MFCC feature is greater than the silence threshold and set it as the sound start frame v0;

[0022] Step 3: For the part where the MFCC features of three or more consecutive frames are all smaller than the silence threshold, the first frame before the first frame that is smaller than the silence threshold is taken as the end frame v1 of the sound;

[0023] Step 4: Clip and save the audio based on the current sound boundary. Note that during the clipping process, in order to facilitate the identification of pauses in the next step as pauses in intonation or between lines, the actual clipped sound end frame is v1+10, unless the end of the audio is less than 10 frames.

[0024] Step 5: After saving the audio in order, continue to loop from step 2.

[0025] In the above-mentioned automatic voice data verification method for speech synthesis, in the step 4, the last frame of the sound is v1+10. If the last frame of the audio is less than 10, no truncation operation is performed, and all the frames at the end of the audio are directly classified as the last cut audio.

[0026] In the above-mentioned method for automatic speech data verification for speech synthesis, the collective operation of performing audio recall processing on the dialogue text according to the dialogue sequence in step 3 includes the following steps:

[0027] Step 1: To merge audio files that have been split due to pauses in speech, first perform silence detection on each audio file. If the last frame of an audio file is not silent, it means that the audio captured there was a pause in speech, and the audio file is marked for processing.

[0028] Step 2: First, take two lines of text to be recalled, text_a and text_b, in the order of the lines of text. Text_a is the audio text that needs to be recalled currently, and text_b is the audio text that needs to be recalled next.

[0029] Step 3: First determine whether wav_b contains "restart", if so, go back to step 2 and start again, otherwise go to the next step;

[0030] Step 4: Take the speech recognition texts wav_a and wav_b of the two audios in order, and use wav_a and wav_b to calculate the similarity distance with text_a and text_b respectively. The similarity distance calculation formula is as follows:

[0031] Match_ab=ED(wav_a,text_b)–|len(wav_a)-len(text_b)|

[0032] Where ED(wav_a,text_b) represents the edit distance between wav_a and text_b, and |len(wav_a)-len(wav_a)| represents the absolute value of the difference in text length between wav_a and text_b.

[0033] Step 5: If audio a is the audio that was edited due to the pause in the first step, determine whether Match_(a+b)b is greater than Match_ab. If Match_(a+b)b is greater than Match_ab, trim the audio frames after the silent frames in audio a, then merge audio a and audio b into a new audio a, merge the text at the same time, and then select the next audio in sequence as audio b. Return to step 3 and start again. If Match_(a+b)b is not greater than Match_ab, trim the audio frames after the silent frames in audio a and proceed to the next step.

[0034] Step 6: Determine the size of Match_aa and Match_ba. If Match_aa is smaller than Match_ba, then change audio b to audio a, select the next audio in sequence as audio b, and return to step 3. Otherwise, compare the size of Match_ba and Match_bb. If Match_ba is smaller than Match_bb and Match_bb is larger than 1 / 2len(wav_b), then audio a is the audio corresponding to text a. Continue to change audio b and text b to audio a and text a, select the next audio in sequence as audio b and text b, and return to step 3. Otherwise, directly select the next audio in sequence as audio b and text b, and return to step 3.

[0035] Step 7: Repeat until all audio lines are recalled.

[0036] In the above-mentioned method for automatic speech data verification for speech synthesis, the method for verifying and processing the dialogue audio data and the corresponding dialogue text for excess and deficiency of words in step 4 includes the following steps:

[0037] Step 1: Use the Transformer_CTC model to train a tone acoustic model with the five tones of pinyin as output. The input is audio and the output is the tone of the correct text.

[0038] Step 2: Input the audio file into the trained Transformer_CTC tone model to obtain the tone output of each audio. At this time, audio a, plus the previous speech recognition result wav_a and the line text text_a, and the tone tone_a output in this step, correspond to three texts;

[0039] Step 3: For audio a, compare text_a, tone_a, and wav_a to identify words that the algorithm always mispronounces and words that the voice actor always mispronounces. If text_a and tone_a are the same length, but different from wav_a, then it is considered that the algorithm made an error. Record the number of words that wav_a has more or less than text_a, and record the frequency. If wav_a and tone_a are the same length, but different from text_a, then it is considered that the voice actor made an error. Record the number of words that text_a has more or less than wav_a, and record the frequency.

[0040] Step 4: Based on the frequency of words that are prone to errors by the voice actor or algorithm, a threshold is selected. Words above the threshold are considered to be habitually misspelled by the voice actor or algorithm.

[0041] Step 5: Compare the lengths of text_a and tone_a again. If they are the same, the word count is correct and no modification is made. If the lengths of text_a and tone_a are different, and the abnormal word belongs to the habitual wrong word of the voice actor / algorithm obtained in the previous step, then determine who is right and who is wrong based on the habitual wrong word, and complete or delete the wrong word in wav_a or text_a based on the habitual wrong word to align the length. Otherwise, use the rules in step 3 to determine whether it is an algorithm error or a voice actor error, and correct wav_a or text_a.

[0042] In the above-mentioned method for automatic speech data verification for speech synthesis, the method for checking and correcting typos in the speech audio data and the corresponding speech text in step 5 includes the following steps:

[0043] Step 1: First, count all the abnormal words that do not correspond to the audio text and WAV, record the frequency, select a threshold based on the frequency, and mark the words above the threshold as habitual errors in the algorithm;

[0044] Step 2: Input the audio a into the speech recognition model to obtain the phoneme prediction vector v_a of the acoustic model for each frame of audio;

[0045] Step 3: Decode and verify the phoneme prediction vector. First, intercept the phoneme prediction vector and decode the first five phoneme values ​​of the corresponding phoneme vector of each frame. If the decoded result completely contains text_a, then text_a is considered correct. Otherwise, proceed to the next step.

[0046] Step 4: If the first five phoneme values ​​of the phoneme vector corresponding to each frame are decoded and a word in text_a cannot be decoded, and the tone of the word is consistent with the tone prediction model, if the word is a habitual error in the algorithm, then text_a is considered correct. Otherwise, the first 10 phoneme values ​​of the phoneme vector corresponding to each frame are decoded. If the decoded result completely contains text_a, then text_a is considered correct. Otherwise, the speech recognition prediction result wav_a is used to modify text_a.

[0047] Step 5: If the first five phoneme values ​​of the phoneme vector corresponding to each frame are decoded and a word in text_a cannot be decoded, and the tone of the word is inconsistent with the tone prediction model, then the speech recognition prediction result wav_a is used to modify text_a.

[0048] The advantages of the present invention are: the present invention utilizes deep learning technology, and based on the characteristics of the speech synthesis data processing process, combines natural language processing and signal processing methods to realize automatic verification and cleaning of speech synthesis data, so as to improve the efficiency and accuracy of speech synthesis data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0050] Figure 1 It is a schematic diagram of the recording process of the present invention;

[0051] Figure 2 It is a schematic diagram of the audio clip silence detection process of the present invention;

[0052] Figure 3 It is a schematic diagram of the effective audio integration process after audio editing of the present invention;

[0053] Figure 4 1. It is a schematic diagram of a process for checking and processing the excess and shortage of words in the speech audio data of the present invention;

[0054] Figure 5 It is a flow chart of checking and correcting typos in word audio data according to the present invention. DETAILED DESCRIPTION

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0056] An automated speech data verification method for speech synthesis comprises the following steps:

[0057] Step A, professional voice actors record audio corpus in a professional recording studio. The text corpus is lines prepared in advance, consisting of independent texts. In order to ensure high recording efficiency, the voice actors need to master the lines in advance before recording, and set the number of lines for single recording. The recording process does not stop. In addition to the audio data corresponding to the normal text, the final recording content may also include other content, such as: pauses in the tone of voice in the text lines, pauses between different lines, voice actor errors that require re-recording of lines, background noise or monologues. In order to facilitate data processing in subsequent steps, the following strategy is adopted for the above-mentioned possible situations: After recording each line, pause for 3 seconds or more. If the voice actor finds that the content is recorded incorrectly, he can say "do it again", then pause for 1 second and re-record the line content.

[0058] The specific recording flow chart is as follows Figure 1 shown.

[0059] In step B, after the voice actor has recorded a complete audio clip, the audio clips are processed to obtain training audio data for subsequent processing and acoustic model training. First, the voice activity detection algorithm, VAD (Voice Activity Detection), is introduced. Mel-scale frequency cepstral coefficients (MFCC features) are used for silence detection. Assume that the audio sampling rate is 16k, the window length is 1024, and the frame shift is 512. Set the silence threshold and perform silence detection according to the following steps:

[0060] 1. For each frame of audio, calculate its corresponding MFCC features.

[0061] 2. Find the first frame whose MFCC feature is greater than the silence threshold and set it as the sound start frame v0.

[0062] 3. For the part where the MFCC features of three or more consecutive frames are all smaller than the silence threshold, the first frame before the first frame that is smaller than the silence threshold is taken as the end frame v1 of the sound.

[0063] 4. Clip and save the audio according to the boundaries of the current sound. Note that when clipping, in order to facilitate the next step of identifying whether the pause is a pause in tone or a pause between lines, the actual last frame of the clipped sound is v1+10 (except when the end of the audio is less than 10 frames).

[0064] 5. After saving the audio in order, continue to loop from step 2.

[0065] The specific flow chart is as follows Figure 2 shown.

[0066] In step C, after trimming the entire recorded audio in the previous step, we obtain short audio files. These short audio files contain the correct line audio, background noise, voice actor narration, and misrecorded line audio. We need to merge the audio that was split apart due to pauses in tone, filter out other useless audio data, and retain the correct line audio. First, perform automatic speech recognition on each short audio file to obtain the text corresponding to each audio file. Then, perform audio recall processing on the line text according to the order of the lines. The specific steps are as follows:

[0067] 1. In order to merge audio files that have been split due to pauses in speech, first perform silence detection on each audio file. If the last frame of the audio file is not a silent frame, it means that the audio captured there was a pause in speech. The audio file is marked for processing.

[0068] 2. First, take two lines of text to be recalled, text_a and text_b, in the order of the lines of text. Text_a is the audio text that needs to be recalled currently, and text_b is the audio text that needs to be recalled next.

[0069] 3. Then first check whether wav_b contains 'restart'. If so, go back to step 2 and start again. Otherwise, go to the next step.

[0070] 4. Take the speech recognition texts wav_a and wav_b of the two audios in order, and use wav_a and wav_b to calculate the similarity distance with text_a and text_b respectively. The similarity distance calculation formula is as follows:

[0071] Match_ab=ED(wav_a,text_b)–|len(wav_a)-len(text_b)|

[0072] Where ED(wav_a,text_b) represents the edit distance between wav_a and text_b, and |len(wav_a)-len(wav_a)| represents the absolute value of the difference in text length between wav_a and text_b.

[0073] 5. If audio a is the audio clipped due to a pause in speech in step 1, determine whether Match_(a+b)b is greater than Match_ab. If so, trim the spoken frames after the silent frames in audio a. Then merge audio a and audio b into a new audio a. Simultaneously merge the text, select the next audio clip in sequence as audio b, and return to step 3. If Match_(a+b)b is not greater than Match_ab, trim the spoken frames after the silent frames in audio a and proceed to the next step.

[0074] 6. Determine the size of Match_aa and Match_ba. If Match_aa is smaller than Match_ba, change audio b to audio a, select the next audio in the order as audio b, and return to step 3. Otherwise, compare Match_ba and Match_bb. If Match_ba is smaller than Match_bb and Match_bb is greater than 1 / 2len(wav_b), then audio a is the audio corresponding to text a. Continue to change audio b and text b to audio a and text a, select the next audio in the order as audio b and text b, and return to step 3. Otherwise, directly select the next audio in the order as audio b and text b, and return to step 3.

[0075] 7. Repeat until all audio lines are recalled.

[0076] The specific algorithm flow chart is as follows Figure 3 shown.

[0077] In step D, we've obtained audio data for each line of dialogue from the previous step. Next, we need to verify the audio and dialogue text to ensure that the training data and labels are free of errors and outliers. Audio text data can generally be divided into four categories: completely correct, audio with too many words, audio with too few words, and audio with typos. This step first detects and processes audio with too many or too few words. The specific steps are as follows:

[0078] 1. Use the Transformer_CTC model to train a tone acoustic model with the five tones of pinyin as output. The input is audio and the output is the tone of the correct text.

[0079] 2. Input the audio file into the trained Transformer_CTC pitch model to obtain the pitch output of each audio. At this time, audio a, plus the previous speech recognition result wav_a and the line text text_a, and the tone tone_a output in this step correspond to three texts.

[0080] 3. For audio a, compare text_a, tone_a, and wav_a to identify words consistently mispronounced by the algorithm and by the voice actor. If text_a and tone_a are the same length but different from wav_a, it's considered an algorithm error. Record the number of words in wav_a that are greater or less than text_a, and the frequency of such errors. If wav_a and tone_a are the same length but different from text_a, it's considered an error by the voice actor. Record the number of words in text_a that are greater or less than wav_a, and the frequency of such errors.

[0081] 4. Based on the frequency of words prone to errors by the voice actor or algorithm, a threshold is selected. Words above the threshold are considered to be frequently mispronounced by the voice actor or algorithm.

[0082] 5. Compare the lengths of text_a and tone_a again. If they are the same, the word count is correct and no corrections are made. If the lengths of text_a and tone_a are different, and the abnormal word is a habitual error by the voice actor / algorithm obtained in the previous step, determine which is correct based on the habitual error and complete or delete the error in wav_a or text_a to align the lengths. Otherwise, use the rules in step 3 to determine whether it is an algorithm error or a voice actor error, and correct wav_a or text_a.

[0083] The specific processing flow chart is as follows Figure 4 shown.

[0084] Step E: After the previous step, the number of words in the text text_a and wav_a of audio a has been determined to be equal. This step mainly checks for typos and corrects them. The specific steps are as follows:

[0085] 1. First, count all the abnormal words that do not correspond to the audio text and wav, record the frequency, select a threshold based on the frequency, and mark the words above the threshold as habitual errors in the algorithm.

[0086] 2. Input the audio a into the speech recognition model to obtain the phoneme prediction vector v_a of the acoustic model for each frame of audio.

[0087] 3. Perform decoding verification based on the phoneme prediction vector. First, intercept the phoneme prediction vector and take the first 5 phoneme values ​​of the corresponding phoneme vector of each frame for decoding. If the decoded result completely contains text_a, text_a is considered to be correct. Otherwise, proceed to the next step of judgment.

[0088] 4. If the first five phoneme values ​​of each frame's corresponding phoneme vector are decoded and a character in text_a cannot be decoded, and the tone of the character is consistent with the tone prediction model, then text_a is considered correct if the character is a habitual error in the algorithm. Otherwise, the first 10 phoneme values ​​of each frame's corresponding phoneme vector are decoded. If the decoded result completely contains text_a, then text_a is considered correct. Otherwise, the speech recognition prediction result wav_a is used to modify text_a.

[0089] 5. If the first five phoneme values ​​of the corresponding phoneme vector of each frame are decoded and a word in text_a cannot be decoded, and the tone of the word is inconsistent with the tone prediction model, the speech recognition prediction result wav_a is used to modify text_a.

[0090] The specific processing flow chart is as follows Figure 5 shown.

[0091] Example

[0092] Step 1: Professional voice actors record audio materials in a professional recording studio;

[0093] Step 2: After the voice actor has recorded a complete audio clip, they begin to edit the audio clips to obtain training audio data one by one.

[0094] Step 3: After segmenting the audio data, perform speech recognition processing on the short audio, and then combine the lines information to filter and merge the short audio to obtain the corresponding short audio data.

[0095] Step 4: After processing the original short audio, we obtain training data lines and audio data pairs one by one. Then, based on the speech recognition results, tone recognition results, and line information, we verify the text length corresponding to the audio to obtain the correct text length of the audio.

[0096] Step 5: By verifying the length of the text corresponding to the audio, the length of the text is determined. Finally, the text content needs to be verified to obtain the correct text content corresponding to the audio.

[0097] Through the above embodiments, it can be seen that the present invention standardizes the recording and editing process, classifies and statistically analyzes the recording data, and optimizes the pause processing and mispronunciation processing process in the voice actor recording, so as to better guide the voice actor's recording process and improve recording efficiency; the present invention can also automatically detect and delete misrecordings and abnormal data in the speech synthesis data recording, such as re-recordings, background noise, etc.; at the same time, the present invention automatically detects and repairs errors and abnormal values ​​in the speech synthesis data, such as missing words, wrong words, and extra words.

[0098] The present invention can realize automatic verification and cleaning of speech synthesis data, improve processing efficiency and accuracy, reduce labor costs, and provide new ideas and solutions for the development of speech synthesis technology.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An automated speech data verification method for speech synthesis, characterized by: The steps include: 1-1: Professional voice actors record audio materials in a professional recording studio; 1-2: After the voice actor has recorded a complete audio clip, they begin to edit the audio clip to obtain training audio data; 1-3: After trimming the entire recorded audio, we obtain several short audio files. We perform automatic speech recognition on each short audio file to obtain the corresponding text for each audio file. We then perform audio recall processing on the line text based on the line order, obtaining several audio data items. 1-4: Verify and process the obtained dialogue audio data and the corresponding dialogue text to ensure that the training data and labels are free of errors and outliers. 1-5: Check and correct typos in the obtained audio data and the corresponding text to ensure that the training data and labels are free of errors and outliers. The collective operation of performing audio recall processing on the dialogue text according to the dialogue sequence in 1-3 above includes the following steps: 1-3-1: To merge audio files that have been split due to pauses in speech, first perform silence detection on each audio file. If the last frame of the audio file is not silent, it means that the capture was due to a pause in speech, and the audio file is marked for processing. 1-3-2: First, take two lines of text to be recalled, text_a and text_b, in the order of the lines of text. Text_a is the audio text that needs to be recalled currently, and text_b is the audio text that needs to be recalled next. 1-3-3: First check whether wav_b contains "restart". If so, go back to step 1-3-2 and start again. Otherwise, go to step 1-3-4. 1-3-4: Take the speech recognition texts wav_a and wav_b of the two audios in order, and use wav_a and wav_b to calculate the similarity distance with text_a and text_b respectively. The similarity distance calculation formula is as follows: Match_ab=ED(wav_a,text_b)–|len(wav_a)-len(text_b)| Where ED(wav_a,text_b) represents the edit distance between wav_a and text_b, and |len(wav_a)-len(text_b)| represents the absolute value of the difference in text length between wav_a and text_b. 1-3-5: If audio a is the audio that was clipped due to a pause in tone in 1-3-1, determine whether Match_(a+b)b is greater than Match_ab. If Match_(a+b)b is greater than Match_ab, trim the audio frames after the silent frames in audio a, then merge audio a and audio b into a new audio a, merge the text at the same time, and then select the next audio in sequence as audio b. Return to 1-3-3 and start again. If Match_(a+b)b is not greater than Match_ab, trim the audio frames after the silent frames in audio a and proceed to 1-3-6. 1-3-6: Determine the size of Match_aa and Match_ba. If Match_aa is smaller than Match_ba, change audio b to audio a, select the next audio as audio b, and return to 1-3-3. Otherwise, compare Match_ba and Match_bb. If Match_ba is smaller than Match_bb and Match_bb is greater than 1 / 2(len(wav_b)), then audio a is the audio corresponding to text a. Continue to change audio b and text b to audio a and text a, select the next audio as audio b and text b, and return to 1-3-3. Otherwise, directly select the next audio as audio b and text b, and return to 1-3-3. 1-3-7: Loop until all audio lines are recalled.

2. The method for automatic speech data verification for speech synthesis according to claim 1, characterized in that: The Chinese text material 1-1 is the lines prepared in advance and consists of independent texts. In order to ensure high recording efficiency, the voice actor needs to master the lines in advance before recording, and set the number of lines for single recording. The recording process does not stop; after recording each line, pause for 3 seconds or more. If the voice actor finds that the content is recorded incorrectly, he can say "do it again", then pause for 1 second and re-record the line content.

3. The method for automatic speech data verification for speech synthesis according to claim 1, wherein: In the above 1-2, a silence boundary detection algorithm, referred to as VAD, is used; and Mel-frequency cepstral coefficients, referred to as MFCC features, are used for silence detection.

4. The method for automatic speech data verification for speech synthesis according to claim 1, wherein: The silence detection comprises the following steps: 4-1: Calculate the MFCC features corresponding to each frame of audio; 4-2: Find the first frame whose MFCC feature is greater than the silence threshold and set it as the sound start frame v0; 4-3: For the part where the MFCC features of three or more consecutive frames are all smaller than the silence threshold, the frame before the first frame that is smaller than the silence threshold is taken as the end frame v1 of the sound; 4-4: Clip and save the audio according to the current sound boundary. Note that when clipping, in order to easily distinguish whether the pause is a pause in tone or a pause between lines, the actual clipped sound end frame is v1+10; 4-5: After saving the audio in order, continue to loop from 4-2.

5. The method for automatic speech data verification for speech synthesis according to claim 4, characterized in that: The last frame of the sound in 4-4 is v1+10. If the last frame of the audio is less than 10, there is no need to cut it. All the frames at the end of the audio are directly classified as the last cut audio.

6. The method for automatic speech data verification for speech synthesis according to claim 1, characterized in that: The method for checking and processing the dialogue audio data and the corresponding dialogue text for more or less words in 1-4 includes the following steps: 1-4-1: Use the Transformer_CTC model to train a tone acoustic model with the five tones of pinyin as output. The input is audio and the output is the tone of the correct text. 1-4-2: Input the audio file into the trained Transformer_CTC tone model to obtain the tone output of each audio. In this case, audio a corresponds to the speech recognition result wav_a, the line text text_a, and the tone tone_a. 1-4-3: For audio a, identify words that the algorithm mispronounced and words that the voice actor mispronounced by comparing text_a, tone_a, and wav_a. If text_a and tone_a are the same length, but different from wav_a, then it is considered an algorithm error. Record the number of words that wav_a has more or less than text_a, and record the frequency. If wav_a and tone_a are the same length, but different from text_a, then it is considered a voice actor error. Record the number of words that text_a has more or less than wav_a, and record the frequency. 1-4-4: Count the frequency of words incorrectly written by the voice actor or algorithm based on 1-4-3, select a threshold, and words above the threshold are considered to be habitually misspelled by the voice actor / algorithm; 1-4-5: Compare the lengths of text_a and tone_a again. If they are the same, the word count is correct and no modification is made. If the lengths of text_a and tone_a are different, and the abnormal word is a habitual wrong word of the voice actor / algorithm obtained in 1-4-4, then determine who is right and who is wrong based on the habitual wrong word, and complete or delete the wrong word in wav_a or text_a based on the habitual wrong word to align the length. Otherwise, use the 1-4-4 rule to determine whether it is an algorithm error or a voice actor error, and correct wav_a or text_a.

7. The method for automatic speech data verification for speech synthesis according to claim 1, characterized in that: The method for checking and correcting typos in the speech audio data and the corresponding speech text in 1-5 comprises the following steps: 1-5-1: First, count all the abnormal words that do not correspond to the audio text and WAV, record the frequency, select a threshold based on the frequency, and mark the words above the threshold as habitual errors in the algorithm; 1-5-2: Input audio a into the speech recognition model to obtain the phoneme prediction vector v_a of the acoustic model for each frame of audio; 1-5-3: Decode and verify the phoneme prediction vector. First, intercept the phoneme prediction vector and decode the first five phoneme values ​​of the corresponding phoneme vector for each frame. If the decoded result completely contains text_a, text_a is considered correct. Otherwise, proceed to 1-5-4. 1-5-4: If the first five phoneme values ​​of the phoneme vector corresponding to each frame are decoded, and a word in text_a cannot be decoded, and the tone of the word is consistent with the tone prediction model, if the word is a habitual error in the algorithm, then text_a is considered correct. Otherwise, the first 10 phoneme values ​​of the phoneme vector corresponding to each frame are decoded. If the decoded result completely contains text_a, then text_a is considered correct. Otherwise, the speech recognition prediction result wav_a is used to modify text_a; 1-5-5: If the first five phoneme values ​​of the corresponding phoneme vector of each frame are decoded, and a word in text_a cannot be decoded, and the tone of the word is inconsistent with the tone prediction model, the speech recognition prediction result wav_a is used to modify text_a.

Images