Method and device for obtaining audio text pairs, electronic device, and storage medium
By obtaining the original text associated with the audio file, performing speech recognition and text alignment, cutting and filtering, the problem of high cost and low accuracy of audio text pair acquisition in the prior art is solved, and efficient and accurate audio text pair acquisition is achieved. Improved the quality of model training.
Patent Information
- Application Number
- CN202311871259.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-31
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2043-12-31
AI Technical Summary
The prior art is difficult to obtain a large number of audio text pairs accurately and efficiently as training data for speech synthesis and speech recognition models, with high manual recording costs and low accuracy of speech recognition converted texts, especially punctuation mark recognition is difficult.
By obtaining the original text associated with the target audio file, performing speech recognition to obtain the time-stamped recognition text, performing text alignment, cutting into initial subparagraphs, counting word error rate filtering, matching audio files, and obtaining accurate audio text pairs.
It realizes accurate and efficient acquisition of audio text pairs, reducing the cost of training data acquisition and improving the accuracy of model training.
Smart Images

Figure CN117975934B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of audio processing technology, and in particular to a method and device for acquiring audio-text pairs, as well as related electronic equipment and storage media. Background Art
[0002] TTS (Text To Speech) is a method of automatically synthesizing speech through text. When training the model, a large number of short audio and text pairs are required as training data. Currently, there are two main ways to obtain these training data:
[0003] 1) Manually record the speech corresponding to the text to obtain training data. This method is costly to obtain data and it is difficult to obtain a large amount of data;
[0004] 2) When there is only voice data, the voice data is converted into corresponding text through voice recognition, and training data is obtained through cross-validation and other methods. The accuracy of samples obtained in this way is low, especially for punctuation marks in text, which are difficult to recognize and it is difficult to obtain accurate punctuation marks.
[0005] Currently, no effective solution has been proposed for accurately and efficiently acquiring a large number of audio-text pairs as training data.
[0006] The description of the background technology is intended to help understand the relevant technology in the relevant field, and does not constitute an admission that the content of the background technology belongs to the prior art. Summary of the invention
[0007] The purpose of this application is to provide a method and device for obtaining audio-text pairs, which can accurately and efficiently obtain a large number of audio-text pairs as training data.
[0008] In a first aspect, the present application provides a method for obtaining an audio-text pair, comprising:
[0009] Get the original text associated with the target audio file;
[0010] Performing speech recognition on the target audio file to obtain a recognition text with a timestamp;
[0011] Performing text alignment on the original text and the recognized text to obtain a text alignment result;
[0012] The text alignment result is cut into a plurality of initial sub-paragraphs;
[0013] Counting the word error rate of each initial subparagraph in the multiple initial subparagraphs, and screening according to the word error rate of each initial subparagraph to obtain multiple target subparagraphs;
[0014] According to the timestamps of each sub-paragraph in the multiple target sub-paragraphs, target audio files are matched for the multiple target sub-paragraphs to obtain multiple audio-text pairs, wherein the audio-text pairs are used as training data to train the model.
[0015] In some embodiments, performing text alignment on the original text and the recognized text to obtain a text alignment result includes:
[0016] When the recognized text at the current position is text content and the original text is empty, deleting the text content at the current position of the recognized text;
[0017] When the original text at the current position is text content and the recognized text is empty, insert the text content at the current position of the original text at the current position of the recognized text;
[0018] When both the recognized text and the original text have text content at the current position point but the text content is inconsistent, the text content at the current position point of the recognized text is converted into a first pinyin, and the text content at the current position point of the original text is converted into a second pinyin, and it is determined whether the first pinyin and the second pinyin are consistent. When the pinyins are consistent, it is determined that the current position point is aligned. When the pinyins are inconsistent, the current position point of the recognized text is replaced with the text content at the current position point of the original text.
[0019] In some embodiments, performing text alignment on the original text and the recognized text to obtain a text alignment result includes:
[0020] Determine whether the recognized text at the current position point is a preset modal particle;
[0021] In the case that the recognized text at the current position point is a preset modal particle, the current position point of the original text is replaced with the preset modal particle at the current position point of the recognized text.
[0022] In some embodiments, performing text alignment on the original text and the recognized text to obtain a text alignment result includes:
[0023] When the original text at the current position is a punctuation mark, and the recognized text is a punctuation mark different from that of the original text, the punctuation mark at the current position of the recognized text is replaced with the punctuation mark at the current position of the original text;
[0024] When the original text at the current position is a punctuation mark and the recognized text is empty, insert the punctuation mark at the current position of the original text at the current position of the recognized text;
[0025] When the recognized text at the current position point is a punctuation mark and the original text has no punctuation mark, the punctuation mark at the current position point of the recognized text is deleted.
[0026] In some embodiments, counting the word error rate of each of the multiple initial sub-paragraphs includes:
[0027] Count the total number of characters in the current initial subparagraph;
[0028] According to the text alignment result, count the number of characters that have been replaced, deleted, or inserted in the current initial subparagraph;
[0029] Determine the ratio of characters with replacement, deletion, and insertion operations to the total number of characters;
[0030] The determined ratio is used as the word error rate of the current initial sub-paragraph.
[0031] In some embodiments, the initial sub-paragraphs are screened according to their word error rates to obtain a plurality of target sub-paragraphs, including:
[0032] Obtaining a preset determination threshold;
[0033] The paragraphs whose sub-error rates exceed the determination threshold in the initial sub-paragraphs are deleted to obtain a plurality of target sub-paragraphs.
[0034] In some embodiments, the text alignment result is cut into a plurality of initial sub-paragraphs, including:
[0035] Obtaining preset segmentation punctuation and / or preset segmentation word length;
[0036] According to preset segmentation punctuation points and / or preset segmentation word lengths, the aligned recognized text is cut into a plurality of initial sub-paragraphs.
[0037] In some embodiments, the audio-text pair is in the form of text first and audio second, and is used as the first training data for training a speech synthesis model.
[0038] In some embodiments, the audio-text pair is in the form of audio first and text second, and is used as the second training data for training the speech recognition model.
[0039] In a second aspect, the present application provides a device for acquiring an audio-text pair, comprising:
[0040] An acquisition module, used for acquiring original text associated with a target audio file;
[0041] A recognition module, used for performing speech recognition on the target audio file to obtain a recognition text with a timestamp;
[0042] An alignment module, used for performing text alignment on the original text and the recognized text to obtain a text alignment result;
[0043] A cutting module, used for cutting the text alignment result into a plurality of initial sub-paragraphs;
[0044] A screening module, used for counting the word error rate of each initial subparagraph in the plurality of initial subparagraphs, and screening according to the word error rate of each initial subparagraph to obtain a plurality of target subparagraphs;
[0045] A generation module is used to match target audio files for multiple target sub-paragraphs according to the timestamps of each sub-paragraph in the multiple target sub-paragraphs to obtain multiple audio-text pairs, wherein the audio-text pairs are used as training data to train the model.
[0046] In some embodiments, the alignment module comprises:
[0047] A first processing unit, configured to delete the text content at the current position of the recognized text when the recognized text at the current position is text content and the original text is empty;
[0048] A second processing unit is used for inserting the text content at the current position of the original text at the current position of the recognized text when the original text at the current position is text content and the recognized text is empty;
[0049] The third processing unit is used for, when both the recognized text and the original text have text content at the current position point but the text contents are inconsistent, converting the text content at the current position point of the recognized text into a first pinyin and converting the text content at the current position point of the original text into a second pinyin, determining whether the first pinyin is consistent with the second pinyin, and determining that the current position point is aligned if the pinyins are consistent, and replacing the current position point of the recognized text with the text content at the current position point of the original text if the pinyins are inconsistent.
[0050] In some embodiments, the cutting module comprises:
[0051] An acquisition unit, used for acquiring a preset segmentation punctuation mark and / or a preset segmentation word length;
[0052] The cutting unit is used to cut the aligned recognized text into a plurality of initial sub-paragraphs according to preset segmentation punctuation points and / or preset segmentation word lengths.
[0053] In a third aspect, the present application provides an electronic device, comprising a processor and a memory for storing processor-executable instructions, wherein the processor implements the steps of the above method when executing the instructions.
[0054] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps of the above method when the computer program / instruction is executed by a processor.
[0055] The method for obtaining audio-text pairs provided in the present application performs text alignment by obtaining the original text associated with the target audio file and the recognized text with a timestamp obtained based on the recognition of the target audio file, and cuts it into multiple initial sub-paragraphs, and then screens based on the word error rate of each initial sub-paragraph to obtain multiple target sub-paragraphs. Further, according to the timestamps of each sub-paragraph in the multiple target sub-paragraphs, the target audio files are matched for the multiple target sub-paragraphs, so as to finally obtain multiple audio-text pairs, and these audio-text pairs can be used as training data to train the model. The above scheme solves the technical problems of high cost and low accuracy in the existing audio-text pair acquisition, and achieves the technical effect of accurately and efficiently obtaining multiple audio-text pairs.
[0056] Other optional features and technical effects of the embodiments of the present application are partially described below, and partially can be understood by reading this document. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use are briefly introduced below. It should be clear that the drawings in the following description only cover some embodiments of the present application, and ordinary technicians in this field can obtain other drawings based on these drawings without performing creative work. The purpose of these drawings is to better illustrate the technical details to help understand the implementation methods of the present application, among which:
[0058] Figure 1 It is a method flow chart of an embodiment of a method for obtaining an audio-text pair provided by the present application;
[0059] Figure 2 It is a logical diagram of text alignment processing provided by this application;
[0060] Figure 3 It is a schematic diagram of long text segmentation provided by this application;
[0061] Figure 4 This is a flowchart of obtaining an audio text from a long text provided by this application;
[0062] Figure 5 is an example diagram of the alignment results provided in this application;
[0063] Figure 6 It is a schematic diagram of the module structure of an embodiment of the device for acquiring audio-text pairs provided by the present application;
[0064] Figure 7It is a hardware structure block diagram of an electronic device for a method for acquiring audio-text pairs provided in this application. DETAILED DESCRIPTION
[0065] The exemplary embodiments of the present application will be described in detail below, and the illustrations of the examples have been shown in the accompanying drawings. When referring to the drawings, unless otherwise specified, the same numbers or marks in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Instead, they are only examples of some aspects of the devices and methods covered by one or more embodiments of this specification as detailed in the claims of this application.
[0066] The term "including" and its variations used in this specification are all intended to be broadly inclusive, i.e., "including but not limited to" the items listed. Unless otherwise stated, the term "or" means "and / or", the term "based on" means relying on, or at least partially relying on, the terms "an example embodiment" and "an embodiment" refer to at least one example embodiment, and the term "another embodiment" means at least one different embodiment. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.
[0067] Figure 1 It is a method flow chart of an embodiment of the method for acquiring audio-text pairs provided by the present application. Although the present application provides method operation steps or device structures as shown in the following embodiments or drawings, more or fewer operation steps or module units may be included in the method or device based on routine or no creative labor. In the steps or structures where there is no necessary causal relationship logically, the execution order of these steps or the module structure of the device is not limited to the execution order or module structure described in the embodiments of the present application and shown in the drawings. When the method or module structure is applied to an actual device or terminal product, it can be connected according to the method or module structure shown in the embodiments or drawings for sequential execution or parallel execution (for example, a parallel processor or a multi-threaded processing environment, or even a distributed processing environment).
[0068] Specifically, Figure 1 As shown, the above-mentioned method for obtaining the audio-text pair may include the following steps:
[0069] Step 101: Obtaining the original text associated with the target audio file;
[0070] Among them, the target audio file can be the audio of an e-book, and the corresponding original text can be an e-book, the target audio can be a TV program, and the corresponding original text is the script of the TV program, and the target audio file can be a movie, and the corresponding original text is the dialogue text corresponding to the movie. All of these resources can be obtained from the Internet or the resource pool. After obtaining these files, they can be stored in an associated manner, for example: movie audio 1-dialogue text 1, movie audio 2-dialogue text 2...e-book audio 1-e-book text 1, e-book audio 2-e-book text 2. These associated stored files can be used as resources for subsequent audio-text pair extraction.
[0071] Step 102: performing speech recognition on the target audio file to obtain a recognition text with a timestamp;
[0072] Specifically, the audio file can be recognized through speech recognition technology, that is, the text corresponding to the audio can be recognized, and the time point of each text field corresponding to the audio, that is, the timestamp, can be recorded, thereby forming a recognized text with a timestamp, and the corresponding audio position can be located based on the text content.
[0073] However, considering that the recognized text is obtained through speech recognition technology, there may sometimes be inaccurate recognition problems, such as text recognition deviation, punctuation recognition errors, etc. In order to correct these possible errors, the original text corresponding to the video file is introduced, and the two are compared to achieve the correction of the recognized text.
[0074] Step 103: performing text alignment on the original text and the recognized text to obtain a text alignment result;
[0075] When aligning the original text and the recognition file, it can be done in the following ways, but not limited to: Figure 2 As shown:
[0076] 1) For text inconsistencies:
[0077] When the recognized text at the current position is text content and the original text is empty, deleting the text content at the current position of the recognized text;
[0078] When the original text at the current position is text content and the recognized text is empty, insert the text content at the current position of the original text at the current position of the recognized text;
[0079] When both the recognized text and the original text have text content at the current position point but the text content is inconsistent, the text content at the current position point of the recognized text is converted into a first pinyin, and the text content at the current position point of the original text is converted into a second pinyin, and it is determined whether the first pinyin and the second pinyin are consistent. When the pinyins are consistent, it is determined that the current position point is aligned. When the pinyins are inconsistent, the current position point of the recognized text is replaced with the text content at the current position point of the original text.
[0080] That is, in the case of different text contents, the original text shall prevail.
[0081] 2) For inconsistent punctuation marks:
[0082] When the original text at the current position is a punctuation mark, and the recognized text is a punctuation mark different from that of the original text, the punctuation mark at the current position of the recognized text is replaced with the punctuation mark at the current position of the original text;
[0083] When the original text at the current position is a punctuation mark and the recognized text is empty, insert the punctuation mark at the current position of the original text at the current position of the recognized text;
[0084] When the recognized text at the current position point is a punctuation mark and the original text has no punctuation mark, the punctuation mark at the current position point of the recognized text is deleted.
[0085] That is, even if the text content is different, the original text shall prevail.
[0086] 3) Regarding the presence of modal particles in the text:
[0087] Determine whether the recognized text at the current position is a preset modal particle; if the recognized text at the current position is a preset modal particle, replace the current position of the original text with the preset modal particle at the current position of the recognized text.
[0088] That is, for modal particles, the recognition text takes precedence.
[0089] By using the consistency processing method in the above text alignment process, the accuracy of the recognized text can be improved.
[0090] Step 104: cutting the text alignment result into a plurality of initial sub-paragraphs;
[0091] That is, for the text alignment result, it is a long text, and the short text required for model training. In order to obtain multiple short texts, the text alignment result can be cut to obtain multiple short texts.
[0092] For example, a preset segmentation punctuation mark and / or a preset segmentation word length can be obtained; according to the preset segmentation punctuation mark and / or the preset segmentation word length, the aligned recognition text is cut into multiple initial sub-paragraphs. For example, "," "." "!" "?" can be set as punctuation marks that trigger segmentation. Then, when segmenting, if these punctuation marks are encountered, these punctuation marks are used as segmentation points for text segmentation. Furthermore, considering that some sentences are too long in actual implementation, if only punctuation marks are used as segmentation points, some sentences will be too long. For this reason, the segmentation word length can be set. For example, the segmentation word length is set to 10 words. Then, if there are too long fields after segmentation based on punctuation marks, these fields are segmented based on the segmentation word length, and the too long fields are segmented into fields with a maximum of 10 words.
[0093] like Figure 3 As shown, the long text is segmented, that is, multiple segmentation points are determined based on "segmentation punctuation and segmentation word length": segmentation point 1, segmentation point 2, etc., so that the text alignment result is divided into multiple initial field falls. Specifically, the text alignment result can be only the recognition text with a timestamp after text alignment.
[0094] Step 105: Counting the word error rate of each of the multiple initial subparagraphs, and screening according to the word error rate of each initial subparagraph to obtain multiple target subparagraphs;
[0095] Considering that the final target sub-paragraph and the generated audio-text pair need to ensure accuracy as much as possible, the accuracy of the final model can be effectively improved. To this end, in this example, the word error rate and word error rate threshold are set to filter texts with more errors in the text alignment results. Specifically, the total number of characters in the current initial sub-paragraph can be counted; according to the text alignment results, the number of characters with replacement, deletion, and insertion operations in the current initial sub-paragraph can be counted; the proportion of the number of characters with replacement, deletion, and insertion operations to the total number of characters is determined; the determined proportion is used as the word error rate of the current initial sub-paragraph. Obtain a preset judgment threshold; delete the paragraphs in the initial sub-paragraph whose sub-error rate exceeds the judgment threshold to obtain multiple target sub-paragraphs.
[0096] Step 106: Match target audio files for the multiple target sub-paragraphs according to the timestamps of each sub-paragraph in the multiple target sub-paragraphs to obtain multiple audio-text pairs, wherein the audio-text pairs are used as training data to train the speech synthesis model.
[0097] The multiple audio-text pairs obtained in the above manner can be used as training data to train the model. In this example, the obtained audio-text pairs can be used to train the speech synthesis model or the speech recognition model, for example:
[0098] 1) In some embodiments, the audio-text pair is in the form of text first and audio second, which is used as the first training data for training the speech synthesis model, so that the trained speech synthesis model will take the text as input and the audio as output.
[0099] 2) In some other embodiments, the audio-text pair is in the form of audio first and text second, which is used as the second training data for training the speech recognition model, so that the trained speech recognition model will use audio as input and text as output.
[0100] The above method is described below in conjunction with a specific embodiment. However, it is worth noting that this specific embodiment is only for better illustrating the present application and does not constitute an improper limitation on the present application.
[0101] In this example, alignment and cutting are performed on long speech (i.e., target audio files) with corresponding text resources (i.e., original text) to obtain training data for the model. For example, some e-book audio has corresponding novel text, and videos such as movies and TV series have corresponding lines and subtitles text. The data in these text resources are relatively accurate, but not completely accurate. Therefore, combined with speech recognition to obtain recognized text, the two are mutually corrected, and short speech-text pairs are cut to obtain a large amount of training data.
[0102] Taking e-books as an example, Figure 4 As shown, it includes: first obtaining the long audio of the e-book, then performing speech recognition on the audio to obtain the recognized text, and at the same time retrieving the e-book text corresponding to the long audio, and then aligning the recognized text and the e-book text to obtain the aligned matching text, and cutting and screening the matching text to obtain a short audio-text pair.
[0103] When performing speech recognition, the text data with timestamps can be recognized by an ASR (Auto Speech Recognition) tool. When performing text alignment, the recognized text and the e-book text can be aligned by NW (Needleman-Wunsch, text alignment algorithm). The alignment result can be as follows: Figure 5 As shown, @ is a fill symbol, cor means correct, sub means replacement, del means deletion, and ins means insertion.
[0104] Specifically, when aligning text, you can follow the following rules:
[0105] 1) Perform text normalization on the recognized text and the e-book text, for example, converting Arabic numerals into Chinese text;
[0106] 2) During the alignment process, if there are filler words or phrases, such as: um, uh, ah, de, ha, oh, heh, etc., it is considered that the corresponding speech is included in the audio file (i.e., exists in the recognized text), but missing in the e-book text. Therefore, these words can be added to the e-book text; ', it will be considered that the corresponding speech is included in the audio, missing in the e-book text, and will be added to the e-book text to match the ASR recognized text;
[0107] 3) For cases of text inconsistency, the inconsistent text can be first converted to pinyin. If the pinyin is the same, they can be aligned together;
[0108] 4) For cases of punctuation mark inconsistency, the e-book text shall prevail.
[0109] Furthermore, when performing cutting, the text can be cut into multiple small segments through punctuation marks, word lengths, etc. Since there are corresponding timestamps in the recognized text, when cutting the text, the audio is also cut accordingly.
[0110] When performing screening, the corresponding character error rate can be counted for each small segment after cutting. For segments with a character error rate higher than the threshold, they can be deleted. Among them, the character error rate can be calculated through the following formula:
[0111]
[0112] Among them, C represents the character error rate, S represents the number of replaced characters, D represents the number of deleted characters, I represents the number of inserted characters, and N represents the total number of characters.
[0113] At the software level, the above-mentioned acquisition device for the audio-text pair can be as Figure 6 shown, including:
[0114] An acquisition module 601, configured to acquire the original text associated with the target audio file;
[0115] A recognition module 602, configured to perform speech recognition on the target audio file to obtain a recognized text with timestamps;
[0116] An alignment module 603, configured to perform text alignment on the original text and the recognized text to obtain a text alignment result;
[0117] A cutting module 604, configured to cut the text alignment result into multiple initial sub-paragraphs;
[0118] A screening module 605, configured to count the character error rate of each initial sub-paragraph among the multiple initial sub-paragraphs, and perform screening according to the character error rate of each initial sub-paragraph to obtain multiple target sub-paragraphs;
[0119] The generation module 606 is used to match the target audio files for the multiple target sub-paragraphs according to the timestamps of each sub-paragraph in the multiple target sub-paragraphs to obtain multiple audio-text pairs, wherein the audio-text pairs are used as training data to train the speech synthesis model.
[0120] In one embodiment, the alignment module 603 may include: a first processing unit, used to delete the text content at the current position of the recognized text when the recognized text at the current position is text content and the original text is empty; a second processing unit, used to insert the text content at the current position of the original text at the current position of the recognized text when the original text at the current position is text content and the recognized text is empty; a third processing unit, used to convert the text content at the current position of the recognized text into a first pinyin, convert the text content at the current position of the original text into a second pinyin, determine whether the first pinyin is consistent with the second pinyin, and determine that the current position is aligned if the pinyins are consistent, and replace the current position of the recognized text with the text content at the current position of the original text if the pinyins are inconsistent.
[0121] In one embodiment, the cutting module 604 may include: an acquisition unit for acquiring preset segmentation punctuation marks and / or preset segmentation word lengths; a cutting unit for cutting the aligned recognized text into multiple initial sub-paragraphs according to the preset segmentation punctuation marks and / or preset segmentation word lengths.
[0122] In one embodiment, text alignment is performed on the original text and the recognized text to obtain a text alignment result, including: determining whether the recognized text at the current position point is a preset modal particle; if the recognized text at the current position point is a preset modal particle, replacing the current position point of the original text with the preset modal particle at the current position point of the recognized text.
[0123] In one embodiment, the alignment module 603 can be specifically used to replace the punctuation mark at the current position of the recognized text with the punctuation mark at the current position of the original text when the original text at the current position is a punctuation mark and the recognized text is a punctuation mark different from that of the original text; to insert the punctuation mark at the current position of the original text at the current position of the recognized text when the original text at the current position is a punctuation mark and the recognized text is empty; and to delete the punctuation mark at the current position of the recognized text when the recognized text at the current position is a punctuation mark and the original text has no punctuation mark.
[0124] In one embodiment, the filtering module 605 can be specifically used to count the total number of characters in the current initial sub-paragraph; based on the text alignment result, count the number of characters that have replacement, deletion, and insertion operations in the current initial sub-paragraph; determine the ratio of the number of characters that have replacement, deletion, and insertion operations to the total number of characters; and use the determined ratio as the character error rate of the current initial sub-paragraph.
[0125] In one implementation, the screening module 605 may be specifically configured to obtain a preset determination threshold; delete paragraphs in the initial subparagraphs whose sub-error rates exceed the determination threshold, and obtain a plurality of target subparagraphs.
[0126] In one embodiment, after the above-mentioned acquisition device matches the target audio files for multiple target sub-paragraphs according to the timestamps of each sub-paragraph in the multiple target sub-paragraphs to obtain multiple audio-text pairs, it can also convert the multiple audio-text pairs into first training data in a manner of audio as input and text as output; and train the speech recognition model through the first training data.
[0127] In one embodiment, after the above-mentioned acquisition device matches the target audio files for multiple target sub-paragraphs according to the timestamps of each sub-paragraph in the multiple target sub-paragraphs to obtain multiple audio-text pairs, it can also convert the multiple audio-text pairs into second training data in a manner of text as input and audio as output; and train the speech synthesis model through the second training data.
[0128] The method embodiments provided in the above embodiments of the present application can be executed in electronic devices, such as mobile terminals, computer terminals or similar computing devices. In addition, the embodiments of the present application also provide a specific implementation of an electronic device that can implement all the steps in the method for obtaining audio text pairs in the above embodiments.
[0129] Figure 7 A schematic diagram of an electronic device 700 that can be used to implement the method of an embodiment of the present application or to implement an embodiment of the present application is shown. In some embodiments, the number of electronic devices may be more or less than the number shown. In some embodiments, a single or multiple electronic devices may be used for implementation. In some embodiments, cloud or distributed electronic devices may also be used for implementation.
[0130] like Figure 7As shown, the electronic device 700 includes a processor 710 and a memory 720. The processor is used to execute programs stored in the memory, and these programs can implement the methods, steps or functions described in the above embodiments when executed by the computer. The processor 710 may include various types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. The processor 710 and the memory 720 are interconnected via a bus 730. The bus 730 can also be connected to an input / output (I / O) interface, etc.
[0131] The processor of the electronic device is used to call the computer program in the memory. When the processor executes the computer program, all the steps in the method for obtaining the audio-text pair in the above embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:
[0132] Step 1: Get the original text associated with the target audio file;
[0133] Step 2: Perform speech recognition on the target audio file to obtain a recognition text with a timestamp;
[0134] Step 3: performing text alignment on the original text and the recognized text to obtain a text alignment result;
[0135] Step 4: Cut the text alignment result into a plurality of initial sub-paragraphs;
[0136] Step 5: Counting the word error rate of each of the multiple initial subparagraphs, and screening according to the word error rate of each initial subparagraph to obtain multiple target subparagraphs;
[0137] Step 6: According to the timestamps of each sub-paragraph in the multiple target sub-paragraphs, target audio files are matched for the multiple target sub-paragraphs to obtain multiple audio-text pairs, wherein the audio-text pairs are used as training data to train the model.
[0138] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps in the method for obtaining an audio-text pair in the above embodiment. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, all the steps in the method for obtaining an audio-text pair in the above embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:
[0139] Step 1: Get the original text associated with the target audio file;
[0140] Step 2: Perform speech recognition on the target audio file to obtain a recognition text with a timestamp;
[0141] Step 3: performing text alignment on the original text and the recognized text to obtain a text alignment result;
[0142] Step 4: Cut the text alignment result into a plurality of initial sub-paragraphs;
[0143] Step 5: Counting the word error rate of each of the multiple initial subparagraphs, and screening according to the word error rate of each initial subparagraph to obtain multiple target subparagraphs;
[0144] Step 6: According to the timestamps of each sub-paragraph in the multiple target sub-paragraphs, target audio files are matched for the multiple target sub-paragraphs to obtain multiple audio-text pairs, wherein the audio-text pairs are used as training data to train the model.
[0145] As can be seen from the above description, the embodiment of the present application performs text alignment by acquiring the original text associated with the target audio file and the recognized text with timestamp obtained based on the recognition of the target audio file, and cuts it into multiple initial sub-paragraphs, and then screens based on the word error rate of each initial sub-paragraph to obtain multiple target sub-paragraphs. Further, according to the timestamps of each sub-paragraph in the multiple target sub-paragraphs, the target audio files are matched for the multiple target sub-paragraphs, so as to finally obtain multiple audio-text pairs, which can be used as training data to train the model. The above scheme solves the technical problems of high cost and low accuracy of the existing audio-text pair acquisition, and achieves the technical effect of accurately and efficiently obtaining multiple audio-text pairs.
[0146] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0147] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0148] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0149] Although the present application provides method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).
[0150] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0151] Although the present specification embodiment provides the method operation steps as described in the embodiment or flow chart, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiment is only one way in the order of execution of many steps, and does not represent a unique execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel (such as a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment) according to the method shown in the embodiment or the accompanying drawings. The term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements not only includes those elements, but also includes other elements not clearly listed, or also includes elements inherent to such process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements.
[0152] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing the embodiments of this specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0153] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.
[0154] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0155] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0157] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0158] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0159] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0160] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0161] The present specification embodiments may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present specification embodiments may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.
[0162] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, in the absence of contradiction, a person skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.
[0163] The above is only an example of the embodiment of the present specification and is not intended to limit the embodiment of the present specification. For those skilled in the art, the embodiment of the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiment of the present specification shall be included in the scope of the claims of the embodiment of the present specification.
Claims
1. A method for obtaining an audio-text pair, characterized in that: include: Get the original text associated with the target audio file; Performing speech recognition on the target audio file to obtain a recognition text with a timestamp; Performing text alignment on the original text and the recognized text to obtain a text alignment result, wherein the original text and the recognized text are mutually corrected, specifically including: normalizing the original text and the recognized text; determining whether the recognized text at the current position is a preset modal particle, and if the recognized text at the current position is a preset modal particle, replacing the current position of the original text with the preset modal particle at the current position of the recognized text; comparing the pinyin when the texts are inconsistent, and if the pinyin is consistent, the recognized text shall prevail, and if the pinyin is inconsistent, the original text shall prevail; if the punctuation marks are inconsistent, the original text shall prevail; The text alignment result is cut into a plurality of initial sub-paragraphs; The word error rate of each initial subparagraph in the multiple initial subparagraphs is counted, and a plurality of target subparagraphs are obtained by screening according to the word error rate of each initial subparagraph; wherein the word error rate is calculated by the following formula: Where C represents the word error rate, S represents the number of replaced characters, D represents the number of deleted characters, I represents the number of inserted characters, and N represents the total number of characters. According to the timestamps of each sub-paragraph in the multiple target sub-paragraphs, target audio files are matched for the multiple target sub-paragraphs to obtain multiple audio-text pairs, wherein the audio-text pairs are used as training data to train the model.
2. The method according to claim 1, characterized in that Performing text alignment on the original text and the recognized text to obtain a text alignment result includes: When the recognized text at the current position is text content and the original text is empty, delete the text content at the current position of the recognized text; When the original text at the current position is text content and the recognized text is empty, insert the text content at the current position of the original text at the current position of the recognized text; When both the recognized text and the original text have text content at the current position point but the text content is inconsistent, the text content at the current position point of the recognized text is converted into a first pinyin, and the text content at the current position point of the original text is converted into a second pinyin, and it is determined whether the first pinyin and the second pinyin are consistent. When the pinyins are consistent, it is determined that the current position point is aligned. When the pinyins are inconsistent, the current position point of the recognized text is replaced with the text content at the current position point of the original text.
3. The method according to claim 1, characterized in that: Performing text alignment on the original text and the recognized text to obtain a text alignment result includes: When the original text at the current position is a punctuation mark, and the recognized text is a punctuation mark different from that of the original text, the punctuation mark at the current position of the recognized text is replaced with the punctuation mark at the current position of the original text; When the original text at the current position is a punctuation mark and the recognized text is empty, insert the punctuation mark at the current position of the original text at the current position of the recognized text; When the recognized text at the current position point is a punctuation mark and the original text has no punctuation mark, the punctuation mark at the current position point of the recognized text is deleted.
4. The method according to claim 1, characterized in that: Counting the word error rate of each of the multiple initial sub-paragraphs includes: Count the total number of characters in the current initial subparagraph; According to the text alignment result, count the number of characters that have been replaced, deleted, or inserted in the current initial subparagraph; Determine the ratio of characters with replacement, deletion, and insertion operations to the total number of characters; The determined ratio is used as the word error rate of the current initial sub-paragraph.
5. The method according to claim 1, characterized in that According to the word error rate of each initial sub-paragraph, multiple target sub-paragraphs are obtained, including: Obtaining a preset determination threshold; The paragraphs whose word error rates in the initial sub-paragraphs exceed the determination threshold are deleted to obtain a plurality of target sub-paragraphs.
6. The method according to claim 1, characterized in that The text alignment result is cut into multiple initial sub-paragraphs, including: Obtaining preset segmentation punctuation and / or preset segmentation word length; According to preset segmentation punctuation points and / or preset segmentation word lengths, the aligned recognized text is cut into a plurality of initial sub-paragraphs.
7. The method according to claim 1, characterized in that The audio-text pair is in the form of text first and audio second, and is used as the first training data for training a speech synthesis model.
8. The method according to claim 1, characterized in that The audio-text pair is in the form of audio first and text second, and is used as second training data for training a speech recognition model.
9. A device for acquiring an audio-text pair, characterized in that: include: An acquisition module, used for acquiring original text associated with a target audio file; A recognition module, used for performing speech recognition on the target audio file to obtain a recognition text with a timestamp; An alignment module is used to perform text alignment on the original text and the recognized text to obtain a text alignment result. The original text and the recognized text are mutually corrected, specifically including: normalizing the original text and the recognized text; determining whether the recognized text at the current position is a preset modal particle, and if the recognized text at the current position is a preset modal particle, replacing the current position of the original text with the preset modal particle at the current position of the recognized text; comparing pinyin when the texts are inconsistent, and if the pinyin is consistent, the recognized text shall prevail, and if the pinyin is inconsistent, the original text shall prevail; if the punctuation marks are inconsistent, the original text shall prevail; A cutting module, used for cutting the text alignment result into a plurality of initial sub-paragraphs; The screening module is used to count the word error rate of each initial sub-paragraph in the multiple initial sub-paragraphs, and screen according to the word error rate of each initial sub-paragraph to obtain multiple target sub-paragraphs; wherein the word error rate is calculated by the following formula: Where C represents the word error rate, S represents the number of replaced characters, D represents the number of deleted characters, I represents the number of inserted characters, and N represents the total number of characters. A generation module is used to match target audio files for multiple target sub-paragraphs according to the timestamps of each sub-paragraph in the multiple target sub-paragraphs to obtain multiple audio-text pairs, wherein the audio-text pairs are used as training data to train the model.
10. The device according to claim 9, characterized in that The alignment module comprises: A first processing unit, configured to delete the text content at the current position of the recognized text when the recognized text at the current position is text content and the original text is empty; A second processing unit is used for inserting the text content at the current position of the original text at the current position of the recognized text when the original text at the current position is text content and the recognized text is empty; The third processing unit is used for, when both the recognized text and the original text have text content at the current position point but the text contents are inconsistent, converting the text content at the current position point of the recognized text into a first pinyin and converting the text content at the current position point of the original text into a second pinyin, determining whether the first pinyin is consistent with the second pinyin, and determining that the current position point is aligned if the pinyins are consistent, and replacing the current position point of the recognized text with the text content at the current position point of the original text if the pinyins are inconsistent.
11. The device according to claim 9, characterized in that The cutting module comprises: An acquisition unit, used for acquiring a preset segmentation punctuation mark and / or a preset segmentation word length; The cutting unit is used to cut the aligned recognized text into a plurality of initial sub-paragraphs according to preset segmentation punctuation points and / or preset segmentation word lengths.
12. An electronic device comprising a processor and a memory for storing instructions executable by the processor, characterized in that: When the processor executes the instructions, the steps of the method according to any one of claims 1 to 8 are implemented.
13. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.
