Audio processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the invention provide an audio processing method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining a first audio and a first text corresponding to the first audio; predicting a first pronunciation sequence of the first text according to the first audio and the first text through a first pronunciation prediction system; tones of pronunciation of characters in the first text marked in the first pronunciation sequence comprise light tones and / or three-tone tones after tone modification; a first three-sound tone in two continuous three-sound tones in the first text is marked as a three-sound tone after tone modification in the first pronunciation sequence; and correcting the light tone in the first pronunciation sequence through a second pronunciation prediction system, and / or correcting the three-tone tone after tone modification in the first pronunciation sequence through a third pronunciation prediction system.

Description

Technical Field

[0001] The present disclosure relates to the field of audio processing, and in particular to an audio processing method, device, electronic device and storage medium. Background Art

[0002] In the field of natural language processing, it is often necessary to predict the pronunciation sequence of a text, which can be a phoneme sequence or a pinyin sequence. There may be a light tone or two consecutive third tones in the text. Two consecutive third tones can be exemplified as: ni3, hao3, where 3 indicates the third tone. For the two consecutive third tones in the text, considering that the first third tone will be toned in the actual reading scenario and will not be read as the third tone, the first third tone is usually marked as the third tone after tones in the pronunciation sequence. In the prior art, the accuracy of predicting the pronunciation sequence is poor due to the interference of the light tone and two consecutive third tones in the text. Summary of the invention

[0003] The embodiments of the present disclosure provide an audio processing method, device, electronic device and storage medium, which can improve the accuracy of predicting the pronunciation sequence of text.

[0004] In a first aspect, an embodiment of the present disclosure provides an audio processing method, including:

[0005] Acquire a first audio and a first text corresponding to the first audio;

[0006] Predicting a first pronunciation sequence of the first text according to the first audio and the first text by a first pronunciation prediction system; the tones of the pronunciation of the characters in the first text marked in the first pronunciation sequence include a light tone and / or a third tone after tone change; the first third tone of two consecutive third tones in the first text is marked as the third tone after tone change in the first pronunciation sequence;

[0007] The unstressed tone in the first pronunciation sequence is corrected by a second pronunciation prediction system, and / or the modulated third tone in the first pronunciation sequence is corrected by a third pronunciation prediction system.

[0008] In a second aspect, an embodiment of the present disclosure provides an audio processing device, including:

[0009] A data acquisition unit, used to acquire a first audio and a first text corresponding to the first audio;

[0010] A pronunciation prediction unit is used to predict a first pronunciation sequence of the first text according to the first audio and the first text through a first pronunciation prediction system; the tones of the pronunciation of the characters in the first text marked in the first pronunciation sequence include a light tone and / or a third tone after tone change; the first third tone of two consecutive third tones in the first text is marked as the third tone after tone change in the first pronunciation sequence;

[0011] The pronunciation correction unit is used to correct the unstressed tone in the first pronunciation sequence through a second pronunciation prediction system, and / or to correct the modulated third tone in the first pronunciation sequence through a third pronunciation prediction system.

[0012] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, enable the processor to implement the steps of the method described in the first aspect above.

[0013] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions implement the steps of the method described in the first aspect when executed by a processor.

[0014] In one or more embodiments of the present disclosure, a first audio and a first text corresponding to the first audio are obtained, and a first pronunciation prediction system is used to predict a first pronunciation sequence of the first text according to the first audio and the first text. The tones of the pronunciation of the characters in the first text marked in the first pronunciation sequence include a light tone and / or a third tone after tone change. The first third tone of two consecutive third tones in the first text is marked as a third tone after tone change in the first pronunciation sequence. The light tone in the first pronunciation sequence is corrected by the second pronunciation prediction system, and / or the third tone after tone change in the first pronunciation sequence is corrected by the third pronunciation prediction system. It can be seen that through this embodiment, not only the first pronunciation sequence of the first text can be predicted, but also the light tone in the first pronunciation sequence can be corrected, and / or the third tone after tone change in the first pronunciation sequence can be corrected, thereby improving the accuracy of the light tone and two consecutive third tones in the predicted text, and improving the accuracy of the pronunciation sequence of the predicted text. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate one or more embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

[0016] Figure 1 A flowchart of an audio processing method provided by an embodiment of the present disclosure;

[0017] Figure 2 A schematic diagram of the pronunciation prediction principle of a first pronunciation prediction system provided by an embodiment of the present disclosure;

[0018] Figure 3 A schematic diagram of the pronunciation prediction principle of a second pronunciation prediction system provided by an embodiment of the present disclosure;

[0019] Figure 4 A schematic diagram of the pronunciation prediction principle of a third pronunciation prediction system provided by an embodiment of the present disclosure;

[0020] Figure 5 A schematic diagram of the structure of an audio processing device provided by an embodiment of the present disclosure;

[0021] Figure 6 A schematic diagram of the structure of an electronic device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of the present disclosure, the technical solutions in one or more embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in one or more embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on one or more embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the protection scope of the present disclosure.

[0023] The embodiments of the present disclosure provide an audio processing method that can improve the accuracy of predicting the pronunciation sequence of a text. In various embodiments of the present disclosure, the pronunciation can be represented by pinyin or phonemes. The pronunciation of each word of a text constitutes the pronunciation sequence of the text. The pronunciation has tones, and the tones of the pronunciation include at least one (level), two (raised), three (turned), and four (falling) tones in pinyin. In the pronunciation sequence, the first, second, third, and fourth tones can be represented by the numbers 1, 2, 3, and 4, respectively. Here, the pronunciation, pronunciation sequence, and tones are explained by a specific example. In a specific illustration, the text is "hello", and the pronunciation sequence of the text is: ni3, hao3, wherein "ni3" is the pronunciation of "you", "hao3" is the pronunciation of "good", and the number 3 indicates that the tone of the pronunciation is three tones.

[0024] Figure 1 A flowchart of an audio processing method provided by an embodiment of the present disclosure is shown in FIG. Figure 1 As shown, the method includes:

[0025] Step S102, obtaining a first audio and a first text corresponding to the first audio;

[0026] Step S104, predicting a first pronunciation sequence of the first text according to the first audio and the first text by a first pronunciation prediction system; the tones of the pronunciation of the characters in the first text marked in the first pronunciation sequence include a light tone and / or a third tone after tone change; the first third tone of two consecutive third tones in the first text is marked as a third tone after tone change in the first pronunciation sequence;

[0027] Step S106, correcting the unstressed tone in the first pronunciation sequence through the second pronunciation prediction system, and / or correcting the third tone after tone change in the first pronunciation sequence through the third pronunciation prediction system.

[0028] In this embodiment, a first audio and a first text corresponding to the first audio are obtained, and a first pronunciation sequence of the first text is predicted according to the first audio and the first text through a first pronunciation prediction system, the tones of the pronunciation of the characters in the first text marked in the first pronunciation sequence include a light tone and / or a third tone after tone change, and the first third tone of two consecutive third tones in the first text is marked as a third tone after tone change in the first pronunciation sequence, and the light tone in the first pronunciation sequence is corrected through a second pronunciation prediction system, and / or, the third tone after tone change in the first pronunciation sequence is corrected through a third pronunciation prediction system. It can be seen that through this embodiment, not only the first pronunciation sequence of the first text can be predicted, but also the light tone in the first pronunciation sequence can be corrected, and / or, the third tone after tone change in the first pronunciation sequence can be corrected, thereby improving the accuracy of the light tone and two consecutive third tones in the predicted text, and improving the accuracy of the pronunciation sequence of the predicted text.

[0029] In the above step S102, the first audio and the first text corresponding to the first audio are obtained. In one embodiment, the first text can be obtained first, and the first text can be a paragraph, an article, or other text that requires a predicted pronunciation sequence, and then the first audio corresponding to the first text is obtained by manually reading the first text. Of course, the first audio corresponding to the first text can also be obtained by other methods, such as based on a neural network, which is not limited in this embodiment. In other embodiments, the first audio can also be obtained first, and then the first text corresponding to the first audio can be obtained by manual translation or neural network, which is not limited here.

[0030] In the above step S104, a first pronunciation sequence of the first text is predicted based on the first audio and the first text by a first pronunciation prediction system. The first pronunciation prediction system includes but is not limited to a grapheme-to-phoneme alignment system, and the grapheme-to-phoneme alignment system includes but is not limited to a G2P (Grapheme-to-Phoneme, word to phoneme) model. The first audio and the first text are input into the first pronunciation prediction system, and the first audio and the first text are processed by the first pronunciation prediction system to obtain a first pronunciation sequence of the first text. The first pronunciation sequence is marked with the pronunciation of each word in the first text, and the pronunciation of each word in the first text can be represented by pinyin or phoneme, and the first pronunciation sequence of the first text includes the pinyin sequence or phoneme sequence of the first text.

[0031] In the first pronunciation sequence, the pronunciations have tones. In addition to the first, second, third, and fourth tones mentioned above, the tones of the pronunciations of the characters in the first text marked in the first pronunciation sequence also include the neutral tone and / or the third tone after tone sandhi. In one embodiment, in addition to the first, second, third, and fourth tones mentioned above, the tones of the pronunciations of the characters in the first text marked in the first pronunciation sequence also include the neutral tone. In another embodiment, in addition to the first, second, third, and fourth tones mentioned above, the tones of the pronunciations of the characters in the first text marked in the first pronunciation sequence also include the third tone after tone sandhi. In yet another embodiment, in addition to the first, second, third, and fourth tones mentioned above, the tones of the pronunciations of the characters in the first text marked in the first pronunciation sequence also include the neutral tone and the third tone after tone sandhi.

[0032] For some modal particles such as "ya", "wa", "ne", the tones of their corresponding pronunciations are the neutral tone. In the first pronunciation sequence, the neutral tone can be represented by the number 5. Thus, the pronunciations of the above-mentioned modal particles are "ya5", "wa5", "ne5" respectively.

[0033] For two consecutive characters with the third tone in the first text, such as the word "here", its corresponding pronunciation sequence can be "zhe3, li3", which are two consecutive third tones. Considering that in the actual reading scenario, the first third tone will be subject to tone sandhi and will not be pronounced as the third tone, for two consecutive characters with the third tone in the first text, the tone of the first character with the third tone is marked as the third tone after tone sandhi in the first pronunciation sequence. The third tone after tone sandhi can be represented by the number 6. Then, in the first pronunciation sequence, the corresponding pronunciation sequence of "here" can be "zhe6, li3".

[0034] In the above step S106, through the second pronunciation prediction system, the neutral tone in the first pronunciation sequence is corrected, and / or, through the third pronunciation prediction system, the third tone after tone sandhi in the first pronunciation sequence is corrected.

[0035] In one embodiment, through the second pronunciation prediction system, the neutral tone in the first pronunciation sequence is corrected. In another embodiment, through the third pronunciation prediction system, the third tone after tone sandhi in the first pronunciation sequence is corrected. In yet another embodiment, through the second pronunciation prediction system, the neutral tone in the first pronunciation sequence is corrected, and, through the third pronunciation prediction system, the third tone after tone sandhi in the first pronunciation sequence is corrected.

[0036] It can be seen that through Figure 1The process in can not only predict the first pronunciation sequence of the first text, but also correct the unstressed tone in the first pronunciation sequence, and / or correct the third tone after tone change in the first pronunciation sequence, thereby improving the accuracy of the unstressed tone and two consecutive third tones in the predicted text, and improving the accuracy of the pronunciation sequence of the predicted text.

[0037] In one embodiment, predicting a first pronunciation sequence of a first text according to a first audio and a first text by a first pronunciation prediction system includes:

[0038] Predicting at least one pronunciation of each word in the first text according to the first audio by a first acoustic module in the first pronunciation prediction system; the tone of the pronunciation includes a light tone and / or three tones after tone change;

[0039] selecting, by a first decoding module in the first pronunciation prediction system, a first target pronunciation for each character in the first text from at least one pronunciation of each character in the first text predicted in the first text;

[0040] A first pronunciation sequence is generated according to the first target pronunciation through a first decoding module.

[0041] Figure 2 A schematic diagram of the pronunciation prediction principle of a first pronunciation prediction system provided by an embodiment of the present disclosure is shown in FIG. Figure 2As shown, the first pronunciation prediction system includes a first acoustic module and a first decoding module, and the first acoustic module includes multiple transformers (neural networks based on attention mechanisms). The first decoding module may include multiple convolutional networks. In the present embodiment, the first audio is input to the first acoustic module, and the first acoustic module processes the first audio to obtain at least one pronunciation of each word in the first text, the pronunciation includes a tone, and the tone of the pronunciation includes a light tone and / or three tones after the tone change, that is, the tone of the pronunciation includes a light tone, or includes three tones after the tone change, or includes a light tone and three tones after the tone change, and the pronunciation can be represented by pinyin or by phoneme. Then, the first text is input to the first decoding module, and the first decoding module compares each word in the first text with each pronunciation of each word in the first text, and determines the matching degree value between each word in the first text and each pronunciation of each word in the first text, the higher the matching degree value, the more matching between the description word and the pronunciation, and the matching degree value can be determined based on the graph path shortest search algorithm. Then, the first decoding module selects the pronunciation with the highest matching degree value from each pronunciation of each word in the first text as the first target pronunciation of each word in the first text according to the matching degree value, and the tone of the first target pronunciation includes the light tone and / or the three tones after tone change, that is, the tone of the first target pronunciation includes the light tone, or includes the three tones after tone change, or includes the light tone and the three tones after tone change. Finally, the first decoding module generates a first pronunciation sequence according to the first target pronunciation of each word in the first text, and the first pronunciation sequence includes the first target pronunciation of each word in the first text.

[0042] It can be seen that through this embodiment, it is possible to predict at least one pronunciation of each word in the first text according to the first audio through the first acoustic module, and select a first target pronunciation for each word in the first text from the at least one pronunciation of each word in the predicted first text according to the first text through the first decoding module, and generate a first pronunciation sequence according to the first target pronunciation, thereby efficiently and quickly predicting the first pronunciation sequence of the first text.

[0043] In one embodiment, the second pronunciation prediction system corrects the unvoiced tone in the first pronunciation sequence, including:

[0044] Predicting, by a second pronunciation prediction system, a second pronunciation sequence of the first text according to the first audio and the first text, the second pronunciation sequence being used to mark the tone of the pronunciation of the characters in the first text as a light tone or a non-light tone;

[0045] The unvoiced tones in the first pronunciation sequence are corrected according to the second pronunciation sequence.

[0046] In this embodiment, the first audio and the first text are first input into the second pronunciation prediction system. The second pronunciation prediction system can predict the second pronunciation sequence of the first text based on the first audio and the first text. The second pronunciation sequence is used to mark the pronunciation tone of the characters in the first text as a light tone or a non-light tone. For example, the second pronunciation sequence is marked with a label of each character in the first text. The label 1 indicates that the pronunciation tone of the character is a light tone, and the label 0 indicates that the pronunciation tone of the character is a non-light tone. The non-light tone refers to other tones except the light tone, including the first, second, third, fourth and third tones after the tone change. Then, the light tone in the first pronunciation sequence is corrected based on the second pronunciation sequence.

[0047] It can be seen that through this embodiment, the second pronunciation sequence of the first text can be predicted by the second pronunciation prediction system, and the second pronunciation sequence is used to mark the tone of the pronunciation of the characters in the first text as a light tone or a non-light tone. The light tone in the first pronunciation sequence is corrected according to the second pronunciation sequence, thereby improving the accuracy of the light tone in the predicted text and improving the accuracy of the pronunciation sequence of the predicted text.

[0048] In one embodiment, predicting a second pronunciation sequence of the first text according to the first audio and the first text by a second pronunciation prediction system includes:

[0049] Predicting, by a second acoustic module in a second pronunciation prediction system, a probability that the tone of pronunciation of each word in the first text is a light tone based on the first audio;

[0050] Determining, by a second decoding module in the second pronunciation prediction system, whether the tone of pronunciation of each character in the first text is a light tone or a non-light tone according to the first text and the predicted probability that the tone of pronunciation of each character in the first text is a light tone;

[0051] A second pronunciation sequence is generated according to the determination result through the second decoding module.

[0052] Figure 3 A schematic diagram of the pronunciation prediction principle of a second pronunciation prediction system provided by an embodiment of the present disclosure is shown in FIG. Figure 3As shown, the second pronunciation prediction system includes a second acoustic module and a second decoding module, and the second acoustic module includes multiple transformers (neural networks based on attention mechanisms). The second decoding module may include multiple convolutional networks. In this embodiment, the first audio is input into the second acoustic module, and the second acoustic module processes the first audio to obtain the probability that the tone of pronunciation of each word in the first text is a light tone. Then, the first text is input into the second decoding module, and the second decoding module determines whether the tone of pronunciation of each word in the first text is a light tone or a non-light tone according to the first text and the predicted probability that the tone of pronunciation of each word in the first text is a light tone. In one example, the second decoding module analyzes each character in the first text one by one to see whether it is in a preset set of light tone characters. For any character in the first text, if the probability that the tone of the pronunciation of the character is a light tone is greater than or equal to a preset probability threshold, and the character is in the preset set of light tone characters, then the tone of the pronunciation of the character is determined to be a light tone. If the probability that the tone of the pronunciation of the character is a light tone is less than the preset probability threshold, or the character is not in the preset set of light tone characters, then the character is determined to be a non-light tone.

[0053] Finally, the second decoding module determines the label of each character based on the determination result. For characters whose pronunciation tone is determined to be a light tone, the label can be set equal to 1. For characters whose pronunciation tone is determined to be a non-light tone, the label can be set equal to 0. The second decoding module generates a second pronunciation sequence based on the label of each character, and the second pronunciation sequence is annotated with the label of each character in the first text.

[0054] It can be seen that through this embodiment, the second acoustic module can predict the probability that the tone of pronunciation of each word in the first text is a light tone based on the first audio, and the second decoding module can determine whether the tone of pronunciation of each word in the first text is a light tone or a non-light tone based on the first text and the predicted probability that the tone of pronunciation of each word in the first text is a light tone, and based on the determination result, the second pronunciation sequence can be efficiently and quickly generated to correct the light tone in the first pronunciation sequence.

[0055] In one embodiment, correcting the unvoiced tone in the first pronunciation sequence according to the second pronunciation sequence includes:

[0056] Determining whether the tone of the pronunciation of the first word in the first text in the first pronunciation sequence is marked as a light tone or a non-light tone; the tone of the pronunciation of the first word in the second pronunciation sequence is marked as a light tone;

[0057] If it is a light tone, the tone of the pronunciation of the first word in the first pronunciation sequence is kept unchanged;

[0058] If it is not a light tone, the tone of the pronunciation of the first character in the first pronunciation sequence is corrected to a light tone.

[0059] In the present embodiment, firstly, a first character is determined in a first text, and the first character is a character whose pronunciation tone is marked as a light tone in the second pronunciation sequence. The number of first characters may be one or more. For each first character, it is determined whether the pronunciation tone of the first character in the first pronunciation sequence is marked as a light tone or a non-light tone. If the pronunciation tone of the first character in the first pronunciation sequence is marked as a light tone, the pronunciation tone of the first character in the first pronunciation sequence is kept unchanged. If the pronunciation tone of the first character in the first pronunciation sequence is marked as a non-light tone, the pronunciation tone of the first character in the first pronunciation sequence is corrected to a light tone.

[0060] For example, for a certain first character, the tone of the pronunciation of the first character in the second pronunciation sequence is marked as a light tone. If the tone of the pronunciation of the first character in the first pronunciation sequence is marked as a light tone, the tone of the pronunciation of the first character in the first pronunciation sequence is kept unchanged. If the tone of the pronunciation of the first character in the first pronunciation sequence is marked as a non-light tone, the tone of the pronunciation of the first character in the first pronunciation sequence is corrected to a light tone.

[0061] It can be seen that through this embodiment, the first character can be identified in the first text. The tone of the pronunciation of the first character is marked as a light tone in the second pronunciation sequence. Taking the second pronunciation sequence as a standard, the tone of the pronunciation of the first character in the first pronunciation sequence is uniformly corrected to a light tone, thereby correcting the light tone in the first pronunciation sequence, improving the accuracy of the light tone in the predicted text, and improving the accuracy of the pronunciation sequence of the predicted text.

[0062] In one embodiment, the third pronunciation prediction system corrects the three tones after the tone change in the first pronunciation sequence, including:

[0063] Predicting a third pronunciation sequence of the first text according to the first audio and the first text by a third pronunciation prediction system, wherein the third pronunciation sequence is used to mark the first third tone of two consecutive third tones in the first text by a second tone;

[0064] The three tones after the tone change in the first pronunciation sequence are corrected according to the third pronunciation sequence.

[0065] In this embodiment, the first audio and the first text are first input into the third pronunciation prediction system. The third pronunciation prediction system can predict the third pronunciation sequence of the first text according to the first audio and the first text. The third pronunciation sequence is used to mark the first third tone of two consecutive third tones in the first text by the second tone. For example, the first third tone of two consecutive third tones in the first text is marked by the number 2 in the third pronunciation sequence.

[0066] In a specific example, for two consecutive third-tone words in the first text, such as the word "here", the corresponding pronunciation sequence can be "zhe3, li3", which is two consecutive third-tone words. Considering that the first third tone will be changed in the actual reading scenario and will not be read as the third tone, for two consecutive third-tone words in the first text, the tone of the first third-tone word is marked as the second tone in the third pronunciation sequence, then the pronunciation sequence corresponding to "here" in the third pronunciation sequence can be "zhe2, li3".

[0067] Next, the three tones after the tone change in the first pronunciation sequence are corrected according to the third pronunciation sequence.

[0068] It can be seen that through this embodiment, the third pronunciation sequence of the first text can be predicted by the third pronunciation prediction system, and the third pronunciation sequence is used to mark the first third tone of two consecutive third tones in the first text by the second tone, and correct the third tone after tone change in the first pronunciation sequence according to the third pronunciation sequence, thereby improving the accuracy of the two consecutive third tones in the predicted text and improving the accuracy of the pronunciation sequence of the predicted text.

[0069] In one embodiment, predicting a third pronunciation sequence of the first text according to the first audio and the first text by a third pronunciation prediction system includes:

[0070] Predicting at least one pronunciation of each word in the first text according to the first audio by a third acoustic module in the third pronunciation prediction system; the tone of the pronunciation includes a second tone corresponding to the first third tone of two consecutive third tones;

[0071] selecting, by a third decoding module in a third pronunciation prediction system, a second target pronunciation for each character in the first text from at least one pronunciation of each character in the first text predicted by the first text;

[0072] A third pronunciation sequence is generated according to the second target pronunciation through a third decoding module.

[0073] Figure 4 A schematic diagram of the pronunciation prediction principle of a third pronunciation prediction system provided by an embodiment of the present disclosure is shown in FIG. Figure 4As shown, the third pronunciation prediction system includes a third acoustic module and a third decoding module, and the third acoustic module includes multiple transformers (neural networks based on attention mechanisms). The third decoding module may include multiple convolutional networks. In this embodiment, the first audio is input into the third acoustic module, and the third acoustic module processes the first audio to obtain at least one pronunciation of each word in the first text, the pronunciation includes a tone, and the tone of the pronunciation includes a second tone corresponding to the first three tones in two consecutive three tones, and the pronunciation can be represented by pinyin or by phonemes. Then, the first text is input into the third decoding module, and the third decoding module compares each word in the first text with each pronunciation of each word in the first text, and determines the matching degree value between each word in the first text and each pronunciation of each word in the first text. The higher the matching degree value, the more matching the word and the pronunciation are, and the matching degree value can be determined based on the graph path shortest search algorithm. Then, the third decoding module selects the pronunciation with the highest matching degree value from each pronunciation of each character in the first text as the second target pronunciation of each character in the first text according to the matching degree value, and the tone of the second target pronunciation includes the second tone corresponding to the first three tones of two consecutive three tones. Finally, the third decoding module generates a third pronunciation sequence according to the second target pronunciation of each character in the first text, and the third pronunciation sequence includes the second target pronunciation of each character in the first text.

[0074] It can be seen that through this embodiment, it is possible to predict at least one pronunciation of each word in the first text based on the first audio through the third acoustic module, and select a second target pronunciation for each word in the first text from the at least one pronunciation of each word in the predicted first text through the third decoding module, and generate a third pronunciation sequence based on the second target pronunciation, thereby efficiently and quickly predicting the third pronunciation sequence of the first text.

[0075] From the previous description, it can be seen that the first pronunciation prediction system, the second pronunciation prediction system and the third pronunciation prediction system have similar structures but different functions. The reason for this difference is that the training data and training process of each system are different. The training data and training process of each system will be introduced later.

[0076] In one embodiment, the three tones after the tone change in the first pronunciation sequence are corrected according to the third pronunciation sequence, including:

[0077] Determine whether the tone of the pronunciation of the second character in the first text in the third pronunciation sequence is marked as the second tone; the tone of the pronunciation of the second character in the first pronunciation sequence is marked as the third tone after the tone change;

[0078] If yes, keep the tone of the pronunciation of the second word in the first pronunciation sequence unchanged;

[0079] If not, the tone of the pronunciation of the second character in the first pronunciation sequence is modified according to the tone of the pronunciation of the second character in the third pronunciation sequence.

[0080] In the present embodiment, first determine the second character in the first text, the second character is a character whose tone of pronunciation is marked as the third tone after tone change (which can be represented by the number 6) in the first pronunciation sequence. The number of the second characters may be one or more. For each second character, determine whether the tone of pronunciation of the second character in the third pronunciation sequence is marked as the second tone. If the tone of pronunciation of the second character in the third pronunciation sequence is marked as the second tone, the tone of pronunciation of the second character in the first pronunciation sequence is kept unchanged. If the tone of pronunciation of the second character in the third pronunciation sequence is not marked as the second tone, the tone of pronunciation of the second character in the first pronunciation sequence is modified according to the tone of pronunciation of the second character in the third pronunciation sequence, and the tone of pronunciation of the second character in the first pronunciation sequence is modified to the tone of pronunciation of the second character in the third pronunciation sequence.

[0081] For example, for a second character, the tone of the pronunciation of the second character in the first pronunciation sequence is marked as the third tone 6 after the tone change. If the tone of the pronunciation of the second character in the third pronunciation sequence is marked as the second tone 2, the tone 6 of the pronunciation of the second character in the first pronunciation sequence is kept unchanged. If the tone of the pronunciation of the second character in the third pronunciation sequence is not marked as the second tone 2, the tone 6 of the pronunciation of the second character in the first pronunciation sequence is modified to the tone of the pronunciation of the second character in the third pronunciation sequence.

[0082] It can be seen that through this embodiment, the second character can be identified in the first text, and the tone of the pronunciation of the second character is marked as the third tone after the tone change in the first pronunciation sequence. If the tone of the pronunciation of the second character in the third pronunciation sequence is not marked as the second tone, the tone of the pronunciation of the second character in the first pronunciation sequence is modified according to the tone of the pronunciation of the second character in the third pronunciation sequence, thereby correcting the third tone after the tone change in the first pronunciation sequence, improving the accuracy of two consecutive third tones in the predicted text, and improving the accuracy of the pronunciation sequence of the predicted text.

[0083] In yet another embodiment, the three tones after the tone change in the first pronunciation sequence are corrected according to the third pronunciation sequence, including:

[0084] Determine whether the tone of the pronunciation of the third character in the first text in the first pronunciation sequence is marked as the third tone after the tone change; the tone of the pronunciation of the third character in the third pronunciation sequence is marked as the second tone corresponding to the first third tone of two consecutive third tones;

[0085] If yes, keep the tone of the third word in the first pronunciation sequence unchanged;

[0086] If not, the tone of the pronunciation of the third character in the first pronunciation sequence is modified to the third tone after the tone change.

[0087] In this embodiment, firstly, the third character is determined in the first text, and the third character is a character whose tone of pronunciation is marked as the second tone (which can be represented by the number 2) corresponding to the first third tone of two consecutive third tones in the third pronunciation sequence. For example, all second tones are identified in the third pronunciation sequence, and if the next tone of the identified second tone is the third tone, then the second tone is determined to be the second tone corresponding to the first third tone of two consecutive third tones, and the character in the first text corresponding to the second tone is the third character. The number of third characters may be one or more.

[0088] For each third character, determine whether the tone of the pronunciation of the third character in the first pronunciation sequence is marked as the third tone after the tone change (which can be represented by the number 6). If the tone of the pronunciation of the third character in the first pronunciation sequence is marked as the third tone after the tone change, then keep the tone of the pronunciation of the third character in the first pronunciation sequence unchanged. If the tone of the pronunciation of the third character in the first pronunciation sequence is not marked as the third tone after the tone change 6, then modify the tone of the pronunciation of the third character in the first pronunciation sequence to the third tone after the tone change 6.

[0089] For example, for a third character, the tone of the third character in the third pronunciation sequence is marked as the second tone 2, and the tone of the next character in the third pronunciation sequence is the third tone 3. If the tone of the third character in the first pronunciation sequence is marked as the third tone 6 after the tone change, the tone 6 of the third character in the first pronunciation sequence is kept unchanged. If the tone of the third character in the first pronunciation sequence is not marked as the third tone 6 after the tone change, the tone of the third character in the first pronunciation sequence is modified to the third tone 6 after the tone change.

[0090] It can be seen that through this embodiment, the third character can be recognized in the first text. If the tone of the pronunciation of the third character in the first pronunciation sequence is not marked as the third tone after the tone change, the tone of the pronunciation of the third character in the first pronunciation sequence is modified to the third tone after the tone change, thereby correcting the third tone after the tone change in the first pronunciation sequence, improving the accuracy of two consecutive third tones in the predicted text, and improving the accuracy of the pronunciation sequence of the predicted text.

[0091] The above introduces the detailed process of the audio processing method. The following introduces the training data and training process of the first pronunciation prediction system, the second pronunciation prediction system and the third pronunciation prediction system.

[0092] In one embodiment, the method further includes:

[0093] Acquire a first sample audio and a first sample pronunciation sequence corresponding to the first sample audio; the tone of the pronunciation of the first sample audio in the first sample pronunciation sequence includes a light tone and / or a third tone after tone change; the first third tone of two consecutive third tones in the first sample audio is marked as a third tone after tone change in the first sample pronunciation sequence;

[0094] Determine a first time alignment relationship between each audio time frame in the first sample audio and each pronunciation in the first sample pronunciation sequence;

[0095] Based on the first time alignment relationship, a first pronunciation prediction system is trained.

[0096] In the present embodiment, first obtain the first sample audio and the first sample pronunciation sequence corresponding to the first sample audio. The first sample audio can be any audio used to train the first pronunciation prediction system, and the first sample pronunciation sequence corresponding to the first sample audio is used to mark the pronunciation of each word in the first sample audio, and the pronunciation has a tone, and the pronunciation can be represented by pinyin or phoneme, and the first sample pronunciation sequence can be a pinyin sequence or a phoneme sequence. In the first sample pronunciation sequence, the tone of the pronunciation of the first sample audio includes a light tone and / or a three-tone tone after a tone change, that is, the tone of the pronunciation of the first sample audio includes at least one of a light tone and a three-tone tone after a tone change. Similar to the above, for the first three-tone tone of two consecutive three-tone tones in the first sample audio, the three-tone tone is marked as a three-tone tone after a tone change in the first sample pronunciation sequence, such as being marked as 6 tones. The first sample pronunciation sequence corresponding to the first sample audio can be obtained by manual marking, and this embodiment is not limited.

[0097] Next, the first time alignment relationship between each audio time frame in the first sample audio and each pronunciation in the first sample pronunciation sequence can be determined by a GMM-HMM (Gaussian Mixture Model-Hidden Markov Model) neural network. The first time alignment relationship represents the corresponding relationship between the duration of each pronunciation in the first sample pronunciation sequence and each audio time frame in the first sample audio. For example, it represents that the duration of the first pronunciation in the first sample pronunciation sequence corresponds to the first and second audio time frames in the first sample audio, and the duration of the second pronunciation in the first sample pronunciation sequence corresponds to the third audio time frame in the first sample audio. Through the first time correspondence, it can be clear which part of the audio time frame in the first sample audio each pronunciation in the first sample pronunciation sequence corresponds to.

[0098] Finally, based on the first time alignment relationship, the first pronunciation prediction system is trained. In one embodiment, the first acoustic module in the first pronunciation prediction system can be trained based on the first time alignment relationship, so that the trained first acoustic module learns which part of the audio time frame in the first sample audio each pronunciation in the first sample pronunciation sequence corresponds to, and learns the features of different pronunciations based on a large amount of first sample audio and the first sample pronunciation sequence corresponding to the first sample audio, thereby predicting the pronunciation corresponding to each time frame of the audio and the tone of the pronunciation.

[0099] During the process of training the first acoustic module, when the first acoustic module learns the pronunciation corresponding to each audio time frame in the first sample audio, it will learn multiple pronunciation results and generate a probability value for each pronunciation result. Combined with the above-mentioned first time alignment relationship, the first acoustic module can generate the maximum probability value for the correct pronunciation result, that is, the pronunciation result with the highest probability corresponding to each audio time frame learned by the first acoustic module is the correct pronunciation result, thereby improving the training accuracy of the first acoustic module.

[0100] Regarding the first decoding module in the first pronunciation prediction system, reference may be made to a general training method, which is not limited in this embodiment.

[0101] It can be seen that through this embodiment, the first pronunciation prediction system can be trained according to the first sample audio and the first sample pronunciation sequence. Since the tones of the pronunciation of the first sample audio in the first sample pronunciation sequence include light tones and / or three tones after tone change, the trained first pronunciation prediction system has the ability to predict the light tones and / or three tones after tone change of the text. Based on the first time alignment relationship, the first pronunciation prediction system is trained, which can make the pronunciation result with the highest probability learned by the first pronunciation prediction system for each audio time frame the correct pronunciation result, thereby improving the training accuracy of the first pronunciation prediction system.

[0102] In one embodiment, the method further includes:

[0103] Acquire the second sample audio and the second sample pronunciation sequence corresponding to the second sample audio; the second sample pronunciation sequence is used to mark the tone of each pronunciation in the second sample audio as a light tone or a non-light tone;

[0104] Determine a second time alignment relationship between each audio time frame in the second sample audio and each pronunciation in the second sample pronunciation sequence;

[0105] Based on the second time alignment relationship, a second pronunciation prediction system is trained.

[0106] In this embodiment, firstly, a second sample audio and a second sample pronunciation sequence corresponding to the second sample audio are obtained. The second sample audio can be any audio used to train the second pronunciation prediction system, and the second sample pronunciation sequence corresponding to the second sample audio is used to mark the tone of pronunciation of each word in the second sample audio as a light tone or a non-light tone, for example, label 1 indicates that the tone is a light tone, and label 0 indicates that the tone is a non-light tone. The second sample pronunciation sequence corresponding to the second sample audio can be obtained by manual annotation, which is not limited in this embodiment. The second sample audio can be the same as or different from the first sample audio, which is not limited here.

[0107] Next, the second time alignment relationship between each audio time frame in the second sample audio and each pronunciation in the second sample pronunciation sequence can be determined through the GMM-HMM neural network. The second time alignment relationship represents the corresponding relationship between the duration of each pronunciation in the second sample pronunciation sequence and each audio time frame in the second sample audio. For example, it represents that the duration of the first pronunciation in the second sample pronunciation sequence corresponds to the first and second audio time frames in the second sample audio, and the duration of the second pronunciation in the second sample pronunciation sequence corresponds to the third audio time frame in the second sample audio. Through the second time correspondence, it can be clear which part of the audio time frame in the second sample audio each pronunciation in the second sample pronunciation sequence corresponds to.

[0108] Finally, based on the second time alignment relationship, the second pronunciation prediction system is trained. In one embodiment, the second acoustic module in the second pronunciation prediction system can be trained based on the second time alignment relationship, so that the trained second acoustic module can learn which part of the audio time frame in the second sample audio each pronunciation in the second sample pronunciation sequence corresponds to based on a large amount of second sample audio and the second sample pronunciation sequence corresponding to the second sample audio, and learn the characteristics of the light tone and the characteristics of the non-light tone, so as to predict whether the tone of the pronunciation corresponding to each time frame of the audio is a light tone.

[0109] Regarding the second decoding module in the second pronunciation prediction system, reference may be made to a general training method, which is not limited in this embodiment.

[0110] It can be seen that through this embodiment, the second pronunciation prediction system can be trained according to the second sample audio and the second sample pronunciation sequence. Since the second sample pronunciation sequence is used to mark the tone of each pronunciation in the second sample audio as a light tone or a non-light tone, the trained second pronunciation prediction system has the ability to predict whether the pronunciation of the text is a light tone or a non-light tone, thereby facilitating the correction of the predicted light tone through the second pronunciation prediction system.

[0111] In one embodiment, the method further includes:

[0112] Obtaining a third sample audio and a third sample pronunciation sequence corresponding to the third sample audio; the first third tone of two consecutive third tones in the third sample audio is marked as the second tone in the third sample pronunciation sequence;

[0113] Determine a third time alignment relationship between each audio time frame in the third sample audio and each pronunciation in the third sample pronunciation sequence;

[0114] Based on the third time alignment relationship, a third pronunciation prediction system is trained.

[0115] In this embodiment, first obtain the third sample audio and the third sample pronunciation sequence corresponding to the third sample audio. The third sample audio can be any audio used to train the third pronunciation prediction system. The third sample pronunciation sequence corresponding to the third sample audio is used to mark the pronunciation of each word in the third sample audio. The pronunciation has a tone, and the pronunciation can be represented by pinyin or phoneme. The third sample pronunciation sequence can be a pinyin sequence or a phoneme sequence. For the first three-tone tone in two consecutive three-tone tones in the third sample audio, the three-tone tone is marked as a two-tone tone 2 in the third sample pronunciation sequence. Therefore, in the third sample pronunciation sequence, the tone of the pronunciation of the third sample audio includes the two-tone tone corresponding to the first three-tone tone in two consecutive three-tone tones. The third sample pronunciation sequence corresponding to the third sample audio can be obtained by manual annotation. For example, the first sample pronunciation sequence corresponding to the first sample audio is first obtained, and then the three-tone tone after the tone change in the first sample pronunciation sequence is modified to a two-tone tone to obtain the third sample pronunciation sequence, and the first sample audio is used as the third sample audio.

[0116] Next, the third time alignment relationship between each audio time frame in the third sample audio and each pronunciation in the third sample pronunciation sequence can be determined through the GMM-HMM neural network. The third time alignment relationship represents the corresponding relationship between the duration of each pronunciation in the third sample pronunciation sequence and each audio time frame in the third sample audio. For example, it represents that the duration of the first pronunciation in the third sample pronunciation sequence corresponds to the first and second audio time frames in the third sample audio, and the duration of the second pronunciation in the third sample pronunciation sequence corresponds to the third audio time frame in the third sample audio. Through the third time correspondence, it can be clear which part of the audio time frame in the third sample audio each pronunciation in the third sample pronunciation sequence corresponds to.

[0117] Finally, based on the third time alignment relationship, the third pronunciation prediction system is trained. In one embodiment, the third acoustic module in the third pronunciation prediction system can be trained based on the third time alignment relationship, so that the trained third acoustic module can learn which part of the audio time frame in the third sample audio each pronunciation in the third sample pronunciation sequence corresponds to based on a large amount of third sample audio and the third sample pronunciation sequence corresponding to the third sample audio, and learn the features of different pronunciations, thereby predicting the pronunciation corresponding to each time frame of the audio and the tone of the pronunciation.

[0118] During the process of training the third acoustic module, when the third acoustic module learns the pronunciation corresponding to each audio time frame in the third sample audio, it will learn multiple pronunciation results and generate a probability value for each pronunciation result. Combined with the above-mentioned third time alignment relationship, the third acoustic module can generate the maximum probability value for the correct pronunciation result, that is, the pronunciation result with the highest probability corresponding to each audio time frame learned by the third acoustic module is the correct pronunciation result, thereby improving the training accuracy of the third acoustic module.

[0119] Regarding the third decoding module in the third pronunciation prediction system, reference may be made to a general training method, which is not limited in this embodiment.

[0120] It can be seen that through this embodiment, the third pronunciation prediction system can be trained according to the third sample audio and the third sample pronunciation sequence. Since the first third tone of two consecutive third tones in the third sample audio is marked as the second tone in the third sample pronunciation sequence, the trained third pronunciation prediction system has the ability to predict the first third tone of two consecutive third tones as the second tone, thereby facilitating the correction of the predicted third tone after tone change through the third pronunciation prediction system.

[0121] In summary, through the above-mentioned audio processing method, not only can the first pronunciation sequence of the first text be predicted, but also the unstressed tone in the first pronunciation sequence can be corrected, and / or the third tone after the tone change in the first pronunciation sequence can be corrected, thereby improving the accuracy of the unstressed tone and two consecutive third tones in the predicted text, and improving the accuracy of the pronunciation sequence of the predicted text. In addition, through the above-mentioned audio processing method, the first decoding module selects a first target pronunciation for each word in the first text according to at least one pronunciation of each word in the predicted first text, and by comparing the text and the multiple possible pronunciations corresponding to the text, the accuracy of the pronunciation of the predicted polyphonic characters can also be improved.

[0122] Figure 5 A structural diagram of an audio processing device provided by an embodiment of the present disclosure is shown in FIG. Figure 5 As shown, the device comprises:

[0123] A data acquisition unit 51 is used to acquire a first audio and a first text corresponding to the first audio;

[0124] The pronunciation prediction unit 52 is used to predict a first pronunciation sequence of the first text according to the first audio and the first text through a first pronunciation prediction system; the tones of the pronunciation of the characters in the first text marked in the first pronunciation sequence include a light tone and / or a third tone after tone change; the first third tone of two consecutive third tones in the first text is marked as a third tone after tone change in the first pronunciation sequence;

[0125] The pronunciation correction unit 53 is used to correct the unstressed tone in the first pronunciation sequence through a second pronunciation prediction system, and / or to correct the modulated third tone in the first pronunciation sequence through a third pronunciation prediction system.

[0126] Optionally, the pronunciation prediction unit 52 is specifically used for:

[0127] Predicting at least one pronunciation of each word in the first text according to the first audio by a first acoustic module in the first pronunciation prediction system; the tones of the pronunciations include a light tone and / or three tones after tone change;

[0128] selecting, by a first decoding module in the first pronunciation prediction system, a first target pronunciation for each character in the first text from at least one pronunciation of each character in the first text predicted by the first text;

[0129] The first pronunciation sequence is generated according to the first target pronunciation through the first decoding module.

[0130] Optionally, the pronunciation correction unit 53 is specifically used for:

[0131] Predicting, by a second pronunciation prediction system, a second pronunciation sequence of the first text according to the first audio and the first text, wherein the second pronunciation sequence is used to mark the tone of pronunciation of a character in the first text as a light tone or a non-light tone;

[0132] The unvoiced tones in the first pronunciation sequence are corrected according to the second pronunciation sequence.

[0133] Optionally, the pronunciation correction unit 53 is specifically used for:

[0134] Predicting, by a third pronunciation prediction system, a third pronunciation sequence of the first text according to the first audio and the first text, the third pronunciation sequence being used to mark the first third tone of two consecutive third tones in the first text with a second tone;

[0135] The three tones after the tone change in the first pronunciation sequence are corrected according to the third pronunciation sequence.

[0136] Optionally, the pronunciation correction unit 53 is further specifically configured to:

[0137] Predicting, by a second acoustic module in the second pronunciation prediction system, a probability that the tone of pronunciation of each word in the first text is a light tone according to the first audio;

[0138] Determining, by a second decoding module in the second pronunciation prediction system, whether the tone of pronunciation of each character in the first text is a light tone or a non-light tone according to the first text and the predicted probability that the tone of pronunciation of each character in the first text is a light tone;

[0139] The second pronunciation sequence is generated according to the determination result through the second decoding module.

[0140] Optionally, the pronunciation correction unit 53 is further specifically configured to:

[0141] Determining whether the tone of the pronunciation of the first character in the first text in the first pronunciation sequence is marked as a light tone or a non-light tone; the tone of the pronunciation of the first character in the second pronunciation sequence is marked as a light tone;

[0142] If it is a light tone, the tone of the pronunciation of the first character in the first pronunciation sequence is kept unchanged;

[0143] If it is not a light tone, the tone of the pronunciation of the first character in the first pronunciation sequence is corrected to a light tone.

[0144] Optionally, the pronunciation correction unit 53 is further specifically configured to:

[0145] Predicting at least one pronunciation of each word in the first text according to the first audio by a third acoustic module in the third pronunciation prediction system; the tone of the pronunciation includes a second tone corresponding to the first third tone of two consecutive third tones;

[0146] selecting, by a third decoding module in the third pronunciation prediction system, a second target pronunciation for each character in the first text from at least one pronunciation of each character in the first text predicted by the first text;

[0147] The third pronunciation sequence is generated according to the second target pronunciation by the third decoding module.

[0148] Optionally, the pronunciation correction unit 53 is further specifically configured to:

[0149] Determine whether the tone of the pronunciation of the second character in the first text in the third pronunciation sequence is marked as the second tone; the tone of the pronunciation of the second character in the first pronunciation sequence is marked as the third tone after the tone change;

[0150] If yes, keeping the tone of the pronunciation of the second character in the first pronunciation sequence unchanged;

[0151] If not, the tone of the pronunciation of the second character in the first pronunciation sequence is modified according to the tone of the pronunciation of the second character in the third pronunciation sequence.

[0152] Optionally, the pronunciation correction unit 53 is further specifically configured to:

[0153] Determine whether the tone of the pronunciation of the third character in the first text in the first pronunciation sequence is marked as the third tone after the tone change; the tone of the pronunciation of the third character in the third pronunciation sequence is marked as the second tone corresponding to the first third tone of two consecutive third tones;

[0154] If yes, keeping the tone of the pronunciation of the third character in the first pronunciation sequence unchanged;

[0155] If not, the tone of the pronunciation of the third character in the first pronunciation sequence is modified to the third tone after the tone change.

[0156] Optionally, the method further includes a first training unit, configured to:

[0157] Acquire a first sample audio and a first sample pronunciation sequence corresponding to the first sample audio; the tone of the pronunciation of the first sample audio in the first sample pronunciation sequence includes a light tone and / or a three-tone tone after tone change; the first three-tone tone of two consecutive three-tone tones in the first sample audio is marked as a three-tone tone after tone change in the first sample pronunciation sequence;

[0158] Determine a first time alignment relationship between each audio time frame in the first sample audio and each pronunciation in the first sample pronunciation sequence;

[0159] Based on the first time alignment relationship, the first pronunciation prediction system is trained.

[0160] Optionally, the method further includes a second training unit, configured to:

[0161] Acquire a second sample audio and a second sample pronunciation sequence corresponding to the second sample audio; the second sample pronunciation sequence is used to mark the tone of each pronunciation in the second sample audio as a light tone or a non-light tone;

[0162] Determine a second time alignment relationship between each audio time frame in the second sample audio and each pronunciation in the second sample pronunciation sequence;

[0163] Based on the second time alignment relationship, the second pronunciation prediction system is trained.

[0164] Optionally, a third training unit is further included, configured to:

[0165] Acquire a third sample audio and a third sample pronunciation sequence corresponding to the third sample audio; the first three-tone of two consecutive three-tones in the third sample audio is marked as a second tone in the third sample pronunciation sequence;

[0166] Determine a third time alignment relationship between each audio time frame in the third sample audio and each pronunciation in the third sample pronunciation sequence;

[0167] Based on the third time alignment relationship, the third pronunciation prediction system is trained.

[0168] The audio processing device in the embodiment of the present disclosure can implement the various processes of the above-mentioned audio processing method embodiment and achieve the same effects and functions, which will not be repeated here.

[0169] An embodiment of the present disclosure further provides an electronic device, Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure, such as Figure 6 As shown, the electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors 601 and memory 602, and one or more applications or data may be stored in the memory 602. Among them, the memory 602 may be a short-term storage or a persistent storage. The application stored in the memory 602 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the electronic device. Furthermore, the processor 601 may be configured to communicate with the memory 602 to execute a series of computer executable instructions in the memory 602 on the electronic device. The electronic device may also include one or more power supplies 603, one or more wired or wireless network interfaces 604, one or more input or output interfaces 605, one or more keyboards 606, etc.

[0170] In a specific embodiment, the electronic device includes a processor; and a memory configured to store computer executable instructions, wherein when the computer executable instructions are executed, the processor implements the following process:

[0171] Acquire a first audio and a first text corresponding to the first audio;

[0172] Predicting a first pronunciation sequence of the first text according to the first audio and the first text by a first pronunciation prediction system; the tones of the pronunciation of the characters in the first text marked in the first pronunciation sequence include a light tone and / or a third tone after tone change; the first third tone of two consecutive third tones in the first text is marked as the third tone after tone change in the first pronunciation sequence;

[0173] The unstressed tone in the first pronunciation sequence is corrected by a second pronunciation prediction system, and / or the modulated third tone in the first pronunciation sequence is corrected by a third pronunciation prediction system.

[0174] The electronic device in the embodiment of the present disclosure can implement each process of the above-mentioned audio processing method embodiment and achieve the same effect and function, which will not be repeated here.

[0175] Another embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the following process is implemented:

[0176] Acquire a first audio and a first text corresponding to the first audio;

[0177] Predicting a first pronunciation sequence of the first text according to the first audio and the first text by a first pronunciation prediction system; the tones of the pronunciation of the characters in the first text marked in the first pronunciation sequence include a light tone and / or a third tone after tone change; the first third tone of two consecutive third tones in the first text is marked as the third tone after tone change in the first pronunciation sequence;

[0178] The unstressed tone in the first pronunciation sequence is corrected by a second pronunciation prediction system, and / or the modulated third tone in the first pronunciation sequence is corrected by a third pronunciation prediction system.

[0179] The storage medium in the embodiment of the present disclosure can implement each process of the above-mentioned audio processing method embodiment and achieve the same effect and function, which will not be repeated here.

[0180] In various embodiments of the present disclosure, the computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0181] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0182] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0183] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0184] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of the present disclosure, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0185] Those skilled in the art will appreciate that one or more embodiments of the present disclosure may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, one or more embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0186] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0187] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0188] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0189] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0190] One or more embodiments of the present disclosure may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present disclosure may also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0191] Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0192] The above description is only an embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the claims of the present disclosure.

Claims

1. An audio processing method, characterized in that: include: Acquire a first audio and a first text corresponding to the first audio; Predicting a first pronunciation sequence of the first text according to the first audio and the first text by a first pronunciation prediction system; the tones of the pronunciation of the characters in the first text marked in the first pronunciation sequence include a light tone and / or a third tone after tone change; the first third tone of two consecutive third tones in the first text is marked as the third tone after tone change in the first pronunciation sequence; The unstressed tone in the first pronunciation sequence is corrected by a second pronunciation prediction system, and / or the modulated third tone in the first pronunciation sequence is corrected by a third pronunciation prediction system.

2. The method according to claim 1, characterized in that The predicting, by the first pronunciation prediction system, the first pronunciation sequence of the first text according to the first audio and the first text comprises: Predicting at least one pronunciation of each word in the first text according to the first audio by a first acoustic module in the first pronunciation prediction system; the tones of the pronunciations include a light tone and / or three tones after tone change; selecting, by a first decoding module in the first pronunciation prediction system, a first target pronunciation for each character in the first text from at least one pronunciation of each character in the first text predicted by the first text; The first pronunciation sequence is generated according to the first target pronunciation through the first decoding module.

3. The method according to claim 1, characterized in that The correcting of the unvoiced tone in the first pronunciation sequence by the second pronunciation prediction system comprises: Predicting, by a second pronunciation prediction system, a second pronunciation sequence of the first text according to the first audio and the first text, wherein the second pronunciation sequence is used to mark the tone of pronunciation of a character in the first text as a light tone or a non-light tone; The unvoiced tones in the first pronunciation sequence are corrected according to the second pronunciation sequence.

4. The method according to claim 1, characterized in that: The method of correcting the three tones after the tone change in the first pronunciation sequence by the third pronunciation prediction system comprises: Predicting, by a third pronunciation prediction system, a third pronunciation sequence of the first text according to the first audio and the first text, the third pronunciation sequence being used to mark the first third tone of two consecutive third tones in the first text with a second tone; The three tones after the tone change in the first pronunciation sequence are corrected according to the third pronunciation sequence.

5. The method according to claim 3, characterized in that: The step of predicting a second pronunciation sequence of the first text according to the first audio and the first text by a second pronunciation prediction system includes: Predicting, by a second acoustic module in the second pronunciation prediction system, a probability that the tone of pronunciation of each word in the first text is a light tone according to the first audio; Determining, by a second decoding module in the second pronunciation prediction system, whether the tone of pronunciation of each character in the first text is a light tone or a non-light tone according to the first text and the predicted probability that the tone of pronunciation of each character in the first text is a light tone; The second pronunciation sequence is generated according to the determination result through the second decoding module.

6. The method according to claim 3, characterized in that The correcting of the unvoiced tone in the first pronunciation sequence according to the second pronunciation sequence comprises: Determining whether the tone of the pronunciation of the first character in the first text in the first pronunciation sequence is marked as a light tone or a non-light tone; the tone of the pronunciation of the first character in the second pronunciation sequence is marked as a light tone; If it is a light tone, the tone of the pronunciation of the first character in the first pronunciation sequence is kept unchanged; If it is not a light tone, the tone of the pronunciation of the first character in the first pronunciation sequence is corrected to a light tone.

7. The method according to claim 4, characterized in that The predicting, by the third pronunciation prediction system, a third pronunciation sequence of the first text according to the first audio and the first text comprises: Predicting at least one pronunciation of each word in the first text according to the first audio by a third acoustic module in the third pronunciation prediction system; the tone of the pronunciation includes a second tone corresponding to the first third tone of two consecutive third tones; selecting, by a third decoding module in the third pronunciation prediction system, a second target pronunciation for each character in the first text from at least one pronunciation of each character in the first text predicted by the first text; The third pronunciation sequence is generated according to the second target pronunciation by the third decoding module.

8. The method according to claim 4, characterized in that The step of correcting the three tones after the tone change in the first pronunciation sequence according to the third pronunciation sequence comprises: Determine whether the tone of the pronunciation of the second character in the first text in the third pronunciation sequence is marked as the second tone; the tone of the pronunciation of the second character in the first pronunciation sequence is marked as the third tone after the tone change; If yes, keeping the tone of the pronunciation of the second character in the first pronunciation sequence unchanged; If not, the tone of the pronunciation of the second character in the first pronunciation sequence is modified according to the tone of the pronunciation of the second character in the third pronunciation sequence.

9. The method according to claim 4, characterized in that The step of correcting the three tones after the tone change in the first pronunciation sequence according to the third pronunciation sequence comprises: Determine whether the tone of the pronunciation of the third character in the first text in the first pronunciation sequence is marked as the third tone after the tone change; the tone of the pronunciation of the third character in the third pronunciation sequence is marked as the second tone corresponding to the first third tone of two consecutive third tones; If yes, keeping the tone of the pronunciation of the third character in the first pronunciation sequence unchanged; If not, the tone of the pronunciation of the third character in the first pronunciation sequence is modified to the third tone after the tone change.

10. The method according to claim 1, characterized in that The method further comprises: Acquire a first sample audio and a first sample pronunciation sequence corresponding to the first sample audio; the tone of the pronunciation of the first sample audio in the first sample pronunciation sequence includes a light tone and / or a three-tone tone after tone change; the first three-tone tone of two consecutive three-tone tones in the first sample audio is marked as a three-tone tone after tone change in the first sample pronunciation sequence; Determine a first time alignment relationship between each audio time frame in the first sample audio and each pronunciation in the first sample pronunciation sequence; Based on the first time alignment relationship, the first pronunciation prediction system is trained.

11. The method according to claim 3, characterized in that The method further comprises: Acquire a second sample audio and a second sample pronunciation sequence corresponding to the second sample audio; the second sample pronunciation sequence is used to mark the tone of each pronunciation in the second sample audio as a light tone or a non-light tone; Determine a second time alignment relationship between each audio time frame in the second sample audio and each pronunciation in the second sample pronunciation sequence; Based on the second time alignment relationship, the second pronunciation prediction system is trained.

12. The method according to claim 4, characterized in that The method further comprises: Acquire a third sample audio and a third sample pronunciation sequence corresponding to the third sample audio; the first three-tone of two consecutive three-tones in the third sample audio is marked as a second tone in the third sample pronunciation sequence; Determine a third time alignment relationship between each audio time frame in the third sample audio and each pronunciation in the third sample pronunciation sequence; Based on the third time alignment relationship, the third pronunciation prediction system is trained.

13. An audio processing device, characterized in that: include: A data acquisition unit, used to acquire a first audio and a first text corresponding to the first audio; A pronunciation prediction unit is used to predict a first pronunciation sequence of the first text according to the first audio and the first text through a first pronunciation prediction system; the tones of the pronunciation of the characters in the first text marked in the first pronunciation sequence include a light tone and / or a third tone after tone change; the first third tone of two consecutive third tones in the first text is marked as the third tone after tone change in the first pronunciation sequence; The pronunciation correction unit is used to correct the unstressed tone in the first pronunciation sequence through a second pronunciation prediction system, and / or to correct the modulated third tone in the first pronunciation sequence through a third pronunciation prediction system.

14. An electronic device, characterized in that: include: processor; as well as, A memory configured to store computer executable instructions, which, when executed, cause the processor to implement the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions, when executed by a processor, implement the steps of the method described in any one of claims 1 to 12.

Images