Audio synthesis method, system, electronic device, and computer-readable storage medium

Abstract

The audio synthesis method, system (30), electronic device (40), and computer-readable storage medium (50) includes the steps of: obtaining a colloquialized text by inserting colloquialized expression content, which includes at least newly added colloquialized content and colloquialized pause intervals, into the text to be processed, depending on the text to be processed (S101); obtaining prosodic features of the colloquialized text and obtaining prosodic pause intervals of the colloquialized text based on the prosodic features (S102); and converting the colloquialized text into target audio based on the colloquialized pause intervals and prosodic pause intervals (S103). The method can improve the degree of anthropomorphism of the audio synthesis.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and particularly to an audio synthesis method, system, electronic device, and computer-readable storage medium.

Background Art

[0002] With the development of smart devices, there are increasingly more smart devices that can support human-computer interaction. During human-computer interaction, usually, it is necessary to synthesize text data into audio for feedback. However, in the conventional human-computer interaction process, generally, only text data is converted into unnatural audio, so the degree of anthropomorphism of audio synthesis is insufficient. In view of this, how to improve the degree of anthropomorphism of audio synthesis has become an urgent problem to be solved.

Summary of the Invention

Problems to be Solved by the Invention

[0003] The main problem to be solved by this application is to provide an audio synthesis method, system, electronic device, and computer-readable storage medium that can improve the degree of anthropomorphism of audio synthesis.

Means for Solving the Problems

[0004] To solve the above problems, a first aspect of this application is an audio synthesis method, which includes: obtaining a spoken expression content including at least a newly added colloquial content and a colloquial pause interval in response to obtaining a text to be processed, and inserting the spoken expression content into the text to be processed to obtain a colloquial text; obtaining the prosodic features of the colloquial text, and obtaining the prosodic pause interval of the colloquial text based on the prosodic features; and converting the colloquial text into a target audio based on the colloquial pause interval and the prosodic pause interval.

[0005] To solve the above problems, the second aspect of the present invention provides an audio synthesis system comprising: a conversion module for obtaining colloquialized text by inserting colloquialized expression content, which includes at least newly added colloquialized content and colloquialized pause intervals, into the text to be processed, in response to the acquisition of the text to be processed; an analysis module for acquiring the prosodic features of the colloquialized text and obtaining the prosodic pause intervals of the colloquialized text based on the prosodic features; and a generation module for converting the colloquialized text into target audio based on the colloquialized pause intervals and the prosodic pause intervals.

[0006] To solve the above problems, the third aspect of the present application provides an electronic device comprising a memory and a processor connected to each other, wherein program instructions are stored in the memory, and the processor is for executing the program instructions so that the audio synthesis method described in the first aspect is realized.

[0007] To solve the above problems, the fourth aspect of the present invention provides a computer-readable storage medium that stores program instructions, wherein when the program instructions are executed by a processor, the audio synthesis method described in the first aspect is realized. [Effects of the Invention]

[0008] According to the above embodiment, once the text to be processed is obtained, colloquialized expression content that includes at least newly added colloquialized content and colloquialized pause intervals is inserted into the text to be processed, thereby converting the text to be processed into colloquialized text. As a result, the colloquialized text can contain more content that will be added during the colloquialized expression and more pauses that will be adopted during the colloquialized expression. The prosodic features of the converted colloquialized text are obtained, enabling the newly added colloquialized content to influence the prosodic features. Furthermore, based on the prosodic features, prosodic pause intervals corresponding to the colloquialized text are obtained, and the prosodic pause intervals are also made more colloquial. Based on the colloquialized pause intervals and prosodic pause intervals, audio synthesis is performed on the colloquialized text containing the newly added colloquialized content, converting the colloquialized text into target audio. This enables the target audio to include the newly added colloquialized content during playback, and the pause intervals are also made more colloquial, thereby improving the degree of personification of the audio synthesis. [Brief explanation of the drawing]

[0009] To more clearly explain the technical aspects of the embodiments of the present application, the drawings that may be used in describing the embodiments are briefly introduced below. However, it is clear that the drawings described below represent only a few embodiments of the present application, and those skilled in the art can obtain other drawings from these without any creative effort. [Figure 1] This is a schematic flowchart of one embodiment of the audio synthesis method according to the present application. [Figure 2] This is a schematic flowchart of another embodiment of the audio synthesis method according to the present application. [Figure 3] This is a schematic diagram of the structure of one embodiment of the audio synthesis system according to the present invention. [Figure 4] This is a schematic diagram of the structure of one embodiment of the electronic device relating to the present application. [Figure 5] This is a schematic diagram of the structure of one embodiment of a computer-readable storage medium relating to the present application. [Modes for carrying out the invention]

[0010] The technical aspects of the embodiments of this application will be described clearly and completely below, with reference to the drawings of the embodiments. It is clear that the embodiments described are only a selection of, and not all, embodiments of this application. Any other embodiments obtained by those skilled in the art without creative work based on the embodiments of this application are all included within the scope of protection of this application.

[0011] In this specification, the terms "system" and "network" are often used interchangeably. In this specification, the term "and / or" simply describes the correlation between related objects, indicating that there are three possible relationships. For example, A and / or B could represent three cases: A exists alone, both A and B exist, or B exists alone. In this specification, the symbol " / " generally indicates that the related objects before and after it are in an "or" relationship. Furthermore, in this specification, "plural" refers to two or more.

[0012] The audio synthesis method according to this application is for synthesizing and obtaining highly anthropomorphic audio, and the implementing entity is a processor capable of processing text and audio. The processor may be a smart terminal, for example, a smart terminal capable of human-computer interaction with a user, but this application does not specifically limit it.

[0013] Referring to Figure 1, Figure 1 is a schematic flowchart of one embodiment of the audio synthesis method according to the present application, which includes the following steps S101 to S103.

[0014] S101 is to obtain colloquialized text by inserting colloquialized expression content, which includes at least newly added colloquialized content and colloquialized pause intervals, into the text to be processed, in accordance with the acquisition of the text to be processed.

[0015] Specifically, once the text to be processed is obtained, colloquialized expressions that include at least newly added colloquialized content and colloquialized pause intervals are inserted into the text to be processed, thereby converting the text to colloquialized text.

[0016] It can be understood that colloquialized texts can contain more content that will be added when colloquializing, as well as more punctuation that will be adopted when colloquializing.

[0017] In one embodiment, once the text to be processed is obtained, it is input into a pre-trained conversion model, which inserts colloquialized expressions into the text to be processed, thereby obtaining colloquialized text output from the conversion model. The conversion model is obtained through training using written text and its corresponding colloquialized expression text.

[0018] In one embodiment, once the text to be processed is obtained, a first hint text is constructed based on the text to be processed, and the first hint text is input into an intelligent analysis model to obtain a colloquialized text fed back by the intelligent analysis model. Of these, the first hint text is intended to prompt the intelligent analysis model to perform colloquialization.

[0019] Selectively, intelligent analytical models are large-scale language models, which may include, but are not limited to, deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory (LSTM), and generative pre-trained Transformer models. Here, we do not specifically limit the construction and deployment of large-scale language models.

[0020] Furthermore, the colloquial expression content includes colloquial newly added content and colloquial pause intervals. The colloquial newly added content includes at least various types of colloquial expression words, and the colloquial pause intervals include pause intervals of multiple types of time lengths. Also, the colloquial expression content may further include colloquial adjustment content. Among them, the colloquial adjustment content is for replacing some content in the text to be processed so that the relatively formal expression content in the text to be processed is replaced by the colloquial adjustment content.

[0021] It should be noted that various types of colloquial expression words include at least interjections, filler words, and repetition words, and the onomatopoeic audio includes at least laughter, inhalation sounds, and tongue clicking sounds.

[0022] S102 is to obtain the prosodic features of the colloquial text and obtain the prosodic pause intervals of the colloquial text based on the prosodic features.

[0023] Specifically, obtain the prosodic features of the converted colloquial text, enable the newly added colloquial content to affect the prosodic features, and further obtain the prosodic pause intervals corresponding to the colloquial text based on the prosodic features, and make the prosodic pause intervals more colloquial.

[0024] In one embodiment, convert the predetermined characters in the colloquial text into a suitable text form, divide the adjusted text into basic units. Among them, the predetermined characters include at least numbers, abbreviations, and currency symbols, the basic units include at least words and punctuation marks. Determine the prosodic features of the colloquial text based on the divided text, and segment the colloquial text based on the prosodic features to obtain the prosodic pause intervals of the colloquial text.

[0025] In one embodiment, obtain the syntax, conversation structure, and information structure of the colloquial text, determine the prosodic features of the colloquial text based on the syntax, conversation structure, and information structure, and segment the colloquial text based on the prosodic features to obtain the prosodic pause intervals of the colloquial text.

[0026] It is understandable that the colloquial pauses inserted into the colloquialized text are unrelated to the text's content, and while these pauses may be masked when acquiring the prosodic features of the colloquialized text, newly added colloquialized content is already present in the colloquialized text and influences the prosodic features during the extraction process, making the prosodic features more closely resemble the colloquialized expression.

[0027] In one implementation scenario, colloquialized text is input into a text analysis model, and the prosodic pause intervals fed back by the text analysis model are obtained. The text analysis model was obtained through training using multiple literary texts and colloquial expression texts corresponding to at least some of the literary texts.

[0028] S103 is to convert the colloquialized text into target audio based on colloquialization pause intervals and prosodic pause intervals.

[0029] Specifically, based on colloquialization pause intervals and prosodic pause intervals, audio synthesis is performed on the colloquialized text containing newly added colloquialized content to convert the colloquialized text into target audio. This ensures that the target audio includes the newly added colloquialized content during playback, and also makes the pause intervals more colloquial.

[0030] In one embodiment, colloquial pause intervals and prosodic pause intervals are merged to obtain text pause intervals for the colloquialized text, a phoneme sequence corresponding to the colloquialized text is obtained based on the text pause intervals and the words in the colloquialized text, and the phoneme sequence is converted into target audio corresponding to the colloquialized text.

[0031] In one embodiment, the colloquialized text is divided into multiple word segments based on colloquialization pause intervals and prosodic pause intervals, the phoneme sequences corresponding to the multiple word segments are determined, and the phoneme sequences are converted into target audio corresponding to the colloquialized text.

[0032] In one implementation scenario, a phoneme sequence is input into an end-to-end audio conversion model, and the target audio output from the audio conversion model is obtained. The audio conversion model is obtained through training using phoneme sequence samples and their corresponding converted audio.

[0033] In one application scenario, the text to be processed is the response text generated during human-computer interaction between the user and the smart device. By converting the text to colloquial language and outputting it as a highly anthropomorphic target audio, the user's human-computer interaction experience becomes more realistic and relatable.

[0034] In another application scenario, the text to be processed is text obtained from user input information, which includes at least one of text, audio, and images. The user input information is ultimately converted into the text to be processed, and the text to be processed is output as a highly personified target audio after undergoing colloquialization. This process adjusts the original expressions within the text to colloquial expressions, converting them into playable audio. For example, if the text to be processed originally contained awkward literary expressions, the acceptance by the user is improved after it is converted into the target audio and played back.

[0035] Selectively, the text to be processed may be a student's written language knowledge text, and when the text to be processed is converted into target audio, students will find it easier to listen to the converted target audio and will be able to learn the knowledge points better. The text to be processed may also be a child's story text, and when the text to be processed is converted into target audio, children will find it easier to listen to the converted target audio and will be able to improve their sense of familiarity when listening to the story. Furthermore, this application will not list in detail the texts to be processed used in other specific application scenarios.

[0036] According to the above embodiment, once the text to be processed is obtained, colloquialized expression content that includes at least newly added colloquialized content and colloquialized pause intervals is inserted into the text to be processed, thereby converting the text to be processed into colloquialized text. As a result, the colloquialized text can contain more content that will be added during the colloquialized expression and more pauses that will be adopted during the colloquialized expression. The prosodic features of the converted colloquialized text are obtained, enabling the newly added colloquialized content to influence the prosodic features. Furthermore, based on the prosodic features, prosodic pause intervals corresponding to the colloquialized text are obtained, and the prosodic pause intervals are also made more colloquial. Based on the colloquialized pause intervals and prosodic pause intervals, audio synthesis is performed on the colloquialized text containing the newly added colloquialized content, converting the colloquialized text into target audio. This enables the target audio to include the newly added colloquialized content during playback, and the pause intervals are also made more colloquial, thereby improving the degree of personification of the audio synthesis.

[0037] Referring to Figure 2, which is a schematic flowchart of another embodiment of the audio synthesis method according to the present invention, the method includes the following steps S201 to S203.

[0038] S201 involves inputting the text to be processed into a text conversion model, and obtaining a colloquialized text fed back by the text conversion model. The text conversion model is obtained through training using parallel text pairs, which include a first training text and a second training text, with the second training text having colloquialized expressions inserted compared to the first training text.

[0039] Specifically, once the text to be processed is obtained, it is input into a text conversion model, and the text to be processed is converted into colloquial text by obtaining colloquial text as feedback from the text conversion model. The text conversion model is obtained through pre-training using parallel text pairs, and the parallel text pairs include the first training text and its corresponding second training text.

[0040] What needs to be explained is that the second training text contains more colloquialized expressions compared to the first training text, and these colloquialized expressions include at least newly added colloquialized content and colloquialized pause intervals. Therefore, the second training text is more colloquial than the first training text, and the colloquialized expressions inserted into the second training text can be used as training tags for supervised training of the text conversion model.

[0041] Furthermore, by utilizing a text conversion model, it is possible to efficiently insert colloquialized expressions into the text to be processed, thus effectively completing the conversion from the text to the colloquialized text.

[0042] In other words, the first training text included in a parallel text pair is typically a literary expression, some of which possess some colloquial features, and the second training text is a colloquial expression generated after colloquial content has been inserted into the first training text. A text conversion model obtained through training using parallel text pairs can convert literary texts into colloquial texts, and also convert texts with some colloquial features into fully colloquial texts.

[0043] What needs to be explained is that new colloquialization additions include at least one of a modal word, filler word, repetition word, and onomatopoeic audio; colloquialization pause intervals include pause intervals of multiple durations; each type of new colloquialization addition is preceded by a corresponding new addition start tag; each type of new colloquialization addition is preceded by a corresponding new addition end tag; and each type of new colloquialization pause interval is positioned with a corresponding pause tag.

[0044] Specifically, the newly added colloquial elements may include not only modal words, filler words, repeated words, and onomatopoeic audio, but may also include other newly added elements such as retroflex consonants (r-sounds), although this application does not specifically limit them. Onomatopoeic audio may also include laughter, inhalation sounds, coughing sounds, and tongue clicks, although this application does not specifically limit them.

[0045] Furthermore, when inserting newly added colloquialized content into the text, a suitable new addition start tag is inserted before each type of newly added colloquialized content, and a new addition end tag is inserted after each type of newly added colloquialized content. For example, the new addition start tag for filler words is fib, the new addition start tag for filler words is fie, the new addition start tag for modal words is yqb, and the new addition start tag for filler words is yqe. This application does not list other specific tags one by one.

[0046] It can be understood that colloquial pause intervals include at least several types of pause intervals of duration, such as pauses, short pauses, long pauses, and very long pauses, and each type of colloquial pause interval is assigned a corresponding pause tag. For example, the pause tag for a short pause is spa, and the pause tag for a long pause is lpa. Other specific tags are not listed individually in this application.

[0047] For the sake of explanation, using parallel text pairs as an example, the first training sample is in formal language and specifically includes "For us, breakfast is definitely soy milk, fried bread, and steamed bread." The second training sample is in colloquial language, and after the colloquialized content is inserted, " <fib>you know <fie>If it were us, in the morning <fib>fundamentally <fie>also <spa>This includes "soy milk, fried bread, and steamed bread." Therefore, the new addition start tag and new addition end tag allow for the classification and positioning of colloquialized new addition content, the pause tag allows for the classification and positioning of colloquialized pause content, and the text conversion model can complete training through rule definition or adaptive learning. After training, the text conversion model can insert colloquialized expressions into the text to be processed and set appropriate tags at the corresponding positions.

[0048] In one embodiment, the text conversion model includes an encoder and a decoder, and the first training text is converted to a first predicted text after passing through the encoder and decoder, respectively. The colloquialized expressions inserted into the second training text are used as training tags for the text conversion model.

[0049] Specifically, the training process of the text conversion model includes inputting a first training text into an encoder to obtain a first encoded text output from the encoder, inputting the first encoded text into a decoder to obtain a first predicted text output from the decoder, comparing the first predicted text with the second training text to determine the first training loss based on the colloquialized expression content contained in the second training text and the predicted expression content inserted into the first predicted text, adjusting the parameters of the encoder and decoder based on the first training loss until the first convergence condition is met, and obtaining a post-trained text conversion model. Therefore, since the text conversion model is a model built on an encoder and decoder and may be acquired through supervised training, the post-trained text conversion model can complete colloquialization conversion for the text to be processed, and the controllability of the colloquialization conversion process is improved.

[0050] In another embodiment, the text conversion model includes a hint building unit and an intelligent analysis model, the hint building unit for building hint text based on at least a first training text and inputting it into the intelligent analysis model, the hint text for prompting the intelligent analysis model to perform colloquialization and output a second predicted text, and if the hint text further includes some parallel text pairs, the parallel text pairs are used as example samples. Of these, the colloquialized expressions inserted into the second training text are used as training tags for the text conversion model.

[0051] Specifically, the hint construction unit within the text conversion model is for constructing hint text, which includes at least the first training text and the colloquialization conversion requirements. If the hint text also includes some parallel text pairs, these parallel text pairs are used as exemplary samples that conform to the colloquialization conversion requirements. When the hint text is provided to the intelligent analysis model, the intelligent analysis model feeds back a second predicted text. The second predicted text is compared with the second training text to determine the second training loss based on the colloquialization expressions contained in the second training text and the predicted expressions inserted into the second predicted text. Based on the second training loss, the intelligent analysis model is fine-tuned until the second convergence condition is met, resulting in a post-trained text conversion model. Therefore, since the text conversion model may be obtained after fine-tuning the intelligent analysis model through supervised training, the post-trained text conversion model can complete colloquialization conversion for the text being processed, reducing the consumption of colloquialization conversion on the terminal.

[0052] In one implementation scenario, fine-tuning of the intelligent analysis model may involve constructing hint text using a zero-shot method. For example, the first training sample is in formal language and specifically includes "If it were us, we'd still have soy milk, fried bread, and steamed bread for breakfast." The hint text constructed using the zero-shot method includes "Perform a colloquialization of the text {If it were us, we'd still have soy milk, fried bread, and steamed bread for breakfast} and set tags at the corresponding positions."

[0053] In another implementation scenario, fine-tuning of the intelligent analysis model may involve constructing hint text using a small-shot method, where the hint text constructed using the small-shot method is, for example, "The literary expression text is 'In Southern cuisine, I love using dip sauces,' and the rewritten colloquial expression text is, <fib>Yeah <fie>In Southern cuisine, <fib>that <fie> <std>I love using dipping sauces and things like that. <yqb>hey <yqe>The output will be ". Please rewrite the following literary text {If it were us, we would still have soy milk, fried bread, and steamed bread for breakfast} in accordance with the above explanation. Therefore, if the hint text contains some parallel text pairs, these parallel text pairs will be used as example samples and can improve the learning accuracy and convergence efficiency of the intelligent analysis model.

[0054] S202 involves obtaining prosodic features of the colloquialized text and obtaining prosodic pause intervals of the colloquialized text based on the prosodic features.

[0055] Specifically, the prosodic features of the colloquialized text are extracted, and these prosodic features are used to determine the prosodic pause intervals of the colloquialized text.

[0056] In one embodiment, colloquialized text is input to a text analysis model, and the text analysis model provides feedback on prosodic pause intervals. The text analysis model then performs text normalization on the colloquialized text, segments the normalized colloquialized text into words, and determines prosodic features based on the segmented colloquialized text. The text analysis model is obtained through training using a first training text.

[0057] Specifically, the conversion from the text to be processed to the target audio is divided into two stages. The first stage is in which the text conversion model performs colloquialization on the text to be processed to obtain the colloquialized text. The second stage is in which the text analysis model analyzes the colloquialized text to obtain prosodic pause intervals, and then completes the audio conversion.

[0058] Furthermore, the text analysis model was obtained through training using the first training text. The text analysis model focuses on the ability to mine prosodic features and to determine prosodic pause intervals based on these prosodic features. After training, the text analysis model performs text normalization on the colloquialized text, segments the normalized colloquialized text into words, and determines prosodic features based on the segmented colloquialized text, thereby obtaining prosodic pause intervals of the colloquialized text based on the prosodic features.

[0059] What needs to be explained is that during text normalization, specified characters within the colloquialized text are converted into a suitable text format, and during word segmentation, the adjusted text is divided into basic units, of which the specified characters include at least numbers, abbreviations, and currency symbols, and the basic units include at least words and punctuation marks, the prosodic features of the colloquialized text are determined based on the segmented text, and the colloquialized text is segmented based on the prosodic features to obtain the prosodic pause intervals of the colloquialized text, thereby improving the accuracy of the prosodic pause intervals.

[0060] S203 is the process of converting colloquialized text into target audio based on colloquialization pause intervals and prosodic pause intervals.

[0061] Specifically, based on the colloquialization pause interval and prosodic pause interval, audio synthesis is performed on the colloquialized text containing newly added colloquialized content to convert the colloquialized text into target audio.

[0062] In one embodiment, the text pause interval of the colloquialized text is determined based on the colloquialized pause interval and the prosodic pause interval, a phoneme sequence corresponding to the colloquialized text is obtained based on the colloquialized text and the corresponding text pause interval, and a target audio corresponding to the colloquialized text is obtained based on the phoneme sequence.

[0063] Specifically, by fusing colloquial pause intervals and prosodic pause intervals to obtain text pause intervals corresponding to the colloquialized text, the text pause intervals become obtained by combining colloquialization features and prosodic features, improving the accuracy of pauses in the colloquialized text. The colloquialized text is then divided using the text pause intervals, the resulting words are obtained, the words are converted into phoneme sequences (i.e., pronunciation is represented by phonemes), and further audio conversion is performed based on the phoneme sequences to obtain target audio corresponding to the colloquialized text, resulting in target audio with a high degree of personification.

[0064] In one implementation scenario, acoustic features are obtained from a phoneme sequence, including at least the fundamental frequency and spectral features. These acoustic features are then converted into audio signals of consecutive speech sampling points to obtain target audio corresponding to the colloquialized text.

[0065] In one specific implementation scenario, the mapping relationship between phoneme sequences and acoustic features is obtained by a feature extraction model, which may include, but is not specifically limited to, at least one of Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM), and Generative Pre-trained Transformers. The conversion from acoustic features to target audio is completed by a vocoder, but is not specifically limited to the algorithm employed by the vocoder.

[0066] Selectively, the conversion between phoneme sequences and target audio may be obtained using an end-to-end audio conversion model, which is obtained by training with phoneme sequence samples and their corresponding converted audio.

[0067] In one implementation scene, multiple types of prosodic pause intervals are assigned corresponding interval length ranks, and multiple types of colloquial pause intervals are compatible with at least some of the interval length ranks. Determining the text pause intervals of colloquial text based on colloquial pause intervals and prosodic pause intervals includes obtaining the text pause intervals of colloquial text based on the interval length ranks corresponding to the colloquial pause intervals and the positions of the colloquial pause intervals within the colloquial text, and the interval length ranks corresponding to the prosodic pause intervals and the positions of the prosodic pause intervals within the colloquial text.

[0068] Specifically, multiple prosodic pause intervals are assigned corresponding interval length ranks, and multiple types of colloquial pause intervals can be adapted to at least some of the interval length ranks. By marking the position of the colloquial pause interval within the colloquialized text and the interval length rank adapted to the colloquial pause interval, and marking the position of the prosodic pause interval within the colloquialized text and the interval length rank adapted to the prosodic pause interval, the text pause interval of the colloquialized text is obtained. By fusing the colloquial pause interval and the prosodic pause interval with each other to obtain a text pause interval with unified rank criteria, higher accuracy is achieved when converting to target audio.

[0069] In one specific application scenario, multiple types of prosodic pause intervals include at least five levels of interval length ranks, of which, in order from shortest to longest, include at least word level L1, phrase level L2, prosodic pause L3, punctuation L4, and paragraph pause L5; and colloquial pause intervals include short pauses, long pauses, and extra-long pauses, of which short pauses correspond to L2, long pauses to L3, and extra-long pauses to L4. In other specific application scenarios, the number of interval length ranks and the specific interval length ranks that fit the colloquial pause intervals can both be customized and set, and this application does not specifically limit this.

[0070] It should be explained that the embodiments or implementation scenarios described in any of the above embodiments are not limited to a single embodiment, and the different embodiments can be combined with each other, but this application does not specifically limit this.

[0071] Unlike the above embodiment, the conversion from the text to be processed to the target audio is divided into two stages. The first stage is in which the text conversion model performs colloquialization conversion on the text to be processed to obtain the colloquialized text. The second stage is in which the text analysis model analyzes the colloquialized text to obtain prosodic pause intervals, and then completes the audio conversion. Of these, the construction and training of the text conversion model includes multiple embodiments. After training, the text conversion model can insert colloquialized expression content into the text to be processed, set appropriate tags at the corresponding positions, fuse the colloquialization pause intervals and prosodic pause intervals to obtain a phoneme sequence, that is, represent pronunciation with phonemes, and then perform audio conversion based on the phoneme sequence to obtain the target audio corresponding to the colloquialized text. As a result, the target audio has a high degree of anthropomorphism.

[0072] Referring to Figure 3, Figure 3 is a schematic diagram of the structure of one embodiment of the audio synthesis system according to the present application, and the audio synthesis system 30 includes a conversion module 301 for obtaining colloquialized text by inserting colloquialized expression content, which includes at least newly added colloquialized content and colloquialized pause intervals, into the text to be processed in response to the text to be processed; an analysis module 302 for obtaining prosodic features of the colloquialized text and obtaining prosodic pause intervals of the colloquialized text based on the prosodic features; and a generation module 303 for converting the colloquialized text into target audio based on the colloquialized pause intervals and prosodic pause intervals.

[0073] According to the above embodiment, once the text to be processed is obtained, the conversion module 301 inserts colloquialized expression content that includes at least newly added colloquialized content and colloquialized pause intervals into the text to be processed, thereby converting the text to be processed into colloquialized text. As a result, the colloquialized text can contain more content that will be added during the colloquialized expression and more pauses that will be adopted during the colloquialized expression. The analysis module 302 obtains the prosodic features of the converted colloquialized text, enabling the newly added colloquialized content to influence the prosodic features. Furthermore, based on the prosodic features, it obtains prosodic pause intervals corresponding to the colloquialized text, making the prosodic pause intervals more colloquial. The generation module 303 performs audio synthesis on the colloquialized text containing the newly added colloquialized content based on the colloquialized pause intervals and prosodic pause intervals, converting the colloquialized text into target audio. This enables the target audio to include the newly added colloquialized content during playback, and also makes the pause intervals more colloquial, thereby improving the degree of personification of the audio synthesis.

[0074] Selectively, the conversion module 301 is also used to input the text to be processed into a text conversion model and obtain colloquialized text fed back by the text conversion model. The text conversion model is obtained through training using parallel text pairs, and the parallel text pairs include a first training text and a second training text, with the second training text having colloquialized expressions inserted compared to the first training text.

[0075] Selectively, the text conversion model includes an encoder and a decoder, and the first predicted text is obtained after the first training text has each obtained an encoder and a decoder. Alternatively, the text conversion model includes a hint building unit and an intelligent analysis model, the hint building unit is for building hint text based on at least the first training text and inputting it into the intelligent analysis model, the hint text is for prompting the intelligent analysis model to perform colloquialization and output a second predicted text, and if the hint text further includes some parallel text pairs, the parallel text pairs are used as example samples, and the colloquialized expressions inserted into the second training text are used as training tags for the text conversion model.

[0076] Selectively, new colloquialization content includes at least one of a modal word, filler word, repetition word, and onomatopoeic audio; colloquialization pause intervals include pause intervals of multiple durations; each type of new colloquialization content is preceded by a corresponding new addition start tag; each type of new colloquialization content is followed by a corresponding new addition end tag; and each type of new colloquialization pause interval is positioned with a corresponding pause tag.

[0077] Selectively, analysis module 302 is also used to input colloquialized text into a text analysis model and obtain prosodic pause intervals fed back by the text analysis model. The text analysis model performs text normalization on the colloquialized text, segments the normalized colloquialized text into words, and determines prosodic features based on the segmented colloquialized text, thereby obtaining prosodic pause intervals of the colloquialized text based on the prosodic features. The text analysis model was obtained through training using the first training text.

[0078] Selectively, the generation module 303 also determines the text pause interval of the colloquialized text based on the colloquialized pause interval and the prosodic pause interval, obtains a phoneme sequence corresponding to the colloquialized text based on the colloquialized text and its corresponding text pause interval, and obtains a target audio corresponding to the colloquialized text based on the phoneme sequence.

[0079] Selectively, multiple types of prosodic pause intervals are assigned corresponding interval length ranks, and multiple types of colloquialized pause intervals are suited to at least some of the interval length ranks. The generation module 303 is also for obtaining text pause intervals in colloquialized text based on the interval length ranks corresponding to colloquialized pause intervals and the positions of the colloquialized pause intervals within the colloquialized text, and the interval length ranks corresponding to prosodic pause intervals and the positions of the prosodic pause intervals within the colloquialized text.

[0080] Referring to Figure 4, which is a schematic diagram of the structure of one embodiment of the electronic device according to the present application, the electronic device 40 includes a memory 401 and a processor 402 connected to each other, with program instructions (not shown) stored in the memory 401, and the processor 402 is for executing the program instructions so that the audio synthesis method in any of the above embodiments is realized. For explanations of related matters, please refer to the detailed description in the embodiment of the above method, and will not be repeated here.

[0081] According to the above embodiment, the degree of anthropomorphism in audio synthesis can be improved.

[0082] Referring to Figure 5, which is a schematic diagram of the structure of one embodiment of a computer-readable storage medium according to the present application, the computer-readable storage medium 50 stores program instructions 500, and when the program instructions 500 are executed by the processor, the audio synthesis method described in any of the above embodiments is realized. For explanations of related matters, please refer to the detailed description in the embodiment of the above method, and will not be repeated here.

[0083] According to the above embodiment, the degree of anthropomorphism in audio synthesis can be improved.

[0084] Units described as separate components may or may not be physically separated, and components presented as units may or may not be physical units; that is, they may be located in the same place or distributed across multiple network units. To achieve the objectives of the embodiments described herein, some or all of the units may be selected as needed.

[0085] Furthermore, each functional unit in each embodiment of the present application may be integrated into a single processing unit, or each unit may be physically provided individually. Moreover, two or more units may be integrated into a single unit. The integrated unit may be implemented in hardware form, or in a form in which software functional units are added to the hardware.

[0086] The integrated unit may be implemented in the form of a software functional unit and, if sold or used as an independent product, may be stored on a computer-readable storage medium. Based on this understanding, essential parts of the technical aspects of the present application, parts that contribute to the prior art, or all or part of such technical aspects can be embodied in the form of a software product. Such computer software product is stored on a single storage medium and includes several instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps in the methods according to each embodiment of the present application. The aforementioned storage mediums include various media capable of storing program code, such as USB flash drives, portable hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0087] The foregoing describes only embodiments of the present application and does not limit the scope of the claims of the present application. Any equivalent structures or flow transformations, or direct or indirect applications to other related technical fields, made using the contents of the specification and drawings of the present application are also included within the scope of the patent protection of the present application.< / yqe> < / yqb> < / std> < / fie> < / fib> < / fie> < / fib> < / spa> < / fie> < / fib> < / fie> < / fib>

Claims

1. An audio synthesis method, In response to obtaining the text to be processed, colloquialized expression content that includes at least newly added colloquialized content and colloquialized pause intervals is inserted into the text to be processed to obtain colloquialized text. The prosodic features of the colloquialized text are obtained, and the prosodic pause intervals of the colloquialized text are obtained based on the prosodic features, An audio synthesis method comprising converting the colloquialized text into target audio based on the colloquialized pause interval and the prosodic pause interval.

2. Inserting the above-mentioned colloquialized expressions into the text to be processed to obtain the colloquialized text is, The method according to claim 1, comprising inputting the text to be processed into a text conversion model to obtain a colloquialized text fed back by the text conversion model, wherein the text conversion model is obtained by training using parallel text pairs, and the parallel text pairs include a first training text and a second training text, and the second training text has the colloquialized expression content inserted into it compared to the first training text.

3. The text conversion model includes an encoder and a decoder, and the first training text passes through the encoder and the decoder respectively to obtain a first predicted text. The method according to claim 2, wherein the colloquialized expression content inserted into the second training text is used as a training tag for the text conversion model.

4. The text conversion model includes a hint building unit and an intelligent analysis model, wherein the hint building unit constructs hint text based at least on the first training text and inputs it into the intelligent analysis model, and the hint text prompts the intelligent analysis model to perform colloquialization and output a second predicted text. The method according to claim 2, wherein the colloquialized expression content inserted into the second training text is used as a training tag for the text conversion model.

5. The method according to claim 4, wherein the hint text includes at least the first training text and the colloquialization requirements, and if the hint text further includes some of the parallel text pairs, the parallel text pairs are exemplary samples that conform to the colloquialization requirements.

6. Obtaining the prosodic features of the aforementioned colloquialized text and obtaining the prosodic pause interval of the colloquialized text based on the prosodic features is: The method according to claim 2, comprising inputting the colloquialized text into a text analysis model to obtain the prosodic pause intervals fed back by the text analysis model, wherein the text analysis model performs text normalization on the colloquialized text, divides the normalized colloquialized text into words, determines the prosodic features based on the colloquialized text after word division, and obtains the prosodic pause intervals of the colloquialized text based on the prosodic features, and the text analysis model is obtained through training using the first training text.

7. The method according to claim 1, wherein the newly added colloquialized content includes at least one of a modal word, a filler word, a repeated word, and an onomatopoeic audio, and the colloquialized pause interval includes pause intervals of multiple different durations.

8. The method according to claim 7, wherein a suitable new addition start tag is attached before each type of the new colloquialized content, a new addition end tag is attached after each type of the new colloquialized content, and a suitable pause tag is attached at the position of each type of the colloquialized pause interval.

9. Based on the aforementioned colloquialization pause interval and prosodic pause interval, converting the colloquialized text into target audio is: Based on the aforementioned colloquialization pause interval and the aforementioned prosodic pause interval, the text pause interval of the colloquialized text is determined, Based on the colloquialized text and the corresponding text pause interval, a phoneme sequence corresponding to the colloquialized text is obtained. The method according to claim 1, comprising obtaining the target audio corresponding to the colloquialized text based on the phoneme sequence.

10. The method according to claim 9, wherein multiple types of prosodic pause intervals are assigned corresponding interval time length ranks, and multiple types of colloquial pause intervals conform to at least some of the interval time length ranks.

11. Based on the aforementioned colloquialization pause interval and prosodic pause interval, determining the text pause interval of the colloquialized text is: The method according to claim 10, comprising obtaining a text pause interval of the colloquialized text based on the interval time length rank corresponding to the colloquialized pause interval and the position of the colloquialized pause interval within the colloquialized text, and the interval time length rank corresponding to the prosodic pause interval and the position of the prosodic pause interval within the colloquialized text.

12. The method according to claim 10, wherein the multiple types of prosodic pause intervals include at least five levels of interval time length ranks, the interval time length ranks including at least word level, phrase level, prosodic pause, punctuation and paragraph pause in order from shortest to longest, and the colloquial pause intervals include short pauses, long pauses and extra-long pauses, of which the short pauses correspond to the phrase level, the long pauses correspond to the prosodic pauses and the extra-long pauses correspond to the punctuation.

13. An audio synthesis system, A conversion module for obtaining colloquialized text by inserting colloquialized expression content, which includes at least newly added colloquialized content and colloquialized pause intervals, into the text to be processed, in accordance with the obtained text to be processed. An analysis module for obtaining prosodic features of the colloquialized text and obtaining prosodic pause intervals of the colloquialized text based on the prosodic features, An audio synthesis system characterized by including a generation module for converting the colloquialized text into target audio based on the colloquialized pause interval and the prosodic pause interval.

14. An electronic device comprising a memory and a processor connected to each other, wherein program instructions are stored in the memory, and the processor is for executing the program instructions so as to realize the audio synthesis method described in any one of claims 1 to 12.

15. A computer-readable storage medium storing program instructions, wherein when the program instructions are executed by a processor, the audio synthesis method described in any one of claims 1 to 12 is realized.

Images