Speech synthesis method and device based on artificial intelligence, and related equipment

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a speech synthesis method and device based on artificial intelligence, electronic equipment and a storage medium. The method comprises the steps of obtaining a to-be-synthesized text and voice prompt data; inputting the to-be-synthesized text to a preset phoneme encoder, and analyzing the to-be-synthesized text through the phoneme encoder to obtain first phoneme data; inputting the voice prompt data to a preset rhythm extractor, and analyzing the voice prompt data through the rhythm extractor to obtain first rhythm data; inputting the voice prompt data to a preset first voice encoder, and analyzing the voice prompt data through the first voice encoder to obtain first voice feature prompt data; embedding the first phoneme data into the first rhythm data to obtain second rhythm data; and processing the second rhythm data and the first voice feature prompt data through a preset second voice decoder to obtain a target synthetic voice. According to the method, the stability of speech synthesis based on the large language model is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a speech synthesis method, device, electronic device and storage medium based on artificial intelligence. Background Art

[0002] Large language models (LLMs) have achieved significant technical progress in natural language generation. As the model size increases, LLMs show strong contextual learning capabilities and are able to handle unseen tasks in a zero-shot or few-shot manner through text instructions (commonly called prompts). The simple yet effective next-token prediction task of LLMs makes it easy to apply to other fields such as vision and speech synthesis. By using a neural codec to convert continuous waveforms into discrete tokens, text-to-speech (TTS) models can be modeled using a decoder-only language model. However, due to the sequence generation nature of language models, codec language models have poor robustness. Although the autoregressive (AR) prediction style of language models enables the model to generate speech with diverse prosodic patterns, it may also lead to failure cases with unnatural prosody. In addition, since there is no strict alignment between text and speech, the model may also miss or repeat words in the input text. Summary of the invention

[0003] The main technical problem solved by the implementation methods of the present application is that the speech synthesis results based on large language models in traditional technologies have poor stability.

[0004] To solve the above technical problems, the first technical solution adopted in the implementation mode of the present application is: to provide a speech synthesis method based on artificial intelligence, including: obtaining a text to be synthesized, and speech prompt data corresponding to the text to be synthesized; inputting the text to be synthesized into a preset phoneme encoder, parsing the text to be synthesized through the phoneme encoder, and obtaining first phoneme data; inputting the speech prompt data into a preset prosody extractor, parsing the speech prompt data through the prosody extractor, and obtaining first prosody data; inputting the speech prompt data into a preset first speech encoder, parsing the speech prompt data through the first speech encoder, and obtaining first speech feature prompt data; embedding the first phoneme data into the first prosody data, and obtaining second prosody data; processing the second prosody data and the first speech feature prompt data through a preset second speech decoder to obtain the target synthesized speech.

[0005] Optionally, the step of parsing the text to be synthesized through the phoneme encoder to obtain the first phoneme data includes: performing text preprocessing and text segmentation processing on the text to be synthesized to obtain a first preprocessed text; using a phoneme dictionary to map syllables in the first preprocessed text to corresponding phonemes to obtain a first phoneme sequence; inputting the first phoneme sequence into the phoneme encoder, and mapping each phoneme to a corresponding phoneme feature vector through the phoneme encoder to obtain the first phoneme data.

[0006] Optionally, the step of inputting the voice prompt data into a preset prosody extractor, parsing the voice prompt data through the prosody extractor, and obtaining first prosody data includes: preprocessing the voice prompt data to obtain first preprocessed voice data; calculating the basic frequency information of the first preprocessed voice data through a preset fundamental frequency estimation algorithm to obtain first pitch data; determining the duration of each syllable in the first preprocessed voice data using an endpoint detection algorithm to obtain first syllable duration data; and processing the first pitch data and the first syllable duration data through the preset prosody extractor to obtain the first prosody data.

[0007] Optionally, the step of parsing the voice prompt data through the first voice encoder to obtain first voice feature prompt data includes: performing denoising and normalization processing on the first voice prompt data to obtain second preprocessed voice data; processing the second preprocessed voice data through a preset framing tool to obtain second voice framing data; performing feature calculation processing on the second voice framing data according to a preset voice feature type to obtain the first voice feature prompt data containing the features of each frame of voice data.

[0008] Optionally, the step of embedding the first phoneme data into the first prosodic data to obtain the second prosodic data includes: removing redundant data and noise data of the first phoneme data and the first prosodic data, and converting the processed first phoneme data and the first prosodic data into a preset data structure; aligning the first phoneme data and the first prosodic data in the time dimension using an alignment tool; and inputting the aligned first phoneme data and the first prosodic data into a preset autoregressive Transformer model to obtain the second prosodic data.

[0009] Optionally, after the step of using an alignment tool to align the first phoneme data and the first prosodic data in the time dimension, the step also includes: parsing the aligned first phoneme data to obtain second syllable duration data for each syllable; comparing whether the second syllable duration data is greater than a preset maximum syllable duration, and if so, updating the current second syllable duration data being traversed to the preset maximum syllable duration; and using the traversed second syllable duration data to update the first phoneme data.

[0010] Optionally, the step of obtaining a target synthesized speech by processing the second prosody data and the first speech feature prompt data through a preset second speech decoder includes: converting the second prosody data and the first speech feature prompt data into corresponding prosody feature vectors and speech prompt feature vectors; inputting the prosody feature vectors and the speech prompt feature vectors into the second speech decoder, wherein the second speech decoder is constructed based on a non-autoregressive Transformer model; processing the prosody feature vectors and the speech prompt feature vectors through a multi-head attention mechanism of the non-autoregressive Transformer model, and adding corresponding position codes according to the speech prompt feature vectors to obtain initial synthesized speech data; and denoising and gain adjusting the initial synthesized speech data to obtain the target synthesized speech.

[0011] To solve the above technical problems, the second technical solution adopted in the implementation mode of the present application is: to provide a speech synthesis device based on artificial intelligence, including: an initial data acquisition module, used to acquire the text to be synthesized, and the speech prompt data corresponding to the text to be synthesized; a first phoneme data module, used to input the text to be synthesized into a preset phoneme encoder, and parse the text to be synthesized through the phoneme encoder to obtain the first phoneme data; a first prosody data module, used to input the voice prompt data into a preset prosody extractor, and parse the voice prompt data through the prosody extractor to obtain the first prosody data; a speech feature prompt module, used to input the voice prompt data into a preset first speech encoder, and parse the voice prompt data through the first speech encoder to obtain the first speech feature prompt data; a second prosody data module, used to embed the first phoneme data into the first prosody data to obtain the second prosody data; a synthesized speech generation module, used to obtain the target synthesized speech by processing the second prosody data and the first speech feature prompt data through a preset second speech decoder.

[0012] To solve the above technical problems, the third technical solution adopted in the implementation mode of the present application is: to provide an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the artificial intelligence-based speech synthesis method as described above.

[0013] In order to solve the above technical problems, the fourth technical solution adopted in the implementation mode of the present application is: providing a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by an electronic device, the electronic device executes the artificial intelligence-based speech synthesis method as described above.

[0014] Different from the related art, the present application obtains the text to be synthesized and voice prompt data; inputs the text to be synthesized into a preset phoneme encoder, and parses the text to be synthesized through the phoneme encoder to obtain first phoneme data; inputs the voice prompt data into a preset rhythm extractor, and parses the voice prompt data through the rhythm extractor to obtain first rhythm data; inputs the voice prompt data into a preset first voice encoder, and parses the voice prompt data through the first voice encoder to obtain first voice feature prompt data; embeds the first phoneme data into the first rhythm data to obtain second rhythm data; and obtains the target synthesized speech by processing the second rhythm data and the first voice feature prompt data through a preset second voice decoder, which not only reduces the word error rate of the synthesized speech, but also further improves the stability of the speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] One or more embodiments are exemplarily described by corresponding drawings, which do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and the figures in the drawings do not constitute proportional limitations unless otherwise stated.

[0016] Figure 1 It is a schematic diagram of the operating environment of the artificial intelligence-based speech synthesis method provided in an embodiment of the present application.

[0017] Figure 2 It is a schematic diagram of the execution flow of the artificial intelligence-based speech synthesis method provided in an embodiment of the present application.

[0018] Figure 3 It is a schematic diagram of the execution flow of generating first phoneme data in the artificial intelligence-based speech synthesis method provided in an embodiment of the present application.

[0019] Figure 4It is a schematic diagram of the execution flow of generating first prosody data in the artificial intelligence-based speech synthesis method provided in an embodiment of the present application.

[0020] Figure 5 It is a schematic diagram of the execution flow of generating first voice feature prompt data in the artificial intelligence-based voice synthesis method provided in an embodiment of the present application.

[0021] Figure 6 It is a schematic diagram of the system structure of the artificial intelligence-based speech synthesis device provided in an embodiment of the present application.

[0022] Figure 7 It is a schematic diagram of the hardware structure of an electronic device that executes an artificial intelligence-based speech synthesis method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0024] It should be noted that, if there is no conflict, the various features in the embodiments of the present application can be combined with each other, all within the scope of protection of the present application. In addition, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order than the module division in the device schematic diagram or the order in the flow chart.

[0025] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used in this specification includes any and all combinations of one or more of the related listed items.

[0026] The defects existing in the above-mentioned solutions are the results obtained by the applicant after practice and careful research. Therefore, the discovery process of the above-mentioned problems and the solutions proposed by this application for the above-mentioned problems below should be the contributions made by the applicant to this application during the disclosure process of this application.

[0027] To facilitate understanding of this embodiment, firstly, a speech synthesis method based on artificial intelligence disclosed in the embodiment of the present application is described in detail. Figure 1 , Figure 1 is a schematic diagram of the operating environment of the speech synthesis method based on artificial intelligence provided in the embodiment of the present application, such as Figure 1As shown, the execution subject of the speech synthesis method based on artificial intelligence provided in the embodiment of the present application is generally an electronic device with certain computing capabilities, such as a computer device. In some possible implementations, the speech synthesis method based on artificial intelligence can be implemented by a processor calling a computer-readable instruction stored in a memory. Among them, Figure 1 The computer device in the term can be a server. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. It can be understood that Figure 1 The number of computer devices in the figure is only for reference and can be expanded to any number according to actual needs.

[0028] Please continue reading Figure 2 , Figure 2 is a schematic diagram of the execution flow of the speech synthesis method based on artificial intelligence provided in the embodiment of the present application, such as Figure 2 As shown, the process includes the following steps S1 to S6.

[0029] S1. Obtain the text to be synthesized and the voice prompt data corresponding to the text to be synthesized.

[0030] The voice prompt data may be a short segment of speaker voice data, which is also used as sample data, that is, the timbre of the final speech synthesis data is almost identical to the timbre of the speaker voice data. The text to be synthesized is the text spoken by the speaker in the final speech synthesis data.

[0031] S2. Input the text to be synthesized into a preset phoneme encoder, and parse the text to be synthesized through the phoneme encoder to obtain first phoneme data.

[0032] As an optional implementation, please continue to refer to Figure 3 , Figure 3 is a schematic diagram of an execution flow of generating first phoneme data in a speech synthesis method based on artificial intelligence provided in an embodiment of the present application, such as Figure 3 As shown, the process includes the following steps S21 to S23.

[0033] S21, performing text preprocessing and text segmentation processing on the text to be synthesized to obtain a first preprocessed text.

[0034] The text preprocessing operations in step S21 may include, but are not limited to: removing special characters, unifying formats, expanding abbreviations, processing numbers, converting units, replacing synonyms, etc.

[0035] S22. Use a phoneme dictionary to map syllables in the first preprocessed text to corresponding phonemes to obtain a first phoneme sequence.

[0036] Among them, the phoneme dictionary is an important tool in the field of speech processing and synthesis. Its main function is to convert written text into corresponding phoneme representation. Phoneme is the smallest sound unit in a language that can distinguish meanings. For example, in English, / p / and / b / are two different phonemes because they can change the meaning of a word (such as "pat" and "bat").

[0037] S23, inputting the first phoneme sequence into a phoneme encoder, mapping each phoneme into a corresponding phoneme feature vector through the phoneme encoder, and obtaining first phoneme data.

[0038] S3. Input the voice prompt data into a preset prosody extractor, and parse the voice prompt data through the prosody extractor to obtain first prosody data.

[0039] The phoneme encoder converts each phoneme into a rich feature vector, which improves the representation ability of phonemes and enables them to better capture the subtle differences between phonemes. By mapping phonemes to feature vectors, the model can more effectively learn the relationship between phonemes and their corresponding pronunciations, thereby reducing overfitting on specific data sets.

[0040] As an optional implementation, please continue to refer to Figure 4 , Figure 4 is a schematic diagram of the execution flow of generating first prosody data in the artificial intelligence-based speech synthesis method provided in an embodiment of the present application, such as Figure 4 As shown, the process includes the following steps S31 to S34.

[0041] S31. Pre-process the voice prompt data to obtain first pre-processed voice data.

[0042] S32. Calculate the basic frequency information of the first preprocessed speech data by using a preset fundamental frequency estimation algorithm to obtain first pitch data.

[0043] Among them, pitch data refers to the basic frequency information contained in the speech signal, which is usually used to represent the frequency characteristics of sound waves and reflects the high and low pitch of the sound. Pitch is one of the most important features in music and language, and is usually directly related to the frequency of the sound wave (in Hertz). High-frequency sound waves are usually perceived as high pitches, while low-frequency sound waves are perceived as low pitches.

[0044] As an optional implementation, the steps for calculating pitch data can be specifically as follows: 1. Preset the fundamental frequency estimation algorithm and select an appropriate fundamental frequency estimation algorithm, such as the Yin algorithm, the Praat tool or the autocorrelation method. 2. Input the preprocessed voice data and input the first preprocessed voice data into the fundamental frequency estimation algorithm. This data should undergo preprocessing steps such as noise elimination and volume normalization to ensure signal quality. 3. Calculate the basic frequency and use the selected algorithm to analyze the audio signal to calculate the basic frequency of each time period to generate pitch data. 4. Output the first pitch data and organize the calculated frequency information into pitch data, usually in the form of a time series, representing the pitch changes at each time point.

[0045] S33. Use an endpoint detection algorithm to determine the duration of each syllable in the first preprocessed speech data to obtain first syllable duration data.

[0046] S34. Process the first pitch data and the first syllable duration data by a preset rhythm extractor to obtain first rhythm data.

[0047] Among them, the prosody extractor can identify prosodic features such as stress, pitch changes, and rhythm in speech, thereby generating more natural and fluent synthesized speech. The extracted prosodic data can be used as input features to optimize subsequent speech synthesis models or recognition models and improve overall performance. Prosodic data can be fused with other features (such as phoneme data, pitch data, etc.) to enhance the expressiveness of the model.

[0048] S4. Input the voice prompt data to a preset first voice encoder, and parse the voice prompt data through the first voice encoder to obtain first voice feature prompt data.

[0049] As an optional implementation, please continue to refer to Figure 5 , Figure 5 is a schematic diagram of an execution flow of generating first voice feature prompt data in the artificial intelligence-based voice synthesis method provided in an embodiment of the present application, such as Figure 5 As shown, the process includes the following steps S41 to S43.

[0050] S41, performing denoising and normalization processing on the first voice prompt data to obtain second pre-processed voice data.

[0051] S42: Process the second preprocessed speech data by using a preset framing tool to obtain second speech framing data.

[0052] S43, performing feature calculation processing on the second speech frame data according to a preset speech feature type to obtain first speech feature prompt data containing features of each frame of speech data.

[0053] Among them, by converting the speech signal into a feature vector, the encoder can capture complex acoustic features in a high-dimensional space and improve the expressiveness of the model. The features processed by the encoder are more suitable for speech recognition tasks and help improve the recognition accuracy of the model. The first speech feature prompt data can be fused with other features (such as rhythm features, pitch features, etc.) to enhance the multimodal processing capabilities of the model.

[0054] S5. Embed the first phoneme data into the first prosody data to obtain second prosody data.

[0055] As an optional implementation, the above step S5 may include the following specific steps S51 to S53.

[0056] S51, removing redundant data and noise data from the first phoneme data and the first prosody data, and converting the processed first phoneme data and the first prosody data into a preset data structure.

[0057] S52: Use an alignment tool to align the first phoneme data and the first prosody data in a time dimension.

[0058] S53, inputting the aligned first phoneme data and the first prosody data into a preset autoregressive Transformer model to obtain second prosody data.

[0059] Among them, the autoregressive Transformer model is a sequence generation model based on the Transformer architecture. Compared with the traditional Transformer model, the autoregressive Transformer adopts a specific strategy when generating sequences, allowing the model to rely on all previously generated outputs when generating each output. When generating a sequence, the autoregressive model uses the previous element of the sequence as input to predict the next element and gradually generates the entire sequence. For example, in text generation, the model generates the next word based on the words that have been generated.

[0060] S6. Process the second prosody data and the first speech feature prompt data by a preset second speech decoder to obtain a target synthesized speech.

[0061] As an optional implementation, the above step S5 may include the following specific steps S61 to S64.

[0062] S61: Convert the second prosody data and the first voice feature prompt data into corresponding prosody feature vectors and voice prompt feature vectors.

[0063] S62. Input the prosody feature vector and the speech prompt feature vector to the second speech decoder, wherein the second speech decoder is constructed based on a non-autoregressive Transformer model.

[0064] Among them, the non-autoregressive Transformer model is an improved Transformer architecture designed to improve the efficiency of sequence generation tasks, especially in fields such as natural language processing and speech synthesis. Unlike the autoregressive model, the non-autoregressive model does not rely on previously generated outputs during the generation process, but is able to generate the entire sequence in parallel. Unlike the autoregressive model, which generates sequences step by step, the non-autoregressive model avoids the cumulative error and delay problems in sequence generation by generating the entire output sequence at once.

[0065] S63. Process the prosodic feature vector and the speech prompt feature vector through the multi-head attention mechanism of the non-autoregressive Transformer model, and add corresponding position codes according to the speech prompt feature vector to obtain initial synthesized speech data.

[0066] S64: Perform denoising and gain adjustment on the initial synthesized speech data to obtain the target synthesized speech.

[0067] Among them, since the entire sequence can be generated in parallel, the above non-autoregressive model has significantly improved the speed of inference, especially when generating long sequences. At the same time, unlike the autoregressive model, the non-autoregressive model does not rely on the previous generation results, thereby reducing the error propagation in the generation process.

[0068] The artificial intelligence-based speech synthesis method provided in the embodiment of the present application obtains a text to be synthesized and speech prompt data; inputs the text to be synthesized into a preset phoneme encoder, and parses the text to be synthesized through the phoneme encoder to obtain first phoneme data; inputs the speech prompt data into a preset prosody extractor, and parses the speech prompt data through the prosody extractor to obtain first prosody data; inputs the speech prompt data into a preset first speech encoder, and parses the speech prompt data through the first speech encoder to obtain first speech feature prompt data; embeds the first phoneme data into the first prosody data to obtain second prosody data; and obtains the target synthesized speech by processing the second prosody data and the first speech feature prompt data through a preset second speech decoder, which not only reduces the word error rate of the synthesized speech, but also further improves the stability of the speech synthesis.

[0069] Please continue reading Figure 6 , Figure 6 is a schematic diagram of the system structure of the speech synthesis device based on artificial intelligence provided in the embodiment of the present application, such as Figure 6 As shown, the device 60 includes: an initial data acquisition module 61, a first phoneme data module 62, a first prosody data module 63, a speech feature prompt module 64, a second prosody data module 65 and a synthetic speech generation module 66.

[0070] The initial data acquisition module 61 is used to acquire the text to be synthesized and the voice prompt data corresponding to the text to be synthesized;

[0071] The first phoneme data module 62 is used to input the text to be synthesized into a preset phoneme encoder, and parse the text to be synthesized through the phoneme encoder to obtain first phoneme data;

[0072] The first prosody data module 63 is used to input the voice prompt data into a preset prosody extractor, and the voice prompt data is parsed by the prosody extractor to obtain first prosody data;

[0073] The voice feature prompt module 64 is used to input the voice prompt data into a preset first voice encoder, and parse the voice prompt data through the first voice encoder to obtain first voice feature prompt data;

[0074] The second prosody data module 65 is used to embed the first phoneme data into the first prosody data to obtain second prosody data;

[0075] The synthesized speech generation module 66 is used to obtain the target synthesized speech by processing the second prosody data and the first speech feature prompt data through a preset second speech decoder.

[0076] As an optional implementation, the first phoneme data module 62 is specifically used to perform text preprocessing and text segmentation processing on the text to be synthesized to obtain a first preprocessed text; use a phoneme dictionary to map syllables in the first preprocessed text to corresponding phonemes to obtain a first phoneme sequence; input the first phoneme sequence into the phoneme encoder, and use the phoneme encoder to map each phoneme to a corresponding phoneme feature vector to obtain the first phoneme data.

[0077] As an optional implementation, the first prosody data module 63 is specifically used to preprocess the voice prompt data to obtain first preprocessed voice data; calculate the basic frequency information of the first preprocessed voice data through a preset fundamental frequency estimation algorithm to obtain first pitch data; use an endpoint detection algorithm to determine the duration of each syllable in the first preprocessed voice data to obtain first syllable duration data; process the first pitch data and the first syllable duration data through the preset prosody extractor to obtain the first prosody data.

[0078] As an optional implementation, the speech feature prompt module 64 is specifically used to perform denoising and normalization processing on the first speech prompt data to obtain second preprocessed speech data; process the second preprocessed speech data through a preset framing tool to obtain second speech framing data; perform feature calculation processing on the second speech framing data according to a preset speech feature type to obtain the first speech feature prompt data containing the features of each frame of speech data.

[0079] As an optional implementation, the second prosodic data module 65 is specifically used to remove redundant data and noise data from the first phoneme data and the first prosodic data, and convert the first phoneme data and the first prosodic data into a preset data structure; use an alignment tool to align the first phoneme data and the first prosodic data in the time dimension; input the aligned first phoneme data and the first prosodic data into a preset autoregressive Transformer model to obtain the second prosodic data.

[0080] As an optional implementation, the second rhythm data module 65 is also used to parse the aligned first phoneme data to obtain the second syllable duration data of each syllable; compare whether the second syllable duration data is greater than the preset maximum syllable duration data, and if greater, update the current second syllable duration data being traversed to the preset maximum syllable duration data; and use the traversed second syllable duration data to update the first phoneme data.

[0081] As an optional implementation, the synthetic speech generation module 66 is specifically used to convert the second prosody data and the first speech feature prompt data into corresponding prosody feature vectors and speech prompt feature vectors; input the prosody feature vectors and the speech prompt feature vectors into the second speech decoder, wherein the second speech decoder is constructed based on a non-autoregressive Transformer model; process the prosody feature vectors and the speech prompt feature vectors through the multi-head attention mechanism of the non-autoregressive Transformer model, and add corresponding position codes according to the speech prompt feature vectors to obtain initial synthetic speech data; denoise and gain adjust the initial synthetic speech data to obtain the target synthetic speech.

[0082] It should be noted that the above-mentioned speech synthesis device based on artificial intelligence can execute the speech synthesis method based on artificial intelligence provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in the embodiment of the speech synthesis device based on artificial intelligence, please refer to the speech synthesis method based on artificial intelligence provided in the embodiment of the present application.

[0083] Please continue reading Figure 7 , Figure 7 is a schematic diagram of the hardware structure of an electronic device 700 for executing a speech synthesis method based on artificial intelligence provided in an embodiment of the present application, such as Figure 7 As shown, the electronic device 700 includes:

[0084] One or more processors 710 and memory 720, Figure 7 A processor 710 is taken as an example.

[0085] The processor 710 and the memory 720 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0086] The memory 720, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the artificial intelligence-based speech synthesis method in the embodiment of the present application. The processor 710 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 720, that is, the artificial intelligence-based speech synthesis method in the above method embodiment is implemented.

[0087] The memory 720 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the speech synthesis device based on artificial intelligence, etc. In addition, the memory 720 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 720 may optionally include a memory remotely arranged relative to the processor 710, and these remote memories may be connected to the speech synthesis device based on artificial intelligence via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0088] The one or more modules are stored in the memory 720, and when executed by the one or more processors 710, perform the XXXXX method in any of the above method embodiments, for example, perform the above described Figure 2 Steps S1 to S6 of the method, Figure 3 Steps S21 to S23 of the method, Figure 4 Steps S31 to S34 of the method, Figure 5 Steps S41 to S43 of the method are implemented Figure 6 The functions of modules 61-66 in.

[0089] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0090] The embodiment of the present application provides a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, such as Figure 7 A processor 710 in the embodiment may enable the one or more processors to execute the artificial intelligence-based speech synthesis method in any of the above method embodiments, for example, to execute the above described Figure 2 Steps S1 to S6 of the method, Figure 3 Steps S21 to S23 of the method, Figure 4 Steps S31 to S34 of the method, Figure 5 Steps S41 to S43 of the method are implemented Figure 6 The functions of modules 61-66 in.

[0091] The present application provides a computer program product, wherein the computer program product includes a computer program stored on a non-volatile computer-readable storage medium, wherein the computer program includes program instructions, and when the program instructions are executed by the electronic device, the electronic device is enabled to execute the artificial intelligence-based speech synthesis method in any of the above method embodiments, for example, to execute the above described Figure 2 Steps S1 to S6 of the method, Figure 3 Steps S21 to S23 of the method, Figure 4 Steps S31 to S34 of the method, Figure 5 Steps S41 to S43 of the method are implemented Figure 6 The functions of modules 61-66 in.

[0092] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0093] Through the description of the above implementation methods, ordinary technicians in this field can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Ordinary technicians in this field can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; under the idea of ​​the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present application as described above, which are not provided in detail for the sake of simplicity; although the present application is described in detail with reference to the above embodiments, ordinary technicians in this field should understand that they can still modify the technical solutions recorded in the above embodiments, or replace some of the technical features by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application. The non-company software tools or components that appear in the embodiments of the present application are only examples and do not represent actual use.

Claims

1. A speech synthesis method based on artificial intelligence, characterized in that: include: Acquire the text to be synthesized and the voice prompt data corresponding to the text to be synthesized; Inputting the text to be synthesized into a preset phoneme encoder, parsing the text to be synthesized by the phoneme encoder to obtain first phoneme data; Inputting the voice prompt data into a preset prosody extractor, parsing the voice prompt data through the prosody extractor to obtain first prosody data; Inputting the voice prompt data into a preset first voice encoder, parsing the voice prompt data through the first voice encoder, and obtaining first voice feature prompt data; embedding the first phoneme data into the first prosodic data to obtain second prosodic data; The target synthesized speech is obtained by processing the second prosody data and the first speech feature prompt data through a preset second speech decoder.

2. The method for speech synthesis based on artificial intelligence according to claim 1, characterized in that: The step of parsing the text to be synthesized by the phoneme encoder to obtain the first phoneme data includes: Performing text preprocessing and text segmentation processing on the text to be synthesized to obtain a first preprocessed text; Using a phoneme dictionary, mapping syllables in the first preprocessed text to corresponding phonemes to obtain a first phoneme sequence; The first phoneme sequence is input into the phoneme encoder, and each phoneme is mapped into a corresponding phoneme feature vector by the phoneme encoder to obtain the first phoneme data.

3. The method for speech synthesis based on artificial intelligence according to claim 1, characterized in that: The step of inputting the voice prompt data into a preset prosody extractor, parsing the voice prompt data by the prosody extractor, and obtaining the first prosody data comprises: Preprocessing the voice prompt data to obtain first preprocessed voice data; Calculate the basic frequency information of the first preprocessed voice data by using a preset fundamental frequency estimation algorithm to obtain first pitch data; Determine the duration of each syllable in the first preprocessed speech data using an endpoint detection algorithm to obtain first syllable duration data; The first pitch data and the first syllable duration data are processed by the preset prosody extractor to obtain the first prosody data.

4. The method for speech synthesis based on artificial intelligence according to claim 1, characterized in that: The step of parsing the voice prompt data by the first voice encoder to obtain first voice feature prompt data includes: Performing denoising and normalization processing on the first voice prompt data to obtain second preprocessed voice data; Processing the second preprocessed voice data by a preset framing tool to obtain second voice framing data; The second speech frame data is subjected to feature calculation processing according to a preset speech feature type to obtain the first speech feature prompt data including the features of each frame of speech data.

5. The method for speech synthesis based on artificial intelligence according to claim 1, characterized in that: The step of embedding the first phoneme data into the first prosodic data to obtain second prosodic data comprises: removing redundant data and noise data from the first phoneme data and the first prosody data, and converting the first phoneme data and the first prosody data into a preset data structure; Using an alignment tool to align the first phoneme data and the first prosodic data in a time dimension; The aligned first phoneme data and the first prosodic data are input into a preset autoregressive Transformer model to obtain the second prosodic data.

6. The method for speech synthesis based on artificial intelligence according to claim 5, characterized in that: After the step of aligning the first phoneme data and the first prosody data in the time dimension using an alignment tool, the method further includes: Parsing the aligned first phoneme data to obtain second syllable duration data of each syllable; Compare whether the second syllable duration data is greater than a preset maximum syllable duration value, and if so, update the current second syllable duration data being traversed to the preset maximum syllable duration value; The first phoneme data is updated using the traversed second syllable duration data.

7. The method for speech synthesis based on artificial intelligence according to claim 1, characterized in that: The step of obtaining a target synthesized speech by processing the second prosody data and the first speech feature prompt data by a preset second speech decoder comprises: Converting the second prosody data and the first voice feature prompt data into corresponding prosody feature vectors and voice prompt feature vectors; Inputting the prosody feature vector and the speech prompt feature vector into the second speech decoder, wherein the second speech decoder is constructed based on a non-autoregressive Transformer model; Processing the prosody feature vector and the speech prompt feature vector through the multi-head attention mechanism of the non-autoregressive Transformer model, and adding corresponding position codes according to the speech prompt feature vector to obtain initial synthesized speech data; The initial synthesized speech data is subjected to denoising and gain adjustment to obtain the target synthesized speech.

8. A speech synthesis device based on artificial intelligence, characterized in that: include: An initial data acquisition module, used to acquire the text to be synthesized and the voice prompt data corresponding to the text to be synthesized; A first phoneme data module, used for inputting the text to be synthesized into a preset phoneme encoder, parsing the text to be synthesized by the phoneme encoder, and obtaining first phoneme data; A first prosody data module, used for inputting the voice prompt data into a preset prosody extractor, parsing the voice prompt data through the prosody extractor, and obtaining first prosody data; A voice feature prompt module, used for inputting the voice prompt data into a preset first voice encoder, parsing the voice prompt data through the first voice encoder, and obtaining first voice feature prompt data; A second prosodic data module, used for embedding the first phoneme data into the first prosodic data to obtain second prosodic data; The synthesized speech generation module is used to obtain the target synthesized speech by processing the second prosody data and the first speech feature prompt data through a preset second speech decoder.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the artificial intelligence-based speech synthesis method described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by an electronic device, the electronic device executes the artificial intelligence-based speech synthesis method described in any one of claims 1 to 7.

Images