Speech synthesis method and device based on artificial intelligence, equipment and storage medium

Abstract

The invention discloses a speech synthesis method and device based on artificial intelligence, equipment and a storage medium. The method comprises the following steps: acquiring a reference Mel spectrum and a target text; inputting the reference Mel spectrum into a variational auto-encoder to obtain a speaker potential representation and an accent potential representation output by the variational auto-encoder; inputting the target text into a pre-trained text encoder to obtain a phoneme embedding representation output by the pre-trained text encoder; inputting the potential speaker representation and the potential accent representation into a target vector quantizer to obtain a speaker embedded representation and an accent embedded representation output by the target vector quantizer; generating a target synthetic Mel spectrum according to the phoneme embedding representation, the speaker embedding representation and the accent embedding representation; and converting the target synthetic Mel spectrum into target speech. The method and the device can be applied to intelligent voice customer service in business scenes such as financial science and technology, medical health, old-age care and the like, so that the flexibility and the individuation degree of voice synthesis are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and speech synthesis technology, and in particular to an artificial intelligence-based speech synthesis method, apparatus, device and storage medium. Background Art

[0002] With the continuous innovation of artificial intelligence technology, speech synthesis technology, especially multi-speaker speech synthesis, has made significant progress in recent years and is widely used in the financial sector as well as in the healthcare and elderly care fields. For example, when a customer calls to inquire about a financial product, the financial service intelligent customer service can flexibly switch between multiple voices based on the customer's preferences. However, while existing speech synthesis systems can imitate accents and emotional expressions, they have difficulty effectively separating speaker and accent characteristics, limiting the flexibility and personalization of speech synthesis. Especially for the needs of minority users, existing speech synthesis systems often have difficulty generating appropriate speech output based on the user's accent or speaking style, affecting the naturalness and comprehensibility of voice interactions.

[0003] Therefore, how to improve the flexibility and personalization of speech synthesis is an urgent problem to be solved. Summary of the Invention

[0004] The embodiments of the present invention provide a speech synthesis method, apparatus, device and storage medium based on artificial intelligence to solve the technical problem that existing speech synthesis technology has difficulty in effectively separating speaker and accent characteristics, thereby limiting the flexibility and personalization of speech synthesis.

[0005] In a first aspect, a speech synthesis method based on artificial intelligence is provided, comprising: Obtain a reference Mel spectrum and a target text, where the reference Mel spectrum is the Mel spectrum corresponding to the speaker and accent selected by the user; Inputting the reference Mel spectrum into a variational autoencoder to obtain a speaker latent representation and an accent latent representation output by the variational autoencoder; Inputting the target text into a pre-trained text encoder to obtain a phoneme embedding representation output by the pre-trained text encoder; Inputting the speaker latent representation and the accent latent representation into a target vector quantizer to obtain a speaker embedding representation and an accent embedding representation output by the target vector quantizer; generating a target synthesized mel-spectrogram based on the phoneme embedding representation, the speaker embedding representation, and the accent embedding representation; The target synthesized Mel spectrum is converted into a target speech.

[0006] In a second aspect, a speech synthesis device based on artificial intelligence is provided, comprising: An acquisition module is used to obtain a reference Mel spectrum and a target text, wherein the reference Mel spectrum is the Mel spectrum corresponding to the speaker and accent selected by the user; A first input module is configured to input the reference Mel-spectrogram into a variational autoencoder to obtain a speaker potential representation and an accent potential representation output by the variational autoencoder; A second input module is configured to input the target text into a pre-trained text encoder to obtain a phoneme embedding representation output by the pre-trained text encoder; a third input module, configured to input the speaker latent representation and the accent latent representation into a target vector quantizer to obtain a speaker embedding representation and an accent embedding representation output by the target vector quantizer; a generation module, configured to generate a target synthesized mel-spectrogram based on the phoneme embedding representation, the speaker embedding representation, and the accent embedding representation; The conversion module is used to convert the target synthesized Mel spectrum into target speech.

[0007] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the artificial intelligence-based speech synthesis method described in the first aspect is implemented.

[0008] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the artificial intelligence-based speech synthesis method described in the first aspect is implemented.

[0009] In one solution implemented by the above-mentioned artificial intelligence-based speech synthesis method, device, equipment and storage medium, a hierarchical latent representation is extracted from the reference Mel spectrum through a variational autoencoder, that is, the speaker latent representation and the accent latent representation are extracted separately, and discretized into independent speaker embedding representations and accent embedding representations through a target vector quantizer, thereby achieving refined control of timbre and pronunciation habits, that is, more detailed modeling can be performed based on the speaker's personality characteristics (such as intonation, speaking speed, etc.) and accent characteristics (such as dialect, regional characteristics, etc.). Unlike traditional methods, which often mix speaker features and accent features, resulting in a lack of flexibility and personalization in the synthesized speech, the variational autoencoder can independently adjust and optimize these features in the same speech processing system, thereby generating more natural speech output that meets the needs of different users. In addition, the target text is processed into a phoneme embedding representation through a pre-trained text encoder, so that the generated target Mel spectrum can not only retain the semantic accuracy of the target text, but also flexibly integrate the timbre and diverse accent features of different speakers. This solves the technical problem that existing speech synthesis technology has difficulty in effectively separating speaker and accent features, which limits the flexibility and personalization of speech synthesis. It shows stronger robustness and adaptability in scenarios such as cross-language accent adaptation and personalized speech cloning, and improves the flexibility and personalization of synthesized speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0011] Figure 1 1 is a schematic diagram of an application environment of a speech synthesis method based on artificial intelligence in one embodiment of the present invention; Figure 2 is a flow chart of a speech synthesis method based on artificial intelligence in one embodiment of the present invention; Figure 3 This is an overall flow chart of a speech synthesis method based on artificial intelligence in one embodiment of the present invention; Figure 4 is another flow chart of a speech synthesis method based on artificial intelligence in one embodiment of the present invention; Figure 5 is another flow chart of a speech synthesis method based on artificial intelligence in one embodiment of the present invention; Figure 6 is another flow chart of a speech synthesis method based on artificial intelligence in one embodiment of the present invention; Figure 7 is another flow chart of a speech synthesis method based on artificial intelligence in one embodiment of the present invention; Figure 8 is a schematic diagram of a speech synthesis device based on artificial intelligence in one embodiment of the present invention; Figure 9 FIG. 1 is a schematic diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0012] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0013] The speech synthesis method based on artificial intelligence provided by the embodiment of the present invention can be applied as follows: Figure 1 Specifically, the speech synthesis method based on artificial intelligence is applied in a speech synthesis system based on artificial intelligence, and the speech synthesis system based on artificial intelligence includes the following: Figure 1 The client and server shown communicate over a network, addressing the technical issues of existing speech synthesis technologies, which struggle to effectively separate speaker and accent characteristics, limiting the flexibility and personalization of speech synthesis. Clients include, but are not limited to, various personal computers, laptops, smartphones, and tablets. The server can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0014] It should be understood that the speech synthesis method based on artificial intelligence can be applied to scenarios such as voice assistants, audio books, and entertainment. A speech synthesis method based on artificial intelligence provided by the present invention will be described in detail below using specific embodiments.

[0015] First, as Figure 2 and Figure 3 As shown, a speech synthesis method based on artificial intelligence is provided, which is applied in Figure 1 The server in the example is used as an example, and the steps are as follows: S10. Obtain a reference Mel frequency spectrum and a target text, where the reference Mel frequency spectrum is the Mel frequency spectrum corresponding to the speaker and accent selected by the user.

[0016] In this embodiment, the reference mel-spectrogram refers to a mel-spectrogram that includes speaker and accent characteristics and is stored in a pre-set speech database. Specifically, when obtaining the reference mel-spectrogram, speaker and accent selection information can be received from the user through interface operation or command input. For example, in the graphical interface of a speech synthesis application, the user can select the speaker and accent through a drop-down menu, search bar, etc. to obtain the corresponding reference mel-spectrogram. The reference mel-spectrogram can also be obtained through other methods, which are not limited here.

[0017] The target text refers to the text entered by the user for speech synthesis. Specifically, the target text can be text content directly entered by the user, such as keyboard input or handwriting, or text imported from an external file, without limitation. File formats include, but are not limited to, TXT, DOC, PDF, and DOCX.

[0018] As an example, in an audiobook scenario, the user can open the corresponding speech synthesis application, in which the graphical interface of the speech synthesis application is provided with a speaker and accent selection area. The user can browse speech samples of different speakers and accents, and select the target speaker and the corresponding accent according to the book style and personal preferences. The speech synthesis system then obtains the reference Mel spectrum corresponding to the speaker and accent from the material library. For the target text, the user can directly enter the target text to be synthesized in the text editing box of the speech synthesis application, or select a locally stored novel document through the "Import File" function to obtain the target text to be synthesized. The above steps can complete the acquisition of the reference Mel spectrum and target text, prepare for the subsequent speech synthesis process, and improve the flexibility and personalization of speech synthesis.

[0019] It should be understood that, if it is an "imported file", the document parsing program inside the speech synthesis application will automatically extract the text content in the "imported file" as the target text.

[0020] S20. Input the reference Mel-spectrogram into a variational autoencoder to obtain a speaker potential representation and an accent potential representation output by the variational autoencoder.

[0021] As an example, after obtaining a reference Mel-spectrogram, the speech synthesis system feeds it into a variational autoencoder (VAE) for processing to obtain the corresponding speaker latent representation and accent latent representation. The speaker latent representation characterizes individual characteristics of the speaker, such as timbre and pitch, while the accent latent representation characterizes the speaker's pronunciation habits, such as whether they have a particular regional dialect. By feeding the reference Mel-spectrogram into the variational autoencoder to obtain the speaker latent representation and accent latent representation, the system decouples the speaker from the accent, significantly improving the accuracy and personalization of speech synthesis.

[0022] S30: Input the target text into a pre-trained text encoder to obtain a phoneme embedding representation output by the pre-trained text encoder.

[0023] In this embodiment, the phoneme embedding representation refers to the feature representation obtained by inputting the target text into a pre-trained text encoder, that is, a low-dimensional representation vector of the text information. The function of the text encoder is to convert natural language text into a format suitable for speech generation, that is, a phoneme embedding representation. The text encoder can be composed of a deep learning model, such as a convolutional neural network (CNN) or a recurrent neural network (RNN), but this is not limited here.

[0024] For example, suppose the target text is: "Today's stock market performance is good, and my confidence has increased." The speech synthesis system will input this target text into a pre-trained text encoder to obtain the corresponding phoneme embedding representation. Specifically, the pre-trained text encoder analyzes each word in the target text and converts this information into a corresponding phoneme embedding representation.

[0025] It should be understood that the embedding representation in the present invention refers to the feature vector. The word embedding is commonly used in English. For example, the phoneme (wo) can be represented as a feature vector (23, 61).

[0026] S40: Input the speaker latent representation and the accent latent representation into a target vector quantizer to obtain a speaker embedding representation and an accent embedding representation output by the target vector quantizer.

[0027] As an example, the role of the target vector quantizer is to discretize the continuous potential representation into a finite number of embedding vectors, thereby effectively reducing the computational complexity while retaining the core timbre and accent information, thereby reducing redundancy in the calculation, reducing computational costs, and improving the speed and quality of speech synthesis.

[0028] As an example, after obtaining the speaker latent representation ( ) and accent latent representation ( ), first, the speaker latent representation ( ) and accent latent representation ( ) are input to the corresponding independent target vector quantizers, such as Figure 4 As shown, that is: S401: Input the speaker latent representation into the first target vector quantizer to obtain a speaker embedding representation output by the first target vector quantizer; S402: Input the accent latent representation into the second target vector quantizer to obtain an accent embedding representation output by the second target vector quantizer.

[0029] Specifically, different target vector quantizers correspond to different vector quantization codebooks, and each vector quantization codebook is composed of a predefined discrete embedding vector, for example, the speaker codebook ( }include =512 = 64-dimensional vector), accent codebook ( }include =256 = 32-dimensional vector). During the quantization process, each target vector quantizer can calculate the Euclidean distance between the input potential representation and each vector in the codebook using the following loss function: = in, Characterizes the stopping gradient operator, Then, the codebook vector with the closest distance is selected as the quantization result to obtain the speaker embedding representation ( ) and accent embedding representation ( ), for example, the speaker embedding is represented as = Calculated; accent embedding is expressed as = It should be understood that the embedding representation here also refers to the feature vector.

[0030] S50: Generate a target synthesized mel-spectrogram based on the phoneme embedding representation, the speaker embedding representation, and the accent embedding representation.

[0031] In this embodiment, after the phoneme embedding representation of the target text and the speaker embedding representation and accent embedding representation corresponding to the reference Mel-spectrogram have been obtained through the processing of steps S20 to S40, the speech synthesis system will then fuse the phoneme embedding representation, speaker embedding representation, and accent embedding representation to generate a target synthesized Mel-spectrogram that not only conforms to the semantics of the target text but also has the speaker and accent characteristics, ensuring the naturalness and personalization of the generated speech in terms of speech quality, timbre, accent, and intonation, and can meet the speech synthesis needs of specific fields.

[0032] As an example, in the financial field, assume the target text is "Today's stock market performance is good, and my confidence has increased." The speaker is a male investment analyst with a Mandarin accent. The speech synthesis system first extracts phonemes from the target text: / today / , / day / , / stock / , / market / , / table / , / present / , / no / , / wrong / , / I / , / trust / , / heart / , / increase / , / strengthen / . These phonemes are then converted into corresponding phoneme embeddings and combined with the speaker embedding (e.g., lower pitch and deeper intonation) and the Mandarin accent embedding (e.g., Mandarin pronunciation rules). The synthesis network then computes the target synthesized mel-spectrogram, ensuring that the synthesized speech not only accurately reflects the information in the target text but also possesses the timbre of the male investment analyst and conforms to Mandarin pronunciation characteristics. This embodiment achieves precise control of timbre and accent through hierarchical feature decoupling and codebook quantization.

[0033] S60: Convert the target synthesized Mel-spectrogram into target speech.

[0034] As an example, after obtaining the target synthesized Mel spectrum, the target synthesized Mel spectrum is further converted into the final target speech. This conversion process can be achieved by step S601, namely: S601: Input the target synthesized Mel spectrum into a pre-trained vocoder, so that the pre-trained vocoder converts the target synthesized Mel spectrum into target speech.

[0035] As an example, a pre-trained vocoder (eg, WaveGlow, HiFi-GAN, or VITS) can be used to convert the target synthesized Mel-spectrogram (ie, spectral features) into an audible audio signal.

[0036] In summary, in one embodiment provided by the present invention, a hierarchical latent representation is extracted from a reference Mel-spectrogram through a variational autoencoder, that is, a speaker latent representation and an accent latent representation are extracted separately, and they are discretized into independent speaker embedding representations and accent embedding representations through a target vector quantizer, thereby achieving refined control of timbre and pronunciation habits, that is, more detailed modeling can be performed based on the speaker's personality characteristics (such as intonation, speaking speed, etc.) and accent characteristics (such as dialect, regional characteristics, etc.). Unlike traditional methods, which often mix speaker features and accent features, resulting in a lack of flexibility and personalization in the synthesized speech, the variational autoencoder can independently adjust and optimize these features in the same speech processing system, thereby generating more natural speech output that meets the needs of different users. In addition, the target text is processed into a phoneme embedding representation through a pre-trained text encoder, so that the generated target Mel spectrum can not only retain the semantic accuracy of the target text, but also flexibly integrate the timbre and diverse accent features of different speakers. This solves the technical problem that existing speech synthesis technology has difficulty in effectively separating speaker and accent features, which limits the flexibility and personalization of speech synthesis. It shows stronger robustness and adaptability in scenarios such as cross-language accent adaptation and personalized speech cloning, and improves the flexibility and personalization of synthesized speech.

[0037] The following is a complete example to fully explain the contents described in the embodiment of steps S10 to S60 of the present invention.

[0038] As an example, in the financial field, suppose a user needs to convert text content into a speech report while performing daily stock market analysis. First, the user selects a speaker and an accent. For example, if they select Xiao Wang as the speaker and Mandarin as the accent, the speech synthesis system will obtain the corresponding reference Mel-spectrogram. Next, the reference Mel-spectrogram is input into a variational autoencoder to obtain the speaker latent representation and the accent latent representation. Furthermore, the analysis content (i.e., the target text), such as "Today, the Shanghai and Shenzhen indices are showing an overall upward trend, with the technology sector showing particularly significant gains. Investors are advised to pay attention to related high-potential stocks," is input into a pre-trained text encoder to obtain a phoneme embedding representation. The speaker latent representation and the accent latent representation are then input into a target vector quantizer to generate a speaker embedding representation and an accent embedding representation. Finally, the speech synthesis system combines the phoneme embedding, speaker embedding, and accent embedding representations to generate a target synthesized Mel-spectrogram. This is then converted into a speech report in Xiao Wang's voice with a Mandarin accent, completing the speech synthesis.

[0039] As another example, consider the healthcare and wellness field. A nurse hopes to learn relevant knowledge in this field by listening to speech. First, the nurse can select a speaker and accent in the speech synthesis system. For example, a female head nurse with a Cantonese accent can be selected. The speech synthesis system then obtains the corresponding reference mel-spectrogram. The reference mel-spectrogram is then input into a variational autoencoder, where it is processed to generate the speaker latent representation and the accent latent representation. Simultaneously, relevant knowledge in the healthcare and wellness field (i.e., target text) is input into a pre-trained text encoder to generate the corresponding phoneme embedding representation. The speaker latent representation and the accent latent representation are then input into the target vector quantizer to generate the speaker embedding and the accent embedding representation. Finally, the speech synthesis system generates a target synthesized Mel-spectrogram by integrating phoneme embedding representation, speaker embedding representation, and accent embedding representation, and then converts it into the voice of a female head nurse in Cantonese. Nurses can learn relevant knowledge in the field of medical health and wellness anytime and anywhere by playing this voice, greatly improving learning flexibility.

[0040] It should be understood that the above is merely an example and does not constitute a limitation of the present invention.

[0041] In one embodiment, if Figure 5 As shown, in step S20, the variational autoencoder includes multiple linear mapping layers; That is, the step of inputting the reference Mel spectrum into a variational autoencoder to obtain a speaker potential representation and an accent potential representation output by the variational autoencoder includes the following steps: S201, normalizing the reference Mel spectrum to obtain a normalized reference Mel spectrum; S202, inputting the normalized reference Mel spectrum into the first linear mapping layer of the variational autoencoder to obtain a first-level feature representation; S203, sequentially inputting the first-level feature representation into the middle multi-layer linear mapping layer of the variational autoencoder to obtain an intermediate-level feature representation; S204, inputting the intermediate-level feature representation into the last linear mapping layer to obtain a final feature representation; S205: Separate the speaker potential representation and the accent potential representation from the final feature representation.

[0042] In this embodiment, the purpose of normalizing the reference Mel-spectrum is to eliminate amplitude differences between different Mel-spectra, making subsequent processing more stable and efficient. For example, assuming the input reference Mel-spectrum values are a set of [30, 45, 50, 20] (unit: dB), after normalization, they can be converted to [0.5, 0.75, 0.83, 0.33] to ensure that the values input to the network have a uniform distribution range.

[0043] As an example, after obtaining a normalized reference Mel-spectrogram, it is input into the first linear mapping layer of the variational autoencoder. This first linear mapping layer performs a linear transformation on the input normalized reference Mel-spectrogram, mapping it into a high-dimensional feature space and outputting a first-level feature representation. This first-level feature representation includes preliminary features for representing the speaker and accent. This first-level feature representation is then sequentially passed to the intermediate layers of the variational autoencoder. Each linear mapping layer further transforms the feature representation, extracting progressively more abstract and deeper features to capture more complex speaker and accent characteristics, thereby enabling more precise latent representation separation. The intermediate-level feature representations, transformed by the intermediate layers, then enter the final linear mapping layer of the variational autoencoder. This final linear mapping layer performs final processing on all input intermediate-level feature representations and outputs a final feature representation. Finally, a latent representation separation operation is performed on the final feature representation to separate speaker features from accent features, generating independent speaker and accent latent representations. For example, during the separation process, the speaker latent representation can be [0.6, 0.8, 0.7], and the accent latent representation can be [0.4, 0.5, 0.2]. By leveraging the multi-layer linear mapping layer of the variational autoencoder, the speaker latent representation and accent latent representation can be efficiently extracted and separated, providing refined control for subsequent speech synthesis.

[0044] In one embodiment, if Figure 6 As shown, step S50, that is, generating a target synthesized Mel-spectrogram based on the phoneme embedding representation, the speaker embedding representation, and the accent embedding representation, includes the following steps: S501, inputting the phoneme embedding representation into a phoneme encoder to obtain a phoneme encoding sequence output by the phoneme encoder; S502: Input the phoneme encoding sequence, the speaker embedding representation, and the accent embedding representation into a variance adapter to obtain a phoneme feature representation output by the variance adapter; S503: Input the phoneme feature representation into a Mel decoder to obtain the target synthesized Mel spectrum generated by the Mel decoder.

[0045] In this embodiment, the phoneme encoder is used to encode the input phoneme embedding representation and form an ordered phoneme encoding sequence.

[0046] The variance adapter is used to receive a phoneme encoding sequence, a speaker embedding representation, and an accent embedding representation, and adjust and adapt the phoneme encoding sequence to incorporate speaker and accent characteristics, and output a phoneme feature representation.

[0047] The Mel decoder is used to convert the phoneme feature representation into the target synthesized Mel spectrum.

[0048] As an example, a phoneme embedding representation is encoded using a phoneme encoder to obtain a phoneme encoding sequence output by the phoneme encoder. The phoneme encoding sequence, speaker embedding representation, and accent embedding representation are then input into a variance adapter. The variance adapter then performs feature adjustment and adaptation based on the phoneme encoding sequence, speaker embedding representation, and accent embedding representation, outputting a phoneme feature representation. Finally, the phoneme feature representation is input into a Mel decoder, which decodes and converts the phoneme feature representation to generate the target synthesized Mel spectrum. This process not only improves the personalization and accuracy of speech synthesis, but also enables the speech synthesis system to flexibly adapt to the timbre and accent characteristics of different speakers, enhancing the naturalness and expressiveness of speech synthesis.

[0049] In one embodiment, if Figure 7 As shown, before step S60, that is, before converting the target synthesized Mel spectrum into target speech, the following steps are included: S60A, calculating a difference value between the reference Mel spectrum and the target synthesized Mel spectrum; S60B, if the difference value is greater than a preset threshold, adjusting the weight matrix and the bias vector of the multi-layer linear mapping layer in the variational autoencoder according to the difference value, wherein the adjustment step is aimed at minimizing the difference between the target synthesized Mel spectrum and the reference Mel spectrum, and adjusting the weight matrix and the bias vector through a backpropagation algorithm; S60C: If the difference value is less than or equal to the preset threshold, convert the target synthesized Mel-spectrogram into target speech.

[0050] In this embodiment, the reference mel-spectrogram represents the desired target speech features, while the target synthesized mel-spectrogram is a mel-spectrogram generated by a multi-layer neural network. The difference value is used to measure the similarity between the two, and can be calculated using mean square error or other methods, which are not limited here.

[0051] As an example, assume that the reference Mel-spectrogram represents a speaker's clear, standard pronunciation, while the target synthesized Mel-spectrogram represents a pre-generated speech sample. By calculating the difference between them, we can quantify the degree of difference between the two spectra. The difference can be calculated using the following formula: in, Characterize the reference Mel spectrum, Characterize the target synthetic mel-spectrogram.

[0052] If the calculated difference value is greater than a preset threshold, it indicates a significant discrepancy between the target synthesized Mel-spectrogram and the reference Mel-spectrogram. The speech synthesis system then adjusts the weight matrix and bias vector in the variational autoencoder based on this difference value. The goal of this adjustment is to minimize the difference between the target synthesized Mel-spectrogram and the reference Mel-spectrogram, thereby optimizing the generated speech quality. This adjustment is performed using a backpropagation algorithm, which calculates gradients based on an objective function (such as the mean squared error function or mean absolute error function). This algorithm then adjusts the weight matrix and bias vector of the variational autoencoder, gradually reducing the difference between the target synthesized Mel-spectrogram and the reference Mel-spectrogram. If the calculated difference value is less than or equal to the preset threshold, it indicates a small or acceptable difference between the target synthesized Mel-spectrogram and the reference Mel-spectrogram. The speech synthesis system then directly converts the target synthesized Mel-spectrogram into the target speech without further adjustment.

[0053] Through the steps of the above embodiment, the speech synthesis system can calculate the difference value and adjust the variational autoencoder parameters as needed before converting the target synthesized mel-spectrogram into the target speech, ensuring that the generated speech is more natural and accurate. Conversely, if the difference value is small, the target synthesized mel-spectrogram can be directly converted into the target speech. This process ensures both the efficiency and quality of speech synthesis and enhances the robustness of the speech system.

[0054] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0055] In a second aspect, a speech synthesis device based on artificial intelligence is provided, which corresponds one-to-one to the speech synthesis method based on artificial intelligence in the above embodiment. Figure 8 As shown, the speech synthesis device based on artificial intelligence includes an acquisition module 101, a first input module 102, a second input module 103, a third input module 104, a generation module 105 and a conversion module 106. The functional modules are described in detail as follows: An acquisition module 101 is configured to acquire a reference Mel spectrum and a target text, wherein the reference Mel spectrum is a Mel spectrum corresponding to a speaker and an accent selected by a user; A first input module 102 is configured to input the reference Mel-spectrogram into a variational autoencoder to obtain a speaker latent representation and an accent latent representation output by the variational autoencoder; A second input module 103 is configured to input the target text into a pre-trained text encoder to obtain a phoneme embedding representation output by the pre-trained text encoder; A third input module 104 is configured to input the speaker latent representation and the accent latent representation into a target vector quantizer to obtain a speaker embedding representation and an accent embedding representation output by the target vector quantizer; A generating module 105 is configured to generate a target synthesized mel-spectrogram based on the phoneme embedding representation, the speaker embedding representation, and the accent embedding representation; The conversion module 106 is configured to convert the target synthesized Mel-spectrogram into target speech.

[0056] In one embodiment, the variational autoencoder includes multiple layers of linear mapping layers; The first input module 102 is further configured to: Normalizing the reference Mel spectrum to obtain a normalized reference Mel spectrum; Inputting the normalized reference Mel spectrum into the first linear mapping layer of the variational autoencoder to obtain a first-level feature representation; Inputting the first-level feature representation into the middle multi-layer linear mapping layer of the variational autoencoder in sequence to obtain an intermediate-level feature representation; Inputting the intermediate-level feature representation into the last linear mapping layer to obtain the final feature representation; The speaker latent representation and the accent latent representation are separated from the final feature representation.

[0057] In one embodiment, the target vector quantizer includes a first target vector quantizer and a second target vector quantizer; The third input module 104 is further configured to: Inputting the speaker latent representation into the first target vector quantizer to obtain a speaker embedding representation output by the first target vector quantizer; The accent latent representation is input into the second target vector quantizer to obtain an accent embedding representation output by the second target vector quantizer.

[0058] In one embodiment, the generating module 105 is further configured to: Inputting the phoneme embedding representation into a phoneme encoder to obtain a phoneme encoding sequence output by the phoneme encoder; Inputting the phoneme encoding sequence, the speaker embedding representation, and the accent embedding representation into a variance adapter to obtain a phoneme feature representation output by the variance adapter; The phoneme feature representation is input into a Mel decoder to obtain the target synthesized Mel spectrum generated by the Mel decoder.

[0059] In one embodiment, the artificial intelligence-based speech synthesis device further includes: A calculation module 107 is configured to calculate a difference between the reference Mel spectrum and the target synthesized Mel spectrum; An adjustment module 108 is configured to adjust the weight matrix and bias vector of the multi-layer linear mapping layer in the variational autoencoder according to the difference value if the difference value is greater than a preset threshold, wherein the adjustment step is to minimize the difference between the target synthesized Mel spectrum and the reference Mel spectrum, and adjust the weight matrix and the bias vector through a backpropagation algorithm; if the difference value is less than or equal to the preset threshold, convert the target synthesized Mel spectrum into a target speech.

[0060] In one embodiment, the conversion module 106 is further configured to: The target synthesized Mel spectrum is input into a pre-trained vocoder, so that the pre-trained vocoder converts the target synthesized Mel spectrum into a target speech.

[0061] For the specific definition of the speech synthesis device based on artificial intelligence, please refer to the definition of the speech synthesis method based on artificial intelligence above, which will not be repeated here. The various modules in the above-mentioned speech synthesis device based on artificial intelligence can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0062] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 9As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data required for executing the artificial intelligence-based speech synthesis method described in the first aspect above. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the artificial intelligence-based speech synthesis method described in the first aspect above is implemented.

[0063] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed: Obtain a reference Mel spectrum and a target text, where the reference Mel spectrum is the Mel spectrum corresponding to the speaker and accent selected by the user; Inputting the reference Mel spectrum into a variational autoencoder to obtain a speaker latent representation and an accent latent representation output by the variational autoencoder; Inputting the target text into a pre-trained text encoder to obtain a phoneme embedding representation output by the pre-trained text encoder; Inputting the speaker latent representation and the accent latent representation into a target vector quantizer to obtain a speaker embedding representation and an accent embedding representation output by the target vector quantizer; generating a target synthesized mel-spectrogram based on the phoneme embedding representation, the speaker embedding representation, and the accent embedding representation; The target synthesized Mel spectrum is converted into a target speech.

[0064] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: Obtain a reference Mel spectrum and a target text, where the reference Mel spectrum is the Mel spectrum corresponding to the speaker and accent selected by the user; Inputting the reference Mel spectrum into a variational autoencoder to obtain a speaker latent representation and an accent latent representation output by the variational autoencoder; Inputting the target text into a pre-trained text encoder to obtain a phoneme embedding representation output by the pre-trained text encoder; Inputting the speaker latent representation and the accent latent representation into a target vector quantizer to obtain a speaker embedding representation and an accent embedding representation output by the target vector quantizer; generating a target synthesized mel-spectrogram based on the phoneme embedding representation, the speaker embedding representation, and the accent embedding representation; The target synthesized Mel spectrum is converted into a target speech.

[0065] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0066] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0067] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A speech synthesis method based on artificial intelligence, characterized in that: include: Obtain a reference Mel spectrum and a target text, where the reference Mel spectrum is the Mel spectrum corresponding to the speaker and accent selected by the user; Inputting the reference Mel spectrum into a variational autoencoder to obtain a speaker latent representation and an accent latent representation output by the variational autoencoder; Inputting the target text into a pre-trained text encoder to obtain a phoneme embedding representation output by the pre-trained text encoder; Inputting the speaker latent representation and the accent latent representation into a target vector quantizer to obtain a speaker embedding representation and an accent embedding representation output by the target vector quantizer; generating a target synthesized mel-spectrogram based on the phoneme embedding representation, the speaker embedding representation, and the accent embedding representation; The target synthesized Mel spectrum is converted into a target speech.

2. The artificial intelligence-based speech synthesis method according to claim 1, wherein: The variational autoencoder includes multiple layers of linear mapping layers; Inputting the reference Mel spectrum into a variational autoencoder to obtain a speaker potential representation and an accent potential representation output by the variational autoencoder includes: Normalizing the reference Mel spectrum to obtain a normalized reference Mel spectrum; Inputting the normalized reference Mel spectrum into the first linear mapping layer of the variational autoencoder to obtain a first-level feature representation; Inputting the first-level feature representation into the middle multi-layer linear mapping layer of the variational autoencoder in sequence to obtain an intermediate-level feature representation; Inputting the intermediate-level feature representation into the last linear mapping layer to obtain the final feature representation; The speaker latent representation and the accent latent representation are separated from the final feature representation.

3. The method for speech synthesis based on artificial intelligence according to claim 1, wherein: The target vector quantizer includes a first target vector quantizer and a second target vector quantizer; Inputting the speaker latent representation and the accent latent representation into a target vector quantizer to obtain a speaker embedding representation and an accent embedding representation output by the target vector quantizer includes: Inputting the speaker latent representation into the first target vector quantizer to obtain a speaker embedding representation output by the first target vector quantizer; The accent latent representation is input into the second target vector quantizer to obtain an accent embedding representation output by the second target vector quantizer.

4. The method for speech synthesis based on artificial intelligence according to claim 1, wherein: Generating a target synthesized Mel spectrum according to the phoneme embedding representation, the speaker embedding representation, and the accent embedding representation includes: Inputting the phoneme embedding representation into a phoneme encoder to obtain a phoneme encoding sequence output by the phoneme encoder; Inputting the phoneme encoding sequence, the speaker embedding representation, and the accent embedding representation into a variance adapter to obtain a phoneme feature representation output by the variance adapter; The phoneme feature representation is input into a Mel decoder to obtain the target synthesized Mel spectrum generated by the Mel decoder.

5. The method for speech synthesis based on artificial intelligence according to claim 1, wherein: Before converting the target synthesized Mel spectrum into target speech, the method includes: Calculating a difference between the reference Mel spectrum and the target synthesized Mel spectrum; If the difference value is greater than a preset threshold, adjusting the weight matrix and the bias vector of the multi-layer linear mapping layer in the variational autoencoder according to the difference value, wherein the adjustment step is to minimize the difference between the target synthesized Mel spectrum and the reference Mel spectrum, and adjust the weight matrix and the bias vector by a backpropagation algorithm; If the difference value is less than or equal to the preset threshold, the target synthesized Mel spectrum is converted into target speech.

6. The method for speech synthesis based on artificial intelligence according to claim 1, wherein: The converting the target synthesized Mel spectrum into a target speech comprises: The target synthesized Mel spectrum is input into a pre-trained vocoder, so that the pre-trained vocoder converts the target synthesized Mel spectrum into a target speech.

7. A speech synthesis device based on artificial intelligence, characterized in that: include: An acquisition module is used to obtain a reference Mel spectrum and a target text, wherein the reference Mel spectrum is the Mel spectrum corresponding to the speaker and accent selected by the user; A first input module is configured to input the reference Mel-spectrogram into a variational autoencoder to obtain a speaker potential representation and an accent potential representation output by the variational autoencoder; A second input module is configured to input the target text into a pre-trained text encoder to obtain a phoneme embedding representation output by the pre-trained text encoder; a third input module, configured to input the speaker latent representation and the accent latent representation into a target vector quantizer to obtain a speaker embedding representation and an accent embedding representation output by the target vector quantizer; a generation module, configured to generate a target synthesized mel-spectrogram based on the phoneme embedding representation, the speaker embedding representation, and the accent embedding representation; The conversion module is used to convert the target synthesized Mel spectrum into target speech.

8. The artificial intelligence-based speech synthesis device according to claim 7, wherein: The variational autoencoder includes multiple layers of linear mapping layers; The first input module is further configured to: Normalizing the reference Mel spectrum to obtain a normalized reference Mel spectrum; Inputting the normalized reference Mel spectrum into the first linear mapping layer of the variational autoencoder to obtain a first-level feature representation; Inputting the first-level feature representation into the middle multi-layer linear mapping layer of the variational autoencoder in sequence to obtain an intermediate-level feature representation; Inputting the intermediate-level feature representation into the last linear mapping layer to obtain the final feature representation; The speaker latent representation and the accent latent representation are separated from the final feature representation.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the artificial intelligence-based speech synthesis method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the artificial intelligence-based speech synthesis method according to any one of claims 1 to 6 is implemented.

Images