Speech synthesis method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computers, in particular to a speech synthesis method and device, electronic equipment and a storage medium. The method comprises the following steps: determining a target timbre feature based on an obtained target timbre label and a trained timbre representation generation model, and determining a first target voice of a preset timbre based on an obtained target text and a trained voice synthesis model; wherein the preset timbre is the timbre of any first sample voice for training the voice synthesis model; furthermore, based on the target timbre feature, the first target voice and a trained voice conversion model, a second target voice of the target timbre can be determined. Thus, through decoupling the timbre representation generation model and the voice synthesis model and using the voice conversion model to perform timbre conversion, a new timbre can be generated, and individual requirements of a user on the timbre of the synthesized voice in different scenes can be met.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a speech synthesis method, device, electronic device and storage medium. Background Art

[0002] In recent years, artificial intelligence has been applied to text-to-speech (TTS) technology, which is a technology that generates natural and expressive speech from text. It is widely used in voice assistants, game NPCs, chatbots, audiobooks, virtual humans, virtual anchors, etc. In TTS technology, it is generally necessary to synthesize speech with a specific timbre.

[0003] Typically, the related technology uses a speech model trained based on data of multiple speakers for speech synthesis. Specifically, the corresponding speaker timbre feature embedding is selected through the speaker ID, and the timbre of the generated speech is controlled according to the corresponding speaker timbre feature embedding. In this context, the timbre of the generated speech is usually limited to the timbre within the training data or can only be the timbre of an existing real human being, which results in the lack of personalization of the synthesized speech and does not meet the user's timbre requirements for the synthesized speech. However, the goal of timbre generation technology is to generate new timbres. Therefore, there is an urgent need for a speech synthesis solution that generates new timbres to meet the user's personalized needs for the timbre of synthesized speech in different scenarios.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] In view of this, the embodiments of the present application at least provide a speech synthesis method, device, electronic device and storage medium, which can generate new timbres and meet the user's personalized needs for the timbre of synthesized speech in different scenarios.

[0006] This application mainly includes the following aspects:

[0007] In a first aspect, an embodiment of the present application provides a speech synthesis method, the method comprising: obtaining a target timbre label and a target text of a speech to be synthesized; determining a target timbre feature based on the target timbre label and a trained timbre representation generation model, and determining a first target speech with a preset timbre based on the target text and a trained speech synthesis model; the preset timbre is the timbre of any first sample speech for training the speech synthesis model; and determining a second target speech with a target timbre based on the target timbre feature, the first target speech, and a trained speech conversion model.

[0008] In a second aspect, an embodiment of the present application also provides a speech synthesis device, which includes: an acquisition module for acquiring a target timbre label and a target text of a speech to be synthesized; a first determination module for determining a target timbre feature based on the target timbre label and a trained timbre representation generation model, and determining a first target speech with a preset timbre based on the target text and a trained speech synthesis model; the preset timbre is the timbre of any first sample speech for training the speech synthesis model; a second determination module for determining a second target speech with a target timbre based on the target timbre feature, the first target speech and a trained speech conversion model.

[0009] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the speech synthesis method described in the first aspect or any possible implementation manner of the first aspect.

[0010] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the speech synthesis method described in the first aspect or any possible implementation manner of the first aspect are executed.

[0011] The speech synthesis method, device, electronic device and storage medium provided in the embodiments of the present application determine the target timbre feature based on the acquired target timbre label and the trained timbre representation generation model, and determine the first target speech of the preset timbre based on the acquired target text and the trained speech synthesis model; wherein the preset timbre is the timbre of any first sample speech of the training speech synthesis model; further, the second target speech of the target timbre can be determined based on the target timbre feature, the first target speech and the trained speech conversion model, compared with the related art of selecting the corresponding speaker timbre feature embedding by the speaker ID, and controlling the timbre of the generated speech according to the corresponding speaker timbre feature embedding. In this context, the timbre of the generated speech is usually limited to the timbre in the training data or can only be the timbre of an existing real human being, which results in the synthesized speech lacking personalization and not meeting the user's timbre requirements for the synthesized speech. In comparison, the present application decouples the timbre representation generation model and the speech synthesis model, and uses the speech conversion model for timbre conversion, so as to generate a new timbre, which can meet the user's personalized requirements for the timbre of the synthesized speech in different scenarios.

[0012] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 A flow chart of a speech synthesis method provided by an embodiment of the present application is shown;

[0015] Figure 2 A flowchart of the training process of the timbre representation generation model in the speech synthesis method provided in the embodiment of the present application;

[0016] Figure 3 One of the functional module diagrams of a speech synthesis device provided in an embodiment of the present application is shown;

[0017] Figure 4 A second functional module diagram of a speech synthesis device provided in an embodiment of the present application is shown;

[0018] Figure 5 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art, under the guidance of the content of the present application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0020] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0021] In order to enable those skilled in the art to use the contents of this application, the following implementation is given in combination with the specific application scenario "speech synthesis". For those skilled in the art, the general principles defined here can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application.

[0022] The following methods, devices, electronic devices or computer-readable storage media of the embodiments of the present application can be applied to any scenario requiring speech synthesis. The embodiments of the present application are not limited to specific application scenarios. Any scheme using the speech synthesis method and device provided by the embodiments of the present application is within the scope of protection of the present application.

[0023] The speech synthesis method improved by the embodiment of the present application can be implemented by a computer device (electronic device), and the computer device can be a terminal or a server. The terminal can be a mobile phone, a tablet computer, a notebook computer, a desktop computer, etc. An application with a speech synthesis function can be installed in the terminal. The server can be a background server of an application with a speech synthesis function, and the server can be a single server or a device group consisting of multiple devices.

[0024] From the perspective of hardware composition, the structure of a computer device may include a processor and a memory. The processor may be a central processing unit (CPU) or a system on chip (SoC), etc. The processor may be used to execute various instructions involved in the method, etc. The memory may include various volatile memories or non-volatile memories, such as solid state disks (SSDs), dynamic random access memories (DRAMs), etc. The memory may be used to store pre-stored data, intermediate data, and result data during the training process, such as sample speech, etc.

[0025] In addition to the processor and memory, the terminal may also include a display component, a communication component, a voice (audio) acquisition component, a voice (audio) output component, etc. The display component may be an independent screen, or a screen integrated with the terminal body, a projector, etc. The screen may be a touch screen or a non-touch screen. The display component is used to display the system interface, application interface, etc., for example, the interface for recording audio, etc.

[0026] The communication component may be a wired network connector, a wireless fidelity (WiFi) module, a Bluetooth module, a cellular network communication module, etc. The communication component may be used to transmit data with other devices, and the other devices may be servers or other terminals.

[0027] The voice collection component may be a microphone for collecting the user's voice. The voice output component may be a speaker, earphone, etc. for playing audio.

[0028] From the perspective of hardware composition, the structure of the server may include a processor and a memory. The processor may be a CPU or a SoC, etc., and the processor may be used to execute various instructions involved in the method, etc. The memory may include various volatile memories or non-volatile memories, such as SSD, DRAM memory, etc. The memory may be used to store pre-stored data, intermediate data, and result data during the training process, such as sample speech, timbre features, predicted speech, etc.

[0029] It is worth noting that the related technology uses a speech model trained based on data of multiple speakers for speech synthesis. Specifically, the corresponding speaker timbre feature embedding is selected through the speaker ID, and the timbre of the generated speech is controlled according to the corresponding speaker timbre feature embedding. In this context, the timbre of the generated speech is usually limited to the timbre within the training data or can only be the timbre of an existing real human being, which results in the lack of personalization of the synthesized speech and does not meet the user's timbre requirements for the synthesized speech. However, the goal of timbre generation technology is to generate new timbres. Therefore, there is an urgent need for a speech synthesis solution that generates new timbres to meet the user's personalized needs for the timbre of synthesized speech in different scenarios.

[0030] In response to the above problems, the embodiments of the present application determine the target timbre features based on the acquired target timbre label and the trained timbre representation generation model, and determine the first target speech of the preset timbre based on the acquired target text and the trained speech synthesis model; wherein the preset timbre is the timbre of any first sample speech of the trained speech synthesis model; further, the second target speech of the target timbre can be determined based on the target timbre features, the first target speech, and the trained speech conversion model. In this way, the present application can generate a new timbre by decoupling the timbre representation generation model and the speech synthesis model, and using the speech conversion model for timbre conversion, which can meet the user's personalized needs for the timbre of synthesized speech in different scenarios.

[0031] The present application provides a speech synthesis method, device, electronic device and storage medium, which can be applied to various scenarios requiring speech synthesis and controllable dubbing timbre, such as public services (information broadcasting, intelligent customer service, etc.), intelligent hardware (such as intelligent speakers, intelligent robots, etc.), smart transportation (voice navigation, intelligent vehicle-mounted equipment, etc.), education (smart classrooms, foreign language learning, etc.), and pan-entertainment (audio reading, film and television dubbing, virtual IP, virtual humans, virtual anchors, game NPCs, etc.). The speech synthesis solution of the present application can synthesize speech with the timbre expected by the user, thereby improving the personalization of the synthesized speech.

[0032] Below, several nouns involved in the embodiments of the present application are introduced:

[0033] Speech synthesis is a technology that converts text into speech / audio. Based on the input text, the corresponding speech audio can be generated. It can be part of the human-computer dialogue, allowing the machine to speak.

[0034] Voice Conversion (VC) is a technology that converts the voice of a source speaker into a target style. That is, a piece of speech is transformed from the source speaker to the target speaker in some way, such as changing the speaker identity, rhythm and emotion, while keeping the language content unchanged. Specifically, it can be achieved by first extracting the source content information and the target speaker information, and then using this information to reconstruct the waveform. Classic voice conversion applications include speaker conversion, dubbing of film and television works, speech assistance system, pronunciation conversion, etc.

[0035] To facilitate the understanding of the present application, the technical solution provided by the present application is described in detail below in conjunction with specific embodiments.

[0036] Figure 1 This is a flow chart of a speech synthesis method provided by an embodiment of the present application. Figure 1 As shown, the speech synthesis method provided in the embodiment of the present application includes the following steps:

[0037] S101: Obtain a target timbre label and a target text of the speech to be synthesized.

[0038] Here, the speech to be synthesized refers to the speech that the user wants to synthesize. The target text is the speech content corresponding to the speech to be synthesized. The target timbre label is the description information of the target timbre of the speech to be synthesized, and the description information includes information describing the target timbre from at least one timbre dimension, for example, describing the timbre from dimensions such as gender, age, voice, pitch, language, spoken language, personality, and occupation. In one example, the target timbre label is such as "lively little Lolita", "cold lady", "uncle with a magnetic voice", etc.

[0039] It should be noted that the target timbre label can be input by the user or automatically generated by the user selecting a timbre description. Here, the input method includes but is not limited to an image input method, a text input method, etc. Among them, the input in the image input method may include: a picture containing a target object, wherein the target object is an object having the timbre that the user expects to synthesize; the input in the text input method may include: a description text describing the timbre that the user expects to synthesize from various timbre dimensions.

[0040] In an optional embodiment, the target timbre label includes descriptive information of the target timbre of the speech to be synthesized from at least one of the following timbre dimensions: gender, age, voice, pitch, language, spoken language, personality, and occupation.

[0041] Among them, gender includes male, female, neutral, etc.; age includes old, middle-aged, young, teenage, childhood, etc.; voice includes hoarse, hoarse (similar to the breathy sound of singing), smoky voice, magnetic, thick (full), rough, sunny, etc.; tone includes sharp (tone), low (tone), gentle, masculine, silly, sweet, coquettish, coquettish (including clip sound), sweet, pure (soft / healing / transparent), lustful, royal (sister), noble (queen / domineering), etc.; language includes Chinese, English, Russian, etc.; spoken language includes dialects; personality includes cold, shy, lively, enthusiastic, etc.; occupation includes doctor, teacher, lawyer, etc.

[0042] S102: Determine the target timbre features based on the target timbre label and the trained timbre representation generation model, and determine the first target speech with a preset timbre based on the target text and the trained speech synthesis model; the preset timbre is the timbre of any first sample speech for training the speech synthesis model.

[0043] S103: Determine a second target voice with a target timbre based on the target timbre feature, the first target voice and the trained voice conversion model.

[0044] It should be noted that in the related technologies of speech synthesis, usually, the corresponding speaker timbre feature embedding is selected by the speaker ID, and the timbre of the generated speech is controlled according to the corresponding speaker timbre feature embedding. In this context, the timbre of the generated speech is usually limited to the timbre within the training data, and the timbre outside the training data cannot be generated.

[0045] In response to this situation, some researchers have devoted themselves to studying a technology that generates speech with the target speaker's timbre by inputting speaker embedding extracted from the target timbre audio. This technology can generate timbre that is not in the training set, but it is also limited to synthesizing the timbre of existing real people. In the field of speech synthesis, the goal of timbre generation technology is actually to generate new timbre, and the generation of this new timbre should not depend on any speaker ID or target timbre of the speech, but directly generate a new timbre representation.

[0046] In addition, the timbre generation technology has a wide range of application prospects in the current explosive growth of artificial intelligence (AIGC) technology, such as making the timbre of virtual people independent of a specific real voice actor, or generating a variety of timbres for a large number of NPCs in the game, or generating a unique timbre for chatbots such as ChatGPT. In short, a good timbre generation system is committed to generating new timbres and has a wide range of application prospects.

[0047] Based on this, in response to the personalized needs of users for the timbre of synthesized speech in different scenarios, the embodiment of the present application provides a speech synthesis solution for generating a new timbre. The implementation process of the speech synthesis solution provided by the embodiment of the present application is described below. The process includes: separating the timbre representation generation model from the speech synthesis model, using the speech synthesis model to generate the speech of a certain speaker, and then using the speech conversion model to convert the speech of a certain person into the speech of the generated timbre. Specifically, firstly synthesize a first target speech of any timbre based on the speech synthesis model and the target text, and generate a target timbre feature (timbre representation) of a new timbre based on the target timbre label and the timbre representation generation model, and further, input the target timbre feature into the speech conversion model to convert the first target speech of the preset timbre synthesized by the speech synthesis model into the second target speech of the target timbre, thereby realizing the generation of a speech of a new timbre, which can meet the personalized needs of users for the timbre of synthesized speech in different scenarios. Here, the speech content of the first target speech is the same as the speech content of the second target speech, and the timbre of the first target speech is different from the timbre of the second target speech. The timbre of the first target speech is the timbre of the training sample (first sample speech) of the speech synthesis model, and the timbre of the second target speech is a new timbre corresponding to the target timbre feature output by the timbre representation generation model.

[0048] Here, the embodiment of the present application regards the timbre representation generation model and the speech synthesis model as two completely independent models and cooperates with the speech conversion model to jointly perform the speech synthesis task, which is different from the related art in which the timbre representation generation model and the speech synthesis model are jointly trained, so that the training data for timbre generation must also be the training data for speech synthesis, which greatly limits the data source, data scale and data diversity of timbre generation. At the same time, since the speech synthesis model needs to be jointly trained with the timbre representation generation model, the speech synthesis model cannot select data without timbre annotation for training, which also limits the expressiveness of the speech synthesis model. In comparison, the embodiment of the present application decouples the timbre representation generation model and the speech synthesis model, that is, separates the timbre representation generation model from the speech synthesis model, so that the training data of the timbre representation generation model no longer depends on the training data of the speech synthesis model, which can greatly expand the scale and source of training data for timbre generation. At the same time, the speech synthesis model can also be trained with more data without considering whether there is timbre annotation, and the speech conversion model itself does not require text annotation and can be trained with big data, which is more suitable as a part of speech generation.

[0049] It should be noted that the timbre representation generation model, speech synthesis model and speech conversion model involved in the embodiments of the present application are trained separately, and the training data between them may have overlapping parts or be completely different. This makes the training data source of each model richer, thereby improving the training effect of the model.

[0050] Here, the purpose of each model is first explained. Specifically, the timbre characterization generation model provided in the embodiment of the present application is a model for generating new timbre feature representations. The timbre characterization generation model includes but is not limited to a diffusion model (Diffusion Model), a convolutional neural network model (CNN), a deep learning model (Attention), a generative adversarial network model (GAN), a variational autoencoder (VAE), and a Gaussian mixture model (GMM). The input of the timbre characterization generation model is a timbre label, and the output is a timbre feature that matches the timbre label.

[0051] Among them, the timbre feature is also called timbre coding feature, timbre representation feature, speaker embedding feature. The timbre feature of the target timbre can reflect the characteristics of the target timbre. The timbre feature is used to quantify the target timbre. The timbre feature can be represented in a vector form, such as a timbre feature vector; the timbre feature can also be represented in a matrix form, such as a timbre feature matrix.

[0052] Here, the speech synthesis model provided in the embodiment of the present application is a model for converting text into speech. The speech synthesis model can be a sound synthesis model. The input of the speech synthesis model is text information, and the output of the speech synthesis model is speech.

[0053] It should be emphasized that the inventive concept of the embodiment of the present application is to input the first target speech of any timbre synthesized by the speech synthesis model and the target timbre characteristics of the new timbre generated by the timbre representation generation model into the speech conversion model to convert the second target speech of the new timbre (target timbre). Therefore, the speech synthesis model in the embodiment of the present application does not care whose timbre the generated speech is, but only requires that the speech synthesis model can synthesize speech with timbre. Therefore, the speech synthesis model can select a conventional TTS model. In this way, the requirements for the selection of the first sample speech for training the speech synthesis model can be relaxed, and the first sample speech can be a speech without timbre annotation. Similarly, the timbre of the first target speech output by the speech synthesis model can be the timbre of any first sample speech in the training data.

[0054] Here, the speech conversion model provided in the embodiment of the present application is a speech-to-speech technology, which can be understood as the input is a speech with timbre A, and the output is a speech with timbre B. In this conversion process, only the timbre is changed, and other things such as speech speed, intonation, and speech content are exactly the same. The speech conversion model can be a zero-shot model. For the embodiment of the present application, the input of the speech conversion model is a speech with an existing timbre and the timbre features of a new timbre, and the output of the speech conversion model is a speech with a new timbre.

[0055] In an embodiment of the present application, the target timbre features are determined based on the acquired target timbre label and the trained timbre representation generation model, and the first target speech of the preset timbre is determined based on the acquired target text and the trained speech synthesis model; wherein the preset timbre is the timbre of any first sample speech of the trained speech synthesis model; further, the second target speech of the target timbre can be determined based on the target timbre features, the first target speech, and the trained speech conversion model. In this way, the present application can generate a new timbre by decoupling the timbre representation generation model and the speech synthesis model, and using the speech conversion model for timbre conversion, which can meet the user's personalized needs for the timbre of synthesized speech in different scenarios.

[0056] In an optional embodiment, the target timbre feature is a timbre feature matrix; the step S102 of determining the target timbre feature based on the target timbre label and the trained timbre representation generation model specifically includes the following steps: inputting the target timbre label into the trained timbre representation generation model to perform timbre data gradient learning to determine the timbre feature matrix.

[0057] It should be noted that the timbre features in the related technology are usually speaker embeddings or voiceprint representations of multi-speaker speech synthesis models. Regardless of which representation is used, a single vector is usually used, which makes the structure of modeling the single vector limited to linear layers, and the generative modeling method is also limited to methods based on the GMM model. It is impossible to use more complex model structures such as CNN and Attention, and it is impossible to apply more powerful generative modeling methods, such as the currently popular Diffusion model. This shortcoming greatly limits the effect of timbre generation.

[0058] In view of the above problems in the related art, the embodiment of the present application improves the modeling of timbre generation, and no longer uses a single vector as the modeling target, but uses a timbre feature matrix as the timbre feature, and adjusts the size of the timbre feature to an M×N timbre feature matrix, such as a 256-dimensional timbre feature can be adjusted to a size of 16×16. In this way, the modeling of speech representation can be similar to the image generation task, and more complex structures and generative modeling methods can be applied. For example, using a Diffusion model based on a UNet structure as a timbre representation generation model can greatly improve the timbre effect of the generated new timbre, so that the timbre effect of the embodiment of the present application can far exceed the timbre effect of the timbre generated by the timbre generation scheme in the related art.

[0059] In a specific implementation, when performing a timbre generation application, the target timbre label can be directly input into a trained timbre representation generation model to perform timbre data gradient learning, thereby generating a timbre feature matrix of the target timbre.

[0060] Figure 2 The flowchart of the training process of the timbre characterization generation model in the speech synthesis method provided in the embodiment of the present application. Figure 2 As shown, the training process of the timbre characterization generation model provided in the embodiment of the present application includes the following steps:

[0061] S201: Acquire a plurality of second sample voices, and determine a real sample timbre feature and a sample timbre label corresponding to each second sample voice.

[0062] S202: For any of the second sample speech, input the sample timbre label of the second sample speech into the timbre representation generation model to be trained to determine the predicted sample timbre feature.

[0063] S203: Training the timbre representation generation model based on a first loss value between the real sample timbre feature and the predicted timbre feature corresponding to each of the second sample voices.

[0064] Here, the training samples used to train the timbre characterization generation model in the embodiment of the present application can be speech samples, and no text samples are required. For the training process of the timbre characterization generation model, specifically, for any second sample speech, the real sample timbre feature and sample timbre label corresponding to the second sample speech can be determined first, the real sample timbre feature is the timbre feature extracted from the second sample speech, and the sample timbre label is the information used to describe the timbre in the second sample speech, such as "the timbre of a hoarse tough guy"; then, the sample timbre label of each second sample speech is input into the timbre characterization generation model to be trained, and the predicted sample timbre feature that matches the sample timbre label to a certain extent can be determined; further, the first loss value between the real sample timbre feature and the predicted timbre feature corresponding to each second sample speech is calculated, and the timbre characterization generation model is trained with the minimization of the first loss value of each second sample speech as the training goal. In addition, if the training end condition is met, the timbre characterization generation model that meets the training end condition is determined as the trained timbre characterization generation model; if the training end condition is not met, that is, when the timbre characterization generation model has not been trained, the timbre features of the predicted samples obtained above are only a predicted value, which may not be accurate enough, and the timbre characterization generation model needs to be further trained. Specifically, the second sample speech can be used to repeat the above process until the training end condition is met.

[0065] There are many options for the training end conditions, and the following are some of them: Condition 1: reaching the set number of training times. Condition 2: the loss value is less than the preset threshold. Condition 3: the loss value no longer has a trend of shrinking. Condition 4: using the timbre features generated by the timbre representation generation model to perform speech synthesis experiments, the synthesized speech is compared with the sample speech used to generate the timbre features, and the matching degree reaches the preset value, or the matching degree of multiple consecutive experiments reaches the specified value.

[0066] Here, the timbre features are represented by vectors or matrices. If represented by vectors, the loss between the predicted and real timbre features can be represented by the angle between the vectors. When the angle between the two vectors is extremely small, it indicates that the difference between the timbre features represented by the two vectors is small. When the angle between the two vectors is large, it indicates that the difference between the timbre features represented by the two vectors is large. For the present application, the first loss value can be determined based on the vector angle between the real sample timbre features and the predicted timbre features. Specifically, the cosine value of the vector angle can be used as the first loss value.

[0067] In an optional embodiment, the real sample timbre features corresponding to the second sample speech are determined according to the following steps: the second sample speech is input into a trained voiceprint extraction model to extract timbre features, and the real sample timbre features corresponding to the second sample speech are determined.

[0068] Here, the voiceprint extraction model is a timbre feature extraction model, and the voiceprint extraction model can be a machine learning model. The network structure used in the voiceprint extraction model can be an emphasized channel attention propagation and aggregation in time delay neural network based speaker verification (ECAPA-TDNN), which is a mainstream voiceprint model.

[0069] Here, for the embodiment of the present application, the timbre features extracted from the sample speech by the trained voiceprint extraction model can be used as the real timbre features. Specifically, when performing voiceprint extraction, the second sample speech can be first processed to obtain Mel spectrum or spectrum data or time domain audio data, and then timbre features can be further extracted to extract timbre features.

[0070] In an optional embodiment, the step of determining the first target speech of a preset timbre based on the target text and a trained speech synthesis model in S102 includes the following steps: determining a phoneme sequence according to the text content of the target text; determining text pronunciation features according to the phoneme sequence; and determining the first target speech of the preset timbre according to the text pronunciation features and the preset timbre features of the preset timbre.

[0071] Here, when performing speech synthesis, it is necessary to input the text content of the target text of the speech to be synthesized, and determine the phoneme sequence corresponding to the text content. For example, the phoneme sequence corresponding to the text "today" is "j, i, n, t, i, a, n". Furthermore, after the phoneme sequence is input into the encoder, the encoder can output the phoneme vector corresponding to the phoneme sequence, that is, the text pronunciation feature. Among them, the text pronunciation feature can also be called a phoneme vector or text encoding. Furthermore, according to the text pronunciation feature and the preset timbre feature, the first target speech of the preset timbre can be obtained, wherein the preset timbre feature is the timbre feature of the preset timbre of a first sample speech arbitrarily selected from the speech synthesis model.

[0072] In one example, for example, the text content of the target text is "Today's weather is very sunny", and the text pronunciation feature corresponding to the target text is the phoneme vector T (t1, t2, ..., t n ), n is a positive integer, and the preset timbre feature is the timbre feature vector H (h1, h2, ..., h m ), m is a positive integer, then based on the phoneme vector T(t1, t2, ..., t n) and timbre feature vector H(h1, h2, …, h m ), a first target speech with a preset timbre can be obtained, wherein the "preset timbre" in the first target speech is consistent with the "preset timbre" in the preset timbre feature, and the "preset timbre" is the timbre of any first sample speech for training the speech synthesis model.

[0073] In an optional embodiment, the speech synthesis model is trained according to the following steps: obtaining multiple sample texts and a first sample speech corresponding to each sample text, and extracting a first sample timbre feature from the first sample speech; for any of the sample texts, inputting the sample text and the first sample timbre feature corresponding to the sample text into the speech synthesis model to be trained to determine a first predicted sample speech; and training the speech synthesis model based on a second loss value between the first sample speech corresponding to each of the sample texts and the first predicted sample speech.

[0074] Here, the speech synthesis model and the timbre characterization generation model in the present application are trained separately, so the first sample speech used to train the speech synthesis model and the second sample speech used to train the timbre characterization generation model may be different.

[0075] The training process of the speech synthesis model is explained below. First, a large number of first sample speech can be obtained, and each first sample speech is annotated with a sample text, which is the speech content of the corresponding first sample speech. The first sample timbre feature can also be extracted from the first sample speech as a speaker embedding feature; then, each sample text and the corresponding first sample timbre feature can be input into the speech synthesis model to be trained to determine the first predicted sample speech; further, the speech synthesis model is trained with minimizing the second loss value between the first sample speech corresponding to the sample text and the first predicted sample speech as the training goal, wherein the training end condition of the speech synthesis model can refer to the training end condition of the timbre representation generation model.

[0076] It should be noted that when training the speech synthesis model, there may be no corresponding relationship between the sample text and the first sample speech in the training data. The key data used in training is essentially only the sample text and any sample timbre features. Here, the reason why the sample text and the first sample speech are corresponded is mainly to consider the loss of the model, that is, the loss between the real first sample speech and the first predicted sample speech corresponding to the model synthesis. Therefore, the training data can select the sample text and the corresponding first sample speech.

[0077] Here, the first sample timbre feature can be extracted from the first sample speech by the voiceprint feature extraction model as the speaker embedding feature of the speech synthesis model, so that when the speech synthesis model is trained, the speech with the speaker embedding feature (timbre feature) can be synthesized.

[0078] In an optional embodiment, the target timbre feature is a timbre feature matrix; the step of determining the second target voice of the target timbre based on the target timbre feature, the first target voice and a trained voice conversion model in S103 includes the following steps: converting the timbre feature matrix into a timbre feature vector; inputting the timbre feature vector and the first target voice into a trained voice conversion model for timbre conversion, and determining the second target voice of the target timbre.

[0079] Here, if the timbre characterization generation model outputs a timbre feature matrix, it is necessary to first convert the timbre feature matrix into a timbre feature vector, and then input the timbre feature vector and the first target speech of the preset timbre output by the speech synthesis model into the trained speech conversion model for timbre transformation. Specifically, "timbre transformation" is based on the timbre feature vector corresponding to the target timbre, replacing the preset timbre in the first target speech with the target timbre, thereby obtaining the second target speech of the target timbre, that is, converting the first target speech of the preset timbre into the second target speech of the target timbre. It should also be made clear that in the process of "timbre transformation", only the timbre is changed, and the speech speed, intonation, speech content, etc. in the speech are exactly the same. Among them, the target timbre is the timbre that matches the target timbre label, that is, the personalized timbre required by the user.

[0080] In an optional embodiment, the speech conversion model is trained according to the following steps: obtaining multiple third sample speech and multiple fourth sample speech, and determining the fourth sample timbre feature corresponding to each of the fourth sample speech; inputting the fourth sample timbre feature and the third sample speech into the speech conversion model to be trained to determine the second predicted sample speech; extracting the fifth sample timbre feature from the second predicted sample speech, and training the speech conversion model according to the third loss value between the fifth sample timbre feature and the fourth sample timbre feature.

[0081] Here, an embodiment of the present application provides two methods for training a speech conversion model. Method one is to train the speech conversion model by minimizing the third loss value between the fifth sample timbre feature extracted from the output second predicted sample speech and the fourth sample timbre feature corresponding to the fourth sample speech.

[0082] It should be noted that the fourth sample timbre feature can be extracted from the fourth sample speech by the voiceprint extraction model, and the fourth sample timbre feature can be used as the real sample timbre feature of the fourth sample speech. The second predicted sample speech and the third sample speech have the same speech content, but different timbres. The timbre of the second predicted sample speech is the timbre corresponding to the fourth sample timbre feature, and the timbre of the third sample speech corresponds to the third sample timbre feature. Among them, the fourth sample timbre feature can be extracted from the fourth sample speech by the voiceprint extraction model as the real sample timbre feature. In the process of training the speech conversion model, the speech conversion model is trained with the goal of minimizing the loss between the timbre feature of the second predicted sample speech and the fourth sample timbre feature, so as to improve the timbre effect of the speech output by the speech conversion model.

[0083] In an optional embodiment, the speech conversion model is trained according to the following steps: obtaining multiple third sample speech and multiple fourth sample speech, and determining the fourth sample timbre feature corresponding to each of the fourth sample speech; inputting the fourth sample timbre feature and the third sample speech into the speech conversion model to be trained, and determining the third predicted sample speech; training the speech conversion model according to the fourth loss value between the third predicted sample speech and the fourth sample speech.

[0084] Here, the embodiments of the present application provide two methods for training the speech conversion model. The second method is to train the speech conversion model based on minimizing the fourth loss value between the output third predicted sample speech and the actual fourth sample speech of the fourth sample timbre feature corresponding to the third predicted sample speech as the training goal, so as to improve the timbre effect of the speech output by the speech conversion model.

[0085] Based on the same application concept, the embodiments of the present application also provide a speech synthesis device corresponding to the speech synthesis method provided in the above embodiments. Since the principle of solving the problem by the device in the embodiments of the present application is similar to the speech synthesis method in the above embodiments of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0086] like Figure 3 , Figure 4 As shown, Figure 3 This is one of the functional module diagrams of a speech synthesis device 300 provided in an embodiment of the present application. Figure 4 FIG2 is a second functional module diagram of a speech synthesis device 300 provided in an embodiment of the present application. Figure 3As shown, the speech synthesis device 300 includes: an acquisition module 310, used to acquire a target timbre label and a target text of a speech to be synthesized; a first determination module 320, used to determine a target timbre feature based on the target timbre label and a trained timbre representation generation model, and to determine a first target speech of a preset timbre based on the target text and a trained speech synthesis model; the preset timbre is the timbre of any first sample speech for training the speech synthesis model; a second determination module 330, used to determine a second target speech of a target timbre based on the target timbre feature, the first target speech and a trained speech conversion model.

[0087] In an optional embodiment, the target timbre label includes descriptive information of the target timbre of the speech to be synthesized from at least one of the following timbre dimensions: gender, age, voice, pitch, language, spoken language, personality, and occupation.

[0088] In an optional embodiment, if Figure 3 As shown, the target timbre feature is a timbre feature matrix; the first determination module 320 is specifically used to input the target timbre label into the trained timbre representation generation model to perform timbre data gradient learning to determine the timbre feature matrix.

[0089] In an optional embodiment, the timbre characterization generation model is one of the following models: a diffusion model, a convolutional neural network model, a deep learning model, a generative adversarial network model, a variational autoencoder, and a Gaussian mixture model.

[0090] In an optional embodiment, if Figure 4 As shown, the speech synthesis device 300 also includes a first training module 340; the first training module 340 is used to train the timbre characterization generation model according to the following steps: obtain multiple second sample speech, and determine the real sample timbre features and sample timbre labels corresponding to each second sample speech; for any second sample speech, input the sample timbre label of the second sample speech into the timbre characterization generation model to be trained to determine the predicted sample timbre features; based on the first loss value between the real sample timbre features and the predicted timbre features corresponding to each second sample speech, train the timbre characterization generation model.

[0091] In an optional embodiment, if Figure 4 As shown, the first training module 340 is also used to determine the real sample timbre features corresponding to the second sample speech according to the following steps: inputting the second sample speech into the trained voiceprint extraction model to extract timbre features, and determining the real sample timbre features corresponding to the second sample speech.

[0092] In an optional embodiment, if Figure 4 As shown, the first determination module 320 is specifically used to determine a phoneme sequence according to the text content of the target text; determine a text pronunciation feature according to the phoneme sequence; and determine a first target speech of the preset timbre according to the text pronunciation feature and the preset timbre feature of the preset timbre.

[0093] In an optional embodiment, if Figure 4 As shown, the speech synthesis device 300 also includes a second training module 350; the second training module 350 is used to train the speech synthesis model according to the following steps: obtain multiple sample texts and a first sample speech corresponding to each sample text, and extract a first sample timbre feature from the first sample speech; for any of the sample texts, input the sample text and the first sample timbre feature corresponding to the sample text into the speech synthesis model to be trained to determine a first predicted sample speech; based on a second loss value between the first sample speech corresponding to each of the sample texts and the first predicted sample speech, train the speech synthesis model.

[0094] In an optional embodiment, the first sample speech and the second sample speech are different.

[0095] In an optional embodiment, if Figure 4 As shown, the target timbre feature is a timbre feature matrix; the second determination module 330 is used to convert the timbre feature matrix into a timbre feature vector; the timbre feature vector and the first target speech are input into a trained speech conversion model for timbre conversion to determine a second target speech of the target timbre.

[0096] In an optional embodiment, if Figure 4 As shown, the speech synthesis device 300 also includes a third training module 360; the third training module 360 ​​is used to train the speech conversion model according to the following steps: obtain multiple third sample speech and multiple fourth sample speech, and determine the fourth sample timbre feature corresponding to each of the fourth sample speech; input the fourth sample timbre feature and the third sample speech into the speech conversion model to be trained to determine the second predicted sample speech; extract the fifth sample timbre feature from the second predicted sample speech, and train the speech conversion model according to the third loss value between the fifth sample timbre feature and the fourth sample timbre feature.

[0097] In an optional embodiment, if Figure 4As shown, the third training module 360 ​​is also used to train the speech conversion model according to the following steps: obtain multiple third sample speech and multiple fourth sample speech, and determine the fourth sample timbre feature corresponding to each of the fourth sample speech; input the fourth sample timbre feature and the third sample speech into the speech conversion model to be trained to determine the third predicted sample speech; train the speech conversion model according to the fourth loss value between the third predicted sample speech and the fourth sample speech.

[0098] In an embodiment of the present application, the target timbre features are determined by the second determination module 320 based on the acquired target timbre label and the trained timbre representation generation model, and the first target speech of the preset timbre is determined based on the acquired target text and the trained speech synthesis model; wherein the preset timbre is the timbre of any first sample speech of the training speech synthesis model; further, the second target speech of the target timbre can be determined by the third determination module 330 based on the target timbre features, the first target speech and the trained speech conversion model. In this way, the present application can generate a new timbre by decoupling the timbre representation generation model and the speech synthesis model, and using the speech conversion model for timbre conversion, which can meet the user's personalized needs for the timbre of synthesized speech in different scenarios.

[0099] Based on the same application concept, see Figure 5 As shown, it is a structural diagram of an electronic device 500 provided in an embodiment of the present application, including: a processor 510, a memory 520 and a bus 530, wherein the memory 520 stores machine-readable instructions executable by the processor 510, and when the electronic device 500 is running, the processor 510 communicates with the memory 520 through the bus 530, and the machine-readable instructions are executed by the processor 510 when running to perform the steps of the speech synthesis method described in any of the above embodiments.

[0100] Specifically, when the machine-readable instructions are executed by the processor 510, the following processing can be performed: obtaining a target timbre label and a target text of the speech to be synthesized; determining a target timbre feature based on the target timbre label and a trained timbre representation generation model, and determining a first target speech with a preset timbre based on the target text and a trained speech synthesis model; the preset timbre is the timbre of any first sample speech for training the speech synthesis model; and determining a second target speech with a target timbre based on the target timbre feature, the first target speech, and the trained speech conversion model.

[0101] In an optional embodiment, when the machine-readable instructions are executed by the processor 510, the following processing may be performed: inputting the target timbre label into the trained timbre representation generation model to perform timbre data gradient learning and determine the timbre feature matrix.

[0102] In an optional embodiment, the machine-readable instructions may perform the following processing when executed by the processor 510: obtaining multiple second sample speech, and determining the real sample timbre features and sample timbre labels corresponding to each second sample speech; for any second sample speech, inputting the sample timbre label of the second sample speech into the timbre representation generation model to be trained to determine the predicted sample timbre features; and training the timbre representation generation model based on the first loss value between the real sample timbre features and the predicted timbre features corresponding to each second sample speech.

[0103] In an optional embodiment, when the machine-readable instructions are executed by the processor 510, the following processing may be performed: inputting the second sample speech into a trained voiceprint extraction model to extract timbre features, and determining the real sample timbre features corresponding to the second sample speech.

[0104] In an optional embodiment, the machine-readable instructions may perform the following processing when executed by the processor 510: determine a phoneme sequence based on the text content of the target text; determine a text pronunciation feature based on the phoneme sequence; determine a first target speech of the preset timbre based on the text pronunciation feature and a preset timbre feature of the preset timbre.

[0105] In an optional embodiment, the machine-readable instructions may perform the following processing when executed by the processor 510: obtain multiple sample texts and a first sample speech corresponding to each sample text, and extract a first sample timbre feature from the first sample speech; for any of the sample texts, input the sample text and the first sample timbre feature corresponding to the sample text into the speech synthesis model to be trained to determine a first predicted sample speech; and train the speech synthesis model based on a second loss value between the first sample speech corresponding to each of the sample texts and the first predicted sample speech.

[0106] In an optional embodiment, when the machine-readable instructions are executed by the processor 510, the following processing can be performed: converting the timbre feature matrix into a timbre feature vector; inputting the timbre feature vector and the first target speech into a trained speech conversion model for timbre conversion, and determining a second target speech of the target timbre.

[0107] In an optional embodiment, the machine-readable instructions may perform the following processing when executed by the processor 510: obtaining a plurality of third sample speech and a plurality of fourth sample speech, and determining a fourth sample timbre feature corresponding to each of the fourth sample speech; inputting the fourth sample timbre feature and the third sample speech into the speech conversion model to be trained to determine a second predicted sample speech; extracting a fifth sample timbre feature from the second predicted sample speech, and training the speech conversion model according to a third loss value between the fifth sample timbre feature and the fourth sample timbre feature.

[0108] In an optional embodiment, the machine-readable instructions may perform the following processing when executed by the processor 510: obtaining a plurality of third sample speech and a plurality of fourth sample speech, and determining a fourth sample timbre feature corresponding to each of the fourth sample speech; inputting the fourth sample timbre feature and the third sample speech into the speech conversion model to be trained to determine a third predicted sample speech; and training the speech conversion model according to a fourth loss value between the third predicted sample speech and the fourth sample speech.

[0109] In an embodiment of the present application, the target timbre features are determined based on the acquired target timbre label and the trained timbre representation generation model, and the first target speech of the preset timbre is determined based on the acquired target text and the trained speech synthesis model; wherein the preset timbre is the timbre of any first sample speech of the trained speech synthesis model; further, the second target speech of the target timbre can be determined based on the target timbre features, the first target speech, and the trained speech conversion model. In this way, the present application can generate a new timbre by decoupling the timbre representation generation model and the speech synthesis model, and using the speech conversion model for timbre conversion, which can meet the user's personalized needs for the timbre of synthesized speech in different scenarios.

[0110] Based on the same application concept, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the speech synthesis method provided in the above embodiment are executed.

[0111] Specifically, when the computer program is executed by the processor, the following processing can be performed: obtaining a target timbre label and a target text of the speech to be synthesized; determining a target timbre feature based on the target timbre label and a trained timbre representation generation model, and determining a first target speech with a preset timbre based on the target text and a trained speech synthesis model; the preset timbre is the timbre of any first sample speech for training the speech synthesis model; and determining a second target speech with a target timbre based on the target timbre feature, the first target speech, and a trained speech conversion model.

[0112] In an optional embodiment, the computer program may perform the following processing when executed by the processor: input the target timbre label into the trained timbre representation generation model to perform timbre data gradient learning and determine the timbre feature matrix.

[0113] In an optional embodiment, the computer program can perform the following processing when executed by the processor: obtain multiple second sample speech, and determine the real sample timbre features and sample timbre labels corresponding to each second sample speech; for any second sample speech, input the sample timbre label of the second sample speech into the timbre representation generation model to be trained to determine the predicted sample timbre features; based on the first loss value between the real sample timbre features and the predicted timbre features corresponding to each second sample speech, train the timbre representation generation model.

[0114] In an optional embodiment, the computer program may perform the following processing when executed by the processor: input the second sample speech into a trained voiceprint extraction model to extract timbre features, and determine the real sample timbre features corresponding to the second sample speech.

[0115] In an optional embodiment, the computer program can perform the following processing when executed by the processor: determine a phoneme sequence based on the text content of the target text; determine a text pronunciation feature based on the phoneme sequence; determine a first target speech of the preset timbre based on the text pronunciation feature and a preset timbre feature of the preset timbre.

[0116] In an optional embodiment, the computer program can perform the following processing when executed by the processor: obtain multiple sample texts and a first sample speech corresponding to each sample text, and extract a first sample timbre feature from the first sample speech; for any of the sample texts, input the sample text and the first sample timbre feature corresponding to the sample text into the speech synthesis model to be trained to determine a first predicted sample speech; and train the speech synthesis model based on a second loss value between the first sample speech corresponding to each of the sample texts and the first predicted sample speech.

[0117] In an optional embodiment, the computer program can perform the following processing when executed by the processor: converting the timbre feature matrix into a timbre feature vector; inputting the timbre feature vector and the first target speech into a trained speech conversion model for timbre conversion, and determining a second target speech of the target timbre.

[0118] In an optional embodiment, the computer program can perform the following processing when executed by the processor: obtain multiple third sample speech and multiple fourth sample speech, and determine the fourth sample timbre feature corresponding to each of the fourth sample speech; input the fourth sample timbre feature and the third sample speech into the speech conversion model to be trained to determine the second predicted sample speech; extract the fifth sample timbre feature from the second predicted sample speech, and train the speech conversion model according to the third loss value between the fifth sample timbre feature and the fourth sample timbre feature.

[0119] In an optional embodiment, the computer program can perform the following processing when executed by the processor: obtain multiple third sample speech and multiple fourth sample speech, and determine the fourth sample timbre feature corresponding to each of the fourth sample speech; input the fourth sample timbre feature and the third sample speech into the speech conversion model to be trained to determine the third predicted sample speech; train the speech conversion model according to the fourth loss value between the third predicted sample speech and the fourth sample speech.

[0120] Here, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, the above-mentioned speech synthesis method can be executed. By decoupling the timbre representation generation model and the speech synthesis model, and using the speech conversion model to perform timbre conversion, a new timbre can be generated, which can meet the user's personalized needs for the timbre of synthesized speech in different scenarios.

[0121] In the embodiment of the present application, the computer program can also execute other machine-readable instructions when run by the processor to execute other methods described in the embodiment. For the specific execution method steps and principles, please refer to the description of the embodiment, which will not be repeated here.

[0122] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0123] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0124] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0125] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0126] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0127] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Obtain the target timbre label and target text of the speech to be synthesized; Determine the target timbre feature based on the target timbre label and the trained timbre representation generation model, and determine the first target speech with a preset timbre based on the target text and the trained speech synthesis model; the preset timbre is the timbre of any first sample speech for training the speech synthesis model; Based on the target timbre feature, the first target voice and the trained voice conversion model, a second target voice with a target timbre is determined.

2. The method according to claim 1, characterized in that The target timbre label includes descriptive information of the target timbre of the speech to be synthesized from at least one of the following timbre dimensions: gender, age, voice, pitch, language, spoken language, personality, and occupation.

3. The method according to claim 1, characterized in that The target timbre feature is a timbre feature matrix; the step of determining the target timbre feature based on the target timbre label and the trained timbre representation generation model includes: The target timbre label is input into the trained timbre characterization generation model to perform timbre data gradient learning to determine the timbre feature matrix.

4. The method according to claim 1, characterized in that: The timbre characterization generation model is one of the following models: a diffusion model, a convolutional neural network model, a deep learning model, a generative adversarial network model, a variational autoencoder, and a Gaussian mixture model.

5. The method according to claim 1, characterized in that The timbre characterization generation model is trained according to the following steps: Acquire multiple second sample voices, and determine a real sample timbre feature and a sample timbre label corresponding to each second sample voice; For any of the second sample speech, inputting the sample timbre label of the second sample speech into the timbre representation generation model to be trained to determine the predicted sample timbre feature; The timbre representation generation model is trained based on a first loss value between the real sample timbre feature and the predicted timbre feature corresponding to each of the second sample voices.

6. The method according to claim 5, characterized in that Determine the real sample timbre feature corresponding to the second sample speech according to the following steps: The second sample speech is input into a trained voiceprint extraction model to extract timbre features, and the real sample timbre features corresponding to the second sample speech are determined.

7. The method according to claim 1, characterized in that The step of determining the first target speech with a preset timbre based on the target text and the trained speech synthesis model includes: Determining a phoneme sequence according to the text content of the target text; Determining text pronunciation features according to the phoneme sequence; A first target voice of the preset timbre is determined according to the text pronunciation feature and the preset timbre feature of the preset timbre.

8. The method according to claim 1, characterized in that The speech synthesis model is trained according to the following steps: Acquire a plurality of sample texts and a first sample speech corresponding to each sample text, and extract a first sample timbre feature from the first sample speech; For any of the sample texts, input the sample text and the first sample timbre feature corresponding to the sample text into the speech synthesis model to be trained to determine a first predicted sample speech; The speech synthesis model is trained based on a second loss value between the first sample speech and the first predicted sample speech corresponding to each of the sample texts.

9. The method according to claim 5, characterized in that The first sample voice and the second sample voice are different.

10. The method according to claim 1, characterized in that The target timbre feature is a timbre feature matrix; and determining the second target voice of the target timbre based on the target timbre feature, the first target voice and the trained voice conversion model includes: Converting the timbre feature matrix into a timbre feature vector; The timbre feature vector and the first target speech are input into a trained speech conversion model for timbre conversion to determine a second target speech of the target timbre.

11. The method according to claim 1, characterized in that: The speech conversion model is trained according to the following steps: Acquire a plurality of third sample voices and a plurality of fourth sample voices, and determine a fourth sample timbre feature corresponding to each of the fourth sample voices; Inputting the fourth sample timbre feature and the third sample speech into the speech conversion model to be trained to determine a second predicted sample speech; A fifth sample timbre feature is extracted from the second predicted sample speech, and the speech conversion model is trained according to a third loss value between the fifth sample timbre feature and the fourth sample timbre feature.

12. The method according to claim 1, characterized in that The speech conversion model is trained according to the following steps: Acquire a plurality of third sample voices and a plurality of fourth sample voices, and determine a fourth sample timbre feature corresponding to each of the fourth sample voices; Inputting the fourth sample timbre feature and the third sample speech into the speech conversion model to be trained to determine a third predicted sample speech; The speech conversion model is trained according to a fourth loss value between the third predicted sample speech and the fourth sample speech.

13. A speech synthesis device, characterized in that: The device comprises: An acquisition module is used to acquire a target timbre label and a target text of the speech to be synthesized; A first determination module is used to determine a target timbre feature based on the target timbre label and the trained timbre representation generation model, and to determine a first target speech with a preset timbre based on the target text and the trained speech synthesis model; the preset timbre is the timbre of any first sample speech for training the speech synthesis model; The second determination module is used to determine a second target voice with a target timbre based on the target timbre feature, the first target voice and a trained voice conversion model.

14. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the speech synthesis method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the speech synthesis method according to any one of claims 1 to 12 are executed.

Images