Speech synthesis method and system, model training method and system, equipment, medium and program product

Abstract

The invention provides a speech synthesis method and system, a model training method and system, equipment, a medium and a program product. The speech synthesis method comprises the following steps: acquiring to-be-processed speech data, a style description text for describing a speaking style and a target character; extracting voice features based on the voice data through a voice synthesis model, and determining speaking style features based on the style description text; based on the voice features and the speaking style features, converting the target characters into target voice; and outputting the target voice. The speech synthesis flexibility can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a speech synthesis, model training method, system, device, medium and program product. Background Art

[0002] With the development of science and technology, speech synthesis technology is now able to generate high-quality speech. However, existing speech synthesis technologies, such as those based on the zero-sample text-to-speech (TTS) model, have the problem of poor flexibility. For example, using the zero-sample TTS model for speech synthesis can only imitate the speaker's timbre, but cannot adjust the voice style. How to improve the flexibility of speech synthesis to meet users' needs for speech synthesis is an urgent problem to be solved. Summary of the invention

[0003] The present application provides a speech synthesis, model training method, system, device, medium and program product, which can improve the flexibility of speech synthesis.

[0004] In a first aspect, the present application provides a speech synthesis method, the method comprising:

[0005] Acquire speech data to be processed, style description text for describing the speaking style, and target text;

[0006] Extracting sound features based on the speech data through a speech synthesis model, and determining speaking style features based on the style description text; converting the target text into a target speech based on the sound features and the speaking style features;

[0007] The target speech is output.

[0008] Optionally, the speech synthesis model includes: a feature extraction module and a speech conversion module, and the feature extraction module includes: a sound encoding sub-model, a style encoding sub-model and a text content encoding sub-model;

[0009] Wherein, the sound coding sub-model is used to extract features from the speech data to obtain the sound features;

[0010] The style encoding sub-model is used to extract features from the style description text to obtain the speaking style features;

[0011] The text content encoding sub-model is used to extract features of the target text to obtain content features of the target text;

[0012] The speech conversion module is used to synthesize the target speech based on the sound feature, the speaking style feature and the content feature.

[0013] Optionally, the extracting features from the style description text to obtain the speaking style features includes:

[0014] Extracting semantic features from the style description text to obtain a target semantic vector; the target semantic vector is used to represent the semantics of the style description text;

[0015] The target semantic vector is converted into a target style feature distribution; the target style feature distribution is used to characterize the speaking style feature.

[0016] Optionally, converting the target semantic vector into a target style feature distribution includes:

[0017] Converting the target semantic vector into a plurality of initial style feature distributions;

[0018] Based on the multiple initial style feature distributions, the target style feature distribution is obtained.

[0019] Optionally, the extracting semantic features of the style description text to obtain a target semantic vector includes:

[0020] The style description text and the preset mark are processed to obtain a word vector corresponding to each word in the style description text and a semantic vector corresponding to the preset mark; the semantic vector corresponding to the preset mark is the target semantic vector.

[0021] Optionally, synthesizing the target speech based on the sound feature, the speaking style feature, and the content feature includes:

[0022] Fusing the speaking style feature with the content feature of the target text to obtain a fused feature;

[0023] The fusion feature and the sound feature are decoded in parallel to obtain the target speech.

[0024] In a second aspect, the present application provides a model training method, the method comprising:

[0025] Acquire a first sample data set; the first sample data set includes: sample speech data, a first sample style description text for describing a speaking style, and sample text;

[0026] The first preset model is trained using the first sample data set to obtain the speech synthesis model as described in any one of claims 1-6.

[0027] Optionally, the speech synthesis model includes: a feature extraction module and a speech conversion module, the first preset model includes: the pre-trained feature extraction module and a first preset sub-model to be trained; the feature extraction module includes: a sound encoding sub-model, a style encoding sub-model and a text content encoding sub-model;

[0028] The using the first sample data set to train a first preset model to obtain the speech synthesis model includes:

[0029] Using the sound coding sub-model, extracting features from the sample speech data to obtain sample sound features;

[0030] Using the style encoding sub-model, extracting features from the first sample style description text to obtain a first sample speaking style feature;

[0031] Using the text content encoding sub-model, extracting features of the sample text to obtain sample content features of the sample text;

[0032] The first preset sub-model is trained using the sample sound feature, the first sample speaking style feature, and the sample content feature to obtain the speech conversion module.

[0033] Optionally, before using the style encoding sub-model to extract features from the first sample style description text to obtain the first sample speaking style features, the method further includes:

[0034] Acquire a second sample data set; the second sample data set includes: a second sample style description text, and a sample style feature distribution corresponding to the second sample style description text; the sample style feature distribution is used to characterize the second sample speaking style feature;

[0035] The second preset model is trained using the second sample data set to obtain the style encoding sub-model.

[0036] Optionally, the using the second sample data set to train a second preset model to obtain the style encoding sub-model includes:

[0037] Extracting semantic features from the second sample style description text to obtain a sample semantic vector; the sample semantic vector is used to represent the semantics of the second sample style description text;

[0038] Converting the sample semantic vector into a plurality of initial style feature distributions;

[0039] Based on the multiple initial style feature distributions and the sample style feature distributions, the second preset model is trained to obtain the style encoding sub-model.

[0040] Optionally, the training of the second preset model based on the multiple initial style feature distributions and the sample style feature distribution to obtain the style encoding sub-model includes:

[0041] Adding noise disturbance to the multiple initial style feature distributions respectively to obtain multiple noise-perturbed style feature distributions;

[0042] The second preset model is trained using the style feature distributions after the multiple noise disturbances and the sample style feature distribution to obtain the style encoding sub-model.

[0043] Optionally, the using the style feature distributions after the multiple noise disturbances and the sample style feature distributions to train the second preset model to obtain the style encoding sub-model includes:

[0044] Repeat the following steps until the style encoding sub-model is obtained:

[0045] The second preset model is used to obtain a prediction weight, a prediction mean and a prediction variance by using the weighted style feature distributions of the plurality of noise disturbances; the sample style feature distribution is characterized by a sample mean and a sample variance;

[0046] Calculating a loss function based on the predicted mean and the predicted variance and the sample mean and the sample variance;

[0047] Based on the loss function and the prediction weight, the second preset model is trained in the next round; the preset weight is the weight of the style feature distribution after the noise disturbance during the next round of training of the second preset model.

[0048] In a third aspect, the present application provides a speech synthesis system, the system comprising: a client, and a server; the server is deployed with a speech synthesis model as described in any one of the first aspects;

[0049] The client is used to receive the voice data to be processed, the style description text for describing the speaking style, and the target text;

[0050] The server is used to respond to the text-to-speech instruction, extract sound features based on the speech data through the speech synthesis model, and determine the speaking style features based on the style description text; convert the target text into the target speech based on the sound features and the speaking style features; and output the target speech through the client.

[0051] Optionally, determining the speaking style feature based on the style description text includes:

[0052] Extracting semantic features from the style description text to obtain a target semantic vector; the target semantic vector is used to represent the semantics of the style description text;

[0053] Converting the target semantic vector into a plurality of initial style feature distributions;

[0054] Based on the multiple initial style feature distributions, a target style feature distribution is obtained, and the target style feature distribution is used to characterize the speaking style feature.

[0055] In a fourth aspect, the present application provides an electronic device, including:

[0056] at least one processor; and a memory communicatively coupled to the at least one processor;

[0057] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to execute the method described in any one of the first aspect and / or the second aspect.

[0058] In a fifth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in any one of the first aspect and / or the second aspect is implemented.

[0059] In a sixth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method as described in any one of the first aspect and / or the second aspect.

[0060] The speech synthesis, model training method, system, equipment, medium and program product provided by the present application can extract features of the speech data to be processed through the speech synthesis model to obtain sound features, and determine the speaking style features based on the style description text used to describe the speaking style. Through the above method, the subsequent speech synthesis model can convert the target text into the target speech based on the sound features and speaking style features. Because the target speech is determined based on the sound features and speaking style features, the sound of the target speech can meet the sound features (i.e., the sound of the speech data is imitated), and the speaking style in the target speech can meet the speaking style features, that is to say, different speech data and different style description texts can make the sound and style of the target speech obtained by the target text conversion different. Compared with the existing speech synthesis method that can only imitate the timbre of the speaker, the present application can imitate the sound of the speech data while realizing the controllable style of the target speech, thereby improving the flexibility of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0062] Figure 1 A flowchart of a speech synthesis method provided in this application;

[0063] Figure 2 A schematic diagram of the architecture of a speech synthesis model provided for this application;

[0064] Figure 3 A flowchart of a model training method provided in this application;

[0065] Figure 4 A schematic diagram of the architecture of another speech synthesis model provided for this application;

[0066] Figure 5 A schematic diagram of the structure of a style feature encoder provided in this application;

[0067] Figure 6 A schematic diagram of the training process of a speech conversion module provided in this application;

[0068] Figure 7 A schematic diagram showing the differences between the speech synthesis system provided in this application and other speech synthesis systems;

[0069] Figure 8A flowchart of a method for performing speech synthesis through a voice assistant provided in this application;

[0070] Fig. 9 A flow chart of a voice navigation method provided in this application;

[0071] Fig.10 A flow chart of a method for voice output provided in this application;

[0072] Fig.11 A schematic diagram of the structure of a speech synthesis device provided in this application;

[0073] Fig.12 A schematic diagram of the structure of a model training device provided in this application;

[0074] Fig.13 A schematic diagram of the hardware structure of an electronic device provided in this application.

[0075] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0077] The following first explains some of the noun concepts involved in this application:

[0078] Zero-sample TTS model: "Zero-sample" means that the TTS model can generate the speech of a specified speaker without the need for the speaker's speech data as a sample to train the TTS model.

[0079] With the development of science and technology, speech synthesis technology has made great progress. For example, existing speech synthesis systems can generate high-quality speech that is close to real human voices. However, existing speech synthesis systems all have the problem of poor flexibility. For example, the existing zero-sample TTS model can only imitate the speaker's timbre (or copy / clone the speaker's voice) and cannot control the style of the synthesized speech. Among them, the style of speech can include, for example: the speed of speech, emotional expression, tone, etc.

[0080] How to improve the flexibility of speech synthesis to meet users' demand for highly flexible speech synthesis is an urgent problem to be solved.

[0081] Considering the above problems existing in the existing speech synthesis methods, the present application proposes a method for speech synthesis based on the speaker's speaking style characteristics and sound characteristics (such as timbre characteristics and rhythm characteristics). Compared with the existing speech synthesis methods that can only imitate the speaker's timbre, the speech synthesis method provided by the present application can synthesize speech of different styles and different sounds as the speaker's speaking style characteristics and sound characteristics change, thereby improving the flexibility of speech synthesis.

[0082] Optionally, the execution entity of the speech synthesis method provided in the present application may be, for example, any electronic device with processing capabilities, such as a terminal or a server.

[0083] The technical solution of the present application is described in detail below in conjunction with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0084] Figure 1 The following is a flow chart of a speech synthesis method provided in this application. Figure 1 As shown, the method may include the following steps:

[0085] S101, obtaining speech data to be processed, style description text for describing a speaking style, and target text.

[0086] Exemplarily, the voice data may be the voice data of the target speaker, or the voice data may be any segment of voice data. Taking the voice data as the voice data of the target speaker as an example, the target speaker may be, for example, any real person, or a virtual person, etc., and the present application does not limit the target speaker. Optionally, the voice data may be the original unprocessed voice data of the target speaker. Alternatively, the voice data may also be, for example, voice data after pre-processing such as noise reduction of the original voice of the target speaker. Optionally, the above-mentioned voice data may be, for example, a segment of voice data of any length.

[0087] Optionally, the above-mentioned speaking style may refer to the unique way and characteristics displayed by the speaker during the speaking process. Exemplarily, the speaking style may include, for example: speaking speed, emotional expression (emotion), gender, pitch (pitch), volume (volume), etc. The style description text is a text used to describe the speaking style. It should be understood that the present application does not limit the language of the above-mentioned voice data, style description text and target text, and the language of the voice data, style description text and target text may be the same or different. The above-mentioned style description text may include text, which is used to describe the speaker's speaking style. It should be understood that the present application does not limit the number of texts included in the style description text.

[0088] Optionally, the target text is text to be converted into speech. Exemplarily, the target text can be at least one text. In some embodiments, the target text can refer to all texts in the target document (or text selected by the user, etc.). Alternatively, the target text can also be text to be converted into speech input by the user. In addition, it should be understood that the application does not limit the number of the target text. Exemplarily, the number of the target text can be related to the data processing capacity of the electronic device, etc.

[0089] Optionally, the electronic device may obtain the above-mentioned voice data, style description text and target text, for example, through an application programming interface (Application Programming Interface, API) or a graphical user interface (Graphical User Interface, GUI). It should be understood that the implementation method of the electronic device obtaining the above-mentioned voice data, style description text and target text may be the same or different, and the present application does not limit this. For example, the electronic device may directly obtain the above-mentioned style description text and target text. Alternatively, the electronic device may also first obtain a voice used to describe the speaker's speaking style, and then convert the voice into the above-mentioned style description text. Alternatively, the electronic device may also first obtain the voice data of reading the above-mentioned target text, and then convert the voice into the above-mentioned target text. In some embodiments, the above-mentioned target text may also be the target text obtained by the electronic device converting the above-mentioned voice data into text.

[0090] Alternatively, taking the electronic device as a server of a speech synthesis system as an example, the server can obtain the above-mentioned speech data, style description text and target text through a client of the speech synthesis system.

[0091] S102: extracting sound features based on the speech data through a speech synthesis model, and determining speaking style features based on the style description text.

[0092] Optionally, the speech synthesis model may be pre-configured in the electronic device, for example. In some embodiments, the speech synthesis model may be a trained machine learning model, such as a neural network model. The speech synthesis model may be used to extract sound features from speech data to obtain the sound features, and to extract speaking style features from style description text to obtain the speaking style features.

[0093] Exemplarily, the sound feature may refer to the unique properties and characteristics of the sound emitted by the voice data. Exemplarily, the sound feature may include, for example, timbre feature, rhythm feature, etc.

[0094] It should be understood that the present application does not determine the order of extracting sound features based on speech data through a speech synthesis model and determining speaking style features based on a style description text.

[0095] S103: Convert the target text into target speech through a speech synthesis model based on sound features and speaking style features.

[0096] Optionally, the above-mentioned sound features may, for example, represent the timbre and rhythm of the speaker's speech, and the above-mentioned speaking style features may, for example, represent the speaking speed, emotion, etc. Therefore, based on the sound features and speaking style features, the target text is converted into the target speech, so that the sound features of the target speech can match the sound, and the style of the target speech can match the style. For example, if the speaker speaks fast, the speaking speed of the target speech is also fast.

[0097] S104: Output the target speech.

[0098] For example, if the electronic device is equipped with a speech output device (such as a speaker, etc.), the electronic device can directly output the target speech through the semantic output device. If the electronic device is a server of a speech synthesis system, the server can convert the target text into a target speech and then send the target speech to a client of the speech synthesis system, so as to output the target speech through the client.

[0099] In the present embodiment, through the speech synthesis model, the speech data to be processed can be feature extracted to obtain sound features, and the speaking style features can be determined based on the style description text used to describe the speaking style. Through the above method, the subsequent speech synthesis model can convert the target text into the target speech based on the sound features and the speaking style features. Because the target speech is determined based on the sound features and the speaking style features, the sound of the target speech can meet the sound features (i.e., the sound of the speech data is imitated), and the speaking style in the target speech can meet the speaking style features, that is to say, different speech data and different style description texts can make the sound and style of the target speech obtained by the target text conversion different. Compared with the existing speech synthesis method that can only imitate the timbre, the present application can imitate the sound of the speech data while realizing the controllable style of the target speech, thereby improving the flexibility of speech synthesis.

[0100] The above speech synthesis model is described in detail below:

[0101] Figure 2 This is a schematic diagram of the architecture of a speech synthesis model provided in this application. Figure 2 As shown, as a possible implementation, the speech synthesis model may include, for example, a feature extraction module and a speech conversion module. The feature extraction module may include, for example, a sound encoding sub-model, a style encoding sub-model and a text content encoding sub-model.

[0102] The above-mentioned sound coding sub-model, for example, can be used to extract features from the above-mentioned speech data to obtain the sound features.

[0103] Exemplarily, the sound coding submodel may include, for example, an audio encoder for encoding speech data, and a timbre extractor (or timbre encoder) for extracting sound features from the encoded audio. The audio encoder may be used to encode the speech data into data recognizable by a computer to obtain the encoded audio. The audio encoder may refer to any existing audio encoder, which will not be described in detail here. Optionally, the timbre extractor may be, for example, a pre-trained neural network model. The pre-trained neural network model may be used to extract sound features from the encoded audio to obtain the sound features corresponding to the speech data. The timbre extractor may also refer to any existing timbre extractor, such as a multi-layer convolutional neural network (CNN), or a multi-reference tone encoder (MRTE).

[0104] The style encoding sub-model mentioned above, for example, can be used to extract features from the style description text to obtain the speaking style features.

[0105] Optionally, the style encoding submodel may, for example, perform semantic extraction on the above-mentioned style description text to obtain the semantics of the style description text. Then the style encoding submodel may, for example, use the semantics as speaking style features. Alternatively, the style encoding submodel may, for example, further convert the semantics into feature distributions to obtain the speaking style features. Optionally, the style encoding submodel may, for example, be a pre-trained neural network model, or any existing model that can extract semantic information from a text. For example, the style encoding submodel may process the style description text based on predefined conversion rules (such as word frequency statistics, etc.) to determine the semantics of each word in the style description text.

[0106] The above-mentioned text content encoding sub-model, for example, can be used to extract features of the above-mentioned target text to obtain content features of the target text.

[0107] Optionally, the text content encoding submodel may include, for example, a text encoder. Exemplarily, the electronic device may, for example, first convert the target text into a word embedding through the text encoder. Then, the electronic device may, for example, positionally encode the word embedding of the target text through the text encoder to determine the order information of the words of the target text, that is, to add the position information to the word embedding. Then, the electronic device may, for example, also use a multi-head self-attention mechanism to focus on other words in the input sequence when encoding each word to improve the accuracy of the content feature extraction of the target text. Through the text encoder, the words in the target text may be converted into a high-dimensional vector for representing contextual information and semantic features, and the high-dimensional vector may be used as the content feature of the target text. The text encoder may, for example, refer to any existing text encoder, such as a Transformer model, etc. In some embodiments, the text encoder may also be referred to as a content encoder.

[0108] The above-mentioned sound coding sub-model, style coding sub-model and text content coding sub-model can be independent of each other and do not interfere with each other. If there is interference between the voice data, the style description text and the target text, for example, if the speech speed of the voice data is very slow, and the style description text describes the speaker's speaking style with a fast speaking speed, it may cause the voice data to interfere with the style description text, thereby reducing the accuracy of the speaking style features. Therefore, by processing the voice data, style description text and target text respectively through the above-mentioned independent and non-interfering decoupled sound coding sub-model, style coding sub-model and text content coding sub-model, the interference between the voice data, style description text and target text is reduced, thereby improving the accuracy of the sound features, speaking style features and content features.

[0109] The speech conversion module can be used to synthesize the target speech based on the above-mentioned sound features, speaking style features and content features.

[0110] In some embodiments, the speech conversion module may first fuse the speech style feature and the content feature to obtain a fused feature, so that the content feature carries the speech style. Then, the speech conversion module may perform speech synthesis operations such as upsampling decoding based on the fused feature and the sound feature to obtain the target speech.

[0111] In this embodiment, the speech synthesis model can process the speech data, style description text and target text respectively through the sound coding sub-model, the style coding sub-model and the text content coding sub-model, so as to realize decoupled coding (that is, using different coding sub-models to encode the speech data, style description text and target text separately) in the coding stage of the speech data, style description text and target text, reduce the interference between the speech data, style description text and target text, realize independent coding, improve the accuracy of the sound features, speaking style features and content features, and then improve the accuracy of the subsequent target speech synthesis based on the sound features, speaking style features and content features, and further improve the user experience.

[0112] The following is a detailed description of how the style encoding sub-model extracts features from the style description text to obtain the speaking style features:

[0113] As a possible implementation manner, the style encoding sub-model may, for example, first perform semantic feature extraction on the style description text to obtain a target semantic vector.

[0114] The target semantic vector can be used to represent the semantics of the style description text. The semantics of the style description text refers to the meaning or concept expressed by a word, phrase, sentence or larger text unit in the style description text.

[0115] Optionally, the target semantic vector may be, for example, a vector, or may be multiple vectors. Exemplarily, taking the target semantic vector as a vector as an example, the style encoding submodel may, for example, perform semantic feature extraction on the style description text to obtain a word vector that can be used to characterize the summary semantic information of the style description text as the target semantic vector. Taking the target semantic vector including multiple vectors as an example, the style encoding submodel may, for example, perform semantic feature extraction on the style description text to obtain a word vector corresponding to each word in the style description text. The electronic device may, for example, use the word vector corresponding to each word in the style description text as the target semantic vector.

[0116] Optionally, the style encoding submodel can, for example, extract semantic features from the style description text through any existing semantic extraction model to obtain a target semantic vector. Optionally, the semantic extraction model can be, for example, a large language model (LLMs) or a small model. Taking the semantic extraction model as a large language model as an example, the semantic extraction model can be, for example, a pre-trained BERT (an existing word encoding model, the full name in English is: Bidirectional Encoder Representations from Transformers) model, or a GPT (a generative model, the full name in English is: Generative Pre-trained Transformer) model, etc. In addition, the semantic extraction model can also be, for example, an ELMo (a model that can learn the context information of the style description text for semantic extraction, the full name in English is: Embeddings from Language Models) model, etc. In some embodiments, whether a large language model or a small language model is deployed in the electronic device as the above-mentioned semantic extraction model can be related to factors such as the computing power of the electronic device or the real-time requirements of the target speech synthesis.

[0117] Taking into account that style description texts with different contents may describe the same style, for example, style description text 1 "He speaks very fast", style description text 2 "He speaks very fast", style description text 3 "He speaks a little fast", these three style description texts all represent one style, namely "fast speaking speed". In order to adapt to style description texts with different contents and obtain the same semantics of style description texts with different contents, therefore, in some embodiments, the style encoding sub-model can also obtain a target semantic vector based on the above-mentioned style description texts and preset tags.

[0118] For example, the style encoding sub-model can process the style description text and the preset tag to obtain the word vector corresponding to each word in the style description text and the semantic vector corresponding to the preset tag. The semantic vector corresponding to the preset tag can be the target semantic vector.

[0119] Among them, the above-mentioned preset mark can represent the summary information of the style description text. In some embodiments, the preset mark can be added at the beginning of the style description text, for example. Then, the style encoding submodel can process the style description text and the preset mark. Exemplarily, the preset mark can be, for example, a classification mark (ClassificationToken, CLS, used to represent a special mark in a text classification task), or any other mark that does not exist in the encoding dictionary. In some embodiments, the CLS mark can be expressed as [CLS], which can be added to the beginning of the style description text sequence to represent the summary information of the style description text.

[0120] Taking the style encoding sub-model as an example, extracting semantic features from the style description text through any existing semantic extraction model to obtain a target semantic vector, the style encoding sub-model can input the style description text and the preset tag into the semantic extraction model. The semantic extraction model can output the word vector corresponding to each word in the style description text, and the semantic vector corresponding to the preset tag. The semantic vector corresponding to the preset tag is a vector. The vector can represent the summary semantic information of the entire style description text.

[0121] Through the above method, by taking the semantic vector corresponding to the preset mark as the target semantic vector, the semantics of the style description text is represented by a vector, and the vector integrates the semantics of the summary information of the style description text, so that when the specific content of the style description text is different but the summary semantics is the same (that is, the expression is the same style), the semantics of the style description text is represented by a target semantic vector, which simplifies the representation of the semantics of the style description text and improves the efficiency of subsequent target speech synthesis based on the speaking style features.

[0122] Optionally, the representation form of the above-mentioned speaking style features can be, for example, a vector or a distribution, which is not limited in this application. Because the distribution of data is a description of features such as data regularity and trends, the speaking style features can be characterized by distribution. Taking the representation form of speaking style features as distribution as an example, after obtaining the target semantic vector, the style encoding submodel can convert the target semantic vector into a target style feature distribution. Among them, the target style feature distribution is used to characterize the above-mentioned speaking style features.

[0123] In some embodiments, taking the target semantic vector as an example, the style encoding sub-model can convert each target semantic vector into a corresponding style feature distribution, that is, the style encoding sub-model can obtain multiple style feature distributions. Then, the style encoding sub-model can calculate the weighted sum of the multiple style feature distributions as the target style feature distribution.

[0124] In some embodiments, taking the target semantic vector as one for example, the style encoding sub-model may directly convert the target semantic vector into a style feature distribution, and use the style feature distribution as the target style feature distribution.

[0125] Alternatively, in some embodiments, the style encoding sub-model may also convert the target semantic vector into a plurality of initial style feature distributions, wherein the number of the target semantic vectors may be multiple or one.

[0126] Optionally, the method of converting the target semantic vector into multiple initial style feature distributions can refer to any existing method of converting a vector into a distribution, for example, the target vector can be converted into multiple initial style feature distributions through a Gaussian Mixture Model (GMM), and the multiple initial style feature distributions can be represented by parameters of the Gaussian distribution (such as mean and covariance matrix). Alternatively, the style encoding sub-model can also convert the target semantic vector into multiple probability density functions through a kernel density estimation (KDE) algorithm, for example, and the probability density function can be used to characterize the initial style feature distribution.

[0127] It should be understood that the present application does not limit the number of multiple initial style feature distributions obtained by converting the target semantic vector. For example, the electronic device can convert a target semantic vector into four initial style feature distributions. Exemplarily, the number of the initial style feature distributions can be pre-configured in the electronic device. The electronic device can convert the above target semantic vector into multiple initial style feature distributions based on the number of the initial style feature distributions.

[0128] Then, the style encoding sub-model may, for example, obtain a target style feature distribution based on the multiple initial style feature distributions.

[0129] Exemplarily, the style encoding sub-model may calculate a weighted sum of the multiple initial style feature distributions as the target style feature distribution. The weights of the initial style feature distributions may be pre-stored in the electronic device. Alternatively, the style encoding sub-model may be a pre-trained machine learning model that can obtain a target style feature distribution based on multiple initial style feature distributions.

[0130] By converting the target semantic vector into multiple initial style feature distributions, it is in line with the one-to-many scenario in actual scenarios (i.e., a style description text may actually correspond to multiple styles. For example, the style description text "she speaks very fast", then there are multiple possibilities for the degree of fast speaking speed, such as first-level fast, second-level fast, third-level fast, etc., which can all be called fast speaking speed). By converting the target semantic vector into multiple initial style feature distributions, it is achieved to obtain multiple initial style feature distributions based on one style description text, that is, multiple styles. Therefore, through the above method, the determination process of the target style feature distribution is more in line with the actual judgment process of the speaking style, and the diversity and accuracy of the speaking style represented by the target style feature distribution are improved.

[0131] The following is a detailed description of how the speech conversion module synthesizes the target speech based on the above sound features, speaking style features and content features:

[0132] As a possible implementation, the speech conversion module may, for example, fuse the above-mentioned speaking style features with the content features of the target text to obtain fused features, and then decode the fused features and the sound features in parallel to obtain the above-mentioned target speech.

[0133] Here, fusion may refer to combining the speaking style feature and the content feature of the target text into one feature.

[0134] Exemplarily, the speech conversion module may, for example, first perform a preliminary fusion of the speech style features and the content features of the target text through a cross attention mechanism to obtain a preliminary fusion feature. Then, the speech conversion module may, for example, adjust the dimension of the preliminary fusion feature through a variable information adapter to obtain a preliminary fusion feature after the dimension is adjusted. Then, the speech conversion module may, for example, perform a cross attention mechanism to fuse the speech style features and the preliminary fusion feature after the dimension is adjusted again to obtain the fusion feature.

[0135] Alternatively, the speech conversion module may, for example, fuse the speaking style features and the content features of the target text through a cross-attention mechanism, and use the result obtained as the fused feature.

[0136] The above-mentioned parallel decoding may refer to a non-autoregressive decoding operation, that is, the speech conversion module can process multiple features at the same time. Compared with sequential decoding, parallel decoding can speed up the efficiency of speech synthesis by processing multiple features at the same time. In some embodiments, the speech conversion module implements parallel decoding, for example, by referring to the decoding method of the existing generative model.

[0137] Through the above method, the speaking style features and the content features of the target text can be fused, so that the speaking style features are carried in the content features of the target text, so that in the decoding process, the target speech that conforms to the speaking style described in the style description text can be generated based on the speaking style features. By decoding the fusion features and the sound features in parallel, because the fusion features fuse the speaking style features, the decoding result, that is, the target speech, is determined based on the speaking style features and the sound features, thereby ensuring that the target speech conforms to the speaking style described in the style description text and has the sound features provided by the speech data. In addition, compared with sequential decoding, by synthesizing the target speech in a parallel decoding manner, the speech conversion module can process the fusion features and the sound features at the same time, thereby improving the efficiency of decoding the target speech based on the fusion features and the sound features.

[0138] As a possible implementation, the speech synthesis model may be, for example, a pre-trained machine learning model. Taking the speech synthesis model as a pre-trained machine learning model as an example, the training method of the speech synthesis model is exemplarily described below:

[0139] First, it should be understood that the execution subject of the training method of the speech synthesis model can also be any electronic device with processing capabilities such as a terminal or a server. Optionally, the electronic device that executes the model training method and the execution subject of the speech synthesis method provided in this application can be the same electronic device or different electronic devices, and this application does not limit this.

[0140] Figure 3 A flow chart of a model training method provided in this application. Figure 3 As shown, the method may include the following steps:

[0141] S201: Obtain a first sample data set.

[0142] The first sample data set may include: sample speech data, a first sample style description text for describing a speaking style, and sample text.

[0143] Exemplarily, the sample voice data may be the voice data of a sample speaker, or the voice data may also be any segment of voice data. Taking the sample voice data as the voice data of a sample speaker as an example, the sample speaker may, for example, be any real person, or a virtual person, etc., and the present application does not limit the sample speaker. Optionally, the above-mentioned first sample data set may, for example, include sample voice data of multiple sample speakers. Optionally, the sample voice data may be the original unprocessed voice data of the sample speaker. Alternatively, the sample voice data may, for example, be the voice data after pre-processing such as noise reduction of the original voice of the sample speaker. Optionally, the above-mentioned sample voice data may, for example, be a segment of sample voice data of any length.

[0144] Exemplarily, the speaking style described in the first sample style description text may include, for example: speaking speed, emotional expression, gender, pitch, volume, etc. The first sample style description text is a text used to describe the speaking style. It should be understood that the present application does not limit the language of the above-mentioned sample voice data, the first sample style description text, and the sample text, and the language of the sample voice data, the first sample style description text, and the sample text may be the same or different. The above-mentioned first sample style description text may include text, which is used to describe the speaking style. It should be understood that the present application does not limit the number of texts included in the first sample style description text.

[0145] Optionally, the sample text is text to be converted into speech. It should be understood that the present application does not limit the number of the sample text. In some embodiments, the sample text may be text converted from sample speech data, for example.

[0146] Optionally, the electronic device may obtain the first sample data set through, for example, a GUI or an API, etc. Optionally, the electronic device may obtain the sample voice data, the sample style description text, and the sample text in the same or different ways.

[0147] S202: Use the first sample data set to train the first preset model to obtain the speech synthesis model.

[0148] The first preset model may include, for example, multiple sub-models. The multiple sub-models may include, for example, sub-models to be trained and pre-trained sub-models. Alternatively, the multiple sub-models may all be sub-models to be trained, that is, the sub-models in the first preset model all need to be trained when the speech synthesis model is trained, rather than being pre-trained.

[0149] Taking the example that the above-mentioned first preset model includes multiple sub-models, and the multiple sub-models include sub-models to be trained and pre-trained sub-models, the pre-trained sub-model included in the first preset model can be, for example, a pre-trained feature extraction module. The function of the feature extraction module can, for example, refer to the feature extraction module described in any of the aforementioned embodiments, and will not be repeated here. The sub-model to be trained included in the first preset model can, for example, be the first preset sub-model to be trained. Among them, after the first preset model is trained through the above-mentioned first sample data set to obtain the speech synthesis model, the first preset sub-model to be trained is the trained first sub-model, and the trained first sub-model can be used as the speech conversion module of the speech synthesis model. Among them, the function of the speech conversion module can, for example, refer to the speech conversion module described in any of the aforementioned embodiments, and will not be repeated here.

[0150] Taking the example that the above-mentioned first preset model includes multiple sub-models, and the multiple sub-models are all sub-models to be trained, the multiple sub-models may, for example, include the above-mentioned first preset sub-model to be trained and the second preset sub-model to be trained. Wherein, after the first preset model is trained through the above-mentioned first sample data set to obtain the speech synthesis model, the first preset sub-model to be trained is the trained first sub-model, and the trained first sub-model can be used as the speech conversion module of the speech synthesis model. Wherein, the function of the speech conversion module, for example, can refer to the speech conversion module described in any of the aforementioned embodiments, and will not be repeated here. After the first preset model is trained through the above-mentioned first sample data set to obtain the speech synthesis model, the second preset sub-model to be trained is the trained second sub-model, and the trained second sub-model can be used as the feature extraction module of the speech synthesis model. Wherein, the function of the feature extraction module, for example, can refer to the feature extraction module described in any of the aforementioned embodiments, and will not be repeated here.

[0151] Through the above method, the first preset model can be trained using the first sample data set to obtain a speech synthesis model that has the function of extracting sound features based on speech data, determining speaking style features based on a style description text, and converting target text into target speech based on the sound features and the speaking style features, thereby laying a foundation for the subsequent synthesis of a target speech having sound features provided by speech data and speaking style features described by a style description text based on the speech synthesis model.

[0152] Next, how to use the first sample data set to train the first preset model to obtain the speech synthesis model is described in detail:

[0153] As a possible implementation method, taking the above-mentioned speech synthesis model including: a feature extraction module and a speech conversion module, and the above-mentioned first preset model including: the pre-trained feature extraction module and the first preset sub-model to be trained as an example, as mentioned above, the feature extraction module may, for example, include: a sound encoding sub-model, a style encoding sub-model and a text content encoding sub-model.

[0154] Optionally, the electronic device can use the sound coding sub-model to perform feature extraction on the sample speech data to obtain sample sound features. The electronic device can use the style coding sub-model to perform feature extraction on the first sample style description text to obtain the first sample speaking style features, and use the text content coding sub-model to perform feature extraction on the sample text to obtain sample content features of the sample text.

[0155] The above-mentioned sound coding sub-model, style coding sub-model and text content coding sub-model may refer to the aforementioned embodiments and will not be described in detail here.

[0156] Then, the electronic device can use the sample sound feature, the first sample speech style feature and the sample content feature to train the first preset sub-model to obtain the speech conversion module. The speech conversion module and the feature extraction module can form the speech synthesis model.

[0157] Optionally, the electronic device uses the above-mentioned sample sound features, the first sample speaking style features and the sample content features to train the first preset sub-model to obtain the speech synthesis model. For example, any existing generative model training method can be referred to, which will not be repeated here.

[0158] As mentioned above, the style encoding sub-model may be a pre-trained neural network model. The following takes the style encoding sub-model as a pre-trained machine learning model as an example to illustrate how to perform model training to obtain the training method of the style encoding sub-model:

[0159] First, it should be understood that the execution subject of the training method of the style coding sub-model can also be any electronic device with processing capabilities such as a terminal or a server. Optionally, the electronic device that executes the training method of the style coding sub-model and the execution subject of the speech synthesis method provided in this application, as well as the execution subject of the training method of the speech synthesis model, can be the same electronic device or different electronic devices, and this application does not limit this.

[0160] Taking the example that the electronic device that executes the training method of the style coding sub-model and the execution subject of the training method of the speech synthesis model provided in the present application can be the same electronic device, as a possible implementation method, the electronic device uses the style coding sub-model to extract features of the first sample style description text to obtain the first sample speaking style features, for example, and can also train the second preset model to obtain the style coding sub-model.

[0161] For example, the electronic device may first obtain a second sample data set, wherein the second sample data set may include: a second sample style description text, and a sample style feature distribution corresponding to the second sample style description text.

[0162] It should be understood that the second sample style description text may be the same as or different from the aforementioned first sample style description text. Exemplarily, the speaking style described in the second sample style description text may include, for example: speaking speed, emotional expression, gender, pitch, volume, etc. The second sample style description text is a text used to describe the speaking style. It should be understood that the present application does not limit the language of the second sample style description text. The above-mentioned second sample style description text may include text, which is used to describe the speaking style. It should be understood that the present application does not limit the number of texts included in the second sample style description text.

[0163] Exemplarily, the sample style feature distribution can be represented by the mean and variance, for example. Alternatively, the sample style feature distribution can also be represented by parameters other than the mean and variance that can characterize the distribution, such as skewness, kurtosis, etc. Alternatively, the sample style feature distribution can be represented by at least one parameter that can characterize the distribution, such as the mean, variance, skewness, kurtosis, etc.

[0164] Optionally, the electronic device may obtain the second sample data set through an API or GUI, etc. Optionally, the present application does not limit the number of sample style description texts included in the second sample data set (one-to-one correspondence between sample style feature distribution and sample style description text).

[0165] Then, the electronic device may use the second sample data set to train the second preset model to obtain the style encoding sub-model.

[0166] Exemplarily, the second preset model may include, for example, a multi-layer convolutional neural network structure. In some embodiments, the electronic device uses the second sample data set to train the second preset model to obtain an implementation of the style coding sub-model, which may refer to any existing model training method.

[0167] Alternatively, in some embodiments, the electronic device may also, for example, first convert the sample style description text into a sample semantic vector, and then perform training of the second preset model based on the sample semantic vector.

[0168] For example, the electronic device may first perform semantic feature extraction on the second sample style description text to obtain a sample semantic vector. The sample semantic vector is used to characterize the semantics of the second sample style description text. As mentioned above, the electronic device may, for example, process the second sample style description text and preset tags through a model with a semantic extraction function to obtain a word vector corresponding to each word in the second sample style description text, as well as a semantic vector corresponding to the preset tag. The semantic vector corresponding to the preset tag may be the above-mentioned sample semantic vector. Alternatively, the electronic device may perform semantic feature extraction on the second sample style description text to obtain a sample semantic vector, for example, the method of "extracting semantic features from the style description text to obtain a target semantic vector" described in any of the aforementioned embodiments may also be referred to.

[0169] Then, the electronic device can convert the sample semantic vector into multiple initial style feature distributions. Optionally, the method of converting the sample semantic vector into multiple initial style feature distributions can refer to the aforementioned method of converting the target semantic vector into multiple initial style feature distributions.

[0170] The electronic device may train the second preset model based on the multiple initial style feature distributions and the aforementioned sample style feature distributions to obtain a style encoding sub-model.

[0171] In this embodiment, the second preset model is trained by using the initial style feature distribution and the sample style feature distribution obtained based on the conversion of multiple sample semantic vectors, so that the style coding sub-model has "a target style feature distribution that characterizes the speaking style features based on multiple initial style feature distributions", which lays a foundation for the electronic device to realize the synthesis of the target speech based on the style coding sub-model.

[0172] In some embodiments, in order to further improve the accuracy of style coding sub-model training and the richness of the target style feature distribution obtained based on the style coding sub-model, the electronic device may, for example, further add noise perturbations to the multiple initial style feature distributions respectively to obtain multiple noise-perturbed style feature distributions, and then use the multiple noise-perturbed style feature distributions and the aforementioned sample style feature distributions to train the second preset model to obtain the above-mentioned style coding sub-model.

[0173] For example, a noise disturbance algorithm may be pre-configured in the electronic device, and the electronic device may use the noise disturbance algorithm to perform noise disturbance addition on multiple initial style feature distributions respectively.

[0174] Exemplarily, the above-mentioned noise perturbation algorithm can refer to any existing method of performing noise perturbation on data, such as diagonal noise perturbation algorithm (Diagonal Noise Perturbation), isotropic noise perturbation (Isotropic Noise Perturbation), across clusters noise perturbation (Across Clusters Noise Perturbation), fixed noise perturbation (Fixed Noise Perturbation), etc.

[0175] Optionally, for the above-mentioned multiple initial style feature distributions, the noise disturbance algorithms used by the electronic device to add noise disturbance to different initial style feature distributions may be the same or different.

[0176] Optionally, the electronic device may, for example, use the multiple noise-perturbed style feature distributions and the aforementioned sample style feature distributions to repeatedly perform the following steps 1 to 3 until the aforementioned style encoding sub-model is obtained:

[0177] Step 1: Through the second preset model, use the weighted "style feature distribution after multiple noise disturbances" to obtain the prediction weight, prediction mean and prediction variance.

[0178] The aforementioned sample style feature distribution can be characterized by the sample mean and sample variance. The above-mentioned predicted mean and predicted variance can be used to characterize the predicted style feature distribution output by the second preset model in this round of model training. For the above-mentioned weighted "style feature distributions after multiple noise disturbances", the weight corresponding to the style feature distribution after each noise disturbance can be the predicted weight output by the second preset model in the previous round of model training. In other words, the predicted weight can be multiple, corresponding to each style feature distribution after noise disturbance.

[0179] Step 2: Calculate the loss function based on the predicted mean and predicted variance as well as the sample mean and sample variance.

[0180] Optionally, the loss function may refer to any existing loss function, such as Mean Squared Error (MSE), Cross-Entropy Loss, Adversarial Loss, etc. The above-mentioned prediction mean and prediction variance characterize the distribution of predicted style features, and the above-mentioned sample mean and sample variance represent the distribution of sample style features. Based on the predicted value and the true value of the second preset model, the loss function may be calculated and the next round of training may be performed.

[0181] Step 3: Based on the loss function and the prediction weight, the second preset model is trained in the next round, wherein the preset weight is the weight of the style feature distribution after the noise disturbance in the next round of training of the second preset model.

[0182] Through the above method, it is possible to train the second preset model based on the style feature distribution after multiple noise disturbances, that is, to train the second preset model through a richer training set, thereby ensuring the richness of the target style feature distribution output by the trained style encoding sub-model when applied.

[0183] Taking the above speech synthesis model as an example, the decoupled feature extraction module is used to achieve zero-shot speaker cloning (i.e., imitation of voice features) and style control. Figure 4 This is a schematic diagram of the architecture of another speech synthesis model provided in this application. Figure 4 As shown, the speech synthesis model can receive input (eg, user input) speech prompts (ie, speech data) and style description text, as well as target text to be converted into a target speech.

[0184] Then, the speech synthesis model can encode the style description text, target text and speech data independently through a decoupled encoder (where the style encoding submodel and semantic extraction submodel belong to the style feature encoder, the text content encoding submodel is the text content encoder, and the timbre extractor and audio encoder belong to the sound feature encoder) to obtain speaking style features (such as emotion, speaking speed, etc.), content features of the target text and sound features (usually including timbre, rhythm and other features of the speech). Through the above-mentioned decoupled encoder, the information of the voice prompt, target text and style description text can be processed independently, reducing information interference.

[0185] In the model training phase, as a possible implementation method, Figure 5 This is a schematic diagram of the structure of a style feature encoder (i.e., the aforementioned style encoding sub-model) provided in this application. Figure 5As shown, the above-mentioned preset mark can be [CLS]. By inputting [CLS] and the sample style description text into the BERT model, the word vector corresponding to [CLS] is obtained as the target semantic vector, which solves the many-to-one problem (the specific content of the style description text is different but the expression is the same style). Then, the style feature encoder can convert the target semantic vector into multiple initial style feature distributions to meet the one-to-many scenario in the actual scenario (refer to the explanation of the previous embodiment, which will not be repeated here). Therefore, this one-to-many scenario is combined with the solution to the above-mentioned many-to-one problem, and the problem of many-to-many style control is solved by the style feature encoder. In addition, in the model training stage, noise disturbance is added to obtain multiple style feature distributions after noise disturbance. Then, based on the style feature distributions after the multiple noise disturbances, the second preset model is trained to obtain a style feature encoder.

[0186] The second preset model may be constructed based on a Gaussian Mixture Model (GMM), for example. In some embodiments, the style feature encoder may also be referred to as a Style Mixture Semantic Density (SMSD) model.

[0187] like Figure 5 As shown, where ω, σ, μ 2 are the aforementioned prediction weight, prediction mean and prediction variance respectively. L represents the loss function. Sampling means that in the application process after the training of the style coding sub-model is completed, the semantic vector corresponding to the preset tag is sampled as the target semantic vector, that is, a target semantic vector is sampled for extracting the speaking style features. In addition, the style codec (a discrete speech representation) is used as the true style of the style coding sub-model training (that is, the taget in the training process). The style extractor can perform operations such as pooling on the style codec, erase the time information, etc., to obtain the global features of the true style, which are used as the sample style feature distribution for training the style coding sub-model.

[0188] The speech conversion module can fuse the speaking style features and the content features of the target text through the cross-fusion unit to obtain the fusion features. Figure 4 As shown, the cross-fusion unit can be used, for example, to initially fuse the speech style features and the content features of the target text through a cross-attention mechanism to obtain a preliminary fusion feature. Figure 4As shown, in the preliminary fusion process, the speaking style features are used as the K (key) and V (value) parameters of the cross-attention mechanism, and the content features of the target text are used as the Q (query) parameters of the cross-attention mechanism. Optionally, the implementation method of the cross-attention mechanism based on the K (key), V (value) and Q (query) can refer to the existing cross-attention mechanism implementation method. Then, the cross-fusion unit can, for example, adjust the dimension of the preliminary fusion feature through a variable information adapter to obtain the preliminary fusion feature after dimension adjustment. Then, the cross-fusion unit can, for example, fuse the above-mentioned speaking style features with the preliminary fusion feature after dimension adjustment again through the cross-attention mechanism to obtain the above-mentioned fusion feature. The execution of the cross-attention mechanism here, such as Figure 4 As shown, the speaking style features can still be used as the K (key) and V (value) parameters of the cross-attention mechanism, and the Q (query) parameters of the cross-attention mechanism here can be determined based on the preliminary fusion features after dimension adjustment. The fusion features can be processed by the codec generator component to generate discrete speech representations (based on the method of generating discrete speech representations by the codec generator component, for example, the existing generative network can be referred to), and then the fusion features and sound features are decoded by the decoding unit to obtain and output the target speech with the specified timbre and style. Among them, the decoding unit can perform parallel decoding, that is, the decoding unit can process multiple feature representations at the same time to improve the speed and efficiency of speech generation.

[0189] also, Figure 6 A schematic diagram of the training process of a speech conversion module provided in this application. Figure 6 As shown, the true target value can also be called ground truth (GT), the frame length is Frame, which represents the duration of a speech segment, for example, a speech segment is 10 seconds (s) or 20s, etc., and NQ represents the number of channels at each moment (equal to the number of quantizers at that moment). Based on the mask iterative generation technology, the true target value can be masked to obtain the masked result to further improve the accuracy and naturalness of the generated speech. Then, based on the fusion feature, the mask is predicted through the codec generation component to achieve the fusion of style features and content features to obtain the target fusion result. Then, the decoding unit can decode the target fusion result and the sound feature in parallel to synthesize the target speech.

[0190] In this embodiment, the speech synthesis model can simultaneously perform zero-sample speaker cloning (i.e., voice imitation) and style control, thereby improving the flexibility and diversity of speech synthesis. And through the decoupled encoder, independent feature extraction of sound features, content features, and speaking style features is achieved, thereby improving the naturalness and diversity of the generated speech. And the above-mentioned style feature encoder generates diversified style speech through mixed distribution sampling, thereby solving the problem of many-to-many (the many-to-many concept can refer to the explanation of the previous embodiment) style control.

[0191] The present application also provides a speech synthesis system, wherein the system may include, for example: a client and a server, wherein the server may be deployed with a speech synthesis model as described in any of the above embodiments.

[0192] As a possible implementation, the client can be used to receive the voice data to be processed, the style description text for describing the speaking style, and the target text. The voice data to be processed, the style description text, and the target text can be explained with reference to the above-mentioned embodiments, and will not be described in detail here.

[0193] It should be understood that the present application does not limit the deployment environment of the client. For example, the client can be deployed in a mobile phone, a computer, a car system, a tablet computer, and other devices.

[0194] In some embodiments, the target text may be a target text input by a user received by the client. Alternatively, the target text may be pre-configured, for example. Taking the audition scenario as an example, the client may display a plurality of candidate texts for the user to select. Then, the client may respond to the text selected by the user as the target text. Then, the server synthesizes the target speech corresponding to the target text of the style and sound (e.g., timbre) specified by the user for the user to audition.

[0195] The server can be used to respond to the text-to-speech instruction, extract sound features based on the speech data through the speech synthesis model, and determine the speaking style features based on the style description text. Then, the server can convert the target text into the target speech based on the sound features and the speaking style features through the speech synthesis model. Then, the server can output the target speech through the above-mentioned client.

[0196] Optionally, the text-to-speech instruction may be triggered by a user through a client, for example. Alternatively, the server may determine that the text-to-speech instruction is received when receiving the “voice data, style description text for describing the speaker's speaking style, and target text” from the client.

[0197] It should be understood that the implementation method of the speech synthesis model determining the speaking style features based on the style description text can refer to the method described in any of the above embodiments. For example, the speech synthesis model can extract semantic features from the style description text to obtain a target semantic vector. As mentioned above, the target semantic vector can be used to characterize the semantics of the style description text. Then, the speech synthesis model can convert the target semantic vector into multiple initial style feature distributions, and obtain a target style feature distribution based on the multiple initial style feature distributions. The target style feature distribution can be used to characterize the above-mentioned speaking style features.

[0198] Through the above method, the speech synthesis system can determine the target style feature distribution used to characterize the above speaking style features after converting the target semantic vector into multiple initial style feature distributions. The process of converting the target semantic vector into multiple initial style feature distributions conforms to the one-to-many scenario in the actual scenario (i.e., a style description text may actually correspond to the possibility of multiple styles, for example, the style description text "she speaks very fast", then there are multiple possibilities for the degree of fast speaking speed, for example, first-level fast, second-level fast, third-level fast, etc., can all be called fast speaking speed). By converting the target semantic vector into multiple initial style feature distributions, it is achieved to obtain multiple initial style feature distributions based on one style description text, that is, multiple styles. Therefore, through the above method, the speech synthesis system is more in line with the actual speaking style judgment process in the process of determining the target style feature distribution, which improves the diversity and accuracy of the speaking style represented by the target style feature distribution.

[0199] In this embodiment, the speech synthesis system can extract features from speech data through a speech synthesis model to obtain the sound features, and determine the speaking style features based on a style description text used to describe the speaker's speaking style. Then, the speech synthesis system can convert the target text into a target speech based on the sound features and the speaking style features. The target speech output by the speech synthesis system can not only conform to the sound features, but also the style of the target speech can conform to the style, so that the style of the synthesized speech can be controlled while the sound can be imitated. Therefore, compared with the existing speech synthesis system that can only imitate the timbre, the speech synthesis system provided by the present application improves the flexibility of speech synthesis. Through the speech synthesis system provided by the present application, the user can provide the speech data of the sound features he wants and the style description text of the speaking style he wants. The speech synthesis system can adjust the sound and style of the output target speech based on the speech data and style description text provided by the user, so as to meet the user's personalized customization needs for speech synthesis and improve the user experience.

[0200] For example, Figure 7 This is a schematic diagram showing the differences between the speech synthesis system provided in this application and other speech synthesis systems. Figure 7 As shown, the text content space (space) is the content feature of the target text. The timbre space belongs to the sound feature. The style space is the speaking style feature. The specific implementation of the residual vector quantizer (RVQ) can refer to the above-mentioned embodiment, which will not be repeated here.

[0201] like Figure 7 As shown, the existing speech synthesis system 1 cannot clone the timbre of the speaker's voice, but can only control the style; the existing speech synthesis system 2 can clone the voice, but cannot control the style of the synthesized speech, for example, the target speech synthesized based on the speech data with a fast speaking speed has a faster speaking speed. The target speech output by the speech synthesis system provided by the present application can not only meet the voice characteristics, but also meet the style of the target speech, so that the style of the synthesized speech can be controlled while imitating the sound.

[0202] In this embodiment, the speech synthesis system provided by the present application can generate speech with controlled timbre, content and style based on a few seconds of voice prompts and simple style description text prompts. Through this speech synthesis system, personalized speech generation services can be provided to users, which can simulate speakers that have never been seen and control the style of their synthesized speech. In addition, the speech synthesis system can be flexibly applied to multiple fields such as intelligent assistants, voice navigation, teaching, media entertainment, etc. to enhance user experience.

[0203] It should be understood that the present application does not limit the application scenarios of the speech synthesis method. For example, the speech synthesis method can be applied to artificial intelligence models such as voice assistants, voice navigation, teaching scenarios, audio books, language models, etc. The following is an exemplary description of the application scenarios of the speech synthesis method:

[0204] Taking the application of the speech synthesis method to a voice assistant as an example, the voice assistant can, for example, provide users with speech synthesis results with controllable sound characteristics and speaking style, thereby increasing the flexibility of the voice assistant and improving the user experience. Optionally, the voice assistant can be deployed in any terminal device or vehicle system with processing capabilities. In addition, the present application does not limit the type of the voice assistant. For example, the voice assistant can be a vehicle voice assistant, a smart home voice assistant, an education voice assistant, a robot voice assistant, a customer service voice assistant, etc.

[0205] Figure 8 A flow chart of a method for speech synthesis using a voice assistant provided in this application. Figure 8 As shown, the method comprises the following steps:

[0206] S301, responding to a voice assistant style customization instruction, receiving, through a style customization interface, voice data to be processed and a style description text for describing a speaking style input by a user.

[0207] Among them, the above-mentioned voice assistant style customization instruction can be triggered by a user through a voice instruction, for example. For example, the terminal device deployed with the voice assistant can respond to the user's voice data describing "open the voice assistant style customization" to trigger the above-mentioned response voice assistant style customization instruction. The voice assistant can respond to the voice assistant style customization instruction and output a style customization interface. Alternatively, taking the terminal device as a touch device as an example, the user can trigger the above-mentioned voice assistant style customization instruction by, for example, opening the voice assistant style customization interface.

[0208] Exemplarily, the above-mentioned style customization interface can be, for example, a voice assistant style customization interface, or a user voice receiving interface. Among them, the user voice receiving interface can be, for example, any voice receiving device such as a microphone of the terminal device. In other words, the voice assistant (or the terminal device deployed with the voice assistant) can, for example, receive the voice data to be processed and the style description text for describing the speaking style input by the user through the above-mentioned voice assistant style customization interface, or the user voice receiving interface.

[0209] S302: Determine the sound features and speaking style features through a speech synthesis model based on the speech data and the style description text.

[0210] Optionally, the voice assistant determines the implementation method of the sound features and speaking style features based on the above-mentioned voice data and style description text through a speech synthesis model. The speech synthesis model described in any of the aforementioned embodiments can be referred to and will not be repeated here.

[0211] Optionally, the voice assistant can, for example, store the sound features and speaking style features after determining the above-mentioned sound features and speaking style features (for example, in the storage space of the terminal device, or in the background of the voice assistant, etc.), so that the sound features and speaking style features can be used to synthesize the target speech during subsequent speech output.

[0212] S303, in response to the voice interaction instruction, determine the target text, and convert the target text into a target voice through the voice synthesis model based on the sound features and speaking style features.

[0213] Exemplarily, taking the voice assistant as a question-and-answer type voice assistant as an example, the voice assistant can be configured with an interactive feedback algorithm for determining the answer to be output to the user based on the user's question. For example, the voice assistant can encode the user's question in response to the voice interaction instruction, and determine the answer to be output to the user based on the interactive feedback algorithm. The answer is the above-mentioned target text. Then, the voice assistant can convert the target text into the target speech based on the above-mentioned sound features and speaking style features through the above-mentioned speech synthesis model.

[0214] Alternatively, the voice assistant may obtain the target text by referring to any existing voice assistant responding to voice interaction instructions to determine a method of outputting the target text as feedback to the user.

[0215] S304: Output the target speech.

[0216] Optionally, the voice assistant can output the target voice through a voice broadcast device of the terminal device, for example, to achieve voice interaction with the user.

[0217] In this embodiment, the voice assistant can respond to the user's custom instructions for the voice assistant, and determine the sound features and speaking style features for customizing the voice assistant based on the voice data and style description text provided by the user, laying the foundation for subsequent interaction with the user based on the sound features and speaking style features. After the voice assistant determines the target text in response to the voice interaction instruction, it can convert the target text into a target voice based on the sound features and speaking style features and output it. Through the above method, because the target voice is determined based on the sound features and speaking style features, the sound of the target voice output by the voice assistant can meet the sound features (i.e., the sound of the voice data is imitated), and the speaking style in the target voice can meet the speaking style features, that is, different voice data and different style description texts can make the sound and style of the target voice converted from the target text different, thereby realizing the customization of the style and sound of the voice assistant, meeting the user's personalized customization needs for the voice assistant, and improving the user experience.

[0218] Taking the application of the speech synthesis method to voice navigation as an example, the navigation software can, based on the speech synthesis method, provide users with customized services for personalized navigation broadcast voice with controllable sound characteristics and speaking style, so as to improve the diversity and personalization of voice broadcast during navigation.

[0219] For example, Fig. 9 A flow chart of a voice navigation method provided by this application. Fig. 9 As shown, the method comprises the following steps:

[0220] S401: In response to a start navigation instruction, determine a target text to be broadcasted during the navigation process.

[0221] Optionally, the execution subject of the voice navigation method can be any existing navigation software. The way in which the navigation software receives the navigation start instruction, the navigation route planning and the target text to be broadcasted during the navigation process can all refer to the existing navigation software and will not be described in detail here.

[0222] S402: Using the speech synthesis model, based on the sound features and speaking style features, convert the target text into a target speech.

[0223] The sound features and speaking style features can be determined in advance by the navigation software based on the voice data to be processed input by the user and the style description text used to describe the speaking style. Specifically, the navigation software determines the implementation method of the sound features and speaking style features based on the voice data to be processed input by the user and the style description text used to describe the speaking style, and the implementation method of converting the above-mentioned target text into the target speech based on the sound features and speaking style features through the above-mentioned speech synthesis model can refer to the speech synthesis model described in any of the above-mentioned embodiments, and will not be repeated here.

[0224] S403: Output the target speech.

[0225] Optionally, the navigation software may output the target voice, for example, through a voice broadcast device of a terminal device or a vehicle computer on which the navigation software is deployed, so as to implement voice navigation broadcast during the navigation process.

[0226] In this embodiment, the voice navigation method can determine the sound features and speaking style features for customizing voice navigation based on the voice data to be processed input by the user and the style description text for describing the speaking style, which lays the foundation for the subsequent output of voice navigation based on the sound features and speaking style features. After the navigation software responds to the start navigation instruction and determines the target text to be broadcasted during the navigation process, it can convert the target text into a target voice and perform voice navigation based on the sound features and speaking style features. Through the above method, because the target voice is determined based on the sound features and speaking style features, the sound of the target voice output by the voice navigation process can meet the sound features (i.e., the sound of the voice data is imitated), and the speaking style in the target voice can meet the speaking style features, that is, different voice data and different style description texts can make the sound and style of the target voice converted from the target text different, thereby realizing the customization of the style and sound of voice navigation, meeting the user's personalized customization needs for voice navigation, and improving user experience.

[0227] Taking the application of the speech synthesis method to audio books as an example, illustratively, Fig.10 A flow chart of a method for voice output provided by this application. Fig.10 As shown, the method comprises the following steps:

[0228] S501. In response to a book listening instruction, the target text is converted into a target speech based on the sound features and the speaking style features through the speech synthesis model.

[0229] Optionally, the above-mentioned audiobook instruction can be used to instruct to output the target reading material (such as novels, news releases, blog content, text in pictures, etc.) through voice. The target text can be, for example, the text in the target reading material.

[0230] The execution subject of the voice output method may be, for example, application software with voice output capability, such as application software that can provide a user with a book listening function.

[0231] The sound features and speaking style features can be determined in advance by the application software based on the voice data to be processed input by the user and the style description text used to describe the speaking style. Specifically, the implementation method of the application software determining the sound features and speaking style features based on the voice data to be processed input by the user and the style description text used to describe the speaking style, and the implementation method of converting the above-mentioned target text into the target speech based on the sound features and speaking style features through the above-mentioned speech synthesis model can refer to the speech synthesis model described in any of the above-mentioned embodiments, and will not be repeated here.

[0232] S502: Output the target speech.

[0233] Optionally, the application software may output the target voice through a voice broadcast device of a terminal device or a vehicle computer on which the application software is deployed, so as to provide the user with a book listening function.

[0234] In this embodiment, the application software providing the listening function can determine the sound features and speaking style features for customizing the listening voice based on the voice data to be processed input by the user and the style description text for describing the speaking style, which lays the foundation for the subsequent output of the listening voice based on the sound features and speaking style features. The application software subsequently responds to the listening instruction, and based on the sound features and speaking style features, converts the text in the target reading material as the target text into the target voice and outputs the voice. Through the above method, because the target voice is determined based on the sound features and speaking style features, the sound of the target voice output by the listening function can meet the sound features (i.e., the sound of the voice data is imitated), and the speaking style in the target voice can meet the speaking style features, that is, different voice data and different style description texts can make the sound and style of the target voice converted from the target text in the target reading material different, thereby realizing the customization of the style and sound of the voice of the listening function, meeting the user's personalized customization needs for the listening function, and improving the user experience.

[0235] Fig.11 This is a schematic diagram of the structure of a speech synthesis device provided in this application. Fig.11 As shown, the speech synthesis device may include: an acquisition module 71 , a processing module 72 , and an output module 73 .

[0236] The acquisition module 71 is used to acquire the speech data to be processed, the style description text for describing the speaking style, and the target text.

[0237] The processing module 72 is used to extract sound features based on the speech data through a speech synthesis model, and determine speaking style features based on the style description text; based on the sound features and the speaking style features, convert the target text into a target speech.

[0238] The output module 73 is used to output the target speech.

[0239] Optionally, taking the example that the speech synthesis model includes: a feature extraction module and a speech conversion module, and the feature extraction module includes: a sound coding sub-model, a style coding sub-model and a text content coding sub-model, wherein the sound coding sub-model is used to perform feature extraction on the speech data to obtain the sound features; the style coding sub-model is used to perform feature extraction on the style description text to obtain the speaking style features; the text content coding sub-model is used to perform feature extraction on the target text to obtain the content features of the target text; and the speech conversion module is used to synthesize the target speech based on the sound features, the speaking style features and the content features.

[0240] Optionally, the processing module 72 is specifically used to extract semantic features from the style description text to obtain a target semantic vector; the target semantic vector is used to represent the semantics of the style description text; and the target semantic vector is converted into a target style feature distribution. The target style feature distribution is used to represent the speaking style feature.

[0241] Optionally, the processing module 72 is specifically configured to convert the target semantic vector into a plurality of initial style feature distributions; and obtain the target style feature distribution based on the plurality of initial style feature distributions.

[0242] Optionally, the processing module 72 is specifically used to process the style description text and the preset mark to obtain the word vector corresponding to each word in the style description text and the semantic vector corresponding to the preset mark. The semantic vector corresponding to the preset mark is the target semantic vector.

[0243] Optionally, the processing module 72 is specifically configured to fuse the speaking style feature with the content feature of the target text to obtain a fused feature; and decode the fused feature and the sound feature in parallel to obtain the target speech.

[0244] The speech synthesis device provided in the present application is used to execute the aforementioned speech synthesis method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0245] Fig.12 This is a schematic diagram of the structure of a model training device provided in this application. Fig.12 As shown, the model training device may include: an acquisition module 81 and a processing module 82.

[0246] The acquisition module 81 is used to acquire a first sample data set, wherein the first sample data set includes: sample speech data, a first sample style description text for describing a speaking style, and sample text.

[0247] The processing module 82 is used to use the first sample data set to train a first preset model to obtain a speech synthesis model as described in any of the aforementioned embodiments.

[0248] Taking the speech synthesis model as an example, the speech synthesis model includes: a feature extraction module and a speech conversion module, the first preset model includes: the pre-trained feature extraction module and the first preset sub-model to be trained; the feature extraction module includes: a sound coding sub-model, a style coding sub-model and a text content coding sub-model, optionally, the processing module 82 is specifically used to use the sound coding sub-model to extract features from the sample speech data to obtain sample sound features; use the style coding sub-model to extract features from the first sample style description text to obtain a first sample speaking style feature; use the text content coding sub-model to extract features from the sample text to obtain sample content features of the sample text; use the sample sound features, the first sample speaking style features and the sample content features to train the first preset sub-model to obtain the speech conversion module.

[0249] Optionally, the processing module 82 is further used to obtain a second sample data set before using the style encoding sub-model to extract features from the first sample style description text to obtain the first sample speech style features; the second sample data set includes: the second sample style description text, and the sample style feature distribution corresponding to the second sample style description text; the second preset model is trained using the second sample data set to obtain the style encoding sub-model. The sample style feature distribution is used to characterize the second sample speech style features.

[0250] Optionally, the processing module 82 is specifically configured to extract semantic features from the second sample style description text to obtain a sample semantic vector; convert the sample semantic vector into a plurality of initial style feature distributions; and train the second preset model based on the plurality of initial style feature distributions and the sample style feature distribution to obtain the style encoding sub-model. The sample semantic vector is used to characterize the semantics of the second sample style description text.

[0251] Optionally, the processing module 82 is specifically used to add noise perturbations to the multiple initial style feature distributions respectively to obtain multiple noise-perturbed style feature distributions; use the multiple noise-perturbed style feature distributions and the sample style feature distribution to train the second preset model to obtain the style encoding sub-model.

[0252] Optionally, the processing module 82 is specifically used to repeatedly execute the following steps until the style encoding sub-model is obtained: through the second preset model, the weighted style feature distribution after the multiple noise disturbances is used to obtain the prediction weight, the prediction mean and the prediction variance; the sample style feature distribution is characterized by the sample mean and the sample variance; based on the prediction mean and the prediction variance and the sample mean and the sample variance, a loss function is calculated; based on the loss function and the prediction weight, the second preset model is trained for the next round; the preset weight is the weight of the style feature distribution after the noise disturbance during the next round of training of the second preset model.

[0253] The model training device provided in this application is used to execute the aforementioned model training method embodiment. Its implementation principle and technical effect are similar and will not be described in detail.

[0254] Fig.13 A schematic diagram of the hardware structure of an electronic device provided in this application. Fig.13 The electronic device 90 shown includes a memory 91, a processor 92, and a communication interface 93. The memory 91, the processor 92, and the communication interface 93 are connected to each other in communication. For example, the memory 91, the processor 92, and the communication interface 93 can be connected to each other in communication by a network connection. Alternatively, the electronic device 90 may further include a bus 94. The memory 91, the processor 92, and the communication interface 93 are connected to each other in communication via the bus 94. Fig.13 The electronic device 90 is a memory 91 , a processor 92 , and a communication interface 93 that are connected to each other via a bus 94 .

[0255] The memory 91 may be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 91 may store a program. When the program stored in the memory 91 is executed by the processor 92, the processor 92 and the communication interface 93 are used to execute the speech synthesis method described in any of the aforementioned embodiments. The memory may also store data required by the speech synthesis method.

[0256] The processor 92 may be a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits.

[0257] The processor 92 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, the speech synthesis and / or model training of the present application may be completed by hardware integrated logic circuits or software instructions in the processor 92. The above-mentioned processor 92 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and may implement or execute the methods, steps and logic block diagrams disclosed in the following embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the following embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 91, and the processor 92 reads the information in the memory 91 and completes the road information processing of the present application in combination with its hardware.

[0258] The communication interface 93 uses a transceiver module such as, but not limited to, a transceiver to implement communication between the electronic device 90 and other devices or a communication network. For example, a data set can be acquired through the communication interface 93 .

[0259] When the electronic device 90 includes a bus 94 , the bus 94 may include a path for transmitting information between various components of the electronic device 90 (eg, the memory 91 , the processor 92 , and the communication interface 93 ).

[0260] The present application also provides a computer-readable storage medium, which may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes. Specifically, the computer-readable storage medium stores program instructions, and the program instructions are used for the methods in the above embodiments.

[0261] The present application also provides a program product, which includes an execution instruction, which is stored in a readable storage medium. At least one processor of the electronic device can read the execution instruction from the readable storage medium, and at least one processor executes the execution instruction so that the electronic device implements the speech synthesis and / or model training methods provided by the various embodiments described above.

[0262] The term "plurality" in this article refers to two or more than two. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship; in the formula, the character " / " indicates that the previous and next associated objects are in a "division" relationship. In addition, it should be understood that in the description of this application, words such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

[0263] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0264] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Acquire speech data to be processed, style description text for describing the speaking style, and target text; Extracting sound features based on the speech data through a speech synthesis model, and determining speaking style features based on the style description text; converting the target text into a target speech based on the sound features and the speaking style features; The target speech is output.

2. The method according to claim 1, characterized in that The speech synthesis model includes: a feature extraction module and a speech conversion module, and the feature extraction module includes: a sound encoding sub-model, a style encoding sub-model and a text content encoding sub-model; Wherein, the sound coding sub-model is used to extract features from the speech data to obtain the sound features; The style encoding sub-model is used to extract features from the style description text to obtain the speaking style features; The text content encoding sub-model is used to extract features of the target text to obtain content features of the target text; The speech conversion module is used to synthesize the target speech based on the sound feature, the speaking style feature and the content feature.

3. The method according to claim 2, characterized in that The extracting features of the style description text to obtain the speaking style features includes: Extracting semantic features from the style description text to obtain a target semantic vector; the target semantic vector is used to represent the semantics of the style description text; The target semantic vector is converted into a target style feature distribution; the target style feature distribution is used to characterize the speaking style feature.

4. The method according to claim 3, characterized in that The converting the target semantic vector into a target style feature distribution comprises: Converting the target semantic vector into a plurality of initial style feature distributions; Based on the multiple initial style feature distributions, the target style feature distribution is obtained.

5. The method according to claim 3 or 4, characterized in that: The extracting semantic features of the style description text to obtain a target semantic vector includes: The style description text and the preset mark are processed to obtain a word vector corresponding to each word in the style description text and a semantic vector corresponding to the preset mark; the semantic vector corresponding to the preset mark is the target semantic vector.

6. The method according to any one of claims 2 to 4, characterized in that: The synthesizing the target speech based on the sound feature, the speaking style feature and the content feature comprises: Fusing the speaking style feature with the content feature of the target text to obtain a fused feature; The fusion feature and the sound feature are decoded in parallel to obtain the target speech.

7. A model training method, characterized in that: The method comprises: Acquire a first sample data set; the first sample data set includes: sample speech data, a first sample style description text for describing a speaking style, and sample text; The first preset model is trained using the first sample data set to obtain the speech synthesis model as described in any one of claims 1-6.

8. The method according to claim 7, characterized in that The speech synthesis model includes: a feature extraction module and a speech conversion module, the first preset model includes: the pre-trained feature extraction module and the first preset sub-model to be trained; the feature extraction module includes: a sound encoding sub-model, a style encoding sub-model and a text content encoding sub-model; The using the first sample data set to train a first preset model to obtain the speech synthesis model includes: Using the sound coding sub-model, extracting features from the sample speech data to obtain sample sound features; Using the style encoding sub-model, extracting features from the first sample style description text to obtain a first sample speaking style feature; Using the text content encoding sub-model, extracting features of the sample text to obtain sample content features of the sample text; The first preset sub-model is trained using the sample sound feature, the first sample speaking style feature, and the sample content feature to obtain the speech conversion module.

9. The method according to claim 8, characterized in that Before extracting features from the first sample style description text using the style encoding sub-model to obtain the first sample speaking style features, the method further includes: Acquire a second sample data set; the second sample data set includes: a second sample style description text, and a sample style feature distribution corresponding to the second sample style description text; the sample style feature distribution is used to characterize the second sample speaking style feature; The second preset model is trained using the second sample data set to obtain the style encoding sub-model.

10. The method according to claim 9, characterized in that The using the second sample data set to train the second preset model to obtain the style encoding sub-model includes: Extracting semantic features from the second sample style description text to obtain a sample semantic vector; the sample semantic vector is used to represent the semantics of the second sample style description text; Converting the sample semantic vector into a plurality of initial style feature distributions; Based on the multiple initial style feature distributions and the sample style feature distributions, the second preset model is trained to obtain the style encoding sub-model.

11. The method according to claim 10, characterized in that The training of the second preset model based on the multiple initial style feature distributions and the sample style feature distribution to obtain the style encoding sub-model includes: Adding noise disturbance to the multiple initial style feature distributions respectively to obtain multiple noise-perturbed style feature distributions; The second preset model is trained using the style feature distributions after the multiple noise disturbances and the sample style feature distribution to obtain the style encoding sub-model.

12. The method according to claim 11, characterized in that The method of training the second preset model using the style feature distributions after the plurality of noise disturbances and the sample style feature distribution to obtain the style encoding sub-model includes: Repeat the following steps until the style encoding sub-model is obtained: The second preset model is used to obtain a prediction weight, a prediction mean and a prediction variance by using the weighted style feature distributions of the plurality of noise disturbances; the sample style feature distribution is characterized by a sample mean and a sample variance; Calculating a loss function based on the predicted mean and the predicted variance and the sample mean and the sample variance; Based on the loss function and the prediction weight, the second preset model is trained in the next round; the preset weight is the weight of the style feature distribution after the noise disturbance during the next round of training of the second preset model.

13. A speech synthesis system, characterized in that: The system comprises: a client, and a server; the server is deployed with a speech synthesis model according to any one of claims 1 to 6; The client is used to receive the voice data to be processed, the style description text for describing the speaking style, and the target text; The server is used to respond to the text-to-speech instruction, extract sound features based on the speech data through the speech synthesis model, and determine the speaking style features based on the style description text; convert the target text into the target speech based on the sound features and the speaking style features; and output the target speech through the client.

14. The speech synthesis system according to claim 13, characterized in that: The determining of the speaking style feature based on the style description text comprises: Extracting semantic features from the style description text to obtain a target semantic vector; the target semantic vector is used to represent the semantics of the style description text; Converting the target semantic vector into a plurality of initial style feature distributions; Based on the multiple initial style feature distributions, a target style feature distribution is obtained, and the target style feature distribution is used to characterize the speaking style feature.

15. An electronic device, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to perform the method described in any one of claims 1 to 12.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 12 is implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images