Speech synthesis method and device, electronic equipment and storage medium

Abstract

The invention provides a speech synthesis method and device, electronic equipment and a storage medium, and relates to the technical field of speech, and the method comprises the steps: inputting an obtained to-be-synthesized text and an emotion attribute into a speech synthesis model, and obtaining a target speech outputted by the speech synthesis model; wherein the speech synthesis model is obtained through training based on a first sample text corresponding to the first sample speech and a first sample emotion feature corresponding to the first sample speech, and the first sample emotion feature is obtained after the first sample speech is input into the emotion coding model; the emotion coding model is obtained by training based on the target code and the minimum mutual information loss of the second sample emotion feature. According to the invention, the emotion coding model can be obtained based on the minimum mutual information loss training, so that the emotion features output by the emotion coding model do not include unrelated information such as timbre and text content, the speech synthesis model can realize fine-grained emotion control, and the control force of speech synthesis is improved.

Description

Technical Field

[0001] The present invention relates to the field of speech technology, and in particular to a speech synthesis method, device, electronic equipment and storage medium. Background Art

[0002] In recent years, text-to-speech (TTS) has been widely used in scenarios such as smart assistants, speakers, car-mounted devices, novel reading, and (short) video dubbing. The speech synthesis needs in these open fields have put forward new requirements for the control and diversity of speech synthesis.

[0003] In the related art, high-quality sound library recording data is usually used to train the speech synthesis model, but the trained speech synthesis model lacks fine-grained control over emotions, thereby reducing the controllability of speech synthesis. Summary of the invention

[0004] The present invention provides a speech synthesis method, device, electronic device and storage medium, which are used to solve the defect of reducing the controllability of speech synthesis in the prior art.

[0005] The present invention provides a speech synthesis method, comprising the following steps.

[0006] Obtain the text to be synthesized and its emotional attributes; Inputting the text to be synthesized and the emotional attribute into a speech synthesis model to obtain a target speech output by the speech synthesis model; Among them, the speech synthesis model is trained based on the first sample text corresponding to the first sample speech and the first sample emotion feature corresponding to the first sample speech, the first sample emotion feature is obtained after the first sample speech is input into the emotion coding model, the emotion coding model is trained based on the minimum mutual information loss of the target coding and the second sample emotion feature, the target coding includes the timbre coding of the second sample speech and / or the text content coding of the second sample text of the second sample speech, and the second sample emotion feature is obtained after the second sample speech is input into the initial emotion coding model.

[0007] According to a speech synthesis method provided by the present invention, obtaining emotional attributes includes: When receiving the attribute description text input by the user, inputting the attribute description text into the text big model to obtain the sentiment attribute output by the text big model; When the emotion template speech input by the user is received, the emotion template speech is input into the emotion coding model to obtain the emotion feature output by the emotion coding model, and the emotion feature is determined as the emotion attribute.

[0008] According to a speech synthesis method provided by the present invention, the emotion coding model is trained based on the following method: Inputting the second sample speech into the speech coding network of the initial emotion coding model to obtain the semantic features output by the speech coding network; Inputting the preset learnable emotion space and the semantic features into the speech coding network of the initial emotion coding model to obtain the second sample emotion feature output by the emotion coding network; Determining a first minimum mutual information loss between the second sample emotion feature and the timbre encoding, and determining a second minimum mutual information loss between the second sample emotion feature and the text content encoding; Based on the first minimum mutual information loss and the second minimum mutual information loss, the network parameters of the emotion coding network are updated to obtain the emotion coding model.

[0009] According to a speech synthesis method provided by the present invention, the network parameters of the emotion coding network are updated based on the first minimum mutual information loss and the second minimum mutual information loss to obtain the emotion coding model, including: Inputting the emotion description text corresponding to the second sample speech into the description text encoding network of the initial emotion encoding model to obtain the emotion description code output by the description text encoding network; Determining a contrastive learning loss based on the second sample emotion feature and the emotion description code; Based on the first minimum mutual information loss, the second minimum mutual information loss and the contrastive learning loss, the network parameters of the emotion encoding network and the network parameters of the description text encoding network are updated to obtain the emotion encoding model.

[0010] According to a speech synthesis method provided by the present invention, the speech synthesis model is trained based on the following method: Inputting the first sample text and the first sample emotion feature into a speech feature extraction network of an initial speech synthesis model to obtain sample speech features output by the speech feature extraction network; Inputting the sample speech feature into the speech decoding network of the initial speech synthesis model to obtain a first predicted speech output by the speech decoding network; Based on the first predicted speech and the first sample speech, the network parameters of the speech feature extraction network and the network parameters of the speech decoding network are updated to obtain the speech synthesis model.

[0011] According to a speech synthesis method provided by the present invention, the step of inputting the text to be synthesized and the emotional attribute into a speech synthesis model to obtain a target speech output by the speech synthesis model comprises: Inputting the text to be synthesized, the emotional attribute and other attributes into the speech synthesis model to obtain the target speech output by the speech synthesis model; the other attributes include at least one of the following: target speech environment, target sound quality level, target language and target speech style, and the sample speech feature is obtained by inputting the first sample text, the first sample emotional feature and other sample attributes corresponding to the first sample speech into the speech feature extraction network; Among them, the other sample attributes include at least one of the following: a sample voice environment corresponding to the first sample voice, a sample sound quality level corresponding to the first sample voice, a sample language corresponding to the first sample voice, and a sample voice style corresponding to the first sample voice.

[0012] According to a speech synthesis method provided by the present invention, the network parameters of the speech feature extraction network and the network parameters of the speech decoding network are updated based on the first predicted speech and the first sample speech to obtain the speech synthesis model, including: Based on the first predicted speech and the first sample speech, updating the network parameters of the speech feature extraction network and the network parameters of the speech decoding network to obtain a reference speech synthesis model; Obtaining a third sample text corresponding to a third sample voice and a third sample emotion feature corresponding to the third sample voice, wherein the third sample voice is a voice recorded by a sample subject at a sample location; Inputting the third sample text, the third sample emotion feature and the identifier of the sample object into the reference speech synthesis model to obtain a second predicted speech output by the reference speech synthesis model; Based on the second predicted speech and the third sample speech, the network parameters of the speech feature extraction network and the network parameters of the speech decoding network in the reference speech synthesis model are updated to obtain the speech synthesis model.

[0013] The present invention also provides a speech synthesis device, comprising: An acquisition unit, used to acquire the text to be synthesized and the sentiment attribute; A synthesis unit, used for inputting the text to be synthesized and the emotional attribute into a speech synthesis model to obtain a target speech output by the speech synthesis model; Among them, the speech synthesis model is trained based on the first sample text corresponding to the first sample speech and the first sample emotion feature corresponding to the first sample speech, the first sample emotion feature is obtained after the first sample speech is input into the emotion coding model, the emotion coding model is trained based on the minimum mutual information loss of the target coding and the second sample emotion feature, the target coding includes the timbre coding of the second sample speech and / or the text content coding of the second sample text of the second sample speech, and the second sample emotion feature is obtained after the second sample speech is input into the initial emotion coding model.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned speech synthesis methods when executing the computer program.

[0015] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the speech synthesis method described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned speech synthesis methods.

[0017] The speech synthesis method, device, electronic device and storage medium provided by the present invention input the acquired text to be synthesized and the emotional attribute into the trained speech synthesis model to obtain the target speech output by the speech synthesis model. Among them, the speech synthesis model is obtained by training based on the first sample text corresponding to the first sample speech and the first sample emotional feature corresponding to the first sample speech, the first sample emotional feature is obtained after the first sample speech is input into the emotional coding model, the emotional coding model is obtained by training based on the minimum mutual information loss of the target coding and the second sample emotional feature, the target coding includes the timbre coding of the second sample speech and / or the text content coding of the second sample text of the second sample speech, and the second sample emotional feature is obtained after the second sample speech is input into the initial emotional coding model. It can be seen that the present invention can obtain the emotional coding model based on the minimum mutual information loss training of the target coding and the second sample emotional feature, so that the emotional features output by the emotional coding model do not include irrelevant information such as timbre and text content, and then the speech synthesis model obtained by training based on the emotional features output by the emotional coding model and the sample text can realize the control of fine-grained emotions, thereby improving the controllability of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flowchart of a speech synthesis method provided by an embodiment of the present invention.

[0020] Figure 2 This is an example diagram of speech synthesis provided by an embodiment of the present invention.

[0021] Figure 3 This is one of the schematic diagrams of the training process of the emotion coding model provided by the embodiment of the present invention.

[0022] Figure 4 Schematic diagram of a public speech training set provided by an embodiment of the present invention.

[0023] Figure 5 This is the second schematic diagram of the training process of the emotion coding model provided by the embodiment of the present invention.

[0024] Figure 6 It is a schematic diagram of the framework of the initial emotion coding model provided by an embodiment of the present invention.

[0025] Figure 7 This is one of the schematic diagrams of the training process of the speech synthesis model provided in an embodiment of the present invention.

[0026] Figure 8 It is a schematic diagram of the framework of the initial speech synthesis model provided by an embodiment of the present invention.

[0027] Fig. 9 This is the second schematic diagram of the training process of the speech synthesis model provided in an embodiment of the present invention.

[0028] Fig.10 It is a structural schematic diagram of a speech synthesis device provided by an embodiment of the present invention.

[0029] Fig.11 It is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] Combine the following Figure 1-Figure 9 The speech synthesis method of the present invention is described. The execution subject of the speech synthesis method can be an electronic device such as a terminal, a tablet computer, a computer, etc., or a speech synthesis device set in the electronic device, and the speech synthesis device can be implemented by software, hardware, or a combination of the two.

[0032] Figure 1 is a flow chart of a speech synthesis method provided by an embodiment of the present invention, such as Figure 1 As shown, the speech synthesis method comprises the following steps: Step 101: Obtain the text to be synthesized and the emotional attributes.

[0033] For example, when a user has a need for speech synthesis, the text to be synthesized and the emotional attributes required for speech synthesis can be input into the electronic device, so that the electronic device obtains the text to be synthesized and the emotional attributes, or the electronic device obtains the previously stored text to be synthesized and the emotional attributes from the memory, for example, the emotional attributes include sad emotions or happy emotions, etc.

[0034] Step 102: input the text to be synthesized and the emotional attribute into a speech synthesis model to obtain a target speech output by the speech synthesis model.

[0035] Among them, the speech synthesis model is trained based on the first sample text corresponding to the first sample speech and the first sample emotion feature corresponding to the first sample speech, the first sample emotion feature is obtained after the first sample speech is input into the emotion coding model, the emotion coding model is trained based on the minimum mutual information loss of the target coding and the second sample emotion feature, the target coding includes the timbre coding of the second sample speech and / or the text content coding of the second sample text of the second sample speech, and the second sample emotion feature is obtained after the second sample speech is input into the initial emotion coding model.

[0036] For example, when the text to be synthesized and the emotional attributes are obtained, the text to be synthesized and the emotional attributes are input into a trained speech synthesis model, and the speech synthesis model generates and outputs the target speech based on the text to be synthesized and the emotional attributes, so that the output target speech includes not only the semantics of the text to be synthesized, but also the emotions corresponding to the emotional attributes.

[0037] It should be noted that the identification of the target speaker may also be input into the speech synthesis model so that the speech synthesis model finally outputs the target speech obtained by adopting the timbre of the target speaker, and the present invention is not limited to this.

[0038] The speech synthesis method provided by the present invention inputs the acquired text to be synthesized and the emotional attribute into the trained speech synthesis model to obtain the target speech output by the speech synthesis model. The speech synthesis model is obtained by training based on the first sample text corresponding to the first sample speech and the first sample emotional feature corresponding to the first sample speech, the first sample emotional feature is obtained after the first sample speech is input into the emotional coding model, the emotional coding model is obtained by training based on the minimum mutual information loss of the target coding and the second sample emotional feature, the target coding includes the timbre coding of the second sample speech and / or the text content coding of the second sample text of the second sample speech, and the second sample emotional feature is obtained after the second sample speech is input into the initial emotional coding model. It can be seen that the present invention can obtain the emotional coding model based on the minimum mutual information loss training of the target coding and the second sample emotional feature, so that the emotional features output by the emotional coding model do not include irrelevant information such as timbre and text content, and then the speech synthesis model obtained by training based on the emotional features output by the emotional coding model and the sample text can realize the control of fine-grained emotions, thereby improving the controllability of speech synthesis.

[0039] In one embodiment, the emotional attributes are obtained in the above step 101, which can be implemented in the following way: The first method is to input the attribute description text input by the user into a large text model to obtain the emotional attribute output by the large text model.

[0040] For example, the user inputs in the form of attribute description text to control the emotional attributes of speech synthesis. For the attribute description text, the attribute description text can be input into the text big model, and the attribute description text is parsed by the text big model. Through prompt design, the text big model outputs the emotional attributes and other attributes involved in the attribute description text. For example, the attribute description text input by the user is "Generate high-quality speech, spoken in English, news style, happy", and the text big model parses out "sound quality level: high, language: English, speech style: news, emotion: happy" from it. For attributes not involved in the attribute description text, the system default attributes can be used, such as "speech environment: recording studio". The parsed emotional attributes, other attributes and default attributes are then input into the speech synthesis model together to obtain the target speech output by the speech synthesis model.

[0041] The second way is that, when the emotion template voice input by the user is received, the emotion template voice is input into the emotion coding model to obtain the emotion feature output by the emotion coding model, and the emotion feature is determined as the emotion attribute.

[0042] For example, when receiving the emotion template voice input by the user, it means that the user wants to control the emotion of the synthesized target voice through the emotion template voice. Under this condition, the voice environment, sound quality level, language, and voice style adopt the specific values ​​or default values ​​of the voice attributes specified by the user; for the emotion attribute, the emotion template voice is input into the trained emotion coding model (also called the fine-grained emotion coding model), and the emotion feature in the emotion template voice is extracted through the emotion coding model, and the emotion feature is determined as the emotion attribute, and then the parsed emotion attribute and the specific value or default value of the voice attribute specified by the user are input into the speech synthesis model together to obtain the target voice output by the speech synthesis model.

[0043] In the third method, the user directly inputs the emotional attribute and other attributes, and the electronic device directly inputs the specific values ​​of the emotional attribute and other attributes input by the user into the speech synthesis model to obtain the target speech output by the speech synthesis model. For attributes not specified by the user, the default value is used.

[0044] Figure 2 is an example diagram of speech synthesis provided by an embodiment of the present invention, such as Figure 2 As shown, any of the three methods mentioned above can be used to obtain different speech attributes, and the speech attributes, the text to be synthesized and the identifier of the target speaker are input into the speech synthesis model, so that the speech synthesis model finally outputs the target speech obtained using the timbre of the target speaker.

[0045] In this embodiment, the emotional attribute that the user wants to control the speech synthesis can be obtained based on different forms of user input, thereby improving the flexibility of speech synthesis.

[0046] In one embodiment, Figure 3 is one of the schematic diagrams of the training process of the emotion coding model provided by the embodiment of the present invention, such as Figure 3 As shown, the emotion coding model is trained based on the following method: Step 301: input the second sample speech into the speech coding network of the initial emotion coding model to obtain the semantic features output by the speech coding network.

[0047] For example, each sample speech in a public speech training set and a high-quality speech training set can be used as the second sample speech, the second sample speech can be input into the speech coding network of the initial emotion coding model, and the semantic features in the second sample speech can be extracted through the speech coding network.

[0048] It should be noted that the public speech training set pretrain-data can be obtained in the following ways: the public speech training set does not have high requirements on speech quality, but it needs to consider the coverage of various speech attributes. Public speech can be recorded or crawled from the Internet. Public speech needs to cover a variety of speech environments, speech styles, speech sources, etc. The present invention can divide speech attributes into five categories: speech environment, sound quality level, language, speech style and fine-grained emotion, and obtain speech attributes through the following three different methods: The first method is to use open source detection tools to automatically identify the category of voice environment, sound quality level and language. Voice environment includes recording studio, conference room, lecture hall, outdoor, etc.; sound quality level is divided into high, medium and low; language is divided according to the actual data coverage, including Chinese, English, Russian, Japanese, Sichuan dialect, Northeastern dialect or Cantonese, etc.

[0049] The second method is to automatically judge the voice style based on the data acquisition channel. For example, the podcast data source can obtain tag information based on the website: finance, history, news, health, etc. The audio novel data source can be divided into fantasy, history, martial arts, suspense, horror, etc. according to the novel type.

[0050] The third method is to automatically obtain sentence-level fine annotations for fine-grained emotions. It can be obtained by manual annotation + self-supervised pre-training. For details, please refer to the following description of the trained emotion encoding model, and the emotion features can be output through the emotion encoding model. Figure 4 is a schematic diagram of a public speech training set provided by an embodiment of the present invention, such as Figure 4 As shown, the public speech training set includes multiple sample voices, the speech environment, sound quality level, language, voice style and fine-grained emotion corresponding to each sample voice.

[0051] It should be noted that the fine-tuned data of the high-quality speech training set can be obtained in the following ways: the sample speech in the high-quality speech training set is high-quality speech collected by professional speakers in a recording studio using professional equipment. Compared with the diverse speech environments, languages, speech styles and emotional coverage of the pretrain data, the high-quality speech training set can only cover some emotions and speech styles, and annotate the sample speech in the high-quality speech training set with emotional description text. The high-quality speech training set is used to build a speech synthesis model for the target speaker.

[0052] Step 302: input the preset learnable emotion space and the semantic features into the emotion coding network of the initial emotion coding model to obtain the second sample emotion features output by the emotion coding network.

[0053] For example, due to the limited number of emotion types, a preset learnable emotion space of fixed dimension is designed in advance. The preset learnable emotion space is composed of multiple emotion vectors, and each emotion vector represents a different emotion. The preset learnable emotion space is input into the emotion coding network of the initial emotion coding model. Usually, the emotion coding network can combine the emotions represented by the preset learnable emotion space to obtain more emotions. However, since more emotions may include emotions that are irrelevant to the input second sample speech, it is necessary to interact information between the emotion coding network and the speech coding network so that the emotion coding network obtains the semantic features output by the speech coding network, and then based on the semantic features and the emotions represented by the preset learnable emotion space, finally outputs the second sample emotion features that are consistent with the semantic features, also called emotion tokens.

[0054] In addition, the second sample speech is input into the timbre coding network of the initial emotion coding model to obtain the timbre coding output by the timbre coding network, and the second sample text corresponding to the second sample speech is input into the text coding network of the initial emotion coding model to obtain the text content coding output by the text coding network.

[0055] Step 303: determine a first minimum mutual information loss between the second sample emotion feature and the timbre encoding, and determine a second minimum mutual information loss between the second sample emotion feature and the text content encoding.

[0056] For example, when the second sample emotion feature corresponding to the second sample speech, the timbre code corresponding to the second sample speech, and the text content code corresponding to the second sample speech are obtained, the first minimum mutual information loss of the second sample emotion feature and the timbre code is calculated based on the following formula (1): , the first minimum mutual information loss The purpose is to remove the timbre-related information in the emotional features of the second sample. Specifically, vCLUB estimation can be used: in, represents the second sample emotion feature corresponding to the i-th frame of the second sample speech, represents the timbre code corresponding to the i-th frame of speech in the second sample speech, Indicates the second sample speech The timbre coding corresponding to the frame speech, Indicated in When known The probability distribution of Indicated in When known The probability distribution of Indicates the total number of frame speech in the second sample speech, also known as batch size. The first minimum mutual information loss The physical meaning of expression is given the second sample emotion feature When the second sample sentiment feature The possibility of the timbre code x belonging to any other frame of speech is as consistent as possible. In other words, the emotional feature of the second sample It has nothing to do with the tone.

[0057] And the second minimum mutual information loss of the second sample sentiment feature and text content encoding is calculated based on the following formula (2): , the second minimum mutual information loss The purpose is to remove the text content related information in the second sample sentiment feature, which can be estimated by vCLUB: in, represents the text content encoding corresponding to the i-th frame of speech in the second sample speech, Indicates the second sample speech The text content encoding corresponding to the frame speech, Indicated in When known The probability distribution of Indicated in When known The second minimum mutual information loss The physical meaning of expression is given the second sample emotion feature When the second sample sentiment feature The probability of the text content encoding t belonging to any other frame of speech is as consistent as possible. In other words, the emotional feature of the second sample Not relevant to the text content.

[0058] Step 304: Based on the first minimum mutual information loss and the second minimum mutual information loss, update the network parameters of the emotion coding network to obtain the emotion coding model.

[0059] For example, when the first minimum mutual information loss and the second minimum mutual information loss are obtained, the network parameters of the emotion coding network can be updated by the sum of the first minimum mutual information loss and the second minimum mutual information loss until a convergence condition is reached, and finally the emotion coding model is obtained.

[0060] In this embodiment, the network parameters of the emotion coding network can be updated based on the first minimum mutual information loss of the second sample emotion feature and timbre coding, and the second minimum mutual information loss of the second sample emotion feature and text content coding, so as to extract emotion-related information from the sample speech, and remove information irrelevant to emotion such as text content and timbre through the minimum mutual information decoupling training criterion, thereby improving the purity of the emotion information, thereby improving the purity of the emotion features output by the trained emotion coding model, and improving the decoupling control capability of different speech attributes.

[0061] In one embodiment, Figure 5 FIG. 2 is a second schematic diagram of the training process of the emotion coding model provided by the embodiment of the present invention. Figure 5 As shown, the above step 304 updates the network parameters of the emotion coding network based on the first minimum mutual information loss and the second minimum mutual information loss to obtain the emotion coding model, which can be specifically implemented by the following steps: Step 3041: input the emotion description text corresponding to the second sample speech into the description text encoding network of the initial emotion encoding model to obtain the emotion description code output by the description text encoding network.

[0062] For example, the annotator needs to annotate the emotion of the second sample speech. The annotator uses the descriptive words or texts that he feels are appropriate as the annotation results based on his own perception of the second sample speech. The descriptive words can be one or more. For example, the annotation result is <feeling a little happy, lively tone, and fast speaking speed>, so as to make a fine-grained description of the emotion of the second sample speech, and the annotation result is called the emotion description text corresponding to the second sample speech. The emotion description text corresponding to the second sample speech is input into the description text encoding network of the initial emotion encoding model, and the emotion description text is encoded by the description text encoding network to obtain the emotion description code output by the description text encoding network.

[0063] Step 3042: Determine the contrastive learning loss based on the second sample emotion feature and the emotion description code.

[0064] For example, when the second sample emotion feature and emotion description code corresponding to the second sample speech are obtained, the contrastive learning loss is calculated based on the following formula (3). The purpose of the contrastive learning loss is to bring the second sample emotion feature and the corresponding emotion description code into the same space. Specifically, the second sample emotion feature is used as an anchor point, and the corresponding emotion description code is used as a positive and negative example. InfoNCE loss is used for training, and the contrastive learning loss is: in, represents the emotion description code corresponding to the i-th frame of speech in the second sample speech, Indicates the second sample speech The emotion description encoding corresponding to the frame speech, f represents the distance measurement function, which can be done by dot product; E is the expectation, which means that the contrastive learning loss calculation is the average result of the training set, which can be approximated by Monte Carlo estimation, usually achieved through gradient descent training based on batch (a group of training samples). The physical meaning of contrastive learning loss is that among all emotion description encodings, the distance between the emotion feature of the second sample and the emotion description encoding matching the emotion feature of the second sample is the shortest.

[0065] Step 3043: Based on the first minimum mutual information loss, the second minimum mutual information loss and the contrastive learning loss, the network parameters of the emotion encoding network and the network parameters of the description text encoding network are updated to obtain the emotion encoding model.

[0066] For example, when the first minimum mutual information loss, the second minimum mutual information loss and the contrastive learning loss are obtained, the total loss can be determined based on the following formula (4): , based on the total loss The network parameters of the emotion encoding network and the network parameters of the description text encoding network are updated until the convergence condition is reached, and finally the emotion encoding model is obtained.

[0067] Figure 6 is a schematic diagram of the framework of the initial emotion coding model provided by the embodiment of the present invention, such as Figure 6 As shown, the initial emotion coding model includes a speech coding network (also called a speech information coding network), an emotion coding network (also called an emotion and mood coding network), a text coding network, a timbre coding network and a description text coding network, wherein speech is input into the speech coding network to obtain the semantic features output by the speech coding network, a preset learnable emotion space is input into the emotion coding network, and information is interacted with the semantic features of the speech coding network to finally obtain the emotion and mood token output by the emotion coding network; the emotion description text of the speech is input into the description text coding network to obtain the emotion description code (also called description code) output by the description text coding network; the text content corresponding to the speech is input into the text coding network to obtain the text content code output by the text coding network; and the speech is input into the timbre coding network to obtain the timbre code output by the timbre coding network.

[0068] It should be noted that the network parameters of the emotion encoding network and the description text encoding network in the initial emotion encoding model need to be trained, and the parameters of other modules can use pre-trained models. The emotion encoding network and the description text encoding network can use traditional text pre-trained models, such as the bidirectional transformer model (Bidirectional Encoder Representations from Transformers, BERT). The speech encoding network can use traditional speech pre-trained models, such as HuBERT, wav2vec, etc.; the text encoding network can use traditional text pre-trained models, such as BERT; the timbre encoding network can use traditional voiceprint models, such as xvector.

[0069] In this embodiment, based on the first minimum mutual information loss of the second sample emotion feature and timbre coding, the second minimum mutual information loss of the second sample emotion feature and text content coding, and the contrastive learning loss determined based on the second sample emotion feature and emotion description coding, the network parameters of the emotion coding network and the network parameters of the description text coding network are updated by combining contrastive learning loss and minimum mutual information loss, which can not only improve the purity of the emotion feature output by the trained emotion coding model, but also improve the accuracy of the emotion description coding output by the emotion coding model. In addition, for unlabeled sample speech, the trained emotion coding model can output accurate emotion features, thereby reducing the workload of labeling.

[0070] In one embodiment, Figure 7 FIG. 1 is one of the flowcharts of the speech synthesis model training process provided by the embodiment of the present invention. Figure 7 As shown, the speech synthesis model is trained based on the following method: Step 701: input the first sample text and the first sample emotion feature into a speech feature extraction network of an initial speech synthesis model to obtain sample speech features output by the speech feature extraction network.

[0071] The initial speech synthesis model may use VALL-E, Fish-speech or chatTTS, etc., and the present invention does not limit this.

[0072] For example, Figure 8 is a schematic diagram of the framework of the initial speech synthesis model provided by an embodiment of the present invention, such as Figure 8As shown, the initial speech synthesis model includes a speech feature extraction network and a speech decoding network from text to semantic token. For example, the speech feature extraction network can be a GPT (Generative Pre-trained Transformer) model. Among them, the speech feature extraction network determines the content and rhythmic style of the synthesized speech. The speech attribute control module mainly acts on the speech feature extraction network, that is, the sample text and the emotional features corresponding to the sample text are input through the speech attribute control module. In addition, the synthesized speech of the previous frame of speech can also be input into the speech feature extraction network, so that the speech feature extraction network can use the synthesized speech of the previous frame of speech as a reference to generate the synthesized speech of the current frame of speech, thereby improving the accuracy of the synthesized speech.

[0073] For example, each sample speech in the public speech training set can be used as the first sample speech, and the first sample text corresponding to the first sample speech and the first sample emotion feature corresponding to the first sample speech can be input into the speech feature extraction network of the initial speech synthesis model to obtain the sample speech features output by the speech feature extraction network.

[0074] Step 702: Input the sample speech features into the speech decoding network of the initial speech synthesis model to obtain a first predicted speech output by the speech decoding network.

[0075] For example, when the sample speech features output by the speech feature extraction network are obtained, the sample speech features are input into the speech decoding network of the initial speech synthesis model, and the sample speech features are decoded by the speech decoding network to obtain the first predicted speech corresponding to the first sample text.

[0076] Step 703: Based on the first predicted speech and the first sample speech, the network parameters of the speech feature extraction network and the network parameters of the speech decoding network are updated to obtain the speech synthesis model.

[0077] For example, when the first predicted speech corresponding to each first sample text is obtained, a first loss function is constructed based on the first predicted speech corresponding to each first sample text and the first sample speech (speech label) corresponding to each first sample text, and the network parameters of the speech feature extraction network and the network parameters of the speech decoding network are updated based on the first loss function until the convergence condition is reached, and finally a speech synthesis model is obtained.

[0078] In this embodiment, the initial speech synthesis model can be trained based on the sample text and sample emotion features corresponding to each sample speech in the public speech training set, so that the final speech synthesis model can achieve fine-grained emotion control, thereby improving the controllability of speech synthesis.

[0079] In one embodiment, the above step 102 inputs the text to be synthesized and the emotional attribute into a speech synthesis model to obtain the target speech output by the speech synthesis model, which can be specifically implemented in the following manner: The text to be synthesized, the emotional attributes and other attributes are input into the speech synthesis model to obtain the target speech output by the speech synthesis model; the other attributes include at least one of the following: target speech environment, target sound quality level, target language and target speech style, and the sample speech feature is obtained after the first sample text, the first sample emotional feature and other sample attributes corresponding to the first sample speech are input into the speech feature extraction network.

[0080] Among them, the other sample attributes include at least one of the following: a sample voice environment corresponding to the first sample voice, a sample sound quality level corresponding to the first sample voice, a sample language corresponding to the first sample voice, and a sample voice style corresponding to the first sample voice.

[0081] For example, in the stage of training the initial speech synthesis model based on a public speech training set, the obtained first sample speech corresponding to the sample speech environment, the sample sound quality level, the sample language and the sample speech style, the first sample emotional characteristics, the first sample text corresponding to the first sample speech, and the identifier of the speaker corresponding to the first sample speech can be spliced ​​to obtain splicing information, and the splicing information can be input into the initial speech synthesis model for training.

[0082] Similarly, the text to be synthesized, emotional attributes, target speech environment, target sound quality level, target language, target speech style and the identification of the target speaker can be spliced ​​to obtain target splicing information, and the target splicing information can be input into the speech synthesis model to obtain the target speech of the target speaker output by the speech synthesis model.

[0083] It should be noted that the first sample emotion feature corresponding to the first sample speech can be obtained in different ways: for the sample speech with emotion description text annotations, directly use the emotion description code of the annotated emotion description text as the first sample emotion feature (emotion token) corresponding to the first sample speech; for the sample speech without emotion description text annotations, input the sample speech into the trained emotion coding model to obtain the emotion token output by the emotion coding model, and use the emotion token as the first sample emotion feature corresponding to the first sample speech.

[0084] In this embodiment, the text to be synthesized, emotional attributes, target speech environment, target sound quality level, target language, target speech style and the identification of the target speaker can be input into the speech synthesis model together to obtain the target speech of the target speaker output by the speech synthesis model, thereby realizing the control of the speech synthesis model over different speech attributes and further improving the control over speech synthesis; in addition, the present invention uniformly divides the speech attributes into emotion, speech environment, sound quality level, language and speech style, and obtains a more comprehensive speech attribute division system, so that the speech synthesis model can control different speech attributes, that is, control the speech attributes of each dimension.

[0085] In one embodiment, Fig. 9 FIG. 2 is a second schematic diagram of a training process of a speech synthesis model provided by an embodiment of the present invention. Fig. 9 As shown, the above step 703 updates the network parameters of the speech feature extraction network and the network parameters of the speech decoding network based on the first predicted speech and the first sample speech to obtain the speech synthesis model, which can be specifically implemented in the following manner: Step 7031: Based on the first predicted speech and the first sample speech, the network parameters of the speech feature extraction network and the network parameters of the speech decoding network are updated to obtain a reference speech synthesis model.

[0086] For example, when the first predicted speech corresponding to each first sample text is obtained, a first loss function is constructed based on the first predicted speech corresponding to each first sample text and the first sample speech (speech label) corresponding to each first sample text, and the network parameters of the speech feature extraction network and the network parameters of the speech decoding network are updated based on the first loss function until the convergence conditions are reached, and finally the speech synthesis model is referenced.

[0087] Step 7032: Obtain a third sample text corresponding to a third sample voice and a third sample emotion feature corresponding to the third sample voice, wherein the third sample voice is the voice recorded by the sample subject at the sample location.

[0088] For example, each sample speech of the high-quality speech training set can be used as the third sample speech, and the third sample text corresponding to the third sample speech can be obtained from the high-quality speech training set. The third sample emotional feature corresponding to the third sample speech can be obtained in different ways: for the third sample speech with emotional description text annotation, the emotional description code of the annotated emotional description text is directly used as the third sample emotional feature (emotion token) corresponding to the third sample speech; for the third sample speech without emotional description text annotation, the third sample speech is input into the trained emotion coding model to obtain the emotion token output by the emotion coding model, and the emotion token is used as the third sample emotional feature corresponding to the third sample speech.

[0089] Step 7033: input the third sample text, the third sample emotional feature and the identifier of the sample object into the reference speech synthesis model to obtain a second predicted speech output by the reference speech synthesis model.

[0090] For example, when the third sample text corresponding to the third sample speech and the third sample emotion feature corresponding to the third sample speech are obtained, the third sample text, the third sample emotion feature and the identification of the sample object are input into the reference speech synthesis model to obtain the second predicted speech corresponding to the third sample text output by the reference speech synthesis model.

[0091] Step 7034: Based on the second predicted speech and the third sample speech, the network parameters of the speech feature extraction network and the network parameters of the speech decoding network in the reference speech synthesis model are updated to obtain the speech synthesis model.

[0092] For example, when the second predicted speech corresponding to each third sample text is obtained, a second loss function is constructed based on the second predicted speech corresponding to each third sample text and the third sample speech (speech label) corresponding to each third sample text, and the network parameters of the speech feature extraction network and the network parameters of the speech decoding network in the reference speech synthesis model are updated based on the second loss function until the convergence conditions are reached, and finally the speech synthesis model is obtained.

[0093] It should be noted that since the public speech training set contains a variety of different speech attributes, the reference speech synthesis model trained based on the public speech training set can cover the control of a variety of different speech attributes. In other words, the reference speech synthesis model has the performance of controlling a variety of different speech attributes. Therefore, even if the high-quality speech training set corresponds to a smaller number of speech attributes, the speech synthesis model finally trained can also control other uncovered speech attributes, thereby realizing speech synthesis controlled by attribute transfer.

[0094] It should be noted that the speech synthesis model can also be trained based on a first proportion of sample speech in a high-quality speech training set and a second proportion of sample speech in a public speech training set, wherein the first proportion is greater than the second proportion, for example, the first proportion is 80% and the second proportion is 20%, and the present invention is not limited to this.

[0095] It should be noted that the present invention also subdivides the way of adding speech attributes. For speech attributes such as speech environment, language and speech style, the user controls the granularity coarsely and adopts coarse-grained discrete category representation. For emotional attributes, users need fine-grained control, and the contrastive learning loss pre-training method is used to pull the emotional features and emotional description codes into the same space, and combined with the constraint of minimum mutual information loss, the information in the emotional features that is irrelevant to the emotions is further reduced, thereby improving the purity of the emotional features.

[0096] In summary, the speech synthesis method provided by the present invention divides the information in the speech except the text into speech attributes such as speech environment, sound quality level, language, speech style, emotion, and speaker timbre. The speech attributes are annotated by automatic annotation or self-supervised decoupling pre-training, and input into the speech synthesis model for control to achieve control of the attributes of the synthesized speech. Specifically, for the speech environment, sound quality level, language, and speech style, since the control accuracy of user requirements is not high, the design category can be used and annotated by automatic identification through information sources or tools; for fine-grained emotions, a large amount of public speech and emotion description texts are collected in advance, and fine-grained emotion features of fixed length are extracted, and the fine-grained emotion coding space and the text sentence-level semantic coding space are aligned by contrast learning pre-training and the minimum mutual information constraints of text content and timbre, and the fine-grained emotion features of attribute decoupling are extracted. After obtaining the different attribute annotations of the speech, they are input into the speech synthesis model together with the text to train the speech synthesis model base with controllable speech attributes. Finally, fine-tuning is performed in combination with the data of the target speaker to obtain a speech synthesis model with controllable speech attributes of the target speaker. During the inference phase, the text to be synthesized and the specified attributes input by the user are input into the trained speech synthesis model to achieve control over different attributes of speech synthesis, thereby improving the controllability and flexibility of the speech synthesis model.

[0097] The speech synthesis device provided by the present invention is described below. The speech synthesis device described below and the speech synthesis method described above can be referred to each other.

[0098] Fig.10 is a schematic diagram of the structure of a speech synthesis device provided by an embodiment of the present invention. Fig.10 As shown, the speech synthesis device 1000 includes an acquisition unit 1001 and a synthesis unit 1002; wherein: An acquisition unit 1001 is used to acquire the text to be synthesized and the emotional attributes; A synthesis unit 1002, used for inputting the text to be synthesized and the emotional attribute into a speech synthesis model to obtain a target speech output by the speech synthesis model; Among them, the speech synthesis model is trained based on the first sample text corresponding to the first sample speech and the first sample emotion feature corresponding to the first sample speech, the first sample emotion feature is obtained after the first sample speech is input into the emotion coding model, the emotion coding model is trained based on the minimum mutual information loss of the target coding and the second sample emotion feature, the target coding includes the timbre coding of the second sample speech and / or the text content coding of the second sample text of the second sample speech, and the second sample emotion feature is obtained after the second sample speech is input into the initial emotion coding model.

[0099] The speech synthesis device provided by the present invention inputs the acquired text to be synthesized and the emotional attribute into the trained speech synthesis model to obtain the target speech output by the speech synthesis model. The speech synthesis model is obtained by training based on the first sample text corresponding to the first sample speech and the first sample emotional feature corresponding to the first sample speech, the first sample emotional feature is obtained after the first sample speech is input into the emotional coding model, the emotional coding model is obtained by training based on the minimum mutual information loss of the target coding and the second sample emotional feature, the target coding includes the timbre coding of the second sample speech and / or the text content coding of the second sample text of the second sample speech, and the second sample emotional feature is obtained after the second sample speech is input into the initial emotional coding model. It can be seen that the present invention can obtain the emotional coding model based on the minimum mutual information loss training of the target coding and the second sample emotional feature, so that the emotional features output by the emotional coding model do not include irrelevant information such as timbre and text content, and then the speech synthesis model obtained by training based on the emotional features output by the emotional coding model and the sample text can realize the control of fine-grained emotions, thereby improving the controllability of speech synthesis.

[0100] Based on any of the above embodiments, the acquiring unit 1001 is specifically used for: When receiving the attribute description text input by the user, inputting the attribute description text into the text big model to obtain the sentiment attribute output by the text big model; When the emotion template speech input by the user is received, the emotion template speech is input into the emotion coding model to obtain the emotion feature output by the emotion coding model, and the emotion feature is determined as the emotion attribute.

[0101] Based on any of the above embodiments, the emotion coding model is trained based on the following method: Inputting the second sample speech into the speech coding network of the initial emotion coding model to obtain the semantic features output by the speech coding network; Inputting the preset learnable emotion space and the semantic features into the speech coding network of the initial emotion coding model to obtain the second sample emotion feature output by the emotion coding network; Determining a first minimum mutual information loss between the second sample emotion feature and the timbre encoding, and determining a second minimum mutual information loss between the second sample emotion feature and the text content encoding; Based on the first minimum mutual information loss and the second minimum mutual information loss, the network parameters of the emotion coding network are updated to obtain the emotion coding model.

[0102] Based on any of the foregoing embodiments, the updating of the network parameters of the emotion coding network based on the first minimum mutual information loss and the second minimum mutual information loss to obtain the emotion coding model includes: Inputting the emotion description text corresponding to the second sample speech into the description text encoding network of the initial emotion encoding model to obtain the emotion description code output by the description text encoding network; Determining a contrastive learning loss based on the second sample emotion feature and the emotion description code; Based on the first minimum mutual information loss, the second minimum mutual information loss and the contrastive learning loss, the network parameters of the emotion encoding network and the network parameters of the description text encoding network are updated to obtain the emotion encoding model.

[0103] Based on any of the above embodiments, the speech synthesis model is trained based on the following method: Inputting the first sample text and the first sample emotion feature into a speech feature extraction network of an initial speech synthesis model to obtain sample speech features output by the speech feature extraction network; Inputting the sample speech feature into the speech decoding network of the initial speech synthesis model to obtain a first predicted speech output by the speech decoding network; Based on the first predicted speech and the first sample speech, the network parameters of the speech feature extraction network and the network parameters of the speech decoding network are updated to obtain the speech synthesis model.

[0104] Based on any of the above embodiments, the synthesis unit 1002 is specifically used for: Inputting the text to be synthesized, the emotional attribute and other attributes into the speech synthesis model to obtain the target speech output by the speech synthesis model; the other attributes include at least one of the following: target speech environment, target sound quality level, target language and target speech style, and the sample speech feature is obtained by inputting the first sample text, the first sample emotional feature and other sample attributes corresponding to the first sample speech into the speech feature extraction network; Among them, the other sample attributes include at least one of the following: a sample voice environment corresponding to the first sample voice, a sample sound quality level corresponding to the first sample voice, a sample language corresponding to the first sample voice, and a sample voice style corresponding to the first sample voice.

[0105] Based on any of the above embodiments, the updating of the network parameters of the speech feature extraction network and the network parameters of the speech decoding network based on the first predicted speech and the first sample speech to obtain the speech synthesis model includes: Based on the first predicted speech and the first sample speech, updating the network parameters of the speech feature extraction network and the network parameters of the speech decoding network to obtain a reference speech synthesis model; Obtaining a third sample text corresponding to a third sample voice and a third sample emotion feature corresponding to the third sample voice, wherein the third sample voice is a voice recorded by a sample subject at a sample location; Inputting the third sample text, the third sample emotion feature and the identifier of the sample object into the reference speech synthesis model to obtain a second predicted speech output by the reference speech synthesis model; Based on the second predicted speech and the third sample speech, the network parameters of the speech feature extraction network and the network parameters of the speech decoding network in the reference speech synthesis model are updated to obtain the speech synthesis model.

[0106] Fig.11 is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Fig.11 As shown, the electronic device may include: a processor 1110, a communication interface 1120, a memory 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120 and the memory 1130 communicate with each other through the communication bus 1140. The processor 1110 may call the logic instructions in the memory 1130 to execute the speech synthesis method, which includes: obtaining the text to be synthesized and the emotion attribute; Inputting the text to be synthesized and the emotional attribute into a speech synthesis model to obtain a target speech output by the speech synthesis model; Among them, the speech synthesis model is trained based on the first sample text corresponding to the first sample speech and the first sample emotion feature corresponding to the first sample speech, the first sample emotion feature is obtained after the first sample speech is input into the emotion coding model, the emotion coding model is trained based on the minimum mutual information loss of the target coding and the second sample emotion feature, the target coding includes the timbre coding of the second sample speech and / or the text content coding of the second sample text of the second sample speech, and the second sample emotion feature is obtained after the second sample speech is input into the initial emotion coding model.

[0107] In addition, the logic instructions in the above-mentioned memory 1130 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0108] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the speech synthesis method provided by the above methods, the method includes: obtaining a text to be synthesized and an emotional attribute; Inputting the text to be synthesized and the emotional attribute into a speech synthesis model to obtain a target speech output by the speech synthesis model; Among them, the speech synthesis model is trained based on the first sample text corresponding to the first sample speech and the first sample emotion feature corresponding to the first sample speech, the first sample emotion feature is obtained after the first sample speech is input into the emotion coding model, the emotion coding model is trained based on the minimum mutual information loss of the target coding and the second sample emotion feature, the target coding includes the timbre coding of the second sample speech and / or the text content coding of the second sample text of the second sample speech, and the second sample emotion feature is obtained after the second sample speech is input into the initial emotion coding model.

[0109] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the speech synthesis method provided by the above methods, the method comprising: obtaining a text to be synthesized and an emotional attribute; Inputting the text to be synthesized and the emotional attribute into a speech synthesis model to obtain a target speech output by the speech synthesis model; Among them, the speech synthesis model is trained based on the first sample text corresponding to the first sample speech and the first sample emotion feature corresponding to the first sample speech, the first sample emotion feature is obtained after the first sample speech is input into the emotion coding model, the emotion coding model is trained based on the minimum mutual information loss of the target coding and the second sample emotion feature, the target coding includes the timbre coding of the second sample speech and / or the text content coding of the second sample text of the second sample speech, and the second sample emotion feature is obtained after the second sample speech is input into the initial emotion coding model.

[0110] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0111] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech synthesis method, characterized in that: include: Obtain the text to be synthesized and its emotional attributes; Inputting the text to be synthesized and the emotional attribute into a speech synthesis model to obtain a target speech output by the speech synthesis model; Among them, the speech synthesis model is trained based on the first sample text corresponding to the first sample speech and the first sample emotional feature corresponding to the first sample speech, the first sample emotional feature is obtained after the first sample speech is input into the emotional coding model, the emotional coding model is trained based on the minimum mutual information loss of the target coding and the second sample emotional feature, the target coding includes the timbre coding of the second sample speech and / or the text content coding of the second sample text of the second sample speech, and the second sample emotional feature is obtained after the second sample speech is input into the initial emotional coding model.

2. The speech synthesis method according to claim 1, characterized in that: Obtaining sentiment attributes includes: When receiving the attribute description text input by the user, inputting the attribute description text into the text big model to obtain the sentiment attribute output by the text big model; When the emotion template speech input by the user is received, the emotion template speech is input into the emotion coding model to obtain the emotion feature output by the emotion coding model, and the emotion feature is determined as the emotion attribute.

3. The speech synthesis method according to claim 1, characterized in that: The emotion coding model is trained based on the following method: Inputting the second sample speech into the speech coding network of the initial emotion coding model to obtain the semantic features output by the speech coding network; Inputting the preset learnable emotion space and the semantic features into the speech coding network of the initial emotion coding model to obtain the second sample emotion feature output by the emotion coding network; Determining a first minimum mutual information loss between the second sample emotion feature and the timbre encoding, and determining a second minimum mutual information loss between the second sample emotion feature and the text content encoding; Based on the first minimum mutual information loss and the second minimum mutual information loss, the network parameters of the emotion coding network are updated to obtain the emotion coding model.

4. The speech synthesis method according to claim 3, characterized in that: The updating of the network parameters of the emotion coding network based on the first minimum mutual information loss and the second minimum mutual information loss to obtain the emotion coding model includes: Inputting the emotion description text corresponding to the second sample speech into the description text encoding network of the initial emotion encoding model to obtain the emotion description code output by the description text encoding network; Determining a contrastive learning loss based on the second sample emotion feature and the emotion description code; Based on the first minimum mutual information loss, the second minimum mutual information loss and the contrastive learning loss, the network parameters of the emotion encoding network and the network parameters of the description text encoding network are updated to obtain the emotion encoding model.

5. The speech synthesis method according to any one of claims 1 to 4, characterized in that: The speech synthesis model is trained based on the following method: Inputting the first sample text and the first sample emotion feature into a speech feature extraction network of an initial speech synthesis model to obtain sample speech features output by the speech feature extraction network; Inputting the sample speech feature into the speech decoding network of the initial speech synthesis model to obtain a first predicted speech output by the speech decoding network; Based on the first predicted speech and the first sample speech, the network parameters of the speech feature extraction network and the network parameters of the speech decoding network are updated to obtain the speech synthesis model.

6. The speech synthesis method according to claim 5, characterized in that: The step of inputting the text to be synthesized and the emotional attribute into a speech synthesis model to obtain a target speech output by the speech synthesis model includes: Inputting the text to be synthesized, the emotional attribute and other attributes into the speech synthesis model to obtain the target speech output by the speech synthesis model; the other attributes include at least one of the following: target speech environment, target sound quality level, target language and target speech style, and the sample speech feature is obtained by inputting the first sample text, the first sample emotional feature and other sample attributes corresponding to the first sample speech into the speech feature extraction network; Among them, the other sample attributes include at least one of the following: a sample voice environment corresponding to the first sample voice, a sample sound quality level corresponding to the first sample voice, a sample language corresponding to the first sample voice, and a sample voice style corresponding to the first sample voice.

7. The speech synthesis method according to claim 5, characterized in that: The updating of the network parameters of the speech feature extraction network and the network parameters of the speech decoding network based on the first predicted speech and the first sample speech to obtain the speech synthesis model includes: Based on the first predicted speech and the first sample speech, updating the network parameters of the speech feature extraction network and the network parameters of the speech decoding network to obtain a reference speech synthesis model; Obtaining a third sample text corresponding to a third sample voice and a third sample emotion feature corresponding to the third sample voice, wherein the third sample voice is a voice recorded by a sample subject at a sample location; Inputting the third sample text, the third sample emotion feature and the identifier of the sample object into the reference speech synthesis model to obtain a second predicted speech output by the reference speech synthesis model; Based on the second predicted speech and the third sample speech, the network parameters of the speech feature extraction network and the network parameters of the speech decoding network in the reference speech synthesis model are updated to obtain the speech synthesis model.

8. A speech synthesis device, characterized in that: include: An acquisition unit, used to acquire the text to be synthesized and the sentiment attribute; A synthesis unit, used for inputting the text to be synthesized and the emotional attribute into a speech synthesis model to obtain a target speech output by the speech synthesis model; Among them, the speech synthesis model is trained based on the first sample text corresponding to the first sample speech and the first sample emotional feature corresponding to the first sample speech, the first sample emotional feature is obtained after the first sample speech is input into the emotional coding model, the emotional coding model is trained based on the minimum mutual information loss of the target coding and the second sample emotional feature, the target coding includes the timbre coding of the second sample speech and / or the text content coding of the second sample text of the second sample speech, and the second sample emotional feature is obtained after the second sample speech is input into the initial emotional coding model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the speech synthesis method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech synthesis method according to any one of claims 1 to 7 is implemented.

Images