Voice generation method and device

Abstract

The embodiment of the invention provides a voice generation method and device, computer equipment, a computer readable storage medium and a computer program product, and belongs to the field of audio processing. The voice generation method comprises the following steps: acquiring a content text and an emotion description text; determining an emotion weight array according to the emotion description text; determining a target emotion vector according to the emotion weight array and a plurality of basic emotion vectors; the target emotion vector and the content text serve as model input, first target voice is generated through a pre-trained audio synthesis model, and the first target voice comprises text content in the content text and emotion features in the emotion description text. According to the technical scheme provided by the embodiment of the invention, emotion control on the first target voice can be realized by utilizing the emotion description text, so that the voice generation stability and the emotion controllability and accuracy in the voice generation process are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of audio processing technology, and in particular to a speech generation method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Emotional speech generation is a key research area in modern speech synthesis technology. While existing speech generation methods often produce natural and fluent speech, they often lack the ability to accurately capture the emotions in the reference audio, and audio creators lack the ability to independently control the emotional attributes of the speech.

[0003] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the Invention

[0004] The embodiments of the present application provide a speech generation method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.

[0005] One aspect of an embodiment of the present application provides a speech generation method for use in a speech generation model, the method comprising: Obtain content text and sentiment description text; Determining a sentiment weight array according to the sentiment description text; Determining a target emotion vector according to the emotion weight array and a plurality of basic emotion vectors; and The target emotion vector and the content text are used as model inputs, and a first target speech is generated through a pre-trained audio synthesis model. The first target speech includes the text content in the content text and the emotion features in the emotion description text.

[0006] Optionally, the sentiment weight array includes a plurality of sentiment weight values; Determine an emotional weight array based on the emotional description text, including: Obtain multiple basic emotion types; and The emotion description text is parsed by using a pre-trained large language model to determine the emotion weight value corresponding to each of the basic emotion types in the emotion description text.

[0007] Optionally, the large language model training method includes: Obtain multiple sample sentiment description texts; The plurality of sample emotion description texts are used as model input, and the emotion distribution corresponding to each of the sample emotion description texts is output through a pre-trained first language model; Adjusting the second language model according to the plurality of sample emotion description texts and the emotion distributions corresponding to the plurality of sample emotion description texts to obtain the large language model; The first language model is a teacher model, and the second language model is a student model.

[0008] Optionally, the multiple basic emotion vectors are obtained by the following operations: Acquire multiple basic emotion types and multiple reference audios corresponding to each of the basic emotion types; Extracting multiple reference emotion vectors for each basic emotion type according to the multiple reference audios corresponding to each basic emotion type, where one reference emotion vector corresponds to one reference audio; and The basic emotion vector of each basic emotion type is determined according to a plurality of reference emotion vectors corresponding to each basic emotion type.

[0009] Optionally, the speech generation model includes an emotion feature extractor and a speaker feature extractor, and the method further includes: Obtain emotional reference audio and speaker reference audio; Extracting emotional features from the emotional reference audio by the emotional feature extractor; Extracting speaker features from the speaker reference audio by the speaker feature extractor; A second target speech is generated by the audio synthesis model according to the emotional characteristics, the speaker characteristics and the content text.

[0010] Optionally, the training method of the speech generation model includes: Obtaining a first training sample, wherein the first training sample includes a plurality of first text-speech pairs and sample speaker audio; Training a basic speech generation model using the first training sample to obtain a first speech generation model, wherein the basic speech generation model includes the speaker feature extractor; Constructing a second speech generation model based on the first language generation model and the emotion feature extractor; wherein the parameter of the speaker feature extractor in the second speech generation model is a first fixed value; Acquire a second training sample, where the second training sample includes a plurality of sample emotion audios and a plurality of sample content texts; Training the second speech generation model using the second training sample to obtain a third speech generation model; Setting the parameters of the emotion feature extractor in the third speech generation model to a second fixed value to obtain a fourth speech generation model; Acquire a third training sample, where the third training sample includes a plurality of second text-speech pairs; The fourth speech generation model is trained using the third training sample to obtain the speech generation model.

[0011] Optionally, training the second speech generation model using the second training sample to obtain a third speech generation model includes: According to the second training sample, obtaining, by the emotion feature extractor, first extracted features corresponding to each of the sample emotion audios, the first extracted features including speaker-related features and the emotion features; Inputting the first extracted features into a speaker classifier through a gradient reversal layer, so as to determine a speaker classification result according to the speaker-related features by the speaker classifier; Calculating the classification loss based on the speaker classification result and the true classification result; Determining a first classification gradient according to the classification loss, wherein the first classification gradient represents a speaker classification effect of the third speech generation model; The first classification gradient is converted into a second classification gradient by the gradient inversion layer, wherein the second classification gradient has an opposite sign to the first classification gradient; The second speech generation model is adjusted according to the second classification gradient to obtain the third speech generation model.

[0012] Another aspect of an embodiment of the present application provides a speech generation device for use in a speech generation model, the device comprising: An acquisition module is used to obtain content text and sentiment description text; A first determining module is used to determine an emotion weight array according to the emotion description text; A second determining module is configured to determine a target emotion vector based on the emotion weight array and a plurality of basic emotion vectors; and A generation module is used to take the target emotion vector and the content text as model inputs and generate a first target speech through a pre-trained audio synthesis model. The first target speech includes the text content in the content text and the emotion features in the emotion description text.

[0013] Another aspect of an embodiment of the present application provides a computer device, including: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein: the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0014] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0015] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the above-mentioned method when executed by a processor.

[0016] The embodiments of the present application using the above-mentioned technical solution may include the following advantages: speech generation is performed based on the target emotion vector determined from the emotion description text, and the generated first target speech can replicate the emotional features in the emotion description text, so that the emotion description text can be used to achieve emotional control of the first target speech, thereby improving the stability of speech generation, as well as the emotional controllability and accuracy in the speech generation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings illustrate exemplary embodiments and constitute a part of the specification. Together with the description of the specification, they serve to explain exemplary implementation of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals designate similar, but not necessarily identical, elements.

[0018] Figure 1 The following schematically shows an operating environment diagram of the speech generation method according to the first embodiment of the present application; Figure 2 The flowchart of the speech generation method according to the first embodiment of the present application is schematically shown; Figure 3 Schematically shows Figure 2 Flowchart of sub-steps of step S202; Figure 4 Schematically shows a newly added flow chart of the speech generation method according to the first embodiment of the present application; Figure 5 Schematically shows another newly added flow chart of the speech generation method according to the first embodiment of the present application; Figure 6 Schematically shows another newly added flow chart of the speech generation method according to the first embodiment of the present application; Figure 7 Schematically shows another newly added flow chart of the speech generation method according to the first embodiment of the present application; Figure 8 Schematically shows Figure 7 Flowchart of sub-steps of step S708; Figure 9 The figure schematically shows the structure of the speech generation model of the speech generation method according to the first embodiment of the present application; Figure 10 The following schematically shows an exemplary application flow chart of the speech generation method according to the first embodiment of the present application; Figure 11 A block diagram schematically shows a speech generating device according to the second embodiment of the present application; and Figure 12 The following schematically shows a hardware architecture diagram of a computer device according to the third embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of this application more clear, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0020] It should be noted that the descriptions of "first", "second", etc. in the embodiments of the present application are for descriptive purposes only and should not be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0021] It should be noted that in all aspects of this application involving data collection, storage, use, transmission, and processing, all data must strictly adhere to the laws, regulations, industry standards, and regulatory requirements of the data's source and use locations, as well as the relevant countries and regions, to ensure the legality and compliance of data activities. During the collection phase, the purpose, method, and scope of collection will be clearly stated to the data subject in a conspicuous manner. Collection will be carried out only after obtaining the data subject's legal authorization, ensuring that the collection process adheres to the "minimum necessary" principle and does not exceed the scope of data collection. During storage, the storage period will be limited, and after the storage purpose has been achieved, data will be promptly deleted, anonymized, or encrypted. During the use phase, strict data security protection mechanisms will be implemented, using field-level desensitization technology to process raw data according to pre-set desensitization rules. A variety of desensitization strategies, such as data generalization, data anonymization, and data encryption, will be employed for different data types to effectively mitigate the risk of sensitive information leakage and ensure that the data ultimately used is securely desensitized data, fully protecting the rights and interests of data subjects and data security. During the transmission and processing phases, the confidentiality and security of data will be ensured during transmission and processing.

[0022] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed. They are only used to facilitate the description of this application and to distinguish each step. Therefore, they cannot be understood as limitations on this application.

[0023] First, an explanation of the terms involved in this application is provided: Text to Speech (TTS): Also known as text-to-speech, this technology converts text input into an audio file. Audio can be output in formats such as PCM, WAV, MP3, and AAC.

[0024] Language models use unsupervised learning to learn and abstract large amounts of text, and then fine-tune them to adapt to specific NLP (natural language processing) tasks. Language models have a wide range of applications, including machine translation, speech recognition, speech generation, and text generation. The core technology behind language models is the Transformer architecture, a neural network architecture that can process contextual information in long texts.

[0025] Multimodal model: An AI system that can process and understand information from multiple different types of data sources, such as text, images, audio, and video.

[0026] Teacher Model: A complex model used to guide or assist in training a smaller or simpler model (student model) in a machine learning task.

[0027] Student Model: A concept in the field of machine learning, which refers to a simplified or optimized model. The purpose is to reduce the consumption of computing resources and improve operational efficiency while maintaining similar performance to a more complex and higher-precision teacher model.

[0028] LoRA method (Low-Rank Adaptation): is a technique for fine-tuning pre-trained models. It achieves flexible adjustment of model behavior by inserting low-rank adapters into each layer of the pre-trained model instead of directly modifying the original parameters of the model.

[0029] Gradient Reversal Layer (GRL): A neural network layer that reverses the direction of gradient propagation during training, allowing the model to suppress or decouple other irrelevant information while learning certain tasks.

[0030] Token Prediction Loss: It is a loss function in natural language processing (NLP) tasks that measures the difference between the sequence predicted by the model (such as words, characters, or phonemes) and the actual sequence.

[0031] Autoencoder: A neural network that learns to effectively represent data through unsupervised learning.

[0032] Variational Autoencoder (VAE): A generative model used to learn the latent representation of data and generate new data samples. It consists of two parts: an encoder and a decoder, and captures the latent features of the data by introducing a probability distribution.

[0033] Mel-Frequency Cepstral Coefficients (MFCCs): A feature extraction method used in speech processing and music information retrieval. By simulating the human ear's perception of sounds of different frequencies, it converts the audio signal into a set of cepstral coefficients that capture key characteristics of the speech signal, such as timbre, pitch, and formants.

[0034] L2 regularization: It is a regularization method used to prevent overfitting of machine learning models. By adding a regularization term to the loss function to limit the size of the model parameters, the model becomes more concise and stable.

[0035] Principal Component Analysis (PCA) is a statistical method that transforms a set of possibly correlated variables into a set of linearly independent variables through orthogonal transformation. These new variables are called principal components.

[0036] Zero-shot learning: is a machine learning method that allows a model to recognize or generate new categories or tasks without direct samples.

[0037] Secondly, to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the following describes the relevant technologies: The goal of emotion-controlled speech synthesis models is to convert text into natural speech with specified emotional characteristics, using natural language descriptions to guide model generation. For example, bidirectional attention and masked decoding mechanisms can be used to distinguish timbre, content, and style, and a hybrid semantic density module can be introduced to address the many-to-many style problem. Alternatively, speech with a specified emotion can be synthesized by prepending a natural language instruction and a special end token to the text to be synthesized. Natural language instructions can also be refined by specifying emotion and emphasizing information, thereby changing the language model's tokens.

[0038] However, the inventors have found that these methods are unable to directly reproduce the emotions in the reference audio, and the speech generation effect is very unstable.

[0039] To this end, the embodiment of the present application provides a speech generation technology solution. In this technology solution, (1) by combining and adjusting the weights of multiple basic emotion vectors, a variety of target emotion vectors can be generated, so that speech generation can adapt to various emotional expression needs, thereby improving the expressiveness and practicality of the speech generation process; (2) based on the obtained target emotion vector, a first target speech can be generated, and the emotion description text and content text can be combined to generate a first target speech with a specified emotion, thereby achieving zero-sample speech generation, meeting the audio creator's demand for emotion control in the speech generation process, and improving the emotion accuracy and controllability of the speech generation process; (3) by setting a speaker classifier and a gradient reversal layer, and calculating the classification loss and reversal gradient to guide the adjustment of the model parameters, the third speech generation model can gradually reduce its dependence on the speaker characteristics during the training process, realize the decoupling of emotion characteristics and speaker characteristics, and enable the final fourth speech generation model to more accurately identify the emotion characteristics in the emotion reference audio when generating speech, improve the ability to extract and apply emotion characteristics, and enhance the emotion control ability and accuracy of the speech generation process. See below for details.

[0040] Finally, for ease of understanding, an exemplary operating environment is provided below.

[0041] like Figure 1 As shown, the operating environment diagram includes: server 2, network 4, client 6, where: Server 2 can be composed of a single or multiple computing devices. These multiple computing devices can include virtualized computing instances. Virtualized computing instances can include virtual machines, such as emulations of computer systems, operating systems, servers, and the like. A computing device can load a virtual machine based on a virtual image and / or other data defining the specific software (e.g., operating system, specialized application, server) used for the emulation. As the demand for different types of processing services changes, different virtual machines can be loaded and / or terminated on one or more computing devices. A hypervisor can be implemented to manage the use of different virtual machines on the same computing device.

[0042] The server 2 can be configured to communicate with the client 6 and the like via a network 4. The network 4 includes various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or the like. The network 4 can include physical links, such as coaxial cable links, twisted pair cable links, fiber optic links, combinations thereof, and the like, or wireless links, such as cellular links, satellite links, Wi-Fi links, and the like.

[0043] Server 2 can provide storage, reading, writing, querying, and deleting services, such as providing voice generation services and model training services to clients.

[0044] Client 6 can be an electronic device running an operating system such as Windows, Android™, or iOS, such as a smartphone, tablet, laptop, virtual reality device, gaming device, set-top box, in-vehicle terminal, or smart TV. Based on these operating systems, various applications can be run, such as audio upload programs, text input programs, and voice playback programs.

[0045] The client 6 may provide / configure a user access page for manipulating the server 2 or uploading objects, etc.

[0046] It should be noted that the above devices are exemplary, and the number and type of devices can be adjusted in different scenarios or according to different needs.

[0047] The following describes the technical solutions of the present application through multiple embodiments, taking the server 2 as the execution subject. It should be noted that these embodiments can be implemented in many different forms and should not be construed as being limited to the embodiments described here.

[0048] Example 1 Figure 2 The flowchart of the speech generation method according to the first embodiment of the present application is schematically shown.

[0049] like Figure 2 As shown, the speech generation method may include steps S200 to S206, which are used in a speech generation model, wherein: Step S200, obtaining content text and emotion description text; Step S202, determining an emotion weight array according to the emotion description text; Step S204, determining a target emotion vector based on the emotion weight array and a plurality of basic emotion vectors; and Step S206: Using the target emotion vector and the content text as model inputs, a pre-trained audio synthesis model is used to generate a first target speech, where the first target speech includes the text content in the content text and the emotion features in the emotion description text.

[0050] The speech generation method provided in this embodiment performs speech generation based on the target emotion vector determined from the emotion description text. The generated first target speech can replicate the emotion features in the emotion description text, so that the emotion description text can be used to achieve emotion control of the first target speech, thereby improving the stability of speech generation, as well as the emotion controllability and accuracy in the speech generation process.

[0051] The following combination Figure 2 , each step in steps S200~S206 and other optional steps are described in detail.

[0052] Step S200 , obtain content text and sentiment description text.

[0053] Content text refers to the text to be generated into speech. It can be natural language text in various formats and languages, such as conversations, press releases, and e-books. Emotional description text can be natural language text in various languages ​​used to control the emotions expressed in the generated speech. For example, the content text could be "The weather is so nice today," while the emotional description text could be "Say this sentence in a lighthearted and cheerful tone."

[0054] In some embodiments, the content text and emotion description text may be directly input by the audio composer in text form, or the audio composer's voice instructions may be converted into content text and emotion description text using speech recognition technology, etc. The emotion description text may also be selected from a plurality of preset candidate emotion description texts.

[0055] After obtaining the content text and the sentiment description text, pre-processing operations such as text cleaning, grammar correction, and semantic understanding can be performed on the content text and the sentiment description text, such as word segmentation and part-of-speech tagging of the input content text and the sentiment description text.

[0056] In some embodiments, the emotion description text may include the emotion type and the emotion intensity level. Emotion types may include basic emotion types such as "happy," "sad," and "angry," and may also include combinations of multiple basic emotion types, such as "happy and excited" or "a little sad but hopeful." The emotion description text may also include a description of a specific emotional scene, such as "the surprise of receiving a gift at a birthday party."

[0057] In some embodiments, the emotional description text may also be obtained by analyzing the content text and extracting the emotional features thereof using a language model configured with an emotional analysis algorithm. For example, if the content text input by the audio creator is "You should never get involved in this again," the emotional features of the content text may be "anger, disgust," and the corresponding emotional description text may be generated as "Say this sentence in a tone of anger and disgust."

[0058] Step S202 , determine the emotion weight array according to the emotion description text.

[0059] The emotion weight array may include weight values ​​corresponding to a plurality of preset basic emotion types. In some embodiments, the emotion description text may be subjected to deep semantic understanding and emotion analysis through a pre-trained language model to obtain the emotion weight array.

[0060] In some embodiments, a quantitative model for sentiment intensity can be established to quantitatively evaluate the sentiment intensity in sentiment description text and determine a sentiment weight array. For example, by analyzing factors such as the intensity of sentiment words (e.g., "very happy" has a higher sentiment intensity than "happy"), the number of sentiment words, and their combination, the sentiment intensity values ​​can be determined and mapped to the sentiment weight array.

[0061] In some embodiments, when the audio creator also provides other modal information, such as emotionally charged speech examples or images, a multimodal model can be used to extract emotional features from the provided speech examples or images. These features are then combined with the sentiment analysis results of the emotional description text to jointly determine the sentiment weight array. For example, by analyzing the emotional intonation, speaking rate, volume, and other characteristics of the speech examples, the results of the textual sentiment analysis can be further verified and supplemented, improving the accuracy of the sentiment weight array.

[0062] In some embodiments, the emotional characteristics of the generated speech can be monitored using speech emotion recognition technology and compared with the target emotional characteristics. Based on the comparison results of the emotional characteristics, the emotional weight array can be adjusted in real time to optimize subsequent speech generation.

[0063] In this embodiment, the emotional description text is converted into an array of emotional weights, which quantifies the emotional characteristics of the emotional description text, thereby determining the specific emotional needs and expectations of the audio creator. Generating audio based on the emotional weight array can improve the emotional accuracy and controllability of the generated audio, meeting the diverse needs of audio creators for speech emotion in different scenarios.

[0064] The above embodiments have mentioned various methods for determining the emotion weight array based on the emotion description text. An exemplary solution is provided below.

[0065] In an optional embodiment, if Figure 3 As shown, the emotion weight array includes multiple emotion weight values, and step S202 includes: S300: Acquire multiple basic emotion types.

[0066] S302: Parse the emotion description text using a pre-trained large language model to determine the emotion weight value corresponding to each of the basic emotion types in the emotion description text.

[0067] Basic emotion types may include "happy," "sad," "angry," "fear," "surprise," "disgust," and "neutral." More detailed emotion types may also be included, such as "joy," "pleasure," "sadness," "melancholy," "rage," "fear," "expectation," and "trust."

[0068] In some embodiments, the basic emotion type set can be dynamically adjusted based on the context or content, determining the actual application scenario of the generated speech, etc. For example, if the content text entered by the audio creator is a poem, special emotion types related to literary emotions such as "lyrical," "passionate," and "euphemistic" can be added to the basic emotion types.

[0069] In some embodiments, an emotion intensity assessment model equipped with an emotion intensity dictionary can be used to label the emotion words according to their intensity in the emotion description text (e.g., "very happy" has a higher intensity than "happy"). These intensity labels can then be weighted to calculate the emotion weight value.

[0070] In some embodiments, multiple pre-trained language models may be used to analyze the sentiment description text, and the sentiment weight prediction results of different models may be fused through weighted averaging or voting mechanisms to determine a sentiment weight array.

[0071] In this embodiment, the emotional features in the emotional description text are converted into emotional weight values ​​of each basic emotional type, which can quantify and concretize the emotions in the text, improve the structural degree of speech generation, and enhance the accuracy and emotional controllability of the generated speech.

[0072] The previous article introduced a method for determining sentiment weight arrays using a large language model. There are various training methods for large language models. The following provides an exemplary training method.

[0073] In an optional embodiment, if Figure 4 As shown, the training method of the large language model includes: S402: Acquire multiple sample emotion description texts.

[0074] S404: Using the plurality of sample emotion description texts as model input, outputting emotion distribution corresponding to each of the sample emotion description texts through a pre-trained first language model.

[0075] S406: Adjust the second language model according to the multiple sample emotion description texts and the emotion distributions corresponding to the multiple sample emotion description texts to obtain the large language model, wherein the first language model is a teacher model and the second language model is a student model.

[0076] Sample sentiment descriptions can cover a variety of topics, styles, and sentiment types. For example, they can be scripted (e.g., "I'm so excited") or descriptive (e.g., "You feel angry and betrayed"). These can be obtained from authorized public sentiment corpora, collected through manual sentiment annotation, or generated using large-scale language models based on pre-defined sentiment types.

[0077] In some embodiments, the sentiment distribution may be the weight distribution of different sentiment types in the sentiment features of the sample sentiment description text. The sentiment distribution may also be the probability distribution of the sample sentiment description text belonging to different sentiment types expressed in the form of a probability vector.

[0078] The first language model (teacher model) can be a large-scale language model with a multi-layer encoder and decoder structure, and the second language model (student model) can be a small-scale language model with fewer encoder and decoder structures.

[0079] In some embodiments, the sentiment distributions output by the student model and the teacher model can be compared using a knowledge distillation loss function, and the student model parameters can be adjusted by minimizing the difference between the two sentiment distributions to obtain a large language model. During the adjustment process, low-rank adapters can be introduced into the student model layers, and some parameters of the student model can be adjusted using the LoRA method.

[0080] In this embodiment, the emotion distribution obtained by the larger-scale teacher model is used to adjust the smaller-scale student model to obtain a large language model. A lightweight and efficient large language model can be trained, which enables it to accurately and quickly extract emotion weight values ​​from emotion description texts, thereby improving the emotion accuracy and efficiency of speech generation.

[0081] Step S204 , determine the target emotion vector according to the emotion weight array and multiple basic emotion vectors.

[0082] In some embodiments, the basic emotion vectors may be predefined vectors extracted from a large-scale emotion dictionary, with each basic emotion vector corresponding to a specific emotion type, such as happiness, sadness, anger, etc. The basic emotion vectors may also be obtained by unsupervised or supervised learning of a large amount of emotion-labeled sample data using an autoencoder or a variational autoencoder (VAE).

[0083] In some embodiments, a target emotion vector can be obtained by weightedly summing the base emotion vector with the corresponding emotion weight values ​​in the emotion weight array. Alternatively, the base emotion vector and the weights can be nonlinearly processed using a multi-layer perceptron (MLP) to combine and obtain the target emotion vector. Alternatively, a self-attention mechanism can be used to calculate attention weights between the base emotion vectors, and then weighted combination can be performed based on these weights.

[0084] In some embodiments, after determining the target emotion vector, it is further possible to verify whether it is consistent with the emotion description text. If not, the weight array or the basic emotion vector can be adjusted.

[0085] In this embodiment, by combining and adjusting the weights of multiple basic emotion vectors, a variety of target emotion vectors can be generated, so that speech generation can adapt to various emotional expression needs, improving the expressiveness and practicality of the speech generation process, and meeting the diverse needs of audio creators for speech emotion control.

[0086] As mentioned above, there are many sources of basic emotion vectors. The following provides an exemplary method for obtaining basic emotion vectors.

[0087] In an optional embodiment, if Figure 5As shown in Figure 2, multiple basic emotion vectors are obtained through the following operations: S500: Acquire multiple basic emotion types and multiple reference audios corresponding to each of the basic emotion types.

[0088] S502 : Extracting multiple reference emotion vectors for each basic emotion type according to the multiple reference audios corresponding to each basic emotion type, where one reference emotion vector corresponds to one reference audio.

[0089] S504: Determine a basic emotion vector for each basic emotion type according to a plurality of reference emotion vectors corresponding to each basic emotion type.

[0090] Basic emotion types may include "happy," "sad," "angry," "fear," "surprise," "disgust," and "neutral." More detailed emotion types may also be included, such as "joy," "pleasure," "sadness," "melancholy," "rage," "fear," "expectation," and "trust."

[0091] Reference audio can be obtained from various sources such as corpora, film and television dramas, and radio dramas, and classified according to basic emotion types using human or emotion recognition models. Reference audio can also be audio recorded from text written by humans based on the corresponding basic emotion types using human or large-scale language models.

[0092] In some embodiments, after obtaining the reference audio, it can also be screened based on the clarity of the reference audio, the purity of emotional expression, the speaker's speaking speed and tone, the audio length, etc.

[0093] In some embodiments, a reference emotion vector can be extracted from the reference audio using acoustic feature extraction methods such as Mel-Frequency Cepstral Coefficients (MFCC). Alternatively, the reference audio can be feature-encoded using convolutional neural networks (CNNs) or recurrent neural networks (RNNs), automatically learning more discriminative emotion feature representations to construct the reference emotion vector.

[0094] In some embodiments, dimensionality reduction algorithms such as principal component analysis (PCA) and linear discriminant analysis (LDA) can be used to reduce the dimensionality of the extracted high-dimensional features and remove redundant information and noise. Evaluation metrics such as information gain and chi-square tests can also be used to select the most relevant feature subsets, improving the compactness and effectiveness of the reference sentiment vector.

[0095] In this embodiment, by extracting reference emotion vectors from real reference audio and synthesizing basic emotion vectors, the obtained basic emotion vectors can accurately and truly represent the emotional characteristics of the corresponding emotion type, thereby improving the emotional accuracy and authenticity of speech generation.

[0096] Step S206 , taking the target emotion vector and the content text as model input, generating a first target speech through a pre-trained audio synthesis model, wherein the first target speech includes the text content in the content text and the emotion features in the emotion description text.

[0097] In some embodiments, the target emotion vector and text features of the content text may be arranged and combined in a certain order to form a unified feature sequence, which is then input into an audio synthesis model to generate the first target speech.

[0098] In some embodiments, an attention mechanism can also be used to dynamically adjust the weights of emotional features at different locations in the text based on the correlation between different parts of the text and emotion, enabling the audio synthesis model to more accurately express emotion when generating the first target speech. For example, higher emotional feature weights can be assigned to words with strong emotional connotations to enhance the emotional expression at corresponding locations in the speech.

[0099] In some embodiments, the speaker's timbre, intonation, speaking speed and other style features can also be used as model inputs, combined with the target emotion vector and content text to generate a first target speech, so that the first target speech has both the specified emotion and the speaker's style.

[0100] In some embodiments, after generating the first target speech, the audio effects device can be used to adjust parameters such as the pitch, volume, and speaking speed of the speech to enhance the emotion of the generated speech. For example, for a happy emotion, the pitch and speaking speed can be appropriately increased, and the volume fluctuations can be increased; for a sad emotion, the pitch and speaking speed can be lowered, and the volume can be reduced.

[0101] In this embodiment, the first target speech is generated based on the acquired target emotion vector. The emotion description text and the content text can be combined to generate the first target speech with the specified emotion, thereby realizing zero-sample speech generation, meeting the audio creator's needs for emotion control in the speech generation process, and improving the emotional accuracy and controllability of the speech generation process.

[0102] The above describes a method for implementing emotion control during speech generation using input emotion description text. In actual applications, emotion control during speech generation can also be implemented through other methods. The following provides an exemplary emotion control method.

[0103] In an optional embodiment, the speech generation model includes an emotion feature extractor and a speaker feature extractor, such as Figure 6 As shown, the method further includes: S600: Acquire emotion reference audio and speaker reference audio.

[0104] S602: Extract emotional features from the emotional reference audio using the emotional feature extractor.

[0105] S604: Extract speaker features from the speaker reference audio using the speaker feature extractor.

[0106] S606: Generate a second target speech through the audio synthesis model according to the emotional feature, the speaker feature and the content text.

[0107] Emotional reference audio can come from professional emotional speech libraries, dialogue clips from movies and TV series, radio dramas, and custom recorded emotional speech. Speaker reference audio can come from the target speaker's professional dubbing works, radio programs, speech videos, and voice clips of specific characters specified by the audio creator.

[0108] In some embodiments, the acquired emotion reference audio and speaker reference audio may be pre-processed to remove interfering factors such as background noise and interfering noise. Voice activity detection may also be performed on the speaker reference audio to determine the speech portion and remove silent segments and non-speech portions.

[0109] In some embodiments, the emotion feature extractor may extract emotion features using various emotion feature extraction algorithms, such as Mel-Frequency Cepstral Coefficient (MFCC)-based emotion feature extraction. Alternatively, the emotion feature extractor may learn emotion feature representations from emotion reference audio using deep learning-based feature extraction models, such as convolutional neural networks (CNNs) or recurrent neural networks (RNNs), to obtain emotion features.

[0110] In some embodiments, a speaker feature extractor built using a deep neural network (DNN) or a variational autoencoder (VAE) can be used to extract speaker features such as timbre and intonation from the speaker reference audio. Multi-scale feature extraction can also be used to extract speaker features at different time and frequency scales.

[0111] In some embodiments, when the emotion reference audio is not obtained, the emotion features can be directly extracted from the speaker reference audio to generate the second target speech.

[0112] In this embodiment, the generation of the second target speech is guided by the emotional features extracted from the emotional reference audio, so that the second target language can replicate the emotions in the emotional reference audio, and the emotional reference audio can be used to achieve emotional control of the second target speech, thereby improving the emotional controllability and accuracy in the speech generation process.

[0113] The above article introduces a method for generating a second target speech using a language generation model. There are many training methods for the language generation model. The following provides an exemplary training method.

[0114] In an optional embodiment, if Figure 7 As shown, the training method of the speech generation model includes: S700: Obtain a first training sample, where the first training sample includes a plurality of first text-speech pairs and sample speaker audio.

[0115] S702: Train a basic speech generation model using the first training sample to obtain a first speech generation model, where the basic speech generation model includes the speaker feature extractor.

[0116] S704: Construct a second speech generation model based on the first language generation model and the emotion feature extractor; wherein the parameters of the speaker feature extractor in the second speech generation model are first fixed values.

[0117] S706: Obtain a second training sample, where the second training sample includes a plurality of sample emotion audios and a plurality of sample content texts.

[0118] S708: Train the second speech generation model using the second training sample to obtain a third speech generation model.

[0119] S710: Set the parameters of the emotion feature extractor in the third speech generation model to second fixed values ​​to obtain a fourth speech generation model.

[0120] S712: Acquire a third training sample, where the third training sample includes a plurality of second text-speech pairs.

[0121] S714: Train the fourth speech generation model using the third training sample to obtain the speech generation model.

[0122] The training samples (including the first training sample, the second training sample, and the third training sample) may include text and audio data collected based on different languages, accents, emotional types, and speaker styles. In some embodiments, the training samples may be augmented by data enhancement techniques such as changing speech rate, pitch, and adding background noise.

[0123] In some embodiments, training efficiency and model performance can be improved through adaptive learning rate adjustment, distributed training, etc. Regularization techniques such as L2 regularization can also be used to prevent model overfitting and improve model generalization.

[0124] In this embodiment, first, by training the basic speech generation model, a first speech generation model with basic speech generation capabilities and speaker feature extraction capabilities can be obtained; then, by training the second speech generation model that freezes the speaker feature extractor on the basis of the first speech generation model and introduces the emotion feature extractor to obtain the third speech generation model, the sensitivity of the third speech generation model to the emotional features in the sample emotional audio can be improved and the emotion features can be decoupled from the speaker features, thereby avoiding mistaking the speaker features for the emotion features; finally, by training the fourth speech generation model that freezes the emotion feature extractor and the speaker feature extractor on the basis of the third speech model through big data samples, the audio generation capability of the speech generation model can be further optimized and the overall quality of speech generation can be improved.

[0125] The previous article introduced a method for training a speech generation model. In practical applications, various optimization methods can be used to further optimize the training process. The following provides an exemplary method.

[0126] In an optional embodiment, if Figure 8 As shown, step S708 includes: S800: According to the second training sample, obtain, by the emotion feature extractor, first extracted features corresponding to each of the sample emotion audios, where the first extracted features include speaker-related features and the emotion features.

[0127] S802: Input the first extracted features into a speaker classifier through a gradient reversal layer, so that the speaker classifier determines a speaker classification result according to the speaker-related features.

[0128] S804: Calculate the classification loss based on the speaker classification result and the true classification result.

[0129] S806: Determine a first classification gradient based on the classification loss, where the first classification gradient represents a speaker classification effect of the third speech generation model.

[0130] S808 , converting the first classification gradient into a second classification gradient through the gradient inversion layer, where the second classification gradient has a sign opposite to that of the first classification gradient.

[0131] S810: Adjust the second speech generation model according to the second classification gradient to obtain the third speech generation model.

[0132] The first extracted feature may be a fixed-length vector or a multidimensional matrix. The speaker feature may include the speaker's timbre, pronunciation habits, etc. In some embodiments, the first extracted feature may also include speech rate features, voice intonation features, volume features, etc.

[0133] Speaker classifiers can be constructed using deep learning structures such as attention mechanisms to improve speaker classification accuracy. In some embodiments, classification loss can be calculated using functions such as cross-entropy loss, homogeneity loss, or focal loss. The classification loss calculation process and formula can also be adjusted based on the training progress or classification difficulty.

[0134] In some embodiments, the coefficient of gradient reversal can be dynamically adjusted to adjust the degree of gradient reversal based on the classification effect and emotional feature extraction effect during the training process. It is also possible to use a smaller degree of reversal at the beginning of training, and gradually increase the degree of reversal as the training progresses, so that the third speech generation model can gradually adapt to the impact of gradient reversal, thereby achieving more effective feature decoupling. Of course, in addition to using the second classification gradient, additional data such as the token prediction loss generated by the second language model when predicting tokens can also be used to adjust the second language model.

[0135] In some embodiments, different parameter fine-tuning strategies can be used to adjust the third speech generation model based on the second classification gradient. For example, parameters related to emotion feature extraction can be fine-tuned using a smaller learning rate to preserve the learned emotion feature representation; whereas parameters related to speaker feature extraction can be fine-tuned using a larger learning rate to more thoroughly remove the influence of speaker features.

[0136] In this embodiment, by setting a speaker classifier and a gradient reversal layer, and calculating the classification loss and the reversal gradient to guide the adjustment of the model parameters, the third speech generation model can gradually reduce its dependence on the speaker characteristics during the training process, realize the decoupling of the emotional characteristics and the speaker characteristics, so that the fourth speech generation model finally obtained can more accurately identify the emotional characteristics in the emotional reference audio when generating speech, improve the extraction and application capabilities of emotional characteristics, and enhance the emotional control ability and accuracy of the speech generation process. The structure of the speech generation model in this embodiment can be as follows Figure 9 As shown in the figure, Emo perceiver Conditioner is the emotion feature extractor, GRL is the gradient reversal layer, SpeakerClassifer is the speaker classifier, and Spk perceiver Conditioner is the speaker feature extractor.

[0137] In order to make this application easier to understand, the following Figure 10 An exemplary application is provided, wherein: S11: The audio creator input a text "The weather is so nice today" (i.e., content text) and an emotional requirement "Please say this in a relaxed and happy tone" (i.e., emotional description text) into the speech generation model; S12. The speech generation model determines that the emotion weight array of the emotion contained in the input text is: "Anger: 0, Happiness: 0.8, Fear: 0, Disgust: 0, Sadness: 0, Surprise: 0.05, Neutral: 0.15"; S13, the speech generation model performs weighted calculation based on the acquired emotion weight array and the pre-acquired basic emotion vectors of each basic emotion to obtain the target emotion vector emovec required by the emotion; S14, the speech generation model generates a speech A (i.e., the first target speech) based on the target emotion vector emovec and the text "The weather is really nice today"; S15. The audio creator inputs an audio clip of anime character M, A1 (the speaker reference audio), and an audio clip with a lighthearted tone, A2 (the emotion reference audio). S16, the speech generation model extracts intonation features and timbre features (i.e., speaker features) of the cartoon character M from the audio A1 through the speaker feature extractor; S17, the speech generation model extracts the emotional feature "relaxed and happy" from audio A2 through the emotional feature extractor; S18. The speech generation model generates speech for the text "The weather is so nice today" based on the extracted emotional features, intonation features, and timbre features, and obtains a speech B (i.e., the second target speech) that is pronounced in a light-hearted and cheerful tone in the style of the anime character M.

[0138] Example 2 Figure 11 The block diagram of the speech generating device according to the second embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Figure 11 As shown, the apparatus 1000 may include: an acquisition module 1100, a first determination module 1200, a second determination module 1300, and a generation module 1400, wherein: An acquisition module 1100 is used to acquire content text and emotion description text; A first determining module 1200 is configured to determine an emotion weight array based on the emotion description text; A second determining module 1300 is configured to determine a target emotion vector based on the emotion weight array and a plurality of basic emotion vectors; and Generation module 1400 is used to take the target emotion vector and the content text as model inputs and generate a first target speech through a pre-trained audio synthesis model. The first target speech includes the text content in the content text and the emotion features in the emotion description text.

[0139] As an optional embodiment, the emotion weight array includes multiple emotion weight values, and the first determining module 1200 is further configured to: Obtain multiple basic emotion types; and The emotion description text is parsed by a pre-trained large language model to determine the emotion weight value corresponding to each of the basic emotion types in the emotion description text.

[0140] As an optional embodiment, the apparatus 1000 further includes a large language model training module, configured to: Obtain multiple sample sentiment description texts; The plurality of sample emotion description texts are used as model input, and the emotion distribution corresponding to each of the sample emotion description texts is output through a pre-trained first language model; Adjusting the second language model according to the plurality of sample emotion description texts and the emotion distributions corresponding to the plurality of sample emotion description texts to obtain the large language model; The first language model is a teacher model, and the second language model is a student model.

[0141] As an optional embodiment, the apparatus 1000 further includes a basic emotion vector acquisition module, configured to: Acquire multiple basic emotion types and multiple reference audios corresponding to each of the basic emotion types; Extracting multiple reference emotion vectors for each basic emotion type according to the multiple reference audios corresponding to each basic emotion type, where one reference emotion vector corresponds to one reference audio; and The basic emotion vector of each basic emotion type is determined according to a plurality of reference emotion vectors corresponding to each basic emotion type.

[0142] As an optional embodiment, the speech generation model includes an emotion feature extractor and a speaker feature extractor, and the apparatus 1000 further includes a second target speech generation module for: Obtain emotional reference audio and speaker reference audio; Extracting emotional features from the emotional reference audio by the emotional feature extractor; Extracting speaker features from the speaker reference audio by the speaker feature extractor; A second target speech is generated by the audio synthesis model according to the emotional characteristics, the speaker characteristics and the content text.

[0143] As an optional embodiment, the apparatus 1000 further includes a speech generation model training module, configured to: Obtaining a first training sample, wherein the first training sample includes a plurality of first text-speech pairs and sample speaker audio; Training a basic speech generation model using the first training sample to obtain a first speech generation model, wherein the basic speech generation model includes the speaker feature extractor; Constructing a second speech generation model based on the first language generation model and the emotion feature extractor; wherein the parameter of the speaker feature extractor in the second speech generation model is a first fixed value; Acquire a second training sample, where the second training sample includes a plurality of sample emotion audios and a plurality of sample content texts; Training the second speech generation model using the second training sample to obtain a third speech generation model; Setting the parameters of the emotion feature extractor in the third speech generation model to a second fixed value to obtain a fourth speech generation model; Acquire a third training sample, where the third training sample includes a plurality of second text-speech pairs; The fourth speech generation model is trained using the third training sample to obtain the speech generation model.

[0144] As an optional embodiment, the large language model training module is further used to: According to the second training sample, obtaining, by the emotion feature extractor, first extracted features corresponding to each of the sample emotion audios, the first extracted features including speaker-related features and the emotion features; Inputting the first extracted features into a speaker classifier through a gradient reversal layer, so as to determine a speaker classification result according to the speaker-related features by the speaker classifier; Calculating the classification loss based on the speaker classification result and the true classification result; Determining a first classification gradient according to the classification loss, wherein the first classification gradient represents a speaker classification effect of the third speech generation model; The first classification gradient is converted into a second classification gradient by the gradient inversion layer, wherein the second classification gradient has an opposite sign to the first classification gradient; The second speech generation model is adjusted according to the second classification gradient to obtain the third speech generation model.

[0145] Example 3 Figure 12 The following schematically shows a hardware architecture diagram of a computer device 10000 suitable for implementing a speech generation method according to the third embodiment of the present application. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle-mounted terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server or a server cluster composed of multiple servers), etc. Figure 12 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other via a system bus. Memory 10010 includes at least one type of computer-readable storage medium, including flash memory, a hard disk, a multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, a magnetic disk, an optical disk, and the like. In some embodiments, memory 10010 may be an internal storage module of computer device 10000, such as a hard disk or memory of computer device 10000. In other embodiments, memory 10010 may also be an external storage device of computer device 10000, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, and the like equipped on computer device 10000. Of course, memory 10010 may also include both internal storage modules and external storage devices of computer device 10000. In this embodiment, the memory 10010 is generally used to store the operating system and various application software installed on the computer device 10000, such as the program code of the speech generation method, etc. In addition, the memory 10010 can also be used to temporarily store various data that has been output or will be output.

[0146] In some embodiments, processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. Processor 10020 is typically used to control the overall operation of computer device 10000, such as performing control and processing related to data exchange or communication with computer device 10000. In this embodiment, processor 10020 is used to execute program code stored in memory 10010 or process data.

[0147] Network interface 10030 may include a wireless network interface or a wired network interface. Network interface 10030 is typically used to establish a communication link between computer device 10000 and other computer devices. For example, network interface 10030 is used to connect computer device 10000 to an external terminal via a network, establishing a data transmission channel and a communication link between computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile Communications (GSM), Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth, or Wi-Fi.

[0148] It should be pointed out that Figure 12 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementing all of the shown components is not a requirement, and more or fewer components may alternatively be implemented.

[0149] In this embodiment, the speech generation method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.

[0150] Example 4 An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the speech generation method in the embodiment are implemented.

[0151] In this embodiment, computer-readable storage media include flash memory, hard disks, multimedia cards, card-type memories (e.g., SD or DX memories), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disks, optical disks, and the like. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as the computer device's hard disk or memory. In other embodiments, the computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, and the like. Of course, the computer-readable storage medium may also include both the internal storage unit and external storage devices of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the speech generation method described in the embodiment. In addition, the computer-readable storage medium may also be used to temporarily store various types of data that has been output or is about to be output.

[0152] Example 5 An embodiment of the present application further provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0153] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented using general-purpose computer devices. They can be concentrated on a single computer device or distributed on a network composed of multiple computer devices. Alternatively, they can be implemented using program codes executable by the computer device, so that they can be stored in a storage device and executed by the computer device. In some cases, the steps shown or described can be performed in a different order than herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0154] It should be noted that the above are only preferred embodiments of the present application and do not limit the scope of patent protection of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the scope of patent protection of the present application.

Claims

1. A speech generation method, characterized in that: Used in a speech generation model, the method includes: Obtain content text and sentiment description text; Determining a sentiment weight array according to the sentiment description text; Determining a target emotion vector according to the emotion weight array and a plurality of basic emotion vectors; and The target emotion vector and the content text are used as model inputs, and a first target speech is generated through a pre-trained audio synthesis model. The first target speech includes the text content in the content text and the emotion features in the emotion description text.

2. The method according to claim 1, characterized in that The sentiment weight array includes a plurality of sentiment weight values; Determine an emotional weight array based on the emotional description text, including: Obtain multiple basic emotion types; and The emotion description text is parsed by a pre-trained large language model to determine the emotion weight value corresponding to each of the basic emotion types in the emotion description text.

3. The method according to claim 2, characterized in that The training method of the large language model includes: Obtain multiple sample sentiment description texts; The plurality of sample emotion description texts are used as model input, and the emotion distribution corresponding to each of the sample emotion description texts is output through a pre-trained first language model; Adjusting the second language model according to the plurality of sample emotion description texts and the emotion distributions corresponding to the plurality of sample emotion description texts to obtain the large language model; The first language model is a teacher model, and the second language model is a student model.

4. The method according to claim 1, wherein The multiple basic emotion vectors are obtained by the following operations: Acquire multiple basic emotion types and multiple reference audios corresponding to each of the basic emotion types; Extracting multiple reference emotion vectors for each basic emotion type according to the multiple reference audios corresponding to each basic emotion type, where one reference emotion vector corresponds to one reference audio; and The basic emotion vector of each basic emotion type is determined according to a plurality of reference emotion vectors corresponding to each basic emotion type.

5. The method according to claim 1, wherein The speech generation model includes an emotion feature extractor and a speaker feature extractor, and the method further includes: Obtain emotional reference audio and speaker reference audio; Extracting emotional features from the emotional reference audio by the emotional feature extractor; Extracting speaker features from the speaker reference audio by the speaker feature extractor; A second target speech is generated by the audio synthesis model according to the emotional characteristics, the speaker characteristics and the content text.

6. The method according to claim 5, characterized in that The training method of the speech generation model includes: Obtaining a first training sample, wherein the first training sample includes a plurality of first text-speech pairs and sample speaker audio; Training a basic speech generation model using the first training sample to obtain a first speech generation model, wherein the basic speech generation model includes the speaker feature extractor; Constructing a second speech generation model based on the first language generation model and the emotion feature extractor; wherein the parameter of the speaker feature extractor in the second speech generation model is a first fixed value; Acquire a second training sample, where the second training sample includes a plurality of sample emotion audios and a plurality of sample content texts; Training the second speech generation model using the second training sample to obtain a third speech generation model; Setting the parameters of the emotion feature extractor in the third speech generation model to a second fixed value to obtain a fourth speech generation model; Acquire a third training sample, where the third training sample includes a plurality of second text-speech pairs; The fourth speech generation model is trained using the third training sample to obtain the speech generation model.

7. The method according to claim 6, characterized in that The second speech generation model is trained using the second training sample to obtain a third speech generation model, including: According to the second training sample, obtaining, by the emotion feature extractor, first extracted features corresponding to each of the sample emotion audios, the first extracted features including speaker-related features and the emotion features; Inputting the first extracted features into a speaker classifier through a gradient reversal layer, so as to determine a speaker classification result according to the speaker-related features by the speaker classifier; Calculating the classification loss based on the speaker classification result and the true classification result; Determining a first classification gradient according to the classification loss, wherein the first classification gradient represents a speaker classification effect of the third speech generation model; The first classification gradient is converted into a second classification gradient by the gradient inversion layer, wherein the second classification gradient has an opposite sign to the first classification gradient; The second speech generation model is adjusted according to the second classification gradient to obtain the third speech generation model.

8. A speech generating device, characterized in that: Used in a speech generation model, the device comprises: An acquisition module is used to obtain content text and sentiment description text; A first determining module is used to determine an emotion weight array according to the emotion description text; A second determining module is configured to determine a target emotion vector based on the emotion weight array and a plurality of basic emotion vectors; and A generation module is used to take the target emotion vector and the content text as model inputs and generate a first target speech through a pre-trained audio synthesis model. The first target speech includes the text content in the content text and the emotion features in the emotion description text.

9. A computer device, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claims 1 to 7 are implemented.

Images