Voice generation method and device, electronic equipment and computer readable storage medium
By introducing supervision training on phoneme duration and sample speech characteristics in the TTS technology, the problem of difficulty in identifying speech emotions in the prior art is solved, and natural emotional speech generation and diverse expression needs are achieved.
Patent Information
- Application Number
- CN202410814374.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-21
- Publication Date
- 2024-11-19
AI Technical Summary
The existing TTS technology requires a large amount of labeled data when converting emotional text to speech, and it is difficult to accurately identify and process the emotional information of the speech itself, resulting in poor emotional control effects and inability to adapt to diverse emotional expression needs.
通过确定目标声音特征信息、目标情感信息和目标文本,利用基于音素持续时长和样本语音特征的语音生成模型进行训练,生成目标语音,结合样本声音特征和样本情感信息进行监督训练,确保语音生成模型 Ability to accurately identify and process emotional information in text.
实现了生成自然的情感表达的高质量语音,适应多样化的情感表达需求,提高了情感语音的生成效果。
Smart Images

Figure CN118982977A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of Internet technology, and in particular to a speech generation method, device, electronic device and computer-readable storage medium. Background Art
[0002] TTS (Text To Speech) technology, as a technology that can convert text information into voice, has been widely used in various fields. For example, in the gaming field, by converting game guides, navigation information and plot dialogues into voice, a more intuitive guidance effect and more natural communication between players and NPCs can be achieved, thereby improving the gaming experience.
[0003] In the related technology, when converting text information based on TTS technology, machine learning algorithms such as support vector machines and naive Bayes can be used to analyze the sentiment of the text and generate corresponding speech; or, deep learning models such as CNN (Convolutional Neural Networks), RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory) or Transformers can be used to extract the sentiment features of the text, and use speech synthesis technology to convert the sentiment features into speech.
[0004] In the process of implementing the relevant technology, the applicant found that the relevant technology has at least the following problems:
[0005] The emotional text-to-speech solution that uses machine learning algorithms and deep learning models requires a large amount of labeled data for training, and simply mining the emotional information of the text tends to ignore the emotional information contained in the speech itself, which to a certain extent affects the effect of emotional control. It is difficult to accurately identify and process the emotional information in the text, and it is also unable to adapt to the diverse emotional expression needs. Summary of the invention
[0006] In view of this, the present application provides a speech generation method, device, electronic device and computer-readable storage medium.
[0007] According to a first aspect of the present application, a speech generation method is provided, the method comprising:
[0008] Determine target sound feature information, target emotion information and target text to be converted into speech;
[0009] The target text, the target sound feature information and the target emotion information are trained by using a speech generation model to obtain a target speech; wherein the speech generation model is obtained by training a training model based on the phoneme duration corresponding to the sample text and the sample speech feature information corresponding to the sample speech; the sample speech feature information includes at least one of the following: sample sound feature information and sample emotion information;
[0010] The target speech is output.
[0011] According to a second aspect of the present application, a speech generation device is provided, the device comprising:
[0012] A determination module, used to determine target sound feature information, target emotion information and target text to be converted into speech;
[0013] A training module is used to train the target text, the target sound feature information and the target emotion information using a speech generation model to obtain a target speech; wherein the speech generation model is obtained by training the training model based on the phoneme duration corresponding to the sample text and the sample speech feature information corresponding to the sample speech; the sample speech feature information includes at least one of the following: sample sound feature information and sample emotion information;
[0014] An output module is used to output the target speech.
[0015] According to a third aspect of the present application, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the methods described in the first aspect when executing the computer program.
[0016] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.
[0017] By means of the above technical scheme, the present application provides a speech generation method, device, electronic device and computer-readable storage medium. The present application determines the target sound feature information, target emotional information and target text to be converted into speech, and uses the speech generation model obtained by training the training model based on the phoneme duration corresponding to the sample text and the sample speech feature information corresponding to the sample speech to train the target text, target sound feature information and target emotional information to obtain the target speech and output the target speech. In the process of constructing the speech generation model, not only the sample text is used for training, but also the phoneme duration and sample speech feature information describing the characteristics of the speech are additionally introduced for supervised training at the same time, so that the speech generation model can accurately recognize and process the emotional information in the text, and can generate relatively natural high-quality speech of a specific speaker with emotional expression when performing speech conversion, so as to achieve a better generation effect of emotional speech, thereby adapting to diverse emotional expression needs.
[0018] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0020] Figure 1 A flow chart of a speech generation method provided by an embodiment of the present application is shown;
[0021] Figure 2A A schematic diagram of another speech generation method provided in an embodiment of the present application is shown;
[0022] Figure 2B A structural schematic diagram of a language emotion model structure provided in an embodiment of the present application is shown;
[0023] Figure 2C A schematic diagram of the structure of a Variance Adaptor provided in an embodiment of the present application is shown;
[0024] Figure 2D A schematic diagram of the structure of a Variance Adaptor provided in an embodiment of the present application is shown;
[0025] Figure 2EA schematic diagram of the structure of a Predictor provided in an embodiment of the present application is shown;
[0026] Figure 3 A schematic diagram of the structure of a speech generating device provided in an embodiment of the present application is shown;
[0027] Figure 4 A schematic diagram of the device structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0028] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.
[0029] The applicant recognizes that the current traditional TTS solution has become mature, and the key lies in achieving more natural emotional control of generated speech. For emotional speech, intuitively, speech with different emotions will also have different reflections on different acoustic features. For example, angry speech will have higher energy and pitch than normal speech, and sad speech will have lower energy and pitch than normal speech. In this regard, the present application not only uses sample text for training, but also introduces additional energy information, pitch information and phoneme duration of phonemes that describe the characteristics of speech for supervised training at the same time, so as to achieve better emotional speech generation effect.
[0030] The present application embodiment provides a method for generating speech, such as Figure 1 As shown, the method includes:
[0031] 101. Determine target sound feature information, target emotion information, and target text to be converted into speech.
[0032] The embodiments of the present application can be applied to servers and / or terminals. The server can be an independent server or a server of basic cloud computing such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, so as to provide text-to-speech services for players in the game and achieve the generation of more natural emotional speech while taking into account a higher speech generation speed.
[0033] When there is a conversion demand, a speech generation instruction can be received, which will indicate which text needs to be converted into speech, and the text is also the target text. The target text is the text that needs to be converted into speech input by the user; the target text can be a character, a word, a sentence or a paragraph of text; the target text can be in different languages such as Chinese, English, Japanese, etc., and the present application does not specifically limit the attributes of the target text. In addition, the user can be a developer of the application, so that the developer can implant the speech obtained by text conversion into the application and use the voice to interact with the user of the application; or the user can also be an individual user such as a game player to meet the diverse needs of individual users. Furthermore, since the user will set the sound feature information to specify the timbre when performing speech conversion, and set the emotional information to specify the emotion, it is necessary to determine the target sound feature information and target emotional information to be converted into speech. Among them, the target sound feature information is used to indicate the timbre characteristics of the target speech obtained by conversion. Since the timbre of each speaker is different, the target sound feature information can be specified by specifying the speaker. The target emotional information is used to indicate the emotional characteristics of the converted target speech, which can be happiness, anger, anxiety, etc. Subsequently, the target text needs to be converted into speech with reference to the target sound feature information and the target emotional information to ensure that the timbre and emotion of the converted target speech meet the conversion requirements.
[0034] 102. The target text, target sound feature information and target emotion information are trained using a speech generation model to obtain the target speech.
[0035] In the embodiment of the present application, in order to realize the conversion of text into speech, a speech generation model is provided. The speech generation model is obtained by training the training model based on the duration of the phonemes corresponding to the sample text and the sample speech feature information corresponding to the sample speech. The sample speech feature information includes at least one of the following: sample sound feature information and sample emotion information.
[0036] It should be noted that the sample text and the sample voice are corresponding, that is, each sample text corresponds to a sample voice, so as to realize the conversion from text to voice. Among them, the sample text can be a single character, a single word or a sentence. For example, a single character "玩 (wán)" can be used as a sample text, and the sample voice corresponding to this sample text is the audio with the pronunciation of "wán"; or a word "玩家 (wán jiā)" as a sample text, and the sample voice corresponding to this sample text is the audio with the pronunciation of "wán jiā"; or a sentence "需要什么帮助?(xūyào shén me bāng zhù)" as a sample text, and the sample voice corresponding to this sample text is the audio with the pronunciation of "xūyào shén me bāng zhù". It should be noted that for the convenience of description, the embodiments of the present application take a sample text as an example for illustration. In the actual application process, a large number of sample texts and their corresponding sample voices may be prepared, and the method provided by the embodiments of the present application is also used for processing, so as to improve the accuracy of the trained voice generation model and enable it to accurately realize the conversion from text to voice. The embodiments of the present application do not limit the specific number of sample texts.
[0037] To achieve more refined model training, it is necessary to determine the phoneme duration of each phoneme in the sample text and the sample voice feature information corresponding to the sample voice, so that the phoneme duration and the sample voice feature information can also participate in the model training, and a standard can be used as a reference in the training process. Among them, a phoneme is the smallest speech unit divided according to the natural attributes of speech. Based on the pronunciation actions in a syllable, one action constitutes one phoneme. Phonemes are divided into two categories: vowels and consonants. For example, the Chinese syllable "啊 (ā)" has only one phoneme; "爱 (ài)" has two phonemes, namely "(à)" and "(i)"; "代 (dài)" has three phonemes, namely "(d)", "(à)" and "(i)". Further, after multiple phonemes are extracted, continue to determine the phoneme duration of each phoneme based on the sample voice, so that the phoneme duration can also participate in the supervised training of the model to achieve a better generation effect of emotional speech. It should be noted that the above description of phonemes is based on Chinese text as an example. In the actual application process, the sample text can also be a language in other languages such as English and Japanese. Assuming the sample text is English, the phoneme extraction and duration determination of the sample text can be carried out according to the 48 phonemes corresponding to the International Phonetic Alphabet of English. The embodiments of the present application do not specifically limit the language of the sample text.
[0038] Furthermore, the sample speech feature information includes at least one of the following: sample sound feature information and sample emotion information. The sample sound feature information is used to characterize the sound features of the sample speech, and can indicate the pitch, timbre, loudness, directionality, sense of distance, etc. of the sample speech. The sample sound feature information can be used to preliminarily determine the emotional color of the sample speech and the speaker's timbre and other information. In order to express the emotional color more accurately, the sample speech feature information can also include sample emotion information. The sample emotion information is used to accurately indicate the emotional color of the sample speech, such as anger, happiness, etc. In this way, by combining the sample speech feature information for model training, the trained model can have the ability to recognize and output emotions, ensuring that the output speech has emotional color.
[0039] After the speech generation model is obtained through the training process described above, the speech generation model can be used to train the target text, target sound feature information and target emotional information to obtain the target speech, thereby realizing the conversion of the target text into the target speech.
[0040] 103. Output the target speech.
[0041] In the embodiment of the present application, when the target voice is output, a section of audio can be output. For example, taking the game scene as an example, when the player triggers the corresponding event, the target voice is played to the player. If the generated target voice is a task description voice, the target voice can be temporarily stored, and when the player triggers the task, the target voice is played to the player, and the strategy of generating voice in advance is adopted to ensure the accuracy of the output voice. In another optional implementation scheme, the target voice of the output can also be played directly, and the game scene is continued as an example, and the voice is generated in real time in the game scene and played to the player, meeting the real-time requirements of the generated voice in some scenes, and ensuring real-time and fast generation of voice. In addition, the target voice can also be combined with video and real-time communication screen for output, or the target voice can also be combined with pre-recorded video, etc. for output, or the target voice can be combined with live screen output, etc., and the present application does not specifically limit the mode of outputting the target voice.
[0042] The method provided in the embodiment of the present application determines the target sound feature information, target emotional information and target text to be converted into speech, and uses the speech generation model obtained by training the training model based on the phoneme duration corresponding to the sample text and the sample speech feature information corresponding to the sample speech to train the target text, target sound feature information and target emotional information to obtain the target speech and output the target speech. In the process of constructing the speech generation model, not only the sample text is used for training, but also the phoneme duration and sample speech feature information describing the characteristics of the speech are additionally introduced for supervised training at the same time, so that the speech generation model can accurately recognize and process the emotional information in the text, and can generate relatively natural high-quality speech of a specific speaker with emotional expression when performing speech conversion, so as to achieve a better generation effect of emotional speech, thereby adapting to diverse emotional expression needs.
[0043] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the specific implementation process of this embodiment, the embodiment of the present application provides another speech generation method, such as Figure 2A As shown, the method includes:
[0044] 201. Obtain sample text and sample speech corresponding to the sample text.
[0045] The embodiments of the present application provide users with a text-to-speech service, achieving the generation of more natural emotional speech while also taking into account a higher speech generation speed.
[0046] In the embodiment of the present application, sample texts are prepared in advance for subsequent model training, and the sample texts all correspond to sample speech, so as to realize the conversion of text to speech. The detailed description of the sample texts and sample speech can be found in the content of step 102 in the above embodiment, which will not be repeated here.
[0047] 202. Perform phoneme extraction on the sample text to obtain multiple phonemes, and align each phoneme with the duration of the audio in the sample speech to obtain the phoneme duration of each phoneme.
[0048] In the embodiment of the present application, since the phoneme duration of the phoneme needs to be additionally introduced when training the speech generation model, it is necessary to extract multiple phonemes of the sample text, and align each phoneme with the duration of the audio in the sample speech to obtain the phoneme duration of each phoneme. The specific description of the phoneme can be found in the content of step 102 of the above embodiment, which will not be repeated here.
[0049] In an alternative embodiment, when extracting phonemes and determining the duration of phonemes, a preset speech-text alignment tool and a preset phoneme table can be obtained. Using the speech-text alignment tool and the preset phoneme table, phoneme extraction is performed on the sample text to obtain multiple phonemes, and each phoneme is aligned with the duration of the audio in the sample speech to obtain the duration of each phoneme. Specifically, the speech-text alignment tool can be the MFA (Montreal Forced Aligner, a speech-text alignment tool), and the content of the preset phoneme table can be set according to the language. Among them, if phoneme extraction is required for Chinese characters, the content of the preset phoneme table can be initial consonants and finals with tone marks; if phoneme extraction is required for English characters, the content of the preset phoneme table can be 20 vowel phonemes and 28 consonant phonemes. The present application does not limit the specific content of the preset phoneme table. Using the MFA tool and the preset phoneme table for phoneme extraction of the text and aligning the phonemes with the duration of the audio, the duration of each phoneme is obtained. Among them, when performing phoneme extraction, the preset phoneme table can be input into the speech-text alignment tool, and the speech-text alignment tool is called to perform phoneme extraction on the sample text according to the preset phoneme table to obtain multiple phonemes. Taking the Chinese character "爱" as an example, the character pinyin "ài" of the character "爱" can be split into two phonemes (à) and (i).
[0050] In another alternative embodiment, after multiple phonemes are extracted, the corresponding duration of each phoneme needs to be determined. Specifically, the speech-text alignment tool can also be used, that is, using the speech-text alignment tool to query the time period corresponding to the audio of each phoneme in the sample speech, and using the time period to determine and align the duration of each phoneme in the audio to obtain the duration of each phoneme. Continuing with the example of the character "爱", the speech-text alignment tool can be used to query the pronunciation part of the character "爱" in the audio, determine the duration of this pronunciation part, and use this duration as the phoneme duration and align it with the character "爱".
[0051] 203. Determine the sample sound feature information corresponding to the sample speech according to the sample speech.
[0052] In the embodiments of the present application, the sample sound feature information is used to characterize the features of the sample speech, and can indicate the pitch, timbre, loudness, directionality, sense of distance, etc. of the sample speech. Through the sample sound feature information, the emotional color of the sample speech and the timbre of the speaker can be initially determined. Specifically, the sample sound feature information includes at least one of the following: fundamental frequency information and energy information. The process of determining the fundamental frequency information will be described first below:
[0053] Generally, sounds are composed of a series of vibrations with different frequencies and amplitudes emitted by the sound-producing body. Among these vibrations, there is a vibration with the lowest frequency, and the sound emitted by it is the fundamental tone, and the rest are overtones. The fundamental tone determines the pitch. Therefore, in the embodiment of the present application, in order to reflect the characteristics of the sound, it is necessary to extract the fundamental tone information corresponding to the sample speech.
[0054] In an optional implementation scheme, the pitch information corresponding to the sample speech can be determined based on the pitch sequence feature. The pitch sequence feature is extracted from the sample speech using a pitch tracking algorithm. The pitch tracking algorithm can be, but is not limited to, a YAAPT (Yet Another Algorithm for Pitch Tracking) algorithm. The pitch sequence feature of the sample speech can be extracted by running the YAAPT algorithm, and then the pitch information corresponding to the sample speech can be determined based on the pitch sequence feature.
[0055] The process of determining energy information is described below:
[0056] The energy of sounds of different emotions is different, so in order to reflect emotional information in the process of text-to-speech conversion, it is necessary to calculate the energy of the sample speech. Among them, the energy of the sample speech can be calculated based on the sample spectrum information of the sample speech to obtain energy information. Specifically, the sample spectrum information is used to indicate the relationship between frequency and energy, which can be the spectrum information, Mel spectrum, etc. of the sample speech. The specific content of the sample spectrum information is not limited in this application. In an optional implementation scheme, the sample spectrum information is the Mel spectrum of the sample speech as an example for explanation. The Mel spectrum is generated by mapping the frequency indicated by the spectrum diagram to the Mel scale. The Mel scale is a perceptual scale that is judged by the audience that the pitches of different frequencies are equal to each other, indicating the human ear's perception of equidistant pitch changes. It is precisely because the Mel scale is designed for the human ear that the Mel spectrum retains the information required for the human ear to understand the original speech to a large extent. In this way, the energy information can be determined using the frequency displayed by the Mel spectrum, and the energy information is used to indicate the pitch of each frame of audio, which is convenient for reflecting emotions. Specifically, since the Mel spectrum can be obtained by mapping the spectrogram to the Mel scale, when determining the energy information, the spectrogram of the sample speech can be obtained, the spectrogram can be converted to obtain the sample spectrum information, and the energy can be calculated based on the relationship between the frequency and energy indicated by the sample spectrum information to obtain the energy information.
[0057] After the pitch information and energy information are calculated through the above process, one or both of them can be used as sample sound feature information.
[0058] 204. Obtain sample emotion information, use the phonemes in the sample text to construct a phoneme sequence of the sample text, use the sample sound feature information and the sample emotion information to process the phoneme sequence, and obtain a high-dimensional vector of the phonemes.
[0059] In an embodiment of the present application, after the basic information required for the training model is obtained through the above process, the model training process is started, and the phoneme duration corresponding to the sample text and the sample speech feature information corresponding to the sample speech are introduced into the training process.
[0060] In order to encode the speaker's timbre and emotion on phonemes, we first use the phonemes in the sample text to construct the phoneme sequence of the sample text, obtain the sample emotional information, and use the sample sound feature information and sample emotional information to process the phoneme sequence to obtain a high-dimensional vector of phonemes that encodes the sound feature information and emotional information, so that the real and accurate sound feature information and emotional information can be integrated into the subsequent training process to supervise and guide the training process.
[0061] In an optional implementation scheme, when generating a phoneme high-dimensional vector, the phoneme sequence is first converted into a discrete high-dimensional vector to obtain an initial high-dimensional vector. Subsequently, the sample sound feature information is converted into a discrete high-dimensional vector to obtain a first high-dimensional vector, and the sample emotional information is converted into a discrete high-dimensional vector to obtain a second high-dimensional vector. Finally, the initial high-dimensional vector is encoded by combining the first high-dimensional vector and the second high-dimensional vector, and the encoded initial high-dimensional vector is used as the phoneme high-dimensional vector.
[0062] 205. Based on the preset language emotion model structure, spectrum prediction is performed on the phoneme high-dimensional vector, the phoneme duration of each phoneme, the sample sound feature information and the sample emotion information to obtain predicted spectrum information.
[0063] In order to improve the accuracy of the generated speech generation model, the embodiment of the present application adopts adversarial training to execute the model training process. Therefore, in order to realize adversarial training, it is necessary to first use the preset language emotion model structure to perform spectral prediction on the phoneme high-dimensional vector, the phoneme duration of each phoneme, the sample sound feature information and the sample emotion information to obtain the predicted spectral information, so as to optimize the language emotion model structure and improve the accuracy of the model in the subsequent statistical analysis of the loss between the predicted spectral information and the sample spectral information.
[0064] In an optional implementation scheme, when generating the predicted spectrum information, the phoneme duration and sample sound feature information of each phoneme are feature predicted based on the language emotion model structure to obtain at least two of the predicted phoneme duration, predicted pitch information, and predicted energy information. Subsequently, the language emotion model structure is used to aggregate at least two of the predicted phoneme duration, predicted pitch information, and predicted energy information onto a phoneme high-dimensional vector, and the aggregated phoneme high-dimensional vector, sample sound feature information, and sample emotion information are decoded to obtain the predicted spectrum information.
[0065] In the actual application process, the processes in step 204 and step 205 can be implemented based on the language emotion model structure. The structure of the language emotion model structure can be as follows: Figure 2B As shown. Among them, Phonemes can be used to indicate a phoneme sequence. After the phoneme sequence is input, Phoneme Embedding is first performed. Phoneme Embedding is used to represent a process of converting a phoneme sequence into a discrete high-dimensional vector. After this process, the phoneme sequence becomes a phoneme Embedding. Phoneme Embedding is a high-dimensional vector (that is, an initial high-dimensional vector) that aggregates phoneme information. Subsequently, the phoneme Embedding is positionally encoded and then input into the encoder, which can be specifically an Encoder. The encoder includes a multi-head attention mechanism layer and a feedforward neural network layer, wherein the multi-head attention mechanism layer can be an MHA (Multi-Head Attention, multi-head attention mechanism) layer, and the feedforward neural network layer can be an FFN (Feedforward Neural, feedforward neural network) layer to implement the encoding operation. After the phoneme Embedding is input into the encoder, the encoder will combine the first high-dimensional vector of the sample sound feature information and the second high-dimensional vector of the sample emotion information to perform a conditional cross-attention mechanism. Among them, the first high-dimensional vector can be represented by spk_emb, the second high-dimensional vector can be represented by emo_emb, and the conditional cross-attention mechanism can be ConditionCrossAttention. The conditional cross-attention mechanism is a mechanism that calculates the attention matrix of the input vector according to certain conditions. Through the conditional cross-attention mechanism, the phoneme Embedding that encodes the sound feature information and emotional information can be obtained. Then, the processed phoneme Embedding and the pre-processed phoneme duration (which can be represented by Duration), pitch information (which can be represented by Pitch), and energy information (which can be represented by Energy) are input together into the network structure used to aggregate other information to a certain high-dimensional vector.
[0066] The network structure used to aggregate other information to a certain high-dimensional vector can be a Variance Adaptor. The specific structure of the Variance Adaptor is as follows: Figure 2C As shown, it includes multiple predictors such as duration predictor, pitch predictor, energy predictor, etc. LR is used to indicate the adaptation of sequence length. Among them, the duration predictor can be Duration Predictor, which can realize the prediction of phoneme duration; the pitch predictor can be Pitch Predictor, which can realize the prediction of fundamental tone information; the energy predictor can be Energy Predictor, which can realize the prediction of energy. Further, as Figure 2D As shown in the figure, the structures of Duration Predictor, Pitch Predictor and Energy Predictor are consistent, including multiple layers such as linear layer, layer normalization + regularization, one-dimensional convolution layer + rectified linear unit, etc., among which the linear layer can be LinearLaryer, the layer normalization + regularization can be LN+Dropout, and the one-dimensional convolution layer + rectified linear unit can be Conv1D+ReLU.
[0067] Through the above process, the prediction of the duration of the phoneme, pitch information and energy information is realized, and the predicted phoneme duration, predicted pitch information and predicted energy information are obtained. Next, the predicted features can be aggregated to the input phoneme Embedding through Variance Adaptor. Finally, the above phoneme Embedding is combined with spk_emb and emo_emb and input into the decoder whose output is Mel spectrum. The decoder whose output is Mel spectrum can be specifically MelDecoder, which decodes the input content to obtain the predicted spectrum information.
[0068] In the above process, the duration, pitch and energy information of the input sample text and sample speech are extracted, and these features are predicted and aggregated using Variance Adaptor to achieve emotional control of speech. The principle is to change the emotional expression of speech by adjusting these features based on the differences in these acoustic features of speech with different emotions.
[0069] In addition, the input phoneme embedding is encoded through the ConditionCrossAttention mechanism, and the sound feature information and emotional information are introduced in combination with spk_emb and emo_emb, so as to realize the speech generation control of different speaker timbres and speech generation control of different emotions. The main implementation principle is to improve the expressiveness of phoneme embedding by weighting and fusing the information of different conditions based on the advantages of the conditional cross attention mechanism, and realize the speech generation control of different conditions (multiple speakers, multiple emotions).
[0070] 206. Calculate the loss function between the predicted spectrum information and the sample spectrum information, update the model parameters of the language emotion model structure according to the loss function until the loss function converges, and construct a speech generation model using the current language emotion model structure.
[0071] In order to improve the quality of the final generated speech, in an embodiment of the present application, an adversarial training method is used to train the speech generation model. The loss function between the predicted spectral information and the sample spectral information is calculated, and the model parameters of the language emotion model structure are updated according to the loss function until the loss function converges. The speech generation model is constructed using the current language emotion model structure.
[0072] In an optional implementation, a Discriminator may be used to calculate the loss function between the predicted spectrum information and the sample spectrum information. Specifically, the input of the Discriminator is the predicted spectrum information and the sample spectrum information, such as Figure 2E As shown, after the input, on the one hand, the Discriminator performs a first type of convolution operation on the predicted spectrum information and the sample spectrum information respectively to obtain a first output vector of the predicted spectrum information and a second output vector of the sample spectrum information. Figure 2E The first type of convolution operation may be an unconditional vector convolution operation, and the unconditional vector convolution operation may specifically be Unconditional_Convs. In this way, the first unconditional output vector of the predicted spectrum information calculated may be used as the first output vector, and the second unconditional output vector of the sample spectrum information calculated may be used as the second output vector. The two output vectors are Figure 2E The unconditional output vector (Unconditional_out) is used in the representation. Next, the second type of convolution operation needs to be performed on the predicted spectrum information and the sample spectrum information respectively to obtain the third output vector of the predicted spectrum information and the fourth output vector of the sample spectrum information. Optionally, continue to refer to Figure 2EThe second type of convolution operation may be a conditional vector convolution operation, and the conditional vector convolution operation may specifically be Conditional_Convs. In this way, the first conditional output vector of the predicted spectrum information calculated may be used as the third output vector, and the second conditional output vector of the sample spectrum information calculated may be used as the fourth output vector. The two output vectors are in Figure 2E The conditional output vector (Conditional_out) is used in the representation.
[0073] On the other hand, Discriminator uses the preset fully connected layer (FullyConnectedLaye) to perform nonlinear transformation on the predicted spectrum information and the sample spectrum information based on the sample sound feature information (spk_emb) and the sample emotion information (emo_emb), respectively, to obtain the first feature output vector of the predicted spectrum information and the second feature output vector of the sample spectrum information. The two output vectors are Figure 2E The feature output vector (Feature_out) is used in the representation.
[0074] Next, based on the Discriminator, the first output vector, the third output vector, the first feature output vector of the predicted spectrum information and the second output vector, the fourth output vector, the second feature output vector of the sample spectrum information are used to calculate the feature matching loss and obtain the feature loss function. Specifically, the feature loss function can be calculated using the following formula 1:
[0075] Formula 1: L fm =mean(L1(Conditional_out pred , ConditionalP_out gt )
[0076] +L1(Unconditional_out pred , Unconditional_out gt )+L1(Feature_out pred , Feature_out gt ))
[0077] Among them, L fm Used to represent the feature loss function, Conditional_out pred Used to represent the third output vector, Conditional_out gt Used to represent the fourth output vector, Unconditional_out pred Used to represent the first output vector, Unconditional_out gtUsed to represent the second output vector, Feature_out pred Used to represent the first feature output vector, Feature_out gt Used to represent the second feature output vector.
[0078] Furthermore, based on the Discriminator, the first output vector and the third output vector of the predicted spectrum information and the second output vector and the fourth output vector of the sample spectrum information are used to perform adversarial loss calculation to obtain an adversarial loss function. Specifically, the adversarial loss function can be calculated using the following formula 2:
[0079] Formula 2: L d =1 / 2(Unconditional_out gt -1) 2 +(Conditional_out gt -1) 2 )+1 / 2((Unconditional_out pred ) 2 +(Conditional_out pred ) 2 )
[0080] Among them, L d Used to represent the adversarial loss function, Unconditional_out gt Used to represent the second output vector, Conditional_out gt Used to represent the fourth output vector, Unconditional_out pred Used to represent the first output vector, Conditional_out pred Used to represent the third output vector.
[0081] After obtaining the feature loss function and the adversarial loss function through the above process, the feature loss function and the adversarial loss function can be used as the loss function between the predicted spectrum information and the sample spectrum information. Next, the model parameters of the language emotion model structure are updated according to the loss function, and the above steps 204 to 206 are re-executed using the updated speech emotion model, and the feature loss function and the adversarial loss function are recalculated as the loss function until the loss function converges, and the speech generation model is constructed using the current language emotion model structure.
[0082] In the above process, the model is trained by adversarial training, the discriminator is used to distinguish the difference between the predicted spectrum information and the sample spectrum information, and the feature matching loss and adversarial loss are calculated to improve the quality of the final generated speech. Based on the idea of generative adversarial network, adversarial training is used to improve the generation ability of the generator, making the generated speech closer to the real speech.
[0083] In addition, precisely because the present application introduces sound feature information and emotional information in the process of training the model, the technical solution of the present application has good scalability for speakers and emotion types. It only needs to add a small amount of training data of the corresponding speakers with the corresponding emotions, such as 50 speech sounds for one emotion, to fine-tune the speech generation model to meet the needs of more scenarios.
[0084] 207. When receiving a speech generation instruction, determine target sound feature information, target emotion information, and target text to be converted into speech.
[0085] When there is a conversion demand, a speech generation instruction can be received, which will indicate which text needs to be converted into speech, and the text is also the target text. The target text is the text that needs to be converted into speech input by the user; the target text can be a character, a word, a sentence or a paragraph of text; the target text can be in different languages such as Chinese, English, Japanese, etc., and the present application does not specifically limit the attributes of the target text. In addition, the user can be a developer of the application, so that the developer can implant the speech obtained by text conversion into the application and use the voice to interact with the user of the application; or the user can also be an individual user such as a game player to meet the diverse needs of individual users. Furthermore, since the user will set the sound feature information to specify the timbre when performing speech conversion, and set the emotional information to specify the emotion, it is necessary to determine the target sound feature information and target emotional information to be converted into speech. Among them, the target sound feature information is used to indicate the timbre characteristics of the target speech obtained by conversion. Since the timbre of each speaker is different, the target sound feature information can be specified by specifying the speaker. The target emotional information is used to indicate the emotional characteristics of the converted target speech, which can be happiness, anger, anxiety, etc. Subsequently, the target text needs to be converted into speech with reference to the target sound feature information and the target emotional information to ensure that the timbre and emotion of the converted target speech meet the conversion requirements.
[0086] In an optional implementation scheme, the speech generation instruction may carry the speaker idx (index) and emotion idx (index) as indexes based on the target text input. The two indexes may be extracted in the speech generation instruction, and the target sound feature information may be queried according to the speaker index, and the target emotion information may be queried according to the emotion index, so as to achieve speech synthesis corresponding to the speaker and the corresponding emotion.
[0087] 208. The target text, target sound feature information and target emotion information are trained using a speech generation model to obtain the target speech.
[0088] In the embodiment of the present application, after obtaining the target text, target sound feature information and target emotion information, the target text, target sound feature information and target emotion information can be trained using the speech generation model to obtain the target speech, thereby realizing the conversion of the target text into the target speech. Specifically, the target text, target sound feature information and target emotion information can be input into the speech generation model together, and the output of the speech generation model is used as the target speech.
[0089] 209. Output the target speech.
[0090] In the embodiment of the present application, when the target voice is output, a section of audio can be output. For example, taking the game scene as an example, when the player triggers the corresponding event, the target voice is played to the player. If the generated target voice is a task description voice, the target voice can be temporarily stored, and when the player triggers the task, the target voice is played to the player, and the strategy of generating voice in advance is adopted to ensure the accuracy of the output voice. In another optional implementation scheme, the target voice of the output can also be played directly, and the game scene is continued as an example, and the voice is generated in real time in the game scene and played to the player, meeting the real-time requirements of the generated voice in some scenes, and ensuring real-time and fast generation of voice. In addition, the target voice can also be combined with video and real-time communication screen for output, or the target voice can also be combined with pre-recorded video, etc. for output, or the target voice can be combined with live screen output, etc., and the present application does not specifically limit the mode of outputting the target voice.
[0091] The method provided in the embodiment of the present application determines the target sound feature information, target emotional information and target text to be converted into speech, and uses the speech generation model obtained by training the training model based on the phoneme duration corresponding to the sample text and the sample speech feature information corresponding to the sample speech to train the target text, target sound feature information and target emotional information to obtain the target speech and output the target speech. In the process of constructing the speech generation model, not only the sample text is used for training, but also the phoneme duration and sample speech feature information describing the characteristics of the speech are additionally introduced for supervised training at the same time, so that the speech generation model can accurately recognize and process the emotional information in the text, and can generate relatively natural high-quality speech of a specific speaker with emotional expression when performing speech conversion, so as to achieve a better generation effect of emotional speech, thereby adapting to diverse emotional expression needs.
[0092] In the process of constructing the speech generation model, the embodiment of the present application not only uses sample text for training, but also introduces additional information about the duration of phonemes and sample speech characteristics that describe speech characteristics for supervised training, so that the speech generation model can accurately identify and process the emotional information in the text, and can generate relatively natural high-quality speech of a specific speaker with emotional expression during speech conversion, achieving better emotional speech generation effects, thereby adapting to diverse emotional expression needs, while also taking into account a higher speech generation speed. Furthermore, as Figure 1 In a specific implementation of the method, the present application provides a speech generating device, such as Figure 3 As shown, the device includes: a determination module 301, a training module 302, and an output module 303.
[0093] The determination module 301 is used to determine the target sound feature information, target emotion information and target text to be converted into speech;
[0094] The training module 302 is used to train the target text, the target sound feature information and the target emotion information using a speech generation model to obtain a target speech; wherein the speech generation model is obtained by training the training model based on the phoneme duration corresponding to the sample text and the sample speech feature information corresponding to the sample speech; the sample speech feature information includes at least one of the following: sample sound feature information and sample emotion information;
[0095] The output module 303 is used to output the target speech.
[0096] In a specific application scenario, the determination module 301 is further used to determine the phoneme duration of each phoneme in the sample text according to the sample text and the sample speech; determine the sample sound feature information and sample emotion information corresponding to the sample speech according to the sample speech; wherein the sample sound feature information includes at least one of the following: pitch information and energy information;
[0097] The training module 302 is further used to train the speech generation model based on the phoneme duration of each phoneme, the sample sound feature information and the sample emotion information;
[0098] And / or, the determination module 301 is used to obtain the speaker index and emotion index carried by the speech generation instruction when receiving the speech generation instruction; query the target sound feature information according to the speaker index; and query the target emotion information according to the emotion index.
[0099] In a specific application scenario, the determination module 301 is used to extract phonemes from the sample text to obtain phonemes corresponding to the sample text; align each of the phonemes with the duration of the audio in the sample speech to obtain the phoneme duration of each of the phonemes.
[0100] In a specific application scenario, the determination module 301 is used to input a preset phoneme table into a speech-to-text alignment tool; call the speech-to-text alignment tool to extract phonemes from the sample text according to the preset phoneme table to obtain the plurality of phonemes;
[0101] And / or, the determination module 301 is used to use the speech-to-text alignment tool to query the time period corresponding to each of the phonemes in the audio of the sample speech; and use the time period to determine the duration of each of the phonemes in the audio and align them to obtain the phoneme duration of each phoneme.
[0102] In a specific application scenario, the sample sound feature information includes at least one of the following: pitch information and energy information;
[0103] The determination module 301 is used for at least one of the following:
[0104] Determine the pitch information corresponding to the sample speech according to the pitch sequence feature, wherein the pitch sequence feature is extracted from the sample speech by using a pitch tracking algorithm;
[0105] Based on the sample spectrum information of the sample speech, energy calculation is performed on the sample speech to obtain the energy information; wherein the sample spectrum information is used to indicate the relationship between the frequency and energy of the sample speech.
[0106] In a specific application scenario, the training module 302 is used to use the phonemes in the sample text to construct a phoneme sequence of the sample text; use the sample sound feature information and the sample emotion information to process the phoneme sequence to obtain a phoneme high-dimensional vector; based on a preset language emotion model structure, perform spectral prediction on the phoneme high-dimensional vector, the phoneme duration of each phoneme, the sample sound feature information and the sample emotion information to obtain predicted spectral information; calculate the loss function between the predicted spectral information and the sample spectral information; update the model parameters of the language emotion model structure according to the loss function until the loss function converges, and use the current language emotion model structure to construct the speech generation model.
[0107] In a specific application scenario, the training module 302 is used to perform discrete high-dimensional vector conversion on the phoneme sequence to obtain an initial high-dimensional vector; perform discrete high-dimensional vector conversion on the sample sound feature information to obtain a first high-dimensional vector; perform discrete high-dimensional vector conversion on the sample emotion information to obtain a second high-dimensional vector; combine the first high-dimensional vector and the second high-dimensional vector, encode the initial high-dimensional vector, and use the encoded initial high-dimensional vector as the phoneme high-dimensional vector.
[0108] In a specific application scenario, the training module 302 is used to perform feature prediction on the phoneme duration and the sample sound feature information of each phoneme based on the language emotion model structure to obtain at least two of the predicted phoneme duration, predicted pitch information and predicted energy information; using the language emotion model structure, at least two of the predicted phoneme duration, the predicted pitch information and the predicted energy information are aggregated into the phoneme high-dimensional vector; and decoding operations are performed on the aggregated phoneme high-dimensional vector, the sample sound feature information and the sample emotion information to obtain the predicted spectrum information.
[0109] In a specific application scenario, the training module 302 performs a first type of convolution operation on the predicted spectrum information and the sample spectrum information respectively to obtain a first output vector of the predicted spectrum information and a second output vector of the sample spectrum information; performs a second type of convolution operation on the predicted spectrum information and the sample spectrum information respectively to obtain a third output vector of the predicted spectrum information and a fourth output vector of the sample spectrum information; uses a preset fully connected layer to perform nonlinear transformation on the predicted spectrum information and the sample spectrum information based on the sample sound feature information and the sample emotion information respectively to obtain a first feature output vector of the predicted spectrum information and a second feature output vector of the sample spectrum information; uses the first output vector, the third output vector, the first feature output vector of the predicted spectrum information and the second output vector, the fourth output vector, and the second feature output vector of the sample spectrum information to perform feature matching loss calculation to obtain a feature loss function; uses the first output vector, the third output vector of the predicted spectrum information and the second output vector, the fourth output vector, and the second feature output vector of the sample spectrum information to perform adversarial loss calculation to obtain an adversarial loss function; uses the feature loss function and the adversarial loss function as the loss function between the predicted spectrum information and the sample spectrum information.
[0110] The device provided in the embodiment of the present application determines the target sound feature information, target emotional information and target text to be converted into speech, and uses the speech generation model obtained by training the training model based on the phoneme duration corresponding to the sample text and the sample speech feature information corresponding to the sample speech to train the target text, target sound feature information and target emotional information to obtain the target speech and output the target speech. In the process of constructing the speech generation model, not only the sample text is used for training, but also the phoneme duration and sample speech feature information describing the characteristics of the speech are additionally introduced for supervised training at the same time, so that the speech generation model can accurately recognize and process the emotional information in the text, and can generate relatively natural high-quality speech of a specific speaker with emotional expression when performing speech conversion, so as to achieve a better generation effect of emotional speech, thereby adapting to diverse emotional expression needs.
[0111] It should be noted that for other corresponding descriptions of the functional units involved in the speech generation device provided in the embodiment of the present application, reference can be made to Figure 1 and FIG. 2A to FIG. 2E The corresponding description in will not be repeated here.
[0112] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0113] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0114] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
[0115] In an exemplary embodiment, see Figure 4 , and also provides an electronic device, which includes a bus, a processor, a memory and a communication interface, and may also include an input and output interface and a display device, wherein each functional unit can communicate with each other through the bus. The memory stores a computer program, and the processor is used to execute the program stored in the memory and execute the speech generation method in the above embodiment.
[0116] A computer-readable storage medium stores a computer program, which implements the steps of the speech generation method when executed by a processor.
[0117] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by hardware, or by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.
[0118] Those skilled in the art will appreciate that the accompanying drawings are merely schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily required for implementing the present application.
[0119] Those skilled in the art will appreciate that the modules in the device in the implementation scenario can be distributed in the device in the implementation scenario according to the implementation scenario description, or can be changed accordingly and located in one or more devices different from the implementation scenario. The modules in the above implementation scenario can be combined into one module, or can be further split into multiple sub-modules.
[0120] The above application serial numbers are for description only and do not represent the advantages or disadvantages of the implementation scenarios.
[0121] The above disclosure only discloses several specific implementation scenarios of the present application. However, the present application is not limited thereto, and any changes that can be conceived by technicians in this field should fall within the scope of protection of the present application.
Claims
1. A speech generation method, characterized in that: include: Determine target sound feature information, target emotion information and target text to be converted into speech; Using a speech generation model to train the target text, the target sound feature information and the target emotion information to obtain a target speech; The speech generation model is obtained by training the training model based on the duration of the phonemes corresponding to the sample text and the sample speech feature information corresponding to the sample speech; the sample speech feature information includes at least one of the following: sample sound feature information and sample emotion information; The target speech is output.
2. The method according to claim 1, characterized in that The method further comprises: Determine the phoneme duration of each phoneme in the sample text according to the sample text and the sample speech; Determine the sample sound feature information and sample emotion information corresponding to the sample sound according to the sample speech; wherein the sample sound feature information includes at least one of the following: pitch information and energy information; Training the speech generation model based on the phoneme duration of each phoneme, the sample sound feature information, and the sample emotion information; And / or, the determining of target sound feature information, target emotion information and target text to be converted into speech includes: When receiving a speech generation instruction, obtaining a speaker index and a sentiment index carried by the speech generation instruction; According to the speaker index, query the target sound feature information; The target emotion information is queried according to the emotion index.
3. The method according to claim 2, characterized in that The step of determining the phoneme duration of each phoneme in the sample text according to the sample text and the sample speech includes: Performing phoneme extraction on the sample text to obtain phonemes corresponding to the sample text; Each of the phonemes is aligned with the duration of the audio in the sample speech to obtain the phoneme duration of each of the phonemes.
4. The method according to claim 3, characterized in that The extracting phonemes from the sample text to obtain phonemes corresponding to the sample text includes: Input the preset phoneme table into the speech-to-text alignment tool; Calling the speech-to-text alignment tool to extract phonemes from the sample text according to the preset phoneme table to obtain the plurality of phonemes; And / or, aligning each of the phonemes with the duration of the audio in the sample speech to obtain the phoneme duration of each of the phonemes includes: Using the speech-to-text alignment tool, querying the time period corresponding to the audio of each phoneme in the sample speech; The duration of each of the phonemes in the audio is determined and aligned using the time period to obtain the phoneme duration of each of the phonemes.
5. The method according to claim 2, characterized in that: The sample sound feature information includes at least one of the following: pitch information and energy information; The step of determining the sample sound feature information corresponding to the sample speech according to the sample speech includes at least one of the following: Determine the pitch information corresponding to the sample speech according to the pitch sequence feature, wherein the pitch sequence feature is extracted from the sample speech by using a pitch tracking algorithm; Based on the sample spectrum information of the sample speech, performing energy calculation on the sample speech to obtain the energy information; The sample spectrum information is used to indicate the relationship between the frequency and energy of the sample speech.
6. The method according to claim 2, characterized in that The training of the speech generation model based on the phoneme duration of each phoneme, the sample sound feature information and the sample emotion information comprises: Using the phonemes in the sample text, construct a phoneme sequence of the sample text; Using the sample sound feature information and the sample emotion information, the phoneme sequence is processed to obtain a phoneme high-dimensional vector; Based on a preset language emotion model structure, spectrum prediction is performed on the phoneme high-dimensional vector, the phoneme duration of each phoneme, the sample sound feature information, and the sample emotion information to obtain predicted spectrum information; Calculating a loss function between the predicted spectrum information and the sample spectrum information; The model parameters of the language emotion model structure are updated according to the loss function until the loss function converges, and the speech generation model is constructed using the current language emotion model structure.
7. The method according to claim 6, characterized in that The method of using the sample sound feature information and the sample emotion information to process the phoneme sequence to obtain a phoneme high-dimensional vector includes: Performing discrete high-dimensional vector conversion on the phoneme sequence to obtain an initial high-dimensional vector; Performing discrete high-dimensional vector conversion on the sample sound feature information to obtain a first high-dimensional vector; Performing discrete high-dimensional vector conversion on the sample emotion information to obtain a second high-dimensional vector; The first high-dimensional vector and the second high-dimensional vector are combined to encode the initial high-dimensional vector, and the encoded initial high-dimensional vector is used as the phoneme high-dimensional vector.
8. The method according to claim 6, characterized in that The method of performing spectrum prediction on the phoneme high-dimensional vector, the phoneme duration of each phoneme, the sample sound feature information, and the sample emotion information based on the preset language emotion model structure to obtain predicted spectrum information includes: Based on the language emotion model structure, feature prediction is performed on the phoneme duration of each of the phonemes and the sample sound feature information to obtain at least two of the predicted phoneme duration, predicted pitch information, and predicted energy information; Adopting the language emotion model structure, aggregating at least two of the predicted phoneme duration, the predicted pitch information and the predicted energy information onto the phoneme high-dimensional vector; The aggregated high-dimensional phoneme vector, the sample sound feature information and the sample emotion information are decoded to obtain the predicted spectrum information.
9. The method according to claim 6, characterized in that The calculating a loss function between the predicted spectrum information and the sample spectrum information includes: Performing a first type of convolution operation on the predicted spectrum information and the sample spectrum information respectively to obtain a first output vector of the predicted spectrum information and a second output vector of the sample spectrum information; Performing a second type of convolution operation on the predicted spectrum information and the sample spectrum information respectively to obtain a third output vector of the predicted spectrum information and a fourth output vector of the sample spectrum information; Using a preset fully connected layer, based on the sample sound feature information and the sample emotion information, the predicted spectrum information and the sample spectrum information are nonlinearly transformed to obtain a first feature output vector of the predicted spectrum information and a second feature output vector of the sample spectrum information; Using the first output vector, the third output vector, the first feature output vector of the predicted spectrum information and the second output vector, the fourth output vector, and the second feature output vector of the sample spectrum information to calculate feature matching loss and obtain a feature loss function; Using the first output vector and the third output vector of the predicted spectrum information and the second output vector and the fourth output vector of the sample spectrum information to perform adversarial loss calculation to obtain an adversarial loss function; The feature loss function and the adversarial loss function are used as loss functions between the predicted spectrum information and the sample spectrum information.
10. A speech generating device, characterized in that: include: A determination module, used to determine target sound feature information, target emotion information and target text to be converted into speech; A training module, used to train the target text, the target sound feature information and the target emotion information using a speech generation model to obtain a target speech; The speech generation model is obtained by training the training model based on the duration of the phonemes corresponding to the sample text and the sample speech feature information corresponding to the sample speech; the sample speech feature information includes at least one of the following: sample sound feature information and sample emotion information; An output module is used to output the target speech.
11. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.
12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.
