Voice generation method, model construction method, device, equipment, medium and product

Abstract

The invention relates to the technical field of artificial intelligence, and provides a voice generation method and device, a model construction method and device, equipment, a medium and a product, and the method comprises the steps: determining emotion feature parameters; based on the emotion feature parameters, constructing an emotion vector of a to-be-generated emotion voice; and inputting the emotion vector and the text vector into a trained emotion voice generation model to obtain a voice token which is output by the emotion voice generation model and corresponds to the text vector, and generating the emotion voice based on the voice token. According to the speech generation method, the model construction method, the device, the equipment, the medium and the product provided by the invention, the emotional vector of the emotional speech to be generated is constructed through the emotional feature parameters, fine modeling of emotional is completed, and the emotional speech can be quickly generated through the strong ability of the emotional speech generation model based on the emotional vector and the text vector. And the generated emotional voice has strong emotional expression ability.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a speech generation method, a model building method, a device, equipment, a medium and a product. Background Art

[0002] Speech generation technology refers to the technology of generating speech from given text. As the application scenarios of intelligent speech technology become more and more extensive, the demand for speech generation is not only neutral emotions, but also more flexible and changeable speech, and speech rich in various emotions. Emotional speech generation is a major research trend in the field of speech generation.

[0003] Existing speech generation technologies often only focus on the accuracy and clarity of speech generation, so the generated speech is usually bland, mechanical, and lacks emotional ups and downs. In addition, there is a lack of existing public emotional speech datasets, and speech emotion annotation is difficult to define, slightly subjective, and expensive, which limits the quality of emotional speech synthesis. Thirdly, existing speech generation technology cannot adjust emotional expression according to different contexts and needs. Summary of the invention

[0004] The present invention provides a speech generation method, a model building method, a device, equipment, a medium and a product, which are used to solve the defect of poor quality of emotional speech synthesis in the prior art.

[0005] The present invention provides a speech generation method, comprising: Determine emotional feature parameters; Based on the emotion feature parameters, construct an emotion vector of the emotion speech to be generated; The emotion vector and the text vector are input into a trained emotion speech generation model to obtain a speech token corresponding to the text vector output by the emotion speech generation model, and the emotion speech is generated based on the speech token.

[0006] According to the speech generation method provided by the present invention, constructing an emotion vector of the emotional speech to be generated based on the emotion feature parameter includes: Transforming the emotion characteristic parameter from the first coordinate system to the second coordinate system to obtain a transformed emotion characteristic parameter; Based on the transformed emotional feature parameters, constructing an emotional intensity vector and an emotional style vector of the emotional speech to be generated; The emotion intensity vector and the emotion style vector are fused to obtain the emotion vector.

[0007] According to the speech generation method provided by the present invention, the step of fusing the emotion intensity vector and the emotion style vector to obtain the emotion vector includes: Based on the emotion feature parameters, determining the emotion category of the emotion speech to be generated; Determining an emotion category vector based on the emotion category of the emotional speech to be generated; The emotion category vector, the emotion intensity vector and the emotion style vector are fused to obtain the emotion vector.

[0008] According to the speech generation method provided by the present invention, the step of transforming the emotion feature parameters from the first coordinate system to the second coordinate system to obtain the transformed emotion feature parameters comprises: Based on the emotion feature parameter and the emotion category of the emotion speech to be generated, determining the origin coordinates of the second coordinate system; Based on the origin coordinates, the emotion feature parameters are transformed from the first coordinate system to the second coordinate system to obtain transformed emotion feature parameters.

[0009] According to the speech generation method provided by the present invention, constructing an emotion vector of the emotional speech to be generated based on the emotion feature parameter includes: In the first coordinate system, determining the single emotion vectors corresponding to the emotion feature parameters respectively; The single emotion vectors are fused to obtain the emotion vector of the emotional speech.

[0010] According to the speech generation method provided by the present invention, the emotion vector and the text vector are input into a trained emotion speech generation model to obtain a speech token corresponding to the text vector output by the emotion speech generation model, including: Extracting a speaker vector corresponding to the first reference speech; The speaker vector, the emotion vector and the text vector are input into a trained emotional speech generation model to obtain speech tokens output by the emotional speech generation model and corresponding to the text vector and the speaker vector respectively.

[0011] According to the speech generation method provided by the present invention, the step of determining the emotion characteristic parameter comprises: Acquiring a second reference voice; Based on the emotion pre-training model, emotion feature parameters of the second reference speech are extracted, and the emotion feature parameters include arousal, valence, and dominance.

[0012] According to the speech generation method provided by the present invention, the emotion characteristic parameters are preset.

[0013] The present invention also provides a method for constructing a speech generation model, comprising: Extracting emotional feature parameters and sample speech vectors of sample speech; Constructing a sample emotion vector based on the emotion feature parameters of the sample speech; Inputting the sample emotion vector, the sample text vector corresponding to the sample speech, and the sample speech vector into an initial large language model to obtain a predicted speech token output by the initial large language model; Based on the difference between the sample speech vector and the predicted speech token, the initial large language model is iterated on parameters to obtain the speech generation model.

[0014] The present invention also provides a speech generating device, comprising: A parameter determination unit, used to determine the emotion characteristic parameters; A vector construction unit, used for constructing an emotion vector of the emotional speech to be generated based on the emotion feature parameters; The speech generation unit is used to input the emotion vector and the text vector into a trained emotion speech generation model, obtain the speech tokens corresponding to the text vector output by the emotion speech generation model, and generate the emotion speech based on the speech tokens.

[0015] The present invention also provides a speech generation model construction device, comprising: A feature extraction unit, used for extracting emotional feature parameters and sample speech vectors of sample speech; A vector construction unit, used to construct a sample emotion vector based on the emotion feature parameters of the sample speech; A speech prediction unit, used for inputting the sample emotion vector, the sample text vector corresponding to the sample speech, and the sample speech vector into an initial large language model to obtain a predicted speech token output by the initial large language model; A parameter iteration unit is used to perform parameter iteration on the initial large language model based on the difference between the sample speech vector and the predicted speech token to obtain the speech generation model.

[0016] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the speech generation method or speech generation model construction method described above is implemented.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the speech generation method or speech generation model construction method as described above is implemented.

[0018] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned speech generation methods or speech generation model construction methods.

[0019] The speech generation method, model construction method, device, equipment, medium and product provided by the present invention construct the emotion vector of the emotional speech to be generated through the emotion feature parameters, complete the refined modeling of emotion, and based on the emotion vector and text vector, through the powerful ability of the emotional speech generation model, the generated emotional speech has strong emotion expression ability. Due to the controllability of the emotion feature parameters, the generated emotional speech also has the ability to control emotions, thereby greatly improving the ability and quality of emotional speech generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 This is one of the flow charts of the speech generation method provided by the present invention.

[0022] Figure 2 This is one of the flow charts of the method for constructing a speech generation model provided by the present invention.

[0023] Figure 3 This is the second flow chart of the speech generation model construction method provided by the present invention.

[0024] Figure 4 It is a flow chart of the emotion vector construction method provided by the present invention.

[0025] Figure 5 It is a structural schematic diagram of the speech generating device provided by the present invention.

[0026] Figure 6 It is a structural schematic diagram of the speech generation model construction device provided by the present invention.

[0027] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] As the application scenarios of intelligent voice technology become more and more extensive, the demand for voice generation includes the expectation to simulate voices of various tones, thereby improving the user experience in the field of intelligent interaction, increasing the attractiveness and immersion of works in games and audiobooks, and making teaching content more vivid in the field of education, thereby improving students' interest in learning and attention. These demands have put forward higher requirements for emotional voice generation.

[0030] However, existing public emotional speech data are difficult to obtain and small in scale. In addition, this type of speech annotation is difficult to define, slightly subjective, and costly. These factors limit the quality of emotional speech synthesis.

[0031] In addition, existing speech generation technologies often only focus on the accuracy and clarity of speech generation and lack emotional modeling. Therefore, the generated speech is usually bland, mechanical, and lacks emotional ups and downs.

[0032] In response to the above problems, an embodiment of the present invention provides a speech generation method, in which the emotion feature parameters are first obtained, and based on the emotion feature parameters, an emotion vector of the emotional speech to be generated is constructed; the emotion vector and the text vector are input into a trained emotion speech generation model to obtain a speech token corresponding to the text vector output by the emotion speech generation model, and the emotion speech is generated based on the speech token. The emotion vector of the emotion speech to be generated is constructed by the emotion feature parameters to complete the refined modeling of emotion. Based on the powerful capability of the emotion speech generation model, the generated emotional speech has strong emotion expression and emotion controllability, thereby greatly improving the ability and quality of emotion speech generation. In addition, this method can implicitly tag the speech with emotion, which greatly reduces the difficulty of emotional speech annotation.

[0033] The embodiments of the present invention can be applied to scenarios where emotional speech generation is required, such as speech cloning, speech conversation, etc. The execution subject of the method can be an electronic device such as a terminal device, a computer, a server, a server cluster, or a specially designed speech generation device, or a speech generation device set in the electronic device, and the speech generation device can be implemented by software, hardware, or a combination of the two.

[0034] In the description of the embodiments of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0035] Figure 1It is one of the flow charts of the speech generation method provided by the present invention, such as Figure 1 As shown, the method comprises the following steps: Step 110, determining emotional characteristic parameters.

[0036] Specifically, the emotional feature parameters can reflect the emotional expression of the speech data, and the emotional feature parameters may include the fundamental frequency, sound intensity, speech rate, intonation, arousal, valence and dominance of the speech data.

[0037] The determination of the emotion feature parameters may be selected from pre-set parameters that can represent a certain emotion, or may be extracted from reference speech with emotion, which is not specifically limited in the embodiment of the present invention.

[0038] In some embodiments, a series of characteristic parameters related to emotions are preset based on theories and experience in the fields of psychology, linguistics, etc. For example, the emotional characteristic parameters including voice attributes such as pitch, volume, speech speed, intonation change, and high-level attributes such as emotional color and emotional intensity can be preset. It is also possible to use existing emotional dictionaries to classify words according to emotional tendencies (such as positive, negative, neutral), and set corresponding emotional characteristic parameters, which can reflect the emotional intensity or polarity of the words in text or speech.

[0039] In other embodiments, the emotional feature parameters can be obtained by extracting emotional features from a reference speech with emotional color selected from a speech library through an emotional recognition algorithm (such as a method based on machine learning or deep learning). Before the emotional feature extraction, the reference speech can also be pre-processed by denoising, standardization, etc., and / or the reference speech can be converted into a format suitable for emotional analysis, such as Mel-frequency cepstral coefficients (MFCC) or linear prediction cepstral coefficients (LPCC), to improve the accuracy of emotional feature extraction.

[0040] Step 120: construct an emotion vector of the emotional speech to be generated based on the emotion feature parameters.

[0041] Step 130, input the emotion vector and the text vector into the trained emotion speech generation model, obtain the speech tokens corresponding to the text vector output by the emotion speech generation model, and generate emotion speech based on the speech tokens.

[0042] Specifically, based on the obtained emotional feature parameters, a multi-dimensional emotional feature space can be constructed. In this space, each dimension represents an emotional feature, such as the high or low fundamental frequency, the strength of the sound intensity, the high or low arousal degree, the high or low dominance degree, etc.

[0043] The emotional feature parameters are mapped into the emotional feature space to obtain a vector representing the emotional state of the speech, that is, the emotional vector of the emotional speech to be generated, which contains all important emotional information in the emotional speech to be generated.

[0044] Here, the text vector refers to the vector of the text corresponding to the emotional speech to be generated. For example, if the text corresponding to the emotional speech to be generated is "The weather is so nice today", then the text vector is the vector corresponding to "The weather is so nice today". For the extraction of text vectors, the text corresponding to the emotional speech can be discretized into text tokens with the help of the vocabulary of the language model. Similarly, each text token also has a text token vector of a specific dimension corresponding to it, and these text token vectors constitute the text vector.

[0045] Then, the emotion vector and the text vector are input into the trained emotion speech generation model to obtain the speech tokens corresponding to the text vector output by the emotion speech generation model. Here, the emotion speech generation model is obtained by training a large language model based on a training data set containing emotion speech. Large language model (LLM), referred to as large model, refers to a natural language processing (NLP) model with a large parameter scale. The number of model parameters and / or the complexity of the model structure exceeds the preset threshold. The model processes large-scale text data during training and has the ability to understand and generate natural language. For example, a large language model can be a qwen series large model, a llama series large model, etc.

[0046] Before executing step 130, the emotional speech generation model may be pre-trained. The training method of the emotional speech generation model includes: obtaining a training data set containing emotional speech, performing feature extraction on the training data through a vector extractor to obtain a vector representation of the training data, inputting the vector representation of the training data into an initial large language model to obtain a predicted speech token output by the initial large language model; based on the difference between the sample speech vector and the predicted speech token, performing parameter iteration on the initial large language model to obtain an emotional speech generation model. The emotional speech generation model thus trained can generate speech tokens containing emotions based on the emotion vector and the text vector.

[0047] Then, the speech tokens output by the large language model are fed into a decoder to generate emotional speech. The decoder can be the decoding part of an audio encoder, such as Encodec or Funcodec.

[0048] The method provided by the embodiment of the present invention constructs the emotion vector of the emotional speech to be generated through the emotion feature parameters, completes the refined modeling of emotion, and based on the emotion vector and the text vector, through the powerful ability of the emotional speech generation model, the generated emotional speech has strong emotion expression ability. Due to the controllability of the emotion feature parameters, the generated emotional speech also has the emotion controllability, thereby greatly improving the ability and quality of emotional speech generation.

[0049] In some embodiments, based on the emotion feature parameters, an emotion vector of the emotion speech to be generated is constructed, that is, step 120 specifically includes: Step 121, transforming the emotion characteristic parameters from the first coordinate system to the second coordinate system to obtain transformed emotion characteristic parameters; Step 122, constructing an emotion intensity vector and an emotion style vector of the emotion speech to be generated based on the transformed emotion feature parameters; Step 123, the emotion intensity vector and the emotion style vector are merged to obtain an emotion vector.

[0050] Specifically, the first coordinate system may be a coordinate system where the original data of the emotion feature parameters are located, and the second coordinate system may be a coordinate system designed according to the analysis or model requirements. For example, the first coordinate system may be a Cartesian coordinate system, and the second coordinate system may be a spherical coordinate system. Considering that the first coordinate system may not be able to model some complex features of emotions, it is necessary to transform the emotion feature parameters from the first coordinate system to the second coordinate system, and the second coordinate system can realize the fine modeling of the emotion vector.

[0051] According to the characteristics of the first coordinate system and the second coordinate system, select a suitable transformation method. For example, if the second coordinate system is low-dimensional, PCA or LDA can be used for dimensionality reduction; if the second coordinate system requires a specific data distribution, standardization or normalization methods can be used. Apply the selected transformation method to transform the emotional feature parameters from the first coordinate system to the second coordinate system. Usually mathematical operations such as matrix multiplication, vector addition and subtraction are involved.

[0052] After obtaining the transformed emotional feature parameters, the emotional intensity vector and emotional style vector of the emotional speech to be generated are constructed. Emotional intensity usually refers to the intensity of emotion, which can be measured by certain characteristic parameters (such as volume, range of speech speed). Emotional style refers to the way or type of emotional expression, such as happiness, sadness, anger, etc., which can be identified by specific feature patterns or combinations.

[0053] For example, statistical methods (such as mean and variance) can be used to construct a sentiment intensity vector; machine learning models (such as classifiers and clusterers) can be used to construct a sentiment style vector.

[0054] According to the characteristics of the emotion intensity vector and the emotion style vector, a suitable fusion method is selected. For example, weighted summation, concatenation, feature mapping and other methods can be used to fuse the emotion intensity vector and the emotion style vector into a comprehensive emotion vector.

[0055] Based on any of the above embodiments, the emotion intensity vector and the emotion style vector are fused to obtain the emotion vector, that is, step 123 specifically includes: Step 123-1, determining the emotion category of the emotional speech to be generated based on the emotion feature parameters; Step 123-2, determining an emotion category vector based on the emotion category of the emotional speech to be generated; Step 123-3, the emotion category vector, the emotion intensity vector and the emotion style vector are merged to obtain the emotion vector.

[0056] Specifically, in order to further improve the quality of emotional speech generation, the emotional category of emotional speech can also be considered when constructing the emotional vector. The emotional category of emotional speech can be determined by emotional feature parameters, for example, the category of emotion can be determined by three feature parameters: arousal, valence, and dominance.

[0057] Then, a sentiment category vector space is defined, in which each dimension represents a sentiment category. According to the determined sentiment category, the corresponding vector position is found in the sentiment category vector space, and the dimension corresponding to the sentiment category in the sentiment category vector is assigned to 1 (or other significant values), and the remaining dimensions are assigned to 0 (or smaller values), so that a sentiment category vector representing a specific sentiment category can be obtained.

[0058] Then, the emotion category vector, emotion intensity vector and emotion style vector are fused to obtain the emotion vector. Common fusion methods include concatenation, weighted summation, feature mapping, etc.

[0059] In some embodiments, the emotion style vector and the emotion category vector may be concatenated and then nonlinearly transformed, and the transformed vector may be added to the emotion intensity vector to obtain the emotion vector. The emotion vector may be expressed as:

[0060] in, It means that the vector operation function is concatenated first and then nonlinear transformation is performed. represents the sentiment vector, represents the emotional style vector, represents the sentiment category vector, Represents the sentiment intensity vector.

[0061] Based on the above embodiment, the emotional characteristic parameters are transformed from the first coordinate system to the second coordinate system to obtain the transformed emotional characteristic parameters, that is, step 121 specifically includes: Step 121-1, determining the origin coordinates of the second coordinate system based on the emotion feature parameters and the emotion category of the emotional speech to be generated; Step 121 - 2 , based on the origin coordinates, transform the emotion feature parameters from the first coordinate system to the second coordinate system to obtain transformed emotion feature parameters.

[0062] Specifically, taking the first coordinate system as the Cartesian coordinate system, the second coordinate system as the spherical coordinate system, and the emotional characteristic parameters including arousal, valence, and dominance as an example, the transformation of the emotional characteristic parameters can be achieved through the following steps: In the Cartesian coordinate system, the emotional feature parameters can be expressed as coordinates ,in Indicates the degree of arousal, Indicates the potency, represents the degree of dominance, Indicates i The first emotion k coordinates. The coordinates of the emotional feature parameters of the emotional category of neutral emotion are averaged to obtain the origin of the second coordinate system. It can be expressed as:

[0063] in, represents the origin of the second coordinate system, Indicates i The first emotion n coordinates, Represents the number of coordinates with neutral sentiment.

[0064] Then the coordinates in the first coordinate system can be transformed into new coordinates with M as the origin , It can be expressed as:

[0065] in, , , are the transformed emotional feature parameters, which represent the transformed arousal, valence and dominance respectively.

[0066] Then, with the help of the coordinate transformation formula, the coordinates of the spherical coordinate system can be obtained as:

[0067]

[0068] Among them, the radius can be expressed as emotional intensity, while the phase ( ) represents the style of emotion. Thus, we can get the emotion intensity vector and emotion style vector.

[0069] Based on any of the above embodiments, based on the emotion feature parameters, an emotion vector of the emotion speech to be generated is constructed, that is, step 120 specifically includes: Step 124, determining the single emotion vectors corresponding to the emotion feature parameters in the first coordinate system; Step 125, the single emotion vectors are fused to obtain an emotion vector.

[0070] Specifically, for the construction of the emotion vector, the coordinate system transformation may not be performed, and the coordinate points corresponding to the emotion feature parameters in the first coordinate system ( ), and obtain the corresponding single emotion vector , and finally the sentiment vector can be expressed as:

[0071] in It means concatenation first and then nonlinear transformation.

[0072] Based on any of the above embodiments, in step 130, the emotion vector and the text vector are input into the trained large language model to obtain the speech token corresponding to the text vector output by the emotion speech generation model, which specifically includes: Step 131, extracting a speaker vector corresponding to the first reference speech; Step 132, input the speaker vector, the emotion vector and the text vector into the trained emotional speech generation model, and obtain the speech tokens output by the emotional speech generation model corresponding to the text vector and the speaker respectively.

[0073] Specifically, in a multi-person interaction scenario, an emotional speech corresponding to any speaker can be generated. For example, an emotional speech with the same timbre as speaker A can be generated for speaker A, and an emotional speech with the same timbre as speaker B can be generated for speaker B. The first reference speech can be a piece of speech data for any speaker, used to extract a speaker vector reflecting the identity of the speaker. The first reference speech can be speech data with emotional color or speech data without emotional color, which is not specifically limited in the embodiment of the present invention.

[0074] Speaker vectors are usually extracted through deep learning models (such as neural networks), which can learn features in speech signals and represent them as high-dimensional vectors. These vectors have certain distribution characteristics in space, which can reflect the differences and similarities between different speakers and realize speaker identification and verification. For example, the speaker vector corresponding to the first reference speech can be extracted through a pre-trained speaker vector extractor such as x-vector.

[0075] Then, the speaker vector, emotion vector and text vector can be concatenated and input into the emotional speech generation model to obtain the speech tokens output by the emotional speech generation model corresponding to the text vector and the speaker respectively.

[0076] In some embodiments, determining the emotion characteristic parameters, that is, step 110 specifically includes: obtaining a second reference speech, and extracting the emotion characteristic parameters of the second reference speech based on the emotion pre-training model, where the emotion characteristic parameters include arousal, valence, and dominance.

[0077] Specifically, the second reference speech may be one or more emotionally charged speech segments selected from a speech library, and the second reference speech is used to extract emotional feature parameters therefrom.

[0078] Here, the sentiment pre-training model may be a deep learning model that has been fully trained and can perform well in sentiment analysis tasks, for example, it may be a wav2vec_ser model.

[0079] The second reference speech is input into the emotion pre-training model, and the model will automatically process the speech data and extract feature information related to emotion, and then extract the emotion feature parameters of the second reference speech based on the feature information.

[0080] The arousal characteristic parameters can be extracted from the energy, amplitude, frequency change and other characteristic information of the speech signal output by the model. Arousal refers to the intensity of an individual's emotional experience or the level of physiological activation. It reflects the degree of body energy activation under emotional state, as well as the individual's alertness and response readiness. High arousal means that the emotional experience is strong, the body energy activation is high, and the individual is in a state of high alertness and response readiness; conversely, low arousal means that the emotional experience is weak, the body energy activation is low, and the individual is in a relatively calm and relaxed state.

[0081] Valence feature parameters can be extracted from the pitch, timbre, speech speed and other characteristic information of the speech signal output by the model. Valence, also known as the polarity or pleasure of emotion, refers to an individual's evaluation of the pleasantness or unpleasantness of emotional experience. It reflects the positive and negative nature of emotions, that is, emotional tendencies such as positive or negative, like or dislike. Positive valence indicates positive emotions such as pleasure, happiness, and satisfaction. Negative valence indicates negative emotions such as pain, sadness, and anger.

[0082] The dominance feature parameters can be extracted from the clarity, coherence, rhythm and other characteristic information of the speech signal output by the model. Dominance refers to the degree of control or influence of an individual on the situation. It reflects the control and influence of an individual on the situation, the external environment and others under the emotional state. High dominance means that the individual has a high degree of control over the situation and can actively respond to and change the situation. Low dominance means that the individual has a low degree of control over the situation and may be in a passive state due to the influence of the situation or others.

[0083] After obtaining the emotional feature parameters of the reference speech, the emotional feature parameters may be normalized, and the value range of each emotional feature parameter may be between 0 and 1.

[0084] The method provided in the embodiment of the present invention can extract the emotion feature parameters of the second reference speech through the emotion pre-training model, and can realize implicit emotion tagging of the speech, which greatly reduces the difficulty of emotion speech annotation.

[0085] In some embodiments, the emotional characteristic parameters are pre-set.

[0086] Specifically, in order to generate more flexible and emotionally controllable emotional speech, the emotional feature parameters in this embodiment can be pre-set. For example, the value range (such as 0-1) of each parameter in the emotional feature parameters and the corresponding emotional description can be pre-set according to the results of psychology or empirical research. According to the emotion of the emotional speech to be generated, the user can customize and flexibly set the value of each emotional feature parameter.

[0087] The method provided by the embodiment of the present invention achieves the generation of emotional speech with more personalized characteristics and significantly improved emotion regulation ability by pre-setting emotional feature parameters, thereby improving the flexibility of emotional speech generation.

[0088] Based on any of the above embodiments, Figure 2 This is one of the flow charts of the method for constructing a speech generation model provided by the present invention. Figure 2 As shown, the model building method includes: Step 210, extracting emotional feature parameters and sample speech vectors of the sample speech; Step 220, constructing a sample emotion vector based on the emotion feature parameters of the sample speech; Step 230, inputting the sample emotion vector, the sample text vector corresponding to the sample speech, and the sample speech vector into the initial large language model to obtain the predicted speech token output by the initial large language model; Step 240, based on the difference between the sample speech vector and the predicted speech token, iterate the parameters of the initial large language model to obtain a speech generation model.

[0089] Specifically, the sample speech may be speech data with emotional color in a speech library, and the emotional feature parameters of the sample speech may be extracted through an emotional pre-training model.

[0090] The extraction of sample speech vectors can be done through some open source audio encoders such as Encodec, Funcodec, etc., which discretize the speech into speech tokens, and each speech token corresponds to a token vector of a specific dimension. These token vectors constitute the sample speech vector.

[0091] In step 220, a sample emotion vector is constructed based on the emotion feature parameters of the sample speech. The implementation method of step 120 may be referred to and will not be described in detail here.

[0092] Then, the sample emotion vector, the sample text vector corresponding to the sample speech, and the sample speech vector are input into the initial large language model to obtain the predicted speech token output by the initial large language model. Here, the predicted speech token includes the corresponding emotion information in the predicted speech.

[0093] On this basis, the difference between the sample speech vector and the predicted speech token is compared, which can be measured by calculating the loss between the two (such as mean square error, cross entropy loss, etc.). Based on the calculated difference, the parameters of the model are updated using the back propagation algorithm. The goal is to reduce the difference between the real speech features and the predicted speech features, thereby improving the accuracy of the model. After many iterations, the model gradually learns to generate speech features similar to the real speech based on the input emotion vector and text vector, thereby completing the training of the speech generation model.

[0094] The method provided by the embodiment of the present invention constructs a sample emotion vector based on the emotion feature parameters of the sample speech, completes the refined modeling of emotion, and through the powerful capabilities of the large language model, the predicted speech tokens generated based on the sample emotion vector have strong emotion expression capabilities. The speech generation model trained in this way can generate speech tokens containing emotion information, thereby greatly improving the ability and quality of emotional speech generation.

[0095] In some embodiments, Figure 3FIG. 2 is a flow chart of the method for constructing a speech generation model provided by the present invention. Figure 3 As shown, a sample speaker vector of the sample speech can also be extracted, and the sample speaker vector, the sample emotion vector, the sample text vector corresponding to the sample speech, and the sample speech vector are input into the initial large language model to obtain the predicted speech token output by the initial large language model.

[0096] Based on any of the above embodiments, a speech generation method is provided, the method comprising: S1, training the emotional speech generation model. Specifically, it includes: extracting the emotional feature parameters and sample speech vector of the sample speech; constructing the sample emotional vector based on the emotional feature parameters of the sample speech; inputting the sample emotional vector, the sample text vector corresponding to the sample speech, and the sample speech vector into the initial large language model to obtain the predicted speech token output by the initial large language model; based on the difference between the sample speech vector and the predicted speech token, iterating the parameters of the initial large language model to obtain the speech generation model.

[0097] S2, determining emotional characteristic parameters, specifically including: obtaining a second reference speech; extracting emotional characteristic parameters of the second reference speech based on an emotional pre-training model, wherein the emotional characteristic parameters include arousal, valence, and dominance.

[0098] S3, constructing an emotion vector of the emotional speech to be generated based on the emotion feature parameters. Figure 4 is a flow chart of the method for constructing the emotion vector provided by the present invention, such as Figure 4 As shown in Figure 2, the construction of the sentiment vector specifically includes: Determine the emotion category of the emotional speech to be generated based on the emotion feature parameters, and determine the emotion category vector based on the emotion category of the emotional speech to be generated; Based on the emotion feature parameters and the emotion category of the emotion speech to be generated, the origin coordinates of the second coordinate system are determined; based on the origin coordinates, the emotion feature parameters are transformed from the first coordinate system to the second coordinate system to obtain the transformed emotion feature parameters; based on the transformed emotion feature parameters, an emotion intensity vector and an emotion style vector of the emotion speech to be generated are constructed; The emotion category vector, emotion intensity vector and emotion style vector are fused to obtain the emotion vector.

[0099] Alternatively, the emotion vector construction specifically includes: in the first coordinate system, determining the single emotion vectors corresponding to the emotion feature parameters respectively; and fusing the single emotion vectors to obtain the emotion vector of the emotional speech.

[0100] S4, inputting the emotion vector and the text vector into the trained emotion speech generation model, obtaining the speech tokens corresponding to the text vector output by the emotion speech generation model, and generating emotion speech based on the speech tokens. Specifically including: Extract the speaker vector corresponding to the first reference speech; input the speaker vector, the emotion vector and the text vector into the trained emotion speech generation model to obtain the speech tokens output by the emotion speech generation model corresponding to the text vector and the speaker vector respectively.

[0101] The speech generation device provided by the present invention is described below. The speech generation device described below and the speech generation method described above can be referenced to each other.

[0102] Based on any of the above embodiments, Figure 5 Schematic diagram of the structure of the speech generating device provided by the present invention. Figure 5 As shown, the speech generating device comprises: A parameter determination unit 510, used to determine an emotion characteristic parameter; A vector construction unit 520, configured to construct an emotion vector of the emotional speech to be generated based on the emotion feature parameters; The speech generation unit 530 is used to input the emotion vector and the text vector into the trained emotion speech generation model, obtain the speech token corresponding to the text vector output by the emotion speech generation model, and generate the emotion speech based on the speech token.

[0103] The device provided by the embodiment of the present invention constructs the emotion vector of the emotional speech to be generated through the emotion feature parameters, completes the refined modeling of emotion, and based on the emotion vector and the text vector, through the powerful ability of the emotional speech generation model, the generated emotional speech has strong emotion expression ability. Due to the controllability of the emotion feature parameters, the generated emotional speech also has the emotion controllability, thereby greatly improving the ability and quality of emotional speech generation.

[0104] Based on any of the above embodiments, the vector construction unit is specifically used for: Transforming the emotion characteristic parameter from the first coordinate system to the second coordinate system to obtain a transformed emotion characteristic parameter; Based on the transformed emotional feature parameters, constructing an emotional intensity vector and an emotional style vector of the emotional speech to be generated; The emotion intensity vector and the emotion style vector are fused to obtain the emotion vector.

[0105] Based on any of the above embodiments, the vector construction unit is specifically used for: Based on the emotion feature parameters, determining the emotion category of the emotion speech to be generated; Determining an emotion category vector based on the emotion category of the emotional speech to be generated; The emotion category vector, the emotion intensity vector and the emotion style vector are fused to obtain the emotion vector.

[0106] Based on any of the above embodiments, the vector construction unit is specifically used for: Based on the emotion feature parameter and the emotion category of the emotion speech to be generated, determining the origin coordinates of the second coordinate system; Based on the origin coordinates, the emotion feature parameters are transformed from the first coordinate system to the second coordinate system to obtain transformed emotion feature parameters.

[0107] Based on any of the above embodiments, the vector construction unit is specifically used for: In the first coordinate system, determining the single emotion vectors corresponding to the emotion feature parameters respectively; The single emotion vectors are fused to obtain the emotion vector of the emotional speech.

[0108] Based on any of the above embodiments, the speech generation unit is specifically used for: Extracting a speaker vector corresponding to the first reference speech; The speaker vector, the emotion vector and the text vector are input into a trained emotional speech generation model to obtain speech tokens output by the emotional speech generation model and corresponding to the text vector and the speaker vector respectively.

[0109] Based on any of the above embodiments, the parameter determination unit is specifically used for: Acquiring a second reference voice; Based on the emotion pre-training model, emotion feature parameters of the second reference speech are extracted, and the emotion feature parameters include arousal, valence, and dominance.

[0110] Based on any of the above embodiments, the emotional characteristic parameters are preset.

[0111] Based on any of the above embodiments, Figure 6 Schematic diagram of the structure of the speech generation model construction device provided by the present invention. Figure 6 As shown, the speech generation model construction device includes: A feature extraction unit 610, used to extract emotional feature parameters and sample speech vectors of the sample speech; A vector construction unit 620, configured to construct a sample emotion vector based on the emotion feature parameters of the sample speech; A speech prediction unit 630, configured to input the sample emotion vector, the sample text vector corresponding to the sample speech, and the sample speech vector into an initial large language model to obtain a predicted speech token output by the initial large language model; The parameter iteration unit 640 is used to perform parameter iteration on the initial large language model based on the difference between the sample speech vector and the predicted speech token to obtain the speech generation model.

[0112] The device provided by the embodiment of the present invention constructs a sample emotion vector based on the emotion feature parameters of the sample speech, completes the refined modeling of emotion, and through the powerful capabilities of the large language model, the predicted speech tokens generated based on the sample emotion vector have strong emotion expression capabilities. The speech generation model trained in this way can generate speech tokens containing emotion information, thereby greatly improving the ability and quality of emotional speech generation.

[0113] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the speech generation method, which includes: determining the emotion feature parameters; constructing the emotion vector of the emotion speech to be generated based on the emotion feature parameters; inputting the emotion vector and the text vector into the trained emotion speech generation model, obtaining the speech token corresponding to the text vector output by the emotion speech generation model, and generating the emotion speech based on the speech token.

[0114] The processor can call the logic instructions in the memory to execute the speech generation model construction method, which includes: extracting the emotion feature parameters and sample speech vector of the sample speech; constructing the sample emotion vector based on the emotion feature parameters of the sample speech; inputting the sample emotion vector, the sample text vector corresponding to the sample speech and the sample speech vector into the initial large language model to obtain the predicted speech token output by the initial large language model; and iterating the parameters of the initial large language model based on the difference between the sample speech vector and the predicted speech token to obtain the speech generation model.

[0115] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0116] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the speech generation method provided by the above-mentioned methods, which method includes: determining emotional feature parameters; constructing an emotional vector of the emotional speech to be generated based on the emotional feature parameters; inputting the emotional vector and the text vector into a trained emotional speech generation model to obtain a speech token corresponding to the text vector output by the emotional speech generation model, and generating the emotional speech based on the speech token.

[0117] When the computer program is executed by the processor, the computer can execute the speech generation model construction method provided by the above methods, which includes: extracting the emotion feature parameters and sample speech vector of the sample speech; constructing the sample emotion vector based on the emotion feature parameters of the sample speech; inputting the sample emotion vector, the sample text vector corresponding to the sample speech and the sample speech vector into the initial large language model to obtain the predicted speech token output by the initial large language model; and iterating the parameters of the initial large language model based on the difference between the sample speech vector and the predicted speech token to obtain the speech generation model.

[0118] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the speech generation method provided by the above-mentioned methods, the method comprising: determining emotion feature parameters; constructing an emotion vector of the emotion speech to be generated based on the emotion feature parameters; inputting the emotion vector and the text vector into a trained emotion speech generation model, obtaining a speech token corresponding to the text vector output by the emotion speech generation model, and generating the emotion speech based on the speech token.

[0119] When the computer program is executed by the processor, it is implemented to execute the speech generation model construction method provided by the above methods, which method includes: extracting the emotion feature parameters and sample speech vector of the sample speech; constructing the sample emotion vector based on the emotion feature parameters of the sample speech; inputting the sample emotion vector, the sample text vector corresponding to the sample speech and the sample speech vector into the initial large language model to obtain the predicted speech token output by the initial large language model; based on the difference between the sample speech vector and the predicted speech token, iterating the parameters of the initial large language model to obtain the speech generation model.

[0120] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0121] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech generation method, characterized in that: include: Determining emotional characteristic parameters, wherein the emotional characteristic parameters include arousal, valence, and dominance; Based on the emotion feature parameters, construct an emotion vector of the emotion speech to be generated; Inputting the emotion vector and the text vector into a trained emotion speech generation model, obtaining a speech token corresponding to the text vector output by the emotion speech generation model, and generating the emotion speech based on the speech token; The emotional speech generation model is obtained by training based on a large language model.

2. The speech generation method according to claim 1, characterized in that: The step of constructing an emotion vector of the emotional speech to be generated based on the emotion feature parameters comprises: Transforming the emotion characteristic parameter from the first coordinate system to the second coordinate system to obtain a transformed emotion characteristic parameter; Based on the transformed emotional feature parameters, constructing an emotional intensity vector and an emotional style vector of the emotional speech to be generated; The emotion intensity vector and the emotion style vector are fused to obtain the emotion vector.

3. The speech generation method according to claim 2, characterized in that: The step of fusing the emotion intensity vector and the emotion style vector to obtain the emotion vector includes: Based on the emotion feature parameters, determining the emotion category of the emotion speech to be generated; Determining an emotion category vector based on the emotion category of the emotional speech to be generated; The emotion category vector, the emotion intensity vector and the emotion style vector are fused to obtain the emotion vector.

4. The speech generation method according to claim 1, characterized in that: The step of constructing an emotion vector of the emotional speech to be generated based on the emotion feature parameters comprises: In the first coordinate system, determining the single emotion vectors corresponding to the emotion feature parameters respectively; The single emotion vectors are fused to obtain the emotion vector of the emotional speech.

5. The speech generation method according to any one of claims 1 to 4, characterized in that: The step of inputting the emotion vector and the text vector into a trained emotion speech generation model to obtain a speech token output by the emotion speech generation model and corresponding to the text vector comprises: Extracting a speaker vector corresponding to the first reference speech; The speaker vector, the emotion vector and the text vector are input into a trained emotional speech generation model to obtain speech tokens output by the emotional speech generation model and corresponding to the text vector and the speaker vector respectively.

6. The speech generation method according to any one of claims 1 to 4, characterized in that: The step of determining the emotional characteristic parameters comprises: Acquiring a second reference voice; Based on the emotion pre-training model, emotion feature parameters of the second reference speech are extracted, and the emotion feature parameters include arousal, valence, and dominance.

7. The speech generation method according to any one of claims 1 to 4, characterized in that: The emotional characteristic parameters are preset.

8. A method for constructing a speech generation model, characterized in that: include: Extracting emotional feature parameters and sample speech vectors of sample speech; Constructing a sample emotion vector based on the emotion feature parameters of the sample speech; Inputting the sample emotion vector, the sample text vector corresponding to the sample speech, and the sample speech vector into an initial large language model to obtain a predicted speech token output by the initial large language model; Based on the difference between the sample speech vector and the predicted speech token, the initial large language model is iterated on parameters to obtain the speech generation model.

9. A speech generating device, characterized in that: include: A parameter determination unit, used to determine emotional characteristic parameters, wherein the emotional characteristic parameters include arousal, valence and dominance; A vector construction unit, used for constructing an emotion vector of the emotional speech to be generated based on the emotion feature parameters; A speech generation unit, used for inputting the emotion vector and the text vector into a trained emotion speech generation model, obtaining a speech token corresponding to the text vector output by the emotion speech generation model, and generating the emotion speech based on the speech token; The emotional speech generation model is obtained by training based on a large language model.

10. A speech generation model construction device, characterized in that: include: A feature extraction unit, used for extracting emotional feature parameters and sample speech vectors of sample speech; A vector construction unit, used to construct a sample emotion vector based on the emotion feature parameters of the sample speech; A speech prediction unit, used for inputting the sample emotion vector, the sample text vector corresponding to the sample speech, and the sample speech vector into an initial large language model to obtain a predicted speech token output by the initial large language model; A parameter iteration unit is used to perform parameter iteration on the initial large language model based on the difference between the sample speech vector and the predicted speech token to obtain the speech generation model.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the speech generation method according to any one of claims 1 to 7 or the speech generation model construction method according to claim 8 is implemented.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech generation method according to any one of claims 1 to 7 or the speech generation model construction method according to claim 8 is implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the speech generation method according to any one of claims 1 to 7 or the speech generation model construction method according to claim 8 is implemented.

Images