An emotion-controllable joint coding VITS speech synthesis method and related device

Abstract

The present invention provides an emotion-controllable joint encoding VITS speech synthesis method and related devices, which relate to the field of speech synthesis technology. The method extracts emotion features from input speech samples and generates an emotion feature vector; constructs a set of emotion category relative ranking functions to generate a relative attribute vector; concatenates the emotion feature vector with the relative attribute vector to obtain a fused emotion feature representation; performs weighted concatenation of the text feature vector and the fused emotion feature representation to generate a joint feature vector; performs joint encoding on the joint feature vector; converts the jointly encoded features into a speech spectrum, and dynamically controls the emotional expression of the synthesized speech by adjusting the weight ratio of each emotion category in the relative attribute vector, thereby achieving a technical effect of effectively, conveniently, and flexibly controlling the emotion of the synthesized speech.

Description

Technical Field

[0001] The present application relates to the field of speech synthesis technology, and in particular to an emotion-controllable joint encoding VITS speech synthesis method and related devices. Background Art

[0002] With the rapid development of artificial intelligence technology, the field of speech synthesis has made significant progress, aiming to generate natural, fluent, and high-quality voice content for users to meet the needs of diverse application scenarios, such as intelligent voice assistants, audiobook production, and voice broadcast systems.

[0003] Traditional speech synthesis methods have evolved through several stages, starting with early rule-based synthesis techniques, which generated speech by splicing pre-recorded speech segments according to specific rules. This approach lacked flexibility and produced less natural speech. Subsequently, statistical parameter speech synthesis methods emerged, using acoustic models and other statistical methods to model speech characteristics. However, this approach still has limitations in terms of naturalness and expressiveness, making it difficult to meet people's demand for more vivid and emotionally expressive speech synthesis.

[0004] Therefore, how to achieve effective, convenient, flexible and controllable synthesis of speech emotions has become a technical problem that needs to be solved urgently. Summary of the Invention

[0005] In order to achieve effective, convenient and flexible control of the emotion of synthesized speech, the present application provides an emotion-controllable joint coding VITS speech synthesis method and related devices.

[0006] In the first aspect, the present application provides an emotion-controllable joint coding VITS speech synthesis method using the following technical solutions:

[0007] An emotion-controllable joint encoding VITS speech synthesis method, comprising:

[0008] S1: Extract emotional features from the input speech sample and generate an emotional feature vector using a preset model;

[0009] S2: constructing a set of emotion category relative ranking functions, quantifying the differences of multiple emotion categories on the acoustic features of the input speech sample, and generating a relative attribute vector;

[0010] S3: concatenating the emotion feature vector and the relative attribute vector to obtain a fused emotion feature representation;

[0011] S4: performing weighted concatenation of the text feature vector and the fused sentiment feature representation to generate a joint feature vector;

[0012] S5: jointly encoding the joint feature vector through a multi-layer Transformer encoder, repeatedly fusing the fused emotional feature representation in each layer encoding process;

[0013] S6: Based on the VITS model, the jointly encoded features are converted into speech spectra, and the time domain speech waveform is generated through the decoder;

[0014] S7: Dynamically control the emotional expression of the synthesized speech by adjusting the weight ratio of each emotion category in the relative attribute vector:.

[0015] Optionally, in step S2, the emotion category relative ranking function set Each ranking function It is implemented by a deep neural network, including two layers of convolutional networks, one layer of LSTM network and two layers of fully connected layers;

[0016] The relative attribute vector The ranking function output value of all sentiment category pairs is obtained by calculating it, which is expressed as:

[0017]

[0018] in , the acoustic eigenvector is , are pre-training parameters, Represents the total number of emotion categories, and each pair of emotion categories is represented by ( , ).

[0019] Optionally, the step of performing weighted concatenation of the text feature vector and the fused sentiment feature representation to generate a joint feature vector includes:

[0020] The text feature vector The emotional feature vector output by the previous splicing emotional feature extractor Perform weighted splicing operation to fuse feature information from two different sources;

[0021] Introducing weight parameters and ,satisfy , weight the text feature vector and sentiment feature vector separately, and then concatenate them in dimensional order to obtain the joint feature vector :

[0022]

[0023]

[0024]

[0025] in, , , and the concatenated joint feature vector , Represents the text feature dimension, Indicates the number of categories, that is, how many class labels there are in the classification task, Indicates the number of combinations between categories. Dimensions representing other modalities / features and Correspondingly, the size of the space used to distinguish features from different sources.

[0026] Optionally, in step S5:

[0027] The multi-layer Transformer encoder has a 6-layer structure, and each layer input is represented by splicing the output of the previous layer and the fusion emotion feature. Implement joint coding, specifically expressed as:

[0028]

[0029] in For the The output features of the layer encoder.

[0030] Optionally, in step S6:

[0031] The posterior encoder of the VITS model takes the Mel spectrum as input to generate latent variables , and transform the latent variable distribution into a prior distribution through the flow model;

[0032] The decoder adopts the HiFiGAN V1 structure and generates time domain waveforms through multi-level upsampling.

[0033] Optionally, the method further includes a model training step:

[0034] Define a comprehensive objective function. The overall objective function consists of multiple sub-loss functions, which are expressed as:

[0035]

[0036] in, 、 、 and is a hyperparameter that weighs the importance of different sub-loss functions and satisfies , including: emotional features integrated into the loss , speech reconstruction loss , fight against losses and regularization loss ;

[0037] The AISHELL-3 or BZNSYP Chinese speech dataset is used for joint optimization to minimize the comprehensive objective function.

[0038] Optionally, the emotional features are integrated into the loss The ranking function output value is calculated by comparing the generated speech and the real emotion category. The specific formula is:

[0039]

[0040] in, It is a preset emotional margin value used to control the degree of distinction between the emotional characteristics of the generated speech and the real emotional category. For each training sample, its corresponding true emotion category, Indicates the number of samples, the total number of speech samples processed in the current batch, represents the sentiment ranking function, which is used to evaluate the relevance score between the generated features and the true sentiment category. Represents the parameters of the sentiment ranking model. The sentiment feature vector extracted after the model generates speech is , the relative attribute vector obtained by the ranking function is , is the number of emotion categories, and the two are concatenated to obtain the fusion emotion feature representation of the generated speech .

[0041] In a second aspect, the present application provides an emotion-controllable joint coding VITS speech synthesis device, comprising:

[0042] The extraction module is used to extract emotional features from the input speech sample and generate an emotional feature vector using a preset model;

[0043] A set construction module is used to construct a set of emotion category relative ranking functions, quantify the differences of multiple emotion categories of the acoustic features of the input speech sample, and generate a relative attribute vector;

[0044] A splicing module, configured to splice the emotion feature vector with the relative attribute vector to obtain a fused emotion feature representation;

[0045] A weighted concatenation module, configured to concatenate the text feature vector with the fused sentiment feature representation to generate a joint feature vector;

[0046] A joint encoding module, configured to jointly encode the joint feature vector using a multi-layer Transformer encoder, repeatedly fusing the fused emotional feature representation in each layer of encoding;

[0047] The conversion module is used to convert the jointly encoded features into a speech spectrum based on the VITS model and generate a time-domain speech waveform through a decoder;

[0048] The control module is used to dynamically control the emotional expression of the synthesized speech by adjusting the weight ratio of each emotion category in the relative attribute vector.

[0049] In a third aspect, the present application provides a computer device, comprising: a memory and a processor, wherein the processor executes the method described above when running computer instructions stored in the memory.

[0050] In a fourth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on a computer, enable the computer to execute the method described above.

[0051] In summary, the present application extracts emotional features from input speech samples and generates an emotional feature vector using a preset model; constructs a set of relative ranking functions for emotional categories, quantifies the differences of multiple emotional categories in the acoustic features of the input speech samples, and generates a relative attribute vector; concatenates the emotional feature vector with the relative attribute vector to obtain a fused emotional feature representation; performs weighted concatenation of the text feature vector with the fused emotional feature representation to generate a joint feature vector; jointly encodes the joint feature vector through a multi-layer Transformer encoder, and repeatedly fuses the fused emotional feature representation in each layer of encoding; based on the VITS model, converts the jointly encoded features into a speech spectrum, and generates a time-domain speech waveform through a decoder; and dynamically controls the emotional expression of the synthesized speech by adjusting the weight ratio of each emotional category in the relative attribute vector. This achieves an effective, convenient, flexible and controllable technical effect on the emotion of the synthesized speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic diagram of the computer device structure of the hardware operating environment involved in the embodiment of the present application;

[0053] Figure 2 This is a flowchart of the first embodiment of the emotion-controllable joint encoding VITS speech synthesis method of the present application;

[0054] Figure 3 This is the ranking function training and reasoning flowchart of this application, where (a) is the ranking function training flowchart, and (b) is the ranking function reasoning flowchart;

[0055] Figure 4 This is the reasoning process diagram of the model of this application;

[0056] Figure 5This is a structural block diagram of the first embodiment of the emotion-controllable joint coding VITS speech synthesis device of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below through the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0058] Reference Figure 1 , Figure 1 This is a schematic diagram of the computer device structure of the hardware operating environment involved in the embodiment of the present application.

[0059] like Figure 1 As shown, the computer device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display and an input unit, such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a wireless fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk storage device. The memory 1005 may also be a storage device independent of the processor 1001.

[0060] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0061] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and an emotion-controllable joint encoding VITS speech synthesis program.

[0062] exist Figure 1In the computer device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in this application can be set in the computer device, and the computer device calls the emotion-controllable joint coding VITS speech synthesis program stored in the memory 1005 through the processor 1001, and executes the emotion-controllable joint coding VITS speech synthesis method provided in the embodiment of this application.

[0063] The present application embodiment provides an emotion controllable joint coding VITS speech synthesis method, referring to Figure 2 , Figure 2 This is a flow chart of the first embodiment of the emotion-controllable joint encoding VITS speech synthesis method of this application.

[0064] In this embodiment, the emotion-controllable joint coding VITS speech synthesis method includes the following steps:

[0065] S1: Extract emotional features from the input speech sample and generate an emotional feature vector using a preset model.

[0066] In the specific implementation, this embodiment designs an emotional feature extractor , using wav2vec2.0 to extract feature vectors that can represent emotional information from a given emotional speech sample. Suppose the input emotional speech sample is First, perform conventional preprocessing on the speech signal, such as sampling and normalization, to make it meet the input requirements of the wav2vec2.0 model. Suppose the overall parameter set of the wav2vec2.0 model is After model processing, the output feature is represented as a vector of a specific dimension. The final output sentiment feature vector dimension is ), and define this output as the sentiment feature vector , its calculation process can be formally expressed as:

[0067] .

[0068] In specific implementation, the steps of training the sentiment category relative ranking function in this embodiment include: problem formulation and data preparation:

[0069] Suppose the emotion category set is ,in is the total number of emotion categories. For any pair of emotion categories (in , ), train a corresponding ranking function.

[0070] Emotional categories and The speech samples constitute subsets and ,in ( ) indicates that it belongs to the emotional category No. Voice samples, ( ) similarly indicates belonging to the sentiment category Voice samples.

[0071] For each speech sample and , extract its acoustic feature vector, recorded as and ( is the dimension of the acoustic eigenvector).

[0072] S2: Constructing a set of emotion category relative ranking functions, quantifying the differences of multiple emotion categories on the acoustic features of the input speech sample, and generating a relative attribute vector.

[0073] It should be noted that in step S2, the emotion category relative ranking function set Each ranking function It is implemented by a deep neural network, including two layers of convolutional networks, one layer of LSTM network and two layers of fully connected layers;

[0074] The relative attribute vector The ranking function output value of all sentiment category pairs is obtained by calculating it, which is expressed as:

[0075]

[0076] in , the acoustic eigenvector is , are pre-training parameters, Represents the total number of emotion categories, and each pair of emotion categories is represented by ( , ).

[0077] In a specific implementation, the step of obtaining the relative attribute vector using the pre-trained ranking function includes: for the input emotional speech sample , it also needs to be converted into an acoustic feature vector representation (the conversion process here can be performed in parallel or sequentially with the wav2vec2.0 feature extraction process. The focus is to ensure the consistency of the feature representation and ultimately present it in a form suitable for subsequent operations), recorded as (in , is the number of time frames, These acoustic feature vectors are input into the ranking function set trained for each pair of emotion categories (assuming that there are There are Different ranking functions, as described above ), obtain the relative attribute vector, which is used to represent the difference between the emotional style of the input speech and other emotional styles. For each pair of emotional categories ( , ) The corresponding ranking function (its parameters has been determined in the training phase), the acoustic feature vector Input in sequence to get the corresponding emotional attribute value , expressed as:

[0078]

[0079] Combine all the emotional attribute values ​​to form a relative attribute vector , for example by arranging these values ​​in order, formally expressed as:

[0080] .

[0081] In specific implementation, Figure 3 The ranking function training and reasoning flowchart shown, where (a) is the ranking function training flowchart, and (b) is the ranking function reasoning flowchart.

[0082] The steps for defining and structuring the ranking function include:

[0083] Defining the ranking function , whose input is from the emotion category or The acoustic feature vector of the speech sample ( Can be or ), the output is a sentiment attribute value , used to accurately quantify the input speech sample relative to the emotion category (reference emotion category) differences in emotional styles.

[0084] Ranking function The deep neural network architecture is used, which includes two convolutional layers, one LSTM layer and two fully connected layers. The parameter set is , then the calculation process of the entire ranking function can be formally expressed as:

[0085]

[0086] First, input the acoustic feature vector Entering the first convolutional layer, the convolution kernel size of this convolutional layer is ( Usually an odd number, such as 3 or 5), the number of convolution kernels (that is, the number of output channels) is , the step size is , the filling method is (It can be "same" or "valid", etc.) The calculation process is as follows:

[0087]

[0088] in is the convolution kernel tensor, is the bias term of the first convolutional layer, is the feature map output by the first convolutional layer ( and depends on factors such as input size, convolution kernel size, step size, and padding method), after After activation function processing, we get .

[0089] then, As the input of the second convolutional layer, the convolution kernel size of the second convolutional layer is , the number of convolution kernels is , the step size is , the filling method is , calculated as:

[0090]

[0091] Also passed Activate, get ,in , is the bias term of the second convolutional layer, and Represents the height and width of the feature map in the second convolutional layer.

[0092] Will Perform dimension transformation and rearrangement operations to make it suitable for input to the LSTM layer. Assume that the sequence obtained after transformation is (in , is the number of time steps), which is input into the LSTM layer. The LSTM layer contains hidden units, which at each time step The calculation process is as follows:

[0093]

[0094]

[0095]

[0096]

[0097]

[0098]

[0099] in 、 、 are the activation vectors of the input gate, forget gate, and output gate, respectively. is the candidate memory cell vector, is the memory cell state vector, is the hidden state vector, represents the sigmoid activation function, represents element-wise multiplication operation, is the corresponding weight matrix (whose dimension is determined by the input feature dimension and the hidden state dimension), are the bias vectors of the corresponding weight matrices. After calculating the entire time series, the final LSTM output hidden state sequence is obtained. , perform an average pooling operation on it to obtain the pooled vector , Refers to the pooled vector The dimension of LSTM is the hidden state sequence output by LSTM. After the average pooling operation, the result is is a vector, It is used to describe how many elements this vector has (i.e., the dimension size). Its value is determined by factors such as the specific model design and input features. It is used in subsequent processing steps such as fully connected layer input to ensure data dimension matching.

[0100] Finally, The first fully connected layer maps it to a dimension of The hidden layer weight matrix is , the bias vector is , calculated as:

[0101]

[0102] The second fully connected layer maps hidden layer 1 to the output layer to obtain the emotional attribute value , the weight matrix is , the bias vector is , calculated as follows:

[0103] .

[0104] S3: Concatenate the emotion feature vector and the relative attribute vector to obtain a fused emotion feature representation.

[0105] In a specific implementation, the step S3 includes: converting the emotional feature vector output by the emotional feature extractor based on wav2vec2.0 and the relative attribute vector obtained by the ranking function Perform splicing operations to obtain the fused emotional feature representation , the splicing operation is expressed as:

[0106]

[0107] Among them, if , ,but .

[0108] S4: Perform weighted concatenation on the text feature vector and the fused sentiment feature representation to generate a joint feature vector.

[0109] It can be understood that the step of weighted concatenation of the text feature vector and the fused sentiment feature representation to generate a joint feature vector includes:

[0110] The text feature vector The emotional feature vector output by the previous splicing emotional feature extractor Perform weighted splicing operation to fuse feature information from two different sources;

[0111] Introducing weight parameters and ,satisfy , weight the text feature vector and sentiment feature vector separately, and then concatenate them in dimensional order to obtain the joint feature vector :

[0112]

[0113]

[0114]

[0115] in, , , and the concatenated joint feature vector , Represents the text feature dimension. After the text is encoded (word embedding, Transformer encoding, etc.), the length of the vector converted into each text sample is a parameter that describes the size of the text feature space. Indicates the number of categories (the first letter of Classification), that is, how many class labels there are in the classification task. The form indicates the calculation of the number of combinations between categories (the number of pairwise combinations when modeling the relationship between categories). Dimensions representing other modalities / features and Correspondingly, the size of the space used to distinguish features from different sources.

[0116] In the specific implementation, the joint feature vector As input, it is fed into the 6-layer Transformer text encoder of VITS2. Let each layer of the Transformer text encoder be ( ), each layer has its corresponding parameter set , mainly including parameters related to the Multi-Head Attention mechanism (MHA) and the Feed-Forward Network (FFN).

[0117] For the first layer Transformer encoder, its input is the joint feature vector First, feature interaction and information extraction are performed through the multi-head attention mechanism layer. The calculation process is as follows:

[0118]

[0119]

[0120]

[0121]

[0122] in, 、 、 They are query, key, and value vectors, respectively, and are transformed linearly (by parameters 、 、 control) from the input get; is the output after multi-head attention mechanism processing, and its dimension is the same as the input The dimensions of .

[0123] Next, the output of the multi-head attention mechanism Send it to the feedforward neural network for further feature transformation, and the calculation is:

[0124]

[0125] Get the final output of the first layer Transformer encoder , the same dimension remains unchanged, that is .

[0126] For the ( ) layer Transformer encoder, whose input contains not only the output of the previous layer , and the previously concatenated emotional feature vector will be introduced again Joint encoding is performed to continuously strengthen the influence of emotional features in the entire encoding process. The specific calculation process is as follows:

[0127]

[0128]

[0129]

[0130]

[0131]

[0132]

[0133] in, It is The input of the layer is obtained by concatenating the output of the previous layer with the sentiment feature vector, and the dimension is (Because the sentiment feature vector is concatenated once); 、 、 They are the corresponding query, key, and value vectors, which are transformed from get; is the output of the multi-head attention mechanism, is the final output after the feedforward neural network, and both dimensions are .

[0134] After the layer-by-layer cascade processing of the 6-layer Transformer text encoder, the feature representation after joint encoding is finally obtained. This feature representation integrates text information and emotional feature information, and fully considers the influence of emotional feature vectors in each layer of encoding process, so that it can be fully integrated into the subsequent speech synthesis process.

[0135] S5: The joint feature vector is jointly encoded through a multi-layer Transformer encoder, and the fused emotional feature representation is repeatedly fused during each layer of encoding.

[0136] It should be noted that, in step S5:

[0137] The multi-layer Transformer encoder has a 6-layer structure, and each layer input is represented by splicing the output of the previous layer and the fusion emotion feature. Implement joint coding, specifically expressed as:

[0138]

[0139] in For the The output features of the layer encoder.

[0140] S6: Based on the VITS model, the jointly encoded features are converted into speech spectrum, and the time domain speech waveform is generated through the decoder.

[0141] It is understandable that in step S6, the posterior encoder of the VITS model generates latent variables using the Mel spectrum as input. , and converts the latent variable distribution into a priori distribution through a flow model; the decoder adopts the HiFiGANV1 structure and generates a time domain waveform through multi-level upsampling.

[0142] S7: Dynamically control the emotional expression of the synthesized speech by adjusting the weight ratio of each emotion category in the relative attribute vector:.

[0143] It should be noted that the method also includes a model training step: defining a comprehensive objective function, which is composed of multiple sub-loss functions, expressed as:

[0144]

[0145] in, 、 、 and is a hyperparameter that weighs the importance of different sub-loss functions and satisfies , including: emotional features integrated into the loss , speech reconstruction loss , fight against losses and regularization loss ;

[0146] The AISHELL-3 or BZNSYP Chinese speech dataset is used for joint optimization to minimize the comprehensive objective function.

[0147] In specific implementation, the mathematical expression of the comprehensive objective function is:

[0148]

[0149] The corresponding formula symbols are explained in Table 1:

[0150]

[0151] Expand in detail:

[0152]

[0153] Explanation of symbols:

[0154] : The true emotion category label of the sample;

[0155] : Traverse all non-real emotion categories;

[0156] :against Ranking function for sentiment category pairs;

[0157] : Generate the fusion emotion feature of speech, which is spliced ​​by two parts:

[0158] : Generate emotional feature vector of speech (dimension );

[0159] : relative attribute vector (dimension );

[0160] : preset sentiment discrimination margin value (hyperparameter);

[0161] Hinge Loss: It ensures that a penalty is imposed when the ranking difference is insufficient.

[0162] All loss weight coefficients satisfy the normalization constraint:

[0163] .

[0164] It should be noted that the speech reconstruction loss is designed to measure the difference between the generated speech and the real speech to ensure that the model can accurately generate the corresponding speech waveform based on the input text and emotional features. samples, for the samples, and its true speech waveform is expressed as ( is the number of time frames, is the speech waveform feature dimension, such as the Mel spectrum feature dimension, etc.), the speech waveform generated by the model is , then the speech reconstruction loss is calculated as follows:

[0165]

[0166] in, Represents a vector The norm is calculated by calculating the sum of the squares of the feature differences between the real speech and the generated speech of each sample and taking the average to obtain the overall speech reconstruction loss. This loss needs to be minimized during the model training process to improve the accuracy of the generated speech.

[0167] It is understandable that in order to further improve the naturalness and quality of generated speech, the adversarial learning mechanism is introduced to build a discriminator. (Its parameters are ) to distinguish real speech from generated speech. The input is a real speech waveform or a generated speech waveform (the dimension is consistent with the speech waveform feature dimension defined above, that is, ), the output is a scalar, which represents the probability that the input speech is the real speech. The calculation process can be expressed as:

[0168]

[0169] in, Indicates real voice, Indicates generated speech.

[0170] The adversarial loss function aims to make the speech generated by the generator as deceptive as possible to the discriminator, while allowing the discriminator to accurately distinguish between real speech and generated speech. It is defined as follows:

[0171]

[0172] in, represents the adversarial loss function, which is used to measure the difference between the generated speech and the real speech under the discriminator;

[0173] Represents the mathematical expectation, which is the average calculation of samples within the distribution;

[0174] Represents real speech samples, which obey the distribution of real speech data ;

[0175] Represents the speech samples generated by the model, which obey the generated speech distribution ;

[0176] represents the discriminator network, which inputs a speech sample and outputs the probability that it is the real speech (in the range [0,1]), are the parameters of the discriminator;

[0177] Represents the natural logarithm, which is used to calculate the loss value of probability.

[0178] By minimizing this adversarial loss function, the generator and discriminator are continuously optimized during the adversarial training process, improving the naturalness of the generated speech and making it closer to real speech.

[0179] In the specific implementation, the real speech loss part ;

[0180] Goal: Make the discriminator output the probability of real speech As close to 1 as possible, Approaching 0 (due to ), with minimal loss.

[0181] Function: Force the discriminator to accurately identify real speech and avoid misclassifying real speech as generated speech.

[0182] Generate speech loss part ;

[0183] Goal: Make the probability of generated speech being identified as real speech As close to 1 as possible, Approaching 0, As the loss approaches negative infinity, the generator tries to minimize the loss. Increase it to make the generated speech closer to the real speech.

[0184] Purpose: Force the generator to generate more realistic speech samples to "fool" the discriminator.

[0185] It should be noted that in order to prevent the model from overfitting and improve the generalization ability of the model, regularization loss is introduced to all trainable parameters in the model (assuming that all parameter sets are ) is constrained, and its calculation is as follows:

[0186]

[0187] Here Represents all trainable parameters in the model By taking the average of the sum of squares of all parameters, the parameter size is constrained to avoid overfitting of the model and improve the generalization ability.

[0188] By adding a regularization loss term to the objective function and combining the corresponding regularization coefficient , during the training process, the size of the model parameters is limited to avoid overfitting of the model on the training data, so that the model can perform better on unseen data.

[0189] It is understandable that the emotional characteristics are integrated into the loss The ranking function output value is calculated by comparing the generated speech and the real emotion category. The specific formula is:

[0190]

[0191] in, It is a preset emotional margin value used to control the degree of distinction between the emotional characteristics of the generated speech and the real emotional category. For each training sample, its corresponding true emotion category, Indicates the number of samples, the total number of speech samples processed in the current batch, represents the sentiment ranking function, which is used to evaluate the relevance score between the generated features and the true sentiment category. Represents the parameters of the sentiment ranking model. The sentiment feature vector extracted after the model generates speech is , the relative attribute vector obtained by the ranking function is , is the number of emotion categories, and the two are concatenated to obtain the fusion emotion feature representation of the generated speech .

[0192] It should be noted that the steps of training the target and loss function in this embodiment also include: training the ranking function The goal is to enable it to accurately predict the relative differences between different emotion categories based on the input acoustic features, that is, for the emotion categories The sample whose output sentiment attribute value should be smaller than that from the sentiment category The sentiment attribute value of the sample output.

[0193] Define the following ranking loss function (RankLossFunction) :

[0194]

[0195] in Is a pre-set margin value (Margin), which is used to enhance the discrimination between samples of different emotion categories. The meaning of this loss function is that for each pair of samples from different emotion categories ( and ) samples, if they come from the emotion category The sentiment attribute value output by the ranking function of the sample is greater than that from the sentiment category The sample output value plus the marginal value , then a corresponding loss will be generated, and this loss should be minimized during model training.

[0196] In addition, to prevent overfitting and make the output of the ranking function more generalizable, add Regularization term, the final target loss function is obtained:

[0197]

[0198] in is the regularization coefficient, which is used to weigh the importance of ranking loss and regularization term. Represents the parameter vector of Norm squared.

[0199] In the specific implementation, the model reasoning process is as shown in the figure Figure 4 The detailed process of the inference phase is shown in Figure 2. During the model's inference phase, emotion extraction is first used to obtain embedded emotion types from the reference speech. A ranking function is then used to introduce features of other emotion types with manually assigned attribute vectors. Subsequently, by varying the percentage of each primary emotion in the attribute vector, the desired emotional effect can be easily synthesized, effectively controlling the emotional rendering of the synthesized speech.

[0200] This embodiment extracts emotional features from input speech samples and generates an emotional feature vector using a preset model; constructs a set of emotional category relative ranking functions, quantifies the differences of multiple emotional categories in the acoustic features of the input speech samples, and generates a relative attribute vector; concatenates the emotional feature vector with the relative attribute vector to obtain a fused emotional feature representation; performs weighted concatenation of the text feature vector and the fused emotional feature representation to generate a joint feature vector; jointly encodes the joint feature vector through a multi-layer Transformer encoder, and repeatedly fuses the fused emotional feature representation in each layer of encoding; based on the VITS model, converts the jointly encoded features into a speech spectrum, and generates a time-domain speech waveform through a decoder; and dynamically controls the emotional expression of the synthesized speech by adjusting the weight ratio of each emotional category in the relative attribute vector. This achieves an effective, convenient, flexible and controllable technical effect on the emotion of the synthesized speech.

[0201] In addition, an embodiment of the present application also proposes a computer-readable storage medium, which stores a program for emotion-controllable joint coding VITS speech synthesis. When the program for emotion-controllable joint coding VITS speech synthesis is executed by a processor, the steps of the method for emotion-controllable joint coding VITS speech synthesis as described above are implemented.

[0202] Reference Figure 5 , Figure 5 This is a structural block diagram of the first embodiment of the emotion-controllable joint encoding VITS speech synthesis device of this application.

[0203] like Figure 5 As shown, the emotion-controllable joint coding VITS speech synthesis device proposed in the embodiment of the present application includes:

[0204] Extraction module 10, used to extract emotional features from input speech samples and generate emotional feature vectors using a preset model;

[0205] A set construction module 20 is used to construct a set of emotion category relative ranking functions, quantify the differences of multiple emotion categories of the acoustic features of the input speech sample, and generate a relative attribute vector;

[0206] a splicing module 30, configured to splice the emotion feature vector and the relative attribute vector to obtain a fused emotion feature representation;

[0207] A weighted concatenation module 40 is configured to perform weighted concatenation on the text feature vector and the fused sentiment feature representation to generate a joint feature vector;

[0208] A joint encoding module 50, configured to jointly encode the joint feature vector using a multi-layer Transformer encoder, repeatedly fusing the fused emotional feature representation in each layer of encoding;

[0209] A conversion module 60 is used to convert the jointly encoded features into a speech spectrum based on the VITS model, and generate a time-domain speech waveform through a decoder;

[0210] The control module 70 is used to dynamically control the emotional expression of the synthesized speech by adjusting the weight ratio of each emotion category in the relative attribute vector:.

[0211] It should be understood that the above is only an example and does not constitute any limitation to the technical solution of the present application. In specific applications, technicians in this field can make settings as needed, and the present application does not impose any restrictions on this.

[0212] This embodiment extracts emotional features from input speech samples and generates an emotional feature vector using a preset model; constructs a set of emotional category relative ranking functions, quantifies the differences of multiple emotional categories in the acoustic features of the input speech samples, and generates a relative attribute vector; concatenates the emotional feature vector with the relative attribute vector to obtain a fused emotional feature representation; performs weighted concatenation of the text feature vector and the fused emotional feature representation to generate a joint feature vector; jointly encodes the joint feature vector through a multi-layer Transformer encoder, and repeatedly fuses the fused emotional feature representation in each layer of encoding; based on the VITS model, converts the jointly encoded features into a speech spectrum, and generates a time-domain speech waveform through a decoder; and dynamically controls the emotional expression of the synthesized speech by adjusting the weight ratio of each emotional category in the relative attribute vector. This achieves an effective, convenient, flexible and controllable technical effect on the emotion of the synthesized speech.

[0213] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this application. In actual applications, technicians in this field can select part or all of it according to actual needs to achieve the purpose of this embodiment scheme, and no restrictions are imposed here.

[0214] In addition, for technical details not fully described in this embodiment, please refer to the method of emotion-controllable joint coding VITS speech synthesis provided in any embodiment of this application, and will not be repeated here.

[0215] In addition, it should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0216] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0217] Through the above description of the embodiments, those skilled in the art will clearly understand that the above-mentioned embodiments and methods can be implemented using software plus the necessary general-purpose hardware platform. Of course, hardware can also be used, but in many cases the former is a more preferred embodiment. Based on this understanding, the technical solution of this application, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, a magnetic disk, or an optical disk) and includes several instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of this application. The above are only preferred embodiments of this application and do not limit the scope of the patent application. Any equivalent structure or equivalent process transformation made using the contents of this application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the scope of patent protection of this application.

Claims

1. An emotion-controllable joint coding VITS speech synthesis method, characterized in that: include: S1: Extract emotional features from the input speech sample and generate an emotional feature vector using a preset model; S2: constructing a set of emotion category relative ranking functions, quantifying the differences of multiple emotion categories on the acoustic features of the input speech sample, and generating a relative attribute vector; S3: concatenating the emotion feature vector and the relative attribute vector to obtain a fused emotion feature representation; S4: performing weighted concatenation of the text feature vector and the fused sentiment feature representation to generate a joint feature vector; S5: jointly encoding the joint feature vector through a multi-layer Transformer encoder, repeatedly fusing the fused emotional feature representation in each layer encoding process; S6: Based on the VITS model, the jointly encoded features are converted into speech spectra, and the time domain speech waveform is generated through the decoder; S7: Dynamically controlling the emotional expression of the synthesized speech by adjusting the weight ratio of each emotion category in the relative attribute vector.

2. The method according to claim 1, characterized in that In step S2, the emotion category relative ranking function set Each ranking function It is implemented by a deep neural network, including two layers of convolutional networks, one layer of LSTM network and two layers of fully connected layers; The relative attribute vector The ranking function output value of all sentiment category pairs is obtained by calculating it, which is expressed as: in , the acoustic eigenvector is , are pre-training parameters, Represents the total number of emotion categories, and each pair of emotion categories is represented by , , .

3. The method according to claim 1, characterized in that The step of weighted concatenating the text feature vector and the fused sentiment feature representation to generate a joint feature vector includes: The text feature vector The emotional feature vector output by the previous splicing emotional feature extractor Perform weighted splicing operation to fuse feature information from two different sources; Introducing weight parameters and ,satisfy , weight the text feature vector and sentiment feature vector separately, and then concatenate them in dimensional order to obtain the joint feature vector : in, , , and the concatenated joint feature vector , Represents the text feature dimension, Indicates the number of categories, that is, how many class labels there are in the classification task, Indicates the number of combinations between categories. Dimensions representing other modalities / features and Correspondingly, the size of the space used to distinguish features from different sources.

4. The method according to claim 1, wherein In the step S5: The multi-layer Transformer encoder has a 6-layer structure, and each layer input is represented by splicing the output of the previous layer and the fusion emotion feature. Implement joint coding, specifically expressed as: in For the The output features of the layer encoder.

5. The method according to claim 1, wherein In the step S6: The posterior encoder of the VITS model takes the Mel spectrum as input to generate latent variables , and transform the latent variable distribution into a prior distribution through the flow model; The decoder adopts the HiFiGAN V1 structure and generates time domain waveforms through multi-level upsampling.

6. The method according to claim 1, characterized in that The method also includes a model training step: Define a comprehensive objective function. The overall objective function consists of multiple sub-loss functions, which are expressed as: in, 、 、 and is a hyperparameter that weighs the importance of different sub-loss functions and satisfies , including: emotional features integrated into the loss , speech reconstruction loss , fight against losses and regularization loss ; The AISHELL-3 or BZNSYP Chinese speech dataset is used for joint optimization to minimize the comprehensive objective function.

7. The method according to claim 6, characterized in that The emotional characteristics are integrated into the loss The ranking function output value is calculated by comparing the generated speech and the real emotion category. The specific formula is: in, It is a preset emotional margin value used to control the degree of distinction between the emotional characteristics of the generated speech and the real emotional category. For each training sample, its corresponding true emotion category, Indicates the number of samples, the total number of speech samples processed in the current batch, represents the sentiment ranking function, which is used to evaluate the relevance score between the generated features and the true sentiment category. Represents the parameters of the sentiment ranking model. The sentiment feature vector extracted after the model generates speech is , the relative attribute vector obtained by the ranking function is , is the number of emotion categories, and the two are concatenated to obtain the fusion emotion feature representation of the generated speech .

8. An emotion-controllable joint coding VITS speech synthesis device, characterized in that: include: The extraction module is used to extract emotional features from the input speech sample and generate an emotional feature vector using a preset model; A set construction module is used to construct a set of emotion category relative ranking functions, quantify the differences of multiple emotion categories of the acoustic features of the input speech sample, and generate a relative attribute vector; A splicing module, configured to splice the emotion feature vector with the relative attribute vector to obtain a fused emotion feature representation; A weighted concatenation module, configured to concatenate the text feature vector with the fused sentiment feature representation to generate a joint feature vector; A joint encoding module, configured to jointly encode the joint feature vector using a multi-layer Transformer encoder, repeatedly fusing the fused emotional feature representation in each layer of encoding; The conversion module is used to convert the jointly encoded features into a speech spectrum based on the VITS model and generate a time-domain speech waveform through a decoder; The control module is used to dynamically control the emotional expression of the synthesized speech by adjusting the weight ratio of each emotion category in the relative attribute vector.

9. A computer device, characterized in that: The device comprises: a memory and a processor, wherein the processor executes the method according to any one of claims 1 to 7 when running computer instructions stored in the memory.

10. A computer-readable storage medium, characterized in that The method comprises instructions which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 7.

Images