A speech generation method and a speech generation device based on timbre features
By generating target timbre vectors through target parsing and fusion models, the problem of accurately capturing subtle emotional differences in existing technologies is solved, achieving high-precision speech generation results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI XIYU JIZHI TECH CO LTD
- Filing Date
- 2025-07-15
- Publication Date
- 2026-07-07
Smart Images
Figure CN120600002B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of audio technology, and in particular to a speech generation method and speech generation device based on timbre features. Background Technology
[0002] With the rapid development of speech synthesis technology, text-based speech synthesis technology has been widely used in scenarios such as intelligent assistants and voice interaction. Current speech synthesis technology not only focuses on the clarity and naturalness of speech, but also pursues fine control over the emotion and timbre of speech.
[0003] Currently, timbre generation typically relies on pre-trained models, which can only achieve basic timbre modulation using a limited number of emotion tags or reference speech. For example, they can distinguish significantly different emotional states such as happiness and sadness. This modulation mainly depends on the prior timbre distribution learned by the pre-trained model. However, because similar emotions have highly overlapping spectral features, existing models struggle to accurately capture subtle differences in emotional expression, such as excitement and happiness. This results in generated speech lacking sufficient expressiveness and discriminatory power, failing to meet users' demands for diverse and high-precision speech generation. Summary of the Invention
[0004] In view of this, the purpose of this application is to provide a speech generation method and speech generation device based on timbre features. The method generates sound feature vectors corresponding to timbre text descriptions through a target parsing model, generates target timbre vectors based on the sound feature vectors through a target fusion model, and then generates target speech with timbre features that meet the user's requirements based on the timbre vectors. This improves the accuracy of speech generation under specified timbre and meets the user's needs for speech generation with diverse timbres and high-precision emotional expression.
[0005] In a first aspect, embodiments of this application provide a speech generation method based on timbre features, the speech generation method comprising:
[0006] Obtain the target parsing model, input the description text into the target parsing model, and obtain the target sound feature vector corresponding to the description text; wherein, the description text includes a target timbre text description;
[0007] The target sound feature vector is input into the target fusion model to generate the target timbre vector;
[0008] The target timbre vector and the text to be converted are input into the speech generation model to obtain the target speech that matches the described text.
[0009] Furthermore, the speech generation method also includes:
[0010] The target parsing model obtains the target voice feature vector and the target emotion space vector corresponding to the description text based on the description text. The target fusion model receives and fuses the target voice feature vector and the target emotion space vector to generate the target timbre vector. The target voice feature vector and the target emotion space vector are independent of each other.
[0011] Furthermore, the target fusion model receives and fuses the target voice feature vector and the target emotion space vector to generate the target timbre vector, including:
[0012] The target sound feature vector is mapped to at least one vector subspace to obtain at least one subvector; wherein each vector subspace represents an acoustic feature in an independent dimension;
[0013] Adjust the weights and / or offsets of each sub-vector according to the target emotion space vector;
[0014] The adjustment results of each sub-vector are combined to obtain the target timbre vector.
[0015] Furthermore, the target fusion model adjusts the weights and / or offsets of each sub-vector based on the target emotion space vector, including:
[0016] A gating unit is configured for each vector subspace, and each vector subspace is connected to an independent gating channel in the gating unit;
[0017] In response to receiving the target emotion space vector, the independent gating channel determines the scaling factor and offset factor of the corresponding vector subspace based on the target emotion space vector;
[0018] The weights and / or offsets of the subvectors in the corresponding vector subspace are adjusted according to the scaling factor and offset factor.
[0019] Furthermore, the target emotion space vector is a three-dimensional emotion space vector, including a first emotion dimension space, a second emotion dimension space, and a third emotion dimension space. The first emotion dimension space, the second emotion dimension space, and the third emotion dimension space are all continuous spaces and independent of each other.
[0020] Furthermore, the speech generation model includes an encoder, an acoustic module, and a decoder. The step of inputting the target timbre vector and the text to be converted into the speech generation model includes:
[0021] The text to be converted is input into the encoder to obtain a text encoding vector. The text encoding vector and the target timbre vector are input into the acoustic module and then into the decoder to obtain the target speech that matches the described text.
[0022] Furthermore, the target analytical model is trained through the following steps:
[0023] Acquire sample speech, input the timbre description sample corresponding to the sample speech into the initial parsing model to obtain the predicted sound feature vector and the predicted emotion space vector; wherein, the predicted sound feature vector is a low-dimensional vector with fixed dimensions;
[0024] A first loss function is established based on the real sound feature vector, the real emotion space vector, the predicted sound feature vector, and the predicted emotion space vector corresponding to the sample speech. The initial parsing model is then iteratively trained based on the first loss function until the preset training completion conditions are met, thus obtaining the trained target parsing model.
[0025] Furthermore, the target fusion model is trained through the following steps:
[0026] The target voice feature vector and the target emotion space vector are input into the initial fusion model to obtain the predicted timbre vector; wherein, the predicted timbre vector is a low-dimensional vector with fixed dimensions;
[0027] A second loss function is established based on the real timbre vector corresponding to the sample speech and the predicted timbre vector. The initial fusion model is then iteratively trained based on the second loss function until the preset training completion conditions are met, thus obtaining the trained target fusion model.
[0028] Furthermore, the speech generation method also includes:
[0029] When training the target parsing model, the target parsing model is kept decoupled from the speech generation model;
[0030] When training the target fusion model, the target fusion model is kept decoupled from the target parsing model and the speech generation model.
[0031] Secondly, embodiments of this application also provide a speech generation device based on timbre features, the speech generation device comprising:
[0032] The sound feature vector determination module is used to obtain a target parsing model, input the descriptive text into the target parsing model, and obtain the target sound feature vector corresponding to the descriptive text; wherein, the descriptive text includes a target timbre text description;
[0033] The timbre vector determination module is used to input the target sound feature vector into the target fusion model to generate the target timbre vector;
[0034] The target speech generation module is used to input the target timbre vector and the text to be converted into the speech generation model to obtain the target speech that matches the description text.
[0035] This application provides a speech generation method and apparatus based on timbre features. First, a target parsing model is obtained, and descriptive text is input into the target parsing model to obtain a target sound feature vector corresponding to the descriptive text; wherein, the descriptive text includes a target timbre text description; then, the target sound feature vector is input into a target fusion model to generate a target timbre vector; finally, the target timbre vector and the text to be converted are input into a speech generation model to obtain target speech that conforms to the descriptive text.
[0036] This application introduces sound feature vectors, generates sound feature vectors corresponding to the timbre text description through a target parsing model, generates target timbre vectors based on the sound feature vectors, and then generates target speech that meets the timbre features required by the user based on the timbre vectors. This improves the accuracy of speech generation under a specified timbre and meets the user's needs for speech generation with diverse timbres and high-precision emotional expression.
[0037] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0038] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0039] Figure 1 A flowchart illustrating a speech generation method based on timbre features provided in an embodiment of this application;
[0040] Figure 2 This is one of the structural schematic diagrams of a speech generation device based on timbre features provided in an embodiment of this application;
[0041] Figure 3 This is a second schematic diagram of a speech generation device based on timbre features provided in an embodiment of this application;
[0042] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0043] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.
[0044] First, the applicable scenarios for this application will be introduced. This application can be applied to the field of audio technology.
[0045] With the rapid development of speech synthesis technology, text-based speech synthesis technology has been widely used in scenarios such as intelligent assistants and voice interaction. Current speech synthesis technology not only focuses on the clarity and naturalness of speech, but also pursues fine control over the emotion and timbre of speech.
[0046] Research has found that current timbre generation typically relies on pre-trained models, which can only achieve basic timbre modulation using a limited number of emotion tags or reference speech. For example, they can distinguish significantly different emotional states such as happiness and sadness. This modulation mainly depends on the prior timbre distribution learned by the pre-trained model. However, because similar emotions have highly overlapping spectral features, existing models struggle to accurately capture subtle differences in emotional expression, such as excitement and happiness. This results in generated speech lacking sufficient expressiveness and discriminatory power, failing to meet users' demands for diverse and high-precision speech generation.
[0047] Based on this, embodiments of this application provide a speech generation method based on timbre features to generate target speech that meets the timbre features required by the user, thereby improving the accuracy of speech generation under specified timbre and meeting the user's needs for speech generation with diverse timbre and high-precision emotional expression.
[0048] Please see Figure 1 , Figure 1 This is a flowchart illustrating a speech generation method based on timbre features provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the speech generation method includes:
[0049] S101, Obtain the target parsing model by inputting the description text into the target parsing model to obtain the target sound feature vector corresponding to the description text.
[0050] Here, the descriptive text includes the target timbre text description. The target timbre text description refers to the text information used to describe the timbre features of the speech to be generated. For example, the target timbre text description can be words that describe timbre from multiple dimensions, such as young, old, female, male, sharp, bright, etc., and this application does not make specific limitations on this.
[0051] For step S101 above, a pre-trained target parsing model is obtained, and the description text is input into the target parsing model to obtain the target sound feature vector corresponding to the description text.
[0052] S102, the target sound feature vector is input into the target fusion model to generate the target timbre vector.
[0053] In specific implementation of step S102, the target sound feature vector output by the target parsing model is input into the pre-trained target fusion model to obtain the target timbre vector.
[0054] Specifically, according to the speech generation method provided in this application, the speech generation method further includes:
[0055] A: Input the descriptive text into the target parsing model to obtain the target sound feature vector and target emotion space vector corresponding to the descriptive text.
[0056] Here, the target parsing model can not only parse the voice features from the descriptive text, but also parse the emotion labels from the descriptive text. Regarding step A above, in specific implementation, the descriptive text is input into the target parsing model. Based on the content of the descriptive text, the target parsing model parses the voice features and emotion labels from the descriptive text, thus obtaining the target voice feature vector and the target emotion space vector.
[0057] Specifically, the target emotion space vector is a three-dimensional emotion space vector, including a first emotion dimension space, a second emotion dimension space, and a third emotion dimension space. The first emotion dimension space, the second emotion dimension space, and the third emotion dimension space are all continuous spaces and independent of each other.
[0058] Here, a continuous-dimensional PAD three-dimensional emotion framework is introduced into the target emotion space vector to represent emotions in detail. The PAD three-dimensional emotion framework (Pleasure-Arousal-Dominance) is an emotion framework used to quantitatively describe human emotional states. It can continuously and smoothly express different categories of emotions. It expresses a variety of emotions based on a three-dimensional coordinate system composed of three core dimensions: Pleasure (Valence) measures the positive or negative degree of emotion, such as happiness being high pleasure and sadness being low pleasure; Activation (Arousal) measures the physiological arousal level of emotion, such as calmness being low activation and mania being high activation; Dominance measures the sense of control or compliance with emotion, such as anger being high dominance and fear being low dominance. The target emotion space vector [P,A,D], composed of these three continuously varying parameters, can reasonably represent most emotions. The target parsing model generates the target emotion space vector, enabling the subsequent target fusion model to understand the changes in timbre of relatively continuously varying emotional details. For example, the acoustic spectral characteristics corresponding to the three emotions of choking, sobbing, and crying are almost the same, but based on the PAD three-dimensional emotion framework, different activation values can be used to represent the subtle differences in the expression of these three emotions. Simultaneously, the spatial results of multiple emotions coupling can be understood based on this target emotion space vector. In a preferred embodiment, the elements in the target emotion space vector [P,A,D] have the same value range; for example, P,A,D∈[-1,1]. Taking the pleasure dimension as an example, a value closer to 1 indicates greater pleasure, and a value closer to -1 indicates less pleasure. The same applies to the activation and dominance dimensions. It should be understood that the above is only an example; P,A,D can also have other values, and the above examples should not be construed as limiting the solution of this application.
[0059] Thus, by introducing the PAD three-dimensional emotion framework, this application not only enables the model to learn the different expressions of subtle emotional differences in timbre, but also allows the model to decouple different dimensions of emotion in situations where multiple emotions are mixed. Taking the emotions of "crying with joy" and "crying with sadness" as examples, in the three-dimensional emotion space, "crying with joy" and "crying with sadness" are two different vectors. Therefore, using the target emotion space vector in timbre expression can more clearly express the difference between the two, and at the same time, it can also enable the model to quickly learn the difference between the two emotions. Meanwhile, the decoupling between the various dimensions of the target emotion space vector can ensure that the model's expression of emotion is not affected by the emotional tone of happiness or sadness when expressing emotions.
[0060] B: Input the target voice feature vector and the target emotion space vector into the target fusion model to generate the target timbre vector.
[0061] Regarding step B above, in specific implementation, the target fusion model fuses the target voice feature vector and the target emotion space vector. The target voice feature vector and the target emotion space vector are input into the target fusion model to generate the target timbre vector. Here, the target voice feature vector and the target emotion space vector are independent of each other.
[0062] Furthermore, the target fusion model introduces a Conditional Fusion Network (CFN). Based on attention or gating mechanisms, the CFN modulates and / or enhances one or more acoustic feature values according to the values of each dimension in different target emotion space vectors. Specifically, regarding step B above, inputting the target voice feature vector and the target emotion space vector into the target fusion model to generate the target timbre vector includes:
[0063] a: Map the target sound feature vector to at least one vector subspace to obtain at least one subvector.
[0064] Regarding step a above, in specific implementation, the target sound feature vector is first mapped to at least one vector subspace to obtain at least one subvector. Here, each vector subspace represents the acoustic features in an independent dimension.
[0065] Specifically, vector subspaces can include: a pitch (F0) subspace, controlling the rise and fall of intonation, such as raising the pitch at the end of an interrogative sentence; an energy subspace, controlling the stress and cadence of pronunciation, such as strengthening stressed syllables to enhance emotional expression; a duration subspace, controlling speech rate and rhythm, such as speaking slowly when the emotion is sadness and speaking quickly when the emotion is excitement; a timbre subspace, used to represent the speaker's identity, such as the timbre characteristics of factors like gender and age, and specific timbre textures, such as hoarseness or mellowness; and a style subspace, capturing personalized pronunciation features, such as dialect, accent, and rhythm preferences. The above vector subspaces are merely examples and should not be construed as limiting the scheme of this application.
[0066] b: The target fusion model adjusts the weights and / or offsets of each sub-vector according to the target emotion space vector.
[0067] Regarding step b above, in practical implementation, the target fusion model adjusts the weights and / or offsets of each sub-vector based on the target emotion space vector. Specifically, the target fusion model uses the target emotion space vector as a condition and dynamically adjusts the weights and / or offsets of sub-vectors in these vector subspaces through attention mechanisms or gating mechanisms. For example, when the target emotion space vector indicates "high activation," corresponding to an emotional arousal state (such as anger), leading to an amplification of fundamental frequency fluctuations and a surge in vocal energy, the target fusion model tends to enhance fundamental frequency F0 modulation and increase vocal energy; when it indicates "low pleasure," corresponding to negative emotions (such as sadness), the target fusion model tends to reduce the average level of fundamental frequency F0 and increase the smoothness of speech rate.
[0068] Specifically, regarding step b above, the target fusion model adjusts the weights and / or offsets of each sub-vector based on the target emotion space vector, including:
[0069] (1) Configure gating units for vector subspaces, with each vector subspace connected to an independent gating channel in the gating unit.
[0070] (2) In response to receiving the target emotion space vector, the independent gating channel determines the scaling factor and offset factor of the corresponding vector subspace based on the target emotion space vector.
[0071] (3) Adjust the weights and / or offsets of the subvectors in the corresponding vector subspace according to the scaling factor and offset factor.
[0072] Regarding steps (1) to (3) above, in specific implementation, the target fusion model configures gating units for the vector subspaces, with each vector subspace connected to an independent gating channel within the gating unit. Specifically, the gating unit can be implemented based on a multilayer perceptron (MLP) and perform end-to-end training together with the target fusion model, enabling better fusion of the target sound feature vector and the target emotion space vector. When the target fusion model receives the target emotion space vector, the independent gating channel determines the scaling factor and / or offset factor of the corresponding vector subspace based on the target emotion space vector, and then adjusts the weights and / or offsets of the subvectors of the corresponding vector subspace according to the scaling factor and offset factor.
[0073] c: The adjustment results of each sub-vector are fused to obtain the target timbre vector.
[0074] Regarding step c above, in specific implementation, after adjusting the weights and / or offsets of each sub-vector in the target fusion model, the adjustment results of each sub-vector are fused to obtain the target timbre vector.
[0075] S103, input the target timbre vector and the text to be converted into the speech generation model to obtain the target speech that matches the description text.
[0076] Regarding step S103 above, in specific implementation, the target timbre vector generated by the target fusion model and the text to be converted are input into the speech generation model to generate target speech that conforms to the descriptive text. Preferably, the speech generation model is a TTS (Text-to-Speech) model. The TTS model is a core technology in the field of artificial intelligence for realizing text-to-speech, and its goal is to automatically convert the input text information into natural and fluent speech that conforms to the characteristics of human language.
[0077] Specifically, the speech generation model includes at least an encoder, an acoustic module, and a decoder. Regarding step S103 above, inputting the target timbre vector and the text to be converted into the speech generation model includes:
[0078] The text to be converted is input into the encoder to obtain a text encoding vector. The text encoding vector and the target timbre vector are input into the acoustic module and then into the decoder to obtain the target speech that matches the described text.
[0079] In the specific implementation of the above steps, the text to be converted is first input into the encoder of the speech generation model to obtain a text encoding vector. Then, the text encoding vector and the target timbre vector are input into the acoustic module of the speech generation model to generate acoustic features (such as Mel-spectrogram). Finally, the acoustic features are input into the decoder to generate the target speech that matches the descriptive text.
[0080] The training process for the three models described above will be explained in detail below:
[0081] Specifically, according to the speech generation method provided in this application, the target parsing model is trained through the following steps:
[0082] I: Obtain sample speech, input the timbre description sample corresponding to the sample speech into the initial parsing model to obtain the predicted sound feature vector and the predicted emotion space vector.
[0083] Regarding step I above, in specific implementation, sample speech is acquired, and the corresponding timbre description sample is input into the initial analytical model to obtain the predicted voice feature vector and the predicted emotion space vector. Here, the predicted voice feature vector can include acoustic feature labels and identity labels. Acoustic feature labels indicate the fundamental frequency, formants, jitter (small fluctuations in fundamental frequency and amplitude), harmonics, speech rate, language, energy variations, and spectral features (such as Mel-Frequency Cepstral Coefficients / MFCCs) of the sample speech. Identity labels indicate the gender and age of the speaker corresponding to the sample speech. It should be noted that, to ensure the decoupling of voice features and the emotion space vector, the voice features do not include emotional factors; they are parameters directly extracted from each sample speech. These voice feature parameters are aggregated into a single voice feature vector.
[0084] During the training phase, if the dimensionality of timbre information is not limited, it may reach or even exceed 4096 dimensions. The processing complexity of high-dimensional data far exceeds the capacity of existing speech generation models. Therefore, due to performance and resource constraints, it is necessary to limit the dimension of the timbre vector. To minimize information loss during dimensionality reduction, this application limits the data dimension in each training step of the model, enabling the model to directly acquire or generate low-dimensional sound features during encoding and decoding. That is, the output dimension of the model is directly limited to a lower dimension, allowing the model to compress and map complete acoustic features into a low-dimensional vector during data inference. In contrast, existing technologies typically generate complete high-dimensional sound features first and then perform dimensionality reduction extraction. Since dimensionality reduction inevitably leads to acoustic feature loss, the solution in this application can improve the situation of acoustic information loss. The predicted sound feature vector is a low-dimensional vector with a fixed dimension, which can be 128 or 256 dimensions, etc., and this application does not specify a particular dimension.
[0085] II: Based on the real sound feature vector, real emotion space vector, predicted sound feature vector, and predicted emotion space vector corresponding to the sample speech, a first loss function is established, and the initial parsing model is iteratively trained based on the first loss function until the preset training completion conditions are met, thus obtaining the trained target parsing model.
[0086] Here, the real voice feature vector can be extracted directly from the sample speech using an acoustic model, such as based on Hubert, Wav2Vec 2.0, or other self-supervised acoustic models. For obtaining the real emotion space vector, it can be achieved by manually annotating the PAD target emotion space vector data of the sample speech, or by fine-tuning the model structure and parameters based on the acoustic model, and then using the manually annotated PAD target emotion space vector and audio to automatically train the PAD target emotion space vector recognition of the sample speech.
[0087] Regarding step II above, in specific implementation, firstly, the real sound feature vector and the real emotion space vector corresponding to the sample speech are obtained. Then, the differences between the real sound feature vector and the predicted sound feature vector, and the differences between the real emotion space vector and the predicted emotion space vector are compared. The two differences are then fused to establish a first loss function. Based on the result of the first loss function, at least one parameter of the initial analytical model is iteratively adjusted through methods such as backpropagation until the model performance meets the training stopping condition. The training stopping condition may include: the number of training rounds reaches a preset number, or the result of the first loss function converges or is minimized. At this point, training is stopped to obtain the target analytical model.
[0088] Here, when training the target parsing model, the target parsing model and the speech generation model are kept decoupled. Specifically, when training the target parsing model, the parameters of the speech generation model are fixed, reducing the number of training parameters while ensuring that the target parsing model and the speech generation model are decoupled.
[0089] Specifically, according to the speech generation method provided in this application, the target fusion model is trained through the following steps:
[0090] i: Input the target voice feature vector and the target emotion space vector into the initial fusion model to obtain the predicted timbre vector.
[0091] Here, the predicted timbre vector is a low-dimensional vector with a fixed dimension. This is because both the sound feature vector and the fused timbre vector need to be low-dimensional in order to preserve the complete timbre information as much as possible. The fixed dimension can be 128-dimensional or 256-dimensional, etc., and this application does not make a specific limitation on it.
[0092] The target fusion model is trained based on the sound feature vector and emotion space vector output by the pre-trained target parsing model. Specifically, in step i above, the target sound feature vector and target emotion space vector output by the target parsing model are input into the initial fusion model to obtain the predicted timbre vector.
[0093] ii: Establish a second loss function based on the real timbre vector corresponding to the sample speech and the predicted timbre vector, and iteratively train the initial fusion model based on the second loss function until the preset training completion condition is met, and obtain the trained target fusion model.
[0094] Regarding step ii above, in specific implementation, the real timbre vector corresponding to the sample speech is obtained, the difference between the real timbre vector and the predicted timbre vector is compared, and the difference is used to establish a second loss function. Based on the result of the second loss function, the parameters of the initial fusion model are adjusted until the training conditions are met. Here, the training conditions can be that the training reaches a preset number of times, or the result of the first loss function converges or is minimized. At this time, the training is stopped to obtain the target fusion model.
[0095] Here, when training the target fusion model, the target fusion model is kept decoupled from the target parsing model and the speech generation model.
[0096] Specifically, the speech generation model is trained based on the target timbre vector output by the pre-trained target fusion model. During training, the target timbre vector output by the target fusion model and the corresponding converted text sample of the sample speech are input into the initial speech generation model to obtain the predicted speech. Then, a third loss function is established based on the sample speech and the predicted speech. At least one parameter of the initial speech generation model is iteratively trained using backpropagation based on the third loss function until a preset training completion condition is met, resulting in a well-trained speech generation model. The preset training completion condition may include: reaching a preset number of training rounds, or the third loss function converging or reaching its minimum, etc., and this application does not impose any special limitations on this.
[0097] The speech generation method provided in this application has data dependencies between the various models. Therefore, in the speech generation method provided in this application, it is preferable to first train a target parsing model, then train a target fusion model based on the sound feature vector and emotion space vector output by the trained target parsing model, and finally train a speech generation module based on the timbre vector output by the trained target fusion model. The desired target timbre text description is input into the target parsing model, the target fusion model fuses the output of the target parsing model to obtain the target timbre vector, and the target timbre vector and the text to be converted are simultaneously input into the speech generation model to generate target speech that conforms to the target timbre features.
[0098] The speech generation method based on timbre features provided in this application firstly obtains a target parsing model and inputs descriptive text into the target parsing model to obtain a target sound feature vector corresponding to the descriptive text; wherein, the descriptive text includes a target timbre text description; then, the target sound feature vector is input into a target fusion model to generate a target timbre vector; finally, the target timbre vector and the text to be converted are input into a speech generation model to obtain target speech that conforms to the descriptive text.
[0099] This application introduces sound feature vectors, generates sound feature vectors corresponding to the timbre text description through a target parsing model, generates target timbre vectors based on the sound feature vectors, and then generates target speech that meets the timbre features required by the user based on the timbre vectors. This improves the accuracy of speech generation under a specified timbre and meets the user's needs for speech generation with diverse timbres and high-precision emotional expression.
[0100] Please see Figure 2 , Figure 3 , Figure 2 This is one of the structural schematic diagrams of a speech generation device based on timbre features provided in an embodiment of this application. Figure 3 This is a second schematic diagram of a speech generation device based on timbre features provided in an embodiment of this application. Figure 2 As shown, the speech generation device 200 includes:
[0101] The sound feature vector determination module 201 is used to obtain a target parsing model, input the descriptive text into the target parsing model, and obtain the target sound feature vector corresponding to the descriptive text; wherein, the descriptive text includes a target timbre text description;
[0102] The first timbre vector determination module 202 is used to input the target sound feature vector into the target fusion model to generate the target timbre vector;
[0103] The target speech generation module 203 is used to input the target timbre vector and the text to be converted into the speech generation model to obtain the target speech that matches the description text.
[0104] Please see Figure 3 The speech generation device 200 further includes a second timbre vector determination module 204, which is used for:
[0105] The descriptive text is input into the target parsing model to obtain the target sound feature vector and the target emotion space vector corresponding to the descriptive text;
[0106] The target voice feature vector and the target emotion space vector are input into the target fusion model to generate the target timbre vector; wherein the target voice feature vector and the target emotion space vector are independent of each other.
[0107] Furthermore, when the second timbre vector determination module 204 inputs the target voice feature vector and the target emotion space vector into the target fusion model to generate the target timbre vector, the second timbre vector determination module 204 is also used for:
[0108] The target sound feature vector is mapped to at least one vector subspace to obtain at least one subvector; wherein each vector subspace represents an acoustic feature in an independent dimension;
[0109] The target fusion model adjusts the weights and / or offsets of each sub-vector according to the target emotion space vector;
[0110] The adjustment results of each sub-vector are combined to obtain the target timbre vector.
[0111] Furthermore, when the second timbre vector determination module 204 is used to adjust the weights and / or offsets of each sub-vector according to the target emotion space vector, the second timbre vector determination module 204 is also used to:
[0112] A gating unit is configured for each vector subspace, and each vector subspace is connected to an independent gating channel in the gating unit;
[0113] In response to receiving the target emotion space vector, the independent gating channel determines the scaling factor and offset factor of the corresponding vector subspace based on the target emotion space vector;
[0114] The weights and / or offsets of the subvectors in the corresponding vector subspace are adjusted according to the scaling factor and offset factor.
[0115] Furthermore, the target emotion space vector is a three-dimensional emotion space vector, including a first emotion dimension space, a second emotion dimension space, and a third emotion dimension space. The first emotion dimension space, the second emotion dimension space, and the third emotion dimension space are all continuous spaces and independent of each other.
[0116] Furthermore, the speech generation model includes an encoder, an acoustic module, and a decoder. When the target speech generation module 203 inputs the target timbre vector and the text to be converted into the speech generation model, the target speech generation module 203 is also used for:
[0117] The text to be converted is input into the encoder to obtain a text encoding vector. The text encoding vector and the target timbre vector are input into the acoustic module and then into the decoder to obtain the target speech that matches the described text.
[0118] Please see Figure 3 The speech generation device 200 further includes a first training module 205, which is used to train the target parsing model through the following steps:
[0119] Acquire sample speech, input the timbre description sample corresponding to the sample speech into the initial parsing model to obtain the predicted sound feature vector and the predicted emotion space vector; wherein, the predicted sound feature vector is a low-dimensional vector with fixed dimensions;
[0120] A first loss function is established based on the real sound feature vector, the real emotion space vector, the predicted sound feature vector, and the predicted emotion space vector corresponding to the sample speech. The initial parsing model is then iteratively trained based on the first loss function until the preset training completion conditions are met, thus obtaining the trained target parsing model.
[0121] Please see Figure 3 The speech generation device 200 further includes a second training module 206, which is used to train the target fusion model through the following steps:
[0122] The target voice feature vector and the target emotion space vector are input into the initial fusion model to obtain the predicted timbre vector; wherein, the predicted timbre vector is a low-dimensional vector with fixed dimensions;
[0123] A second loss function is established based on the real timbre vector corresponding to the sample speech and the predicted timbre vector. The initial fusion model is then iteratively trained based on the second loss function until the preset training completion conditions are met, thus obtaining the trained target fusion model.
[0124] Furthermore, when training the target parsing model, the target parsing model is kept decoupled from the speech generation model; when training the target fusion model, the target fusion model is kept decoupled from the target parsing model and the speech generation model.
[0125] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 400 includes a processor 410, a memory 420, and a bus 430.
[0126] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, they can perform the operations described above. Figure 1 The steps of the speech generation method based on timbre features in the method embodiment shown are specifically implemented in the method embodiment and will not be repeated here.
[0127] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The steps of the speech generation method based on timbre features in the method embodiment shown are specifically implemented in the method embodiment and will not be repeated here.
[0128] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0129] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.
[0130] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0131] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0132] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0133] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A voice generation method based on timbre features, characterized by, The speech generation method includes: A target parsing model is obtained by inputting the description text into the target parsing model to obtain the target sound feature vector corresponding to the description text; wherein, the description text includes a target timbre text description; The target sound feature vector is input into the target fusion model to generate the target timbre vector; The target timbre vector and the text to be converted are input into the speech generation model to obtain the target speech that matches the described text. The speech generation method further includes: The descriptive text is input into the target parsing model to obtain the target sound feature vector and the target emotion space vector corresponding to the descriptive text; The target voice feature vector and the target emotion space vector are input into the target fusion model to generate the target timbre vector; wherein the target voice feature vector and the target emotion space vector are independent of each other; The step of inputting the target voice feature vector and the target emotion space vector into the target fusion model to generate the target timbre vector includes: The target sound feature vector is mapped to at least one vector subspace to obtain at least one subvector; wherein each vector subspace represents an acoustic feature in an independent dimension; The target fusion model adjusts the weights and / or offsets of each sub-vector according to the target emotion space vector; The adjustment results of each sub-vector are combined to obtain the target timbre vector.
2. The voice generation method of claim 1, wherein, The target fusion model adjusts the weights and / or offsets of each sub-vector based on the target emotion space vector, including: A gating unit is configured for each vector subspace, and each vector subspace is connected to an independent gating channel in the gating unit; In response to receiving the target emotion space vector, the independent gating channel determines the scaling factor and offset factor of the corresponding vector subspace based on the target emotion space vector; The weights and / or offsets of the subvectors in the corresponding vector subspace are adjusted according to the scaling factor and offset factor.
3. The voice generation method of claim 2, wherein, The target emotion space vector is a three-dimensional emotion space vector, including a first emotion dimension space, a second emotion dimension space, and a third emotion dimension space. The first emotion dimension space, the second emotion dimension space, and the third emotion dimension space are all continuous spaces and independent of each other.
4. The speech generation method according to any one of claims 1-3, characterized in that, The speech generation model includes an encoder, an acoustic module, and a decoder. Inputting the target timbre vector and the text to be converted into the speech generation model includes: The text to be converted is input into the encoder to obtain a text encoding vector. The text encoding vector and the target timbre vector are input into the acoustic module and then into the decoder to obtain the target speech that matches the described text.
5. The speech generation method according to claim 1, characterized in that, The target analytical model is obtained by training through the following steps: Acquire sample speech, input the timbre description sample corresponding to the sample speech into the initial parsing model to obtain the predicted sound feature vector and the predicted emotion space vector; wherein, the predicted sound feature vector is a low-dimensional vector with fixed dimensions; A first loss function is established based on the real sound feature vector, the real emotion space vector, the predicted sound feature vector, and the predicted emotion space vector corresponding to the sample speech. The initial parsing model is then iteratively trained based on the first loss function until the preset training completion conditions are met, thus obtaining the trained target parsing model.
6. The speech generation method according to claim 5, characterized in that, The target fusion model is trained using the following steps: The target voice feature vector and the target emotion space vector are input into the initial fusion model to obtain the predicted timbre vector; wherein, the predicted timbre vector is a low-dimensional vector with fixed dimensions; A second loss function is established based on the real timbre vector corresponding to the sample speech and the predicted timbre vector. The initial fusion model is then iteratively trained based on the second loss function until the preset training completion conditions are met, thus obtaining the trained target fusion model.
7. The speech generation method according to claim 6, characterized in that, The speech generation method further includes: When training the target parsing model, the target parsing model is kept decoupled from the speech generation model; When training the target fusion model, the target fusion model is kept decoupled from the target parsing model and the speech generation model.
8. A speech generation device based on timbre features, characterized in that, The speech generation device includes: The sound feature vector determination module is used to obtain a target parsing model, input the descriptive text into the target parsing model, and obtain the target sound feature vector corresponding to the descriptive text; wherein, the descriptive text includes a target timbre text description; The timbre vector determination module is used to input the target sound feature vector into the target fusion model to generate the target timbre vector; The target speech generation module is used to input the target timbre vector and the text to be converted into the speech generation model to obtain the target speech that matches the description text. The speech generation device further includes a second timbre vector determination module, which is used for: The descriptive text is input into the target parsing model to obtain the target sound feature vector and the target emotion space vector corresponding to the descriptive text; The target voice feature vector and the target emotion space vector are input into the target fusion model to generate the target timbre vector; wherein the target voice feature vector and the target emotion space vector are independent of each other; When the second timbre vector determination module inputs the target voice feature vector and the target emotion space vector into the target fusion model to generate the target timbre vector, the second timbre vector determination module is further configured to: The target sound feature vector is mapped to at least one vector subspace to obtain at least one subvector; wherein each vector subspace represents an acoustic feature in an independent dimension; The target fusion model adjusts the weights and / or offsets of each sub-vector according to the target emotion space vector; The adjustment results of each sub-vector are combined to obtain the target timbre vector.