Voice generation method and voice generation device based on timbre characteristics

Abstract

The invention provides a tone feature-based voice generation method and device, and the method comprises the steps: obtaining a target analysis model, inputting a description text into the target analysis model, and obtaining a target voice feature vector corresponding to the description text; wherein the description text comprises a target timbre text description; inputting the target sound feature vector into a target fusion model to generate a target tone vector; and inputting the target timbre vector and the to-be-converted text into a voice generation model to obtain a target voice conforming to the description text. Through the method and the device, the target voice meeting the timbre characteristics required by the user is generated, the accuracy of voice generation under the specified timbre is improved, and the voice generation requirements of the user for diversified timbres and high-precision emotion expression are met.

Description

Technical Field

[0001] The present application relates to the field of audio technology, and in particular to a speech generation method and a speech generation device based on timbre features. Background Art

[0002] With the rapid development of speech synthesis technology, text-based speech synthesis has been widely used in scenarios such as intelligent assistants and voice interaction. Current speech synthesis technology not only focuses on speech clarity and naturalness, but also pursues fine-grained control of speech emotion and timbre.

[0003] Currently, timbre generation typically relies on pre-trained models, which can only achieve basic timbre modulation using a small number of emotion labels or reference speech. For example, this modulation relies on the timbre prior distribution learned by the pre-trained model. However, due to the high overlap in spectral features of similar emotions, existing models struggle to accurately capture subtle emotional expressions such as excitement and happiness. As a result, the generated speech lacks sufficient expressiveness and discriminability, making it difficult to meet user demands for diverse, high-precision speech generation. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a speech generation method and speech generation device based on timbre features, wherein a target analysis model is used to generate a sound feature vector corresponding to the timbre text description, and a target fusion model is used to generate a target timbre vector based on the sound feature vector. The speech generation model can generate a target speech that meets the timbre features required by the user based on the timbre vector, thereby improving the accuracy of speech generation under the specified timbre and meeting the user's demand for speech generation with diversified timbre and high-precision emotional expression.

[0005] In a first aspect, an embodiment of the present application provides a method for generating speech based on timbre features, the method comprising: Obtaining a target parsing model, inputting a description text into the target parsing model, and obtaining a target sound feature vector corresponding to the description text; wherein the description text includes a text description of the target timbre; Inputting the target sound feature vector into a target fusion model to generate a target timbre vector; The target timbre vector and the text to be converted are input into a speech generation model to obtain a target speech that conforms to the description text.

[0006] Furthermore, the speech generation method further includes: The target parsing model obtains the target sound feature vector and target emotion space vector corresponding to the description text based on the description text, and the target fusion model receives and fuses the target sound feature vector and the target emotion space vector to generate the target timbre vector; wherein, the target sound feature vector and the target emotion space vector are independent of each other.

[0007] Furthermore, the target fusion model receives and fuses the target sound feature vector and the target emotion space vector to generate the target timbre vector, including: Mapping the target sound feature vector to at least one vector subspace to obtain at least one subvector; wherein each vector subspace represents an acoustic feature in an independent dimension; adjusting the weight and / or offset of each sub-vector according to the target emotion space vector; The adjustment results of each sub-vector are fused to obtain the target timbre vector.

[0008] Furthermore, the target fusion model adjusts the weight and / or offset of each sub-vector according to the target emotion space vector, including: Configuring a gating unit for the vector subspace, wherein each vector subspace is connected to an independent gating channel in the gating unit; In response to receiving the target emotion space vector, the independent gating channel determines a scaling factor and an offset factor of a corresponding vector subspace based on the target emotion space vector; The weight and / or offset of the sub-vector of the corresponding vector subspace is adjusted according to the scaling factor and the offset factor.

[0009] Furthermore, the target emotion space vector is a three-dimensional emotion space vector, including a first emotion dimension space, a second emotion dimension space and a third emotion dimension space, and the first emotion dimension space, the second emotion dimension space and the third emotion dimension space are all continuous spaces and independent of each other.

[0010] Furthermore, the speech generation model includes an encoder, an acoustic module, and a decoder, and the step of inputting the target timbre vector and the text to be converted into the speech generation model includes: The text to be converted is input into the encoder to obtain a text encoding vector, the text encoding vector and the target timbre vector are input into the acoustic module, and the target speech that conforms to the description text is obtained through the decoder.

[0011] Furthermore, the target parsing model is trained by the following steps: Acquire a sample speech, input a timbre description sample corresponding to the sample speech into an initial parsing model, and obtain a predicted sound feature vector and a predicted emotion space vector; wherein the predicted sound feature vector is a low-dimensional vector of fixed dimension; A first loss function is established based on the real sound feature vector, the real emotion space vector, the predicted sound feature vector and the predicted emotion space vector corresponding to the sample speech, and the initial parsing model is iteratively trained based on the first loss function until the preset training completion conditions are met, thereby obtaining the trained target parsing model.

[0012] Furthermore, the target fusion model is obtained by training through the following steps: Inputting the target sound feature vector and the target emotion space vector into an initial fusion model to obtain a predicted timbre vector; wherein the predicted timbre vector is a low-dimensional vector of fixed dimension; A second loss function is established based on the real timbre vector corresponding to the sample speech and the predicted timbre vector, and the initial fusion model is iteratively trained based on the second loss function until the preset training completion condition is met, thereby obtaining the trained target fusion model.

[0013] Furthermore, the speech generation method further includes: When training the target parsing model, keeping the target parsing model decoupled from the speech generation model; When training the target fusion model, the target fusion model is kept decoupled from the target parsing model and the speech generation model.

[0014] In a second aspect, an embodiment of the present application further provides a speech generation device based on timbre features, the speech generation device comprising: A sound feature vector determination module is used to obtain a target parsing model, input a description text into the target parsing model, and obtain a target sound feature vector corresponding to the description text; wherein the description text includes a text description of the target timbre; a timbre vector determination module, configured to input the target sound feature vector into a target fusion model to generate a target timbre vector; The target speech generation module is used to input the target timbre vector and the text to be converted into a speech generation model to obtain a target speech that conforms to the description text.

[0015] The embodiments of the present application provide a speech generation method and a speech generation device based on timbre features. First, a target parsing model is obtained, and a description text is input into the target parsing model to obtain a target sound feature vector corresponding to the description text; wherein the description text includes a target timbre text description; then, the target sound feature vector is input into a target fusion model to generate a target timbre vector; finally, the target timbre vector and the text to be converted are input into a speech generation model to obtain a target speech that conforms to the description text.

[0016] This application introduces sound feature vectors, generates sound feature vectors corresponding to timbre text descriptions through a target parsing model, and generates target timbre vectors based on the sound feature vectors. The speech generation model can then generate target speech that meets the timbre characteristics required by the user based on the timbre vectors, thereby improving the accuracy of speech generation under specified timbre and meeting the user's needs for speech generation with diverse timbres and high-precision emotional expression.

[0017] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A flowchart of a method for generating speech based on timbre features provided in an embodiment of the present application; Figure 2 This is one of the structural diagrams of a speech generation device based on timbre features provided in an embodiment of the present application; Figure 3 This is a second structural diagram of a speech generation device based on timbre features provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for which protection is claimed, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work falls within the scope of protection of the present application.

[0021] First, the application scenarios to which this application is applicable are introduced. This application can be applied in the field of audio technology.

[0022] With the rapid development of speech synthesis technology, text-based speech synthesis has been widely used in scenarios such as intelligent assistants and voice interaction. Current speech synthesis technology not only focuses on speech clarity and naturalness, but also pursues fine-grained control of speech emotion and timbre.

[0023] Research has found that current timbre generation typically relies on pre-trained models, which are only able to achieve basic timbre modulation using a small number of emotion labels or reference speech. For example, distinguishing distinct emotional states like happiness and sadness relies heavily on the timbre prior distribution learned by the pre-trained model. However, due to the high overlap in spectral features of similar emotions, existing models struggle to accurately capture subtle emotional expressions like excitement and happiness. As a result, the generated speech lacks sufficient expressiveness and discriminability, making it difficult to meet user demands for diverse, high-precision speech generation.

[0024] Based on this, the embodiment of the present application provides a speech generation method based on timbre features to generate a target speech that meets the timbre features required by the user, improves the accuracy of speech generation under the specified timbre, and meets the user's demand for speech generation with diverse timbre and high-precision emotional expression.

[0025] See also Figure 1 , Figure 1 This is a flow chart of a method for generating speech based on timbre features provided in an embodiment of the present application. Figure 1 As shown in , the speech generation method provided by the embodiment of the present application includes: S101, obtaining a target parsing model, inputting a description text into the target parsing model, and obtaining a target sound feature vector corresponding to the description text.

[0026] Here, the description text includes a target timbre text description. The target timbre text description refers to text information used to describe the timbre characteristics of the desired speech. For example, the target timbre text description can use words such as young, old, female, male, sharp, bright, etc. to describe the timbre from various dimensions, and this application does not specifically limit this.

[0027] With respect to the above step S101 , a pre-trained target parsing model is obtained, and the description text is input into the target parsing model to obtain a target sound feature vector corresponding to the description text.

[0028] S102: Input the target sound feature vector into a target fusion model to generate a target timbre vector.

[0029] Regarding the above step S102, during specific implementation, the target sound feature vector output by the target parsing model is input into a pre-trained target fusion model to obtain a target timbre vector.

[0030] Specifically, according to the speech generation method provided in this application, the speech generation method further includes: A: Input the description text into the target parsing model to obtain the target sound feature vector and target emotion space vector corresponding to the description text.

[0031] Here, the target parsing model can not only parse the sound features from the description text, but also parse the emotion labels from the description text. Regarding step A above, in the specific implementation, the description text is input into the target parsing model. The target parsing model parses the sound features and emotion labels from the description text based on the content of the description text, thus obtaining the target sound feature vector and target emotion space vector.

[0032] Specifically, the target emotion space vector is a three-dimensional emotion space vector, including a first emotion dimension space, a second emotion dimension space and a third emotion dimension space. The first emotion dimension space, the second emotion dimension space and the third emotion dimension space are all continuous spaces and independent of each other.

[0033] Here, the continuous dimension PAD (Pleasure-Arousal-Dominance) framework is introduced into the target emotion space vector to represent emotions in detail. The PAD (Pleasure-Arousal-Dominance) framework is an emotional framework used to quantitatively describe human emotional states. It can continuously and smoothly express different categories of emotions, expressing multiple emotions based on a three-dimensional coordinate system composed of three core dimensions. Specifically, Pleasure (Valence) measures the positivity or negativity of an emotion, such as happiness (high Pleasure) and sadness (low Pleasure); Activation (Arousal) measures the physiological arousal of an emotion, such as calmness (low Activation) and mania (high Activation); and Dominance (Dominance) measures the sense of control or compliance of an emotion, such as anger (high Dominance) and fear (low Dominance). The target emotion space vector [P, A, D] composed of these three continuously changing parameters can reasonably represent the vast majority of emotions. The target parsing model generates the target emotion space vector, which enables the subsequent target fusion model to understand the changes in timbre of relatively continuously changing emotional details. For example, the acoustic spectrum features corresponding to the three emotions of choking, sobbing, and crying are almost the same, but based on the PAD three-dimensional emotion framework, different activation values ​​can be used to represent the subtle differences in the expression of the three emotions. At the same time, the spatial results of the mutual coupling of multiple emotions can also be understood based on the target emotion space vector. In a preferred embodiment, the value range of each element in the target emotion space vector [P, A, D] is the same. For example, P, A, D∈[-1,1]. Taking the pleasantness dimension as an example, the closer the value is to 1, the more pleasant it is, and the closer the value is to -1, the unpleasant it is. The same is true for the activation and dominance dimensions. It should be understood that the above is only an example. P, A, D can also be other values. The above examples should not be understood as limitations on the present application.

[0034] In this way, by introducing the PAD three-dimensional emotion framework, this application can not only enable the model to learn the different expressions of subtle emotional differences in timbre, but also enable the model to decouple emotions of different dimensions when multiple emotions are expressed in a mixed manner. Taking the two emotions of crying with joy and crying with sadness as examples, "crying with joy" and "crying with sadness" are two different vectors in the three-dimensional emotion space. Therefore, using the target emotion space vector in timbre expression can more clearly express the difference between the two, and also enable the model to quickly learn the difference between the two emotions. At the same time, the various dimensions of the target emotion space vector are decoupled from each other, which can ensure that when the model expresses emotions, the expression effect of the "crying" part will not be affected by the emotional tone of happiness or sadness.

[0035] B: Inputting the target sound feature vector and the target emotion space vector into the target fusion model to generate the target timbre vector.

[0036] Regarding step B above, during implementation, the target fusion model fuses the target sound feature vector and the target emotion space vector, and inputs the target sound feature vector and the target emotion space vector into the target fusion model to generate the target timbre vector. Here, the target sound feature vector and the target emotion space vector are independent of each other.

[0037] Furthermore, the target fusion model introduces a conditional fusion network (CFN). The conditional fusion network, based on an attention mechanism or a gated network mechanism, modulates and / or enhances one or more acoustic feature values ​​according to the values ​​of each dimension in different target emotion space vectors. Specifically, with respect to the above step B, the target sound feature vector and the target emotion space vector are input into the target fusion model to generate the target timbre vector, including: a: Mapping the target sound feature vector to at least one vector subspace to obtain at least one subvector.

[0038] Regarding step a above, in a specific implementation, the target sound feature vector is first mapped to at least one vector subspace to obtain at least one subvector. Here, each vector subspace represents an acoustic feature in an independent dimension.

[0039] Specifically, the vector subspace may include a pitch (F0) subspace, which controls the pitch of the voice, such as performing a rising tone at the end of a question; an energy subspace, which controls the intensity of pronunciation, such as enhancing stressed syllables to strengthen emotional expression; a duration subspace, which controls speech rate and rhythm, such as slow speech when expressing sadness and fast speech when expressing excitement; a timbre subspace, which represents the speaker's identity, such as timbre characteristics and specific timbre textures based on factors such as gender and age, such as hoarseness or mellowness; and a style subspace, which captures personalized pronunciation characteristics, such as dialect, accent, and rhythm preferences. The above vector subspaces are merely examples and should not be construed as limiting the present application.

[0040] b: The target fusion model adjusts the weight and / or offset of each sub-vector according to the target emotion space vector.

[0041] Regarding step b above, during specific implementation, the target fusion model adjusts the weight and / or offset of each subvector based on the target emotion space vector. Specifically, the target fusion model uses the target emotion space vector as a condition and dynamically adjusts the weights and / or offsets of the subvectors in these vector subspaces through an attention mechanism or a gating mechanism. For example, when the target emotion space vector indicates "high activation," corresponding to an emotional arousal state (emotions such as anger), resulting in an amplified fundamental frequency fluctuation and a surge in pronunciation energy, the target fusion model tends to enhance fundamental frequency F0 modulation and a surge in pronunciation energy; when indicating "low pleasure," corresponding to negative emotions (emotions such as sadness), the target fusion model tends to reduce the average level of fundamental frequency F0 and increase the smoothness of the speech rate.

[0042] Specifically, with respect to the above step b, the target fusion model adjusts the weight and / or offset of each sub-vector according to the target emotion space vector, including: (1) A gate control unit is configured for the vector subspace, and each vector subspace is connected to an independent gate control channel in the gate control unit.

[0043] (2) In response to receiving the target emotion space vector, the independent gating channel determines a scaling factor and an offset factor of a corresponding vector subspace based on the target emotion space vector.

[0044] (3) Adjusting the weight and / or offset of the sub-vector of the corresponding vector subspace according to the scaling factor and the offset factor.

[0045] With respect to the above steps (1) to (3), in a specific implementation, the target fusion model configures a gating unit for the vector subspace, and each vector subspace is connected to an independent gating channel in the gating unit. Specifically, the gating unit can be implemented based on a multi-layer perceptron (MLP) and jointly performs end-to-end training with the target fusion model, which can better enable the target fusion model to perform the fusion of the target sound feature vector and the target emotion space vector. When the target fusion model receives the target emotion space vector, the independent gating channel determines the scaling factor and / or offset factor of the corresponding vector subspace based on the target emotion space vector, and then adjusts the weight and / or offset of the subvector of the corresponding vector subspace according to the scaling factor and the offset factor.

[0046] c: Fusing the adjustment results of each sub-vector to obtain the target timbre vector.

[0047] Regarding the above step c, during specific implementation, after the target fusion model adjusts the weight and / or offset of each sub-vector, the adjustment results of each sub-vector are integrated to obtain the target timbre vector.

[0048] S103: Input the target timbre vector and the text to be converted into a speech generation model to obtain a target speech that conforms to the description text.

[0049] Regarding step S103, during implementation, the target timbre vector generated by the target fusion model and the text to be converted are input into a speech generation model to generate a target speech that conforms to the described text. Preferably, the speech generation model uses a TTS (Text-to-Speech) model. The TTS model is a core technology for converting text to speech in the field of artificial intelligence. Its goal is to automatically convert input text information into natural and fluent speech that conforms to human language characteristics.

[0050] Specifically, the speech generation model includes at least an encoder, an acoustic module, and a decoder. With respect to the above step S103, the step of inputting the target timbre vector and the text to be converted into the speech generation model includes: The text to be converted is input into the encoder to obtain a text encoding vector, the text encoding vector and the target timbre vector are input into the acoustic module, and the target speech that conforms to the description text is obtained through the decoder.

[0051] In the specific implementation of the above steps, the text to be converted is first input into the speech generation model's encoder to generate a text encoding vector. This text encoding vector and the target timbre vector are then input into the speech generation model's acoustic module to generate acoustic features (such as a Mel-spectrogram). Finally, these acoustic features are input into the decoder to generate the target speech that matches the described text.

[0052] The following is a detailed description of the training process of the above three models: Specifically, according to the speech generation method provided in this application, the target parsing model is trained by the following steps: I: Obtain sample speech, input the timbre description sample corresponding to the sample speech into the initial parsing model, and obtain a predicted sound feature vector and a predicted emotion space vector.

[0053] Regarding step I above, during implementation, a sample speech is obtained, and the timbre description sample corresponding to the sample speech is input into the initial parsing model to obtain a predicted sound feature vector and a predicted emotion space vector. The predicted sound feature vector may include acoustic feature labels and identity labels. The acoustic feature labels indicate features such as the fundamental frequency, formants, jitter (small fluctuations in fundamental frequency and amplitude), harmonics, speaking rate, language, energy variation, and spectral characteristics (such as Mel-Frequency Cepstral Coefficients (MFCCs)) of the sample speech. The identity label indicates the gender and age of the speaker corresponding to the sample speech. It should be noted that to ensure decoupling of the sound features and emotion space vector, the sound features do not include emotional factors. Instead, they are parameters directly extracted from each sample speech. The sound feature parameters are aggregated into a sound feature vector.

[0054] During the training phase, if the dimension of the timbre information is not restricted, its dimension may reach or even exceed 4096 dimensions. The processing complexity of high-dimensional data far exceeds the tolerance of existing speech generation models. Therefore, due to performance limitations and resource limitations, it is necessary to limit the dimension of the timbre vector. In order to minimize the information loss in the dimensionality reduction process, this application limits the dimension of the data in the training steps of each model, so that the model directly obtains or generates low-dimensional sound features during the encoding and decoding processes, that is, directly limits the output dimension of the model to a lower dimensional level, so that during the data inference process, the model can compress and map the complete acoustic features into low-dimensional vectors at the output. In contrast, the existing technology usually adopts a method of generating complete high-dimensional sound features first and then performing dimensionality reduction extraction processing on them. Since dimensionality reduction processing inevitably leads to acoustic feature loss, the solution of this application can improve the situation of acoustic information loss. Among them, the predicted sound feature vector is a low-dimensional vector of fixed dimension, and the fixed dimension can be 128 dimensions or 256 dimensions, etc., which is not specifically limited in this application.

[0055] II: Establish a first loss function based on the real sound feature vector, the real emotion space vector, the predicted sound feature vector and the predicted emotion space vector corresponding to the sample speech, and iteratively train the initial parsing model based on the first loss function until the preset training completion conditions are met, thereby obtaining the trained target parsing model.

[0056] Here, the true sound feature vector can be extracted directly from the sample speech using an acoustic model, such as one based on HuBERT, Wav2Vec 2.0, or other acoustic self-supervised models. To obtain the true emotion space vector, the PAD target emotion space vector data of the sample speech can be manually annotated. Alternatively, the model structure and parameters can be fine-tuned based on the acoustic model, and the PAD target emotion space vector of the sample speech can be automatically identified using the manually annotated PAD target emotion space vector and audio.

[0057] For the above step II, in the specific implementation, first obtain the real sound feature vector and the real emotion space vector corresponding to the sample speech, then compare the difference between the real sound feature vector and the predicted sound feature vector, and the difference between the real emotion space vector and the predicted emotion space vector, and integrate the above two differences to establish a first loss function. Based on the result of the first loss function, iteratively adjust at least one parameter of the initial parsing model through back propagation and other methods until the model performance meets the training stop condition. The training stop condition here may include: the number of training rounds reaches a preset number, or the result of the first loss function converges or is minimized, then stop training to obtain the target parsing model.

[0058] Here, when training the target parsing model, the target parsing model is kept decoupled from the speech generation model. Specifically, when training the target parsing model, the parameters of the speech generation model are fixed, which reduces the number of training parameters while ensuring that the target parsing model and the speech generation model are decoupled from each other.

[0059] Specifically, according to the speech generation method provided in this application, the target fusion model is obtained by training through the following steps: i: Input the target sound feature vector and the target emotion space vector into the initial fusion model to obtain a predicted timbre vector.

[0060] Here, the predicted timbre vector is a low-dimensional vector of fixed dimension. This is because both the sound feature vector and the fused timbre vector must be low-dimensional to ensure that the low-dimensional timbre vector retains complete timbre information. The fixed dimension can be 128 or 256, etc., and this application does not impose specific restrictions on this.

[0061] The target fusion model is trained based on the sound feature vectors and emotion space vectors output by the trained target parsing model. Regarding step i above, during implementation, the target sound feature vectors and target emotion space vectors output by the target parsing model are input into the initial fusion model to obtain a predicted timbre vector.

[0062] ii: Establish a second loss function based on the real timbre vector corresponding to the sample speech and the predicted timbre vector, and iteratively train the initial fusion model based on the second loss function until the preset training completion condition is met, thereby obtaining the trained target fusion model.

[0063] For the above step ii, during the specific implementation, the real timbre vector corresponding to the sample speech is obtained, the difference between the real timbre vector and the predicted timbre vector is compared, and the difference is used to establish a second loss function. The parameters of the initial fusion model are adjusted based on the result of the second loss function until the training conditions are met. The training conditions here can make the training reach a preset number of times, or the result of the first loss function converges or is minimized. At this time, the training is stopped to obtain the target fusion model.

[0064] Here, when training the target fusion model, the target fusion model is kept decoupled from the target parsing model and speech generation model.

[0065] Specifically, the speech generation model is trained based on the target timbre vector output by the trained target fusion model. When training the speech generation model, the target timbre vector output by the target fusion model and the converted text sample corresponding to the sample speech are input into the initial speech generation model to obtain the predicted speech. Then, a third loss function is established based on the sample speech and the predicted speech, and at least one parameter of the initial speech generation model is iteratively trained by the back propagation method based on the third loss function until the preset training completion condition is met, thereby obtaining a trained speech generation model; wherein, the preset training completion condition may include: the number of training rounds reaches a preset number, or the result of the third loss function converges or is minimized, etc., and this application does not impose any special restrictions on this.

[0066] In the speech generation method provided in the embodiment of the present application, there is a data dependency relationship between the various models. Therefore, in the speech generation method provided in the present application, it is preferred to first train the target parsing model, then train the target fusion model based on the sound feature vector and emotion space vector output by the trained target parsing model, and then train the speech generation module based on the timbre vector output by the trained target fusion model. The required target timbre text description is input into the target parsing model, and the target fusion model fuses the output of the target parsing model to obtain a target timbre vector. The target timbre vector and the text to be converted are simultaneously input into the speech generation model to generate a target speech that meets the target timbre characteristics.

[0067] The embodiment of the present application provides a speech generation method based on timbre features. First, a target parsing model is obtained, and a description text is input into the target parsing model to obtain a target sound feature vector corresponding to the description text; wherein the description text includes a target timbre text description; then, the target sound feature vector is input into a target fusion model to generate a target timbre vector; finally, the target timbre vector and the text to be converted are input into a speech generation model to obtain a target speech that conforms to the description text.

[0068] This application introduces sound feature vectors, generates sound feature vectors corresponding to timbre text descriptions through a target parsing model, and generates target timbre vectors based on the sound feature vectors. The speech generation model can then generate target speech that meets the timbre characteristics required by the user based on the timbre vectors, thereby improving the accuracy of speech generation under specified timbre and meeting the user's needs for speech generation with diverse timbres and high-precision emotional expression.

[0069] See also Figure 2 、 Figure 3 , Figure 2 This is one of the structural diagrams of a speech generation device based on timbre features provided in an embodiment of the present application. Figure 3 This is a second structural diagram of a speech generation device based on timbre features provided in an embodiment of the present application. Figure 2 As shown in , the speech generating device 200 includes: The sound feature vector determination module 201 is used to obtain a target parsing model, input a description text into the target parsing model, and obtain a target sound feature vector corresponding to the description text; wherein the description text includes a text description of the target timbre; A first timbre vector determination module 202 is configured to input the target sound feature vector into a target fusion model to generate a target timbre vector; The target speech generation module 203 is configured to input the target timbre vector and the text to be converted into a speech generation model to obtain a target speech that conforms to the description text.

[0070] See also Figure 3 The speech generation device 200 further includes a second timbre vector determination module 204, and the second timbre vector determination module 204 is configured to: Inputting the description text into the target parsing model to obtain a target sound feature vector and a target emotion space vector corresponding to the description text; The target sound feature vector and the target emotion space vector are input into the target fusion model to generate the target timbre vector; wherein the target sound feature vector and the target emotion space vector are independent of each other.

[0071] Furthermore, when the second timbre vector determination module 204 is used to input the target sound feature vector and the target emotion space vector into the target fusion model to generate the target timbre vector, the second timbre vector determination module 204 is further used to: Mapping the target sound feature vector to at least one vector subspace to obtain at least one subvector; wherein each vector subspace represents an acoustic feature in an independent dimension; The target fusion model adjusts the weight and / or offset of each sub-vector according to the target emotion space vector; The adjustment results of each sub-vector are fused to obtain the target timbre vector.

[0072] Furthermore, when the second timbre vector determining module 204 is used to adjust the weight and / or offset of each sub-vector according to the target emotion space vector, the second timbre vector determining module 204 is further used to: Configuring a gating unit for the vector subspace, wherein each vector subspace is connected to an independent gating channel in the gating unit; In response to receiving the target emotion space vector, the independent gating channel determines a scaling factor and an offset factor of a corresponding vector subspace based on the target emotion space vector; The weight and / or offset of the sub-vector of the corresponding vector subspace is adjusted according to the scaling factor and the offset factor.

[0073] Furthermore, the target emotion space vector is a three-dimensional emotion space vector, including a first emotion dimension space, a second emotion dimension space and a third emotion dimension space, and the first emotion dimension space, the second emotion dimension space and the third emotion dimension space are all continuous spaces and independent of each other.

[0074] Furthermore, the speech generation model includes an encoder, an acoustic module, and a decoder. When the target speech generation module 203 is used to input the target timbre vector and the text to be converted into the speech generation model, the target speech generation module 203 is further used to: The text to be converted is input into the encoder to obtain a text encoding vector, the text encoding vector and the target timbre vector are input into the acoustic module, and the target speech that conforms to the description text is obtained through the decoder.

[0075] See also Figure 3 The speech generation device 200 further includes a first training module 205, which is configured to train the target parsing model through the following steps: Acquire a sample speech, input a timbre description sample corresponding to the sample speech into an initial parsing model, and obtain a predicted sound feature vector and a predicted emotion space vector; wherein the predicted sound feature vector is a low-dimensional vector of fixed dimension; A first loss function is established based on the real sound feature vector, the real emotion space vector, the predicted sound feature vector and the predicted emotion space vector corresponding to the sample speech, and the initial parsing model is iteratively trained based on the first loss function until the preset training completion conditions are met, thereby obtaining the trained target parsing model.

[0076] See also Figure 3 The speech generation device 200 further includes a second training module 206, which is configured to obtain the target fusion model through training by the following steps: Inputting the target sound feature vector and the target emotion space vector into an initial fusion model to obtain a predicted timbre vector; wherein the predicted timbre vector is a low-dimensional vector of fixed dimension; A second loss function is established based on the real timbre vector corresponding to the sample speech and the predicted timbre vector, and the initial fusion model is iteratively trained based on the second loss function until the preset training completion condition is met, thereby obtaining the trained target fusion model.

[0077] Furthermore, when training the target parsing model, the target parsing model is kept decoupled from the speech generation model; when training the target fusion model, the target fusion model is kept decoupled from the target parsing model and the speech generation model.

[0078] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown in FIG, the electronic device 400 includes a processor 410, a memory 420 and a bus 430.

[0079] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, the above-mentioned Figure 1 The specific implementation of the steps of the voice generation method based on timbre features in the method embodiment shown can be found in the method embodiment, and will not be repeated here.

[0080] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The specific implementation of the steps of the voice generation method based on timbre features in the method embodiment shown can be found in the method embodiment, and will not be repeated here.

[0081] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0082] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. There may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interface, indirect coupling or communication connection of devices or units, which may be electrical, mechanical or other forms.

[0083] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0084] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0085] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0086] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-mentioned embodiments within the technical scope disclosed in the present application, or perform equivalent replacements for some of the technical features thereof. These modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for generating speech based on timbre features, characterized in that: The speech generation method comprises: Obtaining a target parsing model, inputting a description text into the target parsing model, and obtaining a target sound feature vector corresponding to the description text; wherein the description text includes a text description of the target timbre; Inputting the target sound feature vector into a target fusion model to generate a target timbre vector; The target timbre vector and the text to be converted are input into a speech generation model to obtain a target speech that conforms to the description text.

2. The speech generation method according to claim 1, wherein: The speech generation method further comprises: Inputting the description text into the target parsing model to obtain a target sound feature vector and a target emotion space vector corresponding to the description text; The target sound feature vector and the target emotion space vector are input into the target fusion model to generate the target timbre vector; wherein the target sound feature vector and the target emotion space vector are independent of each other.

3. The speech generation method according to claim 2, wherein: The step of inputting the target sound feature vector and the target emotion space vector into the target fusion model to generate the target timbre vector includes: Mapping the target sound feature vector to at least one vector subspace to obtain at least one subvector; wherein each vector subspace represents an acoustic feature in an independent dimension; The target fusion model adjusts the weight and / or offset of each sub-vector according to the target emotion space vector; The adjustment results of each sub-vector are fused to obtain the target timbre vector.

4. The speech generation method according to claim 3, wherein: The target fusion model adjusts the weight and / or offset of each sub-vector according to the target emotion space vector, including: Configuring a gating unit for the vector subspace, wherein each vector subspace is connected to an independent gating channel in the gating unit; In response to receiving the target emotion space vector, the independent gating channel determines a scaling factor and an offset factor of a corresponding vector subspace based on the target emotion space vector; The weight and / or offset of the sub-vector of the corresponding vector subspace is adjusted according to the scaling factor and the offset factor.

5. The speech generation method according to claim 4, characterized in that The target emotion space vector is a three-dimensional emotion space vector, including a first emotion dimension space, a second emotion dimension space and a third emotion dimension space. The first emotion dimension space, the second emotion dimension space and the third emotion dimension space are all continuous spaces and independent of each other.

6. The speech generation method according to any one of claims 1 to 5, characterized in that: The speech generation model includes an encoder, an acoustic module, and a decoder. Inputting the target timbre vector and the text to be converted into the speech generation model includes: The text to be converted is input into the encoder to obtain a text encoding vector, the text encoding vector and the target timbre vector are input into the acoustic module, and the target speech that conforms to the description text is obtained through the decoder.

7. The speech generation method according to claim 2, wherein: The target parsing model is trained by the following steps: Acquire a sample speech, input a timbre description sample corresponding to the sample speech into an initial parsing model, and obtain a predicted sound feature vector and a predicted emotion space vector; wherein the predicted sound feature vector is a low-dimensional vector of fixed dimension; A first loss function is established based on the real sound feature vector, the real emotion space vector, the predicted sound feature vector and the predicted emotion space vector corresponding to the sample speech, and the initial parsing model is iteratively trained based on the first loss function until the preset training completion conditions are met, thereby obtaining the trained target parsing model.

8. The speech generation method according to claim 7, characterized in that The target fusion model is obtained by training through the following steps: Inputting the target sound feature vector and the target emotion space vector into an initial fusion model to obtain a predicted timbre vector; wherein the predicted timbre vector is a low-dimensional vector of fixed dimension; A second loss function is established based on the real timbre vector corresponding to the sample speech and the predicted timbre vector, and the initial fusion model is iteratively trained based on the second loss function until the preset training completion condition is met, thereby obtaining the trained target fusion model.

9. The speech generation method according to claim 8, characterized in that The speech generation method further comprises: When training the target parsing model, keeping the target parsing model decoupled from the speech generation model; When training the target fusion model, the target fusion model is kept decoupled from the target parsing model and the speech generation model.

10. A speech generation device based on timbre features, characterized in that: The speech generating device comprises: A sound feature vector determination module is used to obtain a target parsing model, input a description text into the target parsing model, and obtain a target sound feature vector corresponding to the description text; wherein the description text includes a text description of the target timbre; a timbre vector determination module, configured to input the target sound feature vector into a target fusion model to generate a target timbre vector; The target speech generation module is used to input the target timbre vector and the text to be converted into a speech generation model to obtain a target speech that conforms to the description text.

Images