A speech generation method and device based on timbre parameterization control, equipment and storage medium

Abstract

The application discloses a voice generation method and device based on timbre parameterization control, equipment and storage medium, the method comprises the following steps: a timbre extraction model extracts a first timbre vector of reference voice, inputs the first timbre vector into a principal component analysis model, outputs principal component weight parameters, and parameterizes and controls the first timbre vector to obtain a second timbre vector; natural speech content is input into a timbre principal component parameter generation model, and timbre principal component weight parameters are output to parameterize and control the second timbre vector, so that a third timbre vector is obtained; the semantic features of text content and the third timbre vector are input into a voice conversion model, and voice results are output. It can be seen that the model is used to separate the semantic features of text, avoid semantic interference timbre, improve the stability of generation, use the model to output weight parameters, and perform twice parameterization adjustment, improve the controllability of timbre control, the natural voice description corresponding timbre principal component parameter can be parameterized and adjusted, and the explainability of timbre control is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech, more particularly, to a speech generation method and device based on timbre parameterization control, equipment and storage medium. BACKGROUND

[0002] With the rapid development of deep learning, speech synthesis and speech conversion technologies based on neural networks have made significant progress, and can achieve high naturalness and high similarity of speech output. Related technologies are widely used in intelligent voice assistants, virtual digital voice dubbing and other scenarios, providing support for personalized voice needs in various scenarios.

[0003] However, most methods in the prior art do not achieve decoupling of timbre and semantics, resulting in unstable timbre generation. Traditional global speaker embedding only supports sentence-level timbre switching and has no parameterization adjustment capability. The explainability is weak, and in addition to natural language description, reference speech is required, and the generated timbre cannot be parameterized adjusted.

[0004] How to improve the stability, controllability and explainability of timbre control in the process of generating speech is a problem that needs attention. SUMMARY

[0005] In view of the above problems, the present application provides a speech generation method and device based on timbre parameterization control, equipment and storage medium to improve the stability, controllability and explainability of timbre control in the process of generating speech.

[0006] In order to achieve the above purpose, the specific scheme is as follows:

[0007] A speech generation method based on timbre parameterization control, comprising:

[0008] Obtaining a reference speech, based on the timbre description of the reference speech, obtaining a natural speech content and generating a text content;

[0009] Extracting a first timbre vector of the reference speech through a pre-trained timbre extraction model;

[0010] Inputting the first timbre vector into a pre-trained principal component analysis model to output principal component weight parameters, and performing parameterization control on the first timbre vector based on the principal component weight parameters to obtain a second timbre vector;

[0011] Inputting the natural speech content into a pre-trained timbre principal component parameter generation model to output timbre principal component weight parameters, and performing parameterization control on the second timbre vector based on the timbre principal component weight parameters to obtain a third timbre vector;

[0012] extract semantic features of the text content through a pre-trained semantic generation model;

[0013] input the third timbre vector and the semantic features into a pre-trained speech conversion model, and output speech results.

[0014] Optionally, the training process of the speech conversion model comprises:

[0015] obtain first corpus data of a large-scale corpus;

[0016] under the constraint of a text classification loss function, train a speech recognition module of a speech auto-encoding decoding model using the first corpus data, to obtain a semantic extraction model;

[0017] under the constraint of a self-restoration loss function, train a speech vocoder model of the speech auto-encoding decoding model using the first corpus data, to obtain a timbre extraction model;

[0018] fuse the semantic extraction model and the timbre extraction model, to obtain a speech conversion model.

[0019] Optionally, the method further comprises:

[0020] input the third timbre vector and the semantic features into a pre-trained speech generation model, to generate speech results;

[0021] the training process of the speech generation model comprises:

[0022] train the semantic extraction model using the first corpus data, to obtain a semantic generation model;

[0023] input the first corpus data into the timbre extraction model, to output a first timbre vector matrix;

[0024] based on the first timbre vector matrix, jointly use the semantic generation model and the speech conversion model, to obtain a speech generation model.

[0025] Optionally, the training process of the principal component analysis model comprises:

[0026] obtain second corpus data of a high-quality corpus;

[0027] input the second corpus data into the timbre extraction model, to output a second timbre vector matrix;

[0028] perform principal component analysis on the second timbre vector matrix, to obtain a principal component vector matrix and a principal component weight parameter;

[0029] Based on the principal component vector matrix and the principal component weight parameter, a principal component analysis model is trained by using the second corpus data.

[0030] Optionally, the training process of the timbre principal component parameter generation model comprises:

[0031] The timbre vector of the second corpus data is extracted by the timbre extraction model;

[0032] The principal component analysis model is used to perform principal component analysis on the timbre vector to obtain timbre principal component parameters;

[0033] The timbre principal component parameters and the pre-labeled timbre dataset are used to train a large language model to obtain a timbre principal component parameter generation model.

[0034] Optionally, the timbre principal component parameters and the pre-labeled timbre dataset are used to train a large language model to obtain a timbre principal component parameter generation model, comprising:

[0035] Under the constraint of the timbre gender classification loss function, the timbre principal component parameters and the pre-labeled timbre dataset are used to train a large language model to obtain a timbre principal component parameter generation model for explaining timbre gender.

[0036] Optionally, the timbre principal component parameters and the pre-labeled timbre dataset are used to train a large language model to obtain a timbre principal component parameter generation model, comprising:

[0037] Under the constraint of the timbre age classification loss function, the timbre principal component parameters and the pre-labeled timbre dataset are used to train a large language model to obtain a timbre principal component parameter generation model for explaining timbre age.

[0038] A speech generation device based on timbre parameterization control comprises:

[0039] A reference information acquisition unit is configured to acquire reference speech, acquire natural speech content based on a timbre description of the reference speech, and generate text content;

[0040] A timbre vector extraction unit is configured to extract a first timbre vector of the reference speech by using a pre-trained timbre extraction model;

[0041] A first parameterization control unit is configured to input the first timbre vector into a pre-trained principal component analysis model, output principal component weight parameters, and perform parameterization control on the first timbre vector based on the principal component weight parameters to obtain a second timbre vector;

[0042] a second parameterization control unit configured to input the natural voice content into a pre-trained timbre principal component parameter generation model, output a timbre principal component weight parameter, and perform parameterization control on the second timbre vector based on the timbre principal component weight parameter to obtain a third timbre vector;

[0043] a semantic feature extraction unit configured to extract semantic features of the text content by using a pre-trained semantic generation model;

[0044] a voice result output unit configured to input the third timbre vector and the semantic features into a pre-trained voice conversion model, and output a voice result.

[0045] A voice generation device based on timbre parameterization control, comprising a memory and a processor;

[0046] The memory is configured to store a program.

[0047] The processor is configured to execute the program to implement each step of the voice generation method based on timbre parameterization control.

[0048] A storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements each step of the voice generation method based on timbre parameterization control.

[0049] According to the above technical solution, the reference voice is obtained, the natural voice content is obtained and the text content is generated based on the timbre description of the reference voice, the first timbre vector of the reference voice is extracted by using a pre-trained timbre extraction model, the first timbre vector is input into a pre-trained principal component analysis model to output a principal component weight parameter, and the first timbre vector is parameterized based on the principal component weight parameter to obtain a second timbre vector, the natural voice content is input into a pre-trained timbre principal component parameter generation model to output a timbre principal component weight parameter, and the second timbre vector is parameterized based on the timbre principal component weight parameter to obtain a third timbre vector, the semantic features of the text content are extracted by using a pre-trained semantic generation model, the third timbre vector and the semantic features are input into a pre-trained voice conversion model, and a voice result is output. As can be seen, the semantic generation model is used to separate the semantic features of the text, avoid semantic interference of the timbre, improve the stability of generation, the weight parameter is output by using the principal component analysis model, the second and third timbre vectors are obtained by twice parameterization adjustment, the controllability of the timbre control is improved, the timbre principal component parameter corresponding to the natural voice description can be parameterized and adjusted, and the explainability of the timbre control is improved. BRIEF DESCRIPTION OF DRAWINGS

[0050] Various other advantages and benefits will become apparent to those of ordinary skill in the art, upon reading the following detailed description of the preferred embodiment. The accompanying drawings are included to provide a description of preferred embodiments, and are not meant to limit the present application. Moreover, like reference numerals in the various drawings indicate like elements. In the drawings:

[0051] Figure 1 A flow diagram of implementing voice generation based on timbre parameterization control is provided for an embodiment of the present application;

[0052] Figure 2 A data logic diagram of generating voice through various models is provided for an embodiment of the present application.

[0053] Figure 3 A diagram of training a voice conversion model is provided for an embodiment of the present application;

[0054] Figure 4 A diagram of training a voice generation model is provided for an embodiment of the present application;

[0055] Figure 5 A diagram of training a principal component analysis model is provided for an embodiment of the present application;

[0056] Figure 6 A diagram of training a timbre principal component parameter generation model is provided for an embodiment of the present application;

[0057] Figure 7 Another diagram of training a timbre principal component parameter generation model is provided for an embodiment of the present application;

[0058] Figure 8 A device structure diagram of implementing voice generation based on timbre parameterization control is provided for an embodiment of the present application;

[0059] Figure 9 A device structure diagram of implementing voice generation based on timbre parameterization control is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0061] The present application can be implemented based on a terminal with data processing capability, which can be a computer, a cloud, a server, etc.

[0062] Next, in combination with Figure 1 The voice generation method based on timbre parameterization control of the present application can include the following steps:

[0063] Step S110, obtaining a reference voice, obtaining natural voice content and generating text content based on the timbre description of the reference voice.

[0064] Specifically, the reference voice is voice data carrying specific original timbre information, which is the basic input source of timbre extraction. Based on the timbre description of the reference voice, the timbre properties of the reference voice (such as age, gender, timbre style, etc.) are defined, and then the natural voice content reflecting the corresponding timbre characteristics is generated according to the definition information. The text content can be structured text data converted from the natural voice content.

[0065] Step S120, extracting a first timbre vector of the reference voice through a pre-trained timbre extraction model.

[0066] Specifically, the timbre extraction model can capture the core timbre characteristics with exclusive identification in the reference voice, in addition to the voiceprint and tone characteristics, including timbre details that can reflect its uniqueness, such as timbre brightness, thickness, and delicacy, etc. These characteristics directly determine the exclusive timbre properties of the reference voice, ensuring that the extraction result can accurately reflect the core timbre differences of the voice that distinguish it from other voices, and avoiding irrelevant information interference.

[0067] Wherein, the first timbre vector is structured timbre data output after processing the reference voice, representing the original, unadjusted core timbre characteristics of the reference voice, including the voiceprint, tone characteristics, timbre brightness / thickness, etc. Key attributes exclusive to the reference voice.

[0068] Step S130, inputting the first timbre vector into a pre-trained principal component analysis model, outputting principal component weight parameters, and performing parameterization control on the first timbre vector based on the principal component weight parameters to obtain a second timbre vector.

[0069] Specifically, the principal component analysis model can receive the first timbre vector as input, deeply analyze the dimensions directly related to the timbre characteristics in the vector, such as the reference frequency interval of the tone, the brightness gradient of the timbre, and the thickness level of the sound line, etc. These dimensions constitute the key recognition features of the original reference voice timbre. Based on the analysis, the principal component analysis model can filter out the key principal components that play a leading role in the overall timbre style through an algorithm, and filter out redundant dimensions that have weak contribution to the core properties of the timbre, ensuring that the focus of subsequent regulation is on the core timbre properties.

[0070] The principal component weight parameter is directly related to the controllable dimension of timbre. Each weight parameter corresponds to a principal component that determines the core attribute of timbre, such as some parameters corresponding to pitch, brightness / thickness, and some parameters corresponding to age characteristics. For example, the crispness of a teenage voice, the stability of an old voice, or gender characteristics, such as the delicacy of a female voice and the low pitch of a male voice. The parameter value represents the degree of influence of the corresponding principal component on the final timbre, and the numerical adjustment can directionally strengthen or weaken the corresponding timbre attribute.

[0071] The second timbre vector is structured timbre data obtained by adjusting the principal component weight parameter of the first timbre vector output by the principal component analysis model, representing a timbre feature carrier that has been optimized and adapted to the specific core timbre dimension requirements. The adjustment process focuses on the key timbre principal components in the first timbre vector, and directionally strengthens or weakens the corresponding timbre attribute through the weight parameter, making the timbre more suitable for the preliminary control target, and is an intermediate data that bridges the initial timbre and fine-tuning.

[0072] Step S140, input the natural speech content to the pre-trained timbre principal component parameter generation model, output the timbre principal component weight parameter, and perform parameterized control on the second timbre vector based on the timbre principal component weight parameter to obtain a third timbre vector.

[0073] Specifically, the timbre principal component parameter generation model can convert natural speech content into quantifiable timbre control basis. The timbre principal component parameter generation model can receive natural speech content as input, analyze the timbre attribute description therein, and convert unstructured description information into quantifiable timbre principal component weight parameters corresponding to timbre control dimensions. The timbre principal component parameter generation model can be constrained by a timbre gender classification loss function and / or a timbre age classification loss function during training, so the timbre principal component parameter generation model can adjust the timbre gender and / or the timbre age.

[0074] The third timbre vector is structured timbre data obtained by secondarily adjusting the weight parameter of the second timbre vector output by the timbre principal component parameter generation model, representing a customized timbre requirement that completely matches the natural speech description, and a final timbre feature carrier after fine-tuned parameterized control. The adjustment is based on the timbre attribute defined by the natural speech content.

[0075] Step S150, extract the semantic features of the text content through the pre-trained semantic generation model.

[0076] Specifically, the semantic generation model is a model responsible for semantic processing in the speech generation process, which is used to extract semantic features in the text content and realize the decoupling of semantics and timbre information. By receiving the text content converted from the natural speech content as input, the semantic information such as language meaning and syntax logic in the text is analyzed, and the text is converted into structured semantic features. The semantic features can be input into the speech conversion model together with the third timbre vector adjusted by parameters, to ensure that the generated speech can accurately convey the semantic of the text, while avoiding the interference of semantic information on timbre generation.

[0077] Step S160, input the third timbre vector and the semantic features into the pre-trained speech conversion model, and output the speech result.

[0078] Specifically, the speech conversion model can fuse semantic information and customized timbre information to output speech results that meet the requirements. The speech conversion model can fuse the text meaning corresponding to the semantic features and the target timbre corresponding to the timbre vector, to ensure that the generated speech accurately conveys the semantic logic of the original text and strictly restores the customized timbre attributes, avoiding mutual interference between semantics and timbre.

[0079] The data logic of generating speech by each model is as shown in Figure 2 The reference speech is input into the timbre extraction model to obtain the timbre vector, and then input into the principal component analysis model to obtain the principal component weight parameter, and after parameterization control, a new timbre vector is obtained. After inputting the natural speech description into the timbre principal component parameter generation model based on the natural language description, the principal component weight parameter is obtained, and after parameterization control, a new timbre vector is obtained. At the same time, the text content is extracted by the semantic generation model to obtain the semantic features, and the latest timbre vector and semantic features are input into the speech conversion model to generate the final speech result.

[0080] The voice generation method based on timbre parameterization control provided by the embodiment is characterized in that: a reference voice is acquired, natural voice content is acquired and text content is generated based on a timbre description of the reference voice, a first timbre vector of the reference voice is extracted through a pre-trained timbre extraction model, the first timbre vector is input into a pre-trained principal component analysis model, principal component weight parameters are output, the first timbre vector is controlled in a parameterized manner based on the principal component weight parameters, a second timbre vector is obtained, the natural voice content is input into a pre-trained timbre principal component parameter generation model, timbre principal component weight parameters are output, the second timbre vector is controlled in a parameterized manner based on the timbre principal component weight parameters, a third timbre vector is obtained, semantic features of the text content are extracted through a pre-trained semantic generation model, the third timbre vector and the semantic features are input into a pre-trained voice conversion model, and voice results are output. As can be seen, the semantic generation model is used to separate the semantic features of the text, so as to avoid semantic interference of the timbre, improve the stability of generation, the weight parameters are output through the principal component analysis model, the second and third timbre vectors are obtained through twice parameterization adjustment, the controllability of the timbre control is improved, the timbre principal component parameters corresponding to the natural voice description can be adjusted in a parameterized manner, and the explainability of the timbre control is improved.

[0081] In some embodiments of the present application, the training process of the voice conversion model mentioned in the above embodiments is introduced, as shown in the following Figure 3 The process can include the following steps:

[0082] S1, acquiring first corpus data of a large-scale corpus.

[0083] The data of the large-scale corpus is large in scale and wide in coverage, can contain voice data of multiple different speakers, different scenes and different timbre styles, does not need to excessively emphasize the extreme accuracy of a single data, and focuses more on the diversity and volume of the data to meet the needs of the model for learning diversified voice features.

[0084] S2, training a voice recognition module of a voice auto-encoding and decoding model by using the first corpus data under the constraint of a text classification loss function, to obtain a semantic extraction model.

[0085] S3, training a voice vocoder model of the voice auto-encoding and decoding model by using the first corpus data under the constraint of a self-restoration loss function, to obtain a timbre extraction model.

[0086] S4, fusing the semantic extraction model and the timbre extraction model to obtain a voice conversion model.

[0087] It can be understood that the voice conversion model is obtained by fusing the semantic extraction model and the timbre extraction model, and can synchronously obtain accurate semantic data and high-quality timbre data, and can accurately transmit text semantic logic and strictly restore target timbre attributes when generating voice, thereby improving the naturalness and customization accuracy of the output voice.

[0088] On this basis, considering further generating higher-quality voice, the voice conversion model reinforced model output voice result can be used, specifically, the third timbre vector and the semantic feature can be input into the pre-trained voice generation model to generate a voice result.

[0089] The training process of the voice generation model is as shown in Figure 4 , and specifically can include:

[0090] The semantic extraction model is trained using the first corpus data to obtain a semantic generation model. The first corpus data is input into the timbre extraction model to output a first timbre vector matrix. Based on the first timbre vector matrix, the semantic generation model and the voice conversion model are combined to obtain a voice generation model.

[0091] It can be understood that since the semantic generation model and the first timbre vector matrix are both constructed based on the first corpus data, the semantic features and the timbre features have a unified source, which can reduce the fusion conflicts caused by data bias, making the matching of semantics and timbre more coordinated. At the same time, the voice generation model is constructed by synchronously associating the semantic generation model and the voice conversion model, which not only retains the accuracy of semantic extraction and ensures that the voice can accurately transmit the text meaning, but also relies on the first timbre vector matrix to ensure the integrity of the timbre features, effectively decouples the semantics and the timbre, and avoids mutual interference. The finally generated voice not only conforms to the semantic logic, but also can stably restore the target timbre, and takes into account the accuracy and customization, and adapts to the diversified and personalized voice scene requirements.

[0092] In some embodiments of the present application, the training process of the principal component analysis model mentioned in the foregoing embodiments is introduced, and the training process can include: Figure 5

[0093] S1, obtaining second corpus data of a high-quality corpus.

[0094] ​High-quality corpora serve as a data source for enhancing precise timbre analysis and parameterized control. These corpora contain high-quality data with representative timbre features; each data point has clear timbre information with minimal interference, prioritizing accuracy and representativeness over sheer scale. High-quality corpora provide data support for refined timbre-related processing. Firstly, they are used to obtain high-quality, representative timbre vector matrices through timbre extraction models, providing reliable data for speaker vector principal component analysis to obtain accurate principal component vector matrices and weight parameters. Secondly, they combine large-scale model annotation with manual annotation to generate timbre datasets with natural language descriptions.

[0095] S2. Input the second corpus data into the timbre extraction model and output the second timbre vector matrix.

[0096] It is understandable that using a timbre extraction model on a high-quality corpus can yield a high-quality and representative second timbre vector matrix.

[0097] S3. Perform principal component analysis on the second timbre vector matrix to obtain the principal component vector matrix and principal component weight parameters.

[0098] The principal component vector matrix represents the second timbre vector matrix based on a high-quality corpus. It is a set of core timbre dimension vectors extracted through principal component analysis. Each vector corresponds to a key and representative timbre attribute, representing a structured and standardized expression of high-quality timbre features. The principal component weight parameters represent the quantized values ​​corresponding one-to-one with each vector in the principal component vector matrix. They reflect the degree of influence of the corresponding principal component vector on the final timbre style, and their magnitude directly correlates with the strength of the corresponding timbre attribute.

[0099] S4. Based on the principal component vector matrix and principal component weight parameters, a principal component analysis model is trained using data from the second corpus.

[0100] Specifically, during the training of the principal component analysis model, a second corpus of data is introduced to cover diverse timbre samples that are suitable for real-world application scenarios. By inputting the timbre data from the second corpus into the model, the model continuously learns the mapping logic of input timbre features → principal component vector matching → weight parameter output. This continuously optimizes the model's accuracy in decomposing the core dimensions of timbre and the accuracy in calculating weight parameters, ultimately resulting in a principal component analysis model that can stably and accurately process various timbre vectors.

[0101] In some embodiments of this application, the training process of the timbre principal component parameter generation model mentioned in the foregoing embodiments is described, with reference to... Figure 6 The training process of the timbre principal component parameter generation model can specifically include:

[0102] S1. Extract the timbre vectors from the second corpus data using a timbre extraction model.

[0103] It is understandable that by extracting timbre vectors from the second corpus data through a timbre extraction model, the advantages of the second corpus's diverse samples that are adapted to actual application scenarios can be fully explored. This allows for the accurate capture of timbre features under different scenarios and styles, making the generated timbre vectors both rich and scenario-adaptable. This avoids the limitations of timbre features caused by single data and improves the generalization ability of subsequent model processing.

[0104] S2. Use the principal component analysis model to perform principal component analysis on the timbre vector to obtain the timbre principal component parameters.

[0105] Understandably, using principal component analysis (PCA) models to perform PCA on timbre vectors enables precise decomposition and quantifiable control of timbre features. PCA models can filter out the key principal components that determine the core attributes of timbre from complex timbre vectors, eliminate redundant and interfering features, and transform unstructured timbre information into directly adjustable quantifiable parameters, allowing for more precise control over timbre adjustments.

[0106] S3. Using the timbre principal component parameters and the pre-labeled timbre dataset, train the large language model to obtain the timbre principal component parameter generation model.

[0107] Among these, the timbre dataset can be annotated using large models or manually. Large language models, when trained, can be subject to factors such as... Figure 7 The constraints of the timbre gender classification loss function and the timbre age classification loss function are shown.

[0108] Specifically, under the constraint of a timbre gender classification loss function, a large language model can be trained using the timbre principal component parameters and a pre-labeled timbre dataset to obtain a timbre principal component parameter generation model for explaining timbre gender. Similarly, under the constraint of a timbre age classification loss function, a large language model can be trained using the timbre principal component parameters and a pre-labeled timbre dataset to obtain a timbre principal component parameter generation model for explaining timbre age. Furthermore, under the constraints of both timbre gender and timbre age classification loss functions, a large language model can be trained using the timbre principal component parameters and a pre-labeled timbre dataset to obtain a timbre principal component parameter generation model for explaining timbre age and timbre gender.

[0109] The speech generation apparatus based on timbre parameterization control provided in the embodiments of this application will be described below. The speech generation apparatus based on timbre parameterization control described below and the speech generation method based on timbre parameterization control described above can be referred to in correspondence with each other.

[0110] SeeFigure 8 , Figure 8 This is a schematic diagram of a device for implementing speech generation based on timbre parameterization control, as disclosed in an embodiment of this application.

[0111] like Figure 8 As shown, the device may include:

[0112] Reference information acquisition unit 11 is used to acquire reference speech, acquire natural speech content and generate text content based on the timbre description of the reference speech;

[0113] The timbre vector extraction unit 12 is used to extract the first timbre vector of the reference speech through a pre-trained timbre extraction model;

[0114] The first parameterization control unit 13 is used to input the first timbre vector into a pre-trained principal component analysis model, output principal component weight parameters, and perform parameterization control on the first timbre vector based on the principal component weight parameters to obtain a second timbre vector.

[0115] The second parameterization control unit 14 is used to input the natural speech content into a pre-trained timbre principal component parameter generation model, output timbre principal component weight parameters, and perform parameterization control on the second timbre vector based on the timbre principal component weight parameters to obtain a third timbre vector.

[0116] The semantic feature extraction unit 15 is used to extract the semantic features of the text content through a pre-trained semantic generation model;

[0117] The speech result output unit 16 is used to input the third timbre vector and the semantic features into a pre-trained speech conversion model and output the speech result.

[0118] Optionally, the specific implementation logic of other units of the speech generation device based on timbre parameterization control corresponds to the speech generation method based on timbre parameterization control mentioned in the foregoing embodiments, and will not be repeated here.

[0119] The speech generation device based on timbre parameterization control provided in this application embodiment can be applied to speech generation devices based on timbre parameterization control, such as terminals: mobile phones, computers, etc. Optionally, Figure 9 The hardware block diagram of a speech generation device based on timbre parameterization control is shown. (Refer to...) Figure 9 The hardware structure of the xx device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0120] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0121] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0122] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0123] The memory stores a program, which the processor can call. The program is used for:

[0124] Obtain reference speech, and based on the timbre description of the reference speech, obtain natural speech content and generate text content;

[0125] The first timbre vector of the reference speech is extracted using a pre-trained timbre extraction model;

[0126] The first timbre vector is input into a pre-trained principal component analysis model, which outputs principal component weight parameters. Based on the principal component weight parameters, the first timbre vector is parameterized to obtain the second timbre vector.

[0127] The natural speech content is input into a pre-trained timbre principal component parameter generation model, which outputs timbre principal component weight parameters. Based on the timbre principal component weight parameters, the second timbre vector is parameterized to obtain the third timbre vector.

[0128] The semantic features of the text content are extracted using a pre-trained semantic generation model;

[0129] The third timbre vector and the semantic features are input into a pre-trained speech conversion model, and the speech result is output.

[0130] Optionally, the refined and extended functions of the program can be found in the description above.

[0131] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor, the program being used for:

[0132] Obtain reference speech, and based on the timbre description of the reference speech, obtain natural speech content and generate text content;

[0133] The first timbre vector of the reference speech is extracted using a pre-trained timbre extraction model;

[0134] The first timbre vector is input into a pre-trained principal component analysis model, which outputs principal component weight parameters. Based on the principal component weight parameters, the first timbre vector is parameterized to obtain the second timbre vector.

[0135] The natural speech content is input into a pre-trained timbre principal component parameter generation model, which outputs timbre principal component weight parameters. Based on the timbre principal component weight parameters, the second timbre vector is parameterized to obtain the third timbre vector.

[0136] The semantic features of the text content are extracted using a pre-trained semantic generation model;

[0137] The third timbre vector and the semantic features are input into a pre-trained speech conversion model, and the speech result is output.

[0138] Optionally, the refined and extended functions of the program can be found in the description above.

[0139] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0140] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0141] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech generation method based on timbre parameterization control, characterized in that, include: Obtain reference speech, and based on the timbre description of the reference speech, obtain natural speech content and generate text content; The first timbre vector of the reference speech is extracted using a pre-trained timbre extraction model; The first timbre vector is input into a pre-trained principal component analysis model, which outputs principal component weight parameters. The first timbre vector is then parameterized based on the principal component weight parameters to obtain a second timbre vector. The principal component analysis model is used to analyze the dimensions directly related to the timbre characteristics in the vector and output the weight parameters of the adjustable dimensions directly related to the timbre. The natural speech content is input into a pre-trained timbre principal component parameter generation model, which outputs timbre principal component weight parameters. Based on the timbre principal component weight parameters, the second timbre vector is parameterized to obtain a third timbre vector. The timbre principal component parameter generation model is used to parse the timbre attribute description of the natural speech content and output quantifiable timbre principal component weight parameters of the timbre control dimension corresponding to the timbre attribute description. The semantic features of the text content are extracted using a pre-trained semantic generation model; The third timbre vector and the semantic features are input into a pre-trained speech conversion model, and the speech result is output.

2. The method according to claim 1, characterized in that, The training process of the speech conversion model includes: Obtain first corpus data from a large-scale corpus; Under the constraint of the text classification loss function, the speech recognition module of the speech autoencoder-decoder model is trained using the data from the first corpus to obtain the semantic extraction model; Under the constraint of the self-reducing loss function, the speech vocoder model of the speech autoencoder-decoder model is trained using the data from the first corpus to obtain the timbre extraction model. By integrating the semantic extraction model and the timbre extraction model, a speech conversion model is obtained.

3. The method according to claim 2, characterized in that, Also includes: The third timbre vector and the semantic features are input into a pre-trained speech generation model to generate speech results; The training process of the speech generation model includes: The semantic extraction model is trained using the data from the first corpus to obtain a semantic generation model; The first corpus data is input into the timbre extraction model, and the first timbre vector matrix is ​​output. Based on the first timbre vector matrix, the semantic generation model and the speech conversion model are combined to obtain the speech generation model.

4. The method according to claim 1, characterized in that, The training process of the principal component analysis model includes: Obtain high-quality second corpus data; The second corpus data is input into the timbre extraction model, and the second timbre vector matrix is ​​output. Principal component analysis is performed on the second timbre vector matrix to obtain the principal component vector matrix and principal component weight parameters. Based on the principal component vector matrix and the principal component weight parameters, a principal component analysis model is trained using the second corpus data.

5. The method according to claim 4, characterized in that, The training process of the timbre principal component parameter generation model includes: The timbre vector of the second corpus data is extracted using the timbre extraction model. The timbre vector is subjected to principal component analysis using the principal component analysis model to obtain the timbre principal component parameters. Using the principal component parameters of timbre and the pre-labeled timbre dataset, a large language model is trained to obtain a principal component parameter generation model of timbre.

6. The method according to claim 5, characterized in that, Using the principal component parameters of timbre and a pre-labeled timbre dataset, a large language model is trained to obtain a principal component parameter generation model of timbre, including: Under the constraint of the timbre gender classification loss function, the large language model is trained using the timbre principal component parameters and the pre-labeled timbre dataset to obtain a timbre principal component parameter generation model for explaining timbre gender.

7. The method according to claim 5, characterized in that, Using the principal component parameters of timbre and a pre-labeled timbre dataset, a large language model is trained to obtain a principal component parameter generation model of timbre, including: Under the constraint of the timbre age classification loss function, the large language model is trained using the timbre principal component parameters and the pre-labeled timbre dataset to obtain a timbre principal component parameter generation model for interpreting timbre age.

8. A speech generation device based on timbre parameterization control, characterized in that, include: A reference information acquisition unit is used to acquire reference speech, acquire natural speech content and generate text content based on the timbre description of the reference speech; A timbre vector extraction unit is used to extract the first timbre vector of the reference speech through a pre-trained timbre extraction model; The first parameterized control unit is used to input the first timbre vector into a pre-trained principal component analysis model, output principal component weight parameters, and perform parameterized control on the first timbre vector based on the principal component weight parameters to obtain a second timbre vector. The principal component analysis model is used to analyze the dimensions directly related to the timbre characteristics in the vector and output the weight parameters of the adjustable dimensions directly related to the timbre. The second parameterized control unit is used to input the natural speech content into a pre-trained timbre principal component parameter generation model, output timbre principal component weight parameters, and perform parameterized control on the second timbre vector based on the timbre principal component weight parameters to obtain a third timbre vector. The timbre principal component parameter generation model is used to parse the timbre attribute description of the natural speech content and output quantifiable timbre principal component weight parameters of the timbre control dimension corresponding to the timbre attribute description. A semantic feature extraction unit is used to extract semantic features of the text content through a pre-trained semantic generation model; The speech result output unit is used to input the third timbre vector and the semantic features into a pre-trained speech conversion model and output the speech result.

9. A speech generation device based on timbre parameterization control, characterized in that, Including memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement the various steps of the speech generation method based on timbre parameterization control as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the various steps of the speech generation method based on timbre parameterization control as described in any one of claims 1-7.

Images