Voice conversion model training method and device, computer equipment and storage medium

Abstract

The embodiment of the invention provides a voice conversion model training method and device, computer equipment and a storage medium, and belongs to the technical field of artificial intelligence. Comprising the following steps: acquiring sample voice and sample timbre characteristics and sample text characteristics of the sample voice, and acquiring reference voice characteristics and reference timbre characteristics of reference voice; inputting a sample voice into the trained coding sub-model, and outputting a sample voice feature of the sample voice; positively inputting the sample voice feature, the sample text feature and the sample tone feature into a reversible generation sub-model, outputting a target text feature, and determining a first loss value between the target text feature and the sample text feature; reversely inputting the target text feature and the reference tone feature into a reversible generation sub-model, outputting a target voice feature, and determining a second loss value between the target voice feature and the reference voice feature; and iterating the reversible generation sub-model according to the first loss value and the second loss value to obtain a trained reversible generation sub-model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a speech conversion model training method, apparatus, computer equipment and storage medium. Background Art

[0002] With the development of artificial intelligence technology, some artificial intelligence models can be used to achieve voice timbre conversion to generate voice in virtual media content. For example, an artificial intelligence model can obtain the voice content in the voice of an object, and then combine the voice content with the timbre of other objects to generate a new voice to achieve the timbre conversion of the voice content of the object.

[0003] In the related art, the voiceprint features of other objects are extracted through a pre-trained voiceprint feature extraction model, the voiceprint features are used to characterize the timbre, the phonetic posterior-grams (PPG) in the current voice are obtained through a pre-trained speech recognition model, and then the phonetic posterior-grams and voiceprint features are input into the voice conversion model to generate new voice. However, the voiceprint features can only distinguish different timbres, but cannot describe different timbres, and the adaptability of the phonetic posterior-grams and the voice conversion model is poor, which leads to the problem of poor timbre consistency in the new voice generated by the voice conversion model through the phonetic posterior-grams and voiceprint features, that is, the timbre of the current object and the timbre of other objects will exist in the new voice. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a speech conversion model training method, device, computer equipment and storage medium, which improves the timbre consistency of the speech generated after the speech to be converted is converted.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for training a speech conversion model, wherein the speech conversion model includes an encoding sub-model and a reversible generation sub-model, and the method includes:

[0006] Acquire a sample speech and a sample timbre feature and a sample text feature of the sample speech, and acquire a reference speech feature and a reference timbre feature of a reference speech;

[0007] Inputting the sample speech into the trained encoding sub-model, and outputting the sample speech features of the sample speech;

[0008] Forward inputting the sample speech feature, the sample text feature and the sample timbre feature into a reversible generation sub-model, outputting a target text feature, and determining a first loss value between the target text feature and the sample text feature;

[0009] Reversely inputting the target text feature and the reference timbre feature into the reversible generative sub-model, outputting the target speech feature, and determining a second loss value between the target speech feature and the reference speech feature;

[0010] A total loss value is determined according to the first loss value and the second loss value, and model parameters of the reversible generation sub-model are adjusted based on the total loss value to iterate the reversible generation sub-model, and return to execute to obtain sample speech and sample timbre features and sample text features of the sample speech until the total loss value meets a preset loss condition, thereby obtaining a trained reversible generation sub-model, and the trained speech conversion model includes the trained reversible generation sub-model and the trained encoding sub-model.

[0011] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a speech conversion model training device, the device comprising:

[0012] An acquisition module, used to acquire a sample speech and a sample timbre feature and a sample text feature of the sample speech, and to acquire a reference speech feature and a reference timbre feature of a reference speech;

[0013] A first input module, used for inputting the sample speech into the trained encoding sub-model, and outputting the sample speech features of the sample speech;

[0014] A second input module, used for forward inputting the sample speech feature, the sample text feature and the sample timbre feature into the reversible generation sub-model, outputting the target text feature, and determining a first loss value between the target text feature and the sample text feature;

[0015] A third input module, used for reversely inputting the target text feature and the reference timbre feature into the reversible generation sub-model, outputting the target speech feature, and determining a second loss value between the target speech feature and the reference speech feature;

[0016] An iterative module is used to determine a total loss value according to the first loss value and the second loss value, and adjust the model parameters of the reversible generation sub-model based on the total loss value to iterate the reversible generation sub-model, and return to execute to obtain sample speech and sample timbre features and sample text features of the sample speech until the total loss value meets a preset loss condition, thereby obtaining a trained reversible generation sub-model, and the trained speech conversion model includes the trained reversible generation sub-model and the trained encoding sub-model.

[0017] In some implementations, the speech conversion model further includes a decoding sub-model, and the speech conversion model training device further includes a training module for:

[0018] Acquire training speech, input the training speech into the encoding sub-model, and output training speech features of the training speech;

[0019] Inputting the training speech features into the decoding sub-model and outputting the target speech;

[0020] The difference between the target speech and the training speech is determined, and the encoding sub-model and the decoding sub-model are iteratively trained according to the difference to obtain a trained encoding sub-model and a trained decoding sub-model.

[0021] In some embodiments, the training module is further configured to:

[0022] Inputting the target speech into a trained discriminant network, and outputting a target discriminant value corresponding to the target speech;

[0023] The square value of the difference between the target discrimination value and the preset value is determined, and the square value is determined as the difference between the target speech and the training speech.

[0024] In some embodiments, the speech conversion model training device further includes a conversion module, which is used to:

[0025] Acquire a first speech to be converted and a first timbre feature and a first text feature of the first speech;

[0026] Inputting the first speech into the trained encoding sub-model and outputting a first speech feature;

[0027] Forward inputting the first speech feature, the first timbre feature and the first text feature into the trained reversible generative sub-model, and outputting a first target text feature;

[0028] Reversely inputting the first target text feature and the preset timbre feature into the trained reversible generation sub-model to output a second speech feature;

[0029] The second speech feature is input into the trained decoding sub-model, and a second speech is output after the timbre of the first speech is converted.

[0030] In some embodiments, the second input module is used to:

[0031] Determine the time dimension corresponding to the sample speech feature, and adjust the dimension of the sample text feature according to the time dimension to obtain a target sample text feature;

[0032] The sample speech features, the target sample text features and the sample timbre features are forwardly input into a reversible generation sub-model, and the target text features are output.

[0033] In some embodiments, the second input module is used to:

[0034] Determining the KL divergence between the target sample text feature and the target text feature;

[0035] The KL divergence is determined as a first loss value between the target text feature and the sample text feature.

[0036] In some implementations, the speech conversion model further includes a pre-trained timbre extraction model and a pre-trained text model, and an acquisition module for:

[0037] Inputting the sample speech into a pre-trained timbre extraction model, and outputting sample timbre features of the sample speech;

[0038] The sample speech is input into a pre-trained text model, and sample text features of the sample speech are output.

[0039] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores multiple instructions, and the instructions are suitable for a processor to load to execute the speech conversion model training method provided in the embodiment of the present application.

[0040] To achieve the above-mentioned purpose, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the speech conversion model training method of the embodiment of the present application when executing the computer program.

[0041] The speech conversion model training method, device, computer equipment and storage medium proposed in the present application, wherein the speech conversion model includes an encoding sub-model and a reversible generation sub-model, and obtains reference speech features and reference timbre features of a reference speech by obtaining sample speech and sample timbre features and sample text features of the sample speech; the sample speech is input into the trained encoding sub-model, and the sample speech features of the sample speech are output; the sample speech features, sample text features and sample timbre features are forward input into the reversible generation sub-model, and the target text features are output, and a first loss value between the target text features and the sample text features is determined; the target The marked text features and the reference timbre features are reversely input into the reversible generative sub-model, the target speech features are output, and a second loss value between the target speech features and the reference speech features is determined; a total loss value is determined according to the first loss value and the second loss value, and the model parameters of the reversible generative sub-model are adjusted based on the total loss value to iterate the reversible generative sub-model, and return to execute to obtain the sample speech and the sample timbre features and sample text features of the sample speech until the total loss value meets the preset loss condition, and the trained reversible generative sub-model is obtained. The trained speech conversion model includes the trained reversible generative sub-model and the trained encoding sub-model.

[0042] In this way, the sample timbre features and sample text features of the sample speech are obtained, and the sample speech features, sample text features and sample timbre features are forwardly input into the reversible generation sub-model, and the sample timbre features and sample text features are used to guide the reversible generation sub-model to decouple the timbre and text of the sample speech features, accurately extract the target text features in the sample speech features, and then determine the first loss value between the target text features and the real sample text features, and measure the accuracy of the reversible generation sub-model in decoupling the timbre and text in the sample speech. The reversible generation sub-model has a reversible characteristic, and then the target text features and the reference timbre features are reversely input into the reversible generation sub-model, the target speech features are output, and the second loss value between the target speech features and the real reference speech features is determined, and the ability of the reversible generation sub-model to combine the reference timbre features and the target text features is measured by the second loss value, and finally the total loss value is determined according to the first loss value and the second loss value, and the model parameters of the reversible generation sub-model are adjusted based on the total loss value to iterate the reversible generation sub-model. The reversible generative sub-model trained in this way has the ability to decouple the timbre and text of the forward input speech features, and at the same time has the ability to combine text and other timbre to generate speech features. Therefore, compared with the related art of inputting speech posterior graphs and voiceprint features into the speech conversion model to generate new speech, the speech conversion model trained in this application can combine the text in the input speech with the timbre of other people to generate new speech, and the timbre consistency in the new speech is relatively high. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a structural diagram of a speech conversion model provided in an embodiment of the present application;

[0044] Figure 2 It is a flowchart of the speech conversion model training method provided in the embodiment of the present application;

[0045] Figure 3 It is a schematic diagram of the structure of the encoding sub-model and the decoding sub-model provided in the embodiment of the present application;

[0046] Figure 4 It is a flowchart of voice conversion provided by an embodiment of the present application;

[0047] Figure 5 is another flowchart of the speech conversion model training method provided in an embodiment of the present application;

[0048] Figure 6 It is a structural schematic diagram of a speech conversion model training device provided in an embodiment of the present application;

[0049] Figure 7 It is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0053] First, some nouns involved in this application are analyzed:

[0054] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0055] Timbre: A physics term that refers to the fact that different sounds always have distinctive characteristics in terms of waveform. Different objects have different vibration characteristics. Timbre is one of the properties of sound (i.e. loudness, pitch, timbre), which is mainly determined by its overtones. The difference in each person's voice and the sound produced by various musical instruments is caused by different timbres.

[0056] Flow model: a probability generation model based on normalizing flows. The main idea of ​​the Flow model is to transform a simple probability distribution (such as the standard normal distribution) into a complex target distribution through a series of reversible transformations. It consists of multiple reversible transformation layers, which approximate the distribution of the target data by continuously changing the distribution of variables.

[0057] Secondly, describe the technical problems existing in related technologies:

[0058] With the development of artificial intelligence technology, some artificial intelligence models can be used to achieve voice timbre conversion to generate voice in virtual media content. For example, an artificial intelligence model can obtain the voice content in the voice of an object, and then combine the voice content with the timbre of other objects to generate a new voice to achieve the timbre conversion of the voice content of the object.

[0059] In the related art, the voiceprint features of other objects are extracted through a pre-trained voiceprint feature extraction model, the voiceprint features are used to characterize the timbre, the phonetic posterior-grams (PPG) in the current voice are obtained through a pre-trained speech recognition model, and then the phonetic posterior-grams and voiceprint features are input into the voice conversion model to generate new voice. However, the voiceprint features can only distinguish different timbres, but cannot describe different timbres, and the adaptability of the phonetic posterior-grams and the voice conversion model is poor, which leads to the problem of poor timbre consistency in the new voice generated by the voice conversion model through the phonetic posterior-grams and voiceprint features, that is, the timbre of the current object and the timbre of other objects will exist in the new voice.

[0060] In order to solve the above technical problems, the present application provides a speech conversion model, which includes a reversible generation sub-model. The reversible generation sub-model has the functions of forward input and reverse input. When the forward input is used, the reversible generation sub-model can decouple the timbre and text in the speech features, so as to accurately obtain the text features of the input speech. When the text features and the timbre features of the timbre to be converted are reversely input, the reversible generation sub-model can combine the text features and the timbre features of the timbre to be converted, so as to generate a new speech with timbre consistency. The specific training process of the speech conversion model will be described in detail later.

[0061] Specifically, the embodiments of the present application provide a speech conversion model training method, apparatus, computer equipment and storage medium, which can improve the timbre consistency of the speech generated after the speech to be converted undergoes speech conversion.

[0062] See also Figure 1 , Figure 1 It is a scenario diagram of the speech conversion model training method provided in an embodiment of the present application.

[0063] The speech conversion model includes a pre-trained timbre extraction model, a pre-trained text model, an encoding sub-model, a decoding sub-model and a reversible generation sub-model, wherein the reversible generation sub-model can be inputted in both forward and reverse directions. When the reversible generation sub-model is inputted with the speech features, timbre features and text features of the speech in a forward direction, the reversible generation sub-model can decouple the timbre and text in the speech features, thereby outputting the speech features extracted from the speech features. When the reversible generation sub-model is inputted with the text features and timbre features in a reverse direction, the reversible generation sub-model can combine the input text features and timbre features to generate a new speech feature, based on which a new speech can be obtained. The reversible generation sub-model can be a flow model.

[0064] The pre-trained timbre extraction model can extract the timbre features in the speech to obtain the timbre features, which are vectors. The pre-trained text can perform matrix embedding on the input text to generate the text features of the text, which can be vectors. The encoding sub-model can extract the input speech to obtain the speech features in the speech, which can be the spectral features of the speech, specifically, vectors. The decoding sub-model can process the new speech features output by the inverse reversible generation sub-model to obtain new speech.

[0065] Next, we will introduce the operation of the entire speech conversion model in the form of data flow description. First, the current speaker's speech is input into the encoding sub-model, the current speaker's speech features are output, and the current speaker's speech features are forward input into the reversible generation sub-model.

[0066] The speech of the current speaker is input into the pre-trained timbre extraction model, the timbre features of the current speaker are output, and the timbre features of the current speaker are forward input into the reversible generation sub-model.

[0067] Get the text in the current speaker's speech, input the text into the pre-trained text model, output the text features of the current speaker's speech, and then forward input the text features of the current speaker into the reversible generation sub-model.

[0068] After receiving the forward input of the current speaker's voice features, timbre features, and text features, the reversible generative sub-model can decouple the text and timbre in the current speaker's voice features based on the guidance of the current speaker's timbre features and text features to obtain decoupled text features. It should be noted that the decoupled text features are different from the text features output by the pre-trained text model, such as different spatial dimensions, and the decoupled text features are more relevant to the current speaker's voice.

[0069] The voice of the target timbre speaker is input into the pre-trained timbre extraction model, the timbre features of the target timbre speaker are output, and the timbre features of the target timbre speaker and the decoupled text features are reversely input into the reversible generation sub-model. The reversible generation sub-model combines the timbre features of the target timbre speaker and the decoupled text features to generate new speech features.

[0070] The new speech features are input into the decoding sub-model, which can perform dimensionality reduction processing on the new speech features to obtain new speech, the text content of which is the same as that of the current speaker's speech, and the timbre of which is the same as that of the target speaker, thereby achieving the conversion of the timbre of the current speaker's speech.

[0071] From the above content, it can be seen that in this application, the voice conversion model mainly adopts the reversible characteristics of the reversible generation sub-model, so it can better combine the timbre characteristics of the target timbre speaker and the decoupled text features of the current speaker, thereby realizing the conversion of the timbre of the current speaker's voice. Thereby improving the consistency of the timbre of the converted voice.

[0072] The speech conversion model training method, apparatus, computer equipment and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the speech conversion model training method in the embodiments of the present application is described.

[0073] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0074] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0075] The speech conversion model training method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The speech conversion model training method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the speech conversion model training method, etc., but is not limited to the above forms.

[0076] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0077] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0078] See also Figure 2 , Figure 2 : is a flow chart of a speech conversion model training method provided in an embodiment of the present application. The speech conversion model training method may include the following steps:

[0079] Step 110: obtaining a sample speech and a sample timbre feature and a sample text feature of the sample speech, and obtaining a reference speech feature and a reference timbre feature of a reference speech;

[0080] Step 120: input the sample speech into the trained encoding sub-model, and output the sample speech features of the sample speech;

[0081] Step 130: forwardly input the sample speech feature, the sample text feature and the sample timbre feature into the reversible generation sub-model, output the target text feature, and determine a first loss value between the target text feature and the sample text feature;

[0082] Step 140: reversely input the target text feature and the reference timbre feature into the reversible generation sub-model, output the target speech feature, and determine a second loss value between the target speech feature and the reference speech feature;

[0083] Step 150, determine the total loss value according to the first loss value and the second loss value, and adjust the model parameters of the reversible generation sub-model based on the total loss value to iterate the reversible generation sub-model, and return to execute to obtain the sample speech and the sample timbre features and sample text features of the sample speech until the total loss value meets the preset loss condition to obtain the trained reversible generation sub-model.

[0084] Steps 110 to 150 will be described in detail below.

[0085] In step 110, a sample speech and a sample timbre feature and a sample text feature of the sample speech are obtained, and a reference speech feature and a reference timbre feature of a reference speech are obtained.

[0086] Specifically, the timbre features in the speech can be obtained through a pre-trained timbre extraction model, and the text of the speech can be input into a pre-trained text model to obtain the text features of the speech.

[0087] In some implementations, the sample speech is input into a pre-trained timbre extraction model to output sample timbre features of the sample speech; the sample speech is input into a pre-trained text model to output sample text features of the sample speech.

[0088] The reference speech is input into the pre-trained timbre extraction model, and the reference timbre features of the reference speech are output; the reference speech is input into the trained encoding sub-model, and the reference speech features of the reference speech are output.

[0089] It should be noted that the sample voice can be understood as the voice of the current speaker, and the reference voice can be understood as the voice of the target timbre speaker.

[0090] In this application, the encoding sub-model and the decoding sub-model need to be trained together in the first stage. The specific training process may include the following steps:

[0091] (1.1) obtaining training speech, inputting the training speech into the encoding sub-model, and outputting training speech features of the training speech;

[0092] (1.2) Input the training speech features into the decoding sub-model and output the target speech;

[0093] (1.3) Determine the difference between the target speech and the training speech, and iteratively train the encoding sub-model and the decoding sub-model based on the difference to obtain a trained encoding sub-model and a trained decoding sub-model.

[0094] First, a batch of speech data for training needs to be collected. These speech data can come from various sources, such as real speech recorded by recording equipment, speech samples in speech databases, etc. The training speech should have a certain degree of diversity and representativeness, covering different speakers, accents, speaking speeds, emotions and other factors, so that the model can learn speech features and patterns in different situations.

[0095] The purpose of the encoder submodel is to convert the input training speech into a specific feature representation. This feature representation is usually a low-dimensional vector or a set of feature parameters, which can capture the key information in the training speech, such as spectral features, prosodic features, speaker features, etc.

[0096] After being processed by the encoding sub-model, the output training speech features are a compressed and abstract representation of the original training speech. This feature can be seen as an encoding of the training speech, which retains the important information in the original speech while removing some unnecessary details and noise. These features can be used in the subsequent decoding process and as an intermediate representation in the model training process to help the model learn how to convert from input speech to output speech.

[0097] The task of the decoding sub-model is to generate the target speech based on the input training speech features. This target speech should be as close to the original training speech as possible, but there may be some differences because the decoding process is a reconstruction process and may not be able to completely restore the original speech signal.

[0098] The target speech output by the decoding sub-model is a reconstruction of the original training speech. This reconstructed speech can be compared with the original training speech to evaluate the performance of the model and the effect of training.

[0099] In order to evaluate the performance of the model and the effect of training, it is necessary to determine the difference between the target speech and the training speech. This difference can be measured in various ways, such as calculating the mean square error (MSE), signal-to-noise ratio (SNR), perceptual speech quality evaluation (PESQ) and other indicators between them.

[0100] These indicators can reflect the similarity between the target speech and the training speech in different aspects, such as the similarity of audio waveforms, the similarity of spectra, the subjective perception of speech quality, etc.

[0101] The encoding sub-model and the decoding sub-model are iteratively trained based on the difference between the target speech and the training speech. This process is usually implemented through optimization algorithms such as stochastic gradient descent (SGD), Adam, etc.

[0102] In each iteration, the optimization algorithm calculates the update direction and step size of the model parameters based on the current model parameters and the difference between the target speech and the training speech, and then updates the model parameters so that the difference between the target speech and the training speech gradually decreases.

[0103] This process is repeated many times until the performance of the model reaches a certain standard or converges. In each iteration, different training data samples can be used to increase the generalization ability and robustness of the model.

[0104] After multiple iterations of training, the parameters of the encoding sub-model and the decoding sub-model are gradually adjusted so that they can better process the input training speech and generate target speech that is closer to the original training speech. Finally, the trained encoding sub-model and the trained decoding sub-model are obtained, which can be used to encode and decode new speech and implement various speech processing tasks, such as speech synthesis.

[0105] In some implementations, determining the difference between the target speech and the training speech includes:

[0106] (1.3.1) Input the target speech into the trained discriminant network and output the target discriminant value corresponding to the target speech;

[0107] (1.3.2) Determine the square value of the difference between the target discrimination value and the preset value, and determine the square value as the difference between the target speech and the training speech.

[0108] Among them, the encoding sub-model and the decoding sub-model can be regarded as a generation network, and then a pre-trained discriminant network is obtained. The function of the discriminant network is to discriminate the generated speech in the generation network to determine the difference between it and the real speech.

[0109] Specifically, the target speech can be input into the trained discriminant network, and the target discriminant value corresponding to the target speech can be output, which is equivalent to a scoring value. Then the square value of the difference between the target discriminant value and the preset value is determined, and the square value is determined as the difference between the target speech and the training speech. After the difference is obtained, the parameters of the encoding sub-model and the decoding sub-model are adjusted to perform iterative training.

[0110] Please also read Figure 3 , Figure 3 1 is a schematic diagram of the structure of the encoding sub-model and the decoding sub-model provided in the embodiment of the present application, wherein the structure of the encoding sub-model is the same as that of the decoding sub-model.

[0111] The input of the decoding sub-model is sample speech, which is continuous. The continuous sample speech is concatenated and the initial features of the entire concatenated speech are obtained. The initial features are then processed by the tanh function and mapped to the interval [-1,1] to obtain a feature vector, which can make the data have certain nonlinear changes within this range and enhance the expressiveness of the model.

[0112] The initial features are mapped to the interval [0,1] through the sigmoid function to obtain another feature vector, and nonlinearity is introduced in the middle layer. The two feature vectors are point-multiplied to obtain a fusion vector. Then it is input into the convolution layer and the first convolution vector is output.

[0113] The sample speech is input into another convolution layer, the second convolution vector is output, and finally the first convolution vector and the second convolution vector are added to obtain the speech features output by the encoding sub-model. Similarly, the decoding sub-model simply changes its input to the speech features output by the encoding sub-model, and the subsequent data processing logic is consistent.

[0114] In step 120, the sample speech is input into the trained encoding sub-model, and the sample speech features of the sample speech are output.

[0115] The sample speech is input into the trained coding sub-model, and the trained coding sub-model has the ability to accurately extract speech features in the speech, such as extracting spectral features in the speech.

[0116] In step 130, the sample speech features, sample text features and sample timbre features are forwardly input into the reversible generation sub-model, the target text features are output, and the first loss value between the target text features and the sample text features is determined.

[0117] After the first stage of training the encoding sub-model and the decoding sub-model, the reversible generation sub-model needs to be trained in the second stage through the trained encoding sub-model and the decoding sub-model to obtain the trained reversible generation sub-model.

[0118] Among them, the sample speech features, sample text features and sample timbre features can be forward input into the reversible generation sub-model. Since the sample text features and sample timbre features are both real features obtained, the reversible generation sub-model can decouple the sample speech features based on the guidance of the sample text features and the sample timbre features, thereby obtaining the target text features in the sample speech features.

[0119] It is understandable that the spatial dimension of the sample text features is different from the spatial dimension of the results output by the forward reversible generation sub-model. Since the spatial dimension of the results output by the reversible generation sub-model includes the time dimension, while the spatial dimension of the sample text features does not include the time dimension, the spatial dimension of the sample text needs to be adjusted.

[0120] In some implementations, the sample speech features, the sample text features, and the sample timbre features are forwardly input into the reversible generation sub-model, and the target text features are output, including:

[0121] (1.1) Determine the time dimension corresponding to the sample speech feature, and adjust the dimension of the sample text feature according to the time dimension to obtain the target sample text feature;

[0122] (1.2) The sample speech features, target sample text features and sample timbre features are forward input into the reversible generation sub-model, and the target text features are output.

[0123] Sample speech features are usually a set of feature vectors extracted from speech signals, which may include speech spectrum information, prosody information, duration information, etc. Among them, the time dimension refers to the changing characteristics of speech signals on the time axis, such as speech duration, speaking speed, pauses, etc.

[0124] Determining the time dimension corresponding to the sample speech feature can be achieved by analyzing parameters such as the sampling rate and frame rate of the speech signal, or by performing time domain analysis on the speech signal, such as calculating features such as short-time energy and zero-crossing rate.

[0125] After obtaining the time dimension, the dimension of the sample text feature can be adjusted through the time dimension. The target sample text feature obtained after the dimension adjustment not only retains the semantic information of the sample text, but also matches the sample speech feature in the time dimension, providing better input data for subsequent model processing.

[0126] Finally, the sample speech features, target sample text features and sample timbre features are forward input into the reversible generation sub-model, and the target text features are output.

[0127] In some implementations, a first loss value between a target text feature and a sample text feature may be determined. Determining the first loss value between a target text feature and a sample text feature includes:

[0128] (2.1) Determine the KL divergence between the target sample text features and the target text features;

[0129] (2.2) The KL divergence is determined as the first loss value between the target text feature and the sample text feature.

[0130] Among them, the KL divergence between the target sample text feature and the target text feature is determined. KL divergence (Kullback-Leibler divergence), also known as relative entropy, is an indicator used to measure the difference between two probability distributions. The KL divergence is determined as the first loss value between the target text feature and the sample text feature. The target sample text feature can be used to constrain the target text feature extracted by the reversible generation sub-model in the sample speech feature to improve its extraction accuracy.

[0131] In step 140, the target text feature and the reference timbre feature are inversely input into the reversible generation sub-model, the target speech feature is output, and a second loss value between the target speech feature and the reference speech feature is determined.

[0132] From the above content, it can be seen that the reversible generative sub-model has a reversible characteristic, so the target text features and reference timbre features can be reversely input into the reversible generative sub-model to output the target speech features. The target speech features are the speech features predicted by the reversible generative sub-model, while the reference speech features are the real speech features. The second loss value between the two can be determined to measure the prediction accuracy of the reversible generative sub-model.

[0133] In step 150, a total loss value is determined according to the first loss value and the second loss value, and the model parameters of the reversible generation sub-model are adjusted based on the total loss value to iterate the reversible generation sub-model, and return to execute to obtain sample speech and sample timbre features and sample text features of the sample speech until the total loss value meets the preset loss condition, thereby obtaining the trained reversible generation sub-model.

[0134] After obtaining the first loss value and the second loss value, the first loss value and the second loss value can be weighted respectively, and then the weighted sum of the two can be obtained to obtain the total loss, and then determine whether the total loss value is less than the preset loss value. If not, the model parameters of the reversible generation sub-model are adjusted based on the total loss value to iterate the reversible generation sub-model. And return to execute to obtain the sample speech and the sample timbre features and sample text features of the sample speech, that is, return to step 110. Until the total loss value meets the preset loss condition, that is, the total loss value is less than the preset loss value, the trained reversible generation sub-model is obtained.

[0135] The trained speech conversion model includes a trained decoding model, a pre-trained timbre extraction model, a pre-trained text model, a trained reversible generation sub-model and a trained encoding sub-model.

[0136] See also Figure 4 , Figure 4 It is a flowchart of voice conversion provided in an embodiment of the present application.

[0137] In some embodiments, after obtaining the trained reversible generator model, the following steps are further included:

[0138] Step 201, obtaining a first speech to be converted and a first timbre feature and a first text feature of the first speech;

[0139] Step 202: input the first speech into the trained encoding sub-model, and output the first speech feature;

[0140] Step 203: forward input the first speech feature, the first timbre feature and the first text feature into the trained reversible generative sub-model, and output the first target text feature;

[0141] Step 204: reversely input the first target text feature and the preset timbre feature into the trained reversible generation sub-model to output the second speech feature;

[0142] Step 205: Input the second speech feature into the trained decoding sub-model, and output the second speech after performing timbre conversion on the first speech.

[0143] Steps 201 to 205 will be described in detail below.

[0144] In step 201, a first speech to be converted and a first timbre feature and a first text feature of the first speech are obtained.

[0145] The first speech to be converted may be a speech segment read from an audio file, or may be speech data collected in real time by a microphone.

[0146] The first speech to be converted can be input into a pre-trained timbre extraction model to output a first timbre feature, where the first timbre feature is used to describe a unique sound characteristic associated with a speaker in the speech.

[0147] The speech to be converted can be input into a text recognition model to output text, and then the text can be input into a pre-trained text model to output a first text feature of the first speech.

[0148] In step 202, the first speech is input into the trained encoding sub-model, and the first speech feature is output.

[0149] The trained encoding sub-model can convert the input first speech into a specific feature representation, namely the first speech feature. The first speech feature is a low-dimensional vector or a set of feature parameters, which can capture the key information in the speech signal, such as spectral features, prosodic features, speaker features, etc.

[0150] In step 203, the first speech feature, the first timbre feature and the first text feature are forwardly input into the trained reversible generative sub-model, and the first target text feature is output.

[0151] The trained reversible generative sub-model can decouple the timbre and text content of the first speech in the first speech feature based on the guidance of the first timbre feature and the first text feature, thereby obtaining the first target text feature, and the first target text feature is associated with the time dimension of the first speech feature.

[0152] In step 204, the first target text feature and the preset timbre feature are reversely input into the trained reversible generation sub-model to output the second speech feature.

[0153] Through the reversible characteristics of the trained reversible generative sub-model, the first target text feature and the preset timbre feature can be combined to obtain the generated second speech feature. The preset timbre feature is the timbre that the first speech is intended to generate.

[0154] In step 205, the second speech feature is input into the trained decoding sub-model, and the second speech is outputted after the timbre conversion of the first speech.

[0155] The task of the trained decoding sub-model is to generate the corresponding second speech according to the input second speech features. In this way, the timbre conversion of the first speech is realized, and the timbre of the first speech is converted to the preset timbre without changing the text content in the first speech.

[0156] It can be seen from steps 201 to 205 that the speech conversion model can be used to decouple the timbre and text of the first speech, and then combine the decoupled text with the preset timbre to generate a second speech with the preset timbre, and the timbre in the second speech is consistent.

[0157] As can be seen from the above, in the present application, by obtaining sample speech and sample timbre features and sample text features of the sample speech, reference speech features and reference timbre features of the reference speech are obtained; the sample speech is input into the trained encoding sub-model, and the sample speech features of the sample speech are output; the sample speech features, sample text features and sample timbre features are forward input into the reversible generation sub-model, the target text features are output, and a first loss value between the target text features and the sample text features is determined; the target text features and the reference timbre features are reversely input into the reversible generation sub-model, the target speech features are output, and a second loss value between the target speech features and the reference speech features is determined; the total loss value is determined according to the first loss value and the second loss value, and the model parameters of the reversible generation sub-model are adjusted based on the total loss value to iterate the reversible generation sub-model, and return to execute the acquisition of sample speech and sample timbre features and sample text features of the sample speech until the total loss value meets the preset loss condition, and the trained reversible generation sub-model is obtained, and the trained speech conversion model includes the trained reversible generation sub-model and the trained encoding sub-model.

[0158] In this way, the sample timbre features and sample text features of the sample speech are obtained, and the sample speech features, sample text features and sample timbre features are forwardly input into the reversible generation sub-model, and the sample timbre features and sample text features are used to guide the reversible generation sub-model to decouple the timbre and text of the sample speech features, accurately extract the target text features in the sample speech features, and then determine the first loss value between the target text features and the real sample text features, and measure the accuracy of the reversible generation sub-model in decoupling the timbre and text in the sample speech. The reversible generation sub-model has a reversible characteristic, and then the target text features and the reference timbre features are reversely input into the reversible generation sub-model, the target speech features are output, and the second loss value between the target speech features and the real reference speech features is determined, and the ability of the reversible generation sub-model to combine the reference timbre features and the target text features is measured by the second loss value, and finally the total loss value is determined according to the first loss value and the second loss value, and the model parameters of the reversible generation sub-model are adjusted based on the total loss value to iterate the reversible generation sub-model. The reversible generative sub-model trained in this way has the ability to decouple the timbre and text of the forward input speech features, and at the same time has the ability to combine text and other timbre to generate speech features. Therefore, compared with the related art of inputting speech posterior graphs and voiceprint features into the speech conversion model to generate new speech, the speech conversion model trained in this application can combine the text in the input speech with the timbre of other people to generate new speech, and the timbre consistency in the new speech is relatively high.

[0159] See also Figure 5 , Figure 51 is another flow chart of the speech conversion model training method provided in the embodiment of the present application. The speech conversion model training method may include the following steps:

[0160] Step 301: obtain training speech, input the training speech into the encoding sub-model, and output the training speech features of the training speech;

[0161] Step 302: input the training speech features into the decoding sub-model and output the target speech;

[0162] Step 303: determining the difference between the target speech and the training speech, and iteratively training the encoding sub-model and the decoding sub-model according to the difference to obtain a trained encoding sub-model and a trained decoding sub-model;

[0163] Step 304: obtaining a sample speech and a sample timbre feature and a sample text feature of the sample speech, and obtaining a reference speech feature and a reference timbre feature of the reference speech;

[0164] Step 305: input the sample speech into the trained encoding sub-model, and output the sample speech features of the sample speech;

[0165] Step 306: forward input the sample speech feature, the sample text feature and the sample timbre feature into the reversible generation sub-model, output the target text feature, and determine a first loss value between the target text feature and the sample text feature;

[0166] Step 307: reversely input the target text feature and the reference timbre feature into the reversible generation sub-model, output the target speech feature, and determine a second loss value between the target speech feature and the reference speech feature;

[0167] Step 308: determining a total loss value according to the first loss value and the second loss value, and adjusting the model parameters of the reversible generation sub-model based on the total loss value, so as to iterate the reversible generation sub-model, and returning to execute to obtain the sample speech and the sample timbre features and sample text features of the sample speech until the total loss value meets the preset loss condition, and obtaining the trained reversible generation sub-model, wherein the trained speech conversion model includes the trained reversible generation sub-model and the trained encoding sub-model;

[0168] Step 309: obtaining a first speech to be converted and a first timbre feature and a first text feature of the first speech;

[0169] Step 310: input the first speech into the trained encoding sub-model and output the first speech feature;

[0170] Step 311: forwardly input the first speech feature, the first timbre feature and the first text feature into the trained reversible generative sub-model, and output the first target text feature;

[0171] Step 312: reversely input the first target text feature and the preset timbre feature into the trained reversible generation sub-model to output the second speech feature;

[0172] Step 313: Input the second speech feature into the trained decoding sub-model, and output the second speech after performing timbre conversion on the first speech.

[0173] In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described in detail in a certain embodiment, please refer to the above-mentioned speech conversion model training method, which will not be repeated here.

[0174] See also Figure 6 , the embodiment of the present application also provides a speech conversion model training device, which can implement the above-mentioned speech conversion model training method. In the embodiment of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0175] The speech conversion model training device 400 comprises:

[0176] An acquisition module 420 is used to acquire a sample speech and a sample timbre feature and a sample text feature of the sample speech, and to acquire a reference speech feature and a reference timbre feature of a reference speech;

[0177] A first input module 430, used to input the sample speech into the trained encoding sub-model and output the sample speech features of the sample speech;

[0178] A second input module 440, for forward inputting the sample speech feature, the sample text feature and the sample timbre feature into the reversible generation sub-model, outputting the target text feature, and determining a first loss value between the target text feature and the sample text feature;

[0179] A third input module 450, for reversely inputting the target text feature and the reference timbre feature into the reversible generation sub-model, outputting the target speech feature, and determining a second loss value between the target speech feature and the reference speech feature;

[0180] The iterative module 460 is used to determine the total loss value according to the first loss value and the second loss value, and adjust the model parameters of the reversible generation sub-model based on the total loss value to iterate the reversible generation sub-model, and return to execute to obtain the sample speech and the sample timbre features and sample text features of the sample speech until the total loss value meets the preset loss condition, and obtain the trained reversible generation sub-model. The trained speech conversion model includes the trained reversible generation sub-model and the trained encoding sub-model.

[0181] In some implementations, the speech conversion model further includes a decoding sub-model, and the speech conversion model training apparatus 400 further includes a training module 410 for:

[0182] Acquire training speech, input the training speech into the encoding sub-model, and output training speech features of the training speech;

[0183] Input the training speech features into the decoding sub-model and output the target speech;

[0184] The difference between the target speech and the training speech is determined, and the encoding sub-model and the decoding sub-model are iteratively trained according to the difference to obtain a trained encoding sub-model and a trained decoding sub-model.

[0185] In some implementations, the training module 410 is further configured to:

[0186] Input the target speech into the trained discriminant network and output the target discriminant value corresponding to the target speech;

[0187] The square value of the difference between the target discrimination value and the preset value is determined, and the square value is determined as the difference between the target speech and the training speech.

[0188] In some implementations, the speech conversion model training apparatus 400 further includes a conversion module 470, which is used to:

[0189] Acquire a first speech to be converted and a first timbre feature and a first text feature of the first speech;

[0190] Inputting the first speech into the trained encoding sub-model and outputting the first speech feature;

[0191] Forward inputting the first speech feature, the first timbre feature and the first text feature into the trained reversible generative sub-model, and outputting the first target text feature;

[0192] Reversely inputting the first target text feature and the preset timbre feature into the trained reversible generative sub-model to output a second speech feature;

[0193] The second speech feature is input into the trained decoding sub-model, and the second speech is output after the timbre of the first speech is converted.

[0194] In some embodiments, the second input module 440 is used to:

[0195] Determine the time dimension corresponding to the sample speech feature, and adjust the dimension of the sample text feature according to the time dimension to obtain the target sample text feature;

[0196] The sample speech features, target sample text features and sample timbre features are forward input into the reversible generation sub-model, and the target text features are output.

[0197] In some embodiments, the second input module 440 is used to:

[0198] Determine the KL divergence between the target sample text features and the target text features;

[0199] The KL divergence is determined as the first loss value between the target text feature and the sample text feature.

[0200] In some implementations, the speech conversion model further includes a pre-trained timbre extraction model and a pre-trained text model, and the acquisition module 420 is used to:

[0201] Input the sample speech into the pre-trained timbre extraction model, and output the sample timbre features of the sample speech;

[0202] The sample speech is input into the pre-trained text model, and the sample text features of the sample speech are output.

[0203] In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described in detail in a certain embodiment, please refer to the above-mentioned speech conversion model training method, which will not be repeated here.

[0204] As can be seen from the above, the acquisition module 420 acquires the sample speech and the sample timbre features and sample text features of the sample speech, and acquires the reference speech features and reference timbre features of the reference speech; the first input module 430 inputs the sample speech into the trained encoding sub-model, and outputs the sample speech features of the sample speech; the second input module 440 forward inputs the sample speech features, the sample text features and the sample timbre features into the reversible generation sub-model, outputs the target text features, and determines the first loss value between the target text features and the sample text features; the third input module 450 inputs the target text features and the reference timbre features Reverse input is input into the reversible generative sub-model, the target speech feature is output, and a second loss value between the target speech feature and the reference speech feature is determined; the iteration module 460 determines the total loss value according to the first loss value and the second loss value, and adjusts the model parameters of the reversible generative sub-model based on the total loss value to iterate the reversible generative sub-model, and returns to execute to obtain the sample speech and the sample timbre feature and sample text feature of the sample speech until the total loss value meets the preset loss condition, and the trained reversible generative sub-model is obtained. The trained speech conversion model includes the trained reversible generative sub-model and the trained encoding sub-model.

[0205] In this way, the sample timbre features and sample text features of the sample speech are obtained, and the sample speech features, sample text features and sample timbre features are forwardly input into the reversible generation sub-model, and the sample timbre features and sample text features are used to guide the reversible generation sub-model to decouple the timbre and text of the sample speech features, accurately extract the target text features in the sample speech features, and then determine the first loss value between the target text features and the real sample text features, and measure the accuracy of the reversible generation sub-model in decoupling the timbre and text in the sample speech. The reversible generation sub-model has a reversible characteristic, and then the target text features and the reference timbre features are reversely input into the reversible generation sub-model, the target speech features are output, and the second loss value between the target speech features and the real reference speech features is determined, and the ability of the reversible generation sub-model to combine the reference timbre features and the target text features is measured by the second loss value, and finally the total loss value is determined according to the first loss value and the second loss value, and the model parameters of the reversible generation sub-model are adjusted based on the total loss value to iterate the reversible generation sub-model. The reversible generative sub-model trained in this way has the ability to decouple the timbre and text of the forward input speech features, and at the same time has the ability to combine text and other timbre to generate speech features. Therefore, compared with the related art of inputting speech posterior graphs and voiceprint features into the speech conversion model to generate new speech, the speech conversion model trained in this application can combine the text in the input speech with the timbre of other people to generate new speech, and the timbre consistency in the new speech is relatively high.

[0206] The embodiment of the present application also provides a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned speech conversion model training method when executing the computer program. The computer device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0207] See also Figure 6 , Figure 6 The hardware structure of a computer device according to another embodiment is shown, and the computer device includes:

[0208] The processor 501 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0209] The memory 502 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 502, and the processor 501 calls and executes the speech conversion model training method of the embodiment of this application;

[0210] Input / output interface 503, used to implement information input and output;

[0211] Communication interface 504, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0212] A bus 505 that transmits information between the various components of the device (e.g., the processor 501, the memory 502, the input / output interface 503, and the communication interface 504);

[0213] The processor 501 , the memory 502 , the input / output interface 503 and the communication interface 504 are connected to each other in communication within the device via the bus 505 .

[0214] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned speech conversion model training method is implemented.

[0215] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0216] The speech conversion model training method, device, computer equipment and storage medium proposed in the present application, wherein the speech conversion model includes an encoding sub-model and a reversible generation sub-model, and obtains reference speech features and reference timbre features of a reference speech by obtaining sample speech and sample timbre features and sample text features of the sample speech; the sample speech is input into the trained encoding sub-model, and the sample speech features of the sample speech are output; the sample speech features, sample text features and sample timbre features are forward input into the reversible generation sub-model, and the target text features are output, and a first loss value between the target text features and the sample text features is determined; the target The marked text features and the reference timbre features are reversely input into the reversible generative sub-model, the target speech features are output, and a second loss value between the target speech features and the reference speech features is determined; a total loss value is determined according to the first loss value and the second loss value, and the model parameters of the reversible generative sub-model are adjusted based on the total loss value to iterate the reversible generative sub-model, and return to execute to obtain the sample speech and the sample timbre features and sample text features of the sample speech until the total loss value meets the preset loss condition, and the trained reversible generative sub-model is obtained. The trained speech conversion model includes the trained reversible generative sub-model and the trained encoding sub-model.

[0217] In this way, the sample timbre features and sample text features of the sample speech are obtained, and the sample speech features, sample text features and sample timbre features are forwardly input into the reversible generation sub-model, and the sample timbre features and sample text features are used to guide the reversible generation sub-model to decouple the timbre and text of the sample speech features, accurately extract the target text features in the sample speech features, and then determine the first loss value between the target text features and the real sample text features, and measure the accuracy of the reversible generation sub-model in decoupling the timbre and text in the sample speech. The reversible generation sub-model has a reversible characteristic, and then the target text features and the reference timbre features are reversely input into the reversible generation sub-model, the target speech features are output, and the second loss value between the target speech features and the real reference speech features is determined, and the ability of the reversible generation sub-model to combine the reference timbre features and the target text features is measured by the second loss value, and finally the total loss value is determined according to the first loss value and the second loss value, and the model parameters of the reversible generation sub-model are adjusted based on the total loss value to iterate the reversible generation sub-model. The reversible generative sub-model trained in this way has the ability to decouple the timbre and text of the forward input speech features, and at the same time has the ability to combine text and other timbre to generate speech features. Therefore, compared with the related art of inputting speech posterior graphs and voiceprint features into the speech conversion model to generate new speech, the speech conversion model trained in this application can combine the text in the input speech with the timbre of other people to generate new speech, and the timbre consistency in the new speech is relatively high.

[0218] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0219] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0220] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0221] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0222] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0223] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0224] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0225] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0226] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0227] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0228] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A speech conversion model training method, characterized in that: The speech conversion model includes a coding sub-model and a reversible generation sub-model, and the method includes: Acquire a sample speech and a sample timbre feature and a sample text feature of the sample speech, and acquire a reference speech feature and a reference timbre feature of a reference speech; Inputting the sample speech into the trained encoding sub-model, and outputting the sample speech features of the sample speech; Forward inputting the sample speech feature, the sample text feature and the sample timbre feature into a reversible generation sub-model, outputting a target text feature, and determining a first loss value between the target text feature and the sample text feature; Reversely inputting the target text feature and the reference timbre feature into the reversible generative sub-model, outputting the target speech feature, and determining a second loss value between the target speech feature and the reference speech feature; A total loss value is determined according to the first loss value and the second loss value, and model parameters of the reversible generative sub-model are adjusted based on the total loss value to iterate the reversible generative sub-model, and return to execute to obtain sample speech and sample timbre features and sample text features of the sample speech until the total loss value meets a preset loss condition, thereby obtaining a trained reversible generative sub-model.

2. The speech conversion model training method according to claim 1, characterized in that: The speech conversion model further includes a decoding sub-model, and before inputting the sample speech into the trained encoding sub-model and outputting the speech features of the sample speech, further includes: Acquire training speech, input the training speech into the encoding sub-model, and output training speech features of the training speech; Inputting the training speech features into the decoding sub-model and outputting the target speech; The difference between the target speech and the training speech is determined, and the encoding sub-model and the decoding sub-model are iteratively trained according to the difference to obtain the trained encoding sub-model and the trained decoding sub-model.

3. The speech conversion model training method according to claim 2, characterized in that: The determining the difference between the target speech and the training speech comprises: Inputting the target speech into a trained discriminant network, and outputting a target discriminant value corresponding to the target speech; The square value of the difference between the target discrimination value and the preset value is determined, and the square value is determined as the difference between the target speech and the training speech.

4. The speech conversion model training method according to claim 2, characterized in that: After obtaining the trained reversible generator model, the method further includes: Acquire a first speech to be converted and a first timbre feature and a first text feature of the first speech; Inputting the first speech into the trained encoding sub-model and outputting a first speech feature; Forward inputting the first speech feature, the first timbre feature and the first text feature into the trained reversible generative sub-model, and outputting a first target text feature; Reversely inputting the first target text feature and the preset timbre feature into the trained reversible generation sub-model to output a second speech feature; The second speech feature is input into the trained decoding sub-model, and a second speech is output after the timbre of the first speech is converted.

5. The speech conversion model training method according to claim 1, characterized in that: The forward inputting of the sample speech feature, the sample text feature and the sample timbre feature into the reversible generation sub-model and outputting the target text feature comprises: Determine the time dimension corresponding to the sample speech feature, and adjust the dimension of the sample text feature according to the time dimension to obtain a target sample text feature; The sample speech feature, the target sample text feature and the sample timbre feature are forwardly input into the reversible generation sub-model, and the target text feature is output.

6. The speech conversion model training method according to claim 5, characterized in that: The determining of a first loss value between the target text feature and the sample text feature includes: Determining the KL divergence between the target sample text feature and the target text feature; The KL divergence is determined as a first loss value between the target text feature and the sample text feature.

7. The speech conversion model training method according to claim 1, characterized in that: The speech conversion model also includes a pre-trained timbre extraction model and a pre-trained text model. The acquisition of sample speech and sample timbre features and sample text features of the sample speech includes: Inputting the sample speech into the pre-trained timbre extraction model, and outputting the sample timbre features of the sample speech; The sample speech is input into the pre-trained text model, and sample text features of the sample speech are output.

8. A speech conversion model training device, characterized in that: The speech conversion model includes a coding sub-model and a reversible generation sub-model, and the device includes: An acquisition module, used to acquire a sample speech and a sample timbre feature and a sample text feature of the sample speech, and to acquire a reference speech feature and a reference timbre feature of a reference speech; A first input module, used for inputting the sample speech into the trained encoding sub-model, and outputting the sample speech features of the sample speech; A second input module, used for forward inputting the sample speech feature, the sample text feature and the sample timbre feature into the reversible generation sub-model, outputting the target text feature, and determining a first loss value between the target text feature and the sample text feature; A third input module, used for reversely inputting the target text feature and the reference timbre feature into the reversible generation sub-model, outputting the target speech feature, and determining a second loss value between the target speech feature and the reference speech feature; An iteration module is used to determine a total loss value according to the first loss value and the second loss value, and adjust the model parameters of the reversible generation sub-model based on the total loss value to iterate the reversible generation sub-model, and return to execute to obtain sample speech and sample timbre features and sample text features of the sample speech until the total loss value meets a preset loss condition, thereby obtaining a trained reversible generation sub-model.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, which are suitable for loading by a processor to execute the speech conversion model training method according to any one of claims 1 to 7.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the speech conversion model training method described in any one of claims 1 to 7 is implemented.

Images