Tone conversion method, electronic device, and computer-readable storage medium

Abstract

The present application relates to the field of artificial intelligence technology, and in particular to a timbre conversion method, electronic device, and computer-readable storage medium. The timbre conversion method in the embodiment of the present application, after obtaining the source speech and target timbre features, pre-processes the source speech to obtain the original mel spectrum, and then inputs the original mel spectrum into the speech encoder of the speech conversion model for feature extraction to obtain the semantic content features of the source speech, and then inputs the semantic content features and the target timbre features into the speech decoder of the speech conversion model to obtain the target mel spectrum, and finally performs speech conversion on the target mel spectrum to obtain the target speech. The present application first extracts the semantic content features of the source speech, and then performs feature decoding on the semantic content features with the target timbre features, and finally obtains the target speech according to the target mel spectrum, and improves the conversion effect in the timbre conversion process with the efficient speech conversion capability of the speech conversion model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a timbre conversion method, an electronic device, and a computer-readable storage medium. Background Art

[0002] Voice Conversion (VC) is a technology that changes the timbre of speech, converting the timbre of source audio to that of target audio while preserving the content of the speech. VC technology has been widely used in various applications, such as enriching voice databases, assisting pronunciation, and verifying speakers. However, related technologies often produce poor speech quality. Therefore, improving the timbre conversion process has become a pressing issue in the industry. Summary of the Invention

[0003] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a timbre conversion method, an electronic device, and a computer-readable storage medium, which can improve the timbre conversion effect during the timbre conversion process.

[0004] The timbre conversion method according to the source aspect embodiment of the present application includes:

[0005] Obtain source speech and target timbre features;

[0006] Preprocessing the source speech to obtain an original mel spectrum;

[0007] Inputting the original Mel spectrum into the speech encoder of the speech conversion model for feature extraction to obtain the semantic content features of the source speech;

[0008] Inputting the semantic content feature and the target timbre feature into a speech decoder of the speech conversion model, so that the speech decoder performs feature decoding on the semantic content feature based on the target timbre feature to obtain a target mel spectrum;

[0009] Perform speech conversion on the target mel spectrum to obtain the target speech.

[0010] According to some embodiments of the present application, before inputting the original mel spectrum into a speech encoder of a speech conversion model for feature extraction, the method further includes:

[0011] Acquire a training audio set, wherein the training audio set includes training speech formed based on multiple timbres;

[0012] Generating a plurality of training mel spectrograms according to each of the training speech in the training audio set;

[0013] The plurality of training mel spectrograms are input into a basic recognition model for optimization training to obtain the speech conversion model.

[0014] According to some embodiments of the present application, inputting the plurality of training mel spectrograms into a basic recognition model for optimization training to obtain the speech conversion model includes:

[0015] Inputting the training mel spectrum into the basic recognition model for iterative training;

[0016] After each round of iterative training, the model loss function of the basic recognition model is calculated and the basic recognition model is updated;

[0017] When the output value of the model loss function converges to a constant value, the trained speech conversion model is obtained.

[0018] According to some embodiments of the present application, inputting the training mel spectrum into the basic recognition model for iterative training includes:

[0019] Inputting the training mel spectrum into the basic encoder of the basic recognition model to perform feature extraction to obtain training content features and training timbre features;

[0020] The training content features and the training timbre features are input into a basic decoder of the basic recognition model, so that the basic decoder performs feature decoding on the training content features based on the training timbre features to obtain a test mel spectrum.

[0021] According to some embodiments of the present application, after each round of iterative training, calculating the model loss function of the basic recognition model and updating the basic recognition model include:

[0022] After each round of iterative training, calculating the model loss function based on the training Mel spectrum and the test Mel spectrum;

[0023] After obtaining the model loss function, the parameters of the basic recognition model are adjusted for updating.

[0024] According to some embodiments of the present application, inputting the original mel spectrum into a speech encoder of a speech conversion model for feature extraction to obtain semantic content features of the source speech includes:

[0025] Inputting the original Mel spectrum into the first segmented convolutional layer of the speech encoder for dimensionality reduction processing to form an original Mel spectrum feature vector;

[0026] Based on the first feature processing network, decoupling processing is performed on the original Mel spectrum feature vector to obtain a content feature vector;

[0027] The content feature vectors are integrated via a first fully connected layer to obtain the semantic content features.

[0028] According to some embodiments of the present application, inputting the semantic content feature and the target timbre feature into a speech decoder of the speech conversion model, so that the speech decoder performs feature decoding on the semantic content feature based on the target timbre feature to obtain a target mel-spectrogram, includes:

[0029] Inputting the semantic content features and the target timbre features into the second segmented convolutional layer of the speech decoder, so that the second segmented convolutional layer performs vectorization processing on the semantic content features and the target timbre features to obtain the content feature vector and the target timbre feature vector;

[0030] Inputting the content feature vector and the target timbre feature vector into a second feature processing network of the speech decoder, so that the second feature processing network couples the target timbre feature vector with the content feature vector to form a target mel-spectrogram feature vector;

[0031] The target mel spectrum feature vector is integrated through the second fully connected layer of the speech decoder to obtain the target mel spectrum.

[0032] According to some embodiments of the present application, inputting the original mel spectrum into a speech encoder of a speech conversion model for feature extraction to obtain semantic content features of the source speech further includes:

[0033] Based on the first feature processing network, decoupling processing is performed on the original Mel spectrum feature vector to obtain a source timbre feature vector;

[0034] The source timbre feature vectors are integrated via the first fully connected layer to obtain source timbre features.

[0035] In a second aspect, an embodiment of the present application provides an electronic device comprising: a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the timbre conversion method as described in any one of the embodiments of the source aspect of the present application.

[0036] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the timbre conversion method as described in any one of the embodiments of the source aspect of the present application.

[0037] The timbre conversion method, electronic device, and computer-readable storage medium according to the embodiments of the present application have at least the following beneficial effects:

[0038] The timbre conversion method in the embodiment of the present application, after obtaining the source speech and target timbre features, pre-processes the source speech to obtain the original mel spectrum, further inputs the original mel spectrum into the speech encoder of the speech conversion model for feature extraction to obtain the semantic content features of the source speech, and then inputs the semantic content features and the target timbre features into the speech decoder of the speech conversion model, so that the speech decoder performs feature decoding on the semantic content features based on the target timbre features to obtain the target mel spectrum, and finally performs speech conversion on the target mel spectrum to obtain the target speech. The present application first extracts the semantic content features of the source speech through the speech encoder of the speech conversion model, and then performs feature decoding on the semantic content features with the target timbre features through the speech decoder of the speech conversion model, and finally obtains the target speech according to the target mel spectrum, thereby improving the conversion effect in the timbre conversion process with the efficient speech conversion capability of the speech conversion model.

[0039] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become obvious from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0041] Figure 1 A schematic diagram of the flow chart of the timbre conversion method provided in an embodiment of the present application;

[0042] Figure 2 A schematic flow chart of another timbre conversion method provided in an embodiment of the present application;

[0043] Figure 3 A schematic flow chart of another timbre conversion method provided in an embodiment of the present application;

[0044] Figure 4 A schematic flow chart of another timbre conversion method provided in an embodiment of the present application;

[0045] Figure 5 A schematic flow chart of another timbre conversion method provided in an embodiment of the present application;

[0046] Figure 6 A schematic flow chart of another timbre conversion method provided in an embodiment of the present application;

[0047] Figure 7 A schematic flow chart of another timbre conversion method provided in an embodiment of the present application;

[0048] Figure 8 A schematic flow chart of another timbre conversion method provided in an embodiment of the present application;

[0049] Figure 9 Schematic diagram of an electronic device for performing a timbre conversion method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The following describes in detail embodiments of the present application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and are not to be construed as limiting the present application.

[0051] In the description of this application, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. If "source" or "second" is used in the description, it is only used to distinguish technical features and is not to be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0052] In the description of this application, it should be understood that descriptions involving orientations, such as up, down, left, right, front, and back, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on this application.

[0053] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0054] In the description of this application, it should be noted that, unless otherwise explicitly defined, terms such as "set," "install," and "connect" should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meanings of the above terms in this application based on the specific content of the technical solution. In addition, the identification of specific steps below does not represent a limitation on the order of steps and execution logic. The execution order and execution logic between each step should be understood and inferred with reference to the content described in the embodiments.

[0055] Artificial Intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and create new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses theories, methods, technologies, and application systems that use digital computers or digital computer-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0056] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0057] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0058] Voice Conversion (VC) is a technology that changes the timbre of speech, converting the timbre of source audio to that of target audio while preserving the content of the speech. VC technology has been widely used in various applications, such as enriching voice databases, assisting pronunciation, and verifying speakers. However, related technologies often produce poor speech quality. Therefore, improving the timbre conversion process has become a pressing issue in the industry.

[0059] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a timbre conversion method, an electronic device, and a computer-readable storage medium, which can improve the timbre conversion effect during the timbre conversion process.

[0060] Further description will be given below with reference to the accompanying drawings.

[0061] Reference Figure 1 According to the source aspect of the present application, the timbre conversion method includes:

[0062] Step S101, obtaining source speech and target timbre features;

[0063] According to some embodiments provided by the present application, the source voice refers to the voice information that needs to be converted, and the target voice refers to the voice information that serves as the target of the timbre conversion. It should be noted that the source voice is provided by the source speaker, and the timbre characteristics of the source voice are the source timbre characteristics. The target voice is provided by the target speaker, and the timbre characteristics of the target voice are the target timbre characteristics. The timbre conversion method of the embodiment of the present application is used to realize the conversion of the source timbre characteristics into the target timbre characteristics while keeping the semantic content of the source voice unchanged, and obtaining the source voice and target timbre characteristics is the basis of timbre conversion. In some embodiments, there are various ways to obtain the source voice and target timbre characteristics, which can be obtained by querying from a preset voice library, downloading from the Internet, or even by collecting the voices of the source speaker and the target speaker. It should be understood that there are various ways to obtain the source voice and target timbre characteristics, including but not limited to the specific embodiments listed above.

[0064] Step S102, preprocessing the source speech to obtain the original mel spectrum;

[0065] It's important to note that in audio and speech signal processing, we need to convert the signal into a corresponding spectrogram, using the data on the spectrogram as the signal's features. A typical spectrogram has a linear frequency distribution. However, the human ear is sensitive to changes in low-frequency bands and insensitive to changes in high-frequency bands. Therefore, as the frequency of a sound increases linearly, the higher the frequency, the harder it is to hear the difference. This linearly distributed spectrogram cannot reflect the true human auditory perception process in feature extraction. Therefore, instead of using a linear spectrum, a logarithmic spectrum is used to represent the human auditory perception process. This is how the Mel spectrogram (Mel spectrogram) came into being. A Mel spectrogram contains three major characteristics: time-frequency information, perceptually relevant amplitude information, and perceptually relevant frequency information. The semantic content and timbre characteristics of an audio segment are often coupled to these three characteristics: time-frequency information, perceptually relevant amplitude information, and perceptually relevant frequency information. According to some embodiments provided in the present application, the original mel spectrum is the mel spectrum reflecting the source speech, and in some more specific embodiments, the source speech is preprocessed by first performing a short-time Fourier transform (STFT) on the acquired source speech, that is, the sound signal of a section of source speech is framed and windowed, and then a Fourier transform is performed on each frame. The results of each frame are then stacked along another dimension to form a source spectrogram corresponding to the source speech. Furthermore, the source spectrogram is logarithmized by a Mel-scale filter bank to form the original mel spectrum.

[0066] Step S103: inputting the original mel spectrum into the speech encoder of the speech conversion model to extract features and obtain semantic content features of the source speech;

[0067] According to some embodiments provided by this application, the speech conversion model refers to an artificial intelligence model for realizing speech timbre conversion. It should be noted that in some embodiments provided by this application, the speech conversion model includes a speech encoder and a speech decoder. It should be understood that the speech encoder is a series of convolutional networks, which are mainly composed of a series of processing layers such as convolutional layers, pooling layers, and fully connected layers. Among them, the convolutional layer is responsible for obtaining local features of the image, the pooling layer downsamples the image and transmits the scale-invariant features to the next layer, and the main function of the fully connected layer is to map the feature space calculated by the previous layer (convolution, pooling, etc.) to the sample label space, that is, to integrate the feature representation into a value. Its advantage is that it reduces the influence of the feature position on the conversion result and improves the robustness of the entire network. In summary, the speech encoder classifies and analyzes the original Mel spectrum obtained to obtain audio data related to the original Mel spectrum (such as semantic content, timbre features), and the speech decoder is used to restore the Mel spectrum corresponding to the audio data after obtaining these audio data. In some embodiments of the present application, when a portion of the original mel spectrum is first encoded by the encoder to extract the semantic content features therein, and then the semantic content features are restored to the mel spectrum form by the decoder, if the semantic content loss in the two mel spectrums before and after processing is less, it means that the encoder in the speech conversion model can more accurately extract the semantic content features. Similarly, if the semantic content loss in the two mel spectrums before and after processing is greater, it means that the encoder in the speech conversion model is less able to accurately extract the semantic content features.

[0068] Step S104: inputting the semantic content features and the target timbre features into a speech decoder of a speech conversion model, so that the speech decoder decodes the semantic content features based on the target timbre features to obtain a target mel-spectrogram;

[0069] According to some embodiments provided in this application, the target speech refers to the speech information that serves as the target of timbre conversion. The target speech is provided by the target speaker, and the timbre characteristics of the target speech are the target timbre characteristics. It should be noted that the speech encoder of the speech conversion model can decouple and separate the semantic content features in the original mel spectrum from the source timbre features, while the speech decoder of the speech conversion model can perform feature decoding on the separated speech content features based on the target timbre features, thereby obtaining the target mel spectrum, wherein the target mel spectrum includes the target mel spectrum features formed by coupling the speech content features and the target timbre features. Therefore, by further performing speech conversion on the target mel spectrum, the target speech can be obtained.

[0070] Step S105: performing speech conversion on the target mel spectrum to obtain the target speech.

[0071] According to some embodiments provided in the present application, to convert a target mel spectrum into speech, the target mel spectrum needs to be input into a vocoder first, and the converted target speech can be obtained through processing by the vocoder. It should be noted that vocoders, as an important component of speech synthesis, can include, but are not limited to, the following categories: parametric vocoders based on pure signal processing, that is, parametric vocoders based on traditional signal processing algorithms are the process of decoding acoustic features into waveforms using traditional mathematical modeling or inverse transformation; vocoders built based on autoregressive neural networks, such as WaveNet, which is a completely probabilistic and autoregressive model. The predicted distribution of each audio sample is conditioned on all previous samples, and the mapping to the sample value sequence can be directly learned. Therefore, applying WaveNet to vocoders for speech synthesis has a good synthesis effect; vocoders built based on non-autoregressive neural networks. These vocoders are usually several orders of magnitude faster than autoregressive models because they have a highly parallelizable network structure and make full use of modern deep learning hardware (GPUs and TPUs). For example, a trained autoregressive structure vocoder is distilled into a convolution-based non-autoregressive structure vocoder, or a network WaveGlow that relies on streaming to synthesize high-quality speech from Mel-spectrograms is used to build a vocoder.

[0072] The timbre conversion method in the embodiment of the present application, after obtaining the source speech and target timbre features, pre-processes the source speech to obtain the original mel spectrum, further inputs the original mel spectrum into the speech encoder of the speech conversion model for feature extraction to obtain the semantic content features of the source speech, and then inputs the semantic content features and the target timbre features into the speech decoder of the speech conversion model, so that the speech decoder performs feature decoding on the semantic content features based on the target timbre features to obtain the target mel spectrum, and finally performs speech conversion on the target mel spectrum to obtain the target speech. The present application first extracts the semantic content features of the source speech through the speech encoder of the speech conversion model, and then performs feature decoding on the semantic content features with the target timbre features through the speech decoder of the speech conversion model, and finally obtains the target speech according to the target mel spectrum, thereby improving the conversion effect in the timbre conversion process with the efficient speech conversion capability of the speech conversion model.

[0073] Reference Figure 2 According to some embodiments of the present application, before inputting the original mel spectrum into the speech encoder of the speech conversion model for feature extraction, the method further includes:

[0074] Step S201: obtaining a training audio set, where the training audio set includes training speech based on multiple timbres;

[0075] According to some embodiments provided by this application, the training audio set can be divided into two categories: a training audio set based on parallel corpus and a training audio set based on non-parallel corpus, wherein based on parallel corpus means that in the training data set, each audio of speaker A has a corresponding audio of speaker B, and the text content of the two is the same; based on non-parallel corpus means that there is no requirement for the content of the audio, as long as each speaker has a certain number of audios. It should be clarified that in some embodiments of this application, when a part of the original mel spectrum is first encoded by the encoder to extract the semantic content features therein, and then the semantic content features are restored to the mel spectrum form by the decoder, the less the semantic content loss in the two mel spectra before and after processing, the more accurately the encoder in the speech conversion model can extract the semantic content features. Similarly, the more the semantic content loss in the two mel spectra before and after processing, the less accurately the encoder in the speech conversion model can extract the semantic content features. Therefore, in order to optimize the ability of the basic recognition model to accurately extract semantic content features, in some more preferred embodiments of this application, a training audio set based on non-parallel corpus can be used to train the basic recognition model. It should be understood that there are various ways to obtain a training audio set based on non-parallel corpora. These can be downloaded from a pre-set audio database, or conventional audio datasets such as VoxForge and TED-LIUM can be used to optimize and train the basic recognition model. Methods for obtaining a training audio set based on non-parallel corpora may include, but are not limited to, the specific embodiments listed above. In some embodiments, the more comprehensive the human speech audio types included in the training audio set, the more suitable the corresponding training audio set is for optimizing and training the basic recognition model.

[0076] Step S202, generating a plurality of training mel-spectrograms based on the training speech in the training audio set;

[0077] According to some embodiments provided herein, the training audio set includes various types of human speech audio, i.e., individual training voices. It should be noted that each training voice, after preprocessing, generates a corresponding training mel-spectrogram. Therefore, each training voice in the training audio set is preprocessed sequentially to generate multiple training mel-spectrograms.

[0078] Step S203: Input the multiple training mel spectrograms into the basic recognition model for optimization training to obtain a speech conversion model.

[0079] It should be noted that optimization training refers to training that optimizes the ability of the basic recognition model to extract semantic content from the mel spectrum and then restore it. It should be clarified that the link of extracting semantic content is performed in the basic encoder of the basic recognition model, and the link of restoring the mel spectrum according to the semantic content is performed in the basic decoder of the basic recognition model. According to some embodiments provided by the present application, each time a training mel spectrum is input into the basic recognition model for optimization training, the ability of the basic recognition model to extract semantic content from a training mel spectrum and then restore it can be improved. Therefore, inputting multiple training mel spectrums into the basic recognition model for optimization training can improve the ability of the basic recognition model to extract semantic content from various types of training mel spectrums and then restore them.

[0080] Reference Figure 3 According to some embodiments of the present application, multiple training mel spectrograms are input into a basic recognition model for optimization training to obtain a speech conversion model, including:

[0081] Step S301: input the training mel spectrum into the basic recognition model for iterative training;

[0082] It should be noted that the purpose of iterative training is to gradually improve the ability of the basic recognition model to extract semantic content features and then restore them through several rounds of training to extract semantic content features and then restore them. According to some embodiments provided in this application, the process of training the basic recognition model to obtain a speech conversion model includes: inputting the training mel spectrum into the basic recognition model for iterative training, and after each round of iterative training, calculating the model loss function of the basic recognition model and updating the basic recognition model. Among them, the basic recognition model can be obtained by pre-training a preset model that does not have the ability to extract semantic content features and then restore them, or a preset model that already has the ability to preliminarily extract semantic content features and then restore them can be selected.

[0083] Step S302: After each round of iterative training, the model loss function of the basic recognition model is calculated and the basic recognition model is updated;

[0084] It should be noted that the purpose of iterative training is to continuously optimize the ability of the basic recognition model to extract semantic content features and then restore them. After each round of iterative training, the model loss function of the basic recognition model needs to be calculated to clarify that the ability of the basic recognition model to extract semantic content features and then restore them during the optimization training process is gradually improving. It should be noted that during the optimization training process, the basic recognition model needs to be updated after each round of iterative training so that the basic recognition model can perform better in extracting semantic content features and then restoring them in the next round of iterative training. During iterative training, the basic recognition model is updated mainly by adjusting the parameters of the basic recognition model. The parameters of the basic recognition model are the internal parameters of the basic recognition model that are related to the ability to extract semantic content features and then restore them.

[0085] According to some more specific embodiments provided in this application, during the training process of the basic recognition model, the model loss function of the basic recognition model is varied, and can be a variational lower bound function (VLBF), a KL divergence of the basic recognition model, or a reconstruction loss function (Reconstruction Loss) of the basic recognition model. It should be understood that the model loss function for calculating the basic recognition model may include, but is not limited to, the specific embodiments listed above.

[0086] Step S303: When the output value of the model loss function converges to a constant value, a trained speech conversion model is obtained.

[0087] It should be noted that when the output value of the model loss function converges to a fixed value, the iterative training can be stopped and the trained speech conversion model can be obtained, wherein the fixed value refers to: after several rounds of adjustment of the internal parameters of the basic recognition model, the model loss function output of the basic recognition model converges to a value. In some embodiments provided in this application, the output value of the loss function of the basic recognition model will be stable within a certain error range. For example, if the output value of the loss function of the basic recognition model fluctuates within the range of 6% to 8%, the fixed value can be considered to be 7%. It should be understood that the fixed value is not an exact and unchanging value, but a value that changes with the training conditions. When the output value of the loss function of the basic recognition model converges to a fixed value, it is determined that the optimization training has achieved a better effect, and the iterative training can be stopped. The basic recognition model obtained after stopping the iterative training is the speech conversion model. The speech conversion model obtained through the optimization training will have a more accurate ability to extract semantic content features from the mel spectrum and then restore them. According to some embodiments provided in the present application, as iterative training proceeds, the ability of the basic recognition model to extract semantic content features from the mel spectrum and then restore them will be improved. The step of extracting semantic content features is performed in the basic encoder of the basic recognition model, and the step of restoring the mel spectrum according to the semantic content features is performed in the basic decoder of the basic recognition model. Therefore, the underlying reason for the optimization of the basic recognition model is that the ability of the basic encoder to extract semantic content features has been optimized, and the ability of the basic decoder to restore the mel spectrum according to the semantic content features has been optimized.

[0088] Reference Figure 4 According to some embodiments of the present application, the training mel spectrum is input into the basic recognition model for iterative training, including:

[0089] Step S401: Input the training mel spectrum into the basic encoder of the basic recognition model to extract features, thereby obtaining training content features and training timbre features;

[0090] According to some embodiments provided herein, after a training mel-spectrogram is input into a base encoder of a base recognition model, the base encoder of the base recognition model then decouples and separates the training mel-spectrogram, thereby obtaining independent training content features and training timbre features. It should be understood that the training content features are used to characterize the semantic features corresponding to the training mel-spectrogram, while the training timbre features are used to characterize the timbre features corresponding to the training mel-spectrogram.

[0091] Step S402: input the training content features and the training timbre features into a basic decoder of a basic recognition model, so that the basic decoder performs feature decoding on the training content features based on the training timbre features to obtain a test mel spectrum.

[0092] It should be noted that optimization training refers to training that optimizes the ability of the basic recognition model to extract training content features from the mel spectrum and then restore them. It should be clarified that the step of extracting training content features is performed in the basic encoder of the basic recognition model, and the step of restoring the mel spectrum according to the training content features is performed in the basic decoder of the basic recognition model. According to some embodiments provided in the present application, each time a training mel spectrum is input into the basic recognition model for optimization training, the ability of the basic recognition model to extract training content features from a training mel spectrum and then restore it can be improved. Therefore, inputting multiple training mel spectrums into the basic recognition model for optimization training can improve the ability of the basic recognition model to extract training content features from various types of training mel spectrums and then restore them.

[0093] It should be understood that the reason why the basic decoder of the basic recognition model decodes the training content features based on the training timbre features is that the core purpose of optimization training is to train the basic encoder's ability to extract and restore the training content features: the stronger the basic encoder's ability to extract and restore the training content features, the more accurate the training content features extracted from various types of training mel-spectra. Correspondingly, the more accurate the training timbre features extracted by the basic encoder will be. In other words, the basic encoder decouples the training content features and training timbre features from the training mel-spectra. The clearer the division between the training content features and the training timbre features, the stronger the basic encoder's ability to extract the training content features. Similarly, the stronger the basic encoder's ability to extract and restore the training content features, the closer the test mel-spectra restored based on the training content features will be to the training mel-spectra. In other words, the closer the test mel-spectra formed by the basic decoder recouples the decoupled training content features and training timbre features to the training mel-spectra, the stronger the basic decoder's ability to restore the training content features.

[0094] According to some embodiments provided in the present application, the speech conversion model obtained after optimization training has a speech encoder that can clearly decouple and separate the semantic content features and source timbre features from the original mel spectrum, and a speech decoder that can perform feature decoding on the separated semantic content features based on the target timbre features, couple the target timbre features of the target speaker with the semantic content features of the original mel spectrum to obtain the target mel spectrum, and then perform speech conversion on the target mel spectrum to finally obtain the target speech.

[0095] Reference Figure 5 According to some embodiments of the present application, after each round of iterative training, the model loss function of the basic recognition model is calculated and the basic recognition model is updated, including:

[0096] Step S501: After each round of iterative training, the model loss function is calculated based on the training mel spectrum and the test mel spectrum;

[0097] Step S502: After obtaining the model loss function, adjust the parameters of the basic recognition model for updating.

[0098] According to some more specific embodiments provided in the present application, during the training process of the basic recognition model, the training mel spectrum input to the basic recognition model is called the training mel spectrum X, and the mel spectrum output from the basic recognition model is called the test mel spectrum Y, wherein the model loss function of the basic recognition model can be a variational lower bound function (VLBF), and the variational lower bound function L(X t ,S X ; θ, δ) is as follows:

[0099] L(X t ,S x ;θ,δ)=-D KL (t) +E Rec (t)

[0100] Among them, D KL (t) KL divergence, E Rec (t) is the reconstruction loss function of the basic recognition model, t is the number of subframes of the Mel spectrum training (t = 1, 2, ..., T), T is the total number of frames of the Mel spectrum training, and θ and δ are the internal parameters of the basic recognition model.

[0101] In some more specific embodiments, D KL (t) 、E Rec (t) They are:

[0102]

[0103] Among them, q θ (C t |X t ) represents the mel spectrum X trained based on the t-th frame t The semantic content feature C t The conditional probability distribution of N is Gaussian, and the mean and standard deviation are the output of the basic encoder E θ (X t );q δ (Y t |C t ) indicates that the source timbre characteristics Sx and the semantic content feature C of the tth frame t The output of the t-th frame test Mel spectrum Y t The conditional probability distribution of N is Gaussian with a standard deviation of 1 and a mean of the output G of the basic decoder. δ (C t , S x ).

[0104] In the aforementioned L(X t ,S X ;θ,δ)=-D KL (t) +E Rec (t) Based on this, the training goal of the basic recognition model for the model loss function is to maximize VLBF, that is, to solve VLBF according to the following analytical formula Max :

[0105]

[0106] It should be understood that the model loss function for calculating the basic recognition model may include, but is not limited to, the specific embodiments listed above.

[0107] Reference Figure 6 According to some embodiments of the present application, the original mel spectrum is input into the speech encoder of the speech conversion model for feature extraction to obtain the semantic content features of the source speech, including:

[0108] Step S601: inputting the original Mel spectrum into the first segmented convolutional layer of the speech encoder for dimensionality reduction processing to form an original Mel spectrum feature vector;

[0109] Step S602: Decoupling the original mel spectrum feature vector based on the first feature processing network to obtain a content feature vector;

[0110] Step S603: integrating the content feature vectors via the first fully connected layer to obtain semantic content features.

[0111] According to some embodiments provided by the present application, the speech conversion model can use a fully convolutional network (FCN) to adopt a convolutional neural network to realize the transformation from the source timbre feature to the target timbre feature. It should be noted that the fully convolutional network transforms the length of the semantic content feature of the intermediate layer back to the length of the original Mel spectrum feature at the time of input through the transposed convolution layer, so that the conversion result corresponds to the original Mel spectrum feature in the time dimension. However, when the fully convolutional network is used as the speech conversion model, the computing power resources occupied by the speech conversion model are large, and a long delay will be generated during the timbre conversion process. Therefore, in some more preferred embodiments of the present application, a segmented convolution, a feature processing network combined with a fully connected layer is used as the structure of the speech encoder to extract semantic content features and participate in the realization of timbre conversion.

[0112] According to some more specific embodiments provided in this application, the speech encoder of the speech conversion model includes but is not limited to the following three-layer structure:

[0113] The first segmented convolution layer. It should be noted that segmented convolution includes but is not limited to the following advantages: First, efficient training. Since the convolution is divided into multiple paths, each path can be processed separately by a different graphics processing unit (GPU), so the model can be trained on multiple GPUs in parallel. Compared with completing all tasks on a single GPU, such model parallelization on multiple GPUs allows the network to process more images at each step. Second, it saves computing power resources, and the model will be more efficient when running, that is, the model parameters will decrease as the number of filter groups increases. Third, using segmented convolution combined with a feature processing network to process timbre conversion can reduce the delay in the conversion process.

[0114] First feature processing network. It should be noted that the first feature processing network is used to decouple the original Mel spectrum feature vector to obtain the content feature vector. According to some embodiments provided in the present application, there are multiple options for the first feature processing network, including but not limited to: using a recurrent neural network (RNN) as the first feature processing network for extracting the content feature vector, using a long short-term memory network (LSTM) as the first feature processing network for extracting the content feature vector, and using a gated recurrent unit structure (GRU) or other neural network models as the first feature processing network for extracting the content feature vector. It should be noted that ordinary RNNs have the problem of gradient protection or disappearance, so LSTM can maintain the long-term existence of the gradient by introducing a linear self-circulating unit. GRU is one of the RNNs. Like LSTM, it is proposed to solve problems such as long-term memory and gradient in back propagation. Since LSTM has certain advantages in sequence modeling problems, has long-term memory function, is simple to implement, and solves the problems of gradient vanishing and gradient exploding in the long sequence training process, and LSTM can convert the original Mel spectrum frame by frame without causing time delay, in some more preferred embodiments of the present application, LSTM is selected as the first feature processing network, which together with the first segmented convolutional layer and the first fully connected layer forms a speech encoder of the speech conversion model to improve the conversion efficiency of timbre conversion under low computing power resources.

[0115] The first fully connected layer is used to map the feature space calculated by the previous layer to the sample label space, that is, to integrate the feature representation into a single value. This has the advantage of reducing the influence of the feature position on the conversion result and improving the robustness of the entire network. In the embodiment of the present application, the first fully connected layer can integrate the content feature vectors to obtain semantic content features. It should be understood that multiple content feature vectors can form semantic content features after integration.

[0116] Reference Figure 7 According to some embodiments of the present application, inputting semantic content features and target timbre features into a speech decoder of a speech conversion model so that the speech decoder decodes the semantic content features based on the target timbre features to obtain a target mel-spectrogram includes:

[0117] Step S701: inputting the semantic content features and the target timbre features into the second segmented convolutional layer of the speech decoder, so that the second segmented convolutional layer performs vectorization processing on the semantic content features and the target timbre features to obtain a content feature vector and a target timbre feature vector;

[0118] According to some embodiments provided by the present application, the target speech refers to the speech information that serves as the target of timbre conversion, the target speech is provided by the target speaker, and the timbre characteristics of the target speech are the target timbre characteristics. It should be noted that the reason for selecting the second segmented convolution layer is the same as the reason for selecting the first segmented convolution layer, both of which are based on the following advantages of segmented convolution: First, efficient training. Since the convolution is divided into multiple paths, each path can be processed separately by a different GPU, the model can be trained on multiple GPUs in parallel. Compared to completing all tasks on a single GPU, such model parallelization on multiple GPUs allows the network to process more images at each step. Second, it saves computing power resources, and the model will be more efficient when running, that is, the model parameters will decrease as the number of filter groups increases. Third, using segmented convolution in combination with a feature processing network to process timbre conversion can reduce the delay in the conversion process. However, the function of the second segmented convolutional layer is different from that of the first segmented convolutional layer. The first segmented convolutional layer is used to input the original mel-spectrogram into the first segmented convolutional layer of the speech encoder for dimensionality reduction processing to form the original mel-spectrogram feature vector, while the second segmented convolutional layer vectorizes the semantic content features and the target timbre features to obtain the content feature vector and the target timbre feature vector. It should be understood that the process of vectorizing the semantic content features can be regarded as the inverse process of integrating the content feature vector into the semantic content features. Its purpose is to convert the semantic content features back into vector form so that the content feature vector and the target timbre feature vector can be coupled in the subsequent steps to form the target mel-spectrogram feature vector.

[0119] Step S702: Input the content feature vector and the target timbre feature vector into the second feature processing network of the speech decoder, so that the second feature processing network couples the target timbre feature vector with the content feature vector to form a target mel-spectrogram feature vector.

[0120] It should be noted that the second feature processing network is used to couple the target timbre feature vector with the content feature vector to form a target mel spectrum feature vector. According to some embodiments provided by the present application, there are multiple options for the second feature processing network, including but not limited to: using a recurrent neural network (RNN) as the second feature processing network for extracting the content feature vector, using a long short-term memory network (LSTM) as the second feature processing network for extracting the content feature vector, using a gated recurrent unit structure (GRU) or other neural network models as the second feature processing network for extracting the content feature vector. Since LSTM has certain advantages in sequence modeling problems, has a long-term memory function, is simple to implement, solves the problems of gradient vanishing and gradient explosion in the long sequence training process, and LSTM can convert the original mel spectrum frame by frame without causing delay, in some preferred embodiments of the present application, LSTM is selected as the second feature processing network, and together with the second segmented convolutional layer and the second fully connected layer, forms a speech decoder of the speech conversion model to improve the conversion efficiency of timbre conversion under low computing resources.

[0121] Step S703 : The target mel-spectrogram feature vector is integrated through the second fully connected layer of the speech decoder to obtain a target mel-spectrogram.

[0122] It should be noted that the function of the second fully connected layer is different from that of the first fully connected layer. The first fully connected layer is used to integrate the content feature vectors to obtain semantic content features, while the second fully connected layer integrates the target mel-spectrogram feature vectors to obtain the target mel-spectrogram. It should be understood that the process of integrating the target mel-spectrogram feature vectors to obtain the target mel-spectrogram can be regarded as the inverse process of inputting the original mel-spectrogram into the first segmented convolutional layer of the speech encoder for dimensionality reduction processing to form the original mel-spectrogram feature vector. The purpose is that since the target mel-spectrogram vector couples the content feature vector and the target timbre feature vector, integrating the target mel-spectrogram feature vector to obtain the target mel-spectrogram and outputting it can provide the basis for obtaining the final converted target speech.

[0123] According to some embodiments provided in the present application, the speech conversion model obtained after optimization training has a speech encoder that can clearly decouple and separate the semantic content features and source timbre features from the original mel spectrum, and a speech decoder that can perform feature decoding on the separated semantic content features based on the target timbre features, couple the target timbre features of the target speaker with the semantic content features of the original mel spectrum to obtain the target mel spectrum, and then perform speech conversion on the target mel spectrum to finally obtain the target speech.

[0124] Reference Figure 8 According to some embodiments of the present application, the original mel spectrum is input into the speech encoder of the speech conversion model for feature extraction to obtain the semantic content features of the source speech, which also includes:

[0125] Step S801: Decoupling the original Mel spectrum feature vector based on the first feature processing network to obtain a source timbre feature vector;

[0126] Step S802 : integrating the source timbre feature vectors via the first fully connected layer to obtain source timbre features.

[0127] It should be noted that, in the process of inputting the original Mel spectrum into the speech encoder of the speech conversion model for feature extraction to obtain the semantic content features of the source speech, the original Mel spectrum feature vector is also decoupled based on the first feature processing network to obtain the source timbre feature vector, and the source timbre feature vector is integrated through the first fully connected layer to obtain the source timbre feature. It should be understood that the source speech is provided by the source speaker, and the timbre features of the source speech are the source timbre features. In some embodiments of the present application, the source timbre features can be combined with the timbre features of a third person to form a target timbre feature and input into the speech decoder of the speech conversion model, and then the semantic content features are decoded based on the target timbre features. In other embodiments of the present application, after the source timbre features are output from the first fully connected layer, they can also be removed or recycled to the training audio set to expand the training audio set.

[0128] Figure 9 The electronic device 900 provided by an embodiment of the present application is shown. The electronic device 900 includes: a processor 901, a memory 902, and a computer program stored in the memory 902 and executable on the processor 901. When the computer program is executed, it is used to execute the above-mentioned timbre conversion method.

[0129] The processor 901 and the memory 902 may be connected via a bus or other means.

[0130] Memory 902, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs, such as the timbre conversion method described in the embodiments of this application. Processor 901 implements the timbre conversion method described above by executing the non-transitory software program and instructions stored in memory 902.

[0131] The memory 902 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function. The data storage area may store and execute the above-mentioned timbre conversion method. In addition, the memory 902 may include a high-speed random access memory 902, and may also include a non-transitory memory 902, such as at least one storage device memory device, a flash memory device or other non-transitory solid-state memory device. In some embodiments, the memory 902 may optionally include a memory 902 remotely arranged relative to the processor 901, and these remote memories 902 may be connected to the electronic device 900 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0132] The non-transient software program and instructions required to implement the above-mentioned timbre conversion method are stored in the memory 902. When executed by one or more processors 901, the above-mentioned timbre conversion method is executed, for example, Figure 1 Steps S101 to S105 of the method, Figure 2 Steps S201 to S203 of the method, Figure 3 Steps S301 to S303 of the method, Figure 4 Steps S401 to S402 of the method, Figure 5 Steps S501 to S502 of the method, Figure 6 Steps S601 to S603 of the method, Figure 7 Steps S701 to S703 of the method, Figure 8 Method steps S801 to S802.

[0133] The embodiment of the present application also provides a computer-readable storage medium storing computer-executable instructions, which are used to execute the above-mentioned timbre conversion method.

[0134] In one embodiment, the computer readable storage medium stores computer executable instructions, which are executed by one or more control processors, for example, Figure 1 Steps S101 to S105 of the method, Figure 2 Steps S201 to S203 of the method, Figure 3 Steps S301 to S303 of the method, Figure 4 Steps S401 to S402 of the method, Figure 5 Steps S501 to S502 of the method, Figure 6 Steps S601 to S603 of the method, Figure 7 Steps S701 to S703 of the method, Figure 8Method steps S801 to S802.

[0135] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0136] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, storage device storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media generally include computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium. It should also be understood that the various implementations provided in the embodiments of the present application may be arbitrarily combined to achieve different technical effects.

[0137] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above implementation mode. Technical personnel familiar with the art can also make various equivalent modifications or substitutions under the shared conditions that do not violate the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A timbre conversion method, characterized in that: include: Obtain source speech and target timbre features; Preprocessing the source speech to obtain an original mel spectrum; Inputting the original Mel spectrum into the speech encoder of the speech conversion model for feature extraction to obtain the semantic content features of the source speech; Inputting the semantic content feature and the target timbre feature into a speech decoder of the speech conversion model, so that the speech decoder performs feature decoding on the semantic content feature based on the target timbre feature to obtain a target mel spectrum; Performing speech conversion on the target mel spectrum to obtain a target speech; Inputting the original Mel spectrum into a speech encoder of a speech conversion model for feature extraction to obtain semantic content features of the source speech includes: inputting the original Mel spectrum into a first segmented convolutional layer of the speech encoder for dimensionality reduction processing to form an original Mel spectrum feature vector; decoupling the original Mel spectrum feature vector based on a first feature processing network to obtain a content feature vector; and integrating the content feature vectors via a first fully connected layer to obtain the semantic content features; Inputting the original Mel spectrum into the first segmented convolution layer of the speech encoder for dimensionality reduction processing to form an original Mel spectrum feature vector includes: dividing the convolution into multiple paths, each of the paths is processed separately by a different graphics processor; The step of inputting the original Mel spectrum into a speech encoder of a speech conversion model for feature extraction to obtain semantic content features of the source speech further includes: decoupling the original Mel spectrum feature vector based on the first feature processing network to obtain a source timbre feature vector; integrating the source timbre feature vector via the first fully connected layer to obtain a source timbre feature; and combining the source timbre feature with the timbre feature of a third person to form the target timbre feature.

2. The timbre conversion method according to claim 1, wherein: Before inputting the original Mel spectrum into the speech encoder of the speech conversion model for feature extraction, the method further includes: Acquire a training audio set, wherein the training audio set includes training speech formed based on multiple timbres; Generating a plurality of training mel spectrograms according to each of the training speech in the training audio set; The plurality of training mel spectrograms are input into a basic recognition model for optimization training to obtain the speech conversion model.

3. The timbre conversion method according to claim 2, wherein: The step of inputting the plurality of training mel spectrograms into a basic recognition model for optimization training to obtain the speech conversion model includes: Inputting the training mel spectrum into the basic recognition model for iterative training; After each round of iterative training, the model loss function of the basic recognition model is calculated and the basic recognition model is updated; When the output value of the model loss function converges to a constant value, the trained speech conversion model is obtained.

4. The timbre conversion method according to claim 3, wherein: The step of inputting the training mel spectrum into the basic recognition model for iterative training includes: Inputting the training mel spectrum into the basic encoder of the basic recognition model to perform feature extraction to obtain training content features and training timbre features; The training content features and the training timbre features are input into a basic decoder of the basic recognition model, so that the basic decoder performs feature decoding on the training content features based on the training timbre features to obtain a test mel spectrum.

5. The timbre conversion method according to claim 4, wherein: After each round of iterative training, the model loss function of the basic recognition model is calculated and the basic recognition model is updated, including: After each round of iterative training, calculating the model loss function based on the training Mel spectrum and the test Mel spectrum; After obtaining the model loss function, the parameters of the basic recognition model are adjusted for updating.

6. The timbre conversion method according to claim 1, wherein: The step of inputting the semantic content feature and the target timbre feature into a speech decoder of the speech conversion model, so that the speech decoder performs feature decoding on the semantic content feature based on the target timbre feature to obtain a target mel spectrum, comprises: Inputting the semantic content features and the target timbre features into the second segmented convolutional layer of the speech decoder, so that the second segmented convolutional layer performs vectorization processing on the semantic content features and the target timbre features to obtain the content feature vector and the target timbre feature vector; Inputting the content feature vector and the target timbre feature vector into a second feature processing network of the speech decoder, so that the second feature processing network couples the target timbre feature vector with the content feature vector to form a target mel-spectrogram feature vector; The target mel spectrum feature vector is integrated through the second fully connected layer of the speech decoder to obtain the target mel spectrum.

7. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor implements the timbre conversion method according to any one of claims 1 to 6 when executing the computer program.

8. A computer-readable storage medium, characterized in that The storage medium stores a program, and the program is executed by a processor to implement the timbre conversion method according to any one of claims 1 to 6.

Images