Voice conversion model training method, voice conversion method, and device thereof

Abstract

The application discloses a speech conversion model training method, a speech conversion method, a speech conversion model training device, a speech conversion device, an electronic device, and a computer readable storage medium. The training method comprises: obtaining a plurality of audio training data, and extracting first feature training data and second feature training data; obtaining a mel spectrogram corresponding to the audio training data, and obtaining a mask mel picture and a noise mel spectrogram corresponding to the mel spectrogram; inputting the first feature training data, the second feature training data, the mask mel spectrogram, and the noise mel spectrogram corresponding to the audio training data into a speech conversion model to be trained, and obtaining a predicted mel spectrogram; and adjusting model parameters of the speech conversion model to be trained according to the mel spectrogram and the predicted mel spectrogram, and obtaining a trained speech conversion model. The method solves the technical problem that the speech conversion effect is poor due to the fact that the existing technology cannot consider the timbre similarity, noise robustness, and expressiveness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and in particular, to a speech conversion model training method, a speech conversion method, a speech conversion model training device, a speech conversion device, an electronic device, and a computer readable storage medium. BACKGROUND

[0002] Speech conversion is a technology that changes the timbre of a speaker while keeping the content information unchanged, and is widely used in multiple fields. Generally, a speech conversion model is used to convert the source speech of a source speaker into target speech with the timbre of a target speaker.

[0003] Currently, the speech conversion model is mainly trained by using ASR (Automatic Speech Recognition) model features or SSL (Self-Supervised Learning) model features. The ASR model features remove most of the paralinguistic information, non-linguistic information, and noise information in the source speech, which can make the speech conversion model achieve high timbre similarity and noise robustness, but the output target speech often lacks expressiveness. The SSL model features almost retain all the information in the source speech, which can effectively improve the expressiveness of the target speech, but the potential background noise in the SSL model features greatly reduces the noise robustness of the speech conversion model.

[0004] Therefore, there is an urgent need for a speech conversion model that can guarantee high timbre similarity and noise robustness, and improve the expressiveness of the target speech, to solve the technical problem of poor speech conversion effect caused by the inability to balance timbre similarity, noise robustness, and expressiveness in the existing speech conversion model. SUMMARY

[0005] The present application provides a speech conversion model training method, a speech conversion method, a speech conversion model training device, a speech conversion device, an electronic device, and a computer readable storage medium to solve the technical problem of poor speech conversion effect caused by the inability to balance timbre similarity, noise robustness, and expressiveness in the prior art.

[0006] In a first aspect, an embodiment of the present application provides a method for training a speech conversion model, the method further comprising: obtaining a plurality of audio training data, and extracting first feature training data and second feature training data from the audio training data; wherein the first feature training data is used to represent content features corresponding to the audio training data, and the second feature training data is used to represent global features corresponding to the audio training data; obtaining a mel-spectrogram corresponding to the audio training data, and performing local mask processing and noise processing on the mel-spectrogram to obtain a masked mel-spectrogram and a noisy mel-spectrogram corresponding to the mel-spectrogram; inputting the first feature training data, the second feature training data, the masked mel-spectrogram, and the noisy mel-spectrogram corresponding to the audio training data into a speech conversion model to be trained to obtain a predicted mel-spectrogram output by the speech conversion model to be trained; and performing model parameter adjustment on the speech conversion model to be trained according to the mel-spectrogram and the predicted mel-spectrogram corresponding to the audio training data to obtain a trained speech conversion model.

[0007] In a second aspect, an embodiment of the present application provides a speech conversion method, the method comprising: in response to a speech conversion instruction, obtaining first audio data of a to-be-converted speaker and sample audio data of a specified speaker; extracting first feature data and second feature data from the first audio data, and extracting third feature data and fourth feature data from the sample audio data; wherein the first feature data is used to represent content features corresponding to the first audio data, the second feature data is used to represent global features corresponding to the first audio data, the third feature data is used to represent content features corresponding to the sample audio data, and the fourth feature data is used to represent global features corresponding to the sample audio data; performing splicing processing on the first feature data and the third feature data to generate fifth feature data, and performing splicing processing on the second feature data and the fourth feature data to generate sixth feature data; converting the first audio data of the to-be-converted speaker into second audio data of the specified speaker according to the fifth feature data, the sixth feature data, a first mel-spectrogram corresponding to the sample audio data, and a trained speech conversion model; wherein the trained speech conversion model is obtained according to the method for training a speech conversion model.

[0008] In a third aspect, an embodiment of the present application provides a device for training a speech conversion model, the device further comprising: a first data processing unit, a second data processing unit, a model training unit, and a model parameter adjusting unit; the first data processing unit is configured to obtain a plurality of audio training data, and extract first feature training data and second feature training data from the audio training data; wherein the first feature training data is used to represent content features corresponding to the audio training data, and the second feature training data is used to represent global features corresponding to the audio training data; the second data processing unit is configured to obtain a mel spectrogram corresponding to the audio training data, and perform local mask processing and noise processing on the mel spectrogram to obtain a mask mel spectrogram and a noise mel spectrogram corresponding to the mel spectrogram; the model training unit is configured to input the first feature training data, the second feature training data, the mask mel spectrogram, and the noise mel spectrogram corresponding to the audio training data into a speech conversion model to be trained, and obtain a predicted mel spectrogram output by the speech conversion model to be trained; and the model parameter adjusting unit is configured to perform model parameter adjustment on the speech conversion model to be trained according to the mel spectrogram and the predicted mel spectrogram corresponding to the audio training data, and obtain a trained speech conversion model.

[0009] In a fourth aspect, an embodiment of the present application provides a speech conversion device, the device further comprising: a data acquisition unit, a feature extraction unit, a feature splicing unit, and a data conversion unit; the data acquisition unit is configured to obtain first audio data of a to-be-converted speaker and sample audio data of a specified speaker in response to a speech conversion instruction; the feature extraction unit is configured to extract first feature data and second feature data from the first audio data, and extract third feature data and fourth feature data from the sample audio data; wherein the first feature data is used to represent content features corresponding to the first audio data, the second feature data is used to represent global features corresponding to the first audio data, the third feature data is used to represent content features corresponding to the sample audio data, and the fourth feature data is used to represent global features corresponding to the sample audio data; the feature splicing unit is configured to splice the first feature data and the third feature data to generate fifth feature data, and splice the second feature data and the fourth feature data to generate sixth feature data; and the data conversion unit is configured to convert the first audio data of the to-be-converted speaker into second audio data of the specified speaker according to the fifth feature data, the sixth feature data, a first mel spectrogram corresponding to the sample audio data, and a trained speech conversion model; wherein the trained speech conversion model is obtained by training according to the speech conversion model training method.

[0010] In a fifth aspect, an electronic device is provided, and the electronic device includes a memory and a processor. The memory is configured to store one or more computer instructions. The processor is configured to execute the one or more computer instructions to implement the method described above.

[0011] In a sixth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores one or more computer instructions. When the one or more computer instructions are executed by a processor, the one or more computer instructions perform the method described above.

[0012] Compared with the prior art, the training method of the voice conversion model provided in the present application includes: obtaining a plurality of audio training data, and extracting first feature training data and second feature training data from the audio training data; wherein the first feature training data is used to represent the content features corresponding to the audio training data, and the second feature training data is used to represent the global features corresponding to the audio training data; obtaining a mel spectrogram corresponding to the audio training data, and performing local mask processing and noise processing on the mel spectrogram to obtain a mask mel spectrogram and a noise mel spectrogram corresponding to the mel spectrogram; inputting the first feature training data, the second feature training data, the mask mel spectrogram, and the noise mel spectrogram corresponding to the audio training data into the voice conversion model to be trained to obtain a predicted mel spectrogram output by the voice conversion model to be trained; and adjusting the model parameters of the voice conversion model to be trained according to the mel spectrogram and the predicted mel spectrogram corresponding to the audio training data to obtain a trained voice conversion model. First, the method takes the content features corresponding to the audio training data as a training data, so that the voice conversion model to be trained can learn the timbre conversion logic in the training process, thereby improving the timbre similarity of the output result of the voice conversion model. Second, the method takes the global features corresponding to the audio training data as a training data, so that the voice conversion model to be trained can learn the paralanguage information and non-language information in the audio training data in the training process, thereby improving the expressiveness of the output result of the voice conversion model. Third, the method takes the noise mel spectrogram corresponding to the audio training data as a training data, predicts the mask part of the mask mel spectrogram, so that the voice conversion model to be trained can learn to reduce the interference of noise on the predicted result in the training process, thereby improving the noise robustness of the voice conversion model. Fourth, the method does not involve a time sequence alignment pre-operation on the first feature training data, the second feature training data, and the mel spectrogram, so that the voice conversion model to be trained needs to be modeled for time sequence alignment in the training process, thereby reducing the attention of the voice conversion model to non-important information such as noise. In summary, the voice conversion model trained based on the training method of the voice conversion model provided in the present application is a voice conversion model that can guarantee high timbre similarity and noise robustness, and can also improve the expressiveness of the target voice. The technical problem of poor voice conversion effect caused by the inability to balance timbre similarity, noise robustness, and expressiveness in the prior art is solved. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is an application system diagram of the speech conversion model training method provided by the embodiment of the present application;

[0014] Figure 2 is an application system diagram of the speech conversion method provided by the embodiment of the present application;

[0015] Figure 3 is a flowchart of the speech conversion model training method provided by the first embodiment of the present application;

[0016] Figure 4 is a schematic diagram of training the speech conversion model to be trained based on audio training data provided by the first embodiment of the present application;

[0017] Figure 5 is a flowchart of the speech conversion method provided by the second embodiment of the present application;

[0018] Figure 6 is a structural schematic diagram of the speech conversion model training device provided by the third embodiment of the present application;

[0019] Figure 7 is a structural schematic diagram of the speech conversion device provided by the fourth embodiment of the present application;

[0020] Figure 8 is a structural schematic diagram of the electronic device provided by the fifth embodiment of the present application. DETAILED DESCRIPTION

[0021] In the following description, a lot of specific details are set forth in order to facilitate a thorough understanding of the present application. However, the present application can be practiced in many different ways beyond the specific details disclosed herein, and similar modifications and / or substitutions can be made by those skilled in the art without departing from the spirit of the present application, so the present application is not limited to the specific implementations disclosed below.

[0022] With the development of speech processing technology, speech conversion appears, which is a technology of changing the timbre of a speaker while keeping the content information unchanged, and is widely applied in many fields such as dubbing of films and television dramas, privacy protection and personalized speech synthesis. Speech conversion is usually realized based on a speech conversion model, and the speech conversion model can convert the source speech of a source speaker into target speech with the timbre of a target speaker.

[0023] At present, the speech conversion model is divided into a parallel corpus-based speech conversion model and a non-parallel corpus-based speech conversion model according to the type of training data. Since the parallel corpus requires two or more speakers to speak the same content, the training data collection cost is high, and it is difficult to operate in practice. Therefore, the non-parallel corpus-based speech conversion model is the mainstream of speech conversion application. The non-parallel corpus-based speech conversion model is mainly trained based on the ASR (Automatic Speech Recognition, speech recognition) model feature or the SSL (Self Supervised Learning, self-supervised learning) model feature.

[0024] The speech conversion model is trained based on the ASR model feature, specifically, the source speech is input into the ASR model to obtain the bottleneck feature (i.e., the ASR model feature) in the ASR model inference process of the source speech, and then the speech conversion model is trained based on the feature. Since the ASR model feature removes most of the paralinguistic information (such as the pronunciation manner of the source speaker, the tone change, etc.), non-verbal information (such as the laughter, cough, and inhalation of the source speaker), and noise information in the source speech. Therefore, training the language conversion model based on the feature can make the language conversion model achieve high timbre similarity (i.e., the timbre in the target speech is highly similar to the timbre of the target speaker) and noise robustness (i.e., the target speech does not contain noise or only contains slight noise, and the content information in the target speech is not disturbed by the noise and is consistent with the content information in the source speech), but the output target speech often lacks expressiveness (i.e., it cannot express the pronunciation manner, tone change, etc. of the source speaker in the source speech, and the laughter, cough, etc. of the source speaker, and only expresses the target speaker's monotonous reading without emotion).

[0025] The speech conversion model is trained based on the SSL model feature, specifically, the source speech is input into the SSL model to obtain the speech feature (i.e., the SSL model feature) derived by the SSL model through self-supervised learning, and then the speech conversion model is trained based on the feature. Since the SSL model feature almost retains all the information (such as paralinguistic information and non-verbal information) in the source speech. Therefore, training the language conversion model based on the feature can effectively improve the expressiveness of the target speech. However, since the background noise is also retained in the SSL model feature, the noise robustness of the speech conversion model is greatly reduced.

[0026] Therefore, the existing speech conversion model is a trade-off between timbre similarity, noise robustness, and expressiveness, and cannot be achieved at the same time, and there is a problem of poor speech conversion effect.

[0027] Therefore, the application provides a speech conversion model training method, based on which a speech conversion model with high timbre similarity and noise robustness and high expressiveness of target speech can be obtained. First, the method takes the content features corresponding to the audio training data as a training data, so that the speech conversion model to be trained can learn the timbre conversion logic in the training process, thereby improving the timbre similarity of the output results of the speech conversion model. Second, the method takes the global features corresponding to the audio training data as a training data, so that the speech conversion model to be trained can learn the paralanguage information and non-language information in the audio training data in the training process, thereby improving the expressiveness of the output results of the speech conversion model. Third, the method takes the noise mel-spectrogram corresponding to the audio training data as a training data, predicts the mask part of the mask mel-spectrogram, so that the speech conversion model to be trained can learn to reduce the interference of noise on the prediction results in the training process, thereby improving the noise robustness of the speech conversion model. Fourth, the method does not involve the pre-operation of time sequence alignment of the first feature training data, the second feature training data, and the mel-spectrogram, so that the speech conversion model to be trained needs to be modeled in the training process, thereby reducing the attention of the speech conversion model to non-important information such as noise.

[0028] The speech conversion model training method, the speech conversion method, the device thereof, the electronic device, and the computer readable storage medium provided in the application will be further described in detail below with reference to specific embodiments and the accompanying drawings.

[0029] Figure 1 is an application system diagram of the speech conversion model training method provided by the embodiments of the application. As shown in Figure 1 , the system includes a user terminal 101 and a server 102. The user terminal 101 can be any device such as a smartphone, a tablet computer, a notebook computer, a desktop computer, a personal digital assistant (PDA), etc. The server 102 can be a cloud server in communication connection with the user terminal 101. The server 102 is deployed with the speech conversion model training method provided by the application and the speech conversion model to be trained, and in response to receiving the model training instruction and the training sample set sent by the user terminal 101, the speech conversion model to be trained is trained based on the method.

[0030] Figure 2 is an application system diagram of the speech conversion method provided by the embodiments of the application. As shown in Figure 2As shown, the system includes a first user terminal 201, a first service terminal 202 and a second service terminal 203. The first user terminal 201 can be any device such as a smartphone, a tablet computer, a notebook computer, a desktop computer, a personal digital assistant (PDA) and the like. The first service terminal 202 can be a service module inside the first user terminal 201, a service device electrically connected to the first user terminal 201, or a server in communication connection with a plurality of first user terminals 201. The first service terminal 202 is deployed with the speech conversion method provided in the present application, and the second service terminal 203 is deployed with a trained speech conversion model. The first service terminal 202, in response to receiving a speech conversion instruction sent by the first user terminal 201, calls the trained speech conversion model from the second service terminal 203, and converts the speech to be converted based on the method.

[0031] The first embodiment of the present application provides a training method of a speech conversion model, which is deployed in Figure 1 a service terminal 102 as shown, for providing training services for the speech conversion model.

[0032] The model can understand a bionic neural network that mimics the structure and function of a biological neural system, and can be applied to identification, classification, analysis, conversion and the like. The model is composed of a large number of nodes (i.e., neurons), and by continuously optimizing the parameters of each node in the training process, the model can learn the complex relationships in the training data and make accurate predictions on the data to be processed.

[0033] The speech conversion model is a neural network model that can be applied to speech conversion tasks, which can keep the content information of the source speech unchanged, and change the source speaker's timbre to the specified speaker's timbre. In the present embodiment, the speech conversion model can also ensure that the target speech does not contain noise or only has slight noise, and the content information in the target speech is not disturbed by the noise and is consistent with the content information in the source speech. Further, the speech conversion model described in the present embodiment can also ensure that the target speech includes the paralanguage information and nonverbal information in the source speech, and has strong expressiveness.

[0034] The training method of the speech conversion model described in the present embodiment is described in detail as follows:

[0035] Figure 3 is a flowchart of the training method of the speech conversion model provided in the present embodiment. The training method of the speech conversion model provided in the present embodiment is described in detail as follows: Figure 3 The training method of the speech conversion model provided in the present embodiment is described in detail as follows. The embodiments described below are used to explain the technical solutions of the present application, and are not limited to actual use.

[0036] As Figure 3As shown, the training method of the voice conversion model provided in this embodiment includes the following steps S310 to S340.

[0037] In step S310, a plurality of audio training data are obtained, and first feature training data and second feature training data are extracted from the audio training data; wherein the first feature training data is used to represent the content features corresponding to the audio training data, and the second feature training data is used to represent the global features corresponding to the audio training data.

[0038] The content features can be understood as features converted from language information in the audio training data. For example, a word vector representing the semantics of the language information "hello world" in an audio segment. In this embodiment, the content features extracted from the audio training data are defined as the first feature training data.

[0039] The global features can be understood as features converted from all information such as language information, non-language information, paralanguage information, etc. in the audio training data. For example, for an audio segment, the global features include a word vector representing the semantics of the language information, an acoustic feature representing the pronunciation manner and tone change of the paralanguage information, and an emotion feature representing the emotion of the non-language information. In this embodiment, the global features extracted from the audio training data are defined as the second feature training data.

[0040] In an optional implementation, the first feature training data and the second feature training data are extracted from the audio training data, which can specifically include the following steps S311 to S312:

[0041] In step S311, the audio training data are input into a pre-trained speech recognition model to obtain the content features generated by the speech recognition model in the inference process of the audio training data, and the content features are taken as the first feature training data.

[0042] The speech recognition (ASR) model is a neural network model that can convert human speech into text. It identifies and transcribes the words spoken by the speaker by processing and analyzing audio data. In this embodiment, the ASR model is a pre-trained model that has been trained and can accurately infer audio data to obtain text data. Optionally, the pre-trained ASR model can be a model trained by the developer based on a large amount of data, or an existing open source model, without limitation.

[0043] In a specific implementation, the content feature is a bottleneck feature generated in the process of inferring the audio data by the pre-trained ASR model, i.e., a feature generated by a hidden layer of the ASR model. The hidden layer is located in the middle of the ASR model and is designed as a compressed representation for retaining necessary information for speech recognition in the audio data, i.e., language information, and converting the language information into a feature capable of representing semantics thereof, i.e., the content feature.

[0044] In step S312, the audio training data is input into the pre-trained self-supervised learning model to obtain global features generated by the self-supervised learning model after inferring the audio training data, and the global features are taken as second feature training data.

[0045] The self-supervised learning (SSL) model is a learning method without manual annotation of data, which learns useful logic from self-supervision by designing specific tasks (e.g., predicting and attempting to restore a masked part in audio data or outputting a certain audio feature in audio data). In this embodiment, the SSL model is a pre-trained model that has been trained and is capable of accurately inferring global features in audio data. Optionally, the pre-trained SSL model can be a model trained by a developer based on a large amount of data, or an existing open source model, and the specific implementation is not limited.

[0046] In a specific implementation, the global feature is an output feature generated in the process of inferring the audio data by the pre-trained SSL model, i.e., a feature generated by an output layer of the SSL model. The output layer is located at the outermost layer of the SSL model and is responsible for generating a final feature representation, i.e., the global feature.

[0047] In step S320, a mel spectrogram corresponding to the audio training data is obtained, and the mel spectrogram is subjected to local mask processing and noise adding processing to obtain a masked mel spectrogram and a noisy mel spectrogram corresponding to the mel spectrogram.

[0048] The mel spectrogram is a representation method of audio signals, which shows the frequency components of audio at different time points, but uses a mel scale, which is a frequency scale more consistent with human auditory perception. The mel spectrogram is usually obtained by performing short-time Fourier transform (STFT) and mel filter bank on audio data. Conventional techniques can be used to extract the mel spectrogram corresponding to the audio data, and specific details are not described herein.

[0049] In this embodiment, obtaining the mel spectrogram corresponding to the audio training data can be understood as converting the audio training data into a mel spectrogram for subsequent processing, including local mask processing and noise adding processing on the mel spectrogram.

[0050] The local mask processing is a data enhancement technique, which refers to selectively masking or covering certain areas in the mel-spectrogram. This processing can simulate the missing or damaged parts that may exist in the audio data, to help the model learn to better handle incomplete information. In this embodiment, the mel-spectrogram that has undergone local mask processing is defined as a masked mel-spectrogram. For example, a continuous time window can be randomly selected on the time axis, and a number of continuous frequency channels can be randomly selected on the frequency axis, and then the values of these selected areas are set to zero or a certain specific value, thereby creating a masked mel-spectrogram.

[0051] The noise adding processing is also a data enhancement technique, which refers to adding artificially generated noise to the mel-spectrogram. This processing can improve the noise robustness of the model, so that it can better cope with the background noise or other interference factors existing in the actual environment. The noise can be randomly generated, or extracted from real environmental noise samples. The intensity and type of added noise can be adjusted according to the specific application scenario. In this embodiment, the mel-spectrogram that has undergone noise adding processing is defined as a noisy mel-spectrogram.

[0052] Training the mel-spectrogram that has undergone local mask processing and noise adding processing as part of the training data in the model training process can help the model learn more generalization ability, that is, it can still perform well in the conversion when facing audio data generated under complex conditions.

[0053] In step S330, the first feature training data, the second feature training data, the masked mel-spectrogram, and the noisy mel-spectrogram corresponding to the audio training data are input into the speech conversion model to be trained, and a predicted mel-spectrogram output by the speech conversion model to be trained is obtained.

[0054] After obtaining the first feature training data, the second feature training data, the masked mel-spectrogram, and the noisy mel-spectrogram corresponding to the audio training data, these training data are input into the speech conversion model to be trained. The speech conversion model to be trained will infer the mask part of the masked mel-spectrogram based on the first feature training data, the second feature training data, the masked mel-spectrogram, and the noisy mel-spectrogram, and output a predicted mel-spectrogram, that is, the combination of the masked mel-spectrogram and the predicted mask part.

[0055] The method provided in this embodiment does not involve aligning the first feature training data, the second feature training data, and the mel spectrograms (including the mask mel spectrogram and the noise mel spectrogram) in time sequence, that is, after the first feature training data, the second feature training data, the mask mel spectrogram, and the noise mel spectrogram corresponding to the audio training data are input into the speech conversion model to be trained, the speech conversion model to be trained first needs to be modeled in time sequence, that is, in the training process, how to align the first feature training data, the second feature training data, the mask mel spectrogram, and the noise mel spectrogram in time sequence is learned, so that on the basis of the time sequence alignment, the mask part is predicted based on the first feature training data and the second feature training data corresponding to the mask part, and the unmasked part in the mask mel spectrogram and the noise mel spectrogram, and thus the predicted mel spectrogram is output. As described above, the speech conversion model to be trained needs to be modeled in time sequence in the training process, and therefore, the attention of the speech conversion model to the details (such as background noise) other than the speech in the audio training data is reduced, and the noise robustness of the speech conversion model is improved.

[0056] Based on this, in an optional implementation manner, the speech conversion model to be trained can be a diffusion model, such as a flow matching model. The diffusion model is a kind of generative model, which has the ability of modeling generation. In the method provided in this embodiment, the diffusion model is used as the speech conversion model to be trained, which is to realize the time sequence alignment of the input features and the output features by the model automatically.

[0057] In step S340, the model parameter adjustment is performed on the speech conversion model to be trained according to the mel spectrogram and the predicted mel spectrogram corresponding to the audio training data, to obtain the trained speech conversion model.

[0058] After the first feature training data, the second feature training data, the mask mel spectrogram, and the noise mel spectrogram corresponding to the audio training data are input into the speech conversion model to be trained, the predicted mel spectrogram output by the speech conversion model to be trained is obtained. The predicted mel spectrogram can be understood as a predicted value corresponding to the audio training data, and the mel spectrogram can be understood as a true value corresponding to the audio training data. The model parameter adjustment is performed on the speech conversion model to be trained based on the difference between the predicted value and the true value, and thus the speech conversion model trained by the audio training data is obtained.

[0059] Specifically, the model parameter adjustment based on the predicted mel spectrogram and the mel spectrogram aims to minimize the difference between the predicted value and the true value, so as to improve the performance of the speech conversion model to be trained. Optionally, the model parameter can be adjusted by minimizing the loss function between the predicted mel spectrogram and the mel spectrogram based on the gradient descent method, so as to gradually approach the optimal solution.

[0060] In an optional implementation, the method provided by the embodiment further includes the following step S350:

[0061] In step S350, it is determined whether the trained speech conversion model reaches the preset training target. Specifically, in response to the trained speech conversion model reaching the training target, the trained speech conversion model is taken as the speech conversion model that has completed training, and is used for speech conversion on the audio data to be converted; in response to the trained speech conversion model not reaching the training target, the trained speech conversion model is taken as the speech conversion model to be trained, and the iterative training is continued.

[0062] Optionally, in the process of training the speech conversion model to be trained based on the audio training data, it is detected whether the trained speech conversion model reaches the training target according to a preset time interval. If the training target is reached, the trained speech conversion model is the speech conversion model that has completed training, and can be put into use. If the training target is not reached, the trained speech conversion model cannot be put into use, and needs to be iteratively trained based on the audio training data.

[0063] Optionally, the training target is preset by a developer according to the application needs of the speech conversion model. The training target can be a loss function threshold preset for a loss function between a predicted value (i.e., a predicted mel-spectrogram) and a true value (i.e., a mel-spectrogram), can be a similarity threshold set for a similarity between the predicted value (i.e., the predicted mel-spectrogram) and the true value (i.e., the mel-spectrogram), and can be an iteration number threshold set for an iteration number of model training, and the specific implementation is not limited herein. For example, the training target is less than the loss function threshold 0.05. When the loss function between the predicted mel-spectrogram output by the speech conversion model to be trained and the mel-spectrogram corresponding to the audio training data is less than 0.05, it is determined that the trained speech conversion model reaches the training target, the training is terminated, and the trained speech conversion model is taken as the speech conversion model that has completed training and is put into the conversion task of the speech to be converted.

[0064] In an optional implementation, before the step of inputting the first feature training data, the second feature training data, the mask mel-spectrogram, and the noise mel-spectrogram corresponding to the audio training data into the speech conversion model to be trained, and obtaining the predicted mel-spectrogram output by the speech conversion model to be trained, the method provided by the embodiment further includes the following steps S11 to S13:

[0065] In step S11, the first feature training data and the second feature training data corresponding to the audio training data are respectively subjected to dimension reduction processing of a preset channel dimension, to obtain first representation training data corresponding to the first feature training data, and second representation training data corresponding to the second feature training data.

[0066] In step S12, the first feature training data and the second feature training data are padded to be equal in length to the mel spectrogram corresponding to the audio training data according to the time sequence length of the mel spectrogram.

[0067] In step S13, the padded first feature training data, the padded second feature training data, the mask mel spectrogram, and the noise mel spectrogram are spliced in the channel dimension to obtain spliced feature training data corresponding to the audio training data.

[0068] The dimension reduction processing of the feature training data (including the first feature training data and the second feature training data) can be understood as a full connection mapping, that is, the high-dimensional feature training data is mapped to a low dimension. In this embodiment, the feature training data processed by the dimension reduction processing is defined as the feature training data.

[0069] The preset channel dimension is an adjustable value, and the proportion of each feature training data in the overall feature data can be adjusted by setting different dimension reduction processing channel dimensions for different feature training data. Generally, the higher the proportion of the feature training data, the more likely it is to be focused on by the to-be-trained speech conversion model, and the lower the proportion of the feature training data, the lower the attention of the to-be-trained speech conversion model. Therefore, the developer can adjust the channel dimension of different feature training data during the dimension reduction processing to adjust the final training effect of the to-be-trained speech conversion model. For example, the first feature training data is processed to 256 dimensions, and the second feature training data is processed to 64 dimensions. Then, the to-be-trained speech conversion model will pay more attention to the first feature training data during the training process, and reduce the attention to the second feature training data. The finally trained speech conversion model may have higher timbre similarity and noise robustness, but the expressiveness will be slightly weak.

[0070] In this embodiment, the low-dimensional feature training data obtained by dimension reduction processing of the first feature training data is defined as the first feature training data, and the low-dimensional feature training data obtained by dimension reduction processing of the second feature training data is defined as the second feature training data. Optionally, the dimension reduction processing is a preprocessing operation based on an encoder. Specifically, the first feature training data is processed by a content feature encoder to obtain the first feature training data, and the second feature training data is processed by a global feature encoder to obtain the second feature training data. Optionally, the content feature encoder and the global feature encoder can be independent of the speech conversion model, or can be combined with the speech conversion model as a component that does not need to be trained in the model.

[0071] In this embodiment, after the first feature training data and the second feature training data corresponding to the audio training data are subjected to dimension reduction processing to obtain the first representation training data and the second representation training data, the first representation training data and the second representation training data are further subjected to padding processing to make the first representation training data and the second representation training data equal in length to the mel spectrogram in time sequence. The padding processing is different from the time sequence alignment processing. The time sequence alignment processing is usually to expand the data to be equal in length to the mel spectrogram by upsampling (for example, linear interpolation, transposed convolution, etc.), that is, the structure of the data itself is changed to achieve alignment with the mel spectrogram in time sequence. The padding processing does not change the structure of the data itself, but only supplements the feature data to be equal in length to the mel spectrogram by padding special marker symbols or specific constants (for example, padding "0") at the tail of the data sequence, that is, the padded feature training data is equal in length to the mel spectrogram, but not aligned in time sequence.

[0072] In this embodiment, after the first representation training data and the second representation training data are obtained, the first representation training data and the second representation training data are subjected to channel dimension splicing processing with the mask mel spectrogram and the noise mel spectrogram to form a combined feature. In this embodiment, the combined feature after the splicing processing is defined as spliced feature training data. Splicing in the channel dimension can be understood as horizontal splicing, that is, the amount of data is increased in the channel dimension without changing the time step or the spatial size. In this embodiment, the time sequence length of the spliced feature training data is still consistent with the time sequence length of the mel spectrogram.

[0073] Based on this, the first feature training data, the second feature training data, the mask mel spectrogram, and the noise mel spectrogram corresponding to the audio training data are input into the speech conversion model to be trained to obtain a predicted mel spectrogram output by the speech conversion model to be trained. Specifically, the spliced feature training data corresponding to the audio training data can be input into the speech conversion model to be trained to obtain a predicted mel spectrogram output by the speech conversion model to be trained.

[0074] In an optional implementation, to further enhance the expressiveness of the speech conversion model in converting audio, the fundamental frequency data in the audio training data is separately taken as an independent feature to participate in model training. The fundamental frequency data is an inherent attribute of the audio data, is the lowest frequency component in the audio data, and determines the pitch height. Explicitly embodying the fundamental frequency data as an independent training data can enable the speech conversion model to have a better conversion effect on converting singing and other special audio data with strong expressiveness and rich rhythm. Based on this, the first feature training data and the second feature training data are extracted from the audio training data, and specifically, the first feature training data, the second feature training data, and the fundamental frequency data can also be extracted from the audio training data. The first feature training data, the second feature training data, the mask mel spectrogram, and the noise mel spectrogram corresponding to the audio training data are input into the speech conversion model to be trained, and the predicted mel spectrogram output by the speech conversion model to be trained is obtained. Specifically, the first feature training data, the second feature training data, the fundamental frequency data, the mask mel spectrogram, and the noise mel spectrogram corresponding to the audio training data are input into the speech conversion model to be trained, and the predicted mel spectrogram output by the speech conversion model to be trained is obtained.

[0075] In an optional implementation, before the step of inputting the first feature training data, the second feature training data, the fundamental frequency data, the mask mel spectrogram, and the noise mel spectrogram corresponding to the audio training data into the speech conversion model to be trained, and obtaining the predicted mel spectrogram output by the speech conversion model to be trained, the method provided in this embodiment further includes: performing multi-scale fundamental frequency modeling on the fundamental frequency data corresponding to the audio training data to obtain multi-scale fundamental frequency data corresponding to the audio training data.

[0076] The multi-scale fundamental frequency modeling can be understood as performing downsampling of different granularities on the fundamental frequency data to obtain the fundamental frequency data of different granularities. For example, 1-second audio training data samples N fundamental frequency values, realizes that each content word in the audio data corresponds to a fundamental frequency value, performs downsampling on the fundamental frequency data, gradually coarsens the fundamental frequency data, and realizes that each sentence in the audio data corresponds to a fundamental frequency value.

[0077] Based on this, the first feature training data, the second feature training data, the fundamental frequency data, the mask mel spectrogram, and the noise mel spectrogram corresponding to the audio training data are input into the to-be-trained speech conversion model to obtain a predicted mel spectrogram output by the to-be-trained speech conversion model. Specifically, the first feature training data, the second feature training data, the multi-scale fundamental frequency data, the mask mel spectrogram, and the noise mel spectrogram corresponding to the audio training data can be input into the to-be-trained speech conversion model to obtain a predicted mel spectrogram output by the to-be-trained speech conversion model. The speech conversion model training based on the multi-scale fundamental frequency data can enhance the stability and accuracy of pronunciation in the target speech converted by the speech conversion model.

[0078] In an optional implementation, the multi-scale fundamental frequency data, the padded first feature training data, the padded second feature training data, the mask mel spectrogram, and the noise mel spectrogram can be spliced in the channel dimension to obtain spliced feature training data, and then the spliced feature training data is input into the to-be-trained speech conversion model to obtain a predicted mel spectrogram output by the to-be-trained speech conversion model.

[0079] In an optional implementation, after the step of extracting the first feature training data and the second feature training data from the audio training data, the method provided by the embodiment can further include: removing, according to a preset proportion, part of the second feature training data corresponding to the audio training data. Specifically, for each of the plurality of audio training data, after the first feature training data and the second feature training data are extracted, part of the second feature training data is discarded to reduce the dependence of the speech conversion model on the second feature training data. This is because the second feature training data is a global feature in the audio training data, which includes not only language information but also other paralanguage information, non-language information, background noise, etc. If the speech conversion model excessively depends on the second feature training data during the training process, the method of voice timbre conversion cannot be sufficiently learned because too much attention is paid to the language information. The removal proportion of the second feature training data is an adjustable value, and a developer can adjust it according to the output effect of the speech conversion model during the training process, and the specific adjustment is not limited.

[0080] Based on this, for the audio training data from which the second feature training data is removed, the first feature training data, the second feature training data, the mask mel spectrogram, and the noise mel spectrogram corresponding to the audio training data are input into the to-be-trained speech conversion model to obtain a predicted mel spectrogram output by the to-be-trained speech conversion model. Specifically, the first feature training data, the mask mel spectrogram, and the noise mel spectrogram corresponding to the audio training data can be input into the to-be-trained speech conversion model to obtain a predicted mel spectrogram output by the to-be-trained speech conversion model.

[0081] The following illustrates how to train the speech conversion model to be trained based on audio training data in a specific implementation manner.

[0082] Figure 4 is a schematic diagram of training the speech conversion model to be trained based on audio training data provided by the embodiment. As shown in Figure 4 training the speech conversion model to be trained based on audio training data can specifically include the following steps S401 to S412:

[0083] Step S401, input the audio training data 410 into the pre-trained SSL model to obtain the second feature training data 411 corresponding to the audio training data 410.

[0084] Step S402, input the audio training data 410 into the pre-trained ASR model to obtain the first feature training data 412 corresponding to the audio training data 410.

[0085] Step S403, input the audio training data 410 into the fundamental frequency tracking algorithm to obtain the fundamental frequency data 413 corresponding to the audio training data 410.

[0086] Step S404, input the second feature training data 411 into the global feature encoder for dimension reduction preprocessing to obtain the second representation training data 414 corresponding to the second feature training data 411.

[0087] Step S405, input the first feature training data 412 into the content feature encoder for dimension reduction preprocessing to obtain the first representation training data 415 corresponding to the first feature training data 412.

[0088] Step S406, input the fundamental frequency data 413 into the multi-scale fundamental frequency modeling encoder for multi-scale modeling to obtain the multi-scale fundamental frequency data 416 corresponding to the fundamental frequency data 413.

[0089] Step S407, obtain the mel spectrogram 417 corresponding to the audio training data 410.

[0090] Step S408, perform local mask processing on the mel spectrogram 417 to obtain the masked mel spectrogram 418.

[0091] Step S409, perform noise adding processing on the mel spectrogram 417 to obtain the noise mel spectrogram 419.

[0092] Step S410, the second feature training data 414 and the first feature training data 415 are padded to be equal in length to the mel-spectrogram 417, and the padded second feature training data 414, the padded first feature training data 415, the multi-scale fundamental frequency data 416, the mask mel-spectrogram 418, and the noise mel-spectrogram 419 are spliced to obtain the spliced feature training data 420 corresponding to the audio training data 410.

[0093] Step S411, the spliced feature training data 420 is input into the speech conversion model to be trained to obtain a predicted mel-spectrogram 421 output by the speech conversion model to be trained.

[0094] Step S412, based on the mel-spectrogram 417 corresponding to the audio training data 410 and the predicted mel-spectrogram 421, the model parameters of the speech conversion model are adjusted.

[0095] The first embodiment provides an optional training method of a speech conversion model. First, the method takes the content feature corresponding to the audio training data as a training data, so that the speech conversion model to be trained can learn the timbre conversion logic in the training process, and the timbre similarity of the output result of the speech conversion model is improved. Second, the method takes the global feature corresponding to the audio training data as a training data, so that the speech conversion model to be trained can learn the paralanguage information and non-language information in the audio training data in the training process, and the expressiveness of the output result of the speech conversion model is improved. Third, the method takes the noise mel-spectrogram corresponding to the audio training data as a training data, and predicts the mask part of the mask mel-spectrogram, so that the speech conversion model to be trained can learn to reduce the interference of noise on the prediction result in the training process, and the noise robustness of the speech conversion model is improved. Fourth, the method does not involve the time sequence alignment pre-operation of the first feature training data, the second feature training data, and the mel-spectrogram, so that the speech conversion model to be trained needs to model the time sequence alignment in the training process, and the attention of the speech conversion model to non-important information such as noise is reduced. Fifth, in some implementations, the method also takes the fundamental frequency data / multi-scale fundamental frequency data corresponding to the audio training data as a training data, so that the speech conversion model to be trained learns the conversion logic of the pitch in the training process, and further improves the expressiveness of the output result of the speech conversion model, especially for special audio data such as singing, which can ensure the stability and accuracy of singing pronunciation. In summary, the speech conversion model trained based on the training method of the speech conversion model provided in the present application is a speech conversion model that can not only guarantee high timbre similarity and noise robustness, but also improve the expressiveness of the target speech.

[0096] It should be noted that the examples in the first embodiment are only for explaining the methods described in this application and are not intended to limit actual use. The training methods of the speech conversion model provided in this application include, but are not limited to, the methods described in the first embodiment.

[0097] A second embodiment of this application provides a speech conversion method, which is deployed on... Figure 2 The first server 202 shown provides a conversion service for the speech to be converted. Specifically, the method uses the speech conversion model trained in the first embodiment of this application to convert the source speech of the speaker to be converted, generating the target speech of the specified speaker.

[0098] Figure 5 This is a flowchart of the speech conversion method provided in this embodiment. The following is in conjunction with... Figure 5 The speech conversion method provided in this embodiment will be described in detail. The embodiments described below are used to explain the technical solutions of this application and are not intended to limit actual use.

[0099] like Figure 5 As shown, the speech conversion method provided in this embodiment includes the following steps S510 to S540:

[0100] Step S510: In response to the speech conversion instruction, first audio data of the speaker to be converted and sample audio data of the specified speaker are obtained.

[0101] The voice conversion instruction can be understood as the speaker or user to be converted based on... Figure 2 The first user terminal 201 sends an instruction to the first server 202 requesting that the source speech of the speaker to be converted be converted into the target speech of the specified speaker. For example, the speech conversion instruction may be an instruction generated by the first user terminal 201 in response to the speaker to be converted uploading source speech and selecting the specified speaker, and then sent to the second server 202. In this embodiment, the source speech of the speaker to be converted is defined as first audio data, and the target speech of the specified speaker is defined as second audio data.

[0102] The sample audio data of the designated speaker refers to a pre-produced sample voice of the designated speaker, used to provide the timbre of the designated speaker. This sample audio data can be a very short segment of voice, such as a few seconds or tens of seconds. After the speaker to be converted or the user-designated speaker is acquired, the sample audio data of the designated speaker will be acquired simultaneously with the acquisition of the first audio data, so as to convert the first audio data into second audio data with the timbre of the designated speaker.

[0103] Step S520, first feature data and second feature data are extracted from the first audio data, and third feature data and fourth feature data are extracted from the sample audio data; wherein the first feature data is used to represent the content feature corresponding to the first audio data, the second feature data is used to represent the global feature corresponding to the first audio data, the third feature data is used to represent the content feature corresponding to the sample audio data, and the fourth feature data is used to represent the global feature corresponding to the sample audio data.

[0104] In an optional implementation, the first feature data and the second feature data are extracted from the first audio data, and the third feature data and the fourth feature data are extracted from the sample audio data, which can specifically include steps S521 to S524 as follows:

[0105] Step S521, the first audio data is input into the pre-trained speech recognition model to obtain the content feature generated by the speech recognition model in the inference process of the first audio data, and the content feature is taken as the first feature data.

[0106] Step S522, the first audio data is input into the pre-trained self-supervised learning model to obtain the global feature generated by the self-supervised learning model after the inference of the first audio data, and the global feature is taken as the second feature data.

[0107] Step S523, the sample audio data is input into the pre-trained speech recognition model to obtain the content feature generated by the speech recognition model in the inference process of the sample audio data, and the content feature is taken as the third feature data.

[0108] Step S524, the sample audio data is input into the pre-trained self-supervised learning model to obtain the global feature generated by the self-supervised learning model after the inference of the sample audio data, and the global feature is taken as the fourth feature data.

[0109] The specific method of extracting the content feature and the global feature from the audio data based on the pre-trained speech recognition model and the self-supervised learning model has been described in detail in the first embodiment of the present application, and will not be repeated here.

[0110] Step S530, the first feature data and the third feature data are spliced to generate the fifth feature data, and the second feature data and the fourth feature data are spliced to generate the sixth feature data.

[0111] The splicing of the first feature data and the third feature data is splicing in the time sequence dimension, specifically, the third feature data is spliced at the front end of the first feature data to form the fifth feature data. Similarly, the splicing of the second feature data and the fourth feature data is also splicing in the time sequence dimension, specifically, the fourth feature data is spliced at the front end of the second feature data to form the sixth feature data.

[0112] In step S540, the first audio data of the to-be-converted speaker is converted into the second audio data of the specified speaker according to the fifth feature data, the sixth feature data, the first mel-spectrogram corresponding to the sample audio data, and the trained voice conversion model. The trained voice conversion model is obtained by training according to the method of the first embodiment of the present application.

[0113] Specifically, the fifth feature data, the sixth feature data, and the first mel-spectrogram corresponding to the sample audio data are input as output data into the trained voice conversion model, the trained voice conversion model continues the first mel-spectrogram, and further converts the audio based on the mel-spectrogram behind the continued first mel-spectrogram, so that the second audio data with the specified speaker's timbre is obtained. Since the trained voice conversion model is obtained by training based on multiple features (such as content features, global features, and fundamental frequency), the second audio data not only has high timbre similarity and noise robustness, but also has high expressiveness.

[0114] In an optional implementation, before the step of splicing the first feature data and the third feature data to generate the fifth feature data, and splicing the second feature data and the fourth feature data to generate the sixth feature data, the method provided by the embodiment can further include the following steps S21 to S24:

[0115] In step S21, the first feature data and the third feature data are subjected to dimension reduction processing of a first preset channel dimension, to obtain first representation data corresponding to the first feature data, and third representation data corresponding to the third feature data.

[0116] In step S22, the second feature data and the fourth feature data are subjected to dimension reduction processing of a second preset channel dimension, to obtain second representation data corresponding to the second feature data, and fourth representation data corresponding to the fourth feature data.

[0117] In step S23, the third representation data and the fourth representation data are padded to be equal in length to the first mel-spectrogram according to the time sequence length of the first mel-spectrogram.

[0118] In step S24, the first representation data and the second representation data are padded to be equal in length to the second mel-spectrogram according to the time sequence length of the second mel-spectrogram. The second mel-spectrogram is a full-mask mel-spectrogram preset according to the time sequence length of the first audio data.

[0119] The dimension reduction processing has been described in detail in the first embodiment of the present application, and will not be repeated here. It should be noted that the first preset channel dimension and the second preset channel dimension need to be consistent with the channel dimensions preset for the first feature training data and the second feature training data when training the speech conversion model. For example, when training the speech conversion model, the channel dimension set for the first feature training data is 256, and the channel dimension set for the second feature training data is 128. Therefore, in the present embodiment, the first feature data and the third feature data need to be subjected to 256-dimensional dimension reduction processing, and the second feature data and the fourth feature data need to be subjected to 128-dimensional dimension reduction processing. In the present embodiment, the low-dimensional feature data obtained by reducing the dimension of the first feature data is defined as first representation data, and similarly, the low-dimensional feature data obtained by reducing the dimension of the second feature data, the third feature data and the fourth feature data are defined as second representation data, third representation data and fourth representation data, respectively.

[0120] In the present embodiment, after the first feature data, the second feature data, the third feature data and the fourth feature data are subjected to dimension reduction processing, the representation data corresponding to these feature data is also subjected to padding processing. Specifically, the first representation data and the second representation data are padded to be equal in length to the second mel-spectrogram, and the third representation data and the fourth representation data are padded to be equal in length to the first mel-spectrogram. The padding method has been described in detail in the first embodiment of the present application, and will not be repeated here. For example, if the time sequence length corresponding to the first mel-spectrogram is 10 frames, then "0" needs to be padded at the end of the sequence of the third representation data and the fourth representation data, so that the time sequence length of the third representation data and the fourth representation data also reaches 10 frames; if the time sequence length corresponding to the second mel-spectrogram is 1000 frames, then "0" needs to be padded at the end of the sequence of the first representation data and the second representation data, so that the time sequence length of the first representation data and the third representation data also reaches 1000 frames.

[0121] The second mel-spectrogram is a full-mask mel-spectrogram preset according to the time sequence length of the first audio data, and therefore, the second mel-spectrogram can be understood as a blank mel-spectrogram with a certain time sequence length, which is the content to be predicted by the speech conversion model after training.

[0122] Based on this, the first feature data and the third feature data are spliced to generate the fifth feature data, and the second feature data and the fourth feature data are spliced to generate the sixth feature data, which can specifically include: splicing the filled third feature data to the filled first feature data to generate the fifth feature data, and splicing the filled fourth feature data to the filled second feature data to generate the sixth feature data. Specifically, the filled third feature data is spliced before the filled first feature data in the time sequence dimension, and the filled fourth feature data is spliced before the filled second feature data in the time sequence dimension. The fifth feature data and the sixth feature data after splicing are equal in length, and the time sequence length corresponding to the first mel-spectrogram is equal to the time sequence length corresponding to the second mel-spectrogram.

[0123] In an optional implementation, before the step of converting the first audio data of the to-be-converted speaker into the second audio data of the specified speaker according to the fifth feature data, the sixth feature data, the first mel-spectrogram corresponding to the sample audio data, and the voice conversion model after training, the method provided in this embodiment can further include the following steps S31 to S32:

[0124] Step S31: Splice the first mel-spectrogram to the second mel-spectrogram to generate a spliced mel-spectrogram, and the second mel-spectrogram is a full-mask mel-spectrogram preset according to the time sequence length of the first audio data.

[0125] Step S32: Splice the fifth feature data, the sixth feature data, and the spliced mel-spectrogram in the channel dimension to obtain spliced feature data.

[0126] In this implementation, the first mel-spectrogram and the second mel-spectrogram are also spliced, specifically, the first mel-spectrogram is spliced before the second mel-spectrogram in the time sequence dimension to generate a spliced mel-spectrogram. It can be understood that predicting the second mel-spectrogram through the voice conversion model after training is actually a continuation of the first mel-spectrogram.

[0127] Based on the above steps, the fifth feature data, the sixth feature data, and the spliced mel-spectrogram obtained by splicing are spliced in the channel dimension to form a combined feature, which is defined as spliced feature data in this embodiment.

[0128] Based on this, in an optional implementation, the first audio data of the to-be-converted speaker is converted into the second audio data of the specified speaker according to the fifth feature data, the sixth feature data, the first mel-spectrogram corresponding to the sample audio data, and the voice conversion model after training, which can specifically include the following steps S41 to S42:

[0129] Step S41, input the spliced feature data into the trained speech conversion model to obtain a predicted mel-spectrogram corresponding to the second mel-spectrogram output by the trained speech conversion model.

[0130] Step S42, perform audio conversion on the predicted mel-spectrogram to generate second audio data.

[0131] After inputting the spliced feature data into the trained speech conversion model, the model will predict the second mel-spectrogram with full mask and output the prediction result of the second mel-spectrogram. In this embodiment, the prediction result of the second mel-spectrogram is defined as a predicted mel-spectrogram. After obtaining the predicted mel-spectrogram, the pre-trained vocoder can be used to synthesize the audio waveform corresponding to the predicted mel-spectrogram, i.e., the second audio data. The function of the vocoder is to convert the mel-spectrogram into corresponding audio data. The pre-trained vocoder can be a pre-trained one by the developer or an open source model, which is not limited in particular. Converting the mel-spectrogram into audio data based on the vocoder is relatively mature, and will not be described in detail here.

[0132] In an optional implementation, when the first audio data is audio data such as singing with strong expressiveness and rich rhythm, in order to enhance the conversion effect of the trained speech conversion model, the fundamental frequency data can also be extracted from the first audio data, and the fundamental frequency data is used as an input data to ensure the stability and accuracy of the pronunciation of the second audio data. Based on this, the first feature data and the second feature data are extracted from the first audio data, and the third feature data and the fourth feature data are extracted from the sample audio data. Specifically, the first feature data, the second feature data, and the first fundamental frequency data are extracted from the first audio data, and the third feature data, the fourth feature data, and the second fundamental frequency data are extracted from the sample audio data. Then, while splicing the first feature data and the third feature data to generate the fifth feature data, and splicing the second feature data and the fourth feature data to generate the sixth feature data, the first fundamental frequency data and the second fundamental frequency data are spliced to generate the third fundamental frequency data. Specifically, the second fundamental frequency data is spliced before the first fundamental frequency data in the time sequence dimension to generate the third fundamental frequency data. Further, the first audio data of the to-be-converted speaker can be converted into the second audio data of the specified speaker according to the fifth feature data, the sixth feature data, the third fundamental frequency data, the first mel-spectrogram corresponding to the sample audio data, and the trained speech conversion model.

[0133] In an optional implementation, to further improve the pronunciation stability and accuracy of the second audio data, multi-scale fundamental frequency modeling can be performed on the fundamental frequency data. The multi-scale fundamental frequency modeling has been described in the first embodiment of the present application, and will not be described here again. Specifically, before the steps of splicing the first feature data and the third feature data to generate the fifth feature data, splicing the second feature data and the fourth feature data to generate the sixth feature data, and splicing the first fundamental frequency data and the second fundamental frequency data to generate the third fundamental frequency data, the method provided in this embodiment can further include: performing multi-scale fundamental frequency modeling on the first fundamental frequency data and the second fundamental frequency data to obtain first multi-scale fundamental frequency data corresponding to the first fundamental frequency data and second multi-scale fundamental frequency data corresponding to the second fundamental frequency data. Then, while splicing the first feature data and the third feature data to generate the fifth feature data, and splicing the second feature data and the fourth feature data to generate the sixth feature data, the first multi-scale fundamental frequency data and the second multi-scale fundamental frequency data are spliced to generate third multi-scale fundamental frequency data. Further, the first audio data of the to-be-converted speaker is converted into the second audio data of the specified speaker according to the fifth feature data, the sixth feature data, the third multi-scale fundamental frequency data, the first mel spectrogram corresponding to the sample audio data, and the speech conversion model completed training.

[0134] The following illustrates how to convert the first audio data of the to-be-converted speaker into the second audio data of the specified speaker based on the speech conversion model completed training in a specific implementation.

[0135] Converting the first audio data of the to-be-converted speaker into the second audio data of the specified speaker based on the speech conversion model completed training can specifically include the following steps S601 to S613:

[0136] Step S601, in response to receiving a speech conversion instruction, obtaining the first audio data of the to-be-converted speaker and the sample audio data of the specified speaker.

[0137] Step S602, inputting the first audio data into the pre-trained SAR model to obtain the first feature data corresponding to the first audio data; inputting the first audio data into the pre-trained SSL model to obtain the second feature data corresponding to the first audio data; and inputting the first audio data into the fundamental frequency tracking algorithm to obtain the first fundamental frequency data corresponding to the first audio data.

[0138] Step S603, inputting the sample audio data into the pre-trained SAR model to obtain the third feature data corresponding to the sample audio data; inputting the sample audio data into the pre-trained SSL model to obtain the fourth feature data corresponding to the sample audio data; and inputting the sample audio data into the fundamental frequency tracking algorithm to obtain the second fundamental frequency data corresponding to the sample audio data.

[0139] In step S604, the first feature data and the third feature data are input into the content feature encoder for dimension reduction preprocessing to obtain first representation data corresponding to the first feature data and third representation data corresponding to the third feature data.

[0140] In step S605, the second feature data and the fourth feature data are input into the global feature encoder for dimension reduction preprocessing to obtain second representation data corresponding to the second feature data and fourth representation data corresponding to the fourth feature data.

[0141] In step S606, the first fundamental frequency data and the second fundamental frequency data are input into the multi-scale fundamental frequency modeling encoder for multi-scale modeling to obtain first multi-scale fundamental frequency data corresponding to the first fundamental frequency data and second multi-scale fundamental frequency data corresponding to the second fundamental frequency data.

[0142] In step S607, the first mel-spectrogram corresponding to the sample audio data is obtained.

[0143] In step S608, the second mel-spectrogram with a full mask is constructed according to the time sequence length of the first audio data.

[0144] In step S609, the first representation data and the second representation data are padded to be equal in length to the second mel-spectrogram, and the third representation data and the fourth representation data are padded to be equal in length to the first mel-spectrogram.

[0145] In step S610, the third representation data after padding is spliced in the time sequence dimension before the first representation data after padding to generate fifth feature data, the fourth representation data after padding is spliced in the time sequence dimension before the second representation data after padding to generate sixth feature data, the second multi-scale fundamental frequency data is spliced in the time sequence dimension before the first multi-scale fundamental frequency data to generate third multi-scale fundamental frequency data, and the first mel-spectrogram is spliced in the time sequence dimension before the second mel-spectrogram to generate a spliced mel-spectrogram.

[0146] In step S611, the fifth feature data, the sixth feature data, the third multi-scale fundamental frequency data, and the spliced mel-spectrogram are spliced in the channel dimension to generate spliced feature data.

[0147] In step S612, the spliced feature data is input into the speech conversion model after training to obtain a predicted mel-spectrogram output by the model.

[0148] In step S613, the predicted mel-spectrogram is input into the pre-trained vocoder to obtain second audio data converted by the vocoder.

[0149] The second embodiment provides an optional voice conversion method. The method selects the combination of content features (i.e., ASR features extracted by an ASR model) and global features (i.e., SSL features extracted by an SSL model), and provides sufficient content information and acoustic information. On this basis, multi-scale fundamental frequency modeling is added to improve the pronunciation stability of the conversion result, and the performance on singing conversion is particularly obvious. It should be noted that the voice conversion method provided in the embodiment is different from the current conversion scheme based on a voice conversion model. The method provided in the embodiment cancels the strong prior information that the content information and the acoustic information are completely aligned in time sequence, and the voice conversion model needs to complete the alignment modeling between the content information, the acoustic information, and the predicted result by itself. This not only facilitates the voice conversion model to migrate the pronunciation characteristics and style of a specified speaker, but also effectively reduces the attention of the voice conversion model to background noise, thereby realizing a robust and high-performance voice conversion effect.

[0150] It should be noted that the examples in the second embodiment are only for the explanation of the method described in the application and do not limit the actual use. The voice conversion method provided in the application includes but is not limited to the method described in the second embodiment.

[0151] The third embodiment of the application provides a voice conversion model training device. Figure 6 is a structural schematic diagram of the voice conversion model training device provided in the embodiment.

[0152] As Figure 6 shown, the voice conversion model training device provided in the embodiment includes a first data processing unit 601, a second data processing unit 602, a model training unit 603, and a model parameter adjustment unit 604.

[0153] The first data processing unit 601 is configured to obtain a plurality of audio training data, and extract first feature training data and second feature training data from the audio training data. The first feature training data is used to represent the content features corresponding to the audio training data, and the second feature training data is used to represent the global features corresponding to the audio training data.

[0154] Optionally, the first feature training data and the second feature training data are extracted from the audio training data, including:

[0155] The audio training data is input into a pre-trained voice recognition model to obtain the content features generated by the voice recognition model in the inference process of the audio training data. The content features are used as the first feature training data.

[0156] input the audio training data into a pre-trained self-supervised learning model to obtain global features generated by the self-supervised learning model after inference on the audio training data, and use the global features as the second feature training data.

[0157] The second data processing unit 602 is configured to obtain a mel spectrogram corresponding to the audio training data, and perform local mask processing and noise adding processing on the mel spectrogram to obtain a masked mel spectrogram and a noise mel spectrogram corresponding to the mel spectrogram.

[0158] The model training unit 603 is configured to input the first feature training data, the second feature training data, the masked mel spectrogram, and the noise mel spectrogram corresponding to the audio training data into a speech conversion model to be trained to obtain a predicted mel spectrogram output by the speech conversion model to be trained.

[0159] Optionally, before the step of inputting the first feature training data, the second feature training data, the masked mel spectrogram, and the noise mel spectrogram corresponding to the audio training data into the speech conversion model to be trained to obtain a predicted mel spectrogram output by the speech conversion model to be trained, the method further includes:

[0160] performing dimension reduction processing of a preset channel dimension on the first feature training data and the second feature training data corresponding to the audio training data respectively to obtain first representation training data corresponding to the first feature training data and second representation training data corresponding to the second feature training data;

[0161] filling the first representation training data and the second representation training data to be equal in length to the mel spectrogram according to a time sequence length of the mel spectrogram corresponding to the audio training data;

[0162] splicing the filled first representation training data, the filled second representation training data, the masked mel spectrogram, and the noise mel spectrogram in the channel dimension to obtain spliced feature training data corresponding to the audio training data;

[0163] The inputting of the first feature training data, the second feature training data, the masked mel spectrogram, and the noise mel spectrogram corresponding to the audio training data into the speech conversion model to be trained to obtain a predicted mel spectrogram output by the speech conversion model to be trained includes:

[0164] inputting the spliced feature training data corresponding to the audio training data into the speech conversion model to be trained to obtain the predicted mel spectrogram output by the speech conversion model to be trained.

[0165] Optionally, the extracting the first feature training data and the second feature training data from the audio training data comprises:

[0166] extracting the first feature training data, the second feature training data, and fundamental frequency data from the audio training data;

[0167] the inputting the first feature training data, the second feature training data, the mask mel-spectrogram, and the noise mel-spectrogram corresponding to the audio training data into the speech conversion model to be trained, and obtaining a predicted mel-spectrogram output by the speech conversion model to be trained comprises:

[0168] the inputting the first feature training data, the second feature training data, the fundamental frequency data, the mask mel-spectrogram, and the noise mel-spectrogram corresponding to the audio training data into the speech conversion model to be trained, and obtaining the predicted mel-spectrogram output by the speech conversion model to be trained.

[0169] Optionally, before the step of inputting the first feature training data, the second feature training data, the fundamental frequency data, the mask mel-spectrogram, and the noise mel-spectrogram corresponding to the audio training data into the speech conversion model to be trained, and obtaining the predicted mel-spectrogram output by the speech conversion model to be trained, the method further comprises:

[0170] performing multi-scale fundamental frequency modeling on the fundamental frequency data corresponding to the audio training data to obtain multi-scale fundamental frequency data corresponding to the audio training data;

[0171] the inputting the first feature training data, the second feature training data, the fundamental frequency data, the mask mel-spectrogram, and the noise mel-spectrogram corresponding to the audio training data into the speech conversion model to be trained, and obtaining the predicted mel-spectrogram output by the speech conversion model to be trained comprises:

[0172] the inputting the first feature training data, the second feature training data, the multi-scale fundamental frequency data, the mask mel-spectrogram, and the noise mel-spectrogram corresponding to the audio training data into the speech conversion model to be trained, and obtaining the predicted mel-spectrogram output by the speech conversion model to be trained.

[0173] Optionally, after the step of extracting the first feature training data and the second feature training data from the audio training data, the method further comprises:

[0174] removing part of the second feature training data corresponding to the audio training data according to a preset proportion;

[0175] The step of inputting the first feature training data, the second feature training data, the masked Mel spectrogram, and the noise Mel spectrogram corresponding to the audio training data into the speech conversion model to be trained, and obtaining the predicted Mel spectrogram output by the speech conversion model to be trained, includes:

[0176] The first feature training data corresponding to the audio training data, the masked Mel spectrogram, and the noise Mel spectrogram are input into the speech conversion model to be trained to obtain the predicted Mel spectrogram output by the speech conversion model to be trained.

[0177] The model parameter adjustment unit 604 is used to adjust the model parameters of the speech conversion model to be trained according to the Mel spectrogram and the predicted Mel spectrogram corresponding to the audio training data, so as to obtain the trained speech conversion model.

[0178] Optionally, the device further includes a determination unit; the determination unit is used for:

[0179] Determine whether the trained speech conversion model has achieved the preset training objective;

[0180] In response to the trained speech conversion model achieving the training objective, the trained speech conversion model is used as the completed speech conversion model for speech conversion of the audio data to be converted.

[0181] If the trained speech conversion model fails to meet the training objective, the trained speech conversion model is used as the speech conversion model to be trained, and iterative training continues.

[0182] The fourth embodiment of this application provides a speech conversion device. Figure 7 This is a schematic diagram of the speech conversion device provided in this embodiment.

[0183] like Figure 7 As shown, the speech conversion device provided in this embodiment includes: a data acquisition unit 701, a feature extraction unit 702, a feature splicing unit 703, and a data conversion unit 704.

[0184] The data acquisition unit 701 is used to acquire the first audio data of the speaker to be converted and the sample audio data of the specified speaker in response to the speech conversion instruction.

[0185] The feature extraction unit 702 is configured to extract first feature data and second feature data from the first audio data, and extract third feature data and fourth feature data from the sample audio data; wherein the first feature data is used to represent content features corresponding to the first audio data, the second feature data is used to represent global features corresponding to the first audio data, the third feature data is used to represent content features corresponding to the sample audio data, and the fourth feature data is used to represent global features corresponding to the sample audio data.

[0186] Optionally, the extracting the first feature data and the second feature data from the first audio data, and the extracting the third feature data and the fourth feature data from the sample audio data comprises:

[0187] inputting the first audio data into a pre-trained speech recognition model to obtain content features generated by the speech recognition model in an inference process of the first audio data, and taking the content features as the first feature data;

[0188] inputting the first audio data into a pre-trained self-supervised learning model to obtain global features generated by the self-supervised learning model after inference of the first audio data, and taking the global features as the second feature data;

[0189] inputting the sample audio data into the speech recognition model to obtain content features generated by the speech recognition model in an inference process of the sample audio data, and taking the content features as the third feature data;

[0190] inputting the sample audio data into the self-supervised learning model to obtain global features generated by the self-supervised learning model after inference of the sample audio data, and taking the global features as the fourth feature data.

[0191] The feature splicing unit 703 is configured to splice the first feature data and the third feature data to generate fifth feature data, and splice the second feature data and the fourth feature data to generate sixth feature data.

[0192] Optionally, before the splicing the first feature data and the third feature data to generate fifth feature data, and the splicing the second feature data and the fourth feature data to generate sixth feature data, the method further comprises:

[0193] performing dimension reduction processing on the first feature data and the third feature data in a first preset channel dimension to obtain first representation data corresponding to the first feature data and third representation data corresponding to the third feature data;

[0194] performing dimension reduction processing of a second preset channel dimension on the second feature data and the fourth feature data, to obtain second representation data corresponding to the second feature data, and fourth representation data corresponding to the fourth feature data;

[0195] According to the time sequence length of the first mel-spectrogram, the third representation data and the fourth representation data are padded to be equal in length to the first mel-spectrogram;

[0196] According to the time sequence length of the second mel-spectrogram, the first representation data and the second representation data are padded to be equal in length to the second mel-spectrogram, the second mel-spectrogram being a full mask mel-spectrogram preset according to the time sequence length of the first audio data;

[0197] The first feature data and the third feature data are spliced to generate fifth feature data, and the second feature data and the fourth feature data are spliced to generate sixth feature data, including:

[0198] The padded third representation data is spliced before the padded first representation data to generate the fifth feature data, and the padded fourth representation data is spliced before the padded second representation data to generate the sixth feature data.

[0199] The data conversion unit 704 is configured to convert the first audio data of the to-be-converted speaker into the second audio data of the specified speaker according to the fifth feature data, the sixth feature data, the first mel-spectrogram corresponding to the sample audio data, and a trained voice conversion model.

[0200] Optionally, the first feature data and the second feature data are extracted from the first audio data, and the third feature data and the fourth feature data are extracted from the sample audio data, including:

[0201] The first feature data, the second feature data, and first fundamental frequency data are extracted from the first audio data, and the third feature data, the fourth feature data, and second fundamental frequency data are extracted from the sample audio data;

[0202] The first feature data and the third feature data are spliced to generate fifth feature data, and the second feature data and the fourth feature data are spliced to generate sixth feature data, including:

[0203] concatenate the first feature data and the third feature data to generate fifth feature data, concatenate the second feature data and the fourth feature data to generate sixth feature data, and concatenate the first fundamental frequency data and the second fundamental frequency data to generate third fundamental frequency data;

[0204] The first audio data of the to-be-converted speaker is converted into the second audio data of the specified speaker according to the fifth feature data, the sixth feature data, the first mel-spectrogram corresponding to the sample audio data, and the voice conversion model trained.

[0205] The first audio data of the to-be-converted speaker is converted into the second audio data of the specified speaker according to the fifth feature data, the sixth feature data, the third fundamental frequency data, the first mel-spectrogram corresponding to the sample audio data, and the voice conversion model trained.

[0206] Optionally, before the step of concatenating the first feature data and the third feature data to generate the fifth feature data, concatenating the second feature data and the fourth feature data to generate the sixth feature data, and concatenating the first fundamental frequency data and the second fundamental frequency data to generate the third fundamental frequency data, the method further includes:

[0207] The first fundamental frequency data and the second fundamental frequency data are subjected to multi-scale fundamental frequency modeling to obtain first multi-scale fundamental frequency data corresponding to the first fundamental frequency data and second multi-scale fundamental frequency data corresponding to the second fundamental frequency data.

[0208] The first feature data and the third feature data are concatenated to generate the fifth feature data, the second feature data and the fourth feature data are concatenated to generate the sixth feature data, and the first fundamental frequency data and the second fundamental frequency data are concatenated to generate the third fundamental frequency data.

[0209] The first feature data and the third feature data are concatenated to generate the fifth feature data, the second feature data and the fourth feature data are concatenated to generate the sixth feature data, and the first multi-scale fundamental frequency data and the second multi-scale fundamental frequency data are concatenated to generate third multi-scale fundamental frequency data.

[0210] The first audio data of the to-be-converted speaker is converted into the second audio data of the specified speaker according to the fifth feature data, the sixth feature data, the first mel-spectrogram corresponding to the sample audio data, and the voice conversion model trained.

[0211] According to the fifth feature data, the sixth feature data, the third multi-scale fundamental frequency data, the first mel spectrogram corresponding to the sample audio data, and the voice conversion model trained, the first audio data of the to-be-converted speaker is converted into the second audio data of the specified speaker.

[0212] Optionally, before the step of converting the first audio data of the to-be-converted speaker into the second audio data of the specified speaker according to the fifth feature data, the sixth feature data, the first mel spectrogram corresponding to the sample audio data, and the voice conversion model trained, the method further comprises:

[0213] generating a spliced mel spectrogram before splicing the first mel spectrogram to a second mel spectrogram, the second mel spectrogram being a full-mask mel spectrogram preset according to the time length of the first audio data;

[0214] splicing the fifth feature data, the sixth feature data, and the spliced mel spectrogram in the channel dimension to obtain spliced feature data.

[0215] Optionally, the step of converting the first audio data of the to-be-converted speaker into the second audio data of the specified speaker according to the fifth feature data, the sixth feature data, the first mel spectrogram corresponding to the sample audio data, and the voice conversion model trained, comprises:

[0216] inputting the spliced feature data into the voice conversion model trained to obtain a predicted mel spectrogram corresponding to the second mel spectrogram output by the voice conversion model trained;

[0217] performing audio conversion on the predicted mel spectrogram to generate the second audio data.

[0218] The fifth embodiment of the present application provides an electronic device, Figure 8 is a structural schematic diagram of the electronic device provided by the present embodiment.

[0219] As shown in Figure 8 the electronic device provided by the present embodiment comprises a memory 801 and a processor 802.

[0220] The memory 801 is configured to store computer instructions for executing a voice conversion model training method and / or a voice conversion method.

[0221] The processor 802 is configured to execute the computer instructions stored in the memory 801 to perform the following operations:

[0222] obtaining a plurality of audio training data, and extracting first feature training data and second feature training data from the audio training data; wherein the first feature training data is used to represent content features corresponding to the audio training data, and the second feature training data is used to represent global features corresponding to the audio training data;

[0223] obtaining a mel spectrogram corresponding to the audio training data, and performing local mask processing and noise processing on the mel spectrogram to obtain a mask mel spectrogram and a noise mel spectrogram corresponding to the mel spectrogram;

[0224] inputting the first feature training data, the second feature training data, the mask mel spectrogram, and the noise mel spectrogram corresponding to the audio training data into a to-be-trained speech conversion model to obtain a predicted mel spectrogram output by the to-be-trained speech conversion model;

[0225] According to the mel spectrogram corresponding to the audio training data and the predicted mel spectrogram, the model parameters of the to-be-trained speech conversion model are adjusted to obtain a trained speech conversion model.

[0226] Optionally, the following is also performed:

[0227] determining whether the trained speech conversion model reaches a preset training target;

[0228] In response to the trained speech conversion model reaching the training target, the trained speech conversion model is used as a completed training speech conversion model for speech conversion on to-be-converted audio data;

[0229] In response to the trained speech conversion model not reaching the training target, the trained speech conversion model is used as the to-be-trained speech conversion model for continued iterative training.

[0230] Optionally, the first feature training data and the second feature training data are extracted from the audio training data, including:

[0231] inputting the audio training data into a pre-trained speech recognition model to obtain content features generated by the speech recognition model during inference on the audio training data, and using the content features as the first feature training data;

[0232] inputting the audio training data into a pre-trained self-supervised learning model to obtain global features generated by the self-supervised learning model after inference on the audio training data, and using the global features as the second feature training data.

[0233] Optionally, before the step of inputting the first feature training data, the second feature training data, the mask mel-spectrogram, and the noise mel-spectrogram corresponding to the audio training data into a speech conversion model to be trained, and obtaining a predicted mel-spectrogram output by the speech conversion model to be trained, the following steps are further performed:

[0234] performing dimension reduction processing of a preset channel dimension on the first feature training data and the second feature training data corresponding to the audio training data respectively, to obtain first representation training data corresponding to the first feature training data, and second representation training data corresponding to the second feature training data;

[0235] filling the first representation training data and the second representation training data to be equal in length to the mel-spectrogram according to a time sequence length of the mel-spectrogram corresponding to the audio training data;

[0236] concatenating the filled first representation training data, the filled second representation training data, the mask mel-spectrogram, and the noise mel-spectrogram in the channel dimension, to obtain concatenation feature training data corresponding to the audio training data;

[0237] The step of inputting the first feature training data, the second feature training data, the mask mel-spectrogram, and the noise mel-spectrogram corresponding to the audio training data into a speech conversion model to be trained, and obtaining a predicted mel-spectrogram output by the speech conversion model to be trained, includes:

[0238] inputting the concatenation feature training data corresponding to the audio training data into the speech conversion model to be trained, to obtain the predicted mel-spectrogram output by the speech conversion model to be trained.

[0239] Optionally, the step of extracting first feature training data and second feature training data from the audio training data includes:

[0240] extracting the first feature training data, the second feature training data, and fundamental frequency data from the audio training data;

[0241] The step of inputting the first feature training data, the second feature training data, the mask mel-spectrogram, and the noise mel-spectrogram corresponding to the audio training data into a speech conversion model to be trained, and obtaining a predicted mel-spectrogram output by the speech conversion model to be trained, includes:

[0242] input the first feature training data, the second feature training data, the fundamental frequency data, the mask mel-spectrogram, and the noise mel-spectrogram corresponding to the audio training data into the speech conversion model to be trained to obtain the predicted mel-spectrogram output by the speech conversion model to be trained.

[0243] Optionally, before the step of inputting the first feature training data, the second feature training data, the fundamental frequency data, the mask mel-spectrogram, and the noise mel-spectrogram corresponding to the audio training data into the speech conversion model to be trained to obtain the predicted mel-spectrogram output by the speech conversion model to be trained, the following step is further performed:

[0244] performing multi-scale fundamental frequency modeling on the fundamental frequency data corresponding to the audio training data to obtain multi-scale fundamental frequency data corresponding to the audio training data;

[0245] The step of inputting the first feature training data, the second feature training data, the fundamental frequency data, the mask mel-spectrogram, and the noise mel-spectrogram corresponding to the audio training data into the speech conversion model to be trained to obtain the predicted mel-spectrogram output by the speech conversion model to be trained includes:

[0246] The step of inputting the first feature training data, the second feature training data, the fundamental frequency data, the mask mel-spectrogram, and the noise mel-spectrogram corresponding to the audio training data into the speech conversion model to be trained to obtain the predicted mel-spectrogram output by the speech conversion model to be trained includes:

[0247] Optionally, after the step of extracting the first feature training data and the second feature training data from the audio training data, the following step is further performed:

[0248] According to a preset proportion, removing part of the second feature training data corresponding to the audio training data;

[0249] The step of inputting the first feature training data, the second feature training data, the mask mel-spectrogram, and the noise mel-spectrogram corresponding to the audio training data into the speech conversion model to be trained to obtain the predicted mel-spectrogram output by the speech conversion model to be trained includes:

[0250] The step of inputting the first feature training data, the second feature training data, the mask mel-spectrogram, and the noise mel-spectrogram corresponding to the audio training data into the speech conversion model to be trained to obtain the predicted mel-spectrogram output by the speech conversion model to be trained includes:

[0251] Or, the following operations are performed:

[0252] In response to the voice conversion instruction, first audio data of a to-be-converted speaker and sample audio data of a specified speaker are acquired;

[0253] First feature data and second feature data are extracted from the first audio data, and third feature data and fourth feature data are extracted from the sample audio data; wherein the first feature data is used to represent the content feature corresponding to the first audio data, the second feature data is used to represent the global feature corresponding to the first audio data, the third feature data is used to represent the content feature corresponding to the sample audio data, and the fourth feature data is used to represent the global feature corresponding to the sample audio data;

[0254] The first feature data and the third feature data are spliced to generate fifth feature data, and the second feature data and the fourth feature data are spliced to generate sixth feature data;

[0255] According to the fifth feature data, the sixth feature data, the first mel-spectrogram corresponding to the sample audio data, and the trained voice conversion model, the first audio data of the to-be-converted speaker is converted into the second audio data of the specified speaker; wherein the trained voice conversion model is obtained according to the training method of the voice conversion model.

[0256] Optionally, the first feature data and the second feature data are extracted from the first audio data, and the third feature data and the fourth feature data are extracted from the sample audio data, comprising:

[0257] The first audio data is input into a pre-trained speech recognition model to obtain the content feature generated by the speech recognition model in the inference process of the first audio data, and the content feature is taken as the first feature data;

[0258] The first audio data is input into a pre-trained self-supervised learning model to obtain the global feature generated by the self-supervised learning model after inference on the first audio data, and the global feature is taken as the second feature data;

[0259] The sample audio data is input into the speech recognition model to obtain the content feature generated by the speech recognition model in the inference process of the sample audio data, and the content feature is taken as the third feature data;

[0260] The sample audio data is input into the self-supervised learning model to obtain the global feature generated by the self-supervised learning model after inference on the sample audio data, and the global feature is taken as the fourth feature data.

[0261] Optionally, before the step of splicing the first feature data and the third feature data to generate fifth feature data, and splicing the second feature data and the fourth feature data to generate sixth feature data, the following steps are further performed:

[0262] performing dimension reduction processing on the first feature data and the third feature data in a first preset channel dimension, to obtain first representation data corresponding to the first feature data, and third representation data corresponding to the third feature data;

[0263] performing dimension reduction processing on the second feature data and the fourth feature data in a second preset channel dimension, to obtain second representation data corresponding to the second feature data, and fourth representation data corresponding to the fourth feature data;

[0264] filling the third representation data and the fourth representation data to be equal in length to the first mel-spectrogram according to a time sequence length of the first mel-spectrogram;

[0265] filling the first representation data and the second representation data to be equal in length to a second mel-spectrogram according to a time sequence length of the second mel-spectrogram, the second mel-spectrogram being a full-mask mel-spectrogram preset according to a time sequence length of the first audio data;

[0266] the splicing the first feature data and the third feature data to generate fifth feature data, and splicing the second feature data and the fourth feature data to generate sixth feature data, includes:

[0267] splicing the filled third representation data to the filled first representation data to generate the fifth feature data, and splicing the filled fourth representation data to the filled second representation data to generate the sixth feature data.

[0268] Optionally, the extracting the first feature data and the second feature data from the first audio data, and extracting the third feature data and the fourth feature data from the sample audio data, includes:

[0269] extracting the first feature data, the second feature data, and first fundamental frequency data from the first audio data, and extracting the third feature data, the fourth feature data, and second fundamental frequency data from the sample audio data;

[0270] the splicing the first feature data and the third feature data to generate fifth feature data, and splicing the second feature data and the fourth feature data to generate sixth feature data, includes:

[0271] concatenate the first feature data and the third feature data to generate fifth feature data, concatenate the second feature data and the fourth feature data to generate sixth feature data, and concatenate the first fundamental frequency data and the second fundamental frequency data to generate third fundamental frequency data;

[0272] The first audio data of the to-be-converted speaker is converted into the second audio data of the specified speaker according to the fifth feature data, the sixth feature data, the first mel-spectrogram corresponding to the sample audio data, and the voice conversion model trained.

[0273] The first audio data of the to-be-converted speaker is converted into the second audio data of the specified speaker according to the fifth feature data, the sixth feature data, the third fundamental frequency data, the first mel-spectrogram corresponding to the sample audio data, and the voice conversion model trained.

[0274] Optionally, before the step of concatenating the first feature data and the third feature data to generate the fifth feature data, concatenating the second feature data and the fourth feature data to generate the sixth feature data, and concatenating the first fundamental frequency data and the second fundamental frequency data to generate the third fundamental frequency data, the following step is further performed:

[0275] The first fundamental frequency data and the second fundamental frequency data are subjected to multi-scale fundamental frequency modeling to obtain first multi-scale fundamental frequency data corresponding to the first fundamental frequency data and second multi-scale fundamental frequency data corresponding to the second fundamental frequency data.

[0276] The first feature data and the third feature data are concatenated to generate the fifth feature data, the second feature data and the fourth feature data are concatenated to generate the sixth feature data, and the first multi-scale fundamental frequency data and the second multi-scale fundamental frequency data are concatenated to generate third multi-scale fundamental frequency data.

[0277] The first feature data and the third feature data are concatenated to generate the fifth feature data, the second feature data and the fourth feature data are concatenated to generate the sixth feature data, and the first multi-scale fundamental frequency data and the second multi-scale fundamental frequency data are concatenated to generate third multi-scale fundamental frequency data.

[0278] The first audio data of the to-be-converted speaker is converted into the second audio data of the specified speaker according to the fifth feature data, the sixth feature data, the first mel-spectrogram corresponding to the sample audio data, and the voice conversion model trained.

[0279] According to the fifth feature data, the sixth feature data, the third multi-scale fundamental frequency data, the first mel spectrogram corresponding to the sample audio data, and the voice conversion model completed by training, the first audio data of the to-be-converted speaker is converted into the second audio data of the specified speaker.

[0280] Optionally, before the step of converting the first audio data of the to-be-converted speaker into the second audio data of the specified speaker according to the fifth feature data, the sixth feature data, the first mel spectrogram corresponding to the sample audio data, and the voice conversion model completed by training, the following step is further performed:

[0281] A spliced mel spectrogram is generated before the first mel spectrogram is spliced into the second mel spectrogram, the second mel spectrogram being a full-mask mel spectrogram preset according to the time sequence length of the first audio data;

[0282] The fifth feature data, the sixth feature data, and the spliced mel spectrogram are spliced in the channel dimension to obtain spliced feature data.

[0283] Optionally, the step of converting the first audio data of the to-be-converted speaker into the second audio data of the specified speaker according to the fifth feature data, the sixth feature data, the first mel spectrogram corresponding to the sample audio data, and the voice conversion model completed by training includes:

[0284] The spliced feature data is input into the voice conversion model completed by training to obtain a predicted mel spectrogram corresponding to the second mel spectrogram output by the voice conversion model completed by training;

[0285] The predicted mel spectrogram is subjected to audio conversion to generate the second audio data.

[0286] The sixth embodiment of the present application provides a computer readable storage medium, which includes computer instructions, and the computer instructions are used to implement the method described in the embodiments of the present application when executed by a processor.

[0287] It should be noted that the terms such as "first", "second", and the like in the present document are only used to distinguish one entity or operation from another entity or operation, and do not require or imply any actual relationship or sequence between the entities or operations. In addition, the terms "include", "have", "contain", and "comprise" and other similar forms are identical in meaning, and the end of any one or more items following any one of the above terms is open, and any one of the above terms does not mean that the one or more items have been listed exhaustively or are limited to the one or more items listed.

[0288] As used herein, the term "or" includes any and all possible combinations of the associated items, unless the context clearly dictates otherwise. For example, if a database is stated as including A or B, then the database can include A; or B; or both A and B, unless the context clearly dictates otherwise or the combination is impossible. As a second example, if a database is stated as including A, B, or C, then the database can include A; or B; or C; or A and B; or A and C; or B and C; or A and B and C, unless the context clearly dictates otherwise or the combination is impossible.

[0289] It is noted that the above examples can be implemented by hardware or software (program code), or a combination of hardware and software. If implemented in software, the software can be stored in a computer readable medium, as described above. The software, when executed by a processor, can perform the disclosed methods. The computing units and other functional units described in this disclosure can be implemented by hardware or software, or a combination of hardware and software. Those of ordinary skill in the art will also appreciate that the various modules / units described above can be combined into one module / unit, and each of the modules / units described above can be further divided into multiple sub-modules / sub-units.

[0290] In the detailed description above, embodiments have been described with reference to numerous specific details that can vary from implementation to implementation. Certain adaptations and modifications of the described embodiments can occur to those skilled in the art. Other implementations can be apparent to those of ordinary skill in the art from the disclosure of the present application. The true scope of the application is not limited to the specific embodiments described herein, but only to those described by the language of the appended claims, and equivalents thereof. The sequence of steps shown in the drawings is also only for illustrative purposes and is not meant to limit the application to any particular sequence of steps, order. Accordingly, those skilled in the art will recognize that the steps could be performed in a different order, or performed concurrently.

[0291] In the drawings and specification, exemplary embodiments have been presented. However, one or more modifications can be made to these embodiments without departing from the spirit of the disclosure. Accordingly, although specific terms are employed, they are used in a generic and descriptive sense only and not for purposes of limitation.

Claims

1. A method for training a speech conversion model, the method comprising: The method further comprises: obtaining a plurality of audio training data, and extracting first feature training data and second feature training data from the audio training data; wherein the first feature training data is used to represent the content features corresponding to the audio training data, and the second feature training data is used to represent the global features corresponding to the audio training data; obtaining a mel spectrogram corresponding to the audio training data, and performing local mask processing and noise processing on the mel spectrogram to obtain a masked mel spectrogram and a noisy mel spectrogram corresponding to the mel spectrogram; inputting the first feature training data, the second feature training data, the masked mel spectrogram, and the noisy mel spectrogram corresponding to the audio training data into a to-be-trained speech conversion model to obtain a predicted mel spectrogram output by the to-be-trained speech conversion model; adjusting the model parameters of the to-be-trained speech conversion model according to the mel spectrogram and the predicted mel spectrogram corresponding to the audio training data to obtain a trained speech conversion model; wherein, before the step of inputting the first feature training data, the second feature training data, the masked mel spectrogram, and the noisy mel spectrogram corresponding to the audio training data into a to-be-trained speech conversion model to obtain a predicted mel spectrogram output by the to-be-trained speech conversion model, the method further comprises: performing dimension reduction processing of a preset channel dimension on the first feature training data and the second feature training data corresponding to the audio training data respectively to obtain first representation training data corresponding to the first feature training data, and second representation training data corresponding to the second feature training data; filling the first representation training data and the second representation training data to be equal in length to the mel spectrogram according to the time sequence length of the mel spectrogram corresponding to the audio training data; splicing the filled first representation training data, the filled second representation training data, the masked mel spectrogram, and the noisy mel spectrogram in the channel dimension to obtain spliced feature training data corresponding to the audio training data; the step of inputting the first feature training data, the second feature training data, the masked mel spectrogram, and the noisy mel spectrogram corresponding to the audio training data into a to-be-trained speech conversion model to obtain a predicted mel spectrogram output by the to-be-trained speech conversion model, comprises: inputting the spliced feature training data corresponding to the audio training data into the to-be-trained speech conversion model to obtain the predicted mel spectrogram output by the to-be-trained speech conversion model.

2. The method of claim 1, wherein, The method further comprises: determining whether the trained speech conversion model meets a preset training target; in response to the trained speech conversion model meeting the training target, taking the trained speech conversion model as a completed training speech conversion model for speech conversion on to-be-converted audio data; In response to the trained voice conversion model not meeting the training target, the trained voice conversion model is taken as the voice conversion model to be trained, and iterative training is continued.

3. The method of claim 1, wherein, The first feature training data and the second feature training data are extracted from the audio training data, including: inputting the audio training data into a pre-trained speech recognition model to obtain content features generated by the speech recognition model in an inference process on the audio training data, and taking the content features as the first feature training data; inputting the audio training data into a pre-trained self-supervised learning model to obtain global features generated by the self-supervised learning model after inference on the audio training data, and taking the global features as the second feature training data.

4. The method of claim 1, wherein, The first feature training data and the second feature training data are extracted from the audio training data, including: extracting the first feature training data, the second feature training data, and fundamental frequency data from the audio training data; The first feature training data, the second feature training data, the mask mel-spectrogram, and the noise mel-spectrogram corresponding to the audio training data are input into the voice conversion model to be trained to obtain a predicted mel-spectrogram output by the voice conversion model to be trained, including: The first feature training data, the second feature training data, the fundamental frequency data, the mask mel-spectrogram, and the noise mel-spectrogram corresponding to the audio training data are input into the voice conversion model to be trained to obtain the predicted mel-spectrogram output by the voice conversion model to be trained.

5. The method of claim 4, wherein, Before the step of inputting the first feature training data, the second feature training data, the fundamental frequency data, the mask mel-spectrogram, and the noise mel-spectrogram corresponding to the audio training data into the voice conversion model to be trained to obtain the predicted mel-spectrogram output by the voice conversion model to be trained, the method further includes: performing multi-scale fundamental frequency modeling on the fundamental frequency data corresponding to the audio training data to obtain multi-scale fundamental frequency data corresponding to the audio training data; The first feature training data, the second feature training data, the fundamental frequency data, the mask mel-spectrogram, and the noise mel-spectrogram corresponding to the audio training data are input into the voice conversion model to be trained to obtain the predicted mel-spectrogram output by the voice conversion model to be trained, including: The first feature training data, the second feature training data, the multi-scale fundamental frequency data, the mask mel-spectrogram, and the noise mel-spectrogram corresponding to the audio training data are input into the voice conversion model to be trained to obtain the predicted mel-spectrogram output by the voice conversion model to be trained.

6. The method of claim 1, wherein, After the step of extracting the first feature training data and the second feature training data from the audio training data, the method further includes: According to a preset proportion, part of the second feature training data corresponding to the audio training data is removed; The first feature training data, the second feature training data, the mask mel spectrogram, and the noise mel spectrogram corresponding to the audio training data are input into a to-be-trained speech conversion model, and a predicted mel spectrogram output by the to-be-trained speech conversion model is obtained. The first feature training data, the mask mel spectrogram, and the noise mel spectrogram corresponding to the audio training data are input into the to-be-trained speech conversion model, and the predicted mel spectrogram output by the to-be-trained speech conversion model is obtained.

7. A voice conversion method characterized by, The method comprises: In response to a speech conversion instruction, obtaining first audio data of a to-be-converted speaker and sample audio data of a specified speaker; Extracting first feature data and second feature data from the first audio data, and extracting third feature data and fourth feature data from the sample audio data; wherein the first feature data is used to represent the content feature corresponding to the first audio data, the second feature data is used to represent the global feature corresponding to the first audio data, the third feature data is used to represent the content feature corresponding to the sample audio data, and the fourth feature data is used to represent the global feature corresponding to the sample audio data; Splicing the first feature data and the third feature data to generate fifth feature data, and splicing the second feature data and the fourth feature data to generate sixth feature data; Converting the first audio data of the to-be-converted speaker into second audio data of the specified speaker according to the fifth feature data, the sixth feature data, a first mel spectrogram corresponding to the sample audio data, and a speech conversion model trained according to the method of any one of claims 1 to 6.

8. The method of claim 7, wherein, The first feature data and the second feature data are extracted from the first audio data, and the third feature data and the fourth feature data are extracted from the sample audio data, comprising: Inputting the first audio data into a pre-trained speech recognition model to obtain content features generated by the speech recognition model during inference on the first audio data, and taking the content features as the first feature data; Inputting the first audio data into a pre-trained self-supervised learning model to obtain global features generated by the self-supervised learning model after inference on the first audio data, and taking the global features as the second feature data; Inputting the sample audio data into the speech recognition model to obtain content features generated by the speech recognition model during inference on the sample audio data, and taking the content features as the third feature data; Inputting the sample audio data into the self-supervised learning model to obtain global features generated by the self-supervised learning model after inference on the sample audio data, and taking the global features as the fourth feature data.

9. The method of claim 7, wherein, Before the step of splicing the first feature data and the third feature data to generate fifth feature data, and splicing the second feature data and the fourth feature data to generate sixth feature data, the method further comprises: performing dimension reduction processing on the first feature data and the third feature data in a first preset channel dimension to obtain first representation data corresponding to the first feature data and third representation data corresponding to the third feature data; performing dimension reduction processing on the second feature data and the fourth feature data in a second preset channel dimension to obtain second representation data corresponding to the second feature data and fourth representation data corresponding to the fourth feature data; filling the third representation data and the fourth representation data to be equal in length to the first mel-spectrogram according to a time sequence length of the first mel-spectrogram; filling the first representation data and the second representation data to be equal in length to the second mel-spectrogram according to a time sequence length of the second mel-spectrogram, the second mel-spectrogram being a full mask mel-spectrogram preset according to a time sequence length of the first audio data; the step of splicing the first feature data and the third feature data to generate fifth feature data, and splicing the second feature data and the fourth feature data to generate sixth feature data, comprises: splicing the filled third representation data to the filled first representation data to generate the fifth feature data before the first representation data, and splicing the filled fourth representation data to the filled second representation data to generate the sixth feature data before the second representation data.

10. The method of claim 7, wherein, the step of extracting first feature data and second feature data from the first audio data, and extracting third feature data and fourth feature data from the sample audio data, comprises: extracting the first feature data, the second feature data, and first fundamental frequency data from the first audio data, and extracting the third feature data, the fourth feature data, and second fundamental frequency data from the sample audio data; the step of splicing the first feature data and the third feature data to generate fifth feature data, and splicing the second feature data and the fourth feature data to generate sixth feature data, comprises: splicing the first feature data and the third feature data to generate the fifth feature data, splicing the second feature data and the fourth feature data to generate the sixth feature data, and splicing the first fundamental frequency data and the second fundamental frequency data to generate third fundamental frequency data; the step of converting the first audio data of the to-be-converted speaker into the second audio data of the specified speaker according to the fifth feature data, the sixth feature data, the first mel-spectrogram corresponding to the sample audio data, and the voice conversion model completed by training, comprises: According to the fifth feature data, the sixth feature data, the third fundamental frequency data, the first mel-spectrogram corresponding to the sample audio data, and the voice conversion model trained, the first audio data of the to-be-converted speaker is converted into the second audio data of the specified speaker.

11. The method of claim 10, wherein, Before the steps of splicing the first feature data and the third feature data to generate the fifth feature data, splicing the second feature data and the fourth feature data to generate the sixth feature data, and splicing the first fundamental frequency data and the second fundamental frequency data to generate the third fundamental frequency data, the method further comprises: Multi-scale fundamental frequency modeling is performed on the first fundamental frequency data and the second fundamental frequency data to obtain first multi-scale fundamental frequency data corresponding to the first fundamental frequency data and second multi-scale fundamental frequency data corresponding to the second fundamental frequency data. The step of splicing the first feature data and the third feature data to generate the fifth feature data, splicing the second feature data and the fourth feature data to generate the sixth feature data, and splicing the first fundamental frequency data and the second fundamental frequency data to generate the third fundamental frequency data comprises: The step of splicing the first feature data and the third feature data to generate the fifth feature data, splicing the second feature data and the fourth feature data to generate the sixth feature data, and splicing the first fundamental frequency data and the second fundamental frequency data to generate the third fundamental frequency data comprises: The step of converting the first audio data of the to-be-converted speaker into the second audio data of the specified speaker according to the fifth feature data, the sixth feature data, the first mel-spectrogram corresponding to the sample audio data, and the voice conversion model trained comprises: The step of converting the first audio data of the to-be-converted speaker into the second audio data of the specified speaker according to the fifth feature data, the sixth feature data, the third multi-scale fundamental frequency data, the first mel-spectrogram corresponding to the sample audio data, and the voice conversion model trained.

12. The method of claim 7, wherein, Before the step of converting the first audio data of the to-be-converted speaker into the second audio data of the specified speaker according to the fifth feature data, the sixth feature data, the first mel-spectrogram corresponding to the sample audio data, and the voice conversion model trained, the method further comprises: A spliced mel-spectrogram is generated before the first mel-spectrogram is spliced to a second mel-spectrogram, the second mel-spectrogram being a full-mask mel-spectrogram preset according to the time length of the first audio data; The fifth feature data, the sixth feature data, and the spliced mel-spectrogram are spliced in the channel dimension to obtain spliced feature data.

13. The method of claim 12, wherein, The first audio data of the to-be-converted pronunciation person is converted into the second audio data of the specified pronunciation person according to the fifth feature data, the sixth feature data, the first mel spectrogram corresponding to the sample audio data, and the trained voice conversion model. The first audio data of the to-be-converted pronunciation person is converted into the second audio data of the specified pronunciation person according to the fifth feature data, the sixth feature data, the first mel spectrogram corresponding to the sample audio data, and the trained voice conversion model. The first audio data of the to-be-converted pronunciation person is converted into the second audio data of the specified pronunciation person according to the fifth feature data, the sixth feature data, the first mel spectrogram corresponding to the sample audio data, and the trained voice conversion model. 14.A device for training a speech conversion model, comprising: The device further comprises a first data processing unit, a second data processing unit, a model training unit, and a model parameter adjusting unit. The first data processing unit is configured to obtain a plurality of audio training data, and extract first feature training data and second feature training data from the audio training data; wherein the first feature training data is used to represent the content features corresponding to the audio training data, and the second feature training data is used to represent the global features corresponding to the audio training data. The second data processing unit is configured to obtain a mel spectrogram corresponding to the audio training data, and perform local mask processing and noise processing on the mel spectrogram to obtain a mask mel spectrogram and a noise mel spectrogram corresponding to the mel spectrogram. The model training unit is configured to input the first feature training data, the second feature training data, the mask mel spectrogram, and the noise mel spectrogram corresponding to the audio training data into a to-be-trained voice conversion model to obtain a predicted mel spectrogram output by the to-be-trained voice conversion model. Before the step of inputting the first feature training data, the second feature training data, the mask mel spectrogram, and the noise mel spectrogram corresponding to the audio training data into the to-be-trained voice conversion model to obtain the predicted mel spectrogram output by the to-be-trained voice conversion model, The first feature training data and the second feature training data corresponding to the audio training data are respectively subjected to dimension reduction processing of a preset channel dimension to obtain first representation training data corresponding to the first feature training data, and second representation training data corresponding to the second feature training data. According to the time sequence length of the mel spectrogram corresponding to the audio training data, the first representation training data and the second representation training data are filled to be equal in length to the mel spectrogram; The filled first representation training data, the filled second representation training data, the mask mel spectrogram, and the noise mel spectrogram are spliced in the channel dimension to obtain spliced feature training data corresponding to the audio training data. The first audio data of the to-be-converted pronunciation person is converted into the second audio data of the specified pronunciation person according to the fifth feature data, the sixth feature data, the first mel spectrogram corresponding to the sample audio data, and the trained voice conversion model. input the splicing feature training data corresponding to the audio training data into the to-be-trained speech conversion model, to obtain the predicted mel spectrogram output by the to-be-trained speech conversion model; The model parameter adjustment unit is configured to perform model parameter adjustment on the to-be-trained speech conversion model according to the mel spectrogram corresponding to the audio training data and the predicted mel spectrogram, to obtain a trained speech conversion model.

15. A voice conversion apparatus characterized by comprising: The device further includes a data acquisition unit, a feature extraction unit, a feature splicing unit, and a data conversion unit. The data acquisition unit is configured to acquire first audio data of a to-be-converted speaker and sample audio data of a specified speaker in response to a speech conversion instruction. The feature extraction unit is configured to extract first feature data and second feature data from the first audio data, and extract third feature data and fourth feature data from the sample audio data; wherein the first feature data is used to represent content features corresponding to the first audio data, the second feature data is used to represent global features corresponding to the first audio data, the third feature data is used to represent content features corresponding to the sample audio data, and the fourth feature data is used to represent global features corresponding to the sample audio data. The feature splicing unit is configured to splice the first feature data and the third feature data to generate fifth feature data, and splice the second feature data and the fourth feature data to generate sixth feature data. The data conversion unit is configured to convert the first audio data of the to-be-converted speaker into second audio data of the specified speaker according to the fifth feature data, the sixth feature data, a first mel spectrogram corresponding to the sample audio data, and a trained speech conversion model; wherein the trained speech conversion model is obtained by training the method according to any one of claims 1 to 6.

16. An electronic device, comprising: comprising: a memory and a processor; the memory is configured to store one or more computer instructions; the processor is configured to execute the one or more computer instructions to implement the method according to any one of claims 1 to 13.

17. A computer readable storage medium having stored thereon one or more computer instructions, wherein, The instructions are executed by the processor to perform the method according to any one of claims 1 to 13.

Images