Voice conversion method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a speech conversion method and device, electronic equipment and storage medium. The speech conversion method comprises: obtaining to-be-converted speech data and target data, inputting the to-be-converted speech data and the target data into a speech conversion model, wherein the speech conversion model is trained based on non-parallel speech data and parallel speech data in stages, and finally performing speech conversion processing on the to-be-converted speech data and the target data by the speech conversion model to obtain target speech data. The speech conversion method provided by the present disclosure can improve the speech conversion effect.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of voice conversion, and in particular, to a voice conversion method and device, an electronic device, and a storage medium. BACKGROUND

[0002] In the related art, a voice conversion task is to process the personalized features of a user to make it sound like the voice of another user. Currently, voice conversion is usually achieved through a TTS (Text To Speech) network. However, the TTS network model is large, and its network deployment and operation time both require a long time. This results in a long voice conversion process and low voice conversion efficiency. SUMMARY

[0003] The present disclosure provides a voice conversion method, device, electronic device, and storage medium to at least solve the problem of poor voice conversion effect of a voice conversion model in the related art. The technical solutions of the present disclosure are as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a voice conversion method is provided, comprising:

[0005] obtaining to-be-converted voice data and target data;

[0006] inputting the to-be-converted voice data and the target data into a voice conversion model; wherein the voice conversion model is obtained by stage-wise training based on non-parallel voice data and parallel voice data;

[0007] performing voice conversion processing on the to-be-converted voice data and the voice data through the voice conversion model to obtain target voice data.

[0008] In a possible implementation, before the to-be-converted voice data is obtained, the method further comprises:

[0009] obtaining preset training sample data; wherein the preset training sample data comprises non-parallel voice data and parallel voice data;

[0010] training a preset model based on the non-parallel voice data to obtain a pre-trained model;

[0011] training the pre-trained model based on the parallel voice data to obtain the voice conversion model.

[0012] In a possible implementation, the training of the preset model based on the non-parallel voice data to obtain the pre-trained model comprises:

[0013] inputting the i-th non-parallel voice data into the preset model; i∈[1, I], I is the number of non-parallel voice data in the preset training sample data;

[0014] extracting speaker information and semantic information of the ith non-parallel voice data by the preset model; wherein the speaker information at least includes tone, intonation, rhythm;

[0015] combining to obtain first voice data corresponding to the ith non-parallel voice data based on the speaker information and the semantic information of the ith non-parallel voice data;

[0016] modifying the preset model based on the ith non-parallel voice data and the first voice data, to obtain a pre-training model.

[0017] In a possible implementation, the training of the pre-training model based on the parallel voice data to obtain the voice conversion model comprises:

[0018] inputting the mth parallel voice data and the nth parallel voice data into the pre-training model; m∈[1, L], n∈[1, L], L is the number of parallel voice data in the preset training sample data, and m≠n;

[0019] extracting speaker information of the mth parallel voice data and extracting semantic information of the nth parallel voice data by the pre-training model;

[0020] combining to obtain second voice data corresponding to the mth parallel voice data based on the speaker information of the mth parallel voice data and the semantic information of the nth parallel voice data;

[0021] modifying the pre-training model based on the mth parallel voice data and the second voice data to obtain a voice conversion model.

[0022] In a possible implementation, before the inputting of the mth parallel voice data and the nth parallel voice data into the pre-training model, the method further comprises:

[0023] performing dynamic time warping processing on the mth parallel voice data and the nth parallel voice data by the pre-training model to obtain time-aligned mth parallel voice data and nth parallel voice data.

[0024] In a possible implementation, the voice conversion processing on the voice data to be converted and the target data by the voice conversion model to obtain target voice data comprises:

[0025] extracting semantic information of the voice data to be converted by the voice conversion model, and extracting speaker information of the target data by the voice conversion model;

[0026] combine the semantic information of the to-be-converted speech data and the speaker information of the target data to obtain the target speech data.

[0027] According to a second aspect of the embodiments of the present disclosure, a speech conversion device is provided, comprising:

[0028] a first obtaining module configured to obtain to-be-converted speech data and target data;

[0029] an input module configured to input the to-be-converted speech data and the target data into a speech conversion model, wherein the speech conversion model is obtained by stage training based on non-parallel speech data and parallel speech data;

[0030] a conversion module configured to perform speech conversion processing on the to-be-converted speech data and the target data by using the speech conversion model to obtain target speech data.

[0031] In a possible implementation, the first obtaining module further comprises:

[0032] a second obtaining module configured to obtain preset training sample data, wherein the preset training sample data comprises non-parallel speech data and parallel speech data;

[0033] a first training module configured to train a preset model based on the non-parallel speech data to obtain a pre-trained model;

[0034] a second training module configured to train the pre-trained model based on the parallel speech data to obtain the speech conversion model.

[0035] In a possible implementation, the first training module comprises:

[0036] a first input unit configured to input the ith non-parallel speech data into the preset model; i∈[1, I], I is the number of non-parallel speech data in the preset training sample data;

[0037] a first extraction unit configured to extract speaker information and semantic information of the ith non-parallel speech data by using the preset model, wherein the speaker information at least comprises tone, intonation, and rhythm;

[0038] a first combination unit configured to combine the speaker information and the semantic information of the ith non-parallel speech data to obtain first speech data corresponding to the ith non-parallel speech data;

[0039] a first correction unit configured to correct the preset model based on the ith non-parallel speech data and the first speech data to obtain the pre-trained model.

[0040] In a possible implementation, the second training module comprises:

[0041] a second input unit, configured to input the mth parallel speech data and the nth parallel speech data to the pre-training model; m e [1, L], n e [1, L], L is a number of the parallel speech data in the preset training sample data, and m≠n;

[0042] a second extraction unit, configured to extract speaker information of the mth parallel speech data and semantic information of the nth parallel speech data by using the pre-training model;

[0043] a second combination unit, configured to combine the speaker information of the mth parallel speech data and the semantic information of the nth parallel speech data to obtain second speech data corresponding to the mth parallel speech data;

[0044] a second correction unit, configured to correct the pre-training model based on the mth parallel speech data and the second speech data, to obtain a speech conversion model.

[0045] In a possible implementation, the second input unit further comprises:

[0046] a processing unit, configured to perform dynamic time warping processing on the mth parallel speech data and the nth parallel speech data by using the pre-training model, to obtain the mth parallel speech data and the nth parallel speech data after time alignment.

[0047] In a possible implementation, the conversion module comprises:

[0048] a third extraction unit, configured to extract semantic information of the speech data to be converted and speaker information of the target data by using the speech conversion model;

[0049] a third combination unit, configured to combine the semantic information of the speech data to be converted and the speaker information of the target data to obtain the target speech data.

[0050] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, comprising:

[0051] a processor;

[0052] a memory for storing instructions executable by the processor;

[0053] The processor is configured to execute the instructions to implement the speech conversion method according to any one of the first aspect.

[0054] According to a fourth aspect of the embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, when instructions in the non-transitory computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the speech conversion method according to any one of the first aspect.

[0055] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, comprising a computer program, when the computer program is executed by a processor, the speech conversion method according to any one of the first aspect is implemented.

[0056] The embodiments of the present disclosure provide at least the following beneficial effects:

[0057] In the embodiments of the present disclosure, by obtaining the to-be-converted speech data and the target data, and then inputting the to-be-converted speech data and the target data into the speech conversion model, wherein the speech conversion model is obtained by training based on the non-parallel speech data and the parallel speech data in stages, and finally performing speech conversion processing on the to-be-converted speech data and the target data by the speech conversion model to obtain the target speech data. In this way, the speech conversion model is trained based on the non-parallel speech data and the parallel speech data in stages, which can make good use of the advantages of the parallel speech data and the non-parallel speech data, improve the utilization rate of the speech data, and thus improve the fitting degree of the speech conversion model and the application scenario, and further improve the speech conversion effect.

[0058] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0059] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure, and do not constitute an undue limitation on the present disclosure.

[0060] Figure 1 is a flowchart of a speech conversion method according to an exemplary embodiment.

[0061] Figure 2 is a training flowchart of a pre-training model according to an exemplary embodiment.

[0062] Figure 3 is a training flowchart of a speech conversion model according to an exemplary embodiment.

[0063] Figure 4 is a block diagram of a speech conversion device according to an exemplary embodiment.

[0064] Figure 5is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0065] In order for those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings.

[0066] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation described in the following exemplary embodiments does not represent all implementations consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0067] The voice conversion method, device, electronic device and storage medium provided by the embodiments of the present disclosure will be described in detail below in conjunction with the drawings.

[0068] It should be noted that the voice conversion method, device, electronic device and non-transitory computer readable storage medium provided by the embodiments of the present disclosure are applied to voice conversion tasks, and the application scenarios include information security, film dubbing and game voice change. For example, when a user is playing a game, the user can input voice, and the input voice can be converted to voice with a target voice color by a voice conversion model, wherein the target voice color can be specified by pre-setting of the target voice color, including loli voice, ojousama voice, shoujo voice and uncle voice, etc.

[0069] Figure 1 is a flowchart of a voice conversion method according to an exemplary embodiment. The voice conversion method can be applied to an electronic device, for example, a mobile phone, a tablet computer, a computer, etc. As shown in Figure 1 The voice conversion method can include the following steps.

[0070] In step S110, the voice data to be converted and the target data are obtained.

[0071] In the embodiments of the present disclosure, the voice data to be converted and the target data can be acquired. The voice data to be converted can be voice data pre-recorded by a user to an electronic device, can be voice data of the user himself, or can be voice data of other people. The target data can be voice data having voice characteristics that the user wants to convert, can be voice data pre-stored by the electronic device, or can be voice data recorded by the user himself. That is, the target data has multiple options for the user to choose from. After the user selects the target data, the electronic device can acquire the target data according to the user's selection.

[0072] In step S120, the voice data to be converted and the target data are input to the voice conversion model.

[0073] The voice conversion model is trained based on non-parallel voice data and parallel voice data. The non-parallel voice data can be voice data recorded by multiple people for different sentences, and the parallel voice data can be voice data recorded by multiple people for the same sentence.

[0074] In the embodiments of the present disclosure, after the voice data to be converted and the target data are acquired, the voice data to be converted and the target data can be input to the voice conversion model, so that the electronic device can perform voice conversion processing through the voice conversion model.

[0075] In step S130, the voice data to be converted and the target data are processed by the voice conversion model to obtain target voice data.

[0076] In the embodiments of the present disclosure, after the voice data to be converted and the target data are input to the voice conversion model, the voice data to be converted and the target data can be processed by the voice conversion model to obtain target voice data. For example, the semantic information of the voice data to be converted and the speaker information of the target data can be combined to obtain the target voice data, that is, the target voice data can be voice data having the semantic information of the voice data to be converted and the speaker information of the target data; or the speaker information of the voice data to be converted and the semantic information of the target data can be combined to obtain the target voice data, that is, the target voice data can also be voice data having the speaker information of the voice data to be converted and the semantic information of the target data.

[0077] In the embodiments of the present disclosure, by obtaining to-be-converted speech data and target data, and then inputting the to-be-converted speech data and the target data into a speech conversion model, wherein the speech conversion model is obtained by training based on non-parallel speech data and parallel speech data in stages, and finally performing speech conversion processing on the to-be-converted speech data and the target data by the speech conversion model to obtain target speech data. In this way, the speech conversion model is obtained by training based on non-parallel speech data and parallel speech data in stages, which can make good use of the advantages of parallel speech data and non-parallel speech data, improve the utilization rate of speech data, and thus can improve the fitting degree of the speech conversion model and the application scenario, and further improve the speech conversion effect.

[0078] In a possible implementation, before obtaining the to-be-converted speech data, the speech conversion model can also be pre-trained, and the specific implementation can include:

[0079] obtaining preset training sample data; wherein the preset training sample data includes non-parallel speech data and parallel speech data;

[0080] training a preset model based on the non-parallel speech data to obtain a pre-training model;

[0081] training the pre-training model based on the parallel speech data to obtain the speech conversion model.

[0082] The preset model can be a pre-constructed speech conversion model, but has not been trained yet. The pre-training model can be a speech conversion model obtained by training the preset model based on the non-parallel speech data.

[0083] In the embodiments of the present disclosure, considering that the speech conversion model is trained by relying on non-parallel speech data or parallel speech data alone, the speech conversion model obtained has poor speech conversion effect. Before obtaining the speech to be converted, preset training sample data can be obtained, wherein the preset training sample data includes non-parallel speech data and parallel speech data. Then, a preset model is trained based on the non-parallel speech data to obtain a pre-trained model, and finally the pre-trained model is trained based on the parallel speech data to obtain the speech conversion model. In this way, compared with the related art of training the speech conversion model based on non-parallel data or training the speech conversion model by using parallel speech data generated based on non-parallel speech data, the method proposed in the embodiments of the present disclosure does not need to generate parallel speech data based on non-parallel speech data for model training, and can directly train the model by using existing non-parallel speech data and parallel speech data, thereby effectively improving the model training efficiency. Meanwhile, the preset model is first trained based on the non-parallel speech data, and then the model is trained based on the parallel speech data to obtain the speech conversion model, which can better utilize the advantages of the parallel speech data and the non-parallel speech data, reduce the storage size of the speech conversion model and the difficulty of the training process, and improve the utilization rate of the speech data, thereby improving the application scene fitting degree of the speech conversion model and further improving the speech conversion effect.

[0084] In a possible implementation, the specific implementation of training the preset model based on the non-parallel speech data to obtain the pre-trained model can be as follows.

[0085] The i th non-parallel speech data is input into the preset model; i ∈ [1, I], and I is the number of non-parallel speech data in the preset training sample data.

[0086] Speaker information and semantic information of the i th non-parallel speech data are extracted by the preset model; wherein the speaker information at least includes timbre, tone, and rhythm.

[0087] Based on the speaker information and the semantic information of the i th non-parallel speech data, the first speech data corresponding to the i th non-parallel speech data is obtained by combination.

[0088] The preset model is corrected based on the i th non-parallel speech data and the first speech data to obtain the pre-trained model.

[0089] The first speech data can be speech data obtained by combining the speaker information and the speech information of the i th non-parallel speech data.

[0090] In the embodiments of the present disclosure, the i-th non-parallel speech data in the preset training sample can be obtained, and then the i-th non-parallel speech data is input into the preset model, i∈[1, I], I is the number of non-parallel speech data in the preset training sample data. Then, the speaker information and the semantic information of the i-th non-parallel speech data are extracted through the preset model. For example, the speaker information of the i-th non-parallel speech data can be the tone, tone, rhythm, etc. of the speaker. Then, the speaker information and the semantic information of the i-th non-parallel speech data can be combined to obtain the first speech data corresponding to the i-th non-parallel speech data. Finally, the preset model can be modified based on the i-th non-parallel speech data and the first speech data. For example, the difference between the i-th non-parallel speech data and the first speech data can be calculated, and the difference is used as a loss function to modify the preset model. When the loss function reaches a steady state, the trained preset model is determined as a pre-training model.

[0091] Exemplarily, Figure 2 is a training flowchart of a pre-training model according to an exemplary embodiment, as Figure 2 shown, the preset model can include a speaker encoder, a semantic encoder and a decoder. When training the preset model based on non-parallel speech data, the i-th non-parallel speech data can be input into the speaker encoder and the semantic encoder respectively. The speaker information of the i-th non-parallel speech data is extracted through the speaker encoder, and the semantic information of the i-th non-parallel speech data is extracted through the semantic encoder. Then, the speaker information and the semantic information can be input into the decoder, and the speaker information and the speech information are combined through the decoder to generate new speech data, i.e. the first speech data. Then, the difference between the i-th non-parallel speech data and the first speech data can be calculated, i.e. the difference between the spectrum graph of the i-th non-parallel speech data and the spectrum graph of the first speech data is calculated, and the difference is used as a loss function to complete the iterative modification of the preset model. When the loss function reaches a steady state, the pre-training model can be obtained. It can be understood that the structures of the speaker encoder, the semantic encoder and the decoder are the same as those in the related art, and will not be described here.

[0092] In this way, compared with the related art, the model is trained based on the parallel speech data generated by the non-parallel speech data. The method of the present disclosure can train the preset model by the non-parallel data in the preset training sample. In this way, the preset model can be trained directly using the existing non-parallel speech data without generating parallel speech data based on non-parallel speech data, thereby reducing the size and training difficulty of the speech conversion model, improving the application scene fitting degree of the speech conversion model, and further improving the speech conversion effect.

[0093] In a possible implementation, the pre-training model is trained based on parallel speech data, and a specific implementation of the speech conversion model can be as follows.

[0094] The mth parallel speech data and the nth parallel speech data are input into the pre-training model, where m e [1, L], n e [1, L], L is the number of parallel speech data in the preset training sample data, and m≠n;

[0095] Speaker information of the mth parallel speech data is extracted by the pre-training model, and semantic information of the nth parallel speech data is extracted.

[0096] The second speech data corresponding to the mth parallel speech data is obtained by combining the speaker information of the mth parallel speech data and the semantic information of the nth parallel speech data.

[0097] The pre-training model is corrected based on the mth parallel speech data and the second speech data, and the speech conversion model is obtained.

[0098] The second speech data can be speech data obtained by combining the speaker information of the mth parallel speech data and the semantic information of the nth parallel speech data.

[0099] In the embodiments of the present disclosure, after obtaining the pre-training model, the mth parallel speech data and the nth parallel speech data can be obtained from the preset training sample, the mth parallel speech data and the nth parallel speech data are input into the pre-training model, m e [1, L], n e [1, L], L is the number of parallel speech data in the preset training sample data, and m≠n. Then, the speaker information of the mth parallel speech data can be extracted by the pre-training model, and similarly, the speaker information of the mth parallel speech data can be obtained, such as tone, intonation, rhythm, etc., and the semantic information of the nth parallel speech data. The speaker information of the mth parallel speech data and the semantic information of the nth parallel speech data are combined to obtain new speech data corresponding to the mth parallel speech data, i.e. the second speech data. Finally, the pre-training model can be corrected based on the mth parallel speech data and the second speech data, and the speech conversion model is obtained. The correction process of the pre-training model is similar to the correction process of the preset model, which will not be described here.

[0100] Exemplarily, Figure 3 is a training flowchart of a speech conversion model according to an exemplary embodiment, as Figure 3As shown, when training the pre-training model, the mth parallel speech data can be input into the speaker encoder of the pre-training model to obtain speaker information corresponding to the mth parallel speech data; at the same time, or before or after inputting the mth parallel speech data, the nth parallel speech data can be input into the semantic encoder of the pre-training model to obtain semantic information corresponding to the nth parallel speech data. Then, the speaker information corresponding to the mth parallel speech data and the semantic information corresponding to the nth parallel speech data are input into the decoder of the pre-training model, and the decoder is used to combine the speaker information and the semantic information, so as to obtain second speech data. Finally, the difference between the second speech data and the mth parallel speech data is calculated, and the difference is used as a loss function to complete iterative correction of the pre-training model, so as to obtain a speech conversion model.

[0101] In this way, the pre-training model can be trained based on parallel speech data on the basis of training the pre-set model based on non-parallel speech data to obtain a speech conversion model. In this way, the advantages of non-parallel speech data and parallel speech data can be further fully utilized, so as to further improve the utilization rate of speech data, improve the application scene fitting degree of the speech conversion model, and further improve the speech conversion effect.

[0102] In a possible implementation, before the mth parallel speech data and the nth parallel speech data are input into the pre-training model, the method further includes:

[0103] The mth parallel speech data and the nth parallel speech data are dynamically time-warping processed by the pre-training model to obtain time-aligned mth parallel speech data and nth parallel speech data.

[0104] The dynamic time-warping processing can be an alignment processing of the mth parallel speech data and the nth parallel speech data in the same character pronunciation time.

[0105] In the embodiments of the present disclosure, before the mth parallel speech data and the nth parallel speech data are input into the pre-training model, the method further includes: Figure 3As shown, the mth parallel speech data and the nth parallel speech data can be dynamically time warping processed by the pre-trained model, that is, the mth parallel speech data and the nth parallel speech data are aligned at the same time of pronunciation of the same character to obtain the mth parallel speech data and the nth parallel speech data after time alignment. In this way, by dynamically time warping processing the mth parallel speech data and the nth parallel speech data, the mth parallel speech data and the nth parallel speech data can be aligned in time sequence, so as to provide better training data basis for model training, thereby improving the model training efficiency, improving the training effect of the pre-trained model, obtaining a speech conversion model with better training effect, and thereby improving the speech conversion effect when performing the speech conversion task by the speech conversion model.

[0106] In a possible implementation, the specific implementation of performing speech conversion processing on the to-be-converted speech data and the target data by the speech conversion model to obtain the target speech data can be as follows:

[0107] extracting semantic information of the to-be-converted speech data by the speech conversion model and extracting speaker information of the target data by the speech conversion model;

[0108] combining the semantic information of the to-be-converted speech data and the speaker information of the target data to obtain the target speech data.

[0109] In the embodiments of the present disclosure, the semantic information of the to-be-converted speech data can be extracted by the speech conversion model, and the speaker information of the target data can be extracted by the speech conversion model, for example, the semantic information of the to-be-converted speech data can be extracted by the semantic encoder of the speech conversion model, and the semantic information of the to-be-converted speech data can be extracted by the speaker encoder of the speech conversion model. Then, the semantic information of the to-be-converted speech data and the speaker information of the target data can be combined to obtain the target speech data. That is, the target speech data is obtained by combining the semantic information of the to-be-converted speech data and the speaker information of the target data, so that the speech conversion can be realized by the speech conversion model.

[0110] It should be noted that the speaker information of the to-be-converted speech data can also be extracted by the speech conversion model, and the semantic information of the target data can be extracted by the speech conversion model, and then the target speech data can be obtained based on the speaker information of the to-be-converted speech data and the semantic information of the target data.

[0111] In a possible implementation, the speech conversion model is a variational autoencoder VAE.

[0112] In this embodiment, considering the storage size of the speech conversion model, the speech conversion model can be a variational autoencoder (VAE). The VAE model includes a speaker encoder, a semantic encoder, a decoder, and a backpropagation feedback mechanism. Since the VAE model is relatively small and has good real-time performance, it can also meet the requirements of real-time performance, model size, and conversion effect when the speech conversion model is run offline on electronic devices. Therefore, the speech conversion model trained based on VAE can be deployed not only on the server side but also on electronic devices, making the model deployment more flexible. At the same time, the VAE model is small, and its training time is short, thereby improving the model training efficiency.

[0113] It should be noted that similar speech conversion models also include Gaussian networks, GAN (Generative Adversarial Network) networks, TTS (Text to Speech) networks, etc., which can be used to replace the VAE model in the embodiments of this disclosure for other purposes as needed.

[0114] Based on the same inventive concept, embodiments of this disclosure also provide a voice conversion device, such as... Figure 4 As shown, Figure 4 This is a block diagram illustrating a speech conversion device according to an exemplary embodiment. (Refer to...) Figure 4 The voice conversion device 400 may include:

[0115] The first acquisition module 410 is used to acquire the voice data to be converted and the target data;

[0116] The input module 420 is used to input the speech data to be converted and the target data into the speech conversion model; wherein, the speech conversion model is trained in stages based on non-parallel speech data and parallel speech data;

[0117] The conversion module 430 is used to perform speech conversion processing on the speech data to be converted and the target data through a speech conversion model to obtain the target speech data.

[0118] In one possible implementation, prior to the first acquisition module 410, the following further includes:

[0119] The second acquisition module is used to acquire preset training sample data; wherein, the preset training sample data includes non-parallel speech data and parallel speech data;

[0120] The first training module is used to train a preset model based on non-parallel speech data to obtain a pre-trained model.

[0121] The second training module is used to train the pre-trained model based on parallel speech data to obtain a speech conversion model.

[0122] In a possible implementation, the first training module comprises:

[0123] The first input unit is configured to input the ith non-parallel speech data to the preset model; i∈[1, I], I is the number of non-parallel speech data in the preset training sample data;

[0124] The first extraction unit is configured to extract speaker information and semantic information of the ith non-parallel speech data by the preset model; wherein the speaker information at least includes tone, intonation, rhythm;

[0125] The first combination unit is configured to combine the first speech data corresponding to the ith non-parallel speech data based on the speaker information and the semantic information of the ith non-parallel speech data.

[0126] The first correction unit is configured to correct the preset model based on the ith non-parallel speech data and the first speech data, to obtain a pre-training model.

[0127] In a possible implementation, the second training module comprises:

[0128] The second input unit is configured to input the mth parallel speech data and the nth parallel speech data to the pre-training model; m∈[1, L], L is the number of parallel speech data in the preset training sample data, and m≠n;

[0129] The second extraction unit is configured to extract speaker information of the mth parallel speech data by the pre-training model, and extract semantic information of the nth parallel speech data.

[0130] The second combination unit is configured to combine the second speech data corresponding to the mth parallel speech data based on the speaker information of the mth parallel speech data and the semantic information of the nth parallel speech data.

[0131] The second correction unit is configured to correct the pre-training model based on the mth parallel speech data and the second speech data, to obtain a speech conversion model.

[0132] In a possible implementation, before the second input unit, further comprising:

[0133] The processing unit is configured to perform dynamic time warping processing on the mth parallel speech data and the nth parallel speech data by the pre-training model, to obtain the mth parallel speech data and the nth parallel speech data after time alignment.

[0134] In a possible implementation, the conversion module 430 comprises:

[0135] The third extraction unit is used to extract semantic information of the speech data to be converted through the speech conversion model, and to extract speaker information of the target data through the speech conversion model.

[0136] The third combining unit is used to combine the semantic information of the speech data to be converted and the speaker information of the target data to obtain the target speech data.

[0137] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0138] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0139] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device 500 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0140] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0141] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0142] The computing unit 501 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs various methods and processes described above, such as the speech conversion method. For example, in some embodiments, the speech conversion method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded onto the RAM 503 and executed by the computing unit 501, one or more steps of the speech conversion method described above can be performed. Alternatively, in other embodiments, the computing unit 501 can be configured to perform the speech conversion method by any other appropriate means, such as by means of firmware.

[0143] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0144] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0145] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0146] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0147] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0148] The computer system can include clients and servers. This relationship can be. remote, where each server is stored on a remote computer from a client. The clients and the servers can be connected through a communication network. The relationship can be a client-server relationship over a network. Servers can be cloud servers, also known as cloud computing servers or cloud hosts, which are a host product in the cloud computing service system. The servers can be servers of a distributed system, or servers combined with a blockchain.

[0149] It should be understood that the various forms of flow shown above can be reordered, steps added or removed. For example, the steps described in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, which are not limited herein.

[0150] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A voice conversion method characterized by, The method comprises: obtaining to-be-converted speech data and target data; inputting the to-be-converted speech data and the target data into a speech conversion model; wherein the speech conversion model is obtained by training based on non-parallel speech data and parallel speech data in stages; extracting semantic information of the to-be-converted speech data by the speech conversion model, and extracting speaker information of the target data by the speech conversion model; combining the semantic information of the to-be-converted speech data and the speaker information of the target data to obtain target speech data; Before the to-be-converted speech data is obtained, the method further comprises: obtaining preset training sample data; wherein the preset training sample data comprises non-parallel speech data and parallel speech data; The first i non-parallel speech data is input to a preset model; i ∈[1, I ], I is the number of non-parallel speech data in the preset training sample data; extracting the first i The speaker information and semantic information of the non-parallel speech data are extracted, wherein the speaker information at least includes tone, intonation and rhythm. based on the first i utterance information and semantic information of the second i non-parallel speech data, the first speech data corresponding to the second non-parallel speech data is obtained. based on the first i The preset model is corrected based on the first i The difference value of the spectrograms of the first and second non-parallel speech data is calculated, and the difference value is used as a loss function to correct the preset model. training the pre-training model based on the parallel speech data to obtain the speech conversion model.

2. The voice conversion method of claim 1, wherein, Training the pre-training model based on the parallel speech data to obtain the speech conversion model comprises: The first m parallel voice data and the second n parallel voice data are input to the pre-trained model; m∈ [1, L ], n ∈[1, L ], L is the number of the parallel voice data in the preset training sample data, and m ≠ n ; extracting the first m speaker information of the parallel speech data, and extracting the second n semantic information of the parallel speech data; based on the first m speaker information of the parallel speech data, and the semantic information of the second n parallel speech data, the second speech data corresponding to the first m parallel speech data is obtained by combination. based on the first m The pre-training model is corrected based on the first parallel speech data and the second speech data to obtain a speech conversion model.

3. The voice conversion method of claim 2, wherein, The method further includes: m The method further includes: n Before inputting the first parallel speech data and the second parallel speech data into the pre-trained model, the method further includes: The first m parallel speech data and the second n parallel speech data are dynamically time warping processed to obtain the first m parallel speech data and the second n parallel speech data after time alignment.

4. A voice conversion apparatus characterized by comprising: The method comprises: a first obtaining module, configured to obtain to-be-converted speech data and target data; an input module, configured to input the to-be-converted speech data and the target data into a speech conversion model; wherein the speech conversion model is obtained by training based on non-parallel speech data and parallel speech data in stages; a conversion module, configured to perform speech conversion processing on the to-be-converted speech data and the target data by the speech conversion model to obtain target speech data; The conversion module comprises: a third extracting unit, configured to extract semantic information of the to-be-converted speech data by the speech conversion model, and extract speaker information of the target data by the speech conversion model; a third combining unit, configured to combine the semantic information of the to-be-converted speech data and the speaker information of the target data to obtain the target speech data; Before the first obtaining module, the method further comprises: a second obtaining module, configured to obtain preset training sample data; wherein the preset training sample data comprises non-parallel speech data and parallel speech data; a first training module, configured to train a preset model based on the non-parallel speech data to obtain a pre-training model; a second training module, configured to train the pre-training model based on the parallel speech data to obtain the speech conversion model; The first training module comprises: The first input unit is configured to input the first i non-parallel speech data into a preset model; i ∈[1, I ], I is a number of non-parallel speech data in the preset training sample data. The first extraction unit is configured to extract first i The speaker information and the semantic information of the non-parallel voice data, wherein the speaker information at least includes tone, intonation and rhythm. The first combination unit is configured to combine the first i th piece of non-parallel speech data based on the first i th piece of non-parallel speech data, the speaker information of the first i th piece of non-parallel speech data, and the semantic information of the first i th piece of non-parallel speech data to obtain first speech data corresponding to the first i th piece of non-parallel speech data. i The first combination unit is configured to combine the first i th piece of non-parallel speech data based on the first The first correction unit is configured to correct the preset model based on the first i non-parallel speech data and the first speech data, to obtain a pre-training model. i The preset model is corrected based on the first i non-parallel speech data and the first speech data, to obtain a pre-training model.

5. The speech conversion apparatus according to claim 4, wherein The second training module comprises: The second input unit is configured to input the first m parallel speech data and the second n parallel speech data into the pre-trained model. m ∈[1, L ], n ∈[1, L ], L is a number of the parallel speech data in the preset training sample data, and m ≠ n ; a second extraction unit configured to extract the first feature information of the parallel speech data by using the pre-trained model m and the speaker information of the parallel speech data, and extract the semantic information of the parallel speech data n and the speaker information of the parallel speech data, and extract the semantic information of the parallel speech data The second combining unit is used based on the first m Speaker information for each parallel speech data point, and the aforementioned first... n The semantic information of each parallel speech data is combined to obtain the first... m The second voice data corresponding to each parallel voice data; A second correction unit is configured to correct the pre-trained model based on the first m The pre-trained model is corrected based on the first parallel speech data and the second speech data to obtain a speech conversion model.

6. The speech conversion apparatus according to claim 5, wherein Before the second input unit, the method further comprises: Processing unit, used to process the pre-trained model on the first... m Parallel speech data and the first n The parallel speech data is subjected to dynamic time warping to obtain the time-aligned first... m Parallel speech data and the first n Parallel speech data.

7. An electronic device, comprising: The method comprises: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the speech conversion method of any one of claims 1 to 3.

8. A non-transitory computer-readable storage medium, when instructions in the non-transitory computer-readable storage medium are executed by a processor of an electronic device, enabling the electronic device to perform the speech conversion method of any one of claims 1 to 3.

9. A computer program product comprising a computer program, which, when executed by a processor, implements the speech conversion method of any one of claims 1 to 3.

Images