A speech conversion method based on autoencoder framework

Through the speech conversion method based on the automatic encoder framework, Mel spectrum features are extracted and a system containing encoder, decoder and constraint models are built, which solves the problems of speech style conversion and feature separation in the existing technology, and achieves efficient and accurate Voice conversion and reduces training time.

CN113921025BActive Publication Date: 2024-11-26SICHUAN XW BANK CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111262860.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-28
Publication Date
2024-11-26
Estimated Expiration
2041-10-28

Smart Images

  • Figure CN113921025B_ABST
    Figure CN113921025B_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of speech conversion, and specifically relates to a speech conversion method based on an automatic encoder framework. The system framework constructed by the invention comprises only one encoder, so that the Mel spectrum information of the original input is not lost during the training process, and perfect reconstruction is achieved; the extractability of three speech features of content, timbre and rhythm is ensured by combining training through an encoder and a decoder with two constraint models; and the extractability of the three speech features of content, timbre and rhythm is further ensured by adopting a discriminator; when the invention performs speech conversion on audio with different contents, the training of paired data is not involved, and the model training process only reconstructs itself, so efficient speech conversion is achieved; and it is ensured that all speech features are completely separated, and the conversion is performed on speech style.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of speech conversion, and in particular relates to a speech conversion method based on an automatic encoder framework. Background Art

[0002] With the rapid development of intelligent voice technology, audio voice changing technology has become a new and popular technology. It aims to convert a source voice into a target voice with a characteristic voice style and unchanged voice content. For example, a voice changing application changes the voice of a certain audio clip recorded by the user using a certain characteristic target voice changing effect for playback.

[0003] At present, there are very few methods for unpaired data in speech conversion systems. Only AutoVC, One-shotVC, Prosody Transfer and SpeechFLow can achieve speech conversion under unpaired data, and only Prosody Transfer and SpeechFLow can achieve rhythm conversion. However, the above two methods still have two major disadvantages: first, the rhythm conversion effect of audio with different content is poor; second, the training time of the model is long.

[0004] After searching, it was found that the invention patent application with publication number CN111312267A disclosed a method, device, equipment and storage medium for voice style conversion; its process for voice separation is: obtaining source style voice, target style voice and initial conversion voice; according to the voice content loss between the initial conversion voice and the source style voice and the voice style loss between the initial conversion voice and the target style voice, the initial conversion voice is optimized for loss, and the new initial conversion voice is obtained for further loss optimization until the new initial conversion voice meets the preset loss optimization conditions, and the new initial conversion voice is used as the style conversion voice of the source style voice under the target style. The advantage of the technical solution in the above patent application is that it realizes the accurate conversion of the source style voice under the target style, without the need for pre-training of voice style conversion for the target style, ensuring the voice style conversion under the target voice that has not been pre-trained, and improving the comprehensiveness and accuracy of voice style conversion.

[0005] The above technical solution performs voice conversion based on the speaker's timbre, but cannot convert the voice style. Secondly, it cannot guarantee the complete separation of all voice features. Summary of the invention

[0006] The present invention discloses a speech conversion method based on an automatic encoder framework, which is intended to solve the technical problems mentioned in the background technology that the speech style cannot be converted and the separation of all speech features cannot be guaranteed to be completely realized.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0008] A speech conversion method based on an autoencoder framework comprises the following steps:

[0009] Step 1: Based on the open source audio processing library, extract the Mel spectrum features of the audio as a data set;

[0010] Step 2: Build a system framework consisting of an encoder, a decoder, a discriminator, and two constraint models for timbre and rhythm respectively;

[0011] Step 3: Input the data set in step 1 into the system built in step 2, train the constraint model until the loss function converges, and extract the speech features corresponding to the source speaker and the target speaker;

[0012] Step 4: Based on the speech features extracted in step 3, replace the source prosodic code with the target code to complete the speech conversion task.

[0013] The system framework constructed by the present invention includes only one encoder, so that the Mel spectrum information of the original input is not lost during the training process, and perfect reconstruction is achieved; the extractability of the three speech features of content, timbre and rhythm is ensured by combining the training of the encoder and the decoder through two constraint models; and the extractability of the three speech features of content, timbre and rhythm is further ensured by using a discriminator, and the complete separation of speech features is achieved; when the present invention performs speech conversion on audio with different contents, the training of paired data is not involved, and the model training process only reconstructs itself, so efficient speech conversion is achieved.

[0014] Preferably, the encoder, decoder, constraint model and discriminator are all built based on the standard of Informer:AAAI2021best paper. Informer is a fixed model name, and AAAI2021 best paper is the best paper of the 2021 American Association for Artificial Intelligence Conference (AAAI).

[0015] Preferably, the network structure of the system framework in step 2 is composed of 2 or 3 layers of ProSparse Attention modules stacked together. ProSparse Attention is a sparse probability attention mechanism.

[0016] By using the ProSparse Attention module, the model training time is greatly reduced, further improving the efficiency of speech conversion.

[0017] Preferably, step 3 comprises the following steps:

[0018] Step 3.1: The encoder outputs a vector of length 160*1;

[0019] Step 3.2: Divide the vector in step 3.1 into three segments with lengths of 90, 35, and 35, representing the encoding of content, timbre, and rhythm respectively;

[0020] Step 3.3: Input the encoding of timbre and rhythm into the corresponding constraint model, and perform supervised training based on the real timbre vector and rhythm vector;

[0021] Step 3.4: Input the content, timbre and rhythm codes determined in step 3.2 into the discriminator for training until the distribution of the three codes is determined by the discriminator as three different categories.

[0022] The present invention inputs the Mel spectrum into the system for training. During the training process, the intermediate output of the encoder is a vector with a length of 160*1, which will be cut into three segments (90, 35, 35), representing the encoding of content, timbre and rhythm respectively, and then the encoding of timbre and rhythm are respectively input into the corresponding constraint model, and supervised training is performed based on the real timbre vector and rhythm vector, so that the part of the component cut out from the large vector is sufficiently close to the real speech feature.

[0023] Specifically, the loss functions described in step 3 are: the first is the reconstruction loss of the encoder and decoder, the encoder accepts the real Mel spectrum as input, and the training goal of the encoder and decoder is to reconstruct the corresponding Mel spectrum; the second is the loss function of timbre constraint; the third is the loss function of rhythm constraint; the fourth is the loss function of the discriminator for classification tasks.

[0024] Preferably, step 4 comprises the following steps:

[0025] Step 4.1: Let a source audio be A0. After being processed by librosa library, the Mel spectrum corresponding to the audio is X0.

[0026] Step 4.2: Let a target audio be A1. After being processed by librosa library, the Mel spectrum corresponding to the audio is X1.

[0027] Step 4.3: Input X0 to the encoder part of the system, and output a vector V0 with a length of 160*1; input X1 to the encoder part of the model, and output a vector V1 with a length of 160*1;

[0028] Step 4.4: Divide V0 and V1 into V according to the length of 90, 35 and 35 respectively. 0c ,V 0t ,V 0p and V1c ,V 1t ,V 1p , representing the encoding of the content, timbre and rhythm of the source audio and the target audio respectively;

[0029] Step 4.5: Combine the target prosody code, source content code and source timbre code, input them into the decoder part of the system, and output the converted Mel spectrum.

[0030] Preferably, the Mel spectrum corresponding to each audio in step 1 is a 480*80-dimensional matrix.

[0031] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0032] 1. The system framework constructed by the present invention includes only one encoder, so that the Mel spectrum information of the original input is not lost during the training process, and perfect reconstruction is achieved; the extractability of the three speech features of content, timbre and rhythm is ensured by combining the training of the encoder and the decoder through two constraint models; and the discriminator is used to further ensure the extractability of the three speech features of content, timbre and rhythm; when the present invention performs speech conversion on audio with different contents, it does not involve the training of paired data, and the model training process only reconstructs itself, so efficient speech conversion is achieved; and the separation of all speech features and conversion of speech styles are achieved.

[0033] 2. The present invention can realize the training of the speech conversion system without presetting any conditions, and in practical applications, the trained model can be directly called for speech conversion.

[0034] 3. By using the ProSparse Attention module, the model training time is greatly reduced, further improving the efficiency of speech conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0036] Figure 1 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0037] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0038] The following is combined with Figure 1 Describe the embodiments of the present invention in detail;

[0039] A speech conversion method based on an autoencoder framework comprises the following steps:

[0040] Step 1: Based on the open source audio processing library, extract the Mel spectrum features of the audio as a data set; the Mel spectrum corresponding to each audio is a 480*80-dimensional matrix.

[0041] Step 2: Build a system framework consisting of an encoder, a decoder, a discriminator, and two constraint models for timbre and rhythm respectively; the encoder, decoder, constraint model, and discriminator are all built based on the standard of Informer:AAAI2021 bestpaper. The network structure of the system framework consists of 2 or 3 layers of ProSparse Attention modules stacked.

[0042] By using the ProSparse Attention module, the model training time is greatly reduced, further improving the efficiency of speech conversion.

[0043] Step 3: Input the data set in step 1 into the system built in step 2, train the constraint model until the loss function converges, and extract the speech features corresponding to the source speaker and the target speaker;

[0044] The step 3 comprises the following steps:

[0045] Step 3.1: The encoder outputs a vector of length 160*1;

[0046] Step 3.2: Divide the vector in step 3.1 into three segments with lengths of 90, 35, and 35, representing the encoding of content, timbre, and rhythm respectively;

[0047] Step 3.3: Input the encoding of timbre and rhythm into the corresponding constraint model, and perform supervised training based on the real timbre vector and rhythm vector;

[0048] Step 3.4: The content, timbre and rhythm codes determined in step 3.2 are simultaneously input into the discriminator for training until the distribution of the three codes is determined by the discriminator as three different categories.

[0049] The loss functions described in step 3 are: the first is the reconstruction loss of the encoder and decoder. The encoder accepts the real Mel spectrum as input, and the training goal of the encoder and decoder is to reconstruct the corresponding Mel spectrum; the second is the loss function of timbre constraint; the third is the loss function of rhythm constraint; the fourth is the loss function of the discriminator for classification tasks.

[0050] Step 4: Extract speech features based on the model trained in step 3, and replace the source prosodic code with the target code to complete the speech conversion task.

[0051] The step 4 comprises the following steps:

[0052] Step 4.1: Let a source audio be A0. After being processed by librosa library, the Mel spectrum corresponding to the audio is recorded as X0.

[0053] Step 4.2: Let a target audio be A1. After being processed by librosa library, the Mel spectrum corresponding to the audio is X1.

[0054] Step 4.3: Input X0 to the encoder part of the system, and output a vector V0 with a length of 160*1; input X1 to the encoder part of the model, and output a vector V1 with a length of 160*1;

[0055] Step 4.4: Divide V0 and V1 into V according to the length of 90, 35 and 35 respectively. 0c ,V 0t ,V 0p and V 1c ,V 1t ,V 1p , representing the encoding of the content, timbre and rhythm of the source audio and the target audio respectively;

[0056] Step 4.5: Combine the target prosody code, source content code and source timbre code, input them into the decoder part of the system, and output the converted Mel spectrum.

[0057] The system framework constructed by the present invention includes only one encoder, so that the Mel spectrum information of the original input is not lost during the training process, and perfect reconstruction is achieved; the extractability of the three speech features of content, timbre and rhythm is ensured by combining the training of the encoder and the decoder through two constraint models; and the extractability of the three speech features of content, timbre and rhythm is further ensured by using a discriminator; when the present invention performs speech conversion on audio with different contents, the training of paired data is not involved, and the model training process only reconstructs itself, so efficient speech conversion is achieved.

[0058] The present invention inputs the Mel spectrum into the system for training. During the training process, the encoder outputs a vector with a length of 160*1, which will be divided into three segments (90, 35, 35), representing the encoding of content, timbre and rhythm respectively, and then the encoding of timbre and rhythm are respectively input into the corresponding constraint model, and supervised training is performed based on the real timbre vector and rhythm vector, so that the part of the component cut out from the large vector is sufficiently close to the real speech feature.

[0059] The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A speech conversion method based on an autoencoder framework, characterized in that: The following steps are involved: Step 1: Based on the open source audio processing library, extract the Mel spectrum features of the audio as a data set; Step 2: Build a system framework consisting of an encoder, a decoder, a discriminator, and two constraint models for timbre and rhythm respectively; Step 3: Input the data set in step 1 into the system built in step 2, train the constraint model until the loss function converges, and extract the speech features corresponding to the source speaker and the target speaker; Step 4: Based on the speech features extracted in step 3, replace the source prosodic code with the target code to complete the speech conversion task; The step 3 comprises the following steps: Step 3.1: The encoder outputs a vector of length 160*1; Step 3.2: Divide the vector in step 3.1 into three segments with lengths of 90, 35, and 35, representing the encoding of content, timbre, and rhythm respectively; Step 3.3: Input the encoding of timbre and rhythm into the corresponding constraint model, and perform supervised training based on the real timbre vector and rhythm vector; Step 3.4: Input the content, timbre and rhythm codes determined in step 3.2 into the discriminator for training until the distribution of the three codes is determined by the discriminator as three different categories; The step 4 comprises the following steps: Step 4.1: Let a source audio be A0. After being processed by librosa library, the Mel spectrum corresponding to the audio is recorded as X0. Step 4.2: Let a target audio be A1. After being processed by librosa library, the Mel spectrum corresponding to the audio is X1. Step 4.3: Input X0 into the encoder part of the system, and output a vector V0 with a length of 160*1; input X1 into the encoder part of the constraint model, and output a vector V1 with a length of 160*1; Step 4.4: Divide V0 and V1 into V according to the length of 90, 35 and 35 respectively. 0c ,V 0t ,V 0p and V 1c ,V 1t ,V 1p , representing the encoding of the content, timbre and rhythm of the source audio and the target audio respectively; Step 4.5: Combine the target prosody code, source content code and source timbre code, input them into the decoder part of the system, and output the converted Mel spectrum.

2. A method for voice conversion based on an automatic encoder framework according to claim 1, characterized in that: The encoder, decoder, constraint model and discriminator are all built based on the standards of Informer:AAAI2021 best paper.

3. A method for voice conversion based on an automatic encoder framework according to claim 2, characterized in that: The network structure of the system framework in step 2 consists of 2 or 3 layers of ProSparse Attention modules stacked together.

4. A method for voice conversion based on an automatic encoder framework according to claim 1, characterized in that: The loss functions described in step 3 are: the first is the reconstruction loss of the encoder and decoder. The encoder accepts the real Mel spectrum as input, and the training goal of the encoder and decoder is to reconstruct the corresponding Mel spectrum; the second is the loss function of timbre constraint; the third is the loss function of rhythm constraint; the fourth is the loss function of the discriminator for classification tasks.

5. A method for voice conversion based on an automatic encoder framework according to claim 1, characterized in that: The Mel spectrum corresponding to each audio in step 1 is a 480*80-dimensional matrix.