Voice decoupling method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a voice decoupling method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of audio processing. The method comprises the following steps: respectively inputting voice data to be decoupled into a timbre encoder and a rhythm encoder to obtain decoupled timbre information output by the timbre encoder and decoupled rhythm information output by the rhythm encoder; wherein the first reconstruction loss is determined based on the sample audio data of the first speaker and the reconstructed audio data of the first speaker, and the reconstructed audio data of the first speaker is reconstructed based on the target tone information corresponding to the first speaker and the target rhythm information corresponding to the first speaker. According to the invention, the tone keeping capability of the tone encoder can be restrained through the first reconstruction loss, so that the sufficiency of tone decoupling is improved, and the rhythm keeping capability of the rhythm encoder can be restrained, so that the sufficiency of rhythm decoupling is improved; moreover, the voice decoupling generalization can be improved.

Description

Technical Field

[0001] The present invention relates to the field of audio processing technology, and in particular to a speech decoupling method, device, electronic equipment, storage medium and program product. Background Art

[0002] With the rapid development of artificial intelligence, speech synthesis has been applied in more and more scenarios, such as personalized speech synthesis, timbre transfer, rhythm transfer and other speech synthesis directions. Among them, speech synthesis usually requires speech decoupling, which refers to decoupling human speech into three parts: content, rhythm and timbre. In order to achieve high-quality speech synthesis, it is necessary to effectively decouple speech content, timbre and rhythm while maintaining the naturalness of the speech.

[0003] At present, the speech decoupling model is trained by supervised training, and speech decoupling is performed based on the trained speech decoupling model. However, since the supervised training method requires labeling, it cannot be trained based on too many training samples, which makes it impossible to decouple rhythm and timbre well, and it cannot be trained based on a large amount of speaker audio data, resulting in low generalization of speech decoupling. Even if the speech decoupling model is trained by unsupervised training, the decoupling effect achieved by this training method often does not meet expectations, resulting in an inability to decouple rhythm and timbre well. Summary of the invention

[0004] The present invention provides a speech decoupling method, device, electronic device, storage medium and program product, which are used to solve the defect of incomplete timbre and rhythm decoupling in the prior art and realize full speech decoupling.

[0005] The present invention provides a speech decoupling method, comprising: Inputting the speech data to be decoupled into the timbre encoder and the prosody encoder respectively, obtaining the decoupled timbre information output by the timbre encoder and the decoupled prosody information output by the prosody encoder; Wherein, the timbre encoder and the prosody encoder are obtained by training based on a first reconstruction loss and a second reconstruction loss; The first reconstruction loss is determined based on sample audio data of a first speaker and reconstructed audio data of the first speaker, the reconstructed audio data of the first speaker is reconstructed based on target timbre information corresponding to the first speaker and target prosody information corresponding to the first speaker, the target timbre information corresponding to the first speaker is obtained by extracting the sample audio data of the first speaker by the timbre encoder, and the target prosody information corresponding to the first speaker is obtained by extracting the sample audio data of the first speaker by the prosody encoder; the second reconstruction loss is determined based on sample audio data of a second speaker and reconstructed audio data of the second speaker, the reconstructed audio data of the second speaker is reconstructed based on target timbre information corresponding to the second speaker and target prosody information corresponding to the second speaker, the target timbre information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker by the timbre encoder, and the target prosody information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker by the prosody encoder; The first speaker and the second speaker are different speakers.

[0006] According to a speech decoupling method provided by the present invention, the timbre encoder and the prosody encoder are trained based on the following method: Inputting the sample audio data of the first speaker and the sample audio data of the second speaker into the timbre encoder in the acoustic model respectively, obtaining the target timbre information respectively output by the timbre encoder, and inputting the sample audio data of the first speaker and the sample audio data of the second speaker into the prosody encoder in the acoustic model respectively, obtaining the target prosody information respectively output by the prosody encoder; Inputting the target timbre information corresponding to the first speaker and the target prosody information corresponding to the second speaker into the audio decoder in the acoustic model to obtain first reconstructed audio data output by the audio decoder; the audio decoder is used to reconstruct audio data based on the input information; Inputting the first reconstructed audio data into the timbre encoder to obtain reconstructed timbre information output by the timbre encoder, and inputting the first reconstructed audio data into the prosody encoder to obtain reconstructed prosody information output by the prosody encoder; Inputting the reconstructed timbre information and the target prosody information corresponding to the first speaker into the audio decoder to obtain second reconstructed audio data output by the audio decoder, and inputting the target timbre information and the reconstructed prosody information corresponding to the second speaker into the audio decoder to obtain third reconstructed audio data output by the audio decoder; determining a first reconstruction loss based on the sample audio data of the first speaker and the second reconstructed audio data, and determining a second reconstruction loss based on the sample audio data of the second speaker and the third reconstructed audio data; The acoustic model is trained based on the first reconstruction loss and the second reconstruction loss.

[0007] According to a speech decoupling method provided by the present invention, the sample audio data of the first speaker includes different first audio data and second audio data, and the sample audio data of the second speaker includes different third audio data and fourth audio data; The method comprises: inputting the sample audio data of the first speaker and the sample audio data of the second speaker into a timbre encoder in the acoustic model to obtain target timbre information respectively output by the timbre encoder, and inputting the sample audio data of the first speaker and the sample audio data of the second speaker into a prosody encoder in the acoustic model to obtain target prosody information respectively output by the prosody encoder, including: Inputting the second audio data and the fourth audio data into the timbre encoder respectively to obtain target timbre information respectively output by the timbre encoder, and inputting the first audio data and the third audio data into the prosody encoder respectively to obtain target prosody information respectively output by the prosody encoder; The step of inputting the target timbre information corresponding to the first speaker and the target prosody information corresponding to the second speaker into the audio decoder in the acoustic model to obtain first reconstructed audio data output by the audio decoder includes: Inputting the target timbre information corresponding to the second audio data of the first speaker and the target prosody information corresponding to the third audio data of the second speaker into the audio decoder to obtain first reconstructed audio data output by the audio decoder; The step of inputting the reconstructed timbre information and the target prosody information corresponding to the first speaker into the audio decoder to obtain second reconstructed audio data output by the audio decoder, and inputting the target timbre information and the reconstructed prosody information corresponding to the second speaker into the audio decoder to obtain third reconstructed audio data output by the audio decoder includes: Inputting the reconstructed timbre information and the target prosody information corresponding to the first audio data of the first speaker into the audio decoder to obtain second reconstructed audio data output by the audio decoder, and inputting the target timbre information and the reconstructed prosody information corresponding to the fourth audio data of the second speaker into the audio decoder to obtain third reconstructed audio data output by the audio decoder; The determining of a first reconstruction loss based on the sample audio data of the first speaker and the second reconstructed audio data, and determining of a second reconstruction loss based on the sample audio data of the second speaker and the third reconstructed audio data, comprises: A first reconstruction loss is determined based on the first audio data of the first speaker and the second reconstructed audio data, and a second reconstruction loss is determined based on the third audio data of the second speaker and the third reconstructed audio data.

[0008] According to a speech decoupling method provided by the present invention, the second audio data and the fourth audio data are respectively inputted into the timbre encoder to obtain target timbre information respectively outputted by the timbre encoder, and the first audio data and the third audio data are respectively inputted into the prosody encoder to obtain target prosody information respectively outputted by the prosody encoder, including: Inputting the second audio data and the fourth audio data into the timbre encoder respectively to obtain target timbre information respectively output by the timbre encoder, inputting the first audio data and the third audio data into the prosody encoder respectively to obtain target prosody information respectively output by the prosody encoder, and inputting the first text data corresponding to the first audio data and the third text data corresponding to the third audio data into the content encoder in the acoustic model respectively to obtain target content information respectively output by the content encoder; The step of inputting the target timbre information corresponding to the second audio data of the first speaker and the target prosody information corresponding to the third audio data of the second speaker into the audio decoder to obtain first reconstructed audio data output by the audio decoder includes: Inputting the target timbre information corresponding to the second audio data of the first speaker, the target prosody information corresponding to the third audio data of the second speaker, and the target content information corresponding to the third audio data of the second speaker into the audio decoder, and obtaining first reconstructed audio data output by the audio decoder; The step of inputting the reconstructed timbre information and the target prosody information corresponding to the first audio data of the first speaker into the audio decoder to obtain second reconstructed audio data output by the audio decoder, and inputting the target timbre information and the reconstructed prosody information corresponding to the fourth audio data of the second speaker into the audio decoder to obtain third reconstructed audio data output by the audio decoder comprises: The reconstructed timbre information and the target prosody information corresponding to the first audio data of the first speaker, and the target content information corresponding to the first audio data of the first speaker are input into the audio decoder to obtain second reconstructed audio data output by the audio decoder, and the target timbre information and the reconstructed prosody information corresponding to the fourth audio data of the second speaker, and the target content information corresponding to the third audio data of the second speaker are input into the audio decoder to obtain third reconstructed audio data output by the audio decoder.

[0009] According to a speech decoupling method provided by the present invention, the acoustic model is trained based on the first reconstruction loss and the second reconstruction loss, comprising: Inputting the reconstructed timbre information and the target prosody information corresponding to the second speaker into the audio decoder to obtain fourth reconstructed audio data output by the audio decoder, and inputting the target timbre information and the reconstructed prosody information corresponding to the first speaker into the audio decoder to obtain fifth reconstructed audio data output by the audio decoder; determining a third reconstruction loss based on the first reconstructed audio data and the fourth reconstructed audio data, and determining a fourth reconstruction loss based on the first reconstructed audio data and the fifth reconstructed audio data; The acoustic model is trained based on the first reconstruction loss, the second reconstruction loss, the third reconstruction loss, and the fourth reconstruction loss.

[0010] According to a speech decoupling method provided by the present invention, the sample audio data of the first speaker includes different first audio data and second audio data, and the sample audio data of the second speaker includes different third audio data and fourth audio data; The method comprises: inputting the sample audio data of the first speaker and the sample audio data of the second speaker into a timbre encoder in the acoustic model to obtain target timbre information respectively output by the timbre encoder, and inputting the sample audio data of the first speaker and the sample audio data of the second speaker into a prosody encoder in the acoustic model to obtain target prosody information respectively output by the prosody encoder, including: Inputting the second audio data and the fourth audio data into the timbre encoder respectively to obtain target timbre information respectively output by the timbre encoder, inputting the first audio data and the third audio data into the prosody encoder respectively to obtain target prosody information respectively output by the prosody encoder, and inputting the first text data corresponding to the first audio data and the third text data corresponding to the third audio data into the content encoder in the acoustic model respectively to obtain target content information respectively output by the content encoder; The step of inputting the target timbre information corresponding to the first speaker and the target prosody information corresponding to the second speaker into the audio decoder in the acoustic model to obtain first reconstructed audio data output by the audio decoder includes: Inputting the target timbre information corresponding to the second audio data of the first speaker, the target prosody information corresponding to the third audio data of the second speaker, and the target content information corresponding to the third audio data of the second speaker into the audio decoder, and obtaining first reconstructed audio data output by the audio decoder; The step of inputting the reconstructed timbre information and the target prosody information corresponding to the second speaker into the audio decoder to obtain fourth reconstructed audio data output by the audio decoder, and inputting the target timbre information and the reconstructed prosody information corresponding to the first speaker into the audio decoder to obtain fifth reconstructed audio data output by the audio decoder includes: The reconstructed timbre information and the target prosody information corresponding to the third audio data of the second speaker, and the target content information corresponding to the third audio data of the second speaker are input into the audio decoder to obtain fourth reconstructed audio data output by the audio decoder, and the target timbre information and the reconstructed prosody information corresponding to the second audio data of the first speaker, and the target content information corresponding to the third audio data of the second speaker are input into the audio decoder to obtain fifth reconstructed audio data output by the audio decoder.

[0011] According to a speech decoupling method provided by the present invention, the acoustic model is trained based on the first reconstruction loss, the second reconstruction loss, the third reconstruction loss and the fourth reconstruction loss, including: Inputting the first reconstructed audio data, the second reconstructed audio data, the third reconstructed audio data and the fourth reconstructed audio data into a discriminator respectively, and obtaining four discriminant losses output by the discriminator; The acoustic model is trained based on the first reconstruction loss, the second reconstruction loss, the third reconstruction loss, the fourth reconstruction loss and the four discriminant losses.

[0012] According to a speech decoupling method provided by the present invention, the sample audio data of a first speaker and the sample audio data of a second speaker are respectively input into a prosody encoder in an acoustic model to obtain target prosody information respectively output by the prosody encoder, including: Performing timbre change processing on the sample audio data of the first speaker and the sample audio data of the second speaker respectively, to obtain the sample audio data of the first speaker after the timbre change and the sample audio data of the second speaker after the timbre change; The sample audio data of the first speaker after the timbre change and the sample audio data of the second speaker after the timbre change are respectively input into the prosody encoder in the acoustic model to obtain the target prosody information respectively output by the prosody encoder.

[0013] The present invention also provides a speech decoupling device, comprising: A speech decoupling module, used for inputting the speech data to be decoupled into the timbre encoder and the prosody encoder respectively, to obtain the decoupled timbre information output by the timbre encoder and the decoupled prosody information output by the prosody encoder; Wherein, the timbre encoder and the prosody encoder are obtained by training based on a first reconstruction loss and a second reconstruction loss; The first reconstruction loss is determined based on sample audio data of a first speaker and reconstructed audio data of the first speaker, the reconstructed audio data of the first speaker is reconstructed based on target timbre information corresponding to the first speaker and target prosody information corresponding to the first speaker, the target timbre information corresponding to the first speaker is obtained by extracting the sample audio data of the first speaker by the timbre encoder, and the target prosody information corresponding to the first speaker is obtained by extracting the sample audio data of the first speaker by the prosody encoder; the second reconstruction loss is determined based on sample audio data of a second speaker and reconstructed audio data of the second speaker, the reconstructed audio data of the second speaker is reconstructed based on target timbre information corresponding to the second speaker and target prosody information corresponding to the second speaker, the target timbre information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker by the timbre encoder, and the target prosody information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker by the prosody encoder; The first speaker and the second speaker are different speakers.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the speech decoupling method described above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the speech decoupling method as described above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the speech decoupling method described above is implemented.

[0017] The speech decoupling method, device, electronic device, storage medium and program product provided by the present invention, the timbre encoder and the prosody encoder are obtained by training based on a first reconstruction loss and a second reconstruction loss, and the first reconstruction loss is determined based on sample audio data of a first speaker and the reconstructed audio data of the first speaker, the reconstructed audio data of the first speaker is reconstructed based on target timbre information corresponding to the first speaker and target prosody information corresponding to the first speaker, and the target timbre information corresponding to the first speaker is obtained by extracting the sample audio data of the first speaker by the timbre encoder, and the target prosody information corresponding to the first speaker is obtained by extracting the sample audio data of the first speaker by the prosody encoder, so that the first reconstruction loss can constrain the timbre retention capability of the timbre encoder for the first speaker, thereby improving the adequacy of timbre decoupling, and can constrain the prosody retention capability of the prosody encoder for the first speaker, thereby improving the adequacy of prosody decoupling; at the same time, the second reconstruction loss The loss is determined based on the sample audio data of the second speaker and the reconstructed audio data of the second speaker, the reconstructed audio data of the second speaker is reconstructed based on the target timbre information corresponding to the second speaker and the target prosody information corresponding to the second speaker, and the target timbre information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker based on the timbre encoder, and the target prosody information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker based on the prosody encoder, so that the second reconstruction loss can constrain the timbre retention ability of the timbre encoder for the second speaker, thereby improving the adequacy of timbre decoupling, and can constrain the prosody retention ability of the prosody encoder for the second speaker, thereby improving the adequacy of prosody decoupling; in addition, the first speaker and the second speaker are different speakers, so that the model training can be based on the sample audio data of different speakers, that is, it can be trained based on the audio data of a large number of speakers to improve the generalization of speech decoupling. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is one of the flow charts of the speech decoupling method provided by the present invention.

[0020] Figure 2 This is the second flow chart of the speech decoupling method provided by the present invention.

[0021] Figure 3 This is the third flow chart of the speech decoupling method provided by the present invention.

[0022] Figure 4 This is the fourth flow chart of the speech decoupling method provided by the present invention.

[0023] Figure 5 It is a structural schematic diagram of the speech decoupling device provided by the present invention.

[0024] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] With the rapid development of artificial intelligence, speech synthesis technology has also made significant progress, and the application scenarios of speech synthesis are becoming more and more extensive, such as personalized speech synthesis, timbre transfer, rhythm transfer, sound editing and other speech synthesis directions. Among them, speech synthesis usually requires speech decoupling, which refers to decoupling human speech into three parts: content, rhythm and timbre. In order to achieve high-quality speech synthesis, it is necessary to achieve effective decoupling of speech content, timbre and rhythm while maintaining the naturalness of the speech.

[0027] However, these three attributes are closely intertwined in speech signals and difficult to separate directly. Traditional speech decoupling methods often rely on manual feature engineering, which is not only time-consuming and labor-intensive, but also difficult to adapt to different languages ​​and accents.

[0028] It should be noted that in human communication, speech is an important means of conveying information and expressing emotions. Speech signals contain rich information, including not only the content of speech (i.e. text information), but also the speaker's timbre (i.e. the characteristics of the voice) and prosody (i.e. the rhythm and emphasis of speech). These attributes together constitute the uniqueness and diversity of human speech.

[0029] At present, the speech decoupling model is trained by supervised training, and speech decoupling is performed based on the trained speech decoupling model. However, since the supervised training method requires labeling, it cannot be trained based on too many training samples, which makes it impossible to decouple rhythm and timbre well, and it cannot be trained based on a large amount of speaker audio data, resulting in low generalization of speech decoupling. Even if the speech decoupling model is trained by unsupervised training, the decoupling effect achieved by this training method often does not meet expectations, resulting in an inability to decouple rhythm and timbre well.

[0030] For example, a supervised solution is used to decouple content information using the phoneme prediction task, timbre information using the speaker classification task, and rhythm information using the fundamental frequency prediction task. At the same time, a gradient reversal scheme is used to reduce the leakage of other attributes of each branch. However, supervised speech decoupling solutions often require careful design of the proportion of various losses, and the stability of model training is very poor. The speaker prediction task cannot receive massive speaker data, so the timbre generalization is not satisfactory.

[0031] In view of the above problems, the present invention proposes the following embodiments. Figure 1-Figure 4 The speech decoupling method of the present invention is described.

[0032] Figure 1 It is one of the flow charts of the speech decoupling method provided by the present invention, such as Figure 1 As shown, the speech decoupling method includes the following step 110.

[0033] Step 110: input the speech data to be decoupled into a timbre encoder and a prosody encoder respectively, to obtain decoupled timbre information output by the timbre encoder and decoupled prosody information output by the prosody encoder.

[0034] Here, the voice data to be decoupled is audio data to be decoupled, and the voice data to be decoupled may be voice data collected from a speaker. Further, the voice data to be decoupled may be mel spectrum data.

[0035] Here, the timbre encoder is used to decouple the timbre information in the voice data to be decoupled to obtain decoupled timbre information. The prosody encoder is used to decouple the prosody information in the voice data to be decoupled to obtain decoupled prosody information.

[0036] The timbre information decoupled by the timbre encoder includes the global voiceprint information of the speaker. For example, the timbre encoder can obtain a global timbre vector from the mel spectrum data (audio data), that is, timbre is a global attribute that does not change with the length of the audio. Therefore, the timbre information extracted from two voices of the same speaker should represent the timbre of the same speaker and they should be similar enough.

[0037] The prosody information decoupled by the prosody encoder may include, but is not limited to, information such as the speaker's speaking style, rhythm, intonation, and emotions.

[0038] The timbre encoder and the prosody encoder are trained based on the first reconstruction loss and the second reconstruction loss, that is, the timbre encoder and the prosody encoder are trained based on the first reconstruction loss and the second reconstruction loss.

[0039] Among them, the first reconstruction loss is determined based on sample audio data of a first speaker and reconstructed audio data of the first speaker, the reconstructed audio data of the first speaker is reconstructed based on target timbre information corresponding to the first speaker and target prosody information corresponding to the first speaker, the target timbre information corresponding to the first speaker is obtained by extracting the sample audio data of the first speaker based on the timbre encoder, and the target prosody information corresponding to the first speaker is obtained by extracting the sample audio data of the first speaker based on the prosody encoder.

[0040] Here, the sample audio data of the first speaker may be voice data collected from the first speaker.

[0041] The reconstructed audio data of the first speaker is reconstructed based on the target timbre information corresponding to the first speaker and the target prosody information corresponding to the first speaker, and the target timbre information corresponding to the first speaker is obtained by extracting the sample audio data of the first speaker based on the timbre encoder, and the target prosody information corresponding to the first speaker is obtained by extracting the sample audio data of the first speaker based on the prosody encoder. Therefore, if the speech decoupling of the timbre encoder and the prosody encoder is more accurate and sufficient, the first reconstruction loss should be as small as possible. Based on this, the first reconstruction loss can constrain the timbre encoder's ability to maintain the timbre of the first speaker, thereby improving the adequacy of the timbre decoupling, and can constrain the prosody encoder's ability to maintain the prosody of the first speaker, thereby improving the adequacy of the prosody decoupling.

[0042] The second reconstruction loss is determined based on sample audio data of a second speaker and reconstructed audio data of the second speaker, the reconstructed audio data of the second speaker is reconstructed based on target timbre information corresponding to the second speaker and target prosody information corresponding to the second speaker, the target timbre information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker based on the timbre encoder, and the target prosody information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker based on the prosody encoder.

[0043] Here, the sample audio data of the second speaker may be voice data collected from the second speaker.

[0044] The reconstructed audio data of the second speaker is reconstructed based on the target timbre information corresponding to the second speaker and the target prosody information corresponding to the second speaker, and the target timbre information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker through the timbre encoder, and the target prosody information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker through the prosody encoder. Therefore, if the speech decoupling of the timbre encoder and the prosody encoder is more accurate and sufficient, the second reconstruction loss should be as small as possible. Based on this, the second reconstruction loss can constrain the timbre encoder's ability to maintain the timbre of the second speaker, thereby improving the adequacy of the timbre decoupling, and can constrain the prosody encoder's ability to maintain the prosody of the second speaker, thereby improving the adequacy of the prosody decoupling.

[0045] Among them, the first speaker and the second speaker are different speakers, so the model training can be performed based on sample audio data of different speakers, that is, training can be performed based on audio data of a large number of speakers, thereby improving the generalization of speech decoupling, that is, the trained timbre encoder and prosody encoder can be transplanted to various scenarios for the purpose of decoupling.

[0046] Further, the sample audio data of the first speaker includes different first audio data and second audio data, and the sample audio data of the second speaker includes different third audio data and fourth audio data. The first reconstruction loss is determined based on the first audio data of the first speaker and the reconstructed audio data of the first speaker, the reconstructed audio data of the first speaker is reconstructed based on the target timbre information corresponding to the first speaker and the target prosody information corresponding to the first speaker, the target timbre information corresponding to the first speaker is obtained by extracting the second audio data of the first speaker based on the timbre encoder, and the target prosody information corresponding to the first speaker is obtained by extracting the first audio data of the first speaker based on the prosody encoder. The second reconstruction loss is determined based on the third audio data of the second speaker and the reconstructed audio data of the second speaker, the reconstructed audio data of the second speaker is reconstructed based on the target timbre information corresponding to the second speaker and the target prosody information corresponding to the second speaker, the target timbre information corresponding to the second speaker is obtained by extracting the fourth audio data of the second speaker based on the timbre encoder, and the target prosody information corresponding to the second speaker is obtained by extracting the third audio data of the second speaker based on the prosody encoder.

[0047] Based on this, since the first audio data and the second audio data are audio data of the same speaker, the timbre information of the first audio data and the second audio data is the same, but the prosody information of the first audio data and the second audio data is different, and since the third audio data and the fourth audio data are audio data of the same speaker, the timbre information of the third audio data and the fourth audio data is the same, but the prosody information of the third audio data and the fourth audio data is different, therefore, the first reconstruction loss and the second reconstruction loss can constrain the stability of the timbre extraction of different voices of the same person by the timbre encoder, thereby improving the stability of timbre decoupling, and can constrain the stability of the prosody extraction of the prosody encoder for the same prosodic audio of different people, thereby improving the stability of prosody decoupling.

[0048] Furthermore, the speech data to be decoupled is respectively input into the timbre encoder, the prosody encoder and the content encoder to obtain the decoupled timbre information output by the timbre encoder, the decoupled prosody information output by the prosody encoder, and the decoupled content information output by the content encoder.

[0049] Here, the content information decoupled by the content encoder includes the speech content of the speaker.

[0050] In one embodiment, the content encoder includes multiple Conformer layers for capturing the contextual relationship of text data and encoding it into content information.

[0051] Wherein, the timbre encoder, the prosody encoder and the content encoder are trained based on the first reconstruction loss and the second reconstruction loss. The first reconstruction loss is determined based on the sample audio data of the first speaker and the reconstructed audio data of the first speaker, the reconstructed audio data of the first speaker is reconstructed based on the target timbre information corresponding to the first speaker, the target prosody information corresponding to the first speaker and the target content information corresponding to the first speaker, the target timbre information corresponding to the first speaker is obtained by extracting the sample audio data of the first speaker based on the timbre encoder, the target prosody information corresponding to the first speaker is obtained by extracting the sample audio data of the first speaker based on the prosody encoder, and the target content information corresponding to the first speaker is obtained by extracting the text data corresponding to the sample audio data of the first speaker based on the content encoder; The second reconstruction loss is determined based on the sample audio data of the second speaker and the reconstructed audio data of the second speaker, the reconstructed audio data of the second speaker is reconstructed based on the target timbre information corresponding to the second speaker, the target prosody information corresponding to the second speaker and the target content information corresponding to the second speaker, the target timbre information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker based on a timbre encoder, the target prosody information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker based on a prosody encoder, and the target content information corresponding to the second speaker is obtained by extracting text data corresponding to the sample audio data of the second speaker based on a content encoder.

[0052] Based on this, the first reconstruction loss can also constrain the content encoder's ability to maintain the content of the audio data, thereby improving the adequacy of content decoupling. The second reconstruction loss can also constrain the content encoder's ability to maintain the content of the audio data, thereby improving the adequacy of content decoupling.

[0053] Furthermore, during the training process of the prosody encoder, the sample audio data of the first speaker and the sample audio data of the second speaker are respectively processed with timbre change to obtain the sample audio data of the first speaker after the timbre change and the sample audio data of the second speaker after the timbre change; the sample audio data of the first speaker after the timbre change and the sample audio data of the second speaker after the timbre change are respectively input into the prosody encoder to obtain target prosody information output by the prosody encoder; based on this, it is ensured that the prosody encoder focuses only on the extraction of prosody information as much as possible, thereby reducing the leakage of timbre information and content information.

[0054] In the speech decoupling method provided by the embodiment of the present invention, the timbre encoder and the prosody encoder are obtained by training based on the first reconstruction loss and the second reconstruction loss, and the first reconstruction loss is determined based on the sample audio data of the first speaker and the reconstructed audio data of the first speaker, the reconstructed audio data of the first speaker is reconstructed based on the target timbre information corresponding to the first speaker and the target prosody information corresponding to the first speaker, and the target timbre information corresponding to the first speaker is obtained by extracting the sample audio data of the first speaker by the timbre encoder, and the target prosody information corresponding to the first speaker is obtained by extracting the sample audio data of the first speaker by the prosody encoder, so that the first reconstruction loss can constrain the timbre retention capability of the timbre encoder for the first speaker, thereby improving the adequacy of timbre decoupling, and can constrain the prosody retention capability of the prosody encoder for the first speaker, thereby improving the adequacy of prosody decoupling; at the same time, the second reconstruction loss is determined based on the second reconstruction loss. The first speaker and the second speaker are determined by the sample audio data of the speaker and the reconstructed audio data of the second speaker, the reconstructed audio data of the second speaker is reconstructed based on the target timbre information corresponding to the second speaker and the target prosody information corresponding to the second speaker, and the target timbre information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker based on the timbre encoder, and the target prosody information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker based on the prosody encoder, so that the second reconstruction loss can constrain the timbre retention ability of the timbre encoder for the second speaker, thereby improving the adequacy of timbre decoupling, and can constrain the prosody retention ability of the prosody encoder for the second speaker, thereby improving the adequacy of prosody decoupling; in addition, the first speaker and the second speaker are different speakers, so that the model training can be based on the sample audio data of different speakers, that is, it can be trained based on the audio data of a large number of speakers to improve the generalization of speech decoupling.

[0055] Based on any of the above embodiments, Figure 2 FIG. 2 is a flow chart of the voice decoupling method provided by the present invention. Figure 2 As shown, the timbre encoder and the prosody encoder are trained based on the following method, which includes steps 210 to 260.

[0056] Step 210: input the sample audio data of the first speaker and the sample audio data of the second speaker into the timbre encoder in the acoustic model respectively to obtain the target timbre information output by the timbre encoder respectively; and input the sample audio data of the first speaker and the sample audio data of the second speaker into the prosody encoder in the acoustic model respectively to obtain the target prosody information output by the prosody encoder respectively.

[0057] Specifically, the sample audio data of the first speaker is input into the timbre encoder to obtain the target timbre information corresponding to the first speaker output by the timbre encoder; the sample audio data of the second speaker is input into the timbre encoder to obtain the target timbre information corresponding to the second speaker output by the timbre encoder; the sample audio data of the first speaker is input into the rhythm encoder to obtain the target rhythm information corresponding to the first speaker output by the rhythm encoder; the sample audio data of the second speaker is input into the rhythm encoder to obtain the target rhythm information corresponding to the second speaker output by the rhythm encoder.

[0058] Furthermore, the sample audio data of the first speaker and the sample audio data of the second speaker are respectively input into the content encoder in the acoustic model to obtain the target content information respectively output by the content encoder.

[0059] Step 220: input the target timbre information corresponding to the first speaker and the target prosody information corresponding to the second speaker into the audio decoder in the acoustic model to obtain first reconstructed audio data output by the audio decoder.

[0060] The audio decoder is used to reconstruct the audio data based on the input information, that is, to reconstruct the first reconstructed audio data based on the target timbre information corresponding to the first speaker and the target rhythm information corresponding to the second speaker.

[0061] In one embodiment, the audio decoder comprises a plurality of residual convolution modules with skip connections.

[0062] Furthermore, the target timbre information corresponding to the first speaker, the target prosody information corresponding to the second speaker, and the target content information corresponding to the sample audio data of the second speaker are input into the audio decoder to obtain the first reconstructed audio data output by the audio decoder.

[0063] Step 230: input the first reconstructed audio data to the timbre encoder to obtain reconstructed timbre information output by the timbre encoder, and input the first reconstructed audio data to the prosody encoder to obtain reconstructed prosody information output by the prosody encoder.

[0064] It should be noted that, since the first reconstructed audio data corresponds to the timbre of the first speaker, the extracted reconstructed timbre information also corresponds to the timbre of the first speaker; and since the first reconstructed audio data corresponds to the rhythm of the sample audio data of the second speaker, the extracted reconstructed rhythm information also corresponds to the rhythm of the sample audio data of the second speaker.

[0065] Step 240: input the reconstructed timbre information and the target prosody information corresponding to the first speaker into the audio decoder to obtain second reconstructed audio data output by the audio decoder, and input the target timbre information and the reconstructed prosody information corresponding to the second speaker into the audio decoder to obtain third reconstructed audio data output by the audio decoder.

[0066] Furthermore, the reconstructed timbre information, the target prosody information corresponding to the first speaker, and the target content information corresponding to the sample audio data of the first speaker are input into the audio decoder to obtain second reconstructed audio data output by the audio decoder, and the target timbre information, the reconstructed prosody information corresponding to the second speaker, and the target content information corresponding to the sample audio data of the second speaker are input into the audio decoder to obtain third reconstructed audio data output by the audio decoder.

[0067] Step 250: determining a first reconstruction loss based on the sample audio data of the first speaker and the second reconstructed audio data, and determining a second reconstruction loss based on the sample audio data of the second speaker and the third reconstructed audio data.

[0068] Based on this, the first reconstruction loss can constrain the timbre preservation capability of the timbre encoder for the first speaker, thereby improving the adequacy of timbre decoupling, and can constrain the prosody preservation capability of the prosody encoder for the first speaker, thereby improving the adequacy of prosody decoupling. The second reconstruction loss can constrain the timbre preservation capability of the timbre encoder for the second speaker, thereby improving the adequacy of timbre decoupling, and can constrain the prosody preservation capability of the prosody encoder for the second speaker, thereby improving the adequacy of prosody decoupling.

[0069] Step 260: training the acoustic model based on the first reconstruction loss and the second reconstruction loss.

[0070] Specifically, based on the first reconstruction loss and the second reconstruction loss, each network layer in the acoustic model is optimized, that is, the timbre encoder, the prosody encoder and the audio decoder are optimized.

[0071] Furthermore, the first reconstructed audio data, the second reconstructed audio data and the third reconstructed audio data are respectively input to the discriminator to obtain the discrimination loss output by the discriminator; and the acoustic model is trained based on the first reconstruction loss, the second reconstruction loss and the discrimination loss. Based on this, training is performed not only based on the reconstruction loss but also based on the discrimination loss, thereby improving the decoupling adequacy of the timbre encoder, improving the decoupling adequacy of the prosody encoder, and improving the audio reconstruction accuracy of the audio decoder.

[0072] It should be noted that the training process of the acoustic model can be executed on the execution body of the speech decoupling method, or on other training devices, and then the trained timbre encoder and prosody encoder can be deployed on the execution body of the speech decoupling method.

[0073] The speech decoupling method provided by the embodiment of the present invention, through the above-mentioned manner, the first reconstruction loss can constrain the timbre retention capability of the timbre encoder for the first speaker, thereby improving the adequacy of timbre decoupling, and can constrain the prosody retention capability of the prosody encoder for the first speaker, thereby improving the adequacy of prosody decoupling; at the same time, the second reconstruction loss can constrain the timbre retention capability of the timbre encoder for the second speaker, thereby improving the adequacy of timbre decoupling, and can constrain the prosody retention capability of the prosody encoder for the second speaker, thereby improving the adequacy of prosody decoupling; in addition, the first speaker and the second speaker are different speakers, so that the model training can be performed based on the sample audio data of different speakers, that is, the training can be performed based on the audio data of a large number of speakers, thereby improving the generalization of speech decoupling; in addition, the sample audio data of the first speaker and the sample audio data of the second speaker are firstly subjected to audio analysis, and the audio data of the first speaker and the sample audio data of the second speaker are first subjected to audio analysis, and the ... The timbre information is extracted and the rhythm information is extracted, and then the first reconstructed audio data is obtained based on the target timbre information corresponding to the first speaker and the target rhythmic information corresponding to the second speaker. Then, the timbre information and rhythmic information are extracted again for the first reconstructed audio data, and then the second reconstructed audio data is obtained based on the reconstructed timbre information and the target rhythmic information corresponding to the first speaker, and the third reconstructed audio data is obtained based on the target timbre information and the reconstructed rhythmic information corresponding to the second speaker. Thus, the timbre encoder performs timbre extraction twice, and then the first reconstruction loss can constrain the timbre encoder's ability to maintain the timbre of the first speaker twice, thereby further improving the adequacy of timbre decoupling. At the same time, the rhythm encoder performs rhythm extraction twice, and the second reconstruction loss can constrain the rhythm encoder's ability to maintain the rhythm of the second speaker twice, thereby further improving the adequacy of rhythm decoupling.

[0074] Based on any of the above embodiments, in the method, the sample audio data of the first speaker includes different first audio data and second audio data, and the sample audio data of the second speaker includes different third audio data and fourth audio data.

[0075] In other words, two audio data of two different speakers are obtained. Further, each audio data may be Mel-spectrogram data, that is, the first audio data, the second audio data, the third audio data and the fourth audio data are all Mel-spectrogram data. For example, the Mel-spectrogram data is obtained by converting the amplitude spectrum obtained by short-time Fourier transform of the audio to Mel scale.

[0076] Since the first audio data and the second audio data are audio data of the same speaker, the timbre information of the first audio data and the second audio data is the same, but the rhythm information of the first audio data and the second audio data is different. Since the third audio data and the fourth audio data are audio data of the same speaker, the timbre information of the third audio data and the fourth audio data is the same, but the rhythm information of the third audio data and the fourth audio data is different.

[0077] Accordingly, if Figure 3 As shown, the above step 210 includes: inputting the second audio data and the fourth audio data into the timbre encoder respectively, obtaining the target timbre information output by the timbre encoder respectively, and inputting the first audio data and the third audio data into the rhythm encoder respectively, obtaining the target rhythm information output by the rhythm encoder respectively.

[0078] Specifically, the second audio data of the first speaker is input into the timbre encoder to obtain the target timbre information corresponding to the second audio data of the first speaker output by the timbre encoder; the fourth audio data of the second speaker is input into the timbre encoder to obtain the target timbre information corresponding to the fourth audio data of the second speaker output by the timbre encoder; the first audio data of the first speaker is input into the rhythm encoder to obtain the target rhythm information corresponding to the first audio data of the first speaker output by the rhythm encoder; the third audio data of the second speaker is input into the rhythm encoder to obtain the target rhythm information corresponding to the third audio data of the second speaker output by the rhythm encoder.

[0079] Accordingly, if Figure 3 As shown, the above step 220 includes: inputting the target timbre information corresponding to the second audio data of the first speaker and the target prosody information corresponding to the third audio data of the second speaker into the audio decoder to obtain the first reconstructed audio data output by the audio decoder.

[0080] Based on this, the timbre information of the first reconstructed data is the timbre information of the first speaker, and the prosody information of the first reconstructed data is the prosody information of the third audio data of the second speaker.

[0081] Accordingly, if Figure 3 As shown, the above-mentioned step 240 includes: inputting the reconstructed timbre information and the target prosody information corresponding to the first audio data of the first speaker into the audio decoder to obtain second reconstructed audio data output by the audio decoder, and inputting the target timbre information and the reconstructed prosody information corresponding to the fourth audio data of the second speaker into the audio decoder to obtain third reconstructed audio data output by the audio decoder.

[0082] Accordingly, if Figure 3 As shown, the above step 250 includes: determining a first reconstruction loss based on the first audio data of the first speaker and the second reconstructed audio data, and determining a second reconstruction loss based on the third audio data of the second speaker and the third reconstructed audio data.

[0083] In the speech decoupling method provided by the embodiment of the present invention, since the first audio data and the second audio data are audio data of the same speaker, the timbre information of the first audio data and the second audio data are the same, but the prosody information of the first audio data and the second audio data are different. Therefore, the second reconstructed audio data is reconstructed based on the timbre information of the second audio data of the first speaker and the prosody information of the first audio data of the first speaker in the above manner, and then the first reconstruction loss is determined based on the first audio data and the second reconstructed audio data of the first speaker. Therefore, the first reconstruction loss can constrain the stability of the timbre extraction of different voices of the same person by the timbre encoder, thereby improving the timbre decomposition. coupling stability; at the same time, since the third audio data and the fourth audio data are audio data of the same speaker, the timbre information of the third audio data and the fourth audio data is the same, but the prosody information of the third audio data and the fourth audio data is different. Therefore, the third reconstructed audio data is reconstructed based on the timbre information of the fourth audio data of the second speaker and the prosody information of the third audio data of the second speaker in the above manner, and then the second reconstruction loss is determined based on the third audio data of the second speaker and the third reconstructed audio data. Therefore, the second reconstruction loss can constrain the prosody extraction stability of the prosody encoder for the same prosody audio of different people, thereby improving the stability of prosody decoupling.

[0084] Based on any of the above embodiments, Figure 4 As shown, the above step 210 includes: inputting the second audio data and the fourth audio data into the timbre encoder respectively to obtain the target timbre information respectively output by the timbre encoder, and inputting the first audio data and the third audio data into the prosody encoder respectively to obtain the target prosody information respectively output by the prosody encoder, and inputting the first text data corresponding to the first audio data and the third text data corresponding to the third audio data into the content encoder in the acoustic model respectively to obtain the target content information respectively output by the content encoder.

[0085] Here, the first text data or the third text data may be text transcribed by a speech recognition system or may be manually annotated text.

[0086] Specifically, the first text data corresponding to the first audio data is input into the content encoder to obtain the target content information corresponding to the first audio data of the first speaker output by the content encoder; the third text data corresponding to the third audio data is input into the content encoder to obtain the target content information corresponding to the third audio data of the second speaker output by the content encoder.

[0087] Accordingly, if Figure 4 As shown, the above-mentioned step 220 includes: inputting the target timbre information corresponding to the second audio data of the first speaker, the target rhythm information corresponding to the third audio data of the second speaker, and the target content information corresponding to the third audio data of the second speaker into the audio decoder to obtain the first reconstructed audio data output by the audio decoder.

[0088] Based on this, the timbre information of the first reconstructed data is the timbre information of the first speaker, the prosody information of the first reconstructed data is the prosody information of the third audio data of the second speaker, and the content information of the first reconstructed data is the content information of the third audio data of the second speaker. That is, it is ensured that the speakers corresponding to the timbre information and the prosody information are different, but the speakers and audio data corresponding to the prosody information and the content information are the same.

[0089] Accordingly, if Figure 4 As shown, the above-mentioned step 240 includes: inputting the reconstructed timbre information and the target prosody information corresponding to the first audio data of the first speaker, and the target content information corresponding to the first audio data of the first speaker into the audio decoder to obtain second reconstructed audio data output by the audio decoder, and inputting the target timbre information and the reconstructed prosody information corresponding to the fourth audio data of the second speaker, and the target content information corresponding to the third audio data of the second speaker into the audio decoder to obtain third reconstructed audio data output by the audio decoder.

[0090] It should be understood that the second reconstructed audio data should maintain the timbre of the first reconstructed audio data, that is, the timbre of the first speaker, while maintaining the rhythm and content of the first audio data of the first speaker. The third reconstructed audio data should maintain the timbre of the fourth audio data of the second speaker, that is, the timbre of the second speaker, while maintaining the rhythm of the third audio data, that is, the rhythm of the third audio data of the second speaker, while maintaining the content of the third audio data of the second speaker.

[0091] Accordingly, based on the first reconstruction loss and the second reconstruction loss, each network layer in the acoustic model is optimized, that is, the timbre encoder, the prosody encoder, the audio decoder and the audio decoder are optimized.

[0092] Accordingly, the speech data to be decoupled is input into the trained content encoder to obtain the decoupled content information output by the content encoder.

[0093] In the speech decoupling method provided by an embodiment of the present invention, the timbre encoder, prosody encoder and content encoder are trained based on the first reconstruction loss and the second reconstruction loss. Through the above method, the first reconstruction loss and the second reconstruction loss can constrain the content encoder's ability to retain the content of the audio data, thereby improving the adequacy of content decoupling.

[0094] Based on any of the above embodiments, in the method, the above step 260 includes: Inputting the reconstructed timbre information and the target prosody information corresponding to the second speaker into the audio decoder to obtain fourth reconstructed audio data output by the audio decoder, and inputting the target timbre information and the reconstructed prosody information corresponding to the first speaker into the audio decoder to obtain fifth reconstructed audio data output by the audio decoder; determining a third reconstruction loss based on the first reconstructed audio data and the fourth reconstructed audio data, and determining a fourth reconstruction loss based on the first reconstructed audio data and the fifth reconstructed audio data; The acoustic model is trained based on the first reconstruction loss, the second reconstruction loss, the third reconstruction loss, and the fourth reconstruction loss.

[0095] It should be understood that the fourth reconstructed audio data should maintain the timbre of the first reconstructed audio data, that is, the timbre of the first speaker, while maintaining the rhythm and content of the third audio data of the second speaker. The fifth reconstructed audio data should maintain the timbre of the first speaker, while maintaining the rhythm of the third audio data, that is, the rhythm of the third audio data of the second speaker, while maintaining the content of the third audio data of the second speaker.

[0096] Based on this, the third reconstruction loss can constrain the timbre preservation capability of the timbre encoder for the first speaker, thereby improving the adequacy of timbre decoupling, and can constrain the prosody preservation capability of the prosody encoder for the second speaker, thereby improving the adequacy of prosody decoupling. The fourth reconstruction loss can constrain the timbre preservation capability of the timbre encoder for the first speaker, thereby improving the adequacy of timbre decoupling, and can constrain the prosody preservation capability of the prosody encoder for the second speaker, thereby improving the adequacy of prosody decoupling.

[0097] Specifically, based on the first reconstruction loss, the second reconstruction loss, the third reconstruction loss and the fourth reconstruction loss, each network layer in the acoustic model is optimized, that is, the timbre encoder, the prosody encoder and the audio decoder are optimized.

[0098] Furthermore, the sample audio data of the first speaker includes different first audio data and second audio data, and the sample audio data of the second speaker includes different third audio data and fourth audio data. The step of inputting the sample audio data of the first speaker and the sample audio data of the second speaker into the timbre encoder in the acoustic model to obtain target timbre information respectively output by the timbre encoder, and inputting the sample audio data of the first speaker and the sample audio data of the second speaker into the prosody encoder in the acoustic model to obtain target prosody information respectively output by the prosody encoder comprises: inputting the second audio data and the fourth audio data into the timbre encoder to obtain target timbre information respectively output by the timbre encoder, and inputting the first audio data and the third audio data into the prosody encoder to obtain target prosody information respectively output by the prosody encoder. The step of inputting the target timbre information corresponding to the first speaker and the target prosody information corresponding to the second speaker into the audio decoder in the acoustic model to obtain the first reconstructed audio data output by the audio decoder includes: inputting the target timbre information corresponding to the second audio data of the first speaker and the target prosody information corresponding to the third audio data of the second speaker into the audio decoder to obtain the first reconstructed audio data output by the audio decoder. The step of inputting the reconstructed timbre information and the target prosody information corresponding to the second speaker into the audio decoder to obtain the fourth reconstructed audio data output by the audio decoder, and inputting the target timbre information and the reconstructed prosody information corresponding to the first speaker into the audio decoder to obtain the fifth reconstructed audio data output by the audio decoder includes: inputting the reconstructed timbre information and the target prosody information corresponding to the third audio data of the second speaker into the audio decoder to obtain the fourth reconstructed audio data output by the audio decoder, and inputting the target timbre information and the reconstructed prosody information corresponding to the second audio data of the first speaker into the audio decoder to obtain the fifth reconstructed audio data output by the audio decoder.

[0099] Furthermore, the reconstructed timbre information and the target prosody information corresponding to the second speaker, and the target content information corresponding to the sample audio data of the second speaker are input into the audio decoder to obtain the fourth reconstructed audio data output by the audio decoder, and the target timbre information and the reconstructed prosody information corresponding to the first speaker, and the target content information corresponding to the sample audio data of the second speaker are input into the audio decoder to obtain the fifth reconstructed audio data output by the audio decoder.

[0100] The speech decoupling method provided by the embodiment of the present invention, through the above-mentioned manner, the third reconstruction loss can constrain the timbre retaining ability of the timbre encoder for the first speaker, thereby improving the adequacy of the timbre decoupling, and compared with the first reconstruction loss, the reconstruction target is changed from the real audio data to the first reconstructed audio data, and the transmission path of the gradient flow is shorter, thereby improving the stability of the model training, that is, improving the stability of the timbre decoupling; at the same time, the fourth reconstruction loss can constrain the rhythm encoder for the second speaker The rhythm retaining ability, thereby improving the adequacy of the rhythm decoupling, and compared with the second reconstruction loss, the reconstruction target is changed from the real audio data to the first reconstructed audio data, and the transmission path of the gradient flow is shorter, thereby improving the stability of the model training, that is, improving the stability of the rhythm decoupling.

[0101] Based on any of the above embodiments, in the method, the sample audio data of the first speaker includes different first audio data and second audio data, and the sample audio data of the second speaker includes different third audio data and fourth audio data.

[0102] Correspondingly, the above step 210 includes: inputting the second audio data and the fourth audio data into the timbre encoder respectively to obtain target timbre information respectively output by the timbre encoder, and inputting the first audio data and the third audio data into the prosody encoder respectively to obtain target prosody information respectively output by the prosody encoder, and inputting the first text data corresponding to the first audio data and the third text data corresponding to the third audio data into the content encoder in the acoustic model respectively to obtain target content information respectively output by the content encoder; Correspondingly, the above-mentioned step 220 includes: inputting the target timbre information corresponding to the second audio data of the first speaker, the target rhythm information corresponding to the third audio data of the second speaker, and the target content information corresponding to the third audio data of the second speaker into the audio decoder to obtain the first reconstructed audio data output by the audio decoder.

[0103] Correspondingly, the step of inputting the reconstructed timbre information and the target prosody information corresponding to the second speaker into the audio decoder to obtain fourth reconstructed audio data output by the audio decoder, and inputting the target timbre information and the reconstructed prosody information corresponding to the first speaker into the audio decoder to obtain fifth reconstructed audio data output by the audio decoder includes: inputting the reconstructed timbre information and the target prosody information corresponding to the third audio data of the second speaker, and the target content information corresponding to the third audio data of the second speaker into the audio decoder to obtain fourth reconstructed audio data output by the audio decoder, and inputting the target timbre information and the reconstructed prosody information corresponding to the second audio data of the first speaker, and the target content information corresponding to the third audio data of the second speaker into the audio decoder to obtain fifth reconstructed audio data output by the audio decoder.

[0104] In the speech decoupling method provided by an embodiment of the present invention, a timbre encoder, a prosody encoder and a content encoder are obtained by training based on a first reconstruction loss, a second reconstruction loss, a third reconstruction loss and a fourth reconstruction loss. Through the above method, the first reconstruction loss, the second reconstruction loss, the third reconstruction loss and the fourth reconstruction loss can constrain the content encoder's ability to retain the content of the audio data, thereby improving the adequacy of content decoupling.

[0105] To facilitate understanding of the above embodiments, a specific embodiment is described here. First, obtain the first audio data A1 of the first speaker and the second audio data A2 of the first speaker, as well as the third audio data B1 of the second speaker and the fourth audio data B2 of the second speaker, and the first text data corresponding to the first audio data A1. The third text data corresponding to the third audio data B1 Secondly, the second audio data A2 and the fourth audio data B2 are respectively input to the timbre encoder, and the target timbre information corresponding to the second audio data A2 output by the timbre encoder is obtained. The target timbre information corresponding to the fourth audio data B2 The first audio data A1 and the third audio data B1 are respectively input into the prosody encoder, and the target prosody information corresponding to the first audio data A1 output by the prosody encoder is obtained. Target prosody information corresponding to the third audio data B1 , and the first text data corresponding to the first audio data A1 The third text data corresponding to the third audio data B1 The first audio data A1 is respectively input into the content encoder to obtain the target content information corresponding to the first audio data A1 output by the content encoder. Target content information corresponding to the third audio data B1 ; Then, the target timbre information corresponding to the second audio data A2 of the first speaker Target prosody information corresponding to the third audio data B1 of the second speaker , and the target content information corresponding to the third audio data B1 of the second speaker The first reconstructed audio data G1 is input to the audio decoder to obtain the first reconstructed audio data G1 output by the audio decoder; then, the first reconstructed audio data G1 is input to the timbre encoder to obtain the reconstructed timbre information output by the timbre encoder. , and input the first reconstructed audio data G1 into the prosody encoder to obtain the reconstructed prosody information output by the prosody encoder ; After that, the timbre information will be reconstructed Target prosody information corresponding to the first audio data A1 of the first speaker , and the target content information corresponding to the first audio data A1 The audio decoder is input to obtain the second reconstructed audio data G2 output by the audio decoder, and the target timbre information corresponding to the fourth audio data B2 of the second speaker is converted into the target timbre information and reconstruct prosodic information The third reconstructed audio data G3 is input to the audio decoder, and the reconstructed timbre information is obtained. Target prosody information corresponding to the third audio data B1 of the second speaker , and the target content information corresponding to the third audio data B1 of the second speaker The audio decoder is input to obtain the fourth reconstructed audio data G4 output by the audio decoder, and the target timbre information corresponding to the second audio data A2 of the first speaker is converted into the target timbre information G4. and reconstruct prosodic information , and the target content information corresponding to the third audio data B1 of the second speaker Input to the audio decoder to obtain the fifth reconstructed audio data G5 output by the audio decoder.

[0106] After determining the first reconstruction loss, the second reconstruction loss, the third reconstruction loss and the fourth reconstruction loss based on the above, a total reconstruction loss is determined based on the first reconstruction loss, the second reconstruction loss, the third reconstruction loss and the fourth reconstruction loss, and the acoustic model is trained based on the total reconstruction loss.

[0107] Exemplarily, the total reconstruction loss is as follows: ; In the formula, represents the total reconstruction loss, represents the first reconstruction loss, represents the second reconstruction loss, represents the third reconstruction loss, represents the fourth reconstruction loss.

[0108] Among them, the reconstruction losses are as follows: ; ; ; ; In the formula, first audio data representing a first speaker, represents the second reconstructed audio data, Represents the reconstruction of timbre information. represents the target prosody information corresponding to the first audio data A1, represents the target content information corresponding to the first audio data A1, Represents the audio decoder, third audio data representing a second speaker, represents the third reconstructed audio data, represents the target timbre information corresponding to the fourth audio data B2, represents the reconstructed rhythmic information, represents the target content information corresponding to the third audio data B1, the first reconstructed audio data , represents the target timbre information corresponding to the second audio data A2, represents the target prosody information corresponding to the third audio data B1, represents the fourth reconstructed audio data, represents the fifth reconstructed audio data.

[0109] Among them, the first reconstruction loss Constraining the first reconstructed audio data G1 to maintain the timbre of the first speaker A shows the stability of the model in extracting the timbre of different voices of the same person; the second reconstruction loss The ability to preserve the rhythm of the third audio data B1 of the second speaker B in the first reconstructed audio data G1 is constrained, showing the stability of the model in prosody extraction of the same prosodic audio of different speakers; the third reconstruction loss It also constrains the timbre preservation capability of the first reconstructed audio data G1 for the first speaker A, and Compared with the original audio data, the reconstruction target is changed from real audio data to reconstructed audio data, and the gradient flow transmission path is shorter, which helps to improve the stability of training; the fourth reconstruction loss It is also a constraint on the ability to preserve the prosody of the third audio data B1 of the second speaker B in the first reconstructed audio data G1, and In comparison, the reconstruction target is changed from real audio data to reconstructed audio data, and the transmission path of the gradient flow is shorter, which helps to improve the stability of training.

[0110] Based on any of the foregoing embodiments, in the method, training the acoustic model based on the first reconstruction loss, the second reconstruction loss, the third reconstruction loss, and the fourth reconstruction loss includes: Inputting the first reconstructed audio data, the second reconstructed audio data, the third reconstructed audio data, the fourth reconstructed audio data and the fifth reconstructed audio data into a discriminator respectively, and obtaining five discriminant losses output by the discriminator; The acoustic model is trained based on the first reconstruction loss, the second reconstruction loss, the third reconstruction loss, the fourth reconstruction loss and the five discriminant losses.

[0111] Here, the discriminator is used to discriminate the audio reconstruction accuracy of the audio decoder, so as to train the acoustic model based on five discriminant losses, improve the robustness of each network layer in the acoustic model, that is, improve the decoupling adequacy of the timbre encoder, improve the decoupling adequacy of the prosody encoder, and improve the audio reconstruction accuracy of the audio decoder, and further improve the training effect of the acoustic model while improving the audio reconstruction accuracy.

[0112] The discriminator can be pre-trained, that is, the discriminator is updated using its optimized parameters, and then the audio decoder (generator) is optimized based on the discriminant loss (adversarial loss). In other words, the training of the acoustic model includes not only the reconstruction loss but also the discriminant loss (adversarial loss).

[0113] It should be understood that since the audio decoder is reconstructed five times in one training round, five discriminant losses can be obtained based on this, thereby improving the training effect while ensuring the training efficiency.

[0114] The speech decoupling method provided in the embodiment of the present invention, through the above-mentioned method, is trained not only based on the reconstruction loss, but also based on the discrimination loss, thereby improving the decoupling adequacy of the timbre encoder, improving the decoupling adequacy of the prosody encoder, and improving the audio reconstruction accuracy of the audio decoder.

[0115] Based on any of the above embodiments, in the method, in the above step 120, the sample audio data of the first speaker and the sample audio data of the second speaker are respectively input into the prosody encoder in the acoustic model to obtain the target prosody information respectively output by the prosody encoder, including: Performing timbre change processing on the sample audio data of the first speaker and the sample audio data of the second speaker respectively, to obtain the sample audio data of the first speaker after the timbre change and the sample audio data of the second speaker after the timbre change; The sample audio data of the first speaker after the timbre change and the sample audio data of the second speaker after the timbre change are respectively input into the prosody encoder in the acoustic model to obtain the target prosody information respectively output by the prosody encoder.

[0116] In one embodiment, the timbre change processing method is random pitch conversion to obtain the audio data after the pitch is changed.

[0117] Correspondingly, in the above step 230, the first reconstructed audio data is input into the prosody encoder to obtain the reconstructed prosody information output by the prosody encoder, including: performing timbre change processing on the first reconstructed audio data to obtain the first reconstructed audio data after the timbre change; inputting the first reconstructed audio data after the timbre change into the prosody encoder to obtain the reconstructed prosody information output by the prosody encoder.

[0118] It should be noted that the timbre of the input audio data needs to be changed only during the training of the prosody encoder. This is because the training has learned to focus only on the extraction of prosody information, thereby reducing the leakage of timbre information and content information.

[0119] Furthermore, the audio data are all represented by Mel-spectrogram data, and the lower 20 dimensions of the sample audio data of the first speaker after the timbre change and the lower 20 dimensions of the sample audio data of the second speaker after the timbre change are respectively input into the prosody encoder to obtain the target prosody information output by the prosody encoder. Based on this, considering that the lower 20 dimensions of the Mel-spectrogram data retain the complete fundamental frequency and some harmonics, it can represent the prosody features and play a role in information compression, thereby improving the decoupling efficiency of the prosody encoder.

[0120] The speech decoupling method provided by the embodiment of the present invention performs timbre change processing on sample audio data of a first speaker and sample audio data of a second speaker respectively during the training process of a prosody encoder to obtain sample audio data of the first speaker after the timbre change and sample audio data of the second speaker after the timbre change; the sample audio data of the first speaker after the timbre change and the sample audio data of the second speaker after the timbre change are respectively input into the prosody encoder to obtain target prosody information respectively output by the prosody encoder, thereby ensuring that the prosody encoder focuses only on the extraction of prosody information as much as possible, thereby reducing the leakage of timbre information and content information.

[0121] Based on any of the above embodiments, in this method, the audio data is represented by the Mel spectrum data, and when the audio data needs to be input to the prosody encoder, the lower 20 dimensions of the input audio data are input to the prosody encoder. Based on this, considering that the lower 20 dimensions of the Mel spectrum data retain the complete fundamental frequency and some harmonics, it can represent the prosody features and play a role in information compression, thereby improving the decoupling efficiency of the prosody encoder.

[0122] Based on any of the above embodiments, in the method, the prosody encoder includes a first residual module, a downsampling module, a second residual module and a vector quantization module which are connected in sequence.

[0123] In one embodiment, the first residual module and the second residual module each include a plurality of residual convolution modules with jump connections, such as a ResBlock module or a ConvNext module. Furthermore, each convolution layer in the residual convolution module is composed of a plurality of convolution units, and the parameters of each convolution unit are optimized by a back propagation algorithm. The purpose of the convolution operation is to extract different features of the input. The first convolution layer may only be able to extract some low-level features such as edges, lines, and corners. A network with more layers can iteratively extract more complex features from low-level features.

[0124] In one embodiment, the output of the first residual module is input to the first downsampling module, which includes several convolutional layers and pooling layers, which are used to further compress the information of the audio data in the time dimension. The output of the downsampling module is then processed by the second residual module for further processing of hidden features.

[0125] Here, the output of the second residual module is processed by the vector quantization module to quantize the prosody features into a number of discrete vectors, further compressing the information and reducing the difficulty of prosody prediction. At this time, the quantized vectors can be used as prosody information.

[0126] The speech decoupling method provided in the embodiment of the present invention can further improve the decoupling adequacy and accuracy of the prosody encoder through the structure of the above-mentioned prosody encoder.

[0127] Based on the above embodiments, the acoustic model trained by the above training method has the ability of speech decoupling, wherein the timbre encoder can extract the speaker's timbre information in addition to the prosody and content from any audio, the prosody encoder can extract the prosody information in addition to the timbre and content from any audio, and the content encoder can extract the content information from the text. The three types of decoupled information can be combined arbitrarily to complete downstream speech synthesis tasks, such as voice cloning, timbre migration, prosody migration, etc.

[0128] The speech decoupling device provided by the present invention is described below. The speech decoupling device described below and the speech decoupling method described above can be referred to each other.

[0129] Figure 5 is a schematic diagram of the structure of the speech decoupling device provided by the present invention, such as Figure 5 As shown, the speech decoupling device includes a speech decoupling module 510.

[0130] The speech decoupling module 510 is used to input the speech data to be decoupled into the timbre encoder and the prosody encoder respectively, to obtain the decoupled timbre information output by the timbre encoder and the decoupled prosody information output by the prosody encoder.

[0131] Wherein, the timbre encoder and the prosody encoder are trained based on a first reconstruction loss and a second reconstruction loss; the first reconstruction loss is determined based on sample audio data of a first speaker and reconstructed audio data of the first speaker, the reconstructed audio data of the first speaker is reconstructed based on target timbre information corresponding to the first speaker and target prosody information corresponding to the first speaker, the target timbre information corresponding to the first speaker is obtained by extracting the sample audio data of the first speaker by the timbre encoder, and the target prosody information corresponding to the first speaker is obtained by extracting the sample audio data of the first speaker by the prosody encoder; the second reconstruction loss is determined based on sample audio data of a second speaker and reconstructed audio data of the second speaker, the reconstructed audio data of the second speaker is reconstructed based on target timbre information corresponding to the second speaker and target prosody information corresponding to the second speaker, the target timbre information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker by the timbre encoder, and the target prosody information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker by the prosody encoder; the first speaker and the second speaker are different speakers.

[0132] Based on any of the above embodiments, the device further includes a training module, and the training module includes: A first input unit is used to input the sample audio data of the first speaker and the sample audio data of the second speaker into a timbre encoder in the acoustic model, respectively, to obtain target timbre information respectively output by the timbre encoder, and to input the sample audio data of the first speaker and the sample audio data of the second speaker into a prosody encoder in the acoustic model, respectively, to obtain target prosody information respectively output by the prosody encoder; A second input unit is used to input the target timbre information corresponding to the first speaker and the target prosody information corresponding to the second speaker into the audio decoder in the acoustic model to obtain first reconstructed audio data output by the audio decoder; the audio decoder is used to reconstruct audio data based on the input information; a third input unit, configured to input the first reconstructed audio data into the timbre encoder to obtain reconstructed timbre information output by the timbre encoder, and to input the first reconstructed audio data into the prosody encoder to obtain reconstructed prosody information output by the prosody encoder; a fourth input unit, configured to input the reconstructed timbre information and the target prosody information corresponding to the first speaker into the audio decoder to obtain second reconstructed audio data output by the audio decoder, and input the target timbre information and the reconstructed prosody information corresponding to the second speaker into the audio decoder to obtain third reconstructed audio data output by the audio decoder; a loss determining unit, configured to determine a first reconstruction loss based on the sample audio data of the first speaker and the second reconstructed audio data, and to determine a second reconstruction loss based on the sample audio data of the second speaker and the third reconstructed audio data; A model training unit is used to train the acoustic model based on the first reconstruction loss and the second reconstruction loss.

[0133] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the speech decoupling method, which includes: inputting the speech data to be decoupled into the timbre encoder and the prosody encoder respectively, obtaining the decoupled timbre information output by the timbre encoder and the decoupled prosody information output by the prosody encoder; wherein the timbre encoder and the prosody encoder are trained based on a first reconstruction loss and a second reconstruction loss; the first reconstruction loss is determined based on the sample audio data of the first speaker and the reconstructed audio data of the first speaker, the reconstructed audio data of the first speaker is reconstructed based on the target timbre information corresponding to the first speaker and the target prosody information corresponding to the first speaker, and the target timbre information corresponding to the first speaker is based on the timbre encoder for the sample audio of the first speaker The target prosody information corresponding to the first speaker is obtained by extracting the sample audio data of the first speaker based on the prosody encoder; the second reconstruction loss is determined based on the sample audio data of the second speaker and the reconstructed audio data of the second speaker, the reconstructed audio data of the second speaker is reconstructed based on the target timbre information corresponding to the second speaker and the target prosody information corresponding to the second speaker, the target timbre information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker based on the timbre encoder, and the target prosody information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker based on the prosody encoder; the first speaker and the second speaker are different speakers.

[0134] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0135] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the speech decoupling method provided by the above methods, which includes: inputting the speech data to be decoupled into a timbre encoder and a prosody encoder respectively, and obtaining the decoupled timbre information output by the timbre encoder, and the decoupled prosody information output by the prosody encoder; wherein the timbre encoder and the prosody encoder are trained based on a first reconstruction loss and a second reconstruction loss; the first reconstruction loss is determined based on the sample audio data of a first speaker and the reconstructed audio data of the first speaker, and the reconstructed audio data of the first speaker is reconstructed based on the target timbre information corresponding to the first speaker and the target prosody information corresponding to the first speaker, and the first speaker The target timbre information corresponding to the speaker is obtained by extracting the sample audio data of the first speaker by the timbre encoder, and the target prosody information corresponding to the first speaker is obtained by extracting the sample audio data of the first speaker by the prosody encoder; the second reconstruction loss is determined based on the sample audio data of the second speaker and the reconstructed audio data of the second speaker, the reconstructed audio data of the second speaker is reconstructed based on the target timbre information corresponding to the second speaker and the target prosody information corresponding to the second speaker, the target timbre information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker by the timbre encoder, and the target prosody information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker by the prosody encoder; the first speaker and the second speaker are different speakers.

[0136] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the speech decoupling method provided by the above-mentioned methods, the method comprising: inputting the speech data to be decoupled into a timbre encoder and a prosody encoder respectively, obtaining decoupled timbre information output by the timbre encoder, and decoupled prosody information output by the prosody encoder; wherein the timbre encoder and the prosody encoder are trained based on a first reconstruction loss and a second reconstruction loss; the first reconstruction loss is determined based on sample audio data of a first speaker and reconstructed audio data of the first speaker, the reconstructed audio data of the first speaker is reconstructed based on target timbre information corresponding to the first speaker and target prosody information corresponding to the first speaker, and the target timbre information corresponding to the first speaker is based on the The target prosody information corresponding to the first speaker is obtained by extracting the sample audio data of the first speaker by the timbre encoder, and the target prosody information corresponding to the first speaker is obtained based on the sample audio data of the first speaker extracted by the prosody encoder; the second reconstruction loss is determined based on the sample audio data of the second speaker and the reconstructed audio data of the second speaker, the reconstructed audio data of the second speaker is reconstructed based on the target timbre information corresponding to the second speaker and the target prosody information corresponding to the second speaker, the target timbre information corresponding to the second speaker is obtained based on the sample audio data of the second speaker extracted by the timbre encoder, and the target prosody information corresponding to the second speaker is obtained based on the sample audio data of the second speaker extracted by the prosody encoder; the first speaker and the second speaker are different speakers.

[0137] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0138] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech decoupling method, characterized in that: include: Inputting the speech data to be decoupled into a timbre encoder and a prosody encoder respectively, to obtain decoupled timbre information output by the timbre encoder and decoupled prosody information output by the prosody encoder; Wherein, the timbre encoder and the prosody encoder are obtained by training based on a first reconstruction loss and a second reconstruction loss; The first reconstruction loss is determined based on sample audio data of a first speaker and reconstructed audio data of the first speaker, the reconstructed audio data of the first speaker is reconstructed based on target timbre information corresponding to the first speaker and target prosody information corresponding to the first speaker, the target timbre information corresponding to the first speaker is obtained by extracting the sample audio data of the first speaker by the timbre encoder, and the target prosody information corresponding to the first speaker is obtained by extracting the sample audio data of the first speaker by the prosody encoder; the second reconstruction loss is determined based on sample audio data of a second speaker and reconstructed audio data of the second speaker, the reconstructed audio data of the second speaker is reconstructed based on target timbre information corresponding to the second speaker and target prosody information corresponding to the second speaker, the target timbre information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker by the timbre encoder, and the target prosody information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker by the prosody encoder; The first speaker and the second speaker are different speakers.

2. The speech decoupling method according to claim 1, characterized in that: The timbre encoder and the prosody encoder are trained based on the following method: Inputting the sample audio data of the first speaker and the sample audio data of the second speaker into the timbre encoder in the acoustic model respectively, obtaining the target timbre information respectively output by the timbre encoder, and inputting the sample audio data of the first speaker and the sample audio data of the second speaker into the prosody encoder in the acoustic model respectively, obtaining the target prosody information respectively output by the prosody encoder; Inputting the target timbre information corresponding to the first speaker and the target prosody information corresponding to the second speaker into the audio decoder in the acoustic model to obtain first reconstructed audio data output by the audio decoder; the audio decoder is used to reconstruct audio data based on the input information; Inputting the first reconstructed audio data into the timbre encoder to obtain reconstructed timbre information output by the timbre encoder, and inputting the first reconstructed audio data into the prosody encoder to obtain reconstructed prosody information output by the prosody encoder; Inputting the reconstructed timbre information and the target prosody information corresponding to the first speaker into the audio decoder to obtain second reconstructed audio data output by the audio decoder, and inputting the target timbre information and the reconstructed prosody information corresponding to the second speaker into the audio decoder to obtain third reconstructed audio data output by the audio decoder; determining a first reconstruction loss based on the sample audio data of the first speaker and the second reconstructed audio data, and determining a second reconstruction loss based on the sample audio data of the second speaker and the third reconstructed audio data; The acoustic model is trained based on the first reconstruction loss and the second reconstruction loss.

3. The speech decoupling method according to claim 2, characterized in that: The sample audio data of the first speaker includes different first audio data and second audio data, and the sample audio data of the second speaker includes different third audio data and fourth audio data; The method comprises: inputting the sample audio data of the first speaker and the sample audio data of the second speaker into a timbre encoder in the acoustic model to obtain target timbre information respectively output by the timbre encoder, and inputting the sample audio data of the first speaker and the sample audio data of the second speaker into a prosody encoder in the acoustic model to obtain target prosody information respectively output by the prosody encoder, including: Inputting the second audio data and the fourth audio data into the timbre encoder respectively to obtain target timbre information respectively output by the timbre encoder, and inputting the first audio data and the third audio data into the prosody encoder respectively to obtain target prosody information respectively output by the prosody encoder; The step of inputting the target timbre information corresponding to the first speaker and the target prosody information corresponding to the second speaker into the audio decoder in the acoustic model to obtain first reconstructed audio data output by the audio decoder includes: Inputting the target timbre information corresponding to the second audio data of the first speaker and the target prosody information corresponding to the third audio data of the second speaker into the audio decoder to obtain first reconstructed audio data output by the audio decoder; The step of inputting the reconstructed timbre information and the target prosody information corresponding to the first speaker into the audio decoder to obtain second reconstructed audio data output by the audio decoder, and inputting the target timbre information and the reconstructed prosody information corresponding to the second speaker into the audio decoder to obtain third reconstructed audio data output by the audio decoder includes: Inputting the reconstructed timbre information and the target prosody information corresponding to the first audio data of the first speaker into the audio decoder to obtain second reconstructed audio data output by the audio decoder, and inputting the target timbre information and the reconstructed prosody information corresponding to the fourth audio data of the second speaker into the audio decoder to obtain third reconstructed audio data output by the audio decoder; The determining of a first reconstruction loss based on the sample audio data of the first speaker and the second reconstructed audio data, and determining of a second reconstruction loss based on the sample audio data of the second speaker and the third reconstructed audio data, comprises: A first reconstruction loss is determined based on the first audio data of the first speaker and the second reconstructed audio data, and a second reconstruction loss is determined based on the third audio data of the second speaker and the third reconstructed audio data.

4. The speech decoupling method according to claim 3, characterized in that: The step of inputting the second audio data and the fourth audio data into the timbre encoder to obtain target timbre information output by the timbre encoder, and inputting the first audio data and the third audio data into the prosody encoder to obtain target prosody information output by the prosody encoder, respectively, comprises: Inputting the second audio data and the fourth audio data into the timbre encoder respectively to obtain target timbre information respectively output by the timbre encoder, inputting the first audio data and the third audio data into the prosody encoder respectively to obtain target prosody information respectively output by the prosody encoder, and inputting the first text data corresponding to the first audio data and the third text data corresponding to the third audio data into the content encoder in the acoustic model respectively to obtain target content information respectively output by the content encoder; The step of inputting the target timbre information corresponding to the second audio data of the first speaker and the target prosody information corresponding to the third audio data of the second speaker into the audio decoder to obtain first reconstructed audio data output by the audio decoder includes: Inputting the target timbre information corresponding to the second audio data of the first speaker, the target prosody information corresponding to the third audio data of the second speaker, and the target content information corresponding to the third audio data of the second speaker into the audio decoder, and obtaining first reconstructed audio data output by the audio decoder; The step of inputting the reconstructed timbre information and the target prosody information corresponding to the first audio data of the first speaker into the audio decoder to obtain second reconstructed audio data output by the audio decoder, and inputting the target timbre information and the reconstructed prosody information corresponding to the fourth audio data of the second speaker into the audio decoder to obtain third reconstructed audio data output by the audio decoder comprises: The reconstructed timbre information and the target prosody information corresponding to the first audio data of the first speaker, and the target content information corresponding to the first audio data of the first speaker are input into the audio decoder to obtain second reconstructed audio data output by the audio decoder, and the target timbre information and the reconstructed prosody information corresponding to the fourth audio data of the second speaker, and the target content information corresponding to the third audio data of the second speaker are input into the audio decoder to obtain third reconstructed audio data output by the audio decoder.

5. The speech decoupling method according to claim 2, characterized in that: The step of training the acoustic model based on the first reconstruction loss and the second reconstruction loss comprises: Inputting the reconstructed timbre information and the target prosody information corresponding to the second speaker into the audio decoder to obtain fourth reconstructed audio data output by the audio decoder, and inputting the target timbre information and the reconstructed prosody information corresponding to the first speaker into the audio decoder to obtain fifth reconstructed audio data output by the audio decoder; determining a third reconstruction loss based on the first reconstructed audio data and the fourth reconstructed audio data, and determining a fourth reconstruction loss based on the first reconstructed audio data and the fifth reconstructed audio data; The acoustic model is trained based on the first reconstruction loss, the second reconstruction loss, the third reconstruction loss, and the fourth reconstruction loss.

6. The speech decoupling method according to claim 5, characterized in that: The sample audio data of the first speaker includes different first audio data and second audio data, and the sample audio data of the second speaker includes different third audio data and fourth audio data; The method comprises: inputting the sample audio data of the first speaker and the sample audio data of the second speaker into a timbre encoder in the acoustic model to obtain target timbre information respectively output by the timbre encoder, and inputting the sample audio data of the first speaker and the sample audio data of the second speaker into a prosody encoder in the acoustic model to obtain target prosody information respectively output by the prosody encoder, including: Inputting the second audio data and the fourth audio data into the timbre encoder respectively to obtain target timbre information respectively output by the timbre encoder, inputting the first audio data and the third audio data into the prosody encoder respectively to obtain target prosody information respectively output by the prosody encoder, and inputting the first text data corresponding to the first audio data and the third text data corresponding to the third audio data into the content encoder in the acoustic model respectively to obtain target content information respectively output by the content encoder; The step of inputting the target timbre information corresponding to the first speaker and the target prosody information corresponding to the second speaker into the audio decoder in the acoustic model to obtain first reconstructed audio data output by the audio decoder includes: Inputting the target timbre information corresponding to the second audio data of the first speaker, the target prosody information corresponding to the third audio data of the second speaker, and the target content information corresponding to the third audio data of the second speaker into the audio decoder, and obtaining first reconstructed audio data output by the audio decoder; The step of inputting the reconstructed timbre information and the target prosody information corresponding to the second speaker into the audio decoder to obtain fourth reconstructed audio data output by the audio decoder, and inputting the target timbre information and the reconstructed prosody information corresponding to the first speaker into the audio decoder to obtain fifth reconstructed audio data output by the audio decoder includes: The reconstructed timbre information and the target prosody information corresponding to the third audio data of the second speaker, and the target content information corresponding to the third audio data of the second speaker are input into the audio decoder to obtain fourth reconstructed audio data output by the audio decoder, and the target timbre information and the reconstructed prosody information corresponding to the second audio data of the first speaker, and the target content information corresponding to the third audio data of the second speaker are input into the audio decoder to obtain fifth reconstructed audio data output by the audio decoder.

7. The speech decoupling method according to claim 5, characterized in that: The step of training the acoustic model based on the first reconstruction loss, the second reconstruction loss, the third reconstruction loss, and the fourth reconstruction loss comprises: Inputting the first reconstructed audio data, the second reconstructed audio data, the third reconstructed audio data and the fourth reconstructed audio data into a discriminator respectively, and obtaining four discriminant losses output by the discriminator; The acoustic model is trained based on the first reconstruction loss, the second reconstruction loss, the third reconstruction loss, the fourth reconstruction loss and the four discriminant losses.

8. The speech decoupling method according to claim 2, characterized in that: The step of inputting the sample audio data of the first speaker and the sample audio data of the second speaker into the prosody encoder in the acoustic model to obtain the target prosody information outputted by the prosody encoder respectively comprises: Performing timbre change processing on the sample audio data of the first speaker and the sample audio data of the second speaker respectively, to obtain the sample audio data of the first speaker after the timbre change and the sample audio data of the second speaker after the timbre change; The sample audio data of the first speaker after the timbre change and the sample audio data of the second speaker after the timbre change are respectively input into the prosody encoder in the acoustic model to obtain the target prosody information respectively output by the prosody encoder.

9. A speech decoupling device, characterized in that: include: A speech decoupling module, used for inputting the speech data to be decoupled into the timbre encoder and the prosody encoder respectively, to obtain the decoupled timbre information output by the timbre encoder and the decoupled prosody information output by the prosody encoder; Wherein, the timbre encoder and the prosody encoder are obtained by training based on a first reconstruction loss and a second reconstruction loss; The first reconstruction loss is determined based on sample audio data of a first speaker and reconstructed audio data of the first speaker, the reconstructed audio data of the first speaker is reconstructed based on target timbre information corresponding to the first speaker and target prosody information corresponding to the first speaker, the target timbre information corresponding to the first speaker is obtained by extracting the sample audio data of the first speaker by the timbre encoder, and the target prosody information corresponding to the first speaker is obtained by extracting the sample audio data of the first speaker by the prosody encoder; the second reconstruction loss is determined based on sample audio data of a second speaker and reconstructed audio data of the second speaker, the reconstructed audio data of the second speaker is reconstructed based on target timbre information corresponding to the second speaker and target prosody information corresponding to the second speaker, the target timbre information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker by the timbre encoder, and the target prosody information corresponding to the second speaker is obtained by extracting the sample audio data of the second speaker by the prosody encoder; The first speaker and the second speaker are different speakers.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the speech decoupling method according to any one of claims 1 to 8 is implemented.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech decoupling method as claimed in any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the speech decoupling method as claimed in any one of claims 1 to 8 is implemented.

Images