Speech conversion model training and speech conversion method, device and related equipment

Abstract

The present invention discloses a training method for a speech conversion model, which is applied to the field of artificial intelligence. The speech conversion model provided by the present invention includes a first neural network, a third neural network, and a fourth neural network. The method provided by the present invention includes: calculating a first loss of sample speech data after vectorization and after passing through a gradient flipping layer; calculating a second loss of the vectorized sample speech data and the corresponding vectorized text content data; splicing the vectorized sample speech data and the vectorized text content data into a speech latent vector, and converting the speech latent vector into target speech data; calculating a third loss between the target speech data and the sample speech data; adjusting the parameters of the neural network in the speech conversion model according to the first loss, the second loss, and the third loss; and repeating the above steps until the first loss is maximized and the second loss and the third loss are minimized, thereby obtaining a trained speech conversion model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a speech conversion model training and speech conversion method, device and related equipment. Background Art

[0002] Speech conversion decouples the timbre and content of the original speaker's speech and converts it to the target speaker's timbre while preserving the same content. Applications include dubbing for film and television dramas and converting timbre to automatically match different story characters in e-book reading. However, the converted speech produced by traditional technologies still contains the original speaker's timbre information, while the content of the converted speech does not closely match the text of the original speaker's speech. Summary of the Invention

[0003] The embodiments of the present invention provide a speech conversion model training and speech conversion method, apparatus, computer equipment and storage medium to solve the problems of incomplete removal of the original speaker's timbre in the converted speech and low matching degree of text content.

[0004] A method for training a speech conversion model, the speech conversion model comprising a first neural network, a third neural network, and a fourth neural network, the method comprising:

[0005] Inputting the speaker's speech data into the first neural network to obtain a first latent vector;

[0006] Using a gradient flip layer to remove the speaker's timbre information from the first latent vector, and calculating a first loss before and after flipping;

[0007] Inputting text content data corresponding to the speaker's speech data into the second neural network to obtain a second latent vector, and calculating a second loss between the first latent vector and the second latent vector;

[0008] Inputting the speaker's speech data into the third neural network to obtain a third latent vector;

[0009] Inputting the second latent vector and the third latent vector into the fourth neural network to obtain a fourth latent vector, and calculating a third loss between the third latent vector and the fourth latent vector;

[0010] Adjusting parameters of the first neural network according to the first loss and the second loss, and adjusting parameters of the fourth neural network according to the third loss;

[0011] The steps of inputting the speaker's speech data into the first neural network to adjusting the parameters of the first neural network according to the first loss and the second loss are looped until the first loss reaches a maximum and the second loss and the third loss both reach a minimum, thereby obtaining a trained speech conversion model.

[0012] A method for performing speech conversion based on the speech conversion model trained by the above method, the method comprising:

[0013] Inputting the to-be-converted speech data of the original speaker into the first neural network of the speech conversion model to obtain a latent vector of the target speech content;

[0014] Inputting the target speech data of the target speaker into the third neural network in the speech conversion model to obtain the target speech timbre latent vector;

[0015] Concatenating the target speech content latent vector and the target speech timbre latent vector to obtain a concatenated speech latent vector;

[0016] The concatenated speech latent vector is input into a fourth neural network in the speech conversion model to obtain converted speech data.

[0017] A training device for a speech conversion model, comprising:

[0018] A first data acquisition module, configured to input the speaker's speech data into the first neural network to obtain a first latent vector;

[0019] A first loss module, configured to remove the speaker's timbre information from the first latent vector using a gradient flipping layer, and calculate a first loss before and after flipping;

[0020] a second loss module, configured to input text content data corresponding to the speaker's speech data into the second neural network to obtain a second latent vector, and calculate a second loss between the first latent vector and the second latent vector;

[0021] A second data acquisition module is used to input the speaker's speech data into the third neural network to obtain a third latent vector;

[0022] a third loss module, configured to input the second latent vector and the third latent vector into the fourth neural network to obtain a fourth latent vector, and calculate a third loss between the third latent vector and the fourth latent vector;

[0023] a parameter adjustment module, configured to adjust parameters of the first neural network according to the first loss and the second loss, and adjust parameters of the fourth neural network according to the third loss;

[0024] A training module is configured to loop the steps of inputting the speaker's speech data into the first neural network and adjusting the parameters of the first neural network according to the first loss and the second loss, until the first loss reaches a maximum and the second loss and the third loss both reach a minimum, thereby obtaining a trained speech conversion model.

[0025] A device for performing speech conversion based on the speech conversion model provided by the speech conversion model training device, the device comprising:

[0026] A speech content module, configured to input the speech data to be converted from the original speaker into the first neural network in the speech conversion model to obtain a target speech content latent vector;

[0027] A speech timbre module, configured to input the target speech data of the target speaker into the third neural network in the speech conversion model to obtain a target speech timbre latent vector;

[0028] A speech concatenation module, configured to concatenate the target speech content latent vector and the target speech timbre latent vector to obtain a concatenated speech latent vector;

[0029] The speech conversion module is used to input the concatenated speech latent vector into the fourth neural network in the speech conversion model to obtain converted speech data.

[0030] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the training method of the speech conversion model are implemented.

[0031] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the training method for the above-mentioned speech conversion model.

[0032] The aforementioned speech conversion model training and speech conversion method, apparatus, computer device, and storage medium utilize a gradient inversion layer to invert a speaker's speech data to remove the speaker's timbre information. This allows the trained speech conversion model to remove timbre information from the speech data to the greatest extent possible when processing it. Furthermore, the training is assisted by utilizing the textual content corresponding to the speaker's speech data. This allows the trained speech conversion model to retain the textual content of the speech data to the greatest extent possible when processing it. This allows the trained speech conversion model to completely remove the original speaker's timbre information while fully preserving the textual content of the original speaker's speech data during use. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0034] Figure 1 1 is a schematic diagram of an application environment for training a speech conversion model and a speech conversion method according to an embodiment of the present invention;

[0035] Figure 2 is a flow chart of a method for training a speech conversion model in one embodiment of the present invention;

[0036] Figure 3 1 is a schematic diagram of the architecture of a pre-trained speech conversion model according to one embodiment of the present invention;

[0037] Figure 4 is a flow chart of a voice conversion method according to an embodiment of the present invention;

[0038] Figure 5 Schematic diagram of the architecture of a trained speech conversion model in one embodiment of the present invention;

[0039] Figure 6 1 is a schematic structural diagram of a training device for a speech conversion model according to an embodiment of the present invention;

[0040] Figure 7 1 is a schematic structural diagram of a device for voice conversion according to an embodiment of the present invention;

[0041] Figure 8 is a schematic diagram of a type of computer device in one embodiment of the present invention;

[0042] Figure 9 FIG. 2 is a schematic diagram of another type of computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0044] The training of the speech conversion model and the speech conversion method provided in this application can be applied in Figure 1In an application environment, the computer device can communicate with an external device, such as a server, via a network. The computer device may include, but is not limited to, various personal computers, laptops, smartphones, tablet computers, and portable wearable devices. The server may be implemented as a standalone server or a server cluster consisting of multiple servers.

[0045] In one embodiment, if Figure 2 As shown, a training method for a speech conversion model is provided, wherein the speech conversion model includes a first neural network, a third neural network and a fourth neural network. Figure 1 Taking the computer device in FIG. 1 as an example, the method includes the following steps S101 to S107:

[0046] S101: Input the speaker's speech data into the first neural network to obtain a first latent vector.

[0047] Furthermore, the first neural network includes a first convolutional neural network with a first preset number of layers and a first recurrent neural network with a second preset number of layers. Figure 3 As shown, the first neural network includes a first convolutional neural network with three preset convolutional layers and a first recurrent neural network with two preset LSTM (Long Short Term Memory) neural networks. It should be noted that the first recurrent neural network can also use a GRU (Gate Recurrent Unit) neural network.

[0048] Specifically, the speaker's speech data is first processed to convert it into corresponding spectral data. The spectral data may be mel-spectrogram data. The first convolutional neural network is then used to extract spectral features from the first spectral data to obtain a first latent vector. Finally, the first recurrent neural network is used to learn temporal relationships within the first spectral data and add these temporal relationships to the first latent vector.

[0049] S102: Use a gradient flip layer to remove the speaker's timbre information in the first latent vector, and calculate the first loss before and after flipping.

[0050] Furthermore, the gradient flipping layer further includes a classifier configured to classify the gradient-flipped first latent vector, wherein the classification result includes a first probability that the flipped first latent vector still contains the timbre information. Furthermore, parameters of the gradient flipping layer can be optimized based on the first probability to improve the effectiveness of the gradient flipping process in removing the speaker's timbre information.

[0051] Specifically, the classifier is first used to classify the flipped first latent vector to obtain a classification result, which includes a first probability that the flipped first latent vector still contains the timbre information. Next, a determination is made as to whether the first probability is within a preset probability range. If not, the parameters of the gradient flipping layer are optimized based on the first probability. Finally, the steps of classifying the flipped first latent vector using the classifier and determining whether the first probability is within the preset probability range are repeated until the first probability is within the preset probability range.

[0052] S103: Input the text content data corresponding to the speaker's speech data into a second neural network to obtain a second latent vector, and calculate a second loss between the first latent vector and the second latent vector.

[0053] In a specific embodiment, Figure 3 As shown, the second neural network is a first one-dimensional convolutional neural network with a pre-set layer.

[0054] Specifically, the text content data corresponding to the speaker's speech data is first encoded using a pre-set encoder to obtain a first text encoding latent vector representing the text content encoding features. Position encoding information is then added to the text encoding latent vector for each character in the text content data, and importance information is added to each character in the text content data using a multi-head attention mechanism, resulting in a second text encoding latent vector containing both position encoding information and importance information. A pre-set second neural network is then used to extract the spatial features of each character from the text content data and add the spatial features to the second text encoding latent vector. Finally, a pre-set duration predictor is used to expand the second text encoding latent vector so that it aligns with the first latent vector, ultimately obtaining the second latent vector. The duration predictor not only maintains the total duration in the second text encoding latent vector consistent with the total duration in the first latent vector, but also predicts the duration of each character in the text content data within the total duration and adds the duration of each character to the second text encoding latent vector.

[0055] The second loss can be calculated using a preset MAE (Mean Absolute Error) loss function or an MSE (Mean Squared Error) loss function.

[0056] S104: Input the speaker's speech data into the third neural network to obtain a third latent vector.

[0057] Among them, before inputting the speaker's voice data into the third neural network preset in the voice conversion model, the speaker's voice data still needs to be converted into corresponding spectrum data, and the specific implementation method will not be repeated here.

[0058] Furthermore, after inputting the speaker's speech data into a third neural network preset in the speech conversion model to obtain a third latent vector, the method further includes: inputting the third latent vector into a third preset number of linear layers, adapting the size of the third latent vector to the size of the second latent vector, so that the third latent vector and the second latent vector can be spliced.

[0059] In a specific embodiment, Figure 3 As shown, the third neural network includes a pre-set two-layer GRU neural network and a one-layer linear neural network. The two-layer GRU neural network is used to learn the time domain features of the speaker's speech data. The linear layer is used to adjust the size of the third latent vector so that the size of the third latent vector is consistent with the size of the second latent vector.

[0060] S105. Input the second latent vector and the third latent vector into the fourth neural network to obtain a fourth latent vector, and calculate a third loss between the third latent vector and the fourth latent vector.

[0061] In a specific embodiment, Figure 3 As shown, the fourth neural network includes a pre-set second one-dimensional convolutional neural network.

[0062] Furthermore, before inputting the second latent vector and the third latent vector into the fourth neural network preset in the speech conversion model, the second latent vector and the third latent vector are spliced. In the splicing process, the second latent vector representing the content features of the speaker's speech data is used to replace the content feature part of the third latent vector to obtain a spliced ​​latent vector. The position encoding information of each character in the speaker's speech data is added to the spliced ​​latent vector, and the multi-head attention mechanism is used to learn the importance information of each character in the speaker's speech data. The importance information of each character is added to the spliced ​​latent vector. Finally, the spatial features of the speaker's speech data are extracted by the second one-dimensional convolutional neural network and added to the spliced ​​latent vector to obtain the fourth latent vector.

[0063] S106. Adjust parameters of the first neural network according to the first loss and the second loss, and adjust parameters of the fourth neural network according to the third loss.

[0064] The first loss reflects the degree to which the first neural network retains the speaker's voice timbre information when processing the speaker's voice data. That is, during the voice conversion process, the first neural network needs to remove the speaker's timbre information from the speaker's voice data to the greatest extent possible. The second loss reflects the degree to which the first neural network retains the speaker's voice content information when processing the speaker's voice data. That is, during the voice conversion process, the first neural network needs to retain the speaker's voice content information in the speaker's voice data to the greatest extent possible. The third loss reflects the degree of loss caused by the fourth neural network's processing of the concatenated latent vector. That is, during the voice conversion process, the fourth neural network needs to retain the speaker's voice content information and the speaker's timbre information in the concatenated latent vector to the greatest extent possible.

[0065] S107. Loop the steps of inputting the speaker's speech data into the first neural network to adjusting the parameters of the first neural network according to the first loss and the second loss, until the first loss reaches a maximum and the second loss and the third loss both reach a minimum, thereby obtaining a trained speech conversion model.

[0066] It should be noted that in the initial stage of the training method of the speech conversion model, because the first latent vector contains the speaker's timbre information and speech content information, and the second latent vector only contains the speaker's speech content information, the second loss of calculating the first latent vector and the second latent vector will be abnormally large. However, as the number of iterations of the training method increases, the second loss will gradually decrease to a minimum.

[0067] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0068] The speech conversion model training method proposed in this embodiment uses a gradient flipping layer to flip the speaker's speech data to remove the speaker's timbre information. This allows the trained speech conversion model to remove timbre information from the speech data to the greatest extent possible when processing it. Furthermore, the training is supplemented by utilizing the text content information corresponding to the speaker's speech data. This allows the trained speech conversion model to retain the text content information in the speech data to the greatest extent possible when processing it. This ultimately allows the trained speech conversion model to completely remove the original speaker's timbre information while fully preserving the text content information in the original speaker's speech data during use.

[0069] Figure 4 is a flow chart of a method for performing voice conversion using a voice conversion model trained according to the above-mentioned voice conversion model training method in one embodiment of the present invention. According to another embodiment of the present invention, a method for performing voice conversion using a voice conversion model trained according to the above-mentioned voice conversion model training method is proposed, such as Figure 4 As shown, the method includes the following steps S201 to S204.

[0070] S201, inputting the to-be-converted speech data of the original speaker into the first neural network of the speech conversion model to obtain a latent vector of the target speech content;

[0071] S202, inputting the target speech data of the target speaker into the third neural network in the speech conversion model to obtain a target speech timbre latent vector;

[0072] S203, concatenating the target speech content latent vector and the target speech timbre latent vector to obtain a concatenated speech latent vector;

[0073] S204: Input the concatenated speech latent vector into a fourth neural network in the speech conversion model to obtain converted speech data.

[0074] Specifically, in a specific embodiment, Figure 5 As shown, the speech data to be converted is input into the first neural network of the speech conversion model for processing, the target speech data is input into the third neural network of the speech conversion model for processing, and finally the converted speech data model is output through the fourth neural network of the speech conversion model.

[0075] It should be noted that during the training of the speech conversion model, the third neural network receives the same speech data as the first neural network, that is, both receive the speech data of the original speaker. However, when the speech conversion model is in use, the third neural network receives the speech data of the target speaker, while the first neural network still receives the speech data of the original speaker. This is because during the training process, the third loss is used to optimize the parameters of the fourth neural network to ensure that the fourth neural network can effectively synthesize the text content data and timbre feature data into the final target speech data.

[0076] In one embodiment, a training device 100 for a speech conversion model is provided. The speech conversion model includes a first neural network, a third neural network, and a fourth neural network. The training device 100 for the speech conversion model corresponds to the training method for the speech conversion model in the above embodiment. Figure 6As shown, the training device 100 for the speech conversion model includes a first data acquisition module 11, a first loss module 12, a second loss module 13, a second data acquisition module 14, a third loss module 15, a parameter adjustment module 16, and a training module 17. The functional modules are described in detail as follows:

[0077] A first data acquisition module 11 is used to input the speaker's speech data into the first neural network to obtain a first latent vector;

[0078] A first loss module 12 is configured to remove the speaker's timbre information from the first latent vector using a gradient flipping layer, and calculate a first loss before and after flipping;

[0079] a second loss module 13, configured to input text content data corresponding to the speaker's speech data into the second neural network to obtain a second latent vector, and calculate a second loss between the first latent vector and the second latent vector;

[0080] A second data acquisition module 14 is configured to input the speaker's speech data into the third neural network to obtain a third latent vector;

[0081] a third loss module 15, configured to input the second latent vector and the third latent vector into the fourth neural network to obtain a fourth latent vector, and calculate a third loss between the third latent vector and the fourth latent vector;

[0082] a parameter adjustment module 16, configured to adjust parameters of the first neural network according to the first loss and the second loss, and adjust parameters of the fourth neural network according to the third loss;

[0083] The training module 17 is configured to loop the steps of inputting the speaker's speech data into the first neural network and adjusting the parameters of the first neural network according to the first loss and the second loss, until the first loss reaches a maximum and the second loss and the third loss both reach a minimum, thereby obtaining a trained speech conversion model.

[0084] Furthermore, the first data acquisition module 11 further includes:

[0085] A spectrum data acquisition submodule, configured to convert the speaker's speech data to obtain corresponding first spectrum data;

[0086] a spectral feature extraction submodule, configured to extract spectral features of the first spectral data using the first convolutional neural network to obtain a first latent vector;

[0087] A temporal relationship learning submodule is used to learn the temporal relationship in the first spectrum data through the first recurrent neural network and add the temporal relationship to the first latent vector.

[0088] Furthermore, the first loss module 12 further includes:

[0089] a classifier submodule, configured to classify the flipped first latent vector using the classifier to obtain a classification result, wherein the classification result includes a first probability that the flipped first latent vector still contains the timbre information;

[0090] a first probability submodule, configured to determine whether the first probability is within a preset probability range, and if not, optimize the parameters of the gradient flip layer according to the first probability;

[0091] The flipping optimization submodule is configured to loop the steps of classifying the flipped first latent vector using the classifier and determining whether the first probability reaches a preset probability range until the first probability is within the preset probability range.

[0092] Furthermore, the second loss module 13 further includes:

[0093] a text content vector acquisition submodule, configured to encode the text content data corresponding to the speaker's speech data using a preset encoder to obtain a first text encoding latent vector;

[0094] a text content vector processing submodule, configured to add a position code to each character of the text content data in the text encoding latent vector, and to add importance information to each character using a multi-head attention mechanism, to obtain a second text encoding latent vector;

[0095] a spatial feature processing submodule, configured to extract spatial features of each character of the text content data through a preset second neural network, and add the spatial features to the second text encoding latent vector;

[0096] A vector expansion submodule is configured to expand the second text encoding latent vector to align with the first latent vector using a preset duration predictor to obtain the second latent vector.

[0097] Furthermore, the second data acquisition module 14 further includes:

[0098] A vector size adaptation submodule is configured to input the third latent vector into a third preset number of linear layers, and adapt the size of the third latent vector to the size of the second latent vector so that the third latent vector and the second latent vector can be concatenated.

[0099] Furthermore, the third loss module 15 further includes:

[0100] a vector concatenation submodule, configured to concatenate the second latent vector and the third latent vector, wherein the concatenation process replaces the content feature portion of the third latent vector with the second latent vector representing the content feature of the speaker's speech data to obtain a concatenated latent vector;

[0101] A splicing position coding submodule, configured to add position coding information of each character in the speaker's speech data to the spliced ​​latent vector;

[0102] a splicing importance submodule, configured to learn the importance information of each character in the speaker's speech data using a multi-head attention mechanism, and add the importance information of each character to the spliced ​​latent vector;

[0103] The splicing spatial feature submodule is used to extract the spatial features of the speaker's speech data through the second one-dimensional convolutional neural network and add them to the spliced ​​latent vector to obtain the fourth latent vector.

[0104] The specific definition of the speech conversion model training device can be found in the definition of the speech conversion model training method above and will not be repeated here. The various modules in the speech conversion model training device described above can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above modules can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0105] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the training method of the speech conversion model. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a training method of a speech conversion model is implemented.

[0106] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 9As shown. The computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements a method for training a speech conversion model.

[0107] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method for training the speech conversion model in the above embodiment are implemented, such as Figure 2 Alternatively, when the processor executes the computer program, the functions of each module / unit of the training device for the speech conversion model in the above embodiment are realized, for example, Figure 6 The functions of modules 11 to 17 are shown in FIG. 3 and will not be described in detail here to avoid repetition.

[0108] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for training the speech conversion model in the above embodiment are implemented, such as Figure 2 Alternatively, when the computer program is executed by a processor, the functions of each module / unit of the training device for the speech conversion model in the above embodiment are realized, for example, Figure 6 The functions of modules 11 to 17 are shown in FIG. 3 and will not be described in detail here to avoid repetition.

[0109] Figure 7 FIG. 2 is a structural diagram of a speech conversion device 200 according to an embodiment of the present invention. Figure 7 As shown, the device 200 for performing speech conversion based on the speech conversion model provided by the speech conversion model training device 100 includes a speech content module 21, a speech timbre module 22, a speech concatenation module 23, and a speech conversion module 24. The functional modules are described in detail as follows:

[0110] The speech content module 21 is used to input the speech data to be converted by the original speaker into the first neural network in the speech conversion model to obtain the target speech content latent vector;

[0111] A speech timbre module 22 is configured to input the target speech data of the target speaker into the third neural network in the speech conversion model to obtain a target speech timbre latent vector;

[0112] A speech concatenation module 23 is configured to concatenate the target speech content latent vector and the target speech timbre latent vector to obtain a concatenated speech latent vector;

[0113] The speech conversion module 24 is configured to input the concatenated speech latent vector into the fourth neural network in the speech conversion model to obtain converted speech data.

[0114] The meaning of "first" and "second" in the above modules / units is only to distinguish different modules / units, and is not used to limit which module / unit has a higher priority or other limiting meanings. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The division of modules in this application is only a logical division, and other division methods may be used in actual applications.

[0115] For the specific definition of the voice conversion device, please refer to the definition of the voice conversion method above and will not be repeated here. The various modules in the above-mentioned voice conversion device can be implemented in whole or in part by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software so that the processor can call and execute the corresponding operations of the above-mentioned modules.

[0116] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface, and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the speech conversion method. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a speech conversion method is implemented.

[0117] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown. The computer device includes a processor, memory, a network interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements a speech conversion method.

[0118] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the speech conversion method in the above embodiment are implemented, such as Figure 4 Alternatively, when the processor executes the computer program, the functions of each module / unit of the speech conversion device in the above embodiment are realized, for example, Figure 7 The functions of modules 21 to 24 are shown in FIG. 3 and will not be described in detail here to avoid repetition.

[0119] The processor may be a central processing unit (CPU), other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer device, connecting various parts of the entire computer device using various interfaces and lines.

[0120] The memory may be used to store the computer programs and / or modules. The processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory and calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store an operating system, at least one application required for a function (such as a sound playback function, an image playback function, etc.); and the data storage area may store data created based on the use of the mobile phone (such as audio data, video data, etc.).

[0121] The memory may be integrated into the processor or may be provided separately from the processor.

[0122] In one embodiment, a computer-readable storage medium is provided on which a computer program is stored. When the computer program is executed by a processor, the steps of the speech conversion method in the above embodiment are implemented, such as Figure 4 Alternatively, when the computer program is executed by the processor, the functions of each module / unit of the speech conversion device in the above embodiment are realized, such as Figure 7 The functions of modules 21 to 24 are shown in FIG. 3 and will not be described in detail here to avoid repetition.

[0123] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0124] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0125] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A method for training a speech conversion model, characterized in that: The speech conversion model includes a first neural network, a third neural network, and a fourth neural network, and the method includes: Inputting the speaker's speech data into the first neural network to obtain a first latent vector; Using a gradient flip layer to remove the speaker's timbre information from the first latent vector, and calculating a first loss before and after flipping; Inputting text content data corresponding to the speaker's speech data into a preset second neural network to obtain a second latent vector, and calculating a second loss between the first latent vector and the second latent vector; Inputting the speaker's speech data into the third neural network to obtain a third latent vector; Inputting the second latent vector and the third latent vector into the fourth neural network to obtain a fourth latent vector, and calculating a third loss between the third latent vector and the fourth latent vector; Adjusting parameters of the first neural network according to the first loss and the second loss, and adjusting parameters of the fourth neural network according to the third loss; The steps of inputting the speaker's speech data into the first neural network to adjusting the parameters of the first neural network according to the first loss and the second loss are looped until the first loss reaches a maximum and the second loss and the third loss both reach a minimum, thereby obtaining a trained speech conversion model.

2. The method for training a speech conversion model according to claim 1, wherein: The first neural network includes a first convolutional neural network with a first preset number of layers and a first recurrent neural network with a second preset number of layers, and inputting the speaker's speech data into the first neural network to obtain the first latent vector includes: Converting the speaker's speech data to obtain corresponding first spectrum data; extracting spectral features of the first spectral data through the first convolutional neural network to obtain a first latent vector; The first recurrent neural network learns the temporal relationship in the first spectrum data, and adds the temporal relationship to the first latent vector.

3. The method for training a speech conversion model according to claim 1, wherein: The gradient flip layer further includes a classifier, and after removing the speaker's timbre information in the first latent vector using the gradient flip layer, the method further includes: classifying the flipped first latent vector using the classifier to obtain a classification result, wherein the classification result includes a first probability that the flipped first latent vector still contains the timbre information; Determining whether the first probability reaches a preset probability range, and if not, optimizing the parameters of the gradient flip layer according to the first probability; The steps of using the classifier to classify the flipped first latent vector and determining whether the first probability reaches a preset probability range are looped until the first probability is within the preset probability range.

4. The method for training a speech conversion model according to claim 1, wherein: Inputting the text content data corresponding to the speaker's voice data into a preset second neural network to obtain a second latent vector includes: Encoding the text content data corresponding to the speaker's speech data by a preset encoder to obtain a first text encoding latent vector; Adding a position code to each character of the text content data in the text encoding latent vector, and adding importance information to each character using a multi-head attention mechanism to obtain a second text encoding latent vector; extracting spatial features of each character of the text content data through the second neural network, and adding the spatial features to the second text encoding latent vector; The second text encoding latent vector is extended to be aligned with the first latent vector through a preset duration predictor to obtain the second latent vector.

5. The method for training a speech conversion model according to claim 1, wherein: After inputting the speaker's speech data into the third neural network to obtain the third latent vector, the method further includes: The third latent vector is input into a third preset number of linear layers, and the size of the third latent vector is adapted to the size of the second latent vector so that the third latent vector and the second latent vector can be concatenated.

6. A method for performing voice conversion based on a voice conversion model obtained by the method according to any one of claims 1 to 5, characterized in that: The method comprises: Inputting the to-be-converted speech data of the original speaker into the first neural network of the speech conversion model to obtain a latent vector of the target speech content; Inputting the target speech data of the target speaker into the third neural network in the speech conversion model to obtain the target speech timbre latent vector; Concatenating the target speech content latent vector and the target speech timbre latent vector to obtain a concatenated speech latent vector; The concatenated speech latent vector is input into a fourth neural network in the speech conversion model to obtain converted speech data.

7. A training device for a speech conversion model, characterized in that: The speech conversion model includes a first neural network, a third neural network and a fourth neural network, and the device includes: A first data acquisition module, configured to input the speaker's speech data into the first neural network to obtain a first latent vector; A first loss module, configured to remove the speaker's timbre information from the first latent vector using a gradient flipping layer, and calculate a first loss before and after flipping; a second loss module, configured to input text content data corresponding to the speaker's speech data into a preset second neural network to obtain a second latent vector, and calculate a second loss between the first latent vector and the second latent vector; A second data acquisition module is used to input the speaker's speech data into the third neural network to obtain a third latent vector; a third loss module, configured to input the second latent vector and the third latent vector into the fourth neural network to obtain a fourth latent vector, and calculate a third loss between the third latent vector and the fourth latent vector; a parameter adjustment module, configured to adjust parameters of the first neural network according to the first loss and the second loss, and adjust parameters of the fourth neural network according to the third loss; A training module is configured to loop the steps of inputting the speaker's speech data into the first neural network and adjusting the parameters of the first neural network according to the first loss and the second loss, until the first loss reaches a maximum and the second loss and the third loss both reach a minimum, thereby obtaining a trained speech conversion model.

8. The device for performing voice conversion according to the voice conversion model provided in claim 7, characterized in that: include: A speech content module, configured to input the speech data to be converted from the original speaker into the first neural network in the speech conversion model to obtain a target speech content latent vector; A speech timbre module, configured to input the target speech data of the target speaker into the third neural network in the speech conversion model to obtain a target speech timbre latent vector; A speech concatenation module, configured to concatenate the target speech content latent vector and the target speech timbre latent vector to obtain a concatenated speech latent vector; The speech conversion module is used to input the concatenated speech latent vector into the fourth neural network in the speech conversion model to obtain converted speech data.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images