Training of a voice conversion model and voice conversion method and device and related equipment

Abstract

The application discloses a speech conversion model training method applied to the field of artificial intelligence. The speech conversion model provided by the application comprises an encoder, an instantiation normalization layer, a voiceprint extractor and a decoder. The method provided by the application comprises the following steps: inputting a speech sample of an original speaker into the encoder to obtain first speech coding data; the instantiation normalization layer removes the speech attribute of the original speaker in the first speech coding data to obtain a first speech hidden vector; a first voiceprint vector of a target speaker is obtained; the decoder synthesizes the first speech hidden vector and the first voiceprint vector to obtain reconstructed speech data; a first loss of the reconstructed speech data and the speech sample of the original speaker is calculated; whether the first loss reaches a maximum is judged; if not, the parameters of the instantiation normalization layer are optimized, and the foregoing steps are circularly performed until the first loss reaches the maximum, so that a trained speech conversion model is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a speech conversion model training method and device, a speech conversion method and device, and related equipment. BACKGROUND

[0002] Speech conversion is a process of replacing the speech attribute information of the original speaker in the original speaker's speech data with the speech attribute information of the target speaker without changing the content information of the speech, the speech attribute information including the pitch, intensity, duration and timbre of the sound, so that the converted speech of the original speaker is most similar to the speech of the target speaker in terms of hearing, rather than the original speaker, while the content of the converted speech remains consistent with the content of the original speaker's speech. However, the original speaker's speech data processed by the traditional technology still contains some of the original speaker's speech attribute information. SUMMARY

[0003] The present application provides a speech conversion model training method and device, a computer device and a storage medium to solve the problem of incomplete removal of the original speaker's speech attribute in the speech conversion process of the traditional technology.

[0004] A speech conversion model training method, the speech conversion model including an encoder, an instantiation normalization layer, a voiceprint extractor and a decoder, the method comprising:

[0005] processing the original speaker's speech sample to be trained by the encoder to obtain first speech encoding data;

[0006] removing the original speaker's speech attribute in the first speech encoding data by the instantiation normalization layer to obtain a first speech hidden vector;

[0007] obtaining a first voiceprint vector containing the voiceprint information of the target speaker from the speech of the target speaker by the voiceprint extractor;

[0008] synthesizing the first speech hidden vector and the first voiceprint vector by the decoder to obtain reconstructed speech data;

[0009] calculating the loss of the reconstructed speech data and the original speaker's speech sample by a preset first loss function to obtain a first loss;

[0010] determining whether the first loss reaches a maximum, if not, optimizing the parameters of the instantiation normalization layer according to the first loss, and repeating the steps of extracting the original speaker's speech sample to the step of determining whether the first loss reaches a maximum until the first loss reaches a maximum, to obtain a trained speech conversion model.

[0011] A voice conversion method, comprising:

[0012] inputting encoded voice data of an original speaker into an instantiation normalization layer of a voice conversion model trained according to the training method of the voice conversion model to obtain a second voice latent vector;

[0013] inputting encoded voice of a target speaker into the voice conversion model to obtain a second voiceprint vector containing voiceprint information of the target speaker;

[0014] synthesizing the second voice latent vector and the second voiceprint vector by a decoder in the voice conversion model to obtain target reconstructed voice data.

[0015] A training device of a voice conversion model, comprising:

[0016] a first voice processing module configured to extract a voice sample of an original speaker and input the extracted voice sample into the encoder to obtain first voice encoding data;

[0017] a normalization module configured to remove voice attributes of the original speaker in the first voice encoding data by the instantiation normalization layer to obtain a first voice latent vector;

[0018] a first voiceprint acquisition module configured to acquire a first voiceprint vector containing voiceprint information of a target speaker from voice of the target speaker by the voiceprint extractor;

[0019] a reconstructed voice data module configured to synthesize the first voice latent vector and the first voiceprint vector by the decoder to obtain reconstructed voice data;

[0020] a loss calculation module configured to calculate a loss of the reconstructed voice data and the voice sample of the original speaker by a preset first loss function to obtain a first loss;

[0021] a training module configured to determine whether the first loss reaches a maximum, and if not, optimize parameters of the instantiation normalization layer according to the first loss, and repeat the steps from extracting the voice sample of the original speaker to determining whether the first loss reaches the maximum until the first loss reaches the maximum to obtain a trained voice conversion model.

[0022] A device for voice conversion, comprising:

[0023] a second voice processing module configured to input encoded voice data of an original speaker into an instantiation normalization layer of a voice conversion model provided by the training device of the voice conversion model to obtain a second voice latent vector;

[0024] a second voiceprint acquisition module, configured to input the encoded voice of the target speaker into the voice conversion model to acquire a second voiceprint vector containing voiceprint information of the target speaker;

[0025] a voice reconstruction module, configured to synthesize the second voice hidden vector and the second voiceprint vector by the decoder in the voice conversion model to obtain target reconstructed voice data.

[0026] A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the voice conversion model training method and / or the method for voice conversion according to the voice conversion model.

[0027] A computer readable storage medium, which stores a computer program, wherein the computer program is executable by a processor to implement the steps of the voice conversion model training method and / or the method for voice conversion according to the voice conversion model.

[0028] The voice conversion model training method and / or the method for voice conversion according to the voice conversion model, the device, the computer device, and the storage medium, by instantiating the normalization method to process the voice data of the original speaker, separate the voice attribute of the original speaker from the voice data of the original speaker, decouple the voice attribute and the voice content in the voice data of the original speaker, then extract the voiceprint vector of the target speaker from the voice data of the target speaker, and finally synthesize the voiceprint vector and the voice content part of the voice data of the original speaker to obtain the converted reconstructed voice data. The voice attribute of the original speaker in the voice data of the original speaker is removed more thoroughly by the instantiating normalization method and the classifier. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0030] Figure 1 is an application environment diagram of the voice conversion model training method and the voice conversion method in an embodiment of the present application;

[0031] Figure 2 is a flowchart of the voice conversion model training method in an embodiment of the present application;

[0032] Figure 3 is an architecture diagram of the voice conversion model before training in an embodiment of the present application;

[0033] Figure 4 is a flowchart of a voice conversion method in an embodiment of the present application;

[0034] Figure 5 is a structural schematic diagram of a training device of a voice conversion model in an embodiment of the present application;

[0035] Figure 6 is a structural schematic diagram of a voice conversion device in an embodiment of the present application;

[0036] Figure 7 is a schematic diagram of a computer device in an embodiment of the present application;

[0037] Figure 8 is a schematic diagram of another computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0039] The training of the voice conversion model and the voice conversion method provided in the present application can be applied in an application environment as shown in Figure 1 , wherein the computer device can communicate with an external device, such as a server, through a network. The computer device can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers.

[0040] In an embodiment, as shown in Figure 2 , a training method of a voice conversion model is provided, the voice conversion model comprising an encoder, an instantiation normalization layer, a voiceprint extractor and a decoder. The method is described by taking a server in Figure 1 as an example, comprising the following steps S101 to S106.

[0041] S101, extracting a voice sample of an original speaker, inputting the extracted voice sample into the encoder to obtain first voice coding data.

[0042] Specifically, suitable speech data is first extracted from the collected original speaker's speech data as a speech sample, and the screening of the speech sample is based on a pre-set screening rule. The attribute requirements of the speech data in the screening rule include but are not limited to: market, sampling rate, sampling bit number, frame rate, bit rate and channel number. Then, the extracted speech sample of the original speaker will be sent to the pre-set encoder in the speech conversion model for encoding processing to obtain first speech encoding data. It needs to be specially pointed out that the encoder is a part built in the speech conversion model, but in the actual training process and use process, different types of encoders can be replaced according to the actual situation to adapt to various variable demands.

[0043] S102, remove the speech attribute of the original speaker in the first speech encoding data through the instantiation normalization layer to obtain a first speech hidden vector.

[0044] Specifically, the instantiation normalization layer includes a convolutional neural network with a preset first number of layers. After receiving the first speech encoding data, the instantiation normalization layer first calculates the mean and variance of each channel contained in the first speech data, and then inputs the data of each channel contained in the first speech data, the mean of each channel and the method of each channel as a group of speech data to the convolutional neural network with a preset first number of layers in turn to obtain a first speech hidden vector. In a specific embodiment, the convolutional neural network with a preset first number of layers is a one-dimensional convolutional neural network with a preset three layers. The speech attribute includes pitch, intensity, duration and timbre of sound.

[0045] Further, as shown in Figure 3 The instantiation normalization layer also includes a classifier. Before synthesizing the first speech hidden vector and the first voiceprint vector through the decoder, the classifier is used to perform classification operation on the first speech hidden vector. Specifically, the first speech hidden vector is first input to the classifier for classification to obtain a classification result, which includes a first probability that the first speech hidden vector still contains the speech attribute of the original speaker. Then, it is judged whether the first probability is within a preset probability range. If not, the parameters of the instantiation normalization layer are optimized according to the first probability. Finally, the steps from removing the speech attribute of the original speaker in the first speech encoding data through the instantiation normalization layer to judging whether the first probability is within the preset probability range are repeated until the first probability is within the preset probability range. Wherein, the optimization of the parameters of the instantiation normalization layer according to the first probability in the instantiation normalization layer can make the parameters of the instantiation normalization layer be optimized before the end of the single training process in the local cycle, further improving the training efficiency of the speech conversion model.

[0046] S103. Obtain a first voiceprint vector containing the voiceprint information of the target speaker from the speech of the target speaker through the voiceprint extractor.

[0047] Specifically, firstly, the voice data of the target speaker is acquired and split into frame data. Then, the frame data is sequentially input into a deep neural network with a preset second layer, which is included in the voiceprint extractor. Next, voice activation vectors are obtained from the last hidden layer of the deep neural network, and regularization is applied to these vectors to obtain voice frame vectors. Finally, the voice frame vectors are accumulated to obtain the first voiceprint vector. In one specific embodiment, the regularization process uses L2 regularization, but it can also be replaced with L1 regularization or other regularization methods depending on actual needs.

[0048] S104. The first speech latent vector and the first voiceprint vector are synthesized by the decoder to obtain reconstructed speech data.

[0049] The first latent speech vector contains the speech content portion of the reconstructed speech data, and the first voiceprint vector contains the speech attribute portion of the reconstructed speech data. The reconstructed speech data synthesized by the decoder has replaced the original speaker's speech attributes with the target speaker's speech attributes, but retains the speech content portion.

[0050] S105. Calculate the loss between the reconstructed speech data and the original speaker's speech sample using the first loss function to obtain the first loss.

[0051] Specifically, the preset first loss function is as follows:

[0052] L recon =||x-x1||1

[0053] Among them, L recon Let x represent the loss value, x represent the original speaker's speech sample, and x1 represent the reconstructed speech data.

[0054] S106. Determine whether the first loss has reached its maximum. If not, optimize the parameters of the instantiated normalization layer based on the first loss, and repeat the steps from extracting the original speaker's speech samples to determining whether the first loss has reached its maximum until the first loss reaches its maximum, thus obtaining the trained speech conversion model.

[0055] Among them, such as Figure 3As shown, the first loss reflects the loss between the reconstructed speech data and the original speaker's speech, and after training, the loss should be as large as possible, that is, the reconstructed speech data and the original speaker's speech are two completely different people in terms of hearing. According to the first loss, the parameters of the instantiated normalization layer are optimized, so that the instantiated normalization layer removes the original speaker's speech attributes in the original speaker's speech sample more thoroughly, thereby ensuring that after the decoder synthesizes the first speech hidden vector and the first voiceprint vector, the reconstructed speech data contains less original speaker's speech attribute information, that is, the reconstructed speech data contains more target speaker's speech attribute information, and the voice conversion effect is better.

[0056] Further, a second loss between the reconstructed speech and the target speaker's speech can also be calculated. According to the calculation method of the second loss, the second loss must be as small as possible after training, that is, the reconstructed speech and the target speaker's speech are the same person in terms of hearing. Specifically, the second loss between the reconstructed speech data and the target speaker's speech is calculated, and then it is determined whether the second loss reaches the minimum and the first loss reaches the maximum. If not, the parameters of the instantiated normalization layer are optimized according to the first loss and the second loss, and the steps from extracting the original speaker's speech sample to determining whether the second loss reaches the minimum and the first loss reaches the maximum are repeated until the first loss reaches the maximum and the second loss reaches the minimum, and a trained voice conversion model is obtained.

[0057] The training method of the voice conversion model proposed in this embodiment processes the original speaker's speech data through the instantiation normalization method, separates the original speaker's speech attributes from the original speaker's speech data, decouples the speech attributes and speech content in the original speaker's speech data, and uses a classifier to identify the decoupled speech content data, further strengthening the processing effect of the instantiation normalization method. Then, the voiceprint vector of the target speaker is extracted from the target speaker's speech data, and finally the voiceprint vector and the speech content part of the original speaker's speech data are synthesized to obtain the converted reconstructed speech data. The instantiation normalization method and the classifier remove the original speaker's speech attributes in the original speaker's speech data more thoroughly, and at the same time, multiple target reconstructed speech data containing different target speaker's speech attributes can be synthesized according to different target speaker's voiceprint vectors.

[0058] Figure 4is a flowchart of a method of voice conversion performed by a voice conversion model trained according to the voice conversion model training method described above, according to another embodiment of the present application, as shown in Figure 4 The method includes the following steps S201 to S203:

[0059] S201, input the encoded voice data of the original speaker into the instantiation normalization layer of the voice conversion model trained according to the voice conversion model training method described above, to obtain a second voice hidden vector;

[0060] S202, input the encoded voice of the target speaker into the voice conversion model to obtain a second voiceprint vector containing the voiceprint information of the target speaker;

[0061] S203, synthesize the second voice hidden vector and the second voiceprint vector by the decoder in the voice conversion model to obtain target reconstructed voice data.

[0062] It should be particularly noted that in the use process, the voice data of different target speakers can be selected, and after the input of the voice data of the original speaker is completed, the voice conversion process can be realized by selecting the voice data of different target speakers, that is, one or more reconstructed voice data of the voice attributes of different target speakers can be obtained.

[0063] Further, the voice conversion model in this embodiment can also be used to extract the voiceprint vector of the target speaker in advance, and the extracted voiceprint vector of the target speaker can be pre-stored in the voice conversion model. In the voice conversion process, only the voice data of the original speaker needs to be received, and the decoder in the voice conversion model directly synthesizes the pre-stored voiceprint vector of the target speaker and the voice data of the original speaker to obtain target reconstructed voice data, further improving the efficiency of voice conversion.

[0064] Further, in actual application, the voice data of the target speaker contained in the voice conversion model will also increase, that is, as the voice data of the target speaker increases, the voiceprint vector of the same target speaker can also be further trained and optimized, and the voice attributes of the same target speaker can be further strengthened. At the same time, new voice data of the target speaker or new voiceprint vector of the target speaker can be added to the voice conversion model to obtain more reconstructed voice data of the voice attributes of the target speaker.

[0065] In an embodiment, a training device 100 of a voice conversion model is provided, the voice conversion model comprising a vector quantization layer, the voice conversion model comprising an encoder, an instantiation normalization layer, a voiceprint extractor, and a decoder, the training device 100 of the voice conversion model corresponding to the training method of the voice conversion model in the above-mentioned embodiments. As shown in the following figure, the training device 100 of the voice conversion model comprises a first speech processing module 11, a normalization module 12, a first voiceprint acquisition module 13, a reconstructed speech data module 14, a loss calculation module 15, and a training module 16. The functions of each module are described in detail as follows: Figure 5

[0066] The first speech processing module 11 is configured to extract a speech sample of an original speaker, input the extracted speech sample into the encoder, and obtain first speech encoding data;

[0067] The normalization module 12 is configured to remove the speech attribute of the original speaker in the first speech encoding data through the instantiation normalization layer, and obtain a first speech hidden vector;

[0068] The first voiceprint acquisition module 13 is configured to acquire a first voiceprint vector containing voiceprint information of a target speaker from the speech of the target speaker through the voiceprint extractor;

[0069] The reconstructed speech data module 14 is configured to synthesize the first speech hidden vector and the first voiceprint vector through the decoder, and obtain reconstructed speech data;

[0070] The loss calculation module 15 is configured to calculate the loss of the reconstructed speech data and the speech sample of the original speaker through a preset first loss function, and obtain a first loss;

[0071] The training module 16 is configured to determine whether the first loss reaches a maximum, if not, optimize the parameters of the instantiation normalization layer according to the first loss, and repeat the steps from extracting the speech sample of the original speaker to determining whether the first loss reaches the maximum until the first loss reaches the maximum, and obtain a trained voice conversion model.

[0072] Further, the normalization module 12 further comprises:

[0073] A mean and variance calculation submodule is configured to calculate the mean and variance of each channel in the first speech encoding data;

[0074] A first speech hidden vector submodule is configured to take the data of each channel contained in the first speech encoding data, the mean of each channel, and the variance of each channel as a group of speech channel data, input the speech channel data into the preset first number of convolutional neural networks in sequence, and obtain the first speech hidden vector.​

[0075] a first probability sub-module, configured to input the first speech latent vector into the classifier to obtain a classification result, the classification result comprising a first probability that the first speech latent vector still contains the speech attribute of the original speaker;

[0076] a first optimization sub-module, configured to determine whether the first probability is within a preset probability range, and if not, optimize parameters of the instantiated normalization layer according to the first probability;

[0077] a classifier loop sub-module, configured to loop the steps from removing the speech attribute of the original speaker in the first speech encoded data by the instantiated normalization layer to determining whether the first probability is within the preset probability range until the first probability is within the preset probability range.

[0078] Further, the first voiceprint acquisition module 13 further comprises:

[0079] a frame data sub-module, configured to acquire voice data of the target speaker and split the voice data into frame data;

[0080] a deep neural network sub-module, configured to input the frame data into a preset second number of deep neural networks included in the voiceprint extractor in sequence;

[0081] a sound frame vector sub-module, configured to acquire a sound activation vector from a last hidden layer of the deep neural network, and perform regularization processing on the sound activation vector to obtain a sound frame vector;

[0082] a vector accumulation sub-module, configured to accumulate the sound frame vector to obtain the first voiceprint vector.

[0083] Further, the loss calculation module 15 further comprises:

[0084] a first loss function sub-module, configured to calculate the preset first loss according to the following formula:

[0085] L recon =||x-x1||1

[0086] wherein L recon represents a loss value, x represents the speech sample of the original speaker, and x1 represents the reconstructed speech data.

[0087] The specific limitations of the training device of the speech conversion model can refer to the limitations of the training method of the speech conversion model in the foregoing, which will not be described here. Each module in the training device of the speech conversion model can be realized by software, hardware, and a combination thereof, in whole or in part. Each module described above can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so that the processor invokes and executes the operations corresponding to each module.

[0088] In an embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in Figure 7 The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store data involved in the training method of the speech conversion model. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program, when executed by the processor, implements the training method of the speech conversion model.

[0089] In an embodiment, a computer device, which can be a terminal, is provided, and an internal structure diagram of the computer device can be as shown in Figure 8 The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with an external server through a network connection. The computer program, when executed by the processor, implements the training method of the speech conversion model.

[0090] In an embodiment, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor, when executing the computer program, implements the steps of the training method of the speech conversion model in the above embodiments, such as the steps S101 to S106 shown in Figure 2 and other extensions and related steps of the method. Alternatively, the processor, when executing the computer program, implements the functions of each module / unit of the training device of the speech conversion model in the above embodiments, such as the functions of the modules shown in Figure 5The functions of the modules 11-16 shown. To avoid repetition, they will not be described here again.

[0091] In an embodiment, a computer readable storage medium is provided, having stored thereon a computer program, which, when executed by a processor, implements the steps of the method for training a speech conversion model according to the above embodiment, for example Figure 2 The steps S101-S106 shown and other extensions and related steps of the method. Alternatively, the computer program, when executed by a processor, implements the functions of the modules / units of the apparatus for training a speech conversion model according to the above embodiment, for example Figure 5 The functions of the modules 11-16 shown. To avoid repetition, they will not be described here again.

[0092] Figure 6 is a structural schematic diagram of the speech conversion apparatus 200 in an embodiment of the application, as Figure 6 The apparatus 200 for speech conversion according to the speech conversion model provided by the training apparatus 100 comprises a second speech processing module 21, a second voiceprint acquisition module 22 and a speech reconstruction module 23. The functions of the modules are described as follows:

[0093] The second speech processing module 21 is configured to input the encoded speech data of the original speaker into the instantiated normalization layer of the speech conversion model provided by the training apparatus for speech conversion model, to obtain a second speech hidden vector;

[0094] The second voiceprint acquisition module 22 is configured to input the encoded speech of the target speaker into the speech conversion model, to obtain a second voiceprint vector containing the voiceprint information of the target speaker;

[0095] The speech reconstruction module 23 is configured to synthesize the second speech hidden vector and the second voiceprint vector by the decoder in the speech conversion model, to obtain target reconstructed speech data.

[0096] The "first" and "second" in the above modules / units are only used to distinguish different modules / units, and are not used to limit the priority of which module / unit is higher or other limiting meanings. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or modules does not have to be limited to only those steps or modules clearly listed, but can include other steps or modules not clearly listed or inherent to the process, method, product or device. The division of modules in this application is only a logical division, and in actual application, there can be another division mode.

[0097] For specific limitations regarding the speech conversion device, please refer to the limitations on the speech synthesis method above, which will not be repeated here. Each module in the aforementioned speech conversion device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the corresponding operations of each module.

[0098] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores data involved in the speech synthesis method. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a speech synthesis method.

[0099] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements a speech synthesis method.

[0100] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the speech synthesis method described in the above embodiments, for example... Figure 4 The steps S201 to S203 shown, as well as other extensions and related steps of the method, are examples. Alternatively, when the processor executes a computer program, it implements the functions of each module / unit of the speech conversion device in the above embodiments, for example... Figure 6 The functions of modules 21 to 23 are shown. To avoid repetition, they will not be described again here.

[0101] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The processor is a control center of the computer device, and connects various parts of the computer device through various interfaces and lines.

[0102] The memory can be used to store the computer program and / or modules, and the processor realizes various functions of the computer device by running or executing the computer program and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc. The data storage area can store data created according to the use of the mobile phone (such as audio data, video data, etc.), etc.

[0103] The memory can be integrated in the processor, or can be separately arranged from the processor.

[0104] In an embodiment, a computer readable storage medium is provided, and the computer program is stored on the computer readable storage medium. The computer program is executed by the processor to realize the steps of the voice conversion method in the above embodiments, for example Figure 4 the steps S201 to S203 and the extension of other extensions and related steps of the method. Alternatively, the computer program is executed by the processor to realize the functions of the modules / units of the voice conversion device in the above embodiments, for example Figure 6 the functions of the modules 21 to 23. To avoid repetition, details are not described here.

[0105] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0106] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0107] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not limit it. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features. The modification or replacement does not make the essence of the corresponding technical solution deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for training a speech conversion model, the method comprising: The voice conversion model comprises an encoder, an instantiation normalization layer, a voiceprint extractor and a decoder, and the method comprises the following steps: extracting a voice sample of an original speaker, inputting the extracted voice sample into the encoder to obtain first voice coding data; removing the voice attribute of the original speaker in the first voice coding data through the instantiation normalization layer to obtain a first voice hidden vector; obtaining a first voiceprint vector containing voiceprint information of a target speaker from the voice of the target speaker through the voiceprint extractor; synthesizing the first voice hidden vector and the first voiceprint vector through the decoder to obtain reconstructed voice data; calculating the loss of the reconstructed voice data and the voice sample of the original speaker through a preset first loss function to obtain a first loss; calculating a second loss of the reconstructed voice data and the voice of the target speaker; determining whether the second loss reaches a minimum and the first loss reaches a maximum, if not, optimizing the parameters of the instantiation normalization layer according to the first loss and the second loss, and repeating the steps from extracting a voice sample of an original speaker to determining whether the second loss reaches a minimum and the first loss reaches a maximum until the first loss reaches a maximum and the second loss reaches a minimum, to obtain a trained voice conversion model; wherein the instantiation normalization layer comprises a convolutional neural network with a preset first number of layers, and removing the voice attribute of the original speaker in the first voice coding data through the instantiation normalization layer to obtain a first voice hidden vector comprises: calculating the mean and variance of each channel in the first voice coding data; inputting the data of each channel contained in the first voice coding data, the mean of each channel and the variance of each channel as a group of voice channel data into the convolutional neural network with a preset first number of layers in sequence to obtain a first voice hidden vector; wherein the instantiation normalization layer further comprises a classifier, and before synthesizing the first voice hidden vector and the first voiceprint vector through the decoder, the method further comprises: inputting the first voice hidden vector into the classifier for classification to obtain a classification result, wherein the classification result comprises a first probability that the first voice hidden vector still contains the voice attribute of the original speaker; determining whether the first probability is within a preset probability range, if not, optimizing the parameters of the instantiation normalization layer according to the first probability; repeating the steps from removing the voice attribute of the original speaker in the first voice coding data through the instantiation normalization layer to determining whether the first probability is within a preset probability range until the first probability is within a preset probability range.

2. The method of claim 1, wherein, The method of obtaining a first voiceprint vector containing voiceprint information of a target speaker from the voice of the target speaker through a voiceprint extractor comprises: obtaining voice data of the target speaker and splitting the voice data into frame data; inputting the frame data into a deep neural network with a preset second number of layers comprised in the voiceprint extractor in sequence; obtaining a sound frame vector by obtaining a sound activation vector from a last hidden layer of the deep neural network, and performing regularization on the sound activation vector; obtaining the first voiceprint vector by accumulating the sound frame vector.

3. The method of claim 1, wherein, The preset first loss function is as follows: L recon = ||x - xi||1 wherein L recon represents a loss value, x represents the speech sample of the original speaker, and xi represents the reconstructed speech data.

4. A voice conversion method characterized by, The method comprises: inputting the encoded voice data of the original speaker into an instantiation normalization layer of an instance of the voice conversion model obtained by the method of any one of claims 1 to 3 to obtain a second voice hidden vector; inputting the encoded voice of the target speaker into the voice conversion model to obtain a second voiceprint vector containing voiceprint information of the target speaker; obtaining target reconstructed voice data by synthesizing the second voice hidden vector and the second voiceprint vector through the decoder in the voice conversion model. 5.A device for training a speech conversion model, characterized in that, The voice conversion model comprises an encoder, an instantiation normalization layer, a voiceprint extractor, and a decoder, and the device comprises: a first voice processing module configured to extract a voice sample of an original speaker, and input the extracted voice sample into the encoder to obtain first voice encoded data; a normalization module configured to remove voice attributes of the original speaker in the first voice encoded data through the instantiation normalization layer to obtain a first voice hidden vector; a first voiceprint acquisition module configured to acquire a first voiceprint vector containing voiceprint information of a target speaker from a voice of the target speaker through the voiceprint extractor; a reconstructed voice data module configured to synthesize the first voice hidden vector and the first voiceprint vector through the decoder to obtain reconstructed voice data; a loss calculation module configured to calculate a loss of the reconstructed voice data and the voice sample of the original speaker through a preset first loss function to obtain a first loss; a training module configured to calculate a second loss of the reconstructed voice data and a voice of the target speaker; determine whether the second loss reaches a minimum and whether the first loss reaches a maximum; if not, optimize parameters of the instantiation normalization layer according to the first loss and the second loss, and repeat the steps from extracting the voice sample of the original speaker to determining whether the second loss reaches the minimum and whether the first loss reaches the maximum until the first loss reaches the maximum and the second loss reaches the minimum to obtain a trained voice conversion model. The instantiation normalization layer includes a preset first number of layers of a convolutional neural network and a classifier. The normalization module further includes: a mean and variance calculation submodule for calculating the mean and variance of each channel in the first speech encoded data; a first speech hidden vector submodule for inputting the data of each channel included in the first speech encoded data, the mean of each channel, and the variance of each channel as a set of speech channel data, sequentially inputting the speech channel data in groups to the preset first number of layers of the convolutional neural network to obtain the first speech hidden vector; a first probability submodule for inputting the first speech hidden vector to the classifier for classification to obtain a classification result, the classification result including a first probability that the first speech hidden vector still includes the speech attribute of the original speaker; a first optimization submodule for determining whether the first probability is within a preset probability range, and if not, optimizing the parameters of the instantiation normalization layer according to the first probability; and a classifier loop submodule for looping the steps from removing the speech attribute of the original speaker in the first speech encoded data by the instantiation normalization layer to determining whether the first probability is within the preset probability range until the first probability is within the preset probability range.

6. An apparatus for voice conversion, the apparatus comprising: The second speech processing module is configured to input the encoded speech data of the original speaker into an instantiation normalization layer of a speech conversion model obtained by training the training device of the speech conversion model according to claim 5 to obtain a second speech hidden vector. The second voiceprint acquisition module is configured to input the encoded speech of the target speaker into the speech conversion model to obtain a second voiceprint vector including voiceprint information of the target speaker. The speech reconstruction module is configured to synthesize the second speech hidden vector and the second voiceprint vector by the decoder in the speech conversion model to obtain target reconstructed speech data. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 3 and / or claim 4.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 3 and / or claim 4.

8. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 7. ​

Images