Neural network-based speech conversion method, system, device, and storage medium

Abstract

The application belongs to the technical field of artificial intelligence, and provides a speech conversion method, system, device and storage medium based on a neural network, which comprises the following steps: obtaining source speech and target speaker timbre; inputting the source speech into a content encoder to extract semantic content of the source speech and obtain semantic content encoding; inputting the semantic content encoding and the target speaker timbre into a generator to obtain target speech, wherein the speech conversion model is trained by using sample speech and sample timbre. In the embodiment of the application, when training samples are made, only sample speech and sample timbre need to be collected, and the second person's timbre information can be separated from the second person's speech to obtain the target speech. Since the semantic content of the second person is not required, the speaking content of the second person is not limited, and the restriction that the first person and the second person must speak the same content in the prior art is avoided, thereby saving a large amount of data production cost and time and improving the efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a speech conversion method and system based on a neural network, a device, and a storage medium. BACKGROUND

[0002] Speech conversion technology is a technology for converting source speech into target speech while keeping the semantic content unchanged, wherein the source speech is speech uttered by a first person, and the target speech is speech uttered by a second person. In other words, the source speech uttered by the first person is converted into target speech uttered by the second person with the same semantics through speech conversion technology.

[0003] With the rapid development of deep neural network technology, the speech converted by the speech conversion method based on deep learning has high similarity and good quality and fluency. The speech conversion method based on deep learning currently mainly includes two steps, i.e., first, training a conversion model using a large amount of speech data, and then using the trained model to perform speech conversion. However, parallel training data sets are required for training, and the parallel training data sets refer to speech of the same words uttered by the first person and the second person. However, such samples are difficult to collect and time-consuming and laborious. SUMMARY

[0004] The present application provides a speech conversion method and system based on a neural network, a device, and a storage medium, which mainly aims to reduce the production cost of training data sets and synthesize target speech with high naturalness and similarity.

[0005] In a first aspect, an embodiment of the present application provides a speech conversion method based on a neural network, which comprises the following steps:

[0006] obtaining source speech and target speaker timbre;

[0007] inputting the source speech into a content encoder to extract the semantic content of the source speech and obtain semantic content encoding;

[0008] inputting the semantic content encoding and the target speaker timbre into a generator to synthesize the semantic content encoding and the target speaker timbre and obtain target speech, wherein the speech conversion model comprises the content encoder and the generator, and the speech conversion model is trained using sample speech and sample timbre.

[0009] Preferably, the speech conversion model further comprises a discriminator, and the speech conversion model is trained using sample speech and sample timbre, and is realized by the following method:

[0010] inputting the sample speech into the content encoder to obtain semantic content encoding corresponding to the sample speech;

[0011] encoding the semantic content corresponding to the sample voice and the sample timbre into the generator, to obtain converted voice;

[0012] inputting the converted voice into the discriminator, judging whether the converted voice is synthetic, if the discriminator judges that the converted voice is synthetic, adjusting the parameters of the encoder and the generator, and retraining using the adjusted encoder and the adjusted generator until the discriminator judges that the converted voice is not synthetic.

[0013] Preferably, the inputting the converted voice into the discriminator, judging whether the converted voice is synthetic, comprises:

[0014] According to the preset discriminator loss function corresponding to the discriminator, a sample actual loss is calculated, if the sample actual loss is within a preset loss threshold, it is judged that the converted voice is not synthetic voice, otherwise, it is judged that the converted voice is synthetic voice.

[0015] Wherein, the preset discriminator loss function is calculated by the following formula:

[0016]

[0017]

[0018] Wherein, L adv (D) represents the training loss function of the discriminator, x represents the sample voice, represents the converted voice, c represents the semantic content corresponding to the sample voice, represents the sample timbre, D represents the discriminator, G represents the generator, D(x) represents the output of the discriminator when the input is the sample voice, represents the output of the discriminator when the input is the converted voice.

[0019] Preferably, the loss function of the voice conversion model in the training process is calculated by the following formula:

[0020] L(G, E c )=L adv (G, E c )+L fm (G, E c )+L con (G, E c ),

[0021]

[0022]

[0023]

[0024] x=G(c,z),

[0025] wherein, L adv (G,E c ) is a training loss function of the content encoder and the generator, L fm (G,E c ) is a reconstruction loss function, L con (G,E c ) is a language content loss function, E c represents the content encoder, E c (x) represents the output of the content encoder when the sample speech is input, represents the output of the generator when the content encoder corresponding to the sample speech and the timbre of the converted speech are input.

[0026] Preferably, the content encoder is composed of a plurality of content encoding units connected in sequence, and the content encoding units include a first residual network and a down-sampling unit.

[0027] Preferably, the generator is composed of a plurality of generation units connected in sequence, and the generation units include an up-sampling unit and a second residual network.

[0028] Preferably, the second residual network includes a first convolution unit, a second convolution unit, a third convolution unit, a first summation unit, a second summation unit, an inverse tangent calculation unit, a sine calculation unit and a product unit, wherein the input of the first convolution unit is the up-sampled target speaker timbre, the output of the first convolution unit and the semantic content encoding are connected with the input of the first summation unit, the output of the first summation unit is connected with the input of the inverse tangent calculation unit and the sine calculation unit respectively, the output of the inverse tangent calculation unit and the output of the sine calculation unit are connected with the input of the product unit, the output of the product unit is connected with the input of the second convolution unit, the semantic content encoding is connected with the input of the third convolution unit, and the output of the second convolution unit and the output of the third convolution unit are connected with the input of the second summation unit.

[0029] In a second aspect, an embodiment of the present application provides a speech conversion system based on a neural network, comprising:

[0030] a data acquisition module, configured to acquire source speech and target speaker timbre;

[0031] a semantic encoding module, configured to input the source speech into a content encoder, extract semantic content of the source speech, and obtain semantic content encoding.

[0032] a semantic synthesis module, configured to input the semantic content encoding and the target speaker prosody into a generator, and synthesize the semantic content encoding and the target speaker prosody to obtain target speech.

[0033] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the speech conversion method based on a neural network when executing the computer program.

[0034] In a fourth aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program, and the computer program implements the steps of the speech conversion method based on a neural network when executed by a processor.

[0035] The speech conversion method, system, device and storage medium based on a neural network provided by the embodiment of the present application first collect source speech and target speaker prosody, input the source speech into a content encoder, separate the semantic content and the prosody of the source speech through the content encoder, encode the extracted semantic content to obtain semantic content encoding, and finally input the semantic content encoding and the target speaker prosody into a generator to synthesize target speech. The speech conversion model used in the embodiment of the present application is a neural network model, which is trained by sample speech and sample prosody. In the embodiment of the present application, only sample speech and sample prosody need to be collected, and the prosody information of the second person can be separated from the speech of the second person to obtain the prosody information of the second person. Since the semantic content of the second person is not required, the speech content of the second person is not limited, thereby avoiding the limitation that the first person and the second person must speak the same content in the prior art, saving a large amount of data production cost and time, and improving the efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 A flowchart of the speech conversion method based on a neural network provided by the embodiment of the present application is shown in FIG. 1;

[0037] Figure 2 A flowchart of the speech conversion method based on a neural network provided by the embodiment of the present application is shown in FIG. 1;

[0038] Figure 3 A flowchart of the training method of the speech conversion model provided by the embodiment of the present application is shown in FIG. 4;

[0039] Figure 4A structural schematic diagram of the speech conversion model in the embodiment of the present application when applied;

[0040] Figure 5 A structural schematic diagram of the speech conversion model in the embodiment of the present application when trained;

[0041] Figure 6 A structural schematic diagram of the content encoder in the embodiment of the present application;

[0042] Figure 7 A structural schematic diagram of the generator in the embodiment of the present application;

[0043] Figure 8 A structural schematic diagram of the second residual network provided in the embodiment of the present application;

[0044] Figure 9 A structural schematic diagram of the speech conversion system based on a neural network provided in the embodiment of the present application;

[0045] Figure 10 A structural schematic diagram of the computer device provided in the embodiment of the present application.

[0046] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are merely intended to explain the present application and are not intended to limit the present application.

[0048] Figure 1 A flowchart of the speech conversion method based on a neural network provided in the embodiment of the present application is shown in Figure 1 As shown, a user inputs source speech and target speaker timbre at a client, the client collects the source speech and the target speaker timbre, and sends the source speech and the target speaker timbre to a server, the server receives the source speech and the target speaker timbre, executes the speech conversion method based on a neural network, and finally obtains target speech.

[0049] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology and application system for using digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain optimal results.

[0050] The artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric identification technology, speech processing technology, natural language processing technology, and machine learning, deep learning, etc.

[0051] It should be noted that the server can be implemented by an independent server or a server cluster composed of multiple servers. The client can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc., but is not limited thereto. The client and the server can be connected through Bluetooth, a universal serial bus (USB) or other communication connection modes, and the embodiments of the present application do not limit this.

[0052] Voice conversion is an important aspect of artificial intelligence. It is to study how to convert one person's voice into another person's voice without changing the content of the voice. Voice conversion belongs to a general technical field of speech synthesis, which is to convert text into speech or change the properties of speech, such as voice identity, emotion, accent, etc.

[0053] Common speech synthesis methods include statistical modeling-based voice conversion methods and deep learning-based voice conversion methods, and the deep learning-based voice conversion method is more widely used. Deep learning technology is usually data-driven and relies on big data, but the training data needs the first person and the second person to say the same thing, which makes it very difficult to obtain a parallel corpus training set.

[0054] To solve the above problems, Figure 2 A flowchart of a voice conversion method based on a neural network provided by an embodiment of the present application is shown in Figure 2 The method comprises the following steps:

[0055] S210, obtaining source speech and target speaker timbre;

[0056] First, the source speech and the target speaker timbre are obtained. The source speech can be a segment of speech collected from the first person on site, a segment of speech synthesized by a certain method from the first person, or a recorded segment in a film or television play, etc. The specific collection method can be determined according to the actual situation, and the embodiments of the present application do not repeat it here.

[0057] The semantic content of the source speech is taken as an example of "Have you eaten today?" in the embodiment of the present application. The semantic content is speech content irrelevant to the first person. In addition to the semantic content, the source speech also includes timbre, which refers to speech content related to the first person, such as formant, fundamental frequency, intonation, intensity and duration. The speech conversion method only modifies the features related to the first person, such as formant, fundamental frequency (F0), intonation, intensity and duration, and replaces the features related to the first person with features related to the second person. The second person-related features and the first person-related features specifically contain the same indicators, and the speech content irrelevant to the speaker is retained. In this way, the speech conversion process is completed. Here, the second person is the target speaker, and the target speaker timbre is the second person timbre.

[0058] Here, only the target speaker timbre is extracted in the embodiment of the present application, and the target speaker speech is not extracted. As an optional embodiment, the method for extracting the target speaker timbre is as follows: obtaining audio sampling data, and extracting the mel spectrum information corresponding to the human voice pronunciation part from the audio sampling data; calling a pre-trained timbre extraction model to extract a vector matrix set representing the timbre of the audio source object of the audio sampling data from the mel spectrum information. The vector matrix set includes a plurality of vector matrices in the time domain. The mean vector matrix between the plurality of vector matrices in the vector matrix set is obtained as the voiceprint feature information of the audio source object. The target speaker corresponding timbre is generated, and the timbre includes the named identifier of the target speaker and the voiceprint feature information pointed to by the named identifier. The method is used to extract the target speaker timbre in the embodiment of the present application, but the extraction method of the target speaker timbre is not limited, and can be analyzed according to the actual situation.

[0059] S220, inputting the source speech into the content encoder to extract the semantic content of the source speech, and obtaining semantic content encoding;

[0060] Then the source speech is input into the content encoder to extract the semantic content from the source speech, that is, "Have you eaten today?" is extracted from the source speech. In the embodiment of the present application, the content encoder can be an existing speech-to-text content extraction software, such as Baidu intelligent cloud speech recognition, Keda Xunfei recording platform, and Xunjie recording to text, etc., which directly converts speech to text.

[0061] Then the extracted semantic content is encoded to obtain semantic content encoding, and the specific encoding method in the embodiment of the application is: generating a word vector matrix according to the text data, inputting the word vector matrix into a bidirectional recurrent neural network for preprocessing operation to obtain an output vector for representing the context semantic relationship of words, performing convolution operation on the output vector to obtain a convolution processing result, and performing pooling operation on the convolution processing result to obtain a fixed-length vector as the semantic encoding of the text data. The encoding method can be specifically defined according to actual conditions, and is not limited to the unique method given in the embodiment.

[0062] S230, input the semantic content encoding and the target speaker timbre into the generator, synthesize the semantic content encoding and the target speaker timbre to obtain a target voice, wherein the voice conversion model comprises the content encoder and the generator, and the voice conversion model is trained with sample voice and sample timbre.

[0063] Then the semantic content encoding and the target speaker timbre are input into the generator, and the semantic content encoding and the target speaker timbre are synthesized by the generator to obtain the target voice, which is "Have you eaten today?" spoken in the target speaker timbre.

[0064] In the embodiment of the application, the content encoder and the generator are both modules of the voice conversion model, and the voice conversion model is a neural network model. Before applying the voice conversion model, the voice conversion model needs to be trained, that is, the model parameters of the content encoder and the generator are adjusted to meet the preset requirements. After the training is completed, the target voice is obtained by using the content encoder with adjusted parameters and the generator with adjusted parameters.

[0065] Here, it also needs to be noted that the voice conversion model is trained with sample voice and sample timbre, and the sample voice is similar to the source voice. The sample voice can be a voice of a person collected on site, a voice of a person synthesized by a certain method, a recording segment in a film or television play, etc. The specific collection method can be determined according to actual conditions, and the embodiment of the application will not be repeated here.

[0066] The sample timbre is analogized to a target timbre, and the method for extracting the sample timbre is as follows: obtaining audio sampling data, extracting the mel spectrum information corresponding to the human voice sound part from the audio sampling data; calling a pre-trained timbre extraction model to extract a vector matrix set representing the timbre of the sound source object of the audio sampling data from the mel spectrum information, the vector matrix set including a plurality of vector matrices in the time domain; obtaining a mean vector matrix between the plurality of vector matrices in the vector matrix set as the voiceprint feature information of the sound source object; and generating the timbre corresponding to the sample object, i.e., the sample timbre, which includes the naming identifier of the sample object and the voiceprint feature information pointed to by the naming identifier. In the embodiment of the present application, the sample timbre of the sample object is extracted using the method, but the extraction method of the sample timbre is not limited, and can be analyzed according to the actual situation.

[0067] As can be seen from the above, in the embodiment of the present application, when making training samples, only sample speech and sample timbre need to be collected, and the second person's timbre information can be separated from the second person's speech to obtain it. Since the semantic content of the second person is not required, the speaking content of the second person can not be limited, avoiding the limitation that the first person and the second person must speak the same content in the prior art, saving a large amount of data production cost and time, and improving the efficiency.

[0068] The speech conversion method based on the neural network provided in the embodiment of the present application first collects source speech and target speaker timbre, inputs the source speech into a content encoder, separates the semantic content and the timbre of the source speech through the content encoder, encodes the extracted semantic content to obtain semantic content encoding, and finally inputs the semantic content encoding and the target speaker timbre into a generator to synthesize target speech. The speech conversion model used in the embodiment of the present application is a neural network model, which is obtained by training sample speech and sample timbre. In the embodiment of the present application, when making training samples, only sample speech and sample timbre need to be collected, and the second person's timbre information can be separated from the second person's speech to obtain it. Since the semantic content of the second person is not required, the speaking content of the second person can not be limited, avoiding the limitation that the first person and the second person must speak the same content in the prior art, saving a large amount of data production cost and time, and improving the efficiency.

[0069] In addition, test experiments prove that, compared with the speech conversion framework based on variational auto-encoder and GANs in the prior art, the embodiment of the present application avoids the problem of over-smoothing, and has higher naturalness and similarity. At the same time, the embodiment of the present application does not require acoustic features such as mel spectrum, and can directly generate high-quality speech.

[0070] Finally, in the embodiment of the present application, the input voice is converted to output voice, while in the traditional method, the voice is converted to a mel-spectrogram, and then the voice is obtained according to the mel-spectrogram, which needs a two-step conversion process. The embodiment of the present application only needs a one-step conversion process, which avoids the intermediate process of converting only acoustic features in the traditional voice conversion and restoring the acoustic features to voice by using a vocoder.

[0071] On the basis of the above-mentioned embodiments, preferably, Figure 3 The training method flow chart of the voice conversion model provided in the embodiment of the present application is shown in Figure 3 The voice conversion model further comprises a discriminator, and the voice conversion model is obtained by training with sample voice and sample timbre in the following manner:

[0072] S310, inputting the sample voice into the content encoder to obtain semantic content encoding corresponding to the sample voice;

[0073] S320, inputting the semantic content encoding corresponding to the sample voice and the sample timbre into the generator to obtain converted voice;

[0074] S330, inputting the converted voice into the discriminator to determine whether the converted voice is synthetic, and if the discriminator determines that the converted voice is synthetic, adjusting the parameters of the encoder and the generator, and retraining by using the adjusted encoder and the adjusted generator until the discriminator determines that the converted voice is not synthetic.

[0075] Figure 4 The structure schematic diagram of the voice conversion model in the application of the embodiment of the present application comprises a content encoder 410 and a generator 420, Figure 5 The structure schematic diagram of the voice conversion model in the training of the embodiment of the present application further comprises a discriminator 510, which is combined with Figure 4 and Figure 5It can be known that, specifically when the voice conversion model is trained by using the sample voice and the sample timbre, the voice conversion model further includes a discriminator in addition to the content encoder and the generator, and taking one training process as an example, first, the sample voice is input into the content encoder to obtain the semantic content code corresponding to the sample voice, then the semantic content code corresponding to the sample voice and the sample timbre are input into the generator to obtain the converted voice, and finally, the converted voice is input into the discriminator to determine whether the converted voice is synthesized, if the discriminator can determine that the converted voice is synthesized, it indicates that the synthesized voice is not real and natural enough, and the model parameters of the encoder and the generator need to be adjusted, and the encoder after the model parameters are adjusted and the generator after the model parameters are adjusted are used to retrain, until the discriminator cannot determine that the converted voice is synthesized, at this time, the training is ended, and the model parameters of the content encoder and the generator obtained finally are taken as the trained content encoder and the trained generator.

[0076] The voice conversion model in the embodiment of the application belongs to a kind of neural network, and the training process of the voice conversion model can be divided into three steps: defining the structure of the voice conversion model and the output result of forward propagation;Define the loss function and the algorithm of back propagation optimization;Finally, generate a conversation and repeatedly run the back propagation optimization algorithm on the training data.

[0077] Wherein, neuron is the smallest unit of neural network, a neuron can have multiple inputs and an output, the input of each neuron can be the output of other neurons, or the input of the entire neural network. The output of the neural network is the weighted sum of all neuron inputs, and the weight of different inputs is the neuron parameter, and the optimization process of the neural network is the process of optimizing the value of the neuron parameter.

[0078] The effect and optimization target of the neural network are defined by the loss function, and the supervised learning is a way of neural network training, and the result given by the neural network should be as close to the true answer as possible. By adjusting the parameters in the neural network, the training data is fitted, so that the neural network provides prediction ability for unknown samples.

[0079] The back propagation algorithm realizes an iterative process, and at the beginning of each iteration, a part of training data is taken, and the prediction result of the neural network is obtained by the forward propagation algorithm. Because the training data has correct answers, the difference between the prediction result and the correct answer can be calculated. Based on this difference, the back propagation algorithm updates the value of the neural network parameter accordingly, so that it is closer to the true answer.

[0080] After the training process is completed by the above method, the trained target ranking model can be applied.

[0081] On the basis of the above embodiment, preferably, the inputting the converted speech into the discriminator to determine whether the converted speech is synthetic includes:

[0082] According to the preset discriminator loss function corresponding to the discriminator, a sample actual loss is calculated, if the sample actual loss is within a preset loss threshold, it is determined that the converted speech is not synthetic speech, otherwise, it is determined that the converted speech is synthetic speech.

[0083] The preset discriminator loss function is calculated by the following formula:

[0084]

[0085]

[0086] Wherein, L adv (D) represents the training loss function of the discriminator, x represents the sample speech, represents the converted speech, c represents the semantic content corresponding to the sample speech, z represents the sample timbre, D represents the discriminator, G represents the generator, D(x) represents the output of the discriminator when the input is the sample speech, represents the output of the discriminator when the input is the converted speech.

[0087] In the embodiment of the present application, the discriminator calculates the actual loss according to the preset discriminator loss function. Specifically, if the actual loss is less than the preset loss threshold, it means that the converted speech obtained by this training meets the requirements, and this training can be used as the last training. At this time, the content encoder and the generator obtained can constitute the trained speech conversion model. If the actual loss is greater than the preset loss threshold, it means that the converted speech obtained by this training does not meet the requirements, and needs to be retrained, that is, the model parameters of the content encoder and the generator are adjusted, and the input sample speech and sample timbre are processed again until the actual loss obtained after reprocessing is less than the preset loss threshold.

[0088] The preset discriminator loss function can be calculated according to the above formula.

[0089] On the basis of the above embodiment, preferably, the loss function of the speech conversion model in the training process is calculated by the following formula:

[0090] L(G,E c )=L adv (G,E c )+L fm (G,E c )+L con(G, E c ),

[0091]

[0092]

[0093]

[0094] x = G(c, z),

[0095] where L adv (G, E c ) is the training loss function of the content encoder and generator, L fm (G, E c ) is the reconstruction loss function, L con (G, E c ) is the language content loss function, E c represents the content encoder, E c (x) represents the output of the content encoder when the sample speech is input, represents the output of the generator when the content encoder corresponding to the sample speech and the timbre of the converted speech are input.

[0096] Specifically, it is assumed that the sample speech x is composed of semantic content C and sample timbre Z, i.e., x = G(c, z), the language content C describes information that is invariant between different speakers, i.e., timbre, such as phonemes and prosody, etc. In order to convert the speech x of the sample object y to obtain the converted speech of the target object The language content c = E c (x) is obtained by mapping through the content encoder, and the sample timbre is replaced, then

[0097] The goal here is to train the content encoder E c , the generator G and the discriminator D, and directly use one-hot encoding to extract the sample timbre Z.

[0098] The content encoder E c and the generator G generate synthesized speech to deceive the discriminator D, while for a specific speaker, the discriminator D discriminates whether the converted speech is synthesized speech, in order to make it difficult to distinguish between the converted speech and the real speech, the adversarial loss function of the generator G and the discriminator D is defined as:

[0099]

[0100] In order to let the generator G use the speaker timbre information, the reconstruction loss function is increased, which is defined as:

[0101]

[0102] In order to keep the converted speech to maintain the original language content, a language content loss function is proposed, which is defined as:

[0103]

[0104] Therefore, the discriminator D training loss function is L adv (D), content encoder E c , the generator G training loss function is:

[0105] L(G, E c ) = L adv (G, E c ) + L fm (G, E c ) + L con (G, E c ).

[0106] Wherein, L adv (G, E c ) is the training loss function of the content encoder and the generator, L fm (G, E c ) is the reconstruction loss function, L con (G, E c ) is the language content loss function, E c represents the content encoder, E c (x) represents the output of the content encoder when the sample speech is input, represents the output of the generator when the content encoder corresponding to the sample speech and the timbre of the converted speech are input.

[0107] On the basis of the above embodiment, preferably, the content encoder is composed of a plurality of content encoding units connected in sequence and end to end, and the content encoding unit comprises a first residual network and a down-sampling unit.

[0108] The content encoder is composed of a plurality of content encoding units, specifically, the content encoder in the embodiment of the application is composed of four content encoding units connected in sequence and end to end, Figure 6 is a structural schematic diagram of the content encoder in the embodiment of the application, as Figure 6 shown, comprising a first residual network 610 and a down-sampling unit 620, wherein the first residual network extracts and encodes the semantic content of the source speech, and inputs the encoded semantic content into the down-sampling unit to obtain the semantic content code.

[0109] On the basis of the above-mentioned embodiments, preferably, the generator is composed of a plurality of generation units connected in sequence, and the generation unit comprises an up-sampling unit and a second residual network.

[0110] The generator is composed of a plurality of generation units, and specifically, the generator in the embodiments of the present application is composed of four generation units connected in sequence, Figure 7 A structural diagram of the generator in the embodiments of the present application is shown in FIG. 7. Figure 7 As shown in FIG. 7, the generation unit comprises an up-sampling unit 710 and a second residual network 720, the up-sampling unit is used for up-sampling the target timbre, and the second residual network is used for obtaining the speech according to the up-sampling result and the semantic content coding.

[0111] On the basis of the above-mentioned embodiments, preferably, the second residual network comprises a first convolution unit, a second convolution unit, a third convolution unit, a first summation unit, a second summation unit, an inverse tangent calculation unit, a sine calculation unit and a product unit, wherein the input of the first convolution unit is the up-sampled target speaker timbre, the output of the first convolution unit and the semantic content coding are both connected with the input of the first summation unit, the output of the first summation unit is connected with the input of the inverse tangent calculation unit and the sine calculation unit respectively, the output of the inverse tangent calculation unit and the output of the sine calculation unit are both connected with the input of the product unit, the output of the product unit is connected with the input of the second convolution unit, the semantic content coding is connected with the input of the third convolution unit, and the output of the second convolution unit and the output of the third convolution unit are both connected with the input of the second summation unit.

[0112] Figure 8 A structural diagram of the second residual network provided in the embodiments of the present application is shown in FIG. 8. Figure 8 As shown in FIG. 8, the second residual network comprises a first convolution unit 810, a second convolution unit 820, a third convolution unit 830, a first summation unit 840, a second summation unit 850, an inverse tangent calculation unit 860, a sine calculation unit 870 and a product unit 880, wherein the input of the first convolution unit is the up-sampled target speaker timbre, the output of the first convolution unit and the semantic content coding are both connected with the input of the first summation unit, the output of the first summation unit is connected with the input of the inverse tangent calculation unit and the sine calculation unit respectively, the output of the inverse tangent calculation unit and the output of the sine calculation unit are both connected with the input of the product unit, the output of the product unit is connected with the input of the second convolution unit, the semantic content coding is connected with the input of the third convolution unit, and the output of the second convolution unit and the output of the third convolution unit are both connected with the input of the second summation unit.

[0113] The second residual network in the embodiment of the application is easy to optimize, and by stacking multiple generation units, the accuracy can be improved by increasing the depth.

[0114] Figure 9 A structural schematic diagram of a speech conversion system based on a neural network provided by the embodiment of the application is shown in the figure, and the system comprises a data acquisition module 910, a semantic encoding module 920 and a semantic synthesis module 930, wherein: Figure 9

[0115] The data acquisition module 910 is configured to acquire source speech and target speaker timbre.

[0116] The semantic encoding module 920 is configured to input the source speech into a content encoder, extract semantic content of the source speech, and obtain semantic content encoding.

[0117] The semantic synthesis module 930 is configured to input the semantic content encoding and the target speaker timbre into a generator, synthesize the semantic content encoding and the target speaker timbre, and acquire target speech, wherein the speech conversion model comprises the content encoder and the generator, and the speech conversion model is trained by using sample speech and sample timbre.

[0118] The embodiment of the system corresponds to the above method, and details are referred to the above method embodiment, which will not be described here.

[0119] On the basis of the above embodiment, preferably, the speech conversion model further comprises a discriminator, and the semantic synthesis module comprises a first synthesis unit, a second synthesis unit and a third synthesis unit, wherein:

[0120] The first synthesis unit is configured to input the sample speech into the content encoder to acquire semantic content encoding corresponding to the sample speech.

[0121] The second synthesis unit is configured to input the semantic content encoding corresponding to the sample speech and the sample timbre into the generator to acquire converted speech.

[0122] The third synthesis unit is configured to input the converted speech into the discriminator to determine whether the converted speech is synthesized, and if the discriminator determines that the converted speech is synthesized, the parameters of the encoder and the generator are adjusted, and the encoder and the generator are retrained by using the adjusted parameters until the discriminator determines that the converted speech is not synthesized.

[0123] On the basis of the above embodiment, preferably, the third synthesis unit comprises a determination unit.

[0124] ​The judgment unit is configured to calculate a sample actual loss according to a preset discriminator loss function corresponding to the discriminator, and if the sample actual loss is within a preset loss threshold, it is judged that the converted speech is not synthetic speech, otherwise, it is judged that the converted speech is synthetic speech.

[0125] The preset discriminator loss function is calculated by the following formula:

[0126]

[0127]

[0128] L adv (D) represents the training loss function of the discriminator, x represents the sample speech, represents the converted speech, c represents the semantic content corresponding to the sample speech, represents the sample timbre, D represents the discriminator, G represents the generator, D(x) represents the output of the discriminator when the input is the sample speech, represents the output of the discriminator when the input is the converted speech.

[0129] On the basis of the above embodiment, preferably, the loss function of the speech conversion model in the training process is calculated by the following formula:

[0130] L(G,E c )=L adv (G,E c )+L fm (G,E c )+L con (G,E c ),

[0131]

[0132]

[0133]

[0134] x=G(c,z),

[0135] L adv (G,E c ) is the training loss function of the content encoder and the generator, L fm (G,E c ) is the reconstruction loss function, L con (G,E c ) is the language content loss function, E c represents the content encoder, E c(x) represents an output of the content encoder when the sample speech is input, (x) represents an output of the content encoder when the sample speech is input,

[0136] On the basis of the above-mentioned embodiments, preferably, the content encoder is composed of a plurality of content encoding units connected in sequence in a head-to-tail manner, and the content encoding unit comprises a first residual network and a down-sampling unit.

[0137] On the basis of the above-mentioned embodiments, preferably, the generator is composed of a plurality of generation units connected in sequence in a head-to-tail manner, and the generation unit comprises an up-sampling unit and a second residual network.

[0138] On the basis of the above-mentioned embodiments, preferably, the second residual network comprises a first convolution unit, a second convolution unit, a third convolution unit, a first summation unit, a second summation unit, an inverse tangent calculation unit, a sine calculation unit and a product unit, wherein the input of the first convolution unit is the up-sampled target speaker timbre, the output of the first convolution unit and the semantic content encoding are connected with the input of the first summation unit, the output of the first summation unit is connected with the input of the inverse tangent calculation unit and the sine calculation unit respectively, the output of the inverse tangent calculation unit and the output of the sine calculation unit are connected with the input of the product unit, the output of the product unit is connected with the input of the second convolution unit, the semantic content encoding is connected with the input of the third convolution unit, and the output of the second convolution unit and the output of the third convolution unit are connected with the input of the second summation unit.

[0139] The above-mentioned various modules in the neural network-based speech conversion system can be realized by software, hardware and a combination thereof in whole or in part. The above-mentioned various modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above-mentioned various modules.

[0140] Figure 10 A structural schematic diagram of a computer device provided in an embodiment of the present application is shown in FIG. 1, which can be a server, and the internal structure diagram thereof can be as shown in FIG. 2. Figure 10As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a computer storage medium and an internal memory. The computer storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the computer storage medium. The database of the computer device is used to store data generated or obtained in the process of executing the neural network-based speech conversion method, such as source speech and target speaker timbre. The network interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a neural network-based speech conversion method.

[0141] In one embodiment, a computer device is provided, including a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to implement the steps of the neural network-based speech conversion method in the above-mentioned embodiments. Alternatively, the processor executes the computer program to implement the functions of each module / unit in the neural network-based speech conversion system in this embodiment.

[0142] In one embodiment, a computer storage medium is provided, and the computer storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the neural network-based speech conversion method in the above-mentioned embodiments. Alternatively, the computer program is executed by the processor to implement the functions of each module / unit in the neural network-based speech conversion system in this embodiment.

[0143] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0144] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the above-described functions.

[0145] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A speech conversion method based on neural networks, characterized in that, include: Obtain the source speech and the target speaker's timbre; The source speech is input into the content encoder to extract the semantic content of the source speech, thus obtaining the semantic content encoding; The step of inputting the source speech into a content encoder to extract the semantic content of the source speech and obtain semantic content encoding includes: generating a word vector matrix based on the text data of the source speech; inputting the word vector matrix into a bidirectional recurrent neural network for preprocessing to obtain an output vector representing the semantic relationship of word context; performing a convolution operation on the output vector to obtain a convolution processing result; and performing a pooling operation on the convolution processing result to obtain a vector of fixed length as the semantic content encoding of the text data. The semantic content encoding and the target speaker timbre are input into the generator, and the semantic content encoding and the target speaker timbre are synthesized to obtain the target speech. The speech conversion model includes the content encoder and the generator, and the speech conversion model is trained with sample speech and sample timbre. The speech conversion model also includes a discriminator, which is trained using sample speech and sample timbre, and is implemented in the following way. The sample speech is input into the content encoder to obtain the semantic content code corresponding to the sample speech; the semantic content code corresponding to the sample speech and the sample timbre are input into the generator to obtain the converted speech; the converted speech is input into the discriminator to determine whether the converted speech is synthesized. If the discriminator determines that the converted speech is synthesized, the parameters of the encoder and the generator are adjusted, and the adjusted encoder and the adjusted generator are used for retraining until the discriminator determines that the converted speech is not synthesized.

2. The speech conversion method based on neural networks according to claim 1, characterized in that, The step of inputting the converted speech into the discriminator and determining whether the converted speech is synthesized includes: The actual loss of the sample is calculated according to the preset discriminator loss function corresponding to the discriminator. If the actual loss of the sample is within the preset loss threshold, it is determined that the converted speech is not synthesized speech; otherwise, it is determined that the converted speech is synthesized speech. The preset discriminator loss function is calculated using the following formula: Among them, L adv (D) represents the training loss function of the discriminator, and x represents the sample speech. Let c represent the converted speech, and c represent the semantic content corresponding to the sample speech. Let D represent the sample timbre, G represent the discriminator, and D(x) represent the output of the discriminator when the input is the sample speech. This indicates the output of the discriminator when the input is the converted speech.

3. The speech conversion method based on neural networks according to claim 2, characterized in that, The loss function of the speech conversion model during training is calculated using the following formula: L(G,E c )=L adv (G,E c )+L fm (G,E c )+L con (G,E c ), x = G(c,z), Among them, L adv (G,E c L is the training loss function for the content encoder and generator. fm (G,E c ) represents the reconstruction loss function, L con (G,E c E is the language content loss function. c E represents the content encoder. c (x) represents the output of the content encoder when the sample speech is input. This represents the output of the generator when the input is the content encoder corresponding to the input sample speech and the timbre of the converted speech.

4. The speech conversion method based on neural networks according to any one of claims 1 to 3, characterized in that, The content encoder is composed of several content encoding units connected end to end in sequence. Each content encoding unit includes a first residual network and a downsampling unit.

5. The speech conversion method based on neural networks according to any one of claims 1 to 3, characterized in that, The generator is composed of several generation units connected end to end in sequence. Each generation unit includes an upsampling unit and a second residual network.

6. The speech conversion method based on neural networks according to claim 5, characterized in that, The second residual network includes a first convolutional unit, a second convolutional unit, a third convolutional unit, a first summation unit, a second summation unit, an arctangent calculation unit, a sine calculation unit, and a product unit. The input to the first convolutional unit is the upsampled target speaker timbre. The output of the first convolutional unit and the semantic content encoding are both connected to the input of the first summation unit. The output of the first summation unit is connected to the inputs of the arctangent calculation unit and the sine calculation unit, respectively. The outputs of the arctangent calculation unit and the sine calculation unit are both connected to the input of the product unit. The output of the product unit is connected to the input of the second convolutional unit. The semantic content encoding is connected to the input of the third convolutional unit. The outputs of the second convolutional unit and the third convolutional unit are both connected to the input of the second summation unit.

7. A speech conversion system based on a neural network, characterized in that, include: The data acquisition module is used to acquire the source speech and the target speaker's timbre; The semantic encoding module is used to input the source speech into the content encoder, extract the semantic content of the source speech, and obtain the semantic content encoding. The step of inputting the source speech into a content encoder to extract the semantic content of the source speech and obtain semantic content encoding includes: generating a word vector matrix based on the text data of the source speech; inputting the word vector matrix into a bidirectional recurrent neural network for preprocessing to obtain an output vector representing the semantic relationship of word context; performing a convolution operation on the output vector to obtain a convolution processing result; and performing a pooling operation on the convolution processing result to obtain a vector of fixed length as the semantic content encoding of the text data. A semantic synthesis module is used to input the semantic content encoding and the target speaker timbre into a generator, synthesize the semantic content encoding and the target speaker timbre to obtain target speech, wherein the speech conversion model includes the content encoder and the generator, and the speech conversion model is trained with sample speech and sample timbre; The speech conversion model also includes a discriminator, and the semantic synthesis module includes a first synthesis unit, a second synthesis unit, and a third synthesis unit; The first synthesis unit is used to input the sample speech into the content encoder to obtain the semantic content encoding corresponding to the sample speech; The second synthesis unit is used to input the semantic content encoding corresponding to the sample speech and the sample timbre into the generator to obtain the converted speech; The third synthesis unit inputs the converted speech into the discriminator to determine whether the converted speech is synthesized. If the discriminator determines that the converted speech is synthesized, it adjusts the parameters of the encoder and the generator, and retrains using the adjusted encoder and the adjusted generator until the discriminator determines that the converted speech is not synthesized.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the neural network-based speech conversion method as described in any one of claims 1 to 6.

9. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the neural network-based speech conversion method as described in any one of claims 1 to 6.

Images