Voice conversion method and device based on data enhancement, equipment and storage medium

Abstract

The invention relates to voice processing, and provides a voice conversion method and device, equipment and a storage medium, and the method comprises the steps: carrying out the data enhancement of a to-be-converted voice, and obtaining a sample voice; inputting the sample voice into a rhythm encoder for processing to obtain phoneme-level rhythm features and spectrum features of the sample voice; inputting the sample voice into a content encoder for processing to obtain content features of the sample voice; inputting the rhythm features and the content features into an attention-based alignment model for alignment processing to obtain aligned rhythm features; inputting the aligned rhythm features, content features, frequency spectrum features and speaker vectors with set timbres into a decoder for synthesis to obtain synthesized voice; and intercepting part of content corresponding to the voice to be converted from the synthesized voice as target voice. The invention also relates to a block chain, and aims to improve the precision and naturalness of voice conversion.

Description

Technical Field

[0001] The present application relates to the technical field of speech processing, and in particular to a speech conversion method, apparatus, device and storage medium based on data enhancement. Background Art

[0002] Voice conversion is a technology that changes the voice characteristics of a person through voice processing to make it sound like another person's voice while keeping the content of the speech unchanged. In the banking customer service scenario, voice conversion can be applied to customer service robots and voice interaction systems. For example, a customer service robot can analyze the customer's voice characteristics and then use voice conversion technology to convert the robot's voice into a voice that better meets the customer's expectations, thereby improving customer satisfaction and trust.

[0003] However, the speech conversion method used in the related art is difficult to understand the meaning and intention of the customer's voice, resulting in the speech conversion effect being not natural and accurate enough. Therefore, how to improve the accuracy and naturalness of speech conversion is an urgent problem to be solved. Summary of the invention

[0004] The main purpose of this application is to provide a speech conversion method, device, equipment and storage medium based on data enhancement, aiming to improve the accuracy and naturalness of speech conversion.

[0005] In a first aspect, the present application provides a method for speech conversion based on data enhancement, comprising:

[0006] Performing data enhancement on the speech to be converted to obtain a sample speech whose length is greater than that of the speech to be converted;

[0007] Inputting the sample speech into a pre-established prosody encoder for processing to obtain phoneme-level prosody features and spectral features of the sample speech; and

[0008] Inputting the sample speech into a pre-established content encoder for processing to obtain content features of the sample speech;

[0009] Inputting the prosodic feature and the content feature into an attention-based alignment model for alignment processing to obtain an aligned prosodic feature;

[0010] Inputting the aligned prosodic features, the content features, the spectral features and the speaker vector of the set timbre into a pre-established decoder for synthesis to obtain a synthesized speech;

[0011] A portion of the content corresponding to the speech to be converted is intercepted from the synthesized speech as the target speech.

[0012] In a second aspect, the present application further provides a voice conversion device, the voice conversion device comprising:

[0013] A data enhancement module, used for performing data enhancement on the speech to be converted to obtain a sample speech whose length is greater than that of the speech to be converted;

[0014] A first processing module is used to input the sample speech into a pre-established prosody encoder for processing to obtain phoneme-level prosody features and spectral features of the sample speech; and

[0015] A second processing module is used to input the sample speech into a pre-established content encoder for processing to obtain content features of the sample speech;

[0016] A rhythm alignment module, used for inputting the rhythm feature and the content feature into an attention-based alignment model for alignment processing to obtain an aligned rhythm feature;

[0017] A feature synthesis module, used for inputting the aligned prosodic features, the content features, the spectral features and the speaker vector of the set timbre into a pre-established decoder for synthesis to obtain a synthesized speech;

[0018] The speech interception module is used to intercept the partial content corresponding to the speech to be converted from the synthesized speech as the target speech.

[0019] In a third aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the speech conversion method as described above are implemented.

[0020] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the speech conversion method as described above are implemented.

[0021] The present application provides a method, device, equipment and storage medium for speech conversion based on data enhancement. The speech conversion method in the present application performs data enhancement on the speech to be converted to obtain a sample speech whose length is greater than the speech to be converted; the sample speech is input into a pre-established prosody encoder for processing to obtain the prosody features at the phoneme level and the spectral features of the sample speech; and the sample speech is input into a pre-established content encoder for processing to obtain the content features of the sample speech; the prosody features and content features are input into an attention-based alignment model for alignment processing to obtain aligned prosody features; the aligned prosody features, content features, spectral features and speaker vectors with set timbre are input into a pre-established decoder for synthesis to obtain synthesized speech; and part of the content corresponding to the speech to be converted is intercepted from the synthesized speech as the target speech. The embodiment of the present application avoids large deviations in the prosody features at the phoneme level during speech synthesis during speech conversion by means of data enhancement, which can alleviate the decline in speech content quality caused by prosody migration, thereby greatly improving the accuracy and naturalness of speech conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 A flowchart of a voice conversion method provided in an embodiment of the present application;

[0024] Figure 2 for Figure 1 A schematic diagram of the sub-step flow chart of the speech conversion method in FIG.

[0025] Figure 3 A schematic diagram of a reference coding sub-model provided in an embodiment of the present application;

[0026] Figure 4 A schematic diagram of a prosody encoder provided in an embodiment of the present application;

[0027] Figure 5 A schematic diagram of a scenario for implementing the voice conversion method provided in an embodiment of the present application;

[0028] Figure 6 A schematic block diagram of a speech conversion device provided in an embodiment of the present application;

[0029] Figure 7 for Figure 6 A schematic block diagram of a submodule of a speech conversion device in;

[0030] Figure 8 A schematic block diagram of a computer device provided in an embodiment of the present application.

[0031] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0034] Speech conversion aims to convert the voice of one person into another without changing the content of the speech. In the banking customer service scenario, expressive speech conversion technology can be applied to customer service robots and voice interaction systems to improve customer experience and service quality. For example, when a customer calls a bank customer service, the customer service robot can analyze the customer's voice characteristics and emotional characteristics, and then use expressive speech conversion technology to convert the robot's voice into a voice that better meets the customer's expectations, thereby improving customer satisfaction and trust.

[0035] There are still some challenges and problems in the current voice conversion technology, which makes the voice conversion effect not natural and accurate enough. Taking bank customer service as an example, the service provided by the intelligent voice robot needs to be as close to real customer service as possible, which has requirements for timbre and style. However, in the bank customer service scenario, the voice characteristics and emotional characteristics of some customers may be affected by environmental and personal factors, which will have a certain impact on the effect of voice conversion. In addition, for some complex voice features (such as dialects, accents, etc.), the conversion effect of existing technologies still needs to be improved.

[0036] Based on this, the embodiments of the present application provide a data-enhanced voice conversion method, device, equipment and storage medium. Among them, the voice conversion method can be applied to a terminal device or a server, and the terminal device can be an electronic device such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a personal digital assistant and a wearable device; the server can be a single server or a server cluster composed of multiple servers. The embodiments of the present application combine deep learning technology, natural language processing technology and speech synthesis technology to improve the accuracy and naturalness of voice conversion. The embodiments of the present application can be applied to bank customer service scenarios to achieve better customer experience and service quality.

[0037] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0038] Please refer to Figure 1 , Figure 1 A flowchart of the steps of a voice conversion method provided in an embodiment of the present application.

[0039] like Figure 1 As shown, the voice conversion method includes steps S101 to S106.

[0040] Step S101: perform data enhancement on the speech to be converted to obtain a sample speech whose length is greater than the speech to be converted.

[0041] The speech to be converted may be an original speech signal, and the original speech signal may be a time domain waveform or a preprocessed sound spectrum, etc. The sample speech may be the speech to be converted obtained after data enhancement. Data enhancement may refer to splicing another speech data on the basis of the speech to be converted to obtain the sample speech, thereby increasing the length of the speech to be converted, so that the length of the sample speech is greater than the length of the speech to be converted, or the sample speech includes the speech to be converted.

[0042] It is understandable that when the user wishes to change certain properties of the original voice signal, such as converting one person's voice into another person's voice, or converting voice from one language or accent into another language or accent, or changing the emotional color in the voice, for example, from calm to excited, the original voice signal can be used as the voice to be converted, and the voice to be converted can be data enhanced to obtain a sample voice whose length is greater than the voice to be converted.

[0043] It should be noted that the embodiment of the present application extracts the prosodic features at the phoneme level in the subsequent steps. When the prosodic features at the phoneme level are aligned with the content features at the frame level, a large deviation often occurs in the synthesis of the last segment of phonemes. Therefore, by performing data enhancement on the speech to be converted, the prosodic features at the phoneme level of the last segment of the speech to be converted will not be used as the prosodic features at the phoneme level of the last segment of the sample speech, thereby avoiding a large deviation in the synthesis of the last segment of phonemes in the speech to be converted, thereby alleviating the degradation of speech content quality caused by prosodic migration, thereby improving the accuracy and naturalness of speech conversion.

[0044] In one embodiment, the speech to be converted is copied to obtain a copied speech, and the copied speech and the speech to be converted are connected to obtain a sample speech.

[0045] The speech content of the copied speech and the speech to be converted may be consistent. The number of copies may be one or more times, that is, the copied speech is at least one. For example, after copying the speech to be converted once, two speech to be converted with the same content can be obtained, and then the two speech to be converted with the same content are connected to obtain a sample speech whose length in series in the time dimension is twice the length of the speech to be converted. The method of obtaining the copied speech by copying the speech to be converted is simple and convenient to operate, and the implementation cost is low.

[0046] Exemplarily, the length of the speech to be converted is L1, and the speech to be converted is copied to obtain another copied speech with a length of L1. The copied speech and the speech to be converted are connected to obtain a sample speech with a length of twice L1, and the sample speech contains two speech with the same content in the front and back halves.

[0047] In one embodiment, a first voice is obtained, and the first voice is connected to the end of the voice to be converted to obtain a sample voice. The first voice can be flexibly set according to actual conditions. The first voice can be part of the voice data in the voice to be converted, or other voice data. The length of the first voice can be greater than the length of the voice to be converted, or less than or equal to the length of the voice to be converted.

[0048] It should be noted that the first speech is connected to the end of the speech to be converted to ensure that the last segment of the speech content of the speech to be converted will not be at the end of the sample speech, thereby avoiding a large deviation in the synthesis of the last segment of the phonemes of the speech to be converted, thereby ensuring the quality of the speech content of the rhythmic transfer, thereby improving the accuracy and naturalness of the speech conversion.

[0049] Exemplarily, a first speech with a length of L2 is obtained, and the length of the speech to be converted is L1. The first speech is connected to the end of the speech to be converted to obtain a sample speech with a length of L1+L2, the front end content of the sample speech is the speech to be converted, and the back end content is the first speech.

[0050] Step S102: Input the sample speech into a pre-established prosody encoder for processing to obtain phoneme-level prosodic features and spectral features of the sample speech.

[0051] The prosody encoder may be pre-established, and is used to extract prosody features and spectral features of the sample speech, wherein the prosody features are phoneme-level prosody features. The prosody encoder may include a reference encoding sub-model and a spectral encoding sub-model. The pre-established prosody encoder may accurately extract the phoneme-level prosody features and spectral features of the sample speech.

[0052] In one embodiment, the prosody encoder includes a reference coding sub-model and a spectral coding sub-model. Figure 2 As shown, step S102 includes: sub-step S1021 to sub-step S1022.

[0053] Sub-step S1021: input the sample speech into the reference coding sub-model to extract prosodic features, and obtain prosodic features at the phoneme level.

[0054] The reference encoder submodel may be a reference encoder that is close to human ear perception, and the reference encoder submodel may accurately extract the prosodic features of the sample speech to obtain the prosodic features at the phoneme level. The prosodic features at the phoneme level may include pitch, timbre, speaking speed, accent, etc., reflecting the rhythm and intonation of the speech. The reference encoder submodel may also be obtained based on a deep learning model, such as recurrent neural networks (RNNs), long short-term memory networks (LSTMs), or transformers.

[0055] Furthermore, the reference coding sub-model includes a convolution layer, a gated recurrent layer, a linear layer and an activation layer. The sample speech is input into the reference coding sub-model for prosodic feature extraction to obtain the prosodic features at the phoneme level, including: inputting the sample speech into the convolution layer for feature extraction to obtain the speech spectrum of the sample speech; inputting the speech spectrum of the sample speech into the gated recurrent layer for processing to obtain the temporal features of the sample speech; inputting the temporal features of the sample speech into the linear layer for mapping processing to obtain the distribution vectors of the sample speech at different frequencies; inputting the distribution vectors of the sample speech at different frequencies into the activation layer for processing to obtain the prosodic features at the phoneme level.

[0056] It should be noted that the speech spectrum of the sample speech refers to the graphical representation of the frequency components of the speech signal and its intensity distribution, which can be used to describe the acoustic characteristics of the speech. The temporal characteristics of the sample speech refer to the changing characteristics of the speech signal in the time dimension. These characteristics reflect the dynamic behavior of the speech, including the duration of the phoneme, pitch changes, speaking rate, pauses and intonation. The distribution vector of the sample speech at different frequencies usually refers to the spectral representation of the speech signal, which describes the energy or amplitude distribution of the speech signal at each frequency component. The distribution vectors at different frequencies can be obtained by Fourier transforming the speech signal. By referring to the convolutional layer, gated recurrent layer, linear layer and activation layer in the coding sub-model, the accuracy and precision of obtaining the prosodic features at the phoneme level can be improved.

[0057] For example, Figure 3 As shown, Figure 3 A schematic diagram of a reference coding submodel provided in an embodiment of the present application. The reference coding submodel 10 includes a convolutional layer 11, a gated recurrent layer 12, a linear layer 13, and an activation layer 14. When the sample speech is input into the reference coding submodel for prosodic feature extraction, the prosodic features at the phoneme level are obtained through the convolutional layer 11, the gated recurrent layer 12, the linear layer 13, and the activation layer 14.

[0058] Sub-step S1022: input the sample speech into the spectrum coding sub-model to extract spectrum features, so as to obtain spectrum features of the sample speech.

[0059] Among them, spectral feature extraction involves extracting useful information from the spectral representation of the speech signal, that is, the spectral features of the sample speech. The spectral encoding submodel generally refers to a model used to convert the spectral features of the speech signal into a coded representation. Spectral feature extraction can also be implemented through a variety of tools and libraries. Neural networks, especially autoencoders (Autoencoders) or variational autoencoders (VariationalAutoencoders, VAEs), can also be used to learn low-dimensional representations of spectral features. Alternatively, it can also be implemented using techniques such as Linear Prediction (LP) and Linear Predictive Coding (LPC).

[0060] Furthermore, the spectrum coding sub-model includes a fundamental frequency extraction sub-module and an energy extraction sub-module. The sample speech is input into the spectrum coding sub-model to extract spectrum features to obtain spectrum features of the sample speech, including: inputting the sample speech into the fundamental frequency extraction sub-module to extract frequency features to obtain fundamental frequency features of the sample speech; and inputting the sample speech into the energy extraction sub-module to extract energy features to obtain energy features of the sample speech; the fundamental frequency features and energy features are spectrum features of the sample speech.

[0061] Among them, the fundamental frequency feature (F0), usually referred to as the fundamental frequency (F0), corresponds to the basic frequency of the vocal cord vibration. The fundamental frequency is the number of times the vocal cord vibrates per second, measured in Hertz (Hz). The fundamental frequency is the most basic frequency component in the speech signal and determines the pitch of the sound. The energy feature of the sample speech is an acoustic parameter that describes the strength and energy of the speech signal, and can provide important information about the loudness, clarity, and dynamic range of the speech. As part of the speech conversion method, the fundamental frequency feature and energy feature can help improve the accuracy and precision of speech conversion.

[0062] It should be noted that the spectrum coding sub-model may also include a preprocessing sub-module. The sample speech is input into the preprocessing sub-module for preprocessing to obtain the preprocessed sample speech. The preprocessing includes pre-emphasis, framing and windowing of the speech signal in the time domain to reduce transient noise in the signal and enhance the periodicity of the signal.

[0063] It is understandable that the above sub-steps S1021 to S1022 are only an example. In practical applications, the execution order of sub-steps S1021 and S1022 can be determined according to actual conditions, for example, sub-steps S1021 and S1022 can be executed synchronously, or sub-step S1022 can be set to be executed before sub-step S1021.

[0064] In one embodiment, a training method for a prosody encoder includes: iteratively training a preset reference coding sub-model through a first training sample set until the reference coding sub-model converges to obtain a trained reference coding sub-model; and iteratively training a preset spectral coding sub-model through a second training sample set until the spectral coding sub-model converges to obtain a trained spectral coding sub-model; and splicing the trained reference coding sub-model with the trained spectral coding sub-model to obtain a pre-established prosody encoder.

[0065] The first training sample set and the second training sample set may include multiple speech samples of different speakers, different emotions and different language styles, and are obtained by rhythmically annotating the multiple speech samples. The rhythm annotation includes annotations of rhythmic features such as pitch curve, duration, and energy. The model parameters of the reference coding submodel and the spectrum coding submodel are initialized, and the model parameters of the reference coding submodel and the spectrum coding submodel are optimized in iterative training, and a sufficient number of iterations are determined to achieve convergence. After obtaining the trained reference coding submodel and the trained spectrum coding submodel, the trained reference coding submodel and the trained spectrum coding submodel are spliced ​​to obtain a pre-established rhythm encoder, which has good robustness.

[0066] For example, Figure 4 As shown, the prosody encoder includes a reference encoding submodel 10 and a spectrum encoding submodel 20. The reference encoding submodel 10 can be obtained after iterative training through the first training sample set, and the spectrum encoding submodel 20 can be obtained after iterative training through the second training sample set. After the reference encoding submodel 10 and the spectrum encoding submodel 20 are trained, the trained reference encoding submodel 10 and the trained spectrum encoding submodel 20 are spliced ​​to obtain a pre-established prosody encoder.

[0067] Step S103: Input the sample speech into a pre-established content encoder for processing to obtain content features of the sample speech.

[0068] Among them, the role of the content encoder is to extract content features related to the speech content from the sample speech. The content features do not depend on the speaker's individual characteristics, such as pitch, accent, etc., but reflect the semantic content of the speech, such as vocabulary, sentence structure, etc.

[0069] The pre-established content encoder can be an acoustic model based on deep learning, such as CNN (convolutional neural network), RNN (recurrent neural network), LSTM (long short-term memory network), etc., or it can be a multi-task learning model based on deep learning, such as MTL (multi-task learning), or it can be a content encoder based on a transformer (Transformer) or a content encoder based on an encoder-decoder architecture. The embodiments of the present application do not make specific limitations on this.

[0070] In one embodiment, a sample speech is input into a pre-established content encoder for processing to obtain content features of the sample speech, including: pre-processing the input speech to prepare data suitable for the content encoder; extracting acoustic features from the pre-processed audio frames, and inputting the extracted acoustic features into the content encoder; extracting content features through the forward propagation process of the content encoder; and post-processing the content features output by the content encoder to ensure the quality and consistency of the content features.

[0071] Step S104: input the prosodic features and the content features into an attention-based alignment model for alignment processing to obtain aligned prosodic features.

[0072] The granularity of the prosodic features obtained by the reference encoder is different from the granularity of the content features obtained by the content encoder. The prosodic features are at the phoneme level, while the content features are at the frame level. The frame level is the minimum time unit for extracting text data, and the frame level is larger than the phoneme level. Therefore, in order to complete the embedding of prosody in the speech conversion process, the prosodic features and content features need to be aligned.

[0073] It should be noted that the attention-based alignment model is an alignment method that uses a self-attention mechanism to match prosodic features with content features. It can find the temporal correspondence between prosodic features and content features so as to maintain the semantic information of the sample speech during the conversion process, so as to generate a target speech with specific prosodic characteristics.

[0074] In the attention-based alignment model provided in the embodiment of the present application, the model learns how to allocate attention so as to establish a corresponding relationship between the prosodic features and content features of the sample speech. During the training process of the alignment model, the attention-based alignment model learns the attention distribution by minimizing the difference between the prosodic features and content features of the sample speech. This is usually achieved through contrastive loss or similarity loss.

[0075] In one embodiment, an attention mechanism is used to align the attention of the content feature as a query and the prosodic feature as a key and a value, thereby obtaining an aligned prosodic feature. The content feature is input into the attention mechanism as a query. The query is the target that the model attempts to match, reflecting the semantic information of the sample speech. The prosodic feature is input into the attention mechanism as a key and a value. The key is the reference used by the model to match the query, and the value is the information that the model focuses on during the matching process. The attention mechanism calculates the similarity between the content feature query and the prosodic feature key, and assigns attention weights accordingly.

[0076] It should be noted that according to the attention weights, the alignment model performs weighted summation on the values ​​of the prosodic features to obtain the values ​​of the prosodic features most relevant to the content feature query. This weighted summation process is the alignment process, which establishes a corresponding relationship between the prosodic features of the sample speech and the content features, so that the prosodic features and content features of the sample speech can be effectively matched.

[0077] Step S105: input the aligned prosodic features, content features, spectral features and speaker vector of set timbre into a pre-established decoder for synthesis to obtain synthesized speech.

[0078] Among them, the decoder is a key component in the speech conversion technology, and the decoder can be used to generate the final synthesized speech according to the input features and parameters. In the embodiment of the present application, the input features and parameters include aligned prosodic features, content features, spectral features, and speaker vectors with set timbre. The pre-established decoder can be a model based on deep learning, such as a variational autoencoder (VAE), a generative adversarial network (GAN), or a direct vocoder (WaveNet).

[0079] It should be noted that the speaker vector of the set timbre contains the acoustic characteristics of the target speaker, such as formant, vocal tract length, etc. The aligned prosodic features are the aligned prosodic information, including pitch, duration, energy, etc. The aligned prosodic information reflects the prosodic characteristics of the speech, such as rhythm and intonation. The content features reflect the semantic information of the sample speech, such as vocabulary, sentence structure, etc. The spectrum features reflect the acoustic characteristics of the sample speech, such as fundamental frequency features, energy features, etc.

[0080] In one embodiment, obtaining a speaker vector of a set timbre includes: obtaining voice data recorded by a user who sets the timbre; extracting timbre feature data and prosody feature data from the voice data; and obtaining a speaker vector based on the timbre feature data and prosody feature data.

[0081] The user who sets the timbre can be a human customer service representative with a specific timbre, or a professional dubbing staff. The timbre feature data and the prosody feature data can be extracted from the voice data recorded by the user who sets the timbre, and the speaker vector includes the timbre feature data and the prosody feature data. For example, the speaker vector is obtained after the timbre feature data and the prosody feature data are processed. The speaker vector is used in the decoder to realize the migration of timbre and prosody, thereby obtaining synthesized speech.

[0082] Step S106: extracting a portion of the content corresponding to the speech to be converted from the synthesized speech as the target speech.

[0083] The length of the synthesized speech is the same as that of the sample speech. That is to say, the synthesized speech is obtained by voice conversion of the sample speech, so the length of the synthesized speech is greater than the length of the speech to be converted. The sample speech is obtained by data enhancement of the speech to be converted. Corresponding to the data enhancement, in order to obtain part of the content of the speech to be converted, it is necessary to intercept the corresponding content from the synthesized speech, so as to obtain the target speech corresponding to the speech to be converted, so as to avoid the distortion of the speech content of the last part of the speech to be converted.

[0084] In one embodiment, the data enhancement method includes copying the speech to be converted, and the sample speech includes the copied speech to be converted and the original speech to be converted. In this case, a part of the content corresponding to the speech to be converted is intercepted from the synthesized speech as the target speech, including: intercepting the speech content of the first half of the synthesized speech as the target speech.

[0085] In one embodiment, the data enhancement method includes connecting the first speech to the end of the speech to be converted, and the sample speech includes the speech to be converted and the first speech connected in series. In this case, the part of the content corresponding to the speech to be converted is intercepted from the synthesized speech as the target speech, including: intercepting the speech content corresponding to the speech to be converted in the first half of the synthesized speech as the target speech. Alternatively, the part of the content corresponding to the speech to be converted is intercepted from the synthesized speech as the target speech, including: removing the speech content corresponding to the first speech from the synthesized speech to obtain the target speech. That is, the remaining speech content corresponds to the part of the speech to be converted.

[0086] It should be noted that in order to further ensure the privacy and security of the above-mentioned voice to be converted, target voice and other related information, the above-mentioned voice to be converted, target voice and other related information can also be stored in a node of a blockchain. The technical solution of the present application can also be applied to adding other data files stored on the blockchain. The blockchain referred to in the present application is a new application model of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm.

[0087] For example, Figure 5As shown, the speech to be converted is data enhanced to obtain a sample speech whose length is greater than that of the speech to be converted; the sample speech is input into a pre-established prosody encoder 01 for processing to obtain phoneme-level prosody features and spectral features of the sample speech; and the sample speech is input into a pre-established content encoder 02 for processing to obtain content features of the sample speech; the prosody features and content features are input into an attention-based alignment model 03 for alignment processing to obtain aligned prosody features; the aligned prosody features, content features, spectral features and speaker vectors of set timbre are input into a pre-established decoder 04 for synthesis to obtain synthesized speech; and part of the content corresponding to the speech to be converted is intercepted from the synthesized speech as the target speech.

[0088] The speech conversion method provided in the above embodiment performs data enhancement on the speech to be converted to obtain a sample speech whose length is greater than that of the speech to be converted; inputs the sample speech into a pre-established prosody encoder for processing to obtain the prosody features at the phoneme level and the spectral features of the sample speech; and inputs the sample speech into a pre-established content encoder for processing to obtain the content features of the sample speech; inputs the prosody features and content features into an attention-based alignment model for alignment processing to obtain aligned prosody features; inputs the aligned prosody features, content features, spectral features and speaker vectors with set timbre into a pre-established decoder for synthesis to obtain synthesized speech; and extracts part of the content corresponding to the speech to be converted from the synthesized speech as the target speech. The embodiment of the present application avoids large deviations in the prosody features at the phoneme level during speech synthesis during speech conversion by means of data enhancement, which can alleviate the decline in speech content quality caused by prosody migration, thereby greatly improving the accuracy and naturalness of speech conversion.

[0089] Please refer to Figure 6 , Figure 6 A schematic block diagram of a speech conversion device provided in an embodiment of the present application.

[0090] like Figure 6 As shown, the speech conversion device 200 includes:

[0091] The data enhancement module 201 is used to perform data enhancement on the speech to be converted to obtain a sample speech whose length is greater than that of the speech to be converted;

[0092] A first processing module 202 is used to input the sample speech into a pre-established prosody encoder for processing to obtain phoneme-level prosody features and spectral features of the sample speech; and

[0093] The second processing module 203 is used to input the sample speech into a pre-established content encoder for processing to obtain content features of the sample speech;

[0094] A prosody alignment module 204, configured to input the prosody feature and the content feature into an attention-based alignment model for alignment processing to obtain an aligned prosody feature;

[0095] A feature synthesis module 205, for inputting the aligned prosodic features, the content features, the spectral features and the speaker vector of the set timbre into a pre-established decoder for synthesis to obtain a synthesized speech;

[0096] The speech interception module 206 is used to intercept the partial content corresponding to the speech to be converted from the synthesized speech as the target speech.

[0097] In one embodiment, the data enhancement module 201 is further configured to:

[0098] Copying the speech to be converted to obtain a copied speech, and connecting the copied speech and the speech to be converted to obtain the sample speech; or

[0099] A first voice is acquired, and the first voice is connected to the end of the voice to be converted to obtain the sample voice.

[0100] In one embodiment, the prosody encoder comprises a reference coding sub-model and a spectral coding sub-model; Figure 7 As shown, the first processing module 202 includes:

[0101] The prosodic feature extraction submodule 2021 is used to input the sample speech into the reference coding submodel to extract prosodic features and obtain phoneme-level prosodic features.

[0102] The spectrum feature extraction submodule 2022 is used to input the sample speech into the spectrum coding submodel to extract spectrum features, so as to obtain the spectrum features of the sample speech.

[0103] In one embodiment, the spectrum coding sub-model includes a baseband extraction sub-module and an energy extraction sub-module; the spectrum feature extraction sub-module 2022 is further used to:

[0104] Inputting the sample speech into the fundamental frequency extraction submodule to extract frequency features to obtain fundamental frequency features of the sample speech; and

[0105] The sample speech is input into the energy extraction submodule to extract energy features, so as to obtain energy features of the sample speech; the fundamental frequency features and the energy features are the spectrum features of the sample speech.

[0106] In one embodiment, the reference coding sub-model includes a convolutional layer, a gated recurrent layer, a linear layer and an activation layer; the prosody feature extraction sub-module 2021 is further used for:

[0107] Inputting the sample speech into the convolution layer for feature extraction to obtain a speech spectrum of the sample speech;

[0108] Inputting the speech spectrum of the sample speech into the gated recurrent layer for processing to obtain the time series features of the sample speech;

[0109] Inputting the temporal features of the sample speech into the linear layer for mapping processing to obtain distribution vectors of the sample speech at different frequencies;

[0110] The distribution vectors of the sample speech at different frequencies are input into the activation layer for processing to obtain the prosodic features at the phoneme level.

[0111] In one embodiment, the feature synthesis module 205 is further used to:

[0112] Get the voice data recorded by the user with set timbre;

[0113] Extracting timbre feature data and prosody feature data from the speech data;

[0114] The speaker vector is obtained according to the timbre feature data and the prosody feature data.

[0115] In one embodiment, the speech conversion device 200 is further used for:

[0116] Iteratively training a preset reference coding sub-model through a first training sample set until the reference coding sub-model converges to obtain a trained reference coding sub-model; and

[0117] Iteratively training a preset spectrum coding sub-model through a second training sample set until the spectrum coding sub-model converges to obtain a trained spectrum coding sub-model;

[0118] The trained reference coding sub-model is concatenated with the trained spectral coding sub-model to obtain a pre-established prosody encoder.

[0119] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module and unit can refer to the corresponding process in the aforementioned speech conversion method embodiment, and will not be repeated here.

[0120] The apparatus provided in the above embodiment may be implemented in the form of a computer program. The computer program may be Figure 8 Runs on the computer device shown.

[0121] See also Figure 8 , Figure 8A schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0122] like Figure 8 As shown, the computer device includes a processor, a memory and a network interface connected via a system bus, wherein the memory may include a storage medium and an internal memory, and the storage medium may be non-volatile or volatile.

[0123] The storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any one of the speech conversion methods.

[0124] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0125] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any speech conversion method.

[0126] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0127] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0128] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0129] Performing data enhancement on the speech to be converted to obtain a sample speech whose length is greater than that of the speech to be converted;

[0130] Inputting the sample speech into a pre-established prosody encoder for processing to obtain phoneme-level prosody features and spectral features of the sample speech; and

[0131] Inputting the sample speech into a pre-established content encoder for processing to obtain content features of the sample speech;

[0132] Inputting the prosodic feature and the content feature into an attention-based alignment model for alignment processing to obtain an aligned prosodic feature;

[0133] Inputting the aligned prosodic features, the content features, the spectral features and the speaker vector of the set timbre into a pre-established decoder for synthesis to obtain a synthesized speech;

[0134] A portion of the content corresponding to the speech to be converted is intercepted from the synthesized speech as the target speech.

[0135] In one embodiment, when the processor performs data enhancement on the speech to be converted to obtain a sample speech having a length greater than that of the speech to be converted, the processor is used to implement:

[0136] Copying the speech to be converted to obtain a copied speech, and connecting the copied speech and the speech to be converted to obtain the sample speech; or

[0137] A first voice is acquired, and the first voice is connected to the end of the voice to be converted to obtain the sample voice.

[0138] In one embodiment, the prosody encoder includes a reference encoding sub-model and a spectrum encoding sub-model; when the processor implements the step of inputting the sample speech into a pre-established prosody encoder for processing to obtain the prosody features at the phoneme level and the spectrum features of the sample speech, the processor is used to implement:

[0139] Inputting the sample speech into the reference coding sub-model to extract prosodic features to obtain prosodic features at the phoneme level; and

[0140] The sample speech is input into the spectrum coding sub-model to extract spectrum features, so as to obtain the spectrum features of the sample speech.

[0141] In one embodiment, the spectrum coding sub-model includes a fundamental frequency extraction sub-module and an energy extraction sub-module; when the processor implements the step of inputting the sample speech into the spectrum coding sub-model to extract the spectrum feature and obtain the spectrum feature of the sample speech, it is used to implement:

[0142] Inputting the sample speech into the fundamental frequency extraction submodule to extract frequency features to obtain fundamental frequency features of the sample speech; and

[0143] The sample speech is input into the energy extraction submodule to extract energy features, so as to obtain energy features of the sample speech; the fundamental frequency features and the energy features are the spectrum features of the sample speech.

[0144] In one embodiment, the reference coding sub-model includes a convolutional layer, a gated recurrent layer, a linear layer and an activation layer; when the processor implements the step of inputting the sample speech into the reference coding sub-model to extract the prosodic features and obtain the prosodic features at the phoneme level, it is used to implement:

[0145] Inputting the sample speech into the convolution layer for feature extraction to obtain a speech spectrum of the sample speech;

[0146] Inputting the speech spectrum of the sample speech into the gated recurrent layer for processing to obtain the time series features of the sample speech;

[0147] Inputting the temporal features of the sample speech into the linear layer for mapping processing to obtain distribution vectors of the sample speech at different frequencies;

[0148] The distribution vectors of the sample speech at different frequencies are input into the activation layer for processing to obtain the prosodic features at the phoneme level.

[0149] In one embodiment, when obtaining the speaker vector of the set timbre, the processor is used to implement:

[0150] Get the voice data recorded by the user with set timbre;

[0151] Extracting timbre feature data and prosody feature data from the speech data;

[0152] The speaker vector is obtained according to the timbre feature data and the prosody feature data.

[0153] In one embodiment, the processor, when implementing the training method of the prosody encoder, is used to implement:

[0154] Iteratively training a preset reference coding sub-model through a first training sample set until the reference coding sub-model converges to obtain a trained reference coding sub-model; and

[0155] Iteratively training a preset spectrum coding sub-model through a second training sample set until the spectrum coding sub-model converges to obtain a trained spectrum coding sub-model;

[0156] The trained reference coding sub-model is concatenated with the trained spectral coding sub-model to obtain a pre-established prosody encoder.

[0157] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the computer device described above can refer to the corresponding process in the aforementioned speech conversion method embodiment, and will not be repeated here.

[0158] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0159] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the speech conversion method of the present application.

[0160] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0161] Furthermore, the computer-usable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of blockchain nodes, etc. The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods, each of which contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. The blockchain may include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0162] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0163] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0164] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A speech conversion method based on data enhancement, characterized in that: include: Performing data enhancement on the speech to be converted to obtain a sample speech whose length is greater than that of the speech to be converted; Inputting the sample speech into a pre-established prosody encoder for processing to obtain phoneme-level prosody features and spectral features of the sample speech; as well as Inputting the sample speech into a pre-established content encoder for processing to obtain content features of the sample speech; Inputting the prosodic feature and the content feature into an attention-based alignment model for alignment processing to obtain an aligned prosodic feature; Inputting the aligned prosodic features, the content features, the spectral features and the speaker vector of the set timbre into a pre-established decoder for synthesis to obtain a synthesized speech; A portion of the content corresponding to the speech to be converted is intercepted from the synthesized speech as the target speech.

2. The voice conversion method according to claim 1, characterized in that: The step of performing data enhancement on the speech to be converted to obtain a sample speech having a length greater than that of the speech to be converted comprises: Copying the speech to be converted to obtain a copied speech, and connecting the copied speech and the speech to be converted to obtain the sample speech; or A first voice is acquired, and the first voice is connected to the end of the voice to be converted to obtain the sample voice.

3. The voice conversion method according to claim 1, wherein: The prosody encoder includes a reference encoding sub-model and a spectrum encoding sub-model; the sample speech is input into a pre-established prosody encoder for processing to obtain the prosody features at the phoneme level and the spectrum features of the sample speech, including: Inputting the sample speech into the reference coding sub-model to extract prosodic features to obtain prosodic features at the phoneme level; and The sample speech is input into the spectrum coding sub-model to extract spectrum features, so as to obtain the spectrum features of the sample speech.

4. The voice conversion method according to claim 3, characterized in that: The spectrum coding sub-model includes a fundamental frequency extraction sub-module and an energy extraction sub-module; the step of inputting the sample speech into the spectrum coding sub-model to extract spectrum features to obtain the spectrum features of the sample speech includes: Inputting the sample speech into the fundamental frequency extraction submodule to extract frequency features to obtain fundamental frequency features of the sample speech; and The sample speech is input into the energy extraction submodule to extract energy features, so as to obtain energy features of the sample speech; the fundamental frequency features and the energy features are the spectrum features of the sample speech.

5. The voice conversion method according to claim 3, characterized in that: The reference coding sub-model includes a convolutional layer, a gated recurrent layer, a linear layer and an activation layer; the sample speech is input into the reference coding sub-model to extract prosodic features to obtain phoneme-level prosodic features, including: Inputting the sample speech into the convolution layer for feature extraction to obtain a speech spectrum of the sample speech; Inputting the speech spectrum of the sample speech into the gated recurrent layer for processing to obtain the time series features of the sample speech; Inputting the temporal features of the sample speech into the linear layer for mapping processing to obtain distribution vectors of the sample speech at different frequencies; The distribution vectors of the sample speech at different frequencies are input into the activation layer for processing to obtain the prosodic features at the phoneme level.

6. The speech conversion method according to any one of claims 1 to 5, characterized in that: Get the speaker vector of the set timbre, including: Get the voice data recorded by the user with set timbre; Extracting timbre feature data and prosody feature data from the speech data; The speaker vector is obtained according to the timbre feature data and the prosody feature data.

7. The speech conversion method according to any one of claims 1 to 5, characterized in that: The training method of the prosody encoder comprises: Iteratively training a preset reference coding sub-model through a first training sample set until the reference coding sub-model converges to obtain a trained reference coding sub-model; and Iteratively training a preset spectrum coding sub-model through a second training sample set until the spectrum coding sub-model converges to obtain a trained spectrum coding sub-model; The trained reference coding sub-model is concatenated with the trained spectral coding sub-model to obtain a pre-established prosody encoder.

8. A speech conversion device, characterized in that: The voice conversion device comprises: A data enhancement module, used for performing data enhancement on the speech to be converted to obtain a sample speech whose length is greater than that of the speech to be converted; A first processing module is used to input the sample speech into a pre-established prosody encoder for processing to obtain phoneme-level prosody features and spectral features of the sample speech; and A second processing module is used to input the sample speech into a pre-established content encoder for processing to obtain content features of the sample speech; A rhythm alignment module, used for inputting the rhythm feature and the content feature into an attention-based alignment model for alignment processing to obtain an aligned rhythm feature; A feature synthesis module, used for inputting the aligned prosodic features, the content features, the spectral features and the speaker vector of the set timbre into a pre-established decoder for synthesis to obtain a synthesized speech; The speech interception module is used to intercept the partial content corresponding to the speech to be converted from the synthesized speech as the target speech.

9. A computer device, characterized in that: The computer device comprises a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the speech conversion method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the speech conversion method according to any one of claims 1 to 7 is implemented.

Images