Voice conversion method, apparatus, medium, and device
By decoupling the learning of timbre, rhythm, and pitch through a pre-trained speech conversion model, the problem that rhythm and pitch are not considered in existing speech conversion technologies is solved, and high-quality speech conversion results are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN RAISOUND TECH
- Filing Date
- 2025-08-27
- Publication Date
- 2026-07-07
AI Technical Summary
Existing speech conversion methods fail to effectively consider the rhythm and pitch related to speech prosody, resulting in the converted speech losing its naturalness and expressiveness.
By using a pre-trained speech conversion model, audio representations of the source speaker's original speech and the target speaker's target speech are extracted respectively. Then, decoupling learning is performed using a common classifier and an adversarial classifier to achieve deentanglement of timbre, rhythm, and pitch, and the target audio is synthesized.
It maintains the naturalness and expressiveness of the converted speech, improves the naturalness and comprehensibility of speech conversion, and enhances robustness.
Smart Images
Figure CN120748418B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of speech conversion technology, and in particular to a speech conversion method, apparatus, medium and device. Background Technology
[0002] Speech information can be broken down into four parts: content, timbre, pitch, and rhythm. The linguistic content of speech includes the main information contained within the speech sound and can also be transcribed into text. Timbre contains information about the speaker's vocal characteristics, which are closely related to the speaker's identity. Pitch and rhythm are two major components of prosody, expressing the speaker's emotions. Pitch variations convey various aspects of the speaker's tone, while rhythm characterizes the speed at which the speaker pronounces each word or syllable.
[0003] Current speech conversion methods only consider the decoupling of content and timbre representations, neglecting rhythm and pitch representations related to speech prosody. This leads to the leakage of pitch and rhythm-related information into the timbre. This leakage may affect the structure of the converted speech, causing it to lose the naturalness and expressiveness of the source speech.
[0004] Therefore, there is an urgent need for a new speech conversion method that can convert timbre, rhythm and / or pitch during speech conversion. Summary of the Invention
[0005] In view of the above problems, the present invention is proposed to provide a speech conversion method, apparatus, medium and device that overcomes or at least partially solves the above problems.
[0006] Other features and advantages of the invention will become apparent from the following detailed description, or may be learned in part by practice of the invention.
[0007] According to a first aspect of the present invention, a speech conversion method is provided, the speech conversion method comprising:
[0008] Receive a voice conversion instruction; the voice conversion instruction includes the voice conversion type, the original voice, and the target voice.
[0009] A pre-trained speech conversion model is obtained, and task data is determined for input to different encoders in the speech conversion model according to the speech conversion type. The input task data is then encoded by the encoders in the speech conversion model to obtain the encoding features output by each encoder. The task data is extracted based on the original speech or the target speech.
[0010] The encoded features corresponding to the task data extracted from the original speech are input into the common classifier of the speech conversion model to obtain the common features corresponding to the encoded features output by the common classifier. At the same time, the encoded features corresponding to the task data extracted from the target speech are input into the adversarial classifier of the speech conversion model to obtain the adversarial features corresponding to the encoded features output by the adversarial classifier.
[0011] The common features and the adversarial features are input into the speech decoder in the speech conversion model for decoding to obtain the target Mel spectrum; at the same time, the common features and / or the adversarial features are input into the pitch decoder in the speech conversion model to obtain the target pitch contour lines;
[0012] The target audio is synthesized using a vocoder based on the target Mel spectrogram and the target pitch contour lines.
[0013] According to a second aspect of the present invention, a speech conversion apparatus is provided, the speech conversion apparatus comprising:
[0014] The instruction receiving module is used to receive speech conversion instructions; the speech conversion instructions include speech conversion type, original speech, and target speech.
[0015] The feature encoding module is used to acquire a pre-trained speech conversion model and determine the task data input to different encoders in the speech conversion model according to the speech conversion type, so as to encode the input task data through the encoders in the speech conversion model to obtain the encoded features output by each encoder; the task data is extracted based on the original speech or the target speech;
[0016] The feature classification module is used to input the encoded features corresponding to the task data extracted from the original speech into the common classifier of the speech conversion model to obtain the common features corresponding to the encoded features output by the common classifier. At the same time, it inputs the encoded features corresponding to the task data extracted from the target speech into the adversarial classifier of the speech conversion model to obtain the adversarial features corresponding to the encoded features output by the adversarial classifier.
[0017] The feature decoding module is used to input the common features and the adversarial features into the speech decoder in the speech conversion model for decoding processing to obtain the target Mel spectrum; at the same time, the common features and / or the adversarial features are input into the pitch decoder in the speech conversion model to obtain the target pitch contour lines;
[0018] An audio synthesis module is used to synthesize target audio using a vocoder based on the target Mel spectrogram and the target pitch contour lines.
[0019] According to a third aspect of the present invention, a computer-readable storage medium is provided, wherein computer program instructions are stored therein, the computer program instructions being loaded and executed by a processor to perform the operations performed by the method described in any of the preceding claims.
[0020] According to a fourth aspect of the present invention, an electronic device is provided, including a processor and a memory, the memory storing computer program instructions executable by the processor, wherein when the processor executes the computer program instructions, it implements the instructions of any of the methods described above.
[0021] The technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages:
[0022] This invention provides a speech conversion method, apparatus, medium, and device. The method uses a pre-trained speech conversion model to extract corresponding audio representations from the original speech of the source speaker and the target speech of the target speaker according to the speech conversion type. Different representation styles are transmitted in a single speech conversion using the speech conversion model. That is, according to the speech conversion type, speech conversion of timbre or timbre + pitch is realized respectively. This achieves deentanglement learning of content, timbre, rhythm, and pitch representation, so that the converted speech retains the naturalness and expressiveness of the source speech.
[0023] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description
[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0025] Figure 1 A schematic flowchart of a speech conversion method provided in an embodiment of the present invention;
[0026] Figure 2 This is a schematic diagram of the model structure for a content encoder.
[0027] Figure 3 This is a schematic diagram of the model structure of a timbre encoder;
[0028] Figure 4 This is a schematic diagram of the architecture of a speech conversion model.
[0029] Figure 5 A schematic diagram of the model structure for a common classifier;
[0030] Figure 6 A schematic diagram of the model structure for an adversarial classifier;
[0031] Figure 7 This is a model reference diagram for a speech decoder;
[0032] Figure 8 This is a model reference diagram for a pitch decoder;
[0033] Figure 9 A schematic diagram illustrating the process of pre-training the speech conversion model;
[0034] Figure 10 This is a schematic diagram of the principle structure of a speech conversion device provided in an embodiment of the present invention. Detailed Implementation
[0035] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings.
[0036] The accompanying drawings illustrate various structural schematics according to embodiments of the present disclosure. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.
[0037] In the context of this disclosure, when a layer / component is referred to as being "above" another layer / component, that layer / component may be directly above the other layer / component, or there may be an intermediate layer / component between them. Additionally, if a layer / component is "above" another layer / component in one orientation, then when the orientation is reversed, that layer / component may be "below" the other layer / component. In the context of this disclosure, similar or identical components may be denoted by the same or similar reference numerals.
[0038] To better understand the above technical solutions, the following will describe the above technical solutions in detail with reference to specific implementation methods. It should be understood that the embodiments of this disclosure and the specific features in the embodiments are detailed descriptions of the technical solutions of the present invention, rather than limitations on the technical solutions of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.
[0039] Figure 1 This is a flowchart illustrating a speech conversion method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the speech conversion method includes the following steps:
[0040] S1. Receive a voice conversion instruction; the voice conversion instruction includes the voice conversion type, the original voice, and the target voice;
[0041] In this embodiment of the invention, the original speech is speech data obtained from the source speaker, and the target speech is the speech data of the target speaker that needs to be converted. The speech conversion type includes timbre or timbre + pitch, that is, the audio feature type for which the target speech needs to be converted is either timbre only or timbre and pitch are converted simultaneously.
[0042] S2. Obtain the pre-trained speech conversion model, and determine the task data input to different encoders in the speech conversion model according to the speech conversion type, so as to encode the input task data through the encoders in the speech conversion model to obtain the encoding features output by each encoder; the task data is extracted based on the original speech or the target speech;
[0043] In this embodiment of the invention, the pre-trained speech conversion model is used to extract audio representations of the original speech of the source speaker and the target speech of the target speaker, and to synthesize the target audio based on the pre-trained audio representations.
[0044] The speech conversion model includes at least a content encoder, a timbre encoder, a rhythm encoder, a pitch encoder, a speech decoder, and a pitch decoder. The content encoder, timbre encoder, rhythm encoder, and pitch encoder are used to encode the collected speech data of the speaker (including the source speaker and the target speaker) and extract the corresponding audio representation. The outputs of the content encoder, timbre encoder, rhythm encoder, and pitch encoder are used as the inputs of the speech decoder to attempt to reconstruct the input Mel spectrogram. The outputs of the rhythm encoder and pitch encoder are used as the inputs of the pitch encoder to finally decode and attempt to reconstruct the input normalized pitch contour lines. The content encoder, timbre encoder, rhythm encoder, pitch encoder, speech decoder, and pitch decoder are jointly trained during the training phase to minimize reconstruction loss.
[0045] Specifically, in this embodiment of the invention, the task data input into the speech conversion model is encoded using different encoders. For example, the task data is encoded using the content encoder to extract the content code; the task data is encoded using the timbre encoder to extract the timbre code; the task data is encoded using the rhythm encoder to extract the rhythm code; and the task data is encoded using the pitch encoder to extract the pitch code.
[0046] In this embodiment of the invention, determining the task data input to different encoders in the speech conversion model based on the speech conversion type includes: when the speech conversion type is a timbre conversion type, the task data input to different encoders in the speech conversion model is the Mel spectrum of the target speech and the Mel spectrum and pitch contour lines of the original speech; when the speech conversion type is both a timbre conversion type and a pitch conversion type, the task data input to different encoders in the speech conversion model is the Mel spectrum of the target speech, the pitch contour lines, and the Mel spectrum of the original speech.
[0047] Accordingly, when the speech conversion type is timbre conversion, the Mel spectrum of the target speech is input into the timbre encoder in the speech conversion model for encoding, and the resulting encoding feature is timbre encoding; and the Mel spectrum and pitch contour lines of the original speech are input into the content encoder, rhythm encoder, and pitch encoder in the speech conversion model for encoding, and the resulting encoding features are content encoding, rhythm encoding, and pitch encoding, respectively; when the speech conversion type is timbre conversion and pitch conversion, the Mel spectrum and pitch contour lines of the target speech are input into the timbre encoder and pitch encoder in the speech conversion model for encoding, and the resulting encoding features are timbre encoding and pitch encoding, respectively; and the Mel spectrum of the original speech is input into the content encoder and rhythm encoder in the speech conversion model for encoding, and the resulting encoding features are content encoding and rhythm encoding, respectively.
[0048] The content encoder and timbre encoder are based on the AdaIN-VC (Activation Guidance and Adaptive Instance Normalization Voice Conversion) method. AdaIN-VC is a speech conversion method that combines activation guidance and adaptive instance normalization. Figure 2-3 As shown, Figure 2 This is a schematic diagram of the model structure for a content encoder. Figure 3The diagram below is a reference schematic of the model structure of the timbre encoder. The content encoder and the timbre encoder process all frequency information by using a one-dimensional convolutional layer and an average pooling layer to process time frame information. Both the content encoder and the timbre encoder utilize ConvBank to expand the receptive field and capture long-term information.
[0049] The rhythm encoder and pitch encoder have the same structure, consisting of a set of 5×1 convolutional layers, which are normalized. The final output of the convolutional layers is input to a set of bidirectional LSTM layers to reduce the feature dimension, and then the time dimension is reduced by downsampling, thereby generating a hidden rhythm representation or pitch representation.
[0050] The representations corresponding to the content encoding, timbre encoding, rhythm encoding, and pitch encoding can be expressed by the following formula:
[0051] For content encoder E c Extracted content encoding Z c for:
[0052] Z c =E c (S);
[0053] For the tone encoder E s The timbre code Z was extracted. s for:
[0054] Z s =E s (S);
[0055] For rhythm encoder E r The rhythm code Z is extracted. r for:
[0056] Z r =E r ( s );
[0057] For pitch encoder E p The pitch code z is extracted. p for:
[0058] Z p =E p (P).
[0059] In this embodiment of the invention, the content encoding, timbre encoding, rhythm encoding, and pitch encoding are all hidden representations (hidden vectors) or unentangled representations.
[0060] S3. Input the encoding features corresponding to the task data extracted from the original speech into the common classifier of the speech conversion model to obtain the common features corresponding to the encoding features output by the common classifier. At the same time, input the encoding features corresponding to the task data extracted from the target speech into the adversarial classifier of the speech conversion model to obtain the adversarial features corresponding to the encoding features output by the adversarial classifier.
[0061] See Figure 4 As shown, this is a schematic diagram of the architecture of the speech conversion model. After obtaining the target output representation set, this embodiment of the invention further includes: calculating the mutual information of any two audio representations in the target output representation set; and obtaining the corresponding loss function by using the variational comparison log ratio upper limit of the mutual information.
[0062] The embodiments of the present invention determine the correlation between audio representations by calculating the mutual information (MI) of each audio representation and minimize the correlation between each audio representation, so that the separated or decoupled audio representations are as unrelated or have no overlapping components as possible, thereby separating irrelevant information from each other in the acquired speaker's speech data.
[0063] For example, suppose the speaker (or speaker information) u is only related to the timbre code Z s Related to, and related to rhythm encoding Z r Pitch encoding Z p and content encoding Z c Irrelevant, in order to represent speaker-irrelevant audio {Z r Z p Z c The speaker-related audio representation Z s Separation, that is, minimizing f(Z) r Z p Z c ,u) and maximizing f(Z) s ,u).
[0064] For example, minimizing f(Z) r Z p Z c ,u) is characterized as:
[0065] min(f(Z r Z p Z c ,u));
[0066] For example, maximizing f(Z) s ,u) is characterized as:
[0067] max(f(Z s ,u)).
[0068] To achieve the minimization of f(Z) corresponding to the above formula r Z p Z c ,u) and maximizing f(Z) s To achieve the objective of u), this embodiment of the invention employs a common classifier C1 and an adversarial classifier C2, respectively; see reference. Figure 5-6 As shown, Figure 5 This is a schematic diagram of the model structure for a common classifier. Figure 6 The diagram below is a reference illustration of the model structure of the adversarial classifier. The common classifier C1 is used to extract features related to speaker identity (such as timbre), and the adversarial classifier C2 is an adversarial classifier based on gradient inversion layer (GRL) to extract speaker-irrelevant information (content, pitch, and rhythm) to eliminate speaker information in the latent space.
[0069] S4. Input the common features and the adversarial features into the speech decoder in the speech conversion model for decoding processing to obtain the target Mel spectrum; at the same time, input the common features and / or the adversarial features into the pitch decoder in the speech conversion model to obtain the target pitch contour lines;
[0070] See Figure 7 The image shown is a model reference diagram of a speech decoder, wherein the speech decoder D... s The common features and adversarial features are used as inputs, and the target Mel spectrum is output. Characterized as:
[0071]
[0072] See Figure 8 The image shown is a model reference diagram of a pitch decoder, specifically the pitch decoder D. p The common features and / or adversarial features will be used as input to finally decode and generate normalized target pitch contour lines.
[0073] S5. Based on the target Mel spectrogram and the target pitch contour lines, the target audio is synthesized using a vocoder.
[0074] After obtaining the target Mel spectrogram and the target pitch contour, embodiments of the present invention can use a vocoder to synthesize the target audio based on the target Mel spectrogram and the target pitch contour.
[0075] Before receiving the voice conversion command, please refer to Figure 9 As shown, the speech conversion model obtained through pre-training in this embodiment of the invention specifically includes the following steps:
[0076] S61. Obtain a sample speech set; the sample speech set includes at least one sample speech; each sample speech corresponds to a sample label, a sample Mel spectrum, and a sample pitch contour;
[0077] S62. Obtain an initial conversion model containing initial parameters, wherein the initial conversion model includes a rhythm encoder, a content encoder, a timbre encoder, a pitch encoder, a speech decoder, and a pitch decoder.
[0078] S63. Input the sample Mel spectrum to the rhythm encoder, content encoder and timbre encoder to obtain the sample rhythm features, sample content features and sample timbre features output by the rhythm encoder, respectively. At the same time, input the sample pitch contour lines to the pitch encoder to obtain the sample pitch features.
[0079] S64. Input the sample rhythm features, sample content features, and sample pitch features into the common classifier in the initial conversion model to obtain the classification rhythm features, classification content features, and classification pitch features output by the common classifier. At the same time, input the sample timbre features into the adversarial classifier in the initial conversion model to obtain the classification timbre features output by the adversarial classifier.
[0080] S65. Input the classification rhythm feature, classification content feature, classification pitch feature, and classification timbre feature into the speech decoder to obtain the decoded Mel spectrum output by the speech decoder; simultaneously, input the sample rhythm feature and the sample pitch feature into the pitch decoder to obtain the decoded pitch contour lines output by the pitch decoder.
[0081] S66. Obtain the classification loss value corresponding to the common classifier and the adversarial loss value output by the adversarial classifier, and generate the prediction loss value based on the sample Mel spectrum, sample pitch contour, decoded Mel spectrum and decoded pitch contour.
[0082] S67. Generate a model loss value based on the classification loss value, adversarial loss value, and prediction loss value. If the model loss value does not meet the preset conditions, iteratively update the initial parameters until the model loss value meets the preset conditions. Then, record the initial conversion model as the speech conversion model.
[0083] In step S61, the sample speech set consists of one or more sample speech samples. These sample speech samples can originate from multiple sample speech samples produced by the same speaker, or from one or more sample speech samples produced by multiple different speakers. The sample labels are used to improve the accuracy of model training using the sample speech samples. The sample Mel-spectrum represents the speech features of the sample speech samples, and the sample pitch contour lines represent the pitch information of the sample speech samples.
[0084] In step S62, the initial conversion model includes a rhythm encoder, a content encoder, a timbre encoder, a pitch encoder, a speech decoder, a pitch decoder, a common classifier, and an adversarial classifier. The initial conversion model has an initial parameter that is preset and iteratively updated based on the output of the initial conversion model for sample speech during training.
[0085] In step S63, the rhythm encoder, content encoder, and timbre encoder all use the sample Mel spectrum as input features. The rhythm encoder extracts the sample rhythm features from the sample Mel spectrum, the content encoder extracts the text content of the sample speech (converting speech features into text features), and the timbre encoder extracts the sample timbre features representing the speaker from the sample Mel spectrum. The pitch encoder uses sample pitch contour lines as input features, extracting sample pitch features reflecting the speaker's volume and intensity from the pitch contour lines.
[0086] In step S64, the common classifier in this embodiment is used to minimize the influence of sample rhythm features, sample content features, and sample pitch features on the model's speaker prediction. The adversarial classifier is used to maximize the influence of sample timbre features on the model's speaker prediction. In this way, the features input to the speech decoder and pitch decoder are different from the features output by each encoder, that is, the role of sample timbre features is strengthened, while the role of sample rhythm features, sample content features, and sample pitch features is weakened, thereby improving the accuracy of model prediction.
[0087] In step S65, the output of the speech decoder attempts to reconstruct the Mel spectrogram of the input, thus obtaining the decoded Mel spectrogram. The output of the pitch decoder attempts to reconstruct the normalized pitch contour lines of the input, thus obtaining the decoded pitch contour lines. The decoder and encoder are jointly trained to minimize reconstruction loss.
[0088] In step S66, the embodiment of the present invention obtains the classification loss value corresponding to the common classifier and the adversarial loss value output by the adversarial classifier by: inputting the sample rhythm features, sample content features and sample pitch features into the common classifier in the initial conversion model, and then obtaining the classification loss value corresponding to the common classifier based on the classification loss function of the common classifier; and inputting the sample timbre features into the adversarial classifier in the initial conversion model, and then obtaining the adversarial loss value output by the adversarial classifier based on the adversarial loss function of the adversarial classifier.
[0089] Wherein, the classification loss function L corresponding to the common classifier com-cls for:
[0090]
[0091] In the formula, |(·) represents the indicator function, K represents the number of sample speakers, u represents the sample speaker that produced the sample speech x, and p k The probability of speaker k in the sample is represented. The initial parameters characterize the rhythm encoder, content encoder, and pitch encoder in the initial conversion model. The initial parameters characterize the common classifier in the initial transformation model.
[0092] The adversarial loss function L corresponding to the adversarial classifier adv-cls for:
[0093]
[0094] In the formula, |(·) represents the indicator function, K represents the number of sample speakers, u represents the sample speaker that produced the sample speech x, and p k The probability of speaker k in the sample is represented. The initial parameters characterize the timbre encoder in the initial conversion model. The initial parameters characterize the adversarial classifier in the initial transformation model.
[0095] In this embodiment of the invention, the preset condition is, for example, a model loss value (e.g., 0.0001) and / or the number of model training iterations. When the model loss value or the number of model training iterations of the initial conversion model does not reach the preset condition, steps S61-S67 above will be repeated to continuously update the initial parameters until the preset condition is reached, thereby obtaining the trained speech conversion model. In other embodiments of the invention, the preset condition may also be other limiting conditions, and this embodiment of the invention does not limit this.
[0096] The speech conversion method described in this invention has the following advantages compared to the prior art:
[0097] 1. Using a pre-trained speech conversion model, corresponding audio representations are extracted from the original speech of the source speaker and the target speech of the target speaker according to the speech conversion type. Different representation styles are transmitted in a single speech conversion using the speech conversion model. That is, according to the speech conversion type, timbre or timbre + pitch speech conversion is realized respectively. This achieves deentanglement learning of content, timbre, rhythm and pitch representation (the deentanglement learning refers to the use of deep learning technology in the field of speech processing to decouple different features in the speech signal, thereby obtaining a clearer and more useful speech representation), so that the converted speech retains the naturalness and expressiveness of the source speech.
[0098] 2. Based on the mutual information, the corresponding loss function is obtained by using a common classifier and an adversarial classifier. The correlation between each audio representation is determined by the mutual information (MI) of each audio representation. The obtained loss function is used to minimize the correlation between each audio representation, so that the separated or decoupled audio representations are as unrelated as possible or have no overlapping components, which improves the naturalness and intelligibility of one-time speech conversion. Adversarial mutual information learning improves the performance and robustness of unentanglement learning.
[0099] 3. The content encoder, timbre encoder, rhythm encoder, pitch encoder, speech decoder, and pitch decoder in the speech conversion model are jointly trained during the training phase to minimize the reconstruction loss of the Mel spectrogram and pitch contour lines.
[0100] In one specific embodiment, the speech conversion method proposed in this invention can be applied to edge devices. Specifically, after lightweighting the speech conversion model proposed in this invention, the lightweighted speech conversion model is deployed on the edge device to perform speech conversion tasks, thereby enabling the edge device to perform more speech functions. Furthermore, when edge devices are built on in-memory computing chips, higher demands are placed on the space occupied and processing speed of the components or chips on the edge device. By incorporating the lightweight speech conversion model of this invention, the model inference speed can be improved.
[0101] Based on the above embodiments, as a supplement to the above... Figure 1 The present invention provides an embodiment of a speech conversion device to implement the method shown, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices, see reference. Figure 10 As shown, the voice conversion device includes:
[0102] The instruction receiving module 100 is used to receive speech conversion instructions; the speech conversion instructions include speech conversion type, original speech and target speech;
[0103] The feature encoding module 200 is used to acquire a pre-trained speech conversion model and determine the task data input to different encoders in the speech conversion model according to the speech conversion type, so as to encode the input task data through the encoders in the speech conversion model to obtain the encoded features output by each encoder; the task data is extracted based on the original speech or the target speech;
[0104] The feature classification module 300 is used to input the encoded features corresponding to the task data extracted from the original speech into the common classifier of the speech conversion model to obtain the common features corresponding to the encoded features output by the common classifier, and simultaneously input the encoded features corresponding to the task data extracted from the target speech into the adversarial classifier of the speech conversion model to obtain the adversarial features corresponding to the encoded features output by the adversarial classifier.
[0105] The feature decoding module 400 is used to input the common features and the adversarial features into the speech decoder in the speech conversion model for decoding processing to obtain the target Mel spectrum; at the same time, the common features and / or the adversarial features are input into the pitch decoder in the speech conversion model to obtain the target pitch contour lines;
[0106] The audio synthesis module 500 is used to synthesize target audio using a vocoder based on the target Mel spectrogram and the target pitch contour lines.
[0107] The speech conversion device described in this embodiment can execute the speech conversion method provided in the above embodiment. The speech conversion device has the corresponding functional steps and beneficial effects of the speech conversion method described in the above embodiment. For details, please refer to the embodiments of the above speech conversion method. This embodiment will not be repeated here.
[0108] This invention also provides an electronic device, which may include a processor and a memory, wherein the processor and memory can be connected via a bus or other means. The processor may be a Central Processing Unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof. The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the speech conversion method in this invention embodiment. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the speech conversion method in the above method embodiments.
[0109] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. The one or more modules are stored in the memory and, when executed by the processor, perform the speech conversion method as described in the above method embodiments. Specific details of the above electronic device can be understood by referring to the corresponding descriptions and effects in the above method embodiments, and will not be repeated here. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it may include the processes of the embodiments of the above methods. The storage medium may be a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD), etc.; the storage medium may also include a combination of the above types of memory.
[0110] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.
[0111] Similarly, it should be understood that, in order to simplify this disclosure and aid in understanding one or more of the various aspects of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of the invention above. Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and it should be noted that the above embodiments are illustrative of the invention and not restrictive, and that alternative embodiments can be devised by those skilled in the art without departing from its scope.
Claims
1. A speech conversion method, characterized in that, The speech conversion method includes: Receive a voice conversion instruction; the voice conversion instruction includes the voice conversion type, the original voice, and the target voice. A pre-trained speech conversion model is obtained, and task data is determined for input to different encoders in the speech conversion model according to the speech conversion type. The input task data is then encoded by the encoders in the speech conversion model to obtain the encoding features output by each encoder. The task data is extracted based on the original speech or the target speech. The encoded features corresponding to the task data extracted from the original speech are input into the common classifier of the speech conversion model to obtain the common features corresponding to the encoded features output by the common classifier. At the same time, the encoded features corresponding to the task data extracted from the target speech are input into the adversarial classifier of the speech conversion model to obtain the adversarial features corresponding to the encoded features output by the adversarial classifier. The common features and the adversarial features are input into the speech decoder in the speech conversion model for decoding to obtain the target Mel spectrum; at the same time, the common features and / or the adversarial features are input into the pitch decoder in the speech conversion model to obtain the target pitch contour lines; The target audio is synthesized using a vocoder based on the target Mel spectrogram and the target pitch contour lines.
2. The speech conversion method according to claim 1, characterized in that, The task data for determining the input to different encoders in the speech conversion model based on the speech conversion type includes: When the speech conversion type is timbre conversion type, the task data input to different encoders in the speech conversion model is determined to be the Mel spectrum of the target speech and the Mel spectrum and pitch contour lines of the original speech; When the speech conversion type is timbre conversion type and pitch conversion type, the task data input to different encoders in the speech conversion model is determined to be the Mel spectrum of the target speech, the pitch contour lines, and the Mel spectrum of the original speech.
3. The speech conversion method according to claim 2, characterized in that, The process of encoding the input task data using the encoder in the speech conversion model to obtain the encoded features output by each encoder includes: When the speech conversion type is timbre conversion type, the Mel spectrum of the target speech is input into the timbre encoder in the speech conversion model for encoding, and the resulting encoding feature is timbre encoding; and the Mel spectrum and pitch contour lines of the original speech are input into the content encoder, rhythm encoder and pitch encoder in the speech conversion model for encoding respectively, and the resulting encoding features are content encoding, rhythm encoding and pitch encoding respectively. When the speech conversion type is timbre conversion type and pitch conversion type, the Mel spectrum and pitch contour lines of the target speech are input into the timbre encoder and pitch encoder in the speech conversion model for encoding, and the resulting encoding features are timbre encoding and pitch encoding, respectively; and the Mel spectrum of the original speech is input into the content encoder and rhythm encoder in the speech conversion model for encoding, and the resulting encoding features are content encoding and rhythm encoding, respectively.
4. The speech conversion method according to claim 1, characterized in that, Before receiving a voice conversion instruction, the method includes: Obtain a sample speech set; the sample speech set includes at least one sample speech; each sample speech corresponds to a sample label, a sample Mel spectrum, and a sample pitch contour; Obtain an initial conversion model containing initial parameters, wherein the initial conversion model includes a rhythm encoder, a content encoder, a timbre encoder, a pitch encoder, a speech decoder, and a pitch decoder; The sample Mel spectrum is input to the rhythm encoder, content encoder and timbre encoder to obtain the sample rhythm features, sample content features and sample timbre features output by the rhythm encoder, respectively. At the same time, the sample pitch contour lines are input to the pitch encoder to obtain the sample pitch features. The sample rhythm features, sample content features, and sample pitch features are input into the common classifier in the initial conversion model to obtain the classification rhythm features, classification content features, and classification pitch features output by the common classifier. At the same time, the sample timbre features are input into the adversarial classifier in the initial conversion model to obtain the classification timbre features output by the adversarial classifier. The classification rhythm features, classification content features, classification pitch features, and classification timbre features are input into the speech decoder to obtain the decoded Mel spectrum output by the speech decoder; simultaneously, the sample rhythm features and the sample pitch features are input into the pitch decoder to obtain the decoded pitch contour lines output by the pitch decoder. Obtain the classification loss value corresponding to the common classifier and the adversarial loss value output by the adversarial classifier, and generate the prediction loss value based on the sample Mel spectrum, sample pitch contour, decoded Mel spectrum and decoded pitch contour. A model loss value is generated based on the classification loss value, adversarial loss value, and prediction loss value. If the model loss value does not meet the preset conditions, the initial parameters are iteratively updated until the model loss value meets the preset conditions. Then, the initial conversion model is recorded as the speech conversion model.
5. The speech conversion method according to claim 4, characterized in that, The process of obtaining the classification loss value corresponding to the common classifier and the adversarial loss value output by the adversarial classifier includes: After inputting the sample rhythm features, sample content features, and sample pitch features into the common classifier in the initial conversion model, the classification loss value corresponding to the common classifier is obtained based on the classification loss function of the common classifier. After inputting the sample timbre features into the adversarial classifier in the initial conversion model, the adversarial loss value output by the adversarial classifier is obtained based on the adversarial loss function of the adversarial classifier.
6. The speech conversion method according to claim 5, characterized in that, The classification loss function L corresponding to the common classifier com-cls for: In the formula, |(·) represents the indicator function, K represents the number of sample speakers, u represents the sample speaker that produced the sample speech x, and p k The probability of speaker k in the sample is represented. The initial parameters characterize the rhythm encoder, content encoder, and pitch encoder in the initial conversion model. The initial parameters characterize the common classifier in the initial transformation model.
7. The speech conversion method according to claim 5, characterized in that, The adversarial loss function L corresponding to the adversarial classifier adv-cls for: In the formula, |(·) represents the indicator function, K represents the number of sample speakers, u represents the sample speaker that produced the sample speech x, and p k The probability of speaker k in the sample is represented. The initial parameters characterize the timbre encoder in the initial conversion model. The initial parameters characterize the adversarial classifier in the initial transformation model.
8. A speech conversion device, applied to the method described in any one of claims 1-7, characterized in that, The speech conversion device includes: The instruction receiving module is used to receive speech conversion instructions; the speech conversion instructions include speech conversion type, original speech, and target speech. The feature encoding module is used to acquire a pre-trained speech conversion model and determine the task data input to different encoders in the speech conversion model according to the speech conversion type, so as to encode the input task data through the encoders in the speech conversion model to obtain the encoded features output by each encoder; the task data is extracted based on the original speech or the target speech; The feature classification module is used to input the encoded features corresponding to the task data extracted from the original speech into the common classifier of the speech conversion model to obtain the common features corresponding to the encoded features output by the common classifier. At the same time, it inputs the encoded features corresponding to the task data extracted from the target speech into the adversarial classifier of the speech conversion model to obtain the adversarial features corresponding to the encoded features output by the adversarial classifier. The feature decoding module is used to input the common features and the adversarial features into the speech decoder in the speech conversion model for decoding processing to obtain the target Mel spectrum; at the same time, the common features and / or the adversarial features are input into the pitch decoder in the speech conversion model to obtain the target pitch contour lines; An audio synthesis module is used to synthesize target audio using a vocoder based on the target Mel spectrogram and the target pitch contour lines.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that are loaded and executed by a processor to perform the operations described in any one of claims 1-7.
10. An electronic device comprising a processor and a memory, characterized in that, The memory stores computer program instructions that can be executed by the processor, and when the processor executes the computer program instructions, it implements the instructions of the method as described in any one of claims 1-7.