Voice attribute editing method and device
By combining variational autoencoders and stream matching models with adversarial learning and cyclic consistency training, the fidelity and consistency issues in speech attribute editing are solved, improving the naturalness and attribute consistency of speech editing and achieving high-quality speech attribute changes.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-24
- Publication Date
- 2026-03-27
AI Technical Summary
Existing technologies have a gap in fidelity between the generated speaker-related representations and the speaker representations extracted from real speech in speech attribute editing, and there is a mismatch between the training and inference phases, which leads to speech quality degradation in editing scenarios.
A variational autoencoder structure is used to extract attribute-independent residual embeddings from the original speaker representation. Speaker-specific Mel spectra are generated through a flow matching model. Adversarial learning and recurrent consistency training are combined to reconstruct speech attributes to improve naturalness and attribute consistency.
It improves the naturalness and clarity of edited speech, enhances the ability to preserve non-edited attributes, and achieves higher accuracy in attribute editing and speech quality.
Smart Images
Figure CN121747589A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of speech processing and generation technology, and in particular to a method and apparatus for editing speech attributes. Background Technology
[0002] Speech attribute editing aims to modify target attributes (such as age and gender) without altering the semantic content of the speech or the features of non-target speakers. Traditional low-level feature transformation methods based on fundamental frequency and formants are prone to distortion and metallic sound; while editing at the Mel spectrum level based on generative adversarial networks can alleviate some artifacts, it is often accompanied by a decrease in intelligibility. In addition, existing "content-speaker" decoupling frameworks still have two key problems: first, there is a fidelity gap between the generated speaker-related representations and the speaker representations extracted from real speech; second, the use of the original speaker representation during the training phase and the switch to the edited representation during the inference phase leads to a "training-inference mismatch" in the reconstruction model, resulting in degradation in editing scenarios. Summary of the Invention
[0003] To address the problems existing in the prior art, embodiments of this application provide a method, apparatus, computing device, computer storage medium, and product containing a computer program for editing voice attributes, which can improve the naturalness, clarity, and attribute consistency of the edited voice, and enhance the ability to retain non-edited attributes.
[0004] In a first aspect, embodiments of this application provide a speech attribute editing method, comprising: decoupling the input raw speech to obtain semantic features and the original speaker representation; constructing a speaker representation editing module, extracting attribute-independent residual embeddings from the original speaker representation using a variational autoencoder structure, and reconstructing the edited speaker embedding based on the residual embeddings and continuous attribute labels; generating a speaker-independent average acoustic representation based on the semantic features and quantized attribute labels; generating a speaker-specific Mel spectrum based on the average acoustic representation and the edited speaker embedding using a stream matching model; and converting the generated Mel spectrum into the final speech waveform using a pre-trained vocoder.
[0005] In some possible implementations, the decoupling of the input raw speech to obtain semantic features and the original speaker representation includes: using a pre-trained speech recognition model as a semantic encoder to extract the semantic features, while keeping the parameters frozen during training; and using a speaker verification model as a speaker encoder to extract the original speaker representation, wherein the batch normalization layer in the speaker encoder is replaced with a group normalization layer.
[0006] In some possible implementations, a voice tone converter based on a speech conversion model is used to perturb the input speech before it is input into the semantic encoder, thereby enhancing the separation effect between speaker features and semantic features.
[0007] In some possible implementations, the generated speaker-independent average acoustic representation and the generated speaker-specific Mel spectrum together constitute a two-stage speech reconstruction module, the loss function of which is defined as:
[0008]
[0009] In the formula, Characterization flow matching loss, The L2 distance characterizes the average acoustic representation and the true Mel spectrum.
[0010] In some possible implementations, the training process of the speaker representation editing module includes: applying a bounded penalty loss to the extracted residual embedding, the loss function being defined as: In the formula, The vector representing the residual embedding is used; a contrastive learning loss is introduced to enhance the speaker discrimination ability of the residual embedding; an adversarial learning loss is introduced to align the generated speaker embedding with the distribution of the real speaker embedding; the total loss function of the speaker representation editing module is...
[0011] In the formula, Characterizes bounded penalty loss, Characterize the learnable contrastive learning loss weights. Representation contrastive learning loss, Characterize learnable adversarial learning loss weights. Representation adversarial learning loss, Representation of learnable speaker reconstruction loss weights, The speaker represents the reconstruction loss.
[0012] In some possible implementations, the training process of the speaker representation editing module further includes: suppressing attribute information carried by residual embeddings through cycle consistency training.
[0013] In some possible implementations, the recurrent consistency training includes: freezing the encoder of the variational autoencoder; generating a virtual speaker embedding using the decoder based on randomly sampled continuous attribute labels and the residual embedding; calculating the cross-entropy loss between the virtual speaker embedding and the randomly sampled labels using a pre-trained attribute predictor, and updating the decoder parameters via backpropagation to suppress attribute information carried in the residual embedding; the loss function for recurrent consistency training is defined as:
[0014]
[0015] In the formula, Cross-entropy loss characterizing age, Cross-entropy loss representing gender, Characterize learnable adversarial learning loss weights. Representation adversarial learning loss.
[0016] In some possible implementations, the training of the speaker representation editing module includes cyclic optimization training consisting of 5 standard training steps and 1 cycle-consistent training step.
[0017] Secondly, embodiments of this application provide a speech attribute editing device, comprising: an acquisition module for acquiring original speech; a processing module for decoupling the input original speech to obtain semantic features and original speaker representation; the processing module is further configured to construct a speaker representation editing module, extract attribute-independent residual embeddings from the original speaker representation using a variational autoencoder structure, and reconstruct an edited speaker embedding based on the residual embeddings and continuous attribute labels; the processing module is further configured to generate a speaker-independent average acoustic representation based on semantic features and quantized attribute labels; the processing module is further configured to generate a speaker-specific Mel spectrum using a stream matching model based on the average acoustic representation and the edited speaker embedding; and the processing module is further configured to convert the generated Mel spectrum into a final speech waveform using a pre-trained vocoder.
[0018] Thirdly, embodiments of this application provide a computer-readable storage medium including computer-readable instructions that, when read and executed by a computer, cause the computer to perform the method as described in any of the first aspects.
[0019] Fourthly, embodiments of this application provide a computing device, including a processor and a memory, wherein the memory stores computer program instructions, which, when executed by the processor, perform the method as described in any of the first aspects.
[0020] Fifthly, embodiments of this application provide a product comprising a computer program that, when the computer program product is run on a processor, causes the processor to perform the method as described in any of the first aspects. Attached Figure Description
[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 This is a flowchart illustrating a voice attribute editing method provided in an embodiment of this application;
[0023] Figure 2 This is a schematic diagram of another voice attribute editing method provided in an embodiment of this application;
[0024] Figure 3 This is a schematic diagram of a standard SRE training step provided in an embodiment of this application;
[0025] Figure 4 This is a schematic diagram of an SRE cyclic consistent training step provided in an embodiment of this application;
[0026] Figure 5 This is a schematic diagram of the internal process of a two-stage reconstruction module provided in an embodiment of this application;
[0027] Figure 6 This is the average result of a gender editing and age editing task provided in an embodiment of this application;
[0028] Figure 7 This application provides an embodiment of an attribute distribution after age and gender editing;
[0029] Figure 8 This is a schematic diagram of the structure of a voice attribute editing device provided in an embodiment of this application. Detailed Implementation
[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0031] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.
[0032] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.
[0033] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.
[0034] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.
[0035] To facilitate understanding of the embodiments of this application, the following will provide further explanation and description with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.
[0036] First, the technical terms involved in the embodiments of this application will be introduced:
[0037] 1. Semantic features: Content-related representations extracted by a pre-trained speech recognition model.
[0038] 2. Global speaker embedding: Speaker identity-related representation extracted by the speaker encoder.
[0039] 3. Residual embedding (z_res): The attribute-independent part extracted from the speaker's representation.
[0040] 4. Average representation: Speaker-independent Mel spectrum approximation that averages the group articulation and timbre under target attribute conditions.
[0041] 5. Continuous attribute label: A gender / age representation in real or log-odd form output by the attribute predictor.
[0042] Next, the technical solutions provided in the embodiments of this application will be introduced.
[0043] This application provides a speech attribute editing method that combines high-fidelity speaker representation editing (SRE) and two-stage speech reconstruction (TSSR) techniques to achieve continuous / discrete controllable editing of attributes such as age and gender. At the same time, it improves the naturalness, clarity and attribute consistency of the edited speech, and enhances the ability to preserve non-edited attributes.
[0044] For example, Figure 1 A flowchart illustrating a voice attribute editing method provided in an embodiment of this application is shown. Figure 1 As shown, the method may include the following steps:
[0045] S11: Decouple the input raw speech to obtain semantic features and the original speaker representation.
[0046] In this embodiment, to facilitate speaker attribute editing, the speech is first decoupled into speaker representation and semantic features. Simply put, speech decoupling separates the person and content from the speech information, i.e., "who is speaking" and "what was said." The original speaker representation represents the speaker's unique timbre, tone quality, and other identity information, i.e., "who is speaking." Semantic features represent the textual content and linguistic information in the speech, i.e., "what was said." Please refer to [reference needed]. Figure 2 , Figure 2 The text clearly shows that the input speech is decomposed into two independent processing paths.
[0047] Specifically, for semantic feature extraction, the Whisper-small model, pre-trained on a large-scale speech recognition dataset, is used as the semantic encoder. This model was chosen because it can accurately convert speech signals into corresponding text representations, i.e., semantic features, in understanding speech content. To maintain this high-quality content extraction capability during training and to prevent its parameters from being disturbed, a freezing strategy is adopted, meaning that all weights of the semantic encoder remain unchanged in subsequent training.
[0048] For the extraction of the original speaker representation, the ERes2NetV2 model from 3D-Speaker is used as the speaker encoder. This model is a state-of-the-art model in the field of speaker recognition, specifically designed to extract unique features from speech that can distinguish the identities of different speakers. To enhance the model's stability during training, especially its adaptability to small batches of data, all traditional batch normalization layers are replaced with group normalization layers.
[0049] Building upon this, incomplete decoupling may still exist; for example, speaker timbre information may still remain in the semantic features. Therefore, to achieve effective separation of speaker features and semantic features, the approach from the Seed-VC method was adopted. During training, a voice tone converter based on a speech conversion model is first used to perturb the input raw speech, changing its timbre characteristics, and then the processed speech is input into the semantic encoder. This approach is equivalent to introducing interference into the semantic encoder, causing it to ignore the constantly changing speaker timbre and focus on extracting stable and invariant semantic content, thereby effectively enhancing the feature separation effect.
[0050] In this way, a single input speech signal can be decomposed into two information streams: one is the pure semantic feature, representing "what was said", and the other is the original speaker representation, representing "who is speaking".
[0051] S12: Construct a speaker representation editing module, which uses a variational autoencoder structure to extract attribute-independent residual embeddings from the original speaker representation, and reconstructs the edited speaker embeddings based on the residual embeddings and continuous attribute labels.
[0052] In this embodiment, a Speaker Representation Editing (SRE) module is constructed to process the original speaker representation, obtaining an attribute-edited speaker representation, i.e., speaker embedding information. This new embedding needs to ensure attribute controllability and identity preservation. Attribute controllability means that the attribute information it carries (e.g., age, gender, etc.) must be consistent with the target attribute label specified by the user. Identity preservation means that when changing attributes, the identity-related timbre features of the original speaker must be preserved. The SRE adopts a variational autoencoder structure based on generative adversarial training. In the encoding stage, attribute-independent residual embeddings are extracted; in the decoding stage, attribute-controllable speaker embeddings are reconstructed.
[0053] For details, please refer to Figure 3 , Figure 3 This illustration shows a standard SRE training step provided in an embodiment of this application. For example... Figure 3 As shown, the encoder receives the original speaker representation as input, compresses it into a low-dimensional space (e.g., 16-dimensional), and outputs a residual embedding vector, denoted as Z. res Ideally, the residual embedding should contain only speaker identity information, independent of attributes. To achieve this ideal state as closely as possible, a contrastive learning loss L is introduced during training. clThis loss function enhances the differentiation between different speakers by maximizing the cosine similarity between residual embeddings of the same speaker while minimizing the similarity of these representations between different speakers. Furthermore, to improve the robustness of the decoder, standard Gaussian noise is added to the residual embeddings, and a bounded penalty is applied to limit their range and prevent them from becoming too large. The bounded penalty loss function is defined as follows:
[0054]
[0055] The decoder is used to reconstruct the speaker embedding under the joint conditions of residual embedding and continuous attribute labels. Continuous age (e.g., [0-12], [12-20], etc.) and the 2D Logits output of a pre-trained gender classifier (e.g., [0.8, 0.2], etc.) are used as continuous attribute labels to enable continuous attribute editing. To ensure that the generated speaker embedding is sufficiently realistic and indistinguishable from the distribution of real speaker embeddings, an adversarial learning loss L is introduced during training. adv Finally, a domain discriminator is introduced to determine the authenticity of the embeddings, aligning the generated embeddings with the real speaker space. Therefore, the total loss is defined as:
[0056]
[0057] In the formula, Characterizes bounded penalty loss, Characterize the learnable contrastive learning loss weights. Representation contrastive learning loss, Characterize learnable adversarial learning loss weights. Representation adversarial learning loss, Representation of learnable speaker reconstruction loss weights, The speaker represents the reconstruction loss.
[0058] In some possible embodiments, adversarial learning is introduced during speaker representation editing module training to align generated embeddings with the true embedding distribution, and attribute information carried by residual embeddings is suppressed through cycle consistency training.
[0059] In this embodiment, to further separate attribute information from the residual embedding, a cycle-consistent training strategy is introduced, which intersperses additional optimization steps within the standard training iterations. For an example, please refer to [link to example]. Figure 4 , Figure 4 This illustration shows an SRE cyclic consistency training step provided by an embodiment of this application. For example... Figure 4 As shown, in this cyclic consistent training step, the encoder remains frozen, and only the decoder is updated. When updating the decoder, the encoder updates based on randomly sampled continuous attribute labels and the residual embedding vector Z. resA virtual speaker embedding is generated. Then, a pre-trained attribute predictor is used to evaluate the consistency between the virtual speaker embedding and the assigned labels, and the cross-entropy loss between the virtual speaker embedding and randomly assigned attribute labels is calculated. This cross-entropy loss is backpropagated to the decoder, causing the decoder to ignore residual attribute information in the residual embedding. The loss function for this additional training step is defined as:
[0060]
[0061] In the formula, Cross-entropy loss characterizing age, Cross-entropy loss representing gender.
[0062] The speaker indicated that the editing module was optimized during training using a training loop consisting of 5 standard training steps and 1 cycle-consistent training step.
[0063] S13: Generate speaker-independent average acoustic representations based on semantic features and quantized attribute labels.
[0064] In this embodiment, after obtaining the speaker representation and semantic features from the original speech information, a two-stage reconstruction (TSSR) approach is used to further process the speech attributes and perform speech reconstruction. Generating a speaker-independent average acoustic representation is the first stage of the two-stage reconstruction. Its purpose is to generate an acoustic representation that is independent of a specific speaker and can represent the common features of the attribute group. That is, the acoustic representation only contains the common features of semantic content and target attributes, removing the personalized timbre of a specific speaker.
[0065] Specifically, the first step is to discretize the continuous natural attributes. For example, age can be divided into 10 consecutive intervals, such as [0–12, 12–20, …, 90–100], and gender can be divided into two groups (male and female). In this way, the continuous attribute space is transformed into discrete category labels that the model can explicitly learn. Please refer to [reference needed]. Figure 5 , Figure 5 A schematic diagram of the internal process of the two-stage reconstruction module is shown. Figure 5As shown, the averaging generator receives two inputs: semantic features generated in step S1 and discrete attribute labels. The discrete attribute labels represent quantized labels of the target attribute that the model expects to generate, such as "35-year-old male." The averaging generator learns and outputs a speaker-independent average acoustic representation. This representation is used to represent the most representative speech information given the semantic content and target attribute (i.e., discrete attribute labels). For example, for the same sentence "Hello," the generator will generate different speech representations for young men and middle-aged women. This representation is an within-group average acoustic representation, without individual-specific timbre features. To ensure that the generated acoustic representation is a good approximation of real speech in terms of spectral structure, a simple reconstruction loss function is used during the training process of the averaging generator to accurately learn these within-group commonalities. This function requires the L2 distance between the output average acoustic representation and the real Mel spectrogram. .
[0066] S14: Based on average acoustic representation and speaker embedding, a speaker-specific Mel spectrum is generated through a flow matching model.
[0067] In this embodiment, generating speaker-specific Mel spectrograms is the second stage of speech reconstruction. Building upon the first stage, specific speaker timbre details are added to generate speech information. Please continue to refer to... Figure 5 In this stage, the average acoustic representation output from the first stage and the speaker embedding processed by the speaker representation editing module are received as conditional inputs. In this stage, the detail generator uses stream matching to map the speaker-independent representation to a Mel spectrogram of the speaker's speech based on the speaker embedding. Stream matching starts by superimposing the average representation output from the first stage with some Gaussian noise, and, guided by the edited speaker embedding, progressively "de-noises" and refines it, ultimately generating a high-quality Mel spectrogram containing the target speaker's features. Therefore, the overall training loss of the entire TSSR module is defined as:
[0068]
[0069] In the formula, Characterization flow matching loss.
[0070] S15: Convert the generated Mel spectrogram into the final speech waveform using a pre-trained vocoder.
[0071] In this embodiment, a vocoder transforms the speaker-specific high-level acoustic representation of the Mel spectrum into an audible speech signal. The Mel spectrogram is a two-dimensional spectrogram that simulates the human ear's perception of sound frequencies, clearly reflecting the pitch, formants, and other spectral structures of speech, making it ideal as an intermediate representation between the acoustic model and the vocoder. However, the Mel spectrogram itself lacks phase information, and its temporal detail is insufficient for direct conversion into a smooth waveform. The vocoder is used for the inverse transformation from spectral parameters to a waveform, i.e., speech synthesis. For example, the vocoder can be a pre-trained BigVGAN model configuration.
[0072] To verify the advancement of this method, it was compared with two state-of-the-art methods—NANSY++ and VoiceShop. The comparison covered six metrics: (1) AMOS: Using source speech and target attributes as references, 20 participants rated the quality of the generated speech and the compliance with attribute editing instructions on a 1-5 scale; (2) PMOS: Using the same settings as AMOS, but participants were required to evaluate the degree of preservation of uncontrolled information in the speech; (3) WER: Using a pre-trained ASR model (Whisper), the word error rate of the attribute-edited speech was calculated; (4) DNSMOS: The OVRL score in DNSMOS P.835 was used to evaluate the overall speech quality; (5) Age Mean Error (AMAE): Using a pre-trained speech-to-attribute prediction model, the speaker's age was estimated from the generated speech, and the L1 error between the estimated age and the preset age label was calculated. In the gender editing task, a lower AMAE indicates better preservation of the original age; while in the age editing task, it indicates a higher fit between the edited speech and the target age. (6) Gender Classification Accuracy (GCA): Using a pre-trained speech-to-attribute prediction model, the speaker's gender is estimated from the generated speech, and its matching rate with the preset gender label is calculated. In the age editing task, a higher GCA indicates better preservation of the original gender; while in the gender editing task, it represents a higher fit between the edited speech and the target gender. Experiments were conducted on the English subset of the CommonVoice 22.0 dataset and the age- and gender-labeled subset of the VoxCeleb2 dataset. In the CommonVoice dataset, only some speakers have age and gender labels. Following the approach of previous studies, a speech-to-attribute prediction model was first trained on the labeled data, and this model was used to generate weak labels for unlabeled samples. The predicted two-dimensional gender logits were used instead of binary labels as continuous gender labels. Target Mel spectrogram extraction was performed with reference to the pre-trained BigVGAN model configuration and converted into waveforms using the same model. We considered two tasks: gender editing and age editing. The test set was constructed by randomly selecting three speech segments from each of the 109 speakers in the VCTK database. In the gender editing task, we tested two gender labels: [10, -10] (male) and [-10, 10] (female). In the age editing task, we tested four age labels: 12, 35, 55, and 75.
[0073] Figure 6 The average results for gender editing and age editing tasks are shown. Figure 7 The attribute distribution after age and gender editing is shown, with the line representing the trend of the median value. Please refer to [link / reference]. Figure 6 and Figure 7Regarding speech quality, our method achieved the highest AMOS and DNSMOS scores in both tasks, indicating better perceptual quality in the generated speech. Furthermore, in terms of intelligibility based on word error rate (WER), our method significantly reduced the error rate for gender editing to 2.90% and for age editing to 3.34%, both superior to VoiceShop (4.65% / 4.25%) and NANSY++ (8.33% / 7.43%). In terms of attribute control, our method achieved 100.00% accuracy in gender editing and the best performance in age preservation with an AMAE of 15.98. In age editing, our method further reduced the mean absolute error between predicted and target ages to 14.19 years, significantly lower than VoiceShop (19.11 years) and NANSY++ (22.83 years), and achieved the highest GCA value of 96.30%. In addition, Figure 7 The paper also demonstrates the attribute distribution predicted by the speech-to-attribute prediction model for attribute-edited speech. These results show that the speech generated by the proposed model not only matches the target attribute more accurately but also effectively preserves the features of the unedited attribute.
[0074] The above is an introduction to the speech attribute editing method provided in the embodiments of this application. Through adversarial learning and cyclic consistency training, the generated speaker embedding is aligned with the real embedding in terms of fidelity and attribute controllability, thereby improving the accuracy of attribute editing. Through the two-stage reconstruction of "average representation → detail generation", the training-inference mismatch is significantly alleviated, and the naturalness, intelligibility and robustness are improved. It can accurately achieve target attribute changes while maintaining the stability of semantic content and non-target attributes (such as original age or original gender).
[0075] It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. Furthermore, in some possible implementations, each step in the above embodiments may be selectively executed according to actual circumstances; it may be partially or fully executed, without limitation here. All or part of any feature of any embodiment of this application can be freely and arbitrarily combined without contradiction. The combined technical solutions are also within the scope of this application.
[0076] Based on the methods in the above embodiments, this application also provides a voice attribute editing device. For example, Figure 8 A schematic diagram of a voice attribute editing device is shown. Figure 8 As shown, the voice attribute editing device 800 includes an acquisition module 801 and a processing module 802.
[0077] The acquisition module 801 is used to acquire the original audio.
[0078] The processing module 802 is used to decouple the input raw speech and obtain semantic features and the original speaker representation;
[0079] The processing module 802 is also used to construct a speaker representation editing module, which uses a variational autoencoder structure to extract attribute-independent residual embeddings from the original speaker representation, and reconstructs the edited speaker embeddings based on the residual embeddings and continuous attribute labels.
[0080] The processing module 802 is also used to generate speaker-independent average acoustic representations based on semantic features and quantized attribute labels;
[0081] Processing module 802 is also used to generate speaker-specific Mel spectra based on average acoustic representation and edited speaker embedding via a flow matching model;
[0082] The processing module 802 is also used to convert the generated Mel spectrogram into the final speech waveform through a pre-trained vocoder.
[0083] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.
[0084] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.
[0085] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.
[0086] It is understood that the processor in the embodiments of this application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor.
[0087] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.
[0088] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).
[0089] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.
Claims
1. A method for editing voice attributes, characterized in that, The method includes: The original input speech is decoupled to obtain semantic features and original speaker representation; A speaker representation editing module is constructed. An attribute-independent residual embedding is extracted from the original speaker representation using a variational autoencoder structure. The edited speaker embedding is then reconstructed based on the residual embedding and continuous attribute labels. Generating speaker-independent average acoustic representations based on semantic features and quantized attribute labels; Based on the average acoustic representation and the edited speaker embedding, speaker-specific Mel spectra are generated through a flow matching model; The generated Mel spectrogram is converted into the final speech waveform using a pre-trained vocoder.
2. The method according to claim 1, characterized in that, The process of decoupling the input raw speech to obtain semantic features and the original speaker representation includes: A pre-trained speech recognition model is used as a semantic encoder to extract the semantic features, and the parameters are kept frozen during training. The speaker verification model is used as the speaker encoder to extract the original speaker representation, wherein the batch normalization layer in the speaker encoder is replaced with a group normalization layer.
3. The method according to claim 2, characterized in that, A voice tone converter based on a speech conversion model is used to perturb the input speech before it is input into the semantic encoder, thereby enhancing the separation effect between speaker features and semantic features.
4. The method according to claim 1, characterized in that, The generated speaker-independent average acoustic representation and the generated speaker-specific Mel spectrum together constitute a two-stage speech reconstruction module, the loss function of which is defined as: In the formula, Characterization flow matching loss, The L2 distance characterizes the average acoustic representation and the true Mel spectrum.
5. The method according to claim 1, characterized in that, The training process for the speaker representation editing module includes: A bounded penalty loss is applied to the extracted residual embedding, and the loss function is defined as follows: In the formula, A vector representing the residual embedding; Introduce contrastive learning loss to enhance the speaker discrimination ability of residual embedding; An adversarial learning loss is introduced to align the generated speaker embeddings with the distribution of the real speaker embeddings; The total loss function of the speaker representation editing module is: In the formula, Characterizes bounded penalty loss, Characterize the learnable contrastive learning loss weights. Representation contrastive learning loss, Characterize learnable adversarial learning loss weights. Representation adversarial learning loss, Representation of learnable speaker reconstruction loss weights, The speaker represents the reconstruction loss.
6. The method according to claim 1, characterized in that, The training process of the speaker representation editing module also includes: suppressing the attribute information carried by residual embedding through cycle consistency training.
7. The method according to claim 6, characterized in that, The cycle consistency training includes: Freeze the encoder of the variational autoencoder; The decoder is used to generate a virtual speaker embedding based on randomly sampled continuous attribute labels and the residual embedding; The cross-entropy loss between the virtual speaker embedding and the randomly sampled labels is calculated using a pre-trained attribute predictor, and the decoder parameters are updated via backpropagation to suppress attribute information carried in the residual embedding. The loss function for cycle consistency training is defined as: In the formula, Cross-entropy loss characterizing age, Cross-entropy loss representing gender, Characterize learnable adversarial learning loss weights. Representation adversarial learning loss.
8. The method according to claim 1, characterized in that, The training of the speaker representation editing module includes cyclic optimization training with 5 standard training steps and 1 cycle-consistent training step.
9. A voice attribute editing device, characterized in that, The device includes: The acquisition module is used to acquire the raw audio. The processing module is used to decouple the input raw speech and obtain semantic features and the original speaker representation; The processing module is also used to construct a speaker representation editing module, which uses a variational autoencoder structure to extract attribute-independent residual embeddings from the original speaker representation, and reconstructs the edited speaker embeddings based on the residual embeddings and continuous attribute labels. The processing module is also used to generate speaker-independent average acoustic representations based on semantic features and quantized attribute labels; The processing module is also used to generate speaker-specific Mel spectra based on the average acoustic representation and the edited speaker embedding through a flow matching model; The processing module is also used to convert the generated Mel spectrogram into the final speech waveform through a pre-trained vocoder.