A voice environment conversion method and system

Abstract

The application discloses a speech environment conversion method and system, and belongs to the field of speech environment conversion. The speech environment conversion method comprises the following steps: acquiring original audio and target audio; extracting timbre feature vectors, semantic word units and audio environment word units from the original audio and the target audio through an FVQ model; after the semantic word units perform cross attention on the audio environment word units, the semantic word units are sent into a decoder together with the timbre feature vectors, and finally, synthesized audio is obtained. The audio environment and the audio content, and the timbre, are decoupled, so that the functions of speech environment conversion, control of audio environment effect intensity, and superposition of multiple audio environments are realized.

Description

Technical Field

[0001] This invention belongs to the field of speech environment conversion, and specifically relates to a speech environment conversion method and system. Background Technology

[0002] Speech carries a lot of information, not only semantics, but also the speaker's timbre, emotion, style, and environmental information (reverberation, background noise, far-field and channel-related losses, etc.).

[0003] In recent years, speech generation technology has developed rapidly. Breakthroughs have been achieved in speech synthesis, timbre cloning, and timbre conversion. Conditional variational autoencoders, autoregressive models, and diffusion models have emerged, providing detailed modeling of speech synthesis accuracy, timbre similarity, and speaking style, achieving good results. However, ambient sound, an essential element of speech, has received little attention. Appropriate ambient sound is crucial for enhancing the realism of speech. In intelligent outbound calling scenarios, slight workplace noise can bridge the gap with customers and reduce hang-up rates. In dubbing scenarios, adding realistic ambient sound can even produce a convincingly realistic effect.

[0004] Traditional methods for synthesizing speech environments include adding additive noise and adding reverberation using predefined impulse responses. These methods are generally inflexible and can only synthesize predefined speech backgrounds. Speech enhancement models, on the other hand, only focus on removing noise or reverberation from noisy speech, but cannot add appropriate background noise to clean audio.

[0005] Most current speech generation models in the industry can only synthesize audio in quiet environments. While some zero-shot timbre cloning models can synthesize ambient sounds, their performance is poor and they are coupled with other speech attributes, significantly impacting their practicality and convenience. Current industry approaches to synthesizing ambient sounds generally fall into two categories: context-based learning and timbre feature vector-based models. Context-based autoregressive models typically integrate timbre with the speech environment and speaker style, failing to decouple ambient sounds and thus preventing separate cloning of the speech environment. Timbre feature vector-based models (Hierspeech++, Fastspeech, etc.) also face the problem of coupling between the speech environment and timbre. Zero-shot methods, lacking explicit modeling of the speech environment, generally perform poorly. Non-zero-shot methods, due to deep decoupling of timbre and speech environment, require a large amount of corresponding data when expanding timbre and speech environment, greatly limiting scalability.

[0006] Meanwhile, some methods that decouple timbre from the audio environment, by adding environmental feature vectors to speech synthesis, can achieve a certain degree of speech environment cloning. However, the overall architecture still relies on speech synthesis and requires corresponding text to synthesize the final audio. Furthermore, it is ineffective for noisy environments or specific sounds like bird calls; its main effect is the cloning of some reverberation. Additionally, it can only simply clone the corresponding audio environment, without controlling the strength of the added audio environment. The superposition effect between multiple environmental audio elements is also relatively weak. Summary of the Invention

[0007] In view of the shortcomings of the prior art, the purpose of this invention is to provide a voice environment conversion method and system, which solves the problems in the prior art.

[0008] The objective of this invention can be achieved through the following technical solutions:

[0009] A method for speech environment conversion includes the following steps:

[0010] Obtain the original audio and the target audio;

[0011] Using the FVQ model, timbre feature vectors, semantic units, and audio context units are extracted from the original audio and the target audio.

[0012] After semantic lexical units perform cross-attention on audio context lexical units, they are fed into the decoder along with the timbre feature vector to finally obtain the synthesized audio.

[0013] Furthermore, the FVQ model includes: an encoder, a timbre projection layer, a content projection layer, and an audio environment projection layer; the target audio passes through the encoder and the audio environment projection layer in sequence, and is then vector-quantized to obtain audio environment words; the original audio passes through the encoder, and then through the timbre projection layer and the content projection layer respectively to obtain timbre feature vectors and semantic words respectively.

[0014] Furthermore, the encoder body is a convolutional structure with a total of 4 layers and convolution strides of 2, 4, 5, and 5 respectively; the timbre projection layer, content projection layer, and audio environment projection layer all adopt the same transformer structure, but do not share parameters.

[0015] Furthermore, the training of the FVQ model includes the following:

[0016] 1) For timbre information, pooling is performed after the timbre projection layer to obtain the timbre feature vector, and the voiceprint classification loss is calculated by additive angular boundary Softmax; for semantic information, the CTC loss function in speech recognition is added, and the fundamental frequency in the source audio is extracted to calculate the mean square error loss.

[0017] 2) Add multiple GRL layers during the training of audio environment extraction; for timbre, add a GRL layer with CTC loss function and fundamental frequency correlation; for semantics, add a GRL layer for timbre classification.

[0018] 3) By using data augmentation to change the features of speech, the model is trained to extract the corresponding altered features, while other features remain basically unchanged.

[0019] 4) Perform vector quantization on the extracted features, where the codebook size is 4096;

[0020] 5) During data acquisition, while using real recording data, VoiceLDM is also used to generate data for different recording environments by providing different text-based speech environment descriptions.

[0021] Furthermore, the decoder includes: a convolutional enhancement Transformer, a modified stream model, and a HIFT-GAN vocoder;

[0022] The convolutional enhancement Transformer encodes the features extracted from the FVQ model. Then, semantic terms perform cross-attention on audio context terms, and are fed into the modified flow model along with the Mel spectrum of the target audio and the timbre feature vector. After N iterations, the corresponding Mel spectrum information is generated. Finally, the pre-trained HIFT-GAN vocoder is used to synthesize the audio.

[0023] Furthermore, during the decoder training process, a classifier-free guidance technique is used; during training, the audio environment feature vector extracted from the target audio is randomly set to 0 with a probability of p=0.15; during inference, the difference is calculated between using the source audio environment as a condition and not using the audio environment as a condition; then a normalization operation is performed; the specific implementation is shown in the following formula:

[0024] g cfg =g cond +α(g cond -g uncond )

[0025] g final =std(g cfg )*g cfg / std(g cfg )

[0026] In the formula, g cond To use the source's audio environment as a condition for the output, g uncond The output is conditioned on the source's audio environment, where α is the interpolation coefficient; g cfg This represents the result of interpolation without a classifier; std is the standard deviation, and g is the result of interpolation. finalThis is the final result without classifier guidance.

[0027] A voice environment conversion system, comprising:

[0028] Audio acquisition module: Acquires the original audio and the target audio;

[0029] Vector Quantization Module: Extracts timbre feature vectors, semantic units, and audio context units from the original and target audio using the FVQ model;

[0030] In addition, the audio synthesis module: after semantic lexical units perform cross-attention on audio environment lexical units, they are fed into the decoder together with the timbre feature vector to finally obtain the synthesized audio.

[0031] A computer storage medium storing a readable program that, when the program is run, can execute the aforementioned speech environment conversion method.

[0032] An electronic device includes: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus;

[0033] The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to one of the above-described voice environment conversion methods.

[0034] A computer program product includes computer instructions that instruct a computing device to perform an operation corresponding to the above-described speech environment conversion method.

[0035] The beneficial effects of this invention are:

[0036] 1. Compared with traditional methods, the present invention can automatically extract the speech environment information of the source speech based on the given target audio, and replace the sound environment of the source audio with the target audio.

[0037] 2. Compared with the current timbre cloning model, this invention can effectively decouple the audio environment from timbre, content and other information.

[0038] 3. The modified flow method used in this invention to reconstruct audio is faster inference and training than the commonly used diffusion model, and the model is easier to converge. At the same time, it generates higher sound quality compared with audio reconstruction models such as normalized flow.

[0039] 4. This invention can not only mimic overall audio environments such as white noise and reverberation, but also generate audio environments such as whistles and bird calls.

[0040] 5. This invention can not only add specified noise and reverb to clean speech, but also enhance speech if the target audio is clean.

[0041] 6. The decoupling features and audio reconstruction of this invention are independent of each other. Because this invention decouples and vectorizes the attributes of speech, it can not only realize speech environment transformation, but also, with slight extensions, be used for tasks such as timbre transformation and speech style transformation. It also facilitates subsequent upgrades and improvements to the reconstruction model.

[0042] 7. The decoder of this invention uses class-free guidance (CFG) technology during training, which allows for interpolation of the target audio environment during inference, thereby achieving the effect of controlling the strength of environmental noise.

[0043] 8. This invention allows the use of multiple different target audios, from which audio environment information is extracted and interpolated, thereby achieving the superposition of multiple different audio environments. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of the semantic environment conversion process of the present invention;

[0046] Figure 2 This is a schematic diagram of the overall structure and training of the FVQ model of the present invention;

[0047] Figure 3 This is a schematic diagram of the internal structure of the decoder of the present invention;

[0048] Figure 4 This is a schematic diagram of the structure of the convolution-enhanced Transformer of this invention;

[0049] Figure 5 This is a schematic diagram of the modified flow model structure of the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] Example 1

[0052] like Figure 1 As shown, a speech environment conversion method includes the following steps:

[0053] S1, acquire the source audio and the target audio;

[0054] The converted audio retains the content and timbre of the source audio, while the audio context is replaced with the audio environment of the target audio; when multiple audio environments need to be superimposed, multiple target audios can be input.

[0055] S2, through the FVQ (Factor Vector Quantization) model, extracts timbre feature vectors, semantic units, and audio context units from the original audio and the target audio to complete vector quantization;

[0056] like Figure 2 As shown, the FVQ model includes an encoder, a timbre projection layer, a content projection layer, and an audio environment projection layer. The target audio passes through the encoder and the audio environment projection layer in sequence, and is then vector-quantized to obtain audio environment words. After passing through the encoder, the original audio passes through the timbre projection layer and the content projection layer respectively to obtain timbre feature vectors and semantic words.

[0057] The encoder itself is a convolutional structure with four layers and convolutional strides of 2, 4, 5, and 5, achieving a frame rate of 80Hz. The timbre projection layer, content projection layer, and audio environment projection layer all use the same transformer structure but do not share parameters. The transformer structure also includes a bottleneck layer to help decouple relevant information from the model.

[0058] like Figure 2 As shown, the training function of the FVQ model is divided into three parts ( Figure 2 The dashed line portion is used only for training; training the FVQ model includes the following:

[0059] 1) Supervised training

[0060] For timbre information, pooling is performed after the timbre projection layer to obtain the timbre feature vector, and the voiceprint classification loss is calculated by additive angular margin softmax (AAM Softmax). For semantic information, the connectionist temporal classification (CTC) loss function in speech recognition is added, and the mean squared error loss is calculated by extracting the fundamental frequency from the source audio.

[0061] 2) Gradient Reversal Layer (GRL)

[0062] To filter out unwanted information and prevent information leakage, multiple GRL layers (CTC loss function, fundamental frequency, timbre) were added during the training of audio environment extraction to prevent it from learning timbre-related information. Specifically, for timbre, a CTC loss function and fundamental frequency-related GRL were added to prevent it from learning content- and prosody-related information. Similarly, for semantics, a timbre classification GRL layer was added.

[0063] 3) Adversarial training

[0064] By using data augmentation to change certain aspects of speech features (timbre, volume, audio environment), the aim is to train the model to extract significantly different features from the altered features, while keeping other features largely unchanged.

[0065] In this embodiment, a triplet (s) is constructed for each sample. a s o1 ,s o2 Taking audio environment characteristics as an example, where s a The audio is either augmented (possibly with added reverb or noise), or an audio track is randomly selected from the training set according to rules. o1 ,s o2 This is the original audio from the same speech environment (or the same audio, split into two segments); then, using the idea of ​​GAN, an audio environment discriminator is trained, whose loss function is a binary cross-entropy loss.

[0066] 4) FVQ Quantization

[0067] To enable the model to learn relevant information better, the extracted features are vector quantized, with a codebook size of 4096.

[0068] 5) Data Acquisition

[0069] To include as much data as possible with varying levels of noise, VoiceLDM was used to generate data for different recording environments while using real recording data, by providing different text-based speech environment descriptions.

[0070] S3: After semantic lexical units perform cross-attention on audio environment lexical units, they are fed into the decoder together with the timbre feature vector to finally obtain the synthesized audio.

[0071] like Figure 3 As shown, the decoder includes: a convolutional enhancement Transformer, a modified stream model, and a HIFT-GAN vocoder;

[0072] The convolutional enhancement Transformer encodes the features extracted from the FVQ model. Then, semantic lexical units perform cross-attention on audio context lexical units, and are fed into the modified flow model along with the Mel spectrum of the target audio and the timbre feature vector. After N(10) iterations, the corresponding Mel spectrum information is generated. Finally, the pre-trained HIFT-GAN vocoder is used to synthesize the final audio waveform.

[0073] The structure of the convolution-enhanced Transformer is as follows: Figure 4 As shown, convolutional layers are added on top of the transformer, which enables the model to capture global information while also enhancing its ability to process local details.

[0074] The structure of the corrected flow model is as follows: Figure 5 As shown, the overall structure adopts the form of UNet, divided into three parts: downsampling, intermediate layer, and upsampling. Each part uses a transformer and a residual module (ResNet1D). During inference, time step information is introduced, and similar to the generative diffusion model, a multi-step generation method is used to achieve better generation results. The difference is that fewer iterations are required, resulting in faster speed.

[0075] Meanwhile, during the decoder training process, a classifier-free guidance technique is used; during training, the audio environment feature vector extracted from the target audio is randomly set to 0 with a probability of p=0.15; correspondingly, during inference, the outputs using the source audio environment as a condition and those not using the audio environment as a condition are interpolated; then, a normalization operation is performed to control the strength of the audio environment; the specific implementation is as follows:

[0076] g cfg =g cond +α(g cond -g uncond )

[0077] g final =std(g cfg )*g cfg / std(g cfg )

[0078] In the formula, g cond To use the source's audio environment as a condition for the output, g uncond The output is conditioned on the source's audio environment, with α being the interpolation coefficient. cfg This is the result of interpolation after classifier-free guidance. std is the standard deviation, g is the result of interpolation after classifier-free guidance. final This is the final result without classifier guidance.

[0079] Furthermore, audio environment feature vectors extracted from multiple target audio sources can be used for inference and interpolation to achieve the effect of superimposing multiple audio environments.

[0080] Based on a similar inventive concept, embodiments of the present invention also provide a computer storage medium storing a readable program that, when the program is run, can execute the above-described voice environment conversion method.

[0081] Based on a similar inventive concept, this invention provides an electronic device, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus;

[0082] The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to one of the above-described voice environment conversion methods.

[0083] Based on a similar inventive concept, embodiments of the present invention also provide a computer program product, including computer instructions, which instruct a computing device to perform an operation corresponding to the above-described voice environment conversion method.

[0084] Example 2

[0085] Based on the speech environment conversion method in Embodiment 1, this embodiment proposes a speech environment conversion system, specifically including:

[0086] Audio acquisition module: Acquires the original audio and the target audio;

[0087] Vector Quantization Module: Extracts timbre feature vectors, semantic units, and audio context units from the original and target audio using the FVQ model;

[0088] Audio synthesis module: After semantic lexical units perform cross-attention on audio environment lexical units, they are fed into the decoder together with the timbre feature vector to finally obtain the synthesized audio.

[0089] The methods of the present invention can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored on a remote recording medium or a non-transitory machine-readable medium and subsequently stored on a local recording medium, downloaded via a network. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses the code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for performing the methods shown herein.

[0090] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A voice environment conversion method characterized by, The method comprises the following steps: obtaining original audio and target audio; extracting timbre feature vectors, semantic word units and audio environment word units from the original audio and the target audio through a factor vector quantization model; the semantic word units perform cross attention on the audio environment word units to obtain a mixed audio embedding sequence, and the mixed audio embedding sequence is sent into a decoder together with the timbre feature vectors to finally obtain synthesized audio; the factor vector quantization model comprises an encoder, a timbre projection layer, a content projection layer and an audio environment projection layer; the target audio sequentially passes through the encoder and the audio environment projection layer to obtain the audio environment word units through vector quantization; after the original audio passes through the encoder, it respectively passes through the timbre projection layer and the content projection layer to obtain the timbre feature vectors and the semantic word units respectively.

2. The method of claim 1, wherein, The encoder is mainly of a convolutional structure, and has a total of 4 layers with convolutional steps of 2, 4, 5 and 5 respectively; the timbre projection layer, the content projection layer and the audio environment projection layer all adopt the same transformer structure, but do not share parameters.

3. The method of claim 1, wherein, The training of the factor vector quantization model comprises the following contents: 1) for timbre information, pooling is performed after the timbre projection layer to obtain timbre feature vectors, and a voiceprint classification loss is calculated through additive angle boundary Softmax; for semantic information, a CTC loss function in speech recognition is added, and a mean square error loss is calculated by extracting the fundamental frequency of the source audio; 2) multiple GRL layers are added in the training of the audio environment extraction; in terms of timbre, a CTC loss function and a GRL layer related to the fundamental frequency are added; in terms of semantics, a GRL layer for timbre classification is added; 3) the features of the speech are changed through data enhancement, and through training, the corresponding changed features extracted by the model change, while other features remain basically unchanged; 4) the extracted features are vector quantized, and the codebook size is 4096; 5) when acquiring data, real recording data is used, and VoiceLDM is used to generate data of different recording environments by providing different text speech environment descriptions.

4. The method of claim 1, wherein, The decoder comprises a convolutional enhanced transformer, a correction flow model and a HIFT-GAN vocoder; the convolutional enhanced transformer encodes the features extracted from the factor vector quantization model, and then the semantic word units perform cross attention on the audio environment word units, and the mixed audio embedding sequence is sent into the correction flow model together with the mel spectrum of the target audio, to generate corresponding mel spectrum information through N iterations, and finally a pre-trained HIFT-GAN vocoder is used to synthesize audio.

5. The method of claim 4, wherein, In the training process of the decoder, a classifier-free guidance technology is used; in the training process, the audio environment feature vector extracted from the target audio is randomly set to 0 with a probability of p=0.15; in the inference process, the difference between using the source audio environment as a condition and not using the audio environment as a condition is calculated; then a normalization operation is performed; the specific implementation is as follows: g cfg = g cond + a(g cond - g uncond ) g final = std(g cfg )*g cfg / std(g cfg ) where g cond is the output conditioned on the audio environment of the source, g uncond is the output conditioned on the audio environment of the source, a is an interpolation coefficient; g cfg is the result of the interpolation after classifier-free guidance; std is the standard deviation, g final is the final result after classifier-free guidance.

6. A speech environment conversion system characterized by comprising: It comprises: an audio acquisition module for acquiring original audio and target audio; The vector quantization module extracts timbre feature vectors, semantic tokens and audio environment tokens from the original audio and the target audio through a factor vector quantization model; And the audio synthesis module: after the semantic tokens perform cross attention on the audio environment tokens, a mixed audio embedding sequence is obtained, which is sent into a decoder together with the timbre feature vectors, and finally a synthesized audio is obtained. The factor vector quantization model comprises an encoder, a timbre projection layer, a content projection layer and an audio environment projection layer; the target audio sequentially passes through the encoder and the audio environment projection layer, and the audio environment tokens are obtained through vector quantization; after the original audio passes through the encoder, it respectively passes through the timbre projection layer and the content projection layer, and the timbre feature vectors and the semantic tokens are obtained respectively.

7. A computer storage medium storing a readable program, characterized in that, When the program runs, a voice environment conversion method according to any one of claims 1-5 can be executed.

8. An electronic device, comprising: It comprises: a processor, a memory, a communication interface and a communication bus, which complete mutual communication through the communication bus; The memory is used to store at least one executable instruction, which makes the processor execute the operation corresponding to the voice environment conversion method in any one of claims 1-5.

9. A computer program product comprising computer instructions, characterized in that, The computer instructions instruct the computing device to execute the operation corresponding to the voice environment conversion method in any one of claims 1-5.

Images