Voice environment conversion method and system

Abstract

The invention discloses a voice environment conversion method and system, and belongs to the field of voice environment conversion. The voice environment conversion method comprises the following steps: acquiring an original audio and a target audio; extracting timbre feature vectors, semantic lexical elements and audio environment lexical elements from the original audio and the target audio through an FVQ model; after the semantic lexical elements execute cross attention on the audio environment lexical elements, the semantic lexical elements and the timbre feature vectors are sent into a decoder together, and finally the synthesized audio is obtained. Decoupling of the audio environment, the audio content and the tone is achieved, and therefore the functions of voice environment conversion, audio environment effect intensity control and superposition of a plurality of audio environments are achieved.

Description

Technical Field

[0001] The present invention belongs to the field of voice environment conversion, and in particular relates to a voice environment conversion method and system. Background Art

[0002] Speech carries a lot of information, not just semantics, but also the speaker's timbre, emotion, style, and information about the speaking environment (reverberation, background noise, far field, and loss caused by the channel, etc.).

[0003] In recent years, speech generation technology has developed rapidly. Breakthroughs have been made in speech synthesis, timbre cloning, timbre conversion and other technologies. Models such as conditional variational autoencoders, autoregression, and diffusion models have emerged, and detailed modeling has been carried out in terms of speech synthesis accuracy, timbre similarity, and speaking style, and good results have been achieved. However, the ambient sound, which is essential for speech, has received little attention. Appropriate ambient sound is crucial to enhancing the realism of speech. In the intelligent outbound call scenario, slight workplace noise can bring customers closer and reduce the user hang-up rate. In the dubbing scenario, a real ambient sound can even produce a fake effect.

[0004] Traditional methods for synthesizing speech environments include adding additive noise, adding reverberation using predefined impulse responses, etc. These methods are usually not flexible enough and can only synthesize predefined speech backgrounds. Speech enhancement-related models only focus on removing the corresponding noise or reverberation from noisy speech, but cannot add some corresponding background noise to clean audio.

[0005] At present, most of the speech generation models in the industry can only synthesize audio in a quiet environment. Although some zero-sample timbre cloning models can synthesize ambient sound, the effect is poor and coupled with other attributes of the voice, which greatly affects the practicality and convenience. At present, the industry can roughly divide the synthesis of voice environment into two types: based on context learning or based on timbre feature vectors. The autoregressive model based on context learning usually integrates timbre with voice environment, speaker style, etc., and does not achieve the decoupling of ambient sound, so it is naturally impossible to clone the voice environment alone. Models based on timbre feature vectors (Hierspeech++, Fastspeech, etc.) also face the problem of coupling between voice environment and timbre. Among them, the zero-sample method usually has poor effect because it does not explicitly model the voice environment. The non-zero-sample method, on the other hand, requires a large amount of corresponding data when expanding timbre and voice environment because timbre is deeply decoupled from the voice environment, and its scalability is greatly limited.

[0006] At the same time, some work on decoupling timbre and audio environment can achieve the effect of voice environment cloning to a certain extent by adding environmental feature vectors on the basis of speech synthesis, but the overall architecture relies on speech synthesis and needs to synthesize the corresponding final audio through the corresponding text. At the same time, it is also powerless for noisy environments, such as special bird calls, and its main effect is reflected in the cloning of some reverberations. In addition, it can only simply clone the corresponding audio environment, but cannot control the strength of the added audio environment. The superposition effect between multiple environmental audios is also relatively weak. Summary of the invention

[0007] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a method and system for voice environment conversion, which solves the problems in the prior art.

[0008] The purpose of the present invention can be achieved through the following technical solutions:

[0009] A method for converting a voice environment comprises the following steps:

[0010] Get the original audio and the target audio;

[0011] Through the FVQ model, the timbre feature vector, semantic word unit and audio environment word unit are extracted from the original audio and the target audio;

[0012] After the semantic word performs cross attention on the audio environment word, it is sent to the decoder together with the timbre feature vector to finally obtain the synthesized audio.

[0013] Furthermore, the FVQ model includes: an encoder, a timbre projection layer, a content projection layer and an audio environment projection layer; the target audio passes through the encoder and the audio environment projection layer in sequence, and is vector quantized to obtain audio environment word units; after the original audio passes through the encoder, it passes through the timbre projection layer and the content projection layer respectively to obtain timbre feature vectors and semantic word units respectively.

[0014] Furthermore, the encoder body is a convolutional structure with a total of 4 layers, and the convolution steps are 2, 4, 5, and 5 respectively; the timbre projection layer, content projection layer, and audio environment projection layer all adopt the same transformer structure, but do not share parameters.

[0015] Furthermore, the training of the FVQ model includes the following contents:

[0016] 1) For timbre information, pooling is performed after the timbre projection layer to obtain the timbre feature vector, and the voiceprint classification loss is calculated through additive angle boundary Softmax; for semantic information, the CTC loss function in speech recognition is added, and the fundamental frequency in the source audio is extracted to calculate the mean square error loss;

[0017] 2) Add multiple GRL layers in the training of audio environment extraction; in terms of timbre, add CTC loss function and fundamental frequency related GRL layers; in terms of semantics, add timbre classification GRL layers.

[0018] 3) By using data augmentation to change the features of speech, through training, the corresponding features extracted by the model are changed, while other features remain basically unchanged;

[0019] 4) Perform vector quantization on the extracted features, where the codebook size is 4096;

[0020] 5) When acquiring data, in addition to using real recording data, we also use VoiceLDM to generate data for different recording environments by providing different textual speech environment descriptions.

[0021] Further, the decoder includes: a convolution-enhanced Transformer, a modified flow model, and a HIFT-GAN vocoder;

[0022] The convolution-enhanced Transformer encodes the features extracted from the FVQ model, and then the semantic word performs cross-attention on the audio environment word, and then sends it to the modified flow model together with the Mel spectrum of the target audio and the timbre feature vector. After N iterations, the corresponding Mel spectrum information is generated, and finally the pre-trained HIFT-GAN vocoder is used to synthesize the audio.

[0023] Furthermore, in the training process of the decoder, a non-classifier guided technique is used; during the training process, the audio environment feature vector extracted from the target audio is randomly set to 0 with a probability of p = 0.15; during the inference process, the difference between using the source audio environment as a condition and not using the audio environment as a condition is performed; and then a normalization operation is performed; the specific implementation method is shown in the following formula:

[0024] g cfg =g cond +α(g cond -g uncond )

[0025] g final = std(g cfg )*g cfg / std(g cfg )

[0026] In the formula, g cond To use the source audio environment as the conditional output, g uncond is the output using the source audio environment as the condition, α is the interpolation coefficient; g cfg is the result of interpolation after bootstrapping without a classifier; std is the standard deviation, g finalis the final result without classifier guidance.

[0027] A voice environment conversion system, comprising:

[0028] Audio acquisition module: obtain original audio and target audio;

[0029] Vector quantization module: extracts timbre feature vectors, semantic words and audio environment words from the original audio and target audio through the FVQ model;

[0030] And, the audio synthesis module: after the semantic word performs cross attention on the audio environment word, it is sent to the decoder together with the timbre feature vector to finally obtain the synthesized audio.

[0031] A computer storage medium stores a readable program, which can execute the above-mentioned voice environment conversion method when the program is running.

[0032] An electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus;

[0033] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned voice environment conversion method.

[0034] A computer program product includes computer instructions, wherein the computer instructions instruct a computing device to execute operations corresponding to the above-mentioned voice environment conversion method.

[0035] Beneficial effects of the present invention:

[0036] 1. Compared with the traditional method, the present invention can automatically extract the speech environment information of the source speech according to the given target audio, and replace the sound environment of the source audio with the target audio.

[0037] 2. Compared with the current timbre cloning model, the present invention can well decouple the audio environment from timbre, content and other information.

[0038] 3. The modified stream used in the present invention to reconstruct audio is faster than the currently commonly used diffusion model inference training, the model is easier to converge, and the generated sound quality is higher than that of audio reconstruction model methods such as standardized streams.

[0039] 4. The present invention can not only imitate the overall audio environment such as white noise and reverberation, but also generate audio environments such as whistle sounds and bird calls.

[0040] 5. The present invention can not only add specified noise and reverberation to clean speech, but also can achieve speech enhancement effect if the target audio is clean audio.

[0041] 6. The decoupling feature and audio reconstruction of the present invention are independent of each other. Since the present invention decouples and vectorizes the attributes of speech, it can not only realize speech environment conversion, but also be used for tasks such as timbre conversion and speech style conversion with a little expansion. It is also convenient for subsequent upgrading and improvement of the reconstruction model.

[0042] 7. Since the decoder of the present invention adopts the class free guidance (CFG) technology in training, it allows the target audio environment to be differentiated during reasoning, thereby achieving the effect of controlling the intensity of the environmental noise.

[0043] 8. The present invention allows the use of multiple different target audios, extracts audio environment information from them, and performs difference, thereby achieving the superposition of multiple different audio environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 It is a schematic diagram of the semantic environment conversion process of the present invention;

[0046] Figure 2 It is the overall structure and training diagram of the FVQ model of the present invention;

[0047] Figure 3 It is a schematic diagram of the internal structure of the decoder of the present invention;

[0048] Figure 4 It is a schematic diagram of the structure of the convolution-enhanced Transformer of the present invention;

[0049] Figure 5 It is a schematic diagram of the structure of the modified flow model of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] Example 1

[0052] like Figure 1 As shown, a method for converting a voice environment comprises the following steps:

[0053] S1, obtain source audio and target audio;

[0054] The converted audio retains the content and timbre of the source audio, and the audio environment is replaced with the audio environment of the target audio; when multiple audio environments need to be superimposed, multiple target audios can be input.

[0055] S2, extracting timbre feature vectors, semantic words and audio environment words from the original audio and target audio through the FVQ (factor vector quantization) model to complete vector quantization;

[0056] like Figure 2 As shown in the figure, the FVQ model includes an encoder, a timbre projection layer, a content projection layer and an audio environment projection layer; the target audio passes through the encoder and the audio environment projection layer in turn, and is vector quantized to obtain the audio environment word unit; after the original audio passes through the encoder, it passes through the timbre projection layer and the content projection layer respectively to obtain the timbre feature vector and the semantic word unit respectively.

[0057] The encoder is mainly a convolutional structure with 4 layers in total. The convolution steps are 2, 4, 5, and 5, respectively, so as to achieve a frame rate of 80 Hz. The timbre projection layer, content projection layer, and audio environment projection layer all use the same transformer structure, but do not share parameters. At the same time, the transformer structure also contains a bottleneck layer structure to help the model decouple related information.

[0058] like Figure 2 As shown, the training function of the FVQ model is divided into three parts: Figure 2 The dotted part is only used for training); the training of the FVQ model includes the following:

[0059] 1) Supervised training

[0060] For timbre information, pooling is performed after the timbre projection layer to obtain the timbre feature vector, and the voiceprint classification loss is calculated through Additive Angular Margin Softmax (AAM Softmax). For semantic information, the Connectionist Temporal Classification (CTC) loss function in speech recognition is added, and the fundamental frequency in the source audio is extracted to calculate the mean square error loss.

[0061] 2) Gradient Reversal Layer (GRL)

[0062] In order to filter out unnecessary information and prevent information leakage, multiple GRL layers (CTC loss function, fundamental frequency, timbre) are added to the training of audio environment extraction to prevent it from learning information related to timbre and content. In terms of timbre, CTC loss function and fundamental frequency-related GRL are added to prevent it from learning information related to content and rhythm. Similarly, in terms of semantics, a GRL layer for timbre classification is added.

[0063] 3) Adversarial training

[0064] By using data augmentation to change the characteristics of a certain aspect of speech (timbre, volume, audio environment), it is hoped that through training, the corresponding changed features extracted by the model will undergo significant changes, while other features remain basically unchanged.

[0065] In this embodiment, a triplet (s a ,s o1 ,s o2 ); Taking the audio environment characteristics as an example, s a is the audio after data enhancement (perhaps with the addition of reverberation or noise), or a piece of audio is randomly selected from the training set according to the rules, s o1 ,s o2 is the original audio of the same speech environment (or the same audio is used and split into two segments); then, using the idea of ​​GAN, an audio environment discriminator is trained, and its loss function is a binary cross entropy loss.

[0066] 4) FVQ quantification

[0067] In order to enable the model to better learn relevant information, the extracted features are vector quantized, with a codebook size of 4096.

[0068] 5) Data Acquisition

[0069] In order to include as much data as possible with various noises, while using real recording data, we also use VoiceLDM to generate data in different recording environments by providing different text speech environment descriptions.

[0070] In S3, the semantic word performs cross attention on the audio environment word and is sent to the decoder together with the timbre feature vector to finally obtain the synthesized audio.

[0071] like Figure 3 As shown, the decoder includes: convolution-enhanced Transformer, modified flow model and HIFT-GAN vocoder;

[0072] The convolution-enhanced Transformer encodes the features extracted from the FVQ model. After that, the semantic word performs cross-attention on the audio environment word. The mel spectrum of the target audio and the timbre feature vector are sent to the modified flow model. After N(10) iterations, the corresponding mel spectrum information is generated. Finally, the pre-trained HIFT-GAN vocoder is used to synthesize the final audio waveform.

[0073] Among them, the structure of the convolution-enhanced Transformer is as follows Figure 4 As shown in the figure, a convolutional layer is added to the transformer, so that the model can capture global information while also enhancing its ability to process local details.

[0074] The structure of the Rectified Flow model is as follows Figure 5 As shown in the figure, the overall structure adopts the form of Unet, which is divided into three parts: downsampling, middle layer, and upsampling. Each part uses transformer and residual module (resnet1d). Time step information is introduced during inference, and multi-step generation is used similar to the generation diffusion model to achieve better generation effect. The difference is that fewer iterations are required and the speed is faster.

[0075] At the same time, in the training process of the decoder, the classifier-free guidance technology is used; during the training process, the audio environment feature vector extracted from the target audio is randomly set to 0 with a probability of p = 0.15; accordingly, in the process of inference, the output using the source audio environment as a condition and not using the audio environment as a condition is interpolated; then a normalization operation is performed to achieve the purpose of controlling the strength of the audio environment; the specific implementation method is as follows:

[0076] g cfg =g cond +α(g cond -g uncond )

[0077] g final = std(g cfg )*g cfg / std(g cfg )

[0078] In the formula, g cond To use the source audio environment as the conditional output, g uncond is the output using the source audio environment as the condition, and α is the interpolation coefficient. cfg is the result of interpolation after bootstrapping without classifier. std is the standard deviation, g final is the final result without classifier guidance.

[0079] Furthermore, the audio environment feature vectors extracted from multiple target audios may be used for inference and difference, thereby achieving the effect of superimposing multiple audio environments.

[0080] Based on similar inventive concepts, an embodiment of the present invention further provides a computer storage medium storing a readable program, which can execute the above-mentioned voice environment conversion method when the program is running.

[0081] Based on similar inventive concepts, an embodiment of the present invention provides an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus;

[0082] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned voice environment conversion method.

[0083] Based on similar inventive concepts, an embodiment of the present invention further provides a computer program product, including computer instructions, wherein the computer instructions instruct a computing device to execute operations corresponding to the above-mentioned voice environment conversion method.

[0084] Example 2

[0085] Based on a voice environment conversion method in Example 1, this embodiment proposes a voice environment conversion system, which specifically includes:

[0086] Audio acquisition module: obtain original audio and target audio;

[0087] Vector quantization module: extracts timbre feature vectors, semantic words and audio environment words from the original audio and target audio through the FVQ model;

[0088] Audio synthesis module: After the semantic word performs cross attention on the audio environment word, it is sent to the decoder together with the timbre feature vector to finally obtain the synthesized audio.

[0089] The method of the present invention may be implemented in hardware, firmware, or as software or computer code that may be stored in a recording medium (such as a CDROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded over a network and will be stored in a local recording medium, so that the method described herein may be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that a computer, processor, microprocessor controller, or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, processor, or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown herein.

[0090] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A method for converting a speech environment, characterized in that: The following steps are involved: Get the original audio and the target audio; Through the FVQ model, the timbre feature vector, semantic word unit and audio environment word unit are extracted from the original audio and the target audio; After the semantic word performs cross attention on the audio environment word, it is sent to the decoder together with the timbre feature vector to finally obtain the synthesized audio.

2. A method for converting a voice environment according to claim 1, characterized in that: The FVQ model includes: an encoder, a timbre projection layer, a content projection layer and an audio environment projection layer; the target audio passes through the encoder and the audio environment projection layer in sequence, and is vector quantized to obtain an audio environment word unit; after the original audio passes through the encoder, it passes through the timbre projection layer and the content projection layer respectively to obtain a timbre feature vector and a semantic word unit respectively.

3. A method for converting a voice environment according to claim 2, characterized in that: The encoder body is a convolutional structure with a total of 4 layers, and the convolution steps are 2, 4, 5, and 5 respectively; the timbre projection layer, content projection layer, and audio environment projection layer all adopt the same transformer structure, but do not share parameters.

4. A method for converting a voice environment according to claim 2, characterized in that: The training of the FVQ model includes the following: 1) For timbre information, pooling is performed after the timbre projection layer to obtain the timbre feature vector, and the voiceprint classification loss is calculated through additive angle boundary Softmax; for semantic information, the CTC loss function in speech recognition is added, and the fundamental frequency in the source audio is extracted to calculate the mean square error loss; 2) Add multiple GRL layers in the training of audio environment extraction; in terms of timbre, add CTC loss function and fundamental frequency related GRL layers; in terms of semantics, add timbre classification GRL layers; 3) By using data augmentation to change the features of speech, through training, the corresponding features extracted by the model are changed, while other features remain basically unchanged; 4) Perform vector quantization on the extracted features, where the codebook size is 4096; 5) When acquiring data, in addition to using real recording data, we also use VoiceLDM to generate data for different recording environments by providing different textual speech environment descriptions.

5. A method for converting a voice environment according to claim 1, characterized in that: The decoder includes: a convolution-enhanced Transformer, a modified flow model, and a HIFT-GAN vocoder; The convolution-enhanced Transformer encodes the features extracted from the FVQ model, and then the semantic word performs cross-attention on the audio environment word, and then sends it to the modified flow model together with the Mel spectrum of the target audio and the timbre feature vector. After N iterations, the corresponding Mel spectrum information is generated, and finally the pre-trained HIFT-GAN vocoder is used to synthesize the audio.

6. A method for converting a voice environment according to claim 5, characterized in that: In the decoder training process, the classifier-free bootstrap technique is used; during the training process, the audio environment feature vector extracted from the target audio is randomly set to 0 with a probability of p = 0.15; during the inference process, the difference between using the source audio environment as a condition and not using the audio environment as a condition is performed; and then a normalization operation is performed; The specific implementation method is shown in the following formula: g cfg =g cond +α(g cond -g uncond ) g final =std(g cfg )*g cfg / std(g cfg ) In the formula, g cond To use the source audio environment as the conditional output, g uncond is the output using the source audio environment as the condition, α is the interpolation coefficient; g cfg is the result of interpolation after bootstrapping without a classifier; std is the standard deviation, g final is the final result without classifier guidance.

7. A voice environment conversion system, characterized in that: include: Audio acquisition module: obtain original audio and target audio; Vector quantization module: extracts timbre feature vectors, semantic words and audio environment words from the original audio and target audio through the FVQ model; And, the audio synthesis module: after the semantic word performs cross attention on the audio environment word, it is sent to the decoder together with the timbre feature vector to finally obtain the synthesized audio.

8. A computer storage medium storing a readable program, characterized in that: When the program is running, it can execute the voice environment conversion method described in any one of claims 1 to 6.

9. An electronic device, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to a voice environment conversion method as described in any one of claims 1-6.

10. A computer program product comprising computer instructions, characterized in that: The computer instructions instruct the computing device to execute operations corresponding to a voice environment conversion method as described in any one of claims 1-6.

Images