Feature semantic distinguishing and model single-step screening mimicry audio generation method

Abstract

The invention discloses a mimicry audio generation method based on feature semantic distinguishing and model single-step screening, and belongs to the technical field of computer audio synthesis and signal processing. The implementation method comprises the following steps: 1, extracting Mel spectrum features from an audio sample, forming a frequency domain spectrogram of the audio through short-time Fourier transform, and performing frequency domain mapping and logarithmic compression on the frequency domain spectrogram by using a Mel filter bank to form a Mel spectrogram; 2, training an audio codec configured with a multi-stage residual quantizer by using fusion loss of semantic classification loss and supervised reconstruction loss to obtain audio features with semantic differentiation; 3, training the audio diffusion model with multi-task diffusion loss; and 4, performing single-step screening on the voice audio by using the trained audio diffusion model in combination with the language audio similarity to obtain a de-noised audio. Compared with the prior art, the technical problem of efficiently generating the high-quality audio consistent with the input semantics when the high-quality vivid scene sound effect is simulated is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to a feature semantic distinction and model single-step screening pseudo-audio generation method, belonging to the technical field of computer audio synthesis and signal processing, and applied to virtual reality and augmented reality scene audio generation. BACKGROUND

[0002] In current audio generation technology, the synthesis of pseudo-audio scene audio has been facing multiple challenges, mainly including unstable sound quality, mismatch between generation results and semantics, and low generation efficiency. Existing audio generation methods usually rely on neural network-based models such as generative adversarial networks and variational autoencoders, which have made certain progress in high-quality audio generation, but still have certain difficulties in model training stability, audio generation quality, and semantic consistency.

[0003] When simulating high-quality realistic scene sound effects, existing methods are difficult to guarantee the diversity and accuracy of generated audio. Diffusion modeling as a new generation method has made significant progress in image and text generation in recent years, but its application in audio generation is still relatively rare. Existing methods based on diffusion models usually generate data gradually through an iterative process and generate multiple audios through multiple iterations. Finally, the pre-trained model is used to screen the audio most relevant to the semantic label, but this process still faces problems such as low generation efficiency and complex model training.

[0004] Therefore, when simulating high-quality realistic scene sound effects, how to efficiently generate high-quality audio consistent with the input semantics has become a problem to be solved. SUMMARY

[0005] The purpose of the present application is to solve the technical problem of efficiently generating high-quality audio consistent with the input semantics when simulating high-quality realistic scene sound effects, and to propose a feature semantic distinction and model single-step screening pseudo-audio generation method. The present application aims to improve audio generation quality through feature semantic distinction and to improve audio efficiency through pre-trained model single-step screening.

[0006] The working principle of the present application is as follows: the present application is based on a diffusion model-based generation method that combines semantic feature distinction audio codec for feature extraction, and uses diffusion modeling to model noisy features under given conditions to restore original features. By calling the diffusion model multiple times to generate multiple audios only with single-step denoising, the audio most consistent with the semantics is screened out and multiple-step denoising is completed.

[0007] The purpose of the present application is achieved through the following technical solutions:

[0008] The application discloses a feature semantic distinction and model single-step screening phonetic audio generation method.

[0009] Step 1: the audio sample is subjected to mel spectrum feature extraction, a frequency domain spectrum of the audio is formed through short-time Fourier transform, frequency domain mapping and logarithmic compression of the frequency domain spectrum are performed on the frequency domain spectrum through a mel filter bank, and a mel spectrum is formed;

[0010] Step 1.1: audio data is collected to form an audio sample;

[0011] Step 1.2: mel spectrum features of the audio sample are extracted, and a frequency domain spectrum of the audio is obtained through Fourier transform;

[0012] Step 1.2.1: the mel spectrum features of the audio are extracted through a classification loss guided residual audio codec;

[0013] Step 1.2.2: the mel spectrum features of the audio are subjected to short-time Fourier transform to obtain a frequency domain spectrum of the audio;

[0014] Step 1.3: frequency domain mapping and logarithmic compression are performed on the frequency domain spectrum through a mel filter bank to obtain a mel spectrum;

[0015] Step 2: an audio codec configured with a multi-level residual quantizer is trained through a fusion loss formed by fusing a semantic classification loss and a supervised reconstruction loss, and audio features with semantic distinction are obtained;

[0016] Step 2.1: an audio codec configured with a multi-level residual quantizer is constructed, a semantic classification loss L cls and a supervised reconstruction loss L recons are fused to form a fusion loss, and the audio codec is optimized through the fusion loss;

[0017] Step 2.1.1: an audio codec composed of an audio encoder E and an audio decoder D is constructed;

[0018] Step 2.1.2: the audio codec is configured with a multi-level residual quantizer, and a quantization error L quant of the audio features as shown in formula (1) is obtained.

[0019] L quant =∑ l sum||z l –Q l (z l )|| 2 (1)

[0020] Wherein, z l represents the lth layer feature, Q l represents the selection of z lquantization function closest to the symbol;

[0021] Step 2.1.3: classifying the quantized audio features with a nonlinear transformation of a semantic-aware multi-layer perceptron (MLP);

[0022] Step 2.1.3.1: obtaining a semantic-aware semantic classification loss L cls ;

[0023] L cls =∑ i -t i log(MLP(z quant ) (2)

[0024] where t i is the label of the ith pseudo-audio and z quant is the output of all layers of quantizers;

[0025] Step 2.1.3.2: incorporating the semantic classification loss into the multi-layer perceptron (MLP);

[0026] Step 2.1.3.3: classifying the audio features with the semantic-aware multi-layer perceptron (MLP);

[0027] Step 2.1.4: optimizing the audio codec with a fusion loss formed by the semantic classification loss L cls and the supervised reconstruction loss L recons ;

[0028] Step 2.1.4.1: configuring the supervised reconstruction loss L recons for the audio codec;

[0029] Step 2.1.4.2: fusing the semantic classification loss L cls with the supervised reconstruction loss L recons to form a fusion loss as shown in equation (3);

[0030] L vae = L recons +λ quant L quant +λ cks L cls (3)

[0031] where λ quant and λ cls are balancing hyperparameters;

[0032] Step 2.1.4.3: optimizing the audio codec with the fusion loss;

[0033] Step 2.2: input the mel-spectrogram into the audio codec for training to obtain audio features with semantic distinction;

[0034] Step 2.2.1: the mel-spectrogram is compressed by two-dimensional convolution of the audio encoder E to obtain audio features z = E(x);

[0035] Step 2.2.2: the audio features are sampled by two-dimensional transposed convolution of the audio decoder D to obtain the mel-spectrum of the audio;

[0036] Step 3: train the audio diffusion model with multi-task diffusion loss by using the pseudo-audio label t and the audio features with semantic distinction;

[0037] Step 3.1: construct the audio diffusion model with multi-task diffusion loss;

[0038] Step 3.1.1: construct the audio diffusion model with Dual-Unet architecture;

[0039] Step 3.1.2: add noise to the audio features with semantic distinction by using the linear noise scheduling method as shown in equation (4);

[0040]

[0041] where t ∈ [1, T] is the diffusion step number, β min = 0.0001, β max = 0.02;

[0042] Step 3.1.3: optimize the audio diffusion model by using the multi-task diffusion loss as shown in equation (5);

[0043] L diff = L data + λ quant L noise (5)

[0044] where L data is the loss function of the data branch of the dual branch, which calculates the mean square error between the Unet output and the target data; L noise is the loss function of the noise branch of the dual branch, which calculates the mean square error between the Unet output and the target noise;

[0045] Step 3.2: set the pseudo-audio label t, input the pseudo-audio label t and the audio features with semantic distinction into the audio diffusion model with multi-task diffusion loss for training;

[0046] Step 4: use the trained audio diffusion model to combine the language audio similarity for one-step screening of speech audio to obtain denoised audio;

[0047] Step 4.1: denoising a plurality of randomly generated Gaussian noises using the trained audio diffusion model to form a single-step denoised audio set;

[0048] Step 4.2: single-step screening of the language audio set using the language audio similarity as shown in formula (6);

[0049] s = cos(c(a) + f t (t)) (6)

[0050] Wherein, f a and f t respectively represent the feature encoders of the audio and pseudo phonetic labels;

[0051] Step 4.3: setting the number of denoising steps to M, executing steps 4.1 to 4.2 in a loop iteration manner until all denoising steps are completed, and obtaining the completely denoised audio;

[0052] Compared with the prior art, the following beneficial effects are achieved:

[0053] 1. The pseudo phonetic label is introduced to enhance the semantic distinguishability of the audio representation, and combined with the multi-level residual quantizer, the fidelity and robustness of the audio reconstruction are significantly improved, providing support for the downstream generation task;

[0054] 2. In terms of generation model architecture, a multi-task diffusion modeling framework based on an improved Dual-Unet of Unet structure is constructed. The method introduces a multi-task objective function based on the traditional diffusion model, simultaneously predicts noise and clean data, and uses double up-sampling branches based on the U-net structure to model and improve the modeling accuracy and denoising ability of the denoising model. The generation quality of the pseudo phonetic audio is significantly improved;

[0055] 3. The contrastive language-audio pre-training model is introduced to construct a generation screening mechanism based on semantic similarity. At the same time, a single-step screening strategy is designed, which can efficiently screen high-quality samples in the inference stage compared with screening after completing all denoising steps of the audio, improving the generation efficiency while ensuring the synthesis effect; BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is a schematic diagram of the system framework of the present application;

[0057] Figure 2 is a flowchart of the present application. DETAILED DESCRIPTION

[0058] For the purpose of better illustrating the objects and advantages of the present application, the following further describes the application in connection with the drawings and examples. It should be noted that the implementation of the present application is not limited to the following examples, and any form of variation or change made to the present application will fall within the scope of protection of the present application.

[0059] Embodiment

[0060] As shown in the embodiment, a feature semantic distinction and model single-step screening prosodic audio generation method, the specific implementation steps are as follows: Figure 1

[0061] Step 1: Extract the Mel spectrum features of the audio sample, form the frequency domain spectrum graph of the audio by short-time Fourier transform, and use the Mel filter bank to perform frequency domain mapping and logarithmic compression on the frequency domain spectrum graph to form the Mel spectrum graph;

[0062] Step 1.1: Collect audio data to form an audio sample;

[0063] Step 1.2: Extract the Mel spectrum features of the audio sample, and obtain the frequency domain spectrum graph of the audio by Fourier transform;

[0064] Step 1.2.1: Use the classification loss guided residual audio codec to extract the Mel spectrum features of the audio;

[0065] Step 1.2.2: Obtain the frequency domain spectrum graph of the audio by short-time Fourier transform of the Mel spectrum features of the audio;

[0066] Step 1.3: Use the Mel filter bank to perform frequency domain mapping and logarithmic compression on the frequency domain spectrum graph to obtain the Mel spectrum graph;

[0067] In the embodiment, as shown in the embodiment, the audio of DCASE-2023 task 7 is transformed into a frequency domain spectrum graph by short-time Fourier transform, and the frequency domain spectrum graph is mapped and compressed by a Mel filter bank to form a Mel spectrum graph; Figure 2

[0068] Step 2: Train the audio codec configured with a multi-level residual quantizer using the fusion loss of the semantic classification loss and the supervised reconstruction loss to obtain audio features with semantic distinction;

[0069] Step 2.1: Construct an audio codec configured with a multi-level residual quantizer, and use the semantic classification loss L cls and the supervised reconstruction loss L recons to form a fusion loss to optimize the audio codec;

[0070] Step 2.1.1: Construct an audio codec composed of an audio encoder E and an audio decoder D of a deep convolutional neural network; ​​

[0071] Step 2.1.2: Configure the multi-level residual quantizer for the audio codec, obtain the quantization error L of the audio feature as shown in equation (1) quant ;

[0072] L quant =∑ l sum||z l –Q l (z l )|| 2 (1)

[0073] where z l represents the l-th layer feature, Q l represents the quantization function that selects the nearest quantization symbol to z l ;

[0074] Step 2.1.3: Classify the quantized audio feature using the nonlinear transformation of the multi-layer perception MLP with semantic differentiation;

[0075] Step 2.1.3.1: Obtain the semantic classification loss L cls of semantic differentiation in the manner as shown in equation (2)

[0076] L cls =∑ i -t i log(MLP(z quant ) (2)

[0077] where t i is the label of the i-th pseudo audio, and z quant is the output after all layer quantizers;

[0078] Step 2.1.3.2: Add the semantic classification loss to the multi-layer perception MLP;

[0079] Step 2.1.3.3: Classify the audio feature using the multi-layer perception MLP with semantic differentiation;

[0080] Step 2.1.4: Optimize the audio codec using the fusion loss formed by the semantic classification loss L cls and the supervised reconstruction loss L recons ;

[0081] Step 2.1.4.1: Configure the supervised reconstruction loss L recons for the audio codec;

[0082] Step 2.1.4.2: Fuse the semantic classification loss L cls and the supervised reconstruction loss L recons to form the fusion loss as shown in equation (3)

[0083] L vae = L recons + λ quant L quant + λ cls L cls (3)

[0084] where λ quant and λ cls are balanced hyperparameters;

[0085] Step 2.1.4.3: optimizing the audio codec with the fusion loss;

[0086] Step 2.2: inputting the mel-spectrogram into the audio codec for training to obtain audio features with semantic distinction;

[0087] Step 2.2.1: compressing the mel-spectrogram using two-dimensional convolution of the audio encoder E to obtain audio features z = E(x);

[0088] Step 2.2.2: sampling the audio features using two-dimensional transposed convolution of the audio decoder D to obtain the mel-spectrum of the audio;

[0089] In the embodiment, the mel-spectrogram calculated from the pseudo-audio of DCASE-2023 Task 7 is input into the audio codec, and the total loss is composed of the reconstruction loss, the quantization loss and the classification loss. The model is trained using the total loss function, and then the audio uses the trained model to use the quantized features as the features of the audio, preparing for the generation of the next step.

[0090] Step 3: training the audio diffusion model with multi-task diffusion loss using pseudo-audio labels t and audio features with semantic distinction;

[0091] Step 3.1: constructing an audio diffusion model with multi-task diffusion loss;

[0092] Step 3.1.1: constructing an audio diffusion model with Dual-Unet architecture;

[0093] Step 3.1.2: adding noise to the audio features with semantic distinction using the linear noise scheduling method shown in equation (4);

[0094]

[0095] where t ∈ [1, T] is the number of diffusion steps, β min = 0.0001, and β max = 0.02;

[0096] Step 3.1.3: The audio diffusion model is optimized by using a multi-task diffusion loss as shown in formula (5);

[0097] L diff =L data +λ quant L noise (5)

[0098] Wherein L data is the loss function of the data branch of the double branch, which calculates the mean square error between the Unet output and the target data; L noise is the loss function of the noise branch of the double branch, which calculates the mean square error between the Unet output and the target noise;

[0099] Step 3.2: Set the pseudo-audio label t, and input the pseudo-audio label t and the audio features with semantic distinction into the audio diffusion model with multi-task diffusion loss for training;

[0100] In the embodiment, the label and feature of the pseudo-audio in step 2 are used as training data, and the pseudo-audio label and the Gaussian noise added data are input into the Dual-Unet model of the diffusion model to obtain the trained denoising model Dual-Unet.

[0101] Step 4: Use the trained audio diffusion model to combine the language audio similarity to screen the speech audio in one step, and obtain the denoised audio;

[0102] Step 4.1: Use the trained audio diffusion model to denoise a plurality of randomly generated Gaussian noises to form a set of one-step denoised audios;

[0103] Step 4.2: Use the language audio similarity as shown in formula (6) to screen the language audio set in one step;

[0104] s=cos(c(a)+f t (t)) (6)

[0105] Wherein f a and f t represent the feature encoders of the audio and pseudo-audio labels respectively;

[0106] Step 4.3: Set the denoising step number M, and execute steps 4.1 to 4.2 in a loop iteration manner until all denoising steps are completed, and obtain the completely denoised audio;

[0107] In the embodiment, the trained diffusion model is used to denoise a plurality of randomly generated Gaussian noises to obtain audio features, and then restore the original audio. The pre-trained model is used to compare and select the audio most related to the audio label and complete the subsequent denoising process.

[0108] In order to further illustrate the superiority of the present application, the description is made in the form of combining experimental data.

[0109] Table 1 Linear probe classification of audio extraction features

[0110] Method Number of symbols Number of quantizer layers Classification accuracy (↑) Baseline system 2048 2 58.13％ Baseline system + classification loss 2048 2 91.65％

[0111] Table 2 Objective evaluation of the number of screening steps on audio quality

[0112]

[0113]

[0114] Table 3 Influence of the number of screening steps on subjective evaluation of audio quality

[0115]

[0116] The results show that the classification loss guided audio codec realized by the present application can make the extracted audio features have obvious semantic distinguishability, and it can be seen from the classification accuracy rate of Table 1 that the audio features have better distinguishability after using the classification loss. It can be seen from Table 2 that the average CLAP score, the average FAD and the number of function evaluations (NFE) all indicate that the more the number of screening steps, the better the objective indicators, but the obvious marginal effect has appeared when the number of screening steps is 1, although the obvious decline has appeared in the improvement of the objective indicators with the significant increase of the number of screening steps. In addition, it can be shown from the audio quality, the category matching degree and the diversity of three subjective indicators in Table 3 that the use of the pre-training model for screening and the screening of only one step can ensure the improvement of the efficiency and the generation quality.

Claims

1. A method for generating onomatopoeic audio based on feature semantic differentiation and single-step model selection, characterized in that: Includes the following steps, Step 1: Extract Mel spectral features from the audio samples, generate the frequency domain spectrum of the audio through short-time Fourier transform, and perform frequency domain mapping and logarithmic compression on the frequency domain spectrum using a Mel filter bank to form a Mel spectrogram; Step 2: Train the audio codec with a multi-level residual quantizer by fusing semantic classification loss and supervised reconstruction loss to obtain semantically discriminative audio features; Step 2.1: Construct an audio codec with a multi-level residual quantizer, utilizing semantic classification loss L. cls With supervision and reconstruction loss L recons The resulting fusion loss is used to optimize the audio codec; Step 2.2: Input the Mel spectrogram into the audio codec for training to obtain semantically discriminative audio features; Step 3: Train the audio diffusion model with multi-task diffusion loss using the onomatopoeic audio tag t and semantically discriminative audio features; Step 3.1: Construct an audio diffusion model with multi-task diffusion loss; Step 3.2: Set the onomatopoeic audio label t, and input the onomatopoeic audio label t and the semantically distinguishable audio features into the audio diffusion model with multi-task diffusion loss for training; Step 4: Use the trained audio diffusion model combined with language audio similarity to perform single-step filtering of speech audio to obtain denoised frequencies.

2. The method for generating simulated audio based on feature semantic differentiation and single-step model selection as described in claim 1, characterized in that: Step 1 is implemented as follows: Step 1.1: Collect audio data to form audio samples; Step 1.2: Extract the Mel spectrum features of the audio samples and obtain the frequency domain spectrum of the audio through Fourier transform; Step 1.3: Use the Mel filter bank to perform frequency domain mapping and logarithmic compression on the frequency domain spectrum to obtain the Mel spectrum.

3. The method for generating simulated audio based on feature semantic differentiation and single-step model selection as described in claim 2, characterized in that: Step 1.2 is implemented as follows: Step 1.2.1: Extract Mel spectrum features of the audio using a residual audio codec guided by classification loss; Step 1.2.2: Obtain the frequency domain spectrum of the audio by using the Mel spectrum features of the audio through short-time Fourier transform.

4. The method for generating simulated audio based on feature semantic differentiation and single-step model selection as described in claim 1, characterized in that: Step 2.1 is implemented as follows: Step 2.1.1: Construct an audio codec consisting of an audio encoder E and an audio decoder D, using a deep convolutional neural network; Step 2.1.2: Configure a multi-level residual quantizer for the audio codec to obtain the quantization error L of the audio features as shown in equation (1). quant ; L quant =∑ l sum||z l –Q l (z l )|| 2 (1) Among them, z l Let Q represent the features of the l-th layer. l Indicates selection z l The quantization function closest to the symbol; Step 2.1.3: Classify the quantized audio features using the nonlinear transformation of a multilayer perceptron (MLP) with added semantic discrimination; Step 2.1.4: Utilize semantic classification loss L cls With supervision and reconstruction loss L recons The resulting fusion loss is used to optimize the audio codec.

5. The method for generating simulated audio based on feature semantic differentiation and single-step model selection as described in claim 4, characterized in that: The implementation method for step 2.1.3 is as follows: Step 2.1.3.1: Obtain the semantic classification loss L for semantic distinction using the method shown in Equation (2). cls ; L cls =∑ i -t i log(MLP(z quant ) (2) Among them, t i It is the tag for the i-th onomatopoeic audio, z quant It is the output after passing through all the layer quantizers; Step 2.1.3.2: Add the semantic classification loss to the multilayer perceptron (MLP); Step 2.1.3.3: Classify audio features using a semantically discriminative multilayer perceptron (MLP).

6. The method for generating simulated audio based on feature semantic differentiation and single-step model selection as described in claim 4, characterized in that: The implementation method for step 2.1.4 is as follows: Step 2.1.4.1: Configure the supervised reconstruction loss L for the audio codec recons ; Step 2.1.4.2: Apply semantic classification loss L cls With supervision and reconstruction loss L recons The fusion is performed, resulting in the fusion loss shown in equation (3); L vae =L recons +λ quant L quant +λ cls L cls (3) Where, λ quant and λ cls To balance the hyperparameters. Step 2.1.4.3: Optimize the audio codec using fusion loss.

7. The method for generating simulated audio based on feature semantic differentiation and single-step model selection as described in claim 1, characterized in that: Step 2.2 is implemented as follows: Step 2.2.1: The Mel spectrogram is compressed using the two-dimensional convolution of the audio encoder E to obtain the audio features z = E(x); Step 2.2.2: Audio features are sampled using the two-dimensional transpose convolution of the audio decoder D to obtain the Mel spectrum of the audio.

8. The method for generating simulated audio based on feature semantic differentiation and single-step model selection as described in claim 1, characterized in that: Step 3.1 is implemented as follows: Step 3.1.1: Construct an audio diffusion model based on the Dual-Unet architecture; Step 3.1.2: Add noise to the audio features with semantic distinction using the linear noise scheduling method shown in Equation (4); Where t∈[1,T] is the number of diffusion steps, β min =0.0001, β max =0.02; Step 3.1.3: Optimize the audio diffusion model using the multi-task diffusion loss shown in Equation (5); L diff =L data +λ quant L noise (5) Where L data It is a loss function for the two-branch data branch, calculating the mean square error between the Unet output and the target data; L noise It is a loss function of the noise branch in a two-branch system, and the mean square error of the Unet output and the target noise is calculated.

9. The method for generating simulated audio based on feature semantic differentiation and single-step model selection as described in claim 1, characterized in that: Step 4 is implemented as follows: Step 4.1: Use the trained audio diffusion model to denoise multiple randomly generated Gaussian noises to form a single-step denoised audio set; Step 4.2: Use the language audio similarity as shown in Equation (6) to perform a single-step screening of the language audio set; s=cos(c(a)+f t (t)) (6) Among them, f a and f t These represent feature encoders for audio and onomatopoeia tags, respectively. Step 4.3: Set the number of denoising steps to M, and execute steps 4.1 to 4.2 in a loop until all denoising steps are completed, and obtain the fully denoised audio.

Images