Audio generation model training method, audio generation method and device

Abstract

The present disclosure provides a training method, an audio generation method, and an apparatus for an audio generation model, belonging to the field of audio technology. Through the above-mentioned training process, a latent space stream matching method can be used to train the generative model. During the model training process, not only is the rapid convergence of the prediction guided by the descriptive text or video, but also, based on the self-attention mechanism of the generative model, a cross-attention mechanism is combined to achieve a simpler expression in the form of a predicted scale spectrum, thereby ensuring the accuracy of the generated content while improving the training efficiency.

Description

Technical Field

[0001] The present disclosure relates to the field of audio technology, and in particular to a training method for an audio generation model, an audio generation method, and an audio generation device. Background Art

[0002] With the rapid development of multimedia technology, audio processing techniques are also advancing, particularly in the fields of text-to-audio (TTA) and video-to-audio (VTA). These technologies have a wide range of applications, from online education and entertainment to virtual reality. However, in the pursuit of a more immersive experience, traditional mono audio output is gradually revealing its limitations, especially in conveying spatial and directional perception. Therefore, generating two-channel stereo sound has become a hot topic of research.

[0003] The current solution is usually to train a sound generation model directly through video conditions or text conditions during TTA or VTA. Figure 1 This method generates stereo audio with azimuth information end-to-end. While this method can meet basic requirements to a certain extent, the lack of sufficient stereo audio for training often results in poor content accuracy in the generated audio. Therefore, a training method for audio generation models that can improve accuracy is urgently needed. Summary of the Invention

[0004] The present disclosure provides a training method, an audio generation method, and an apparatus for an audio generation model, which can improve the efficiency of training while ensuring the accuracy of the generated content. The technical solution of the present disclosure is as follows:

[0005] According to one aspect of an embodiment of the present disclosure, a method for training an audio generation model is provided, comprising:

[0006] Acquire training data, where the training data includes a plurality of two-channel stereo data, and the training data also includes at least one of a sample video from which the plurality of two-channel stereo data are derived and a description text of the sample video;

[0007] Obtaining a proportional spectrum of each of the two-channel stereo data, where the proportional spectrum represents a ratio of a difference to a sum of frequency spectra of left and right channel data in the two-channel stereo data;

[0008] Adding noise to the plurality of ratio spectra based on Gaussian distribution;

[0009] Based on the sample videos and the two-channel stereo data, a plurality of first sample sequences are generated, where the first sample sequences represent the sample videos, the average frequency spectra of the two-channel stereo data, and the ratio spectrum after adding noise;

[0010] generating a second sample sequence based on at least one of the sample video and a description text of the sample video;

[0011] Using the first sample sequence as a main input of a model and the second sample sequence as an auxiliary input of the model, the first sample sequence and the second sample sequence are processed by an audio generative model to obtain a prediction vector field corresponding to each of the first sample sequences;

[0012] The audio generation model is trained based on the noisy scale spectrum, the prediction vector field corresponding to each of the first sample sequences, and the Gaussian distribution.

[0013] In some embodiments, the audio generation model includes a self-attention layer and a cross-attention layer, the query vector, key vector and value vector of the self-attention layer are all obtained based on the first sample sequence, the query vector of the cross-attention layer is obtained based on the first sample sequence, and the key vector and value vector of the cross-attention layer are both obtained based on the second sample sequence.

[0014] In some embodiments, adding noise to the plurality of ratio spectra comprises:

[0015] Noise is added to the plurality of scale spectra based on the Gaussian distribution and a scaling factor selected between 0 and 1.

[0016] In some embodiments, generating a plurality of first sample sequences based on the respective sample videos and the two-channel stereo data comprises:

[0017] generating the proportional spectrum based on a sum and a difference between frequency spectra of left and right channel data of the two-channel stereo data;

[0018] Outputting the mean of the frequency spectra of the left and right channel data of the two-channel stereo data as the average frequency spectrum of the two-channel stereo data;

[0019] Extracting video features of the sample video, and aligning the video features with the frequency spectra of the left and right channel data to obtain aligned video features;

[0020] The scale spectrum, the average spectrum, and the aligned video features are concatenated in dimension to obtain the first sample sequence.

[0021] In some embodiments, generating the second sample sequence based on at least one of a video feature of the sample video and a descriptive text feature of the video includes:

[0022] If the sample video has a description text, extracting the description text features of the sample video through a text encoder;

[0023] The video features of the sample video and the description text features of the sample video are spliced ​​in length to obtain the second sample sequence.

[0024] According to another aspect of an embodiment of the present disclosure, a training device for an audio generation model is provided, comprising:

[0025] A first data acquisition unit is configured to acquire training data, wherein the training data includes a plurality of two-channel stereo data, and the training data also includes at least one of a sample video from which the plurality of two-channel stereo data are sourced and a description text of the sample video;

[0026] a proportional spectrum acquiring unit configured to acquire a proportional spectrum of each of the two-channel stereo data, wherein the proportional spectrum represents a ratio of a difference to a sum of frequency spectra of left and right channel data in the two-channel stereo data;

[0027] a noise adding unit configured to add noise to the plurality of proportional spectra based on Gaussian distribution;

[0028] A first sequence generating unit is configured to generate a plurality of first sample sequences based on the sample videos and the two-channel stereo data, wherein the first sample sequences represent the average spectrum of the sample videos and the two-channel stereo data and the ratio spectrum after adding noise;

[0029] A second sequence generating unit is configured to generate a second sample sequence based on at least one of the sample video and a description text of the sample video;

[0030] The training unit is configured to use the first sample sequence as the main input of the model and the second sample sequence as the auxiliary input of the model, process the first sample sequence and the second sample sequence through the audio generation model to obtain a prediction vector field corresponding to each first sample sequence; and train the audio generation model based on the noisy scale spectrum, the prediction vector field corresponding to each first sample sequence, and the Gaussian distribution.

[0031] In some embodiments, the audio generation model includes a self-attention layer and a cross-attention layer,

[0032] The query vector, key vector and value vector of the self-attention layer are all obtained based on the first sample sequence, the query vector of the cross-attention layer is obtained based on the first sample sequence, and the key vector and value vector of the cross-attention layer are both obtained based on the second sample sequence.

[0033] In some embodiments, the noise adding unit is configured to add noise to the plurality of scale spectra based on the Gaussian distribution and a scaling factor selected from 0 to 1.

[0034] In some embodiments, the first sequence generating unit is configured to generate the proportional spectrum based on the sum and difference between the frequency spectra of the left and right channel data of the two-channel stereo data;

[0035] Outputting the mean of the frequency spectra of the left and right channel data of the two-channel stereo data as the average frequency spectrum of the two-channel stereo data;

[0036] Extracting video features of the sample video, and aligning the video features with the frequency spectra of the left and right channel data to obtain aligned video features;

[0037] The scale spectrum, the average spectrum, and the aligned video features are concatenated in dimension to obtain the first sample sequence.

[0038] In some embodiments, the second sequence generating unit is configured to extract descriptive text features of the sample video through a text encoder if there is descriptive text in the sample video;

[0039] The video features of the sample video and the description text features of the sample video are spliced ​​in length to obtain the second sample sequence.

[0040] According to another aspect of the present disclosure, there is provided an audio generation method, including:

[0041] Get mono audio data;

[0042] performing multiple rounds of iterative processing on the monophonic audio data based on an audio generative model trained based on a latent space stream matching algorithm to obtain a target proportional spectrum, wherein the target proportional spectrum represents a ratio of a difference to a sum of frequency spectra of left and right channel data in the two-channel stereo data to be generated;

[0043] generating left and right channel data based on the target proportional spectrum and the monophonic audio data;

[0044] Based on the left and right channel data, two-channel stereo data corresponding to the mono audio data is obtained.

[0045] In some embodiments, performing multiple rounds of iterative processing on the monophonic audio data based on the audio generation model to obtain a target proportional spectrum includes:

[0046] During each iteration, the scale spectrum obtained in the previous iteration is input into the audio generation model for processing to obtain an intermediate scale spectrum and vector field. The vector field is sampled based on the ODE (Ordinary Differential Equations) solver, and the intermediate scale spectrum is updated based on the sampling points. The scale spectrum and vector field obtained in the current iteration are output.

[0047] According to another aspect of the present disclosure, there is provided an audio generating apparatus, including:

[0048] A second data acquisition unit is configured to acquire mono audio data;

[0049] a proportional spectrum generating unit configured to perform multiple rounds of iterative processing on the monophonic audio data based on an audio generative model trained based on a latent space stream matching algorithm to obtain a target proportional spectrum, wherein the target proportional spectrum represents a ratio of a difference to a sum of frequency spectra of left and right channel data in the two-channel stereo data to be generated;

[0050] a left and right channel generating unit, configured to generate left and right channel data based on the target ratio spectrum and the monophonic audio data;

[0051] The stereo generating unit is configured to obtain two-channel stereo data corresponding to the mono audio data based on the left and right channel data.

[0052] In some embodiments, the scale spectrum generation unit is configured to input the scale spectrum obtained in the previous round of iteration into the audio generation model for processing during each round of iteration to obtain an intermediate scale spectrum and vector field, sample the vector field based on the ODE solver, update the intermediate scale spectrum based on the sampling points, and output the scale spectrum and vector field obtained in this round of iteration.

[0053] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device including:

[0054] one or more processors;

[0055] a memory for storing program codes executable by the processor;

[0056] The processor is configured to execute the program code to implement the above-mentioned audio generation model training method or audio generation method.

[0057] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When the program code in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device can execute the above-mentioned audio generation model training method or audio generation method.

[0058] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program / instruction, which implements the above-mentioned audio generation model training method or audio generation method when executed by a processor.

[0059] Through the above training process, the generative model can be trained using latent space flow matching. During the model training process, not only is the rapid convergence of the prediction guided by the descriptive text or video, but the cross-attention mechanism is also combined with the self-attention mechanism of the generative model to achieve a simpler expression in the form of a predicted scale spectrum, thereby ensuring the accuracy of the generated content while improving the training efficiency.

[0060] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0062] Figure 1 The figure is a schematic diagram of an implementation environment according to an exemplary embodiment.

[0063] Figure 2 The figure is a flowchart of a method for training an audio generation model according to an exemplary embodiment.

[0064] Figure 3 It is a structural diagram of model training according to an exemplary embodiment.

[0065] Figure 4 The figure is a flowchart of an audio generation method according to an exemplary embodiment.

[0066] Figure 5 The figure is a schematic diagram showing stereo generation according to an exemplary embodiment.

[0067] Figure 6 The figure is a block diagram of a training device for an audio generation model according to an exemplary embodiment.

[0068] Figure 7 The figure is a block diagram of an audio generating apparatus according to an exemplary embodiment.

[0069] Figure 8 It is a block diagram of a terminal according to an exemplary embodiment.

[0070] Figure 9The figure is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0071] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0072] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0073] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, storage, and display, etc.), and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the task execution progress and objects involved in this disclosure are obtained with full authorization.

[0074] LFM (latent flow matching) is a generative model training method that efficiently generates new samples by learning the optimal transmission path (flow) between data distributions in the latent space.

[0075] VTA (Video-to-Audio) is a model that generates corresponding audio (such as ambient sound or dialogue) from video. It achieves cross-modal generation from vision to hearing by learning the association between vision and sound.

[0076] Text-to-Audio (TTA) is a technology that generates corresponding audio (such as environmental sound effects, music, or speech) from text. By modeling the relationship between language and sound, it achieves cross-modal generation from text to auditory content.

[0077] Wav-VAE is a variational autoencoder (VAE)-based model for generating and reconstructing audio at the raw waveform level, achieving high-quality sound modeling by learning the latent representation of audio data.

[0078] Figure 1FIG. 1 is a schematic diagram of an implementation environment according to an exemplary embodiment. Taking the electronic device as a terminal as an example, see Figure 1 The implementation environment specifically includes: a terminal 101 and a server 102. The terminal 101 and the server 102 can be directly or indirectly connected via wired or wireless communication, which is not limited in the present disclosure.

[0079] Terminal 101 is at least one of a smartphone, smartwatch, desktop computer, laptop computer, MP3 player, MP4 player, and portable computer. Terminal 101 runs an application that supports model training. Users can log in to this application through terminal 101 to access services provided by this application. For example, users can perform model training through this application on terminal 101. Of course, terminal 101 can also run an application that supports the invocation of an audio generation model, enabling the generation of stereo audio data by invoking the audio generation model.

[0080] Terminal 101 generally refers to one of multiple terminals. This embodiment uses terminal 101 as an example. Those skilled in the art will appreciate that the number of terminals may be greater or lesser. For example, there may be a few terminals, or dozens, hundreds, or even more. This embodiment does not limit the number or device type of terminals.

[0081] The server 102 is at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The server 102 is used to provide background services for model training. The server 102 can be connected to the terminal 101 via a wireless network or a wired network. The server 102 can provide database services, etc. for the terminal. Of course, it can also provide other types of services, such as data preprocessing, etc. In some embodiments, the number of the above-mentioned servers can be more or less, and the embodiments of the present disclosure are not limited to this. Of course, the server 102 also includes other functional servers to provide more comprehensive and diversified services. Of course, in other embodiments, the server 102 can also be used as a carrier for model training to implement the model training process of the terminal 101 mentioned above, so as to provide the trained audio generation model to the terminal for use.

[0082] Figure 2 is a flowchart of a method for training an audio generation model provided in an embodiment of the present application. Figure 3 FIG. 1 is a structural diagram of a model training according to an exemplary embodiment. Figure 2 and Figure 3 , the method includes the following steps.

[0083] In step 201, training data is acquired, where the training data includes a plurality of two-channel stereo data and at least one of sample videos from which the plurality of two-channel stereo data are derived and description texts of the sample videos.

[0084] The training data may include multiple batches of training data, each batch of training data including multiple binaural stereo data. The training data may also include at least one of sample videos and descriptive text from which the binaural stereo data originates. During the model training process, a batch of training data may be input each time to perform an iteration, thereby adjusting the parameters of the audio generation model.

[0085] In step 202 , a proportional spectrum of the two-channel stereo data of each sample video is obtained, where the proportional spectrum represents the ratio of the difference to the sum of the frequency spectra of the left and right channel data in the two-channel stereo data.

[0086] In an embodiment of the present application, spectra are extracted for the left and right channel data of the two-channel stereo data. The spectra can be in the form of mel-spectrograms or linear spectra. After the extracted spectra are obtained, a proportional spectrum can be obtained based on the spectra of the left and right channel data. For example, using mel-spectrograms as an example, the formula for the proportional spectrum can be expressed as follows: Formula 1.

[0087]

[0088] Subratio Used to represent the ratio spectrum, Used to represent the left channel data in two-channel stereo data. Used to represent the right channel data in two-channel stereo data.

[0089] The proportional spectrum calculated by formula 1 has a value between -1 and 1, which is convenient for model prediction. In addition, the size of the proportional spectrum is the same as that of the single-channel spectrum, and the prediction difficulty is lower than that of simultaneously predicting the dual-channel spectrum, which greatly reduces the calculation difficulty.

[0090] In step 203 , noise is added to the multiple ratio spectra based on Gaussian distribution.

[0091] In some embodiments, adding noise to the plurality of scale spectra includes adding noise to the plurality of scale spectra based on a Gaussian distribution and a scaling factor selected from 0 to 1.

[0092] It should be noted that in the embodiments of this application, LFM (Latent Flow Matching) is used to train the audio generative model. This is equivalent to learning a continuous probability stream to transform a simple prior distribution (such as a standard Gaussian distribution) into a complex data distribution. During training, a "noise-and-denoise" process is used to gradually add noise to the input until it becomes pure noise. The inference process of the trained audio generative model can be considered the process of denoising the input. To achieve this, it is necessary to add noise to the samples during training to obtain the model input.

[0093] For the distribution from the latent space The data sampled in , the flow matching model defines a Gaussian distribution arrive The probability path , in the form of an ordinary differential equation (ODE): ,in is a time-dependent vector field, and . Probability Path Describes the time Time latent variables The probability distribution of In the embodiment of the present application, an optimal transmission path is adopted, which is defined by the linear transformation between the data distribution and the Gaussian distribution as shown in the following formula 2.

[0094] Formula 2:

[0095] in, is a small constant, is the above-mentioned ratio spectrum, t is the cosine distribution in the interval A time of random sampling, which is the scaling factor, is a Gaussian distribution, is the ratio spectrum after adding noise.

[0096] In addition, using the Optimal Transport technique to construct probabilistic paths can further speed up training and improve the generalization ability of the model.

[0097] In step 204, a plurality of first sample sequences are generated based on the sample videos and the two-channel stereo data. The first sample sequences represent the average spectrum and the ratio spectrum after adding noise of the sample videos and the two-channel stereo data.

[0098] When extracting video features from the sample video, a video encoder can be used. The most basic clip model can be used as the video encoder to reduce the overall training cost. It is understandable that if the training data does not include the sample video, there is no need to splice the video features of the sample video, and the preset features can be used directly to complete the video features.

[0099] In some embodiments, when generating the first sample sequence, the following process can be adopted: the mean of the frequency spectra of the left and right channel data of the two-channel stereo data is output as the average frequency spectrum of the two-channel stereo data; the video features of the sample video are extracted, and the video features are aligned with the frequency spectra of the left and right channel data to obtain aligned video features; the proportional spectrum, the average spectrum and the aligned video features are spliced ​​in the dimension to obtain the first sample sequence.

[0100] The video features may originally have a small dimension, which may make them impossible to stitch. To achieve the purpose of stitching, the video features and the spectrum can be aligned to achieve the purpose of stitching. This alignment can be achieved by interpolating the video features, which is not limited in the present embodiment.

[0101] In step 205, a second sample sequence is generated based on at least one of a video feature of the sample video and a descriptive text feature of the video.

[0102] In step 205, if there is a descriptive text in the sample video, the descriptive text features of the sample video are extracted through a text encoder; the video features of the sample video and the descriptive text features of the sample video are spliced ​​in length to obtain a second sample sequence. Among them, the text encoder can adopt the most basic T5 model to reduce the complexity during training. For video features, they can be reduced in dimension to achieve splicing with descriptive text features to avoid phenomena such as excessive length. It is understandable that if the training data does not include sample videos or descriptive texts, the second sample sequence can be obtained based on the included features. In some embodiments, if necessary, the features of insufficient length can be padded to obtain a second sample sequence of the required length.

[0103] In step 206, the first sample sequence is used as the main input of the model and the second sample sequence is used as the auxiliary input of the model. The first sample sequence and the second sample sequence are processed by the audio generation model to obtain the prediction vector field corresponding to each first sample sequence.

[0104] In some embodiments, the audio generation model includes a self-attention layer and a cross-attention layer. The query vector, key vector, and value vector of the self-attention layer are all derived based on the first sample sequence. The query vector of the cross-attention layer is derived based on the first sample sequence, and the key vector and value vector of the cross-attention layer are both derived based on the second sample sequence. By combining the self-attention and cross-attention mechanisms for processing, long-range dependencies within the sequence can be taken into account, associations can be established across different time steps and feature dimensions, and long-term dependencies in the data can be captured, thereby better estimating the vector field. The system can also consider interactions between different input sequences, enhancing guidance through the processing of multimodal information, and thus generating a more accurate vector field.

[0105] It should be noted that the above-mentioned first sample sequence, as the main input of the audio generation model, participates in the calculation of each layer of the audio generation model, while the second sample sequence, as an auxiliary input, only participates in the calculation in the cross-attention layer, and is mainly used to generate the key vector and value vector in the cross-attention layer. Since the second sample sequence includes at least one of the descriptive text features and video features of the sample video, it can guide the diffusion process to converge in the case of multimodality, thereby improving the controllability of the generation.

[0106] The audio generation model is a generative model. It should be noted that the audio generation model can be understood as a velocity field, which can obtain the model prediction vector field based on the noisy samples after random noise addition. Its input is the first sample sequence as the noisy samples and the second sample sequence as the constraint condition, and its output is the predicted vector field. In addition, the random selection in the noise addition process is t , will also serve as the input of the transformer. For example, the audio generation model can be DiT (Diffusion Transformer), and its basic architecture is a transformer. The transformer includes an input layer, a self-attention layer, a cross-attention layer, and an output layer, wherein the input layer can encode the input and perform position encoding and other processing, and the self-attention layer can execute the self-attention mechanism based on the first sample sequence processed by the input layer. Accordingly, it can also include a feedforward network, etc., for further processing the output. After the self-attention layer is processed, the cross-attention layer is based on the output of the self-attention layer and uses the second sample sequence as the driving condition to execute the cross-attention mechanism, integrating the conditional information into the processing process of the model to obtain the cross-attention feature, and through the processing of the output layer, to output the predicted vector field. It should be noted that technicians can control the dimension of the model, the depth of the model and other parameters according to actual needs, so as to control the size of the model, and the embodiments of the present application are not limited to this.

[0107] In step 207, the audio generation model is trained based on the noisy scale spectrum, the prediction vector field corresponding to each first sample sequence, and the Gaussian distribution.

[0108] It should be noted that during the training process of the audio generation model, the time is learned by minimizing the following objective function The vector field at :

[0109] Objective function:

[0110] in, is a parameterized representation of the audio generation model, Refers to finding the parameters that minimize the expression , It's about time and expectations, represents the prediction vector field, and It's time The target vector field at .

[0111] The goal of flow matching is to learn a vector field function so that starting from any initial point, the flow along this vector field can eventually reach the target distribution, which is the Gaussian distribution in this embodiment of the application. Therefore, the purpose of the above objective function is to minimize the mean square error between the predicted vector field and the target vector field, and it can refer to the predicted vector field and time. The difference between the target vector fields at and the optimization target is used to update the model parameters through the back propagation algorithm, so that the model can gradually learn the correct vector field, thereby achieving accurate mapping from noise distribution to data distribution, and making the vector field predicted by the model able to describe the real flow of data as accurately as possible.

[0112] Through the above training process, the generative model can be trained using latent space flow matching. During the model training process, not only is the rapid convergence of the prediction guided by the descriptive text or video, but the cross-attention mechanism is also combined with the self-attention mechanism of the generative model to achieve a simpler expression in the form of a predicted scale spectrum, thereby ensuring the accuracy of the generated content while improving the training efficiency.

[0113] In addition, the improved training efficiency of this application not only solves the current problem of insufficient training data, but also provides the ability to flexibly adjust sound characteristics in different scenarios.

[0114] Figure 4 This is a flowchart of an audio generation method provided by an embodiment of the present application, see Figure 4 , the method includes the following steps.

[0115] In step 401, mono audio data is acquired.

[0116] In the embodiment of the present application, the mono audio data may refer to any type of audio data, and the embodiment of the present application does not limit this. In some embodiments, if the mono audio data has a source video, the source video may also be obtained as part of the input.

[0117] That is, the input sequence is obtained by splicing the spectrum of the monophonic audio data and the video features of the source video. Of course, in order to obtain input sequences of equal length, a sequence of a preset length can also be spliced. In addition, in the embodiment of the present application, the audio generation model also has the following features: Figure 2 In the embodiment shown, the first sample sequence and the second sample sequence are input, where the input sequence is equivalent to the first sample sequence, and the second sample sequence used as a condition may also include at least one of the video features and the descriptive text features of the source video of the mono audio data. The processing process in the model is the same as the above steps and will not be repeated here.

[0118] In step 402, the monophonic audio data is iteratively processed for multiple rounds based on an audio generation model to obtain a target proportional spectrum. The audio generation model is trained based on a latent space stream matching algorithm. The target proportional spectrum represents the ratio of the difference and sum between the frequency spectra of the left and right channel data in the two-channel stereo data to be generated.

[0119] In an embodiment of the present application, the audio generative model can obtain the target scaled spectrum through multiple rounds of iterative processing. During each round of iteration, the scaled spectrum obtained in the previous round of iteration is input into the audio generative model for processing to obtain an intermediate scaled spectrum and vector field. The vector field is sampled using an ODE solver, the intermediate scaled spectrum is updated based on the sampling points, and the scaled spectrum and vector field obtained in the current round of iteration are output.

[0120] The above process involves updating the output to generate new input for the model. For a given iteration, the scale spectrum obtained in the previous iteration is input into the audio generative model for processing. The audio generative model, also known as a velocity field model, can output both the scale spectrum and the corresponding vector field. The ODE solver uses the vector field to calculate the sample position or sample difference at the next time t2 based on the current sample position and time t1. Based on this sample position or sample difference, it updates the intermediate scale spectrum, which serves as the input for the next iteration. Through continuous iteration, the ODE solver iteratively processes the input mono audio data along the path defined by the vector field to obtain the target scale spectrum. This process can be viewed as the vector field model and ODE solver gradually denoising the input scale spectrum in reverse time order, ultimately generating the target scale spectrum at time 0.

[0121] In step 403, left and right channel data are generated based on the target ratio spectrum and the monophonic audio data.

[0122] In the embodiment of the present application, when the target proportional spectrum is obtained, taking the spectrum as a Mel spectrum as an example, the Mel spectrum of the left and right channels can be calculated using the following formulas 3 and 4.

[0123] Formula 3:

[0124] Formula 4 ：

[0125] in, mel is the Mel spectrum of mono audio data, is the generated left channel data, The generated right channel data.

[0126] Based on the target ratio spectrum and monophonic audio data, we can generate left and right channel spectra. This spectrum is then converted into waveform audio, yielding left and right channel data. This waveform is more resistant to noise than WAV-VAE. For waveform conversion, lightweight vocoders such as BigVGAN, HiFi-GAN, Parallel WaveGAN, nsf-HifiGAN, or Vocos can be used to ensure real-time performance and sound quality.

[0127] In step 404, two-channel stereo data corresponding to the mono audio data is obtained based on the left and right channel data.

[0128] For the left and right channel data, the waveforms are spliced ​​in the dimension to obtain the two-channel stereo data. This process can be done by Figure 5 The process shown is embodied.

[0129] In the technical solution provided in the embodiment of the present application, an audio generation model trained by a latent spatial stream matching algorithm is utilized to process given mono audio data under the condition of describing text or video to obtain a proportional spectrum that can represent the left and right channel data. This ensures the accuracy of the audio content while retaining the quality of the original audio and enhances the spatial sense and realism of the user experience.

[0130] During the inference process, experiments have shown that the audio generation method adopted in the embodiment of the present application uses the Euler ODE solver to generate the required samples from the Gaussian distribution in only 10 steps, which greatly improves the inference efficiency.

[0131] Figure 6 FIG1 is a block diagram of a training device for an audio generation model according to an exemplary embodiment. Figure 6 , the training device of the audio generation model includes:

[0132] A first data acquisition unit 601 is configured to acquire training data, wherein the training data includes a plurality of two-channel stereo data, and the training data also includes at least one of a sample video from which the plurality of two-channel stereo data are derived and a description text of the sample video;

[0133] a proportional spectrum acquiring unit 602 configured to acquire a proportional spectrum of each of the two-channel stereo data, wherein the proportional spectrum represents a ratio of a difference to a sum of frequency spectra of left and right channel data in the two-channel stereo data;

[0134] a noise adding unit 603 configured to add noise to the plurality of proportional spectra based on Gaussian distribution;

[0135] A first sequence generating unit 604 is configured to generate a plurality of first sample sequences based on the sample videos and the two-channel stereo data, wherein the first sample sequences represent the average spectrum and the noise-added proportional spectrum of the sample videos and the two-channel stereo data;

[0136] A second sequence generating unit 605 is configured to generate a second sample sequence based on at least one of the sample video and the description text of the sample video;

[0137] The training unit 606 is configured to use the first sample sequence as the main input of the model and the second sample sequence as the auxiliary input of the model, process the first sample sequence and the second sample sequence through the audio generation model to obtain a prediction vector field corresponding to each first sample sequence; and train the audio generation model based on the noisy scale spectrum, the prediction vector field corresponding to each first sample sequence, and the Gaussian distribution.

[0138] In some embodiments, the audio generation model includes a self-attention layer and a cross-attention layer, the query vector, key vector and value vector of the self-attention layer are all obtained based on the first sample sequence, the query vector of the cross-attention layer is obtained based on the first sample sequence, and the key vector and value vector of the cross-attention layer are both obtained based on the second sample sequence.

[0139] In some embodiments, the noise adding unit is configured to add noise to the plurality of scale spectra based on a Gaussian distribution and a scaling factor selected from 0 to 1.

[0140] In some embodiments, the first sequence generating unit is configured to generate the proportional spectrum based on the sum and difference between the frequency spectra of the left and right channel data of the two-channel stereo data;

[0141] Outputting the mean of the frequency spectra of the left and right channel data of the two-channel stereo data as the average frequency spectrum of the two-channel stereo data;

[0142] Extracting video features of the sample video, and aligning the video features with the frequency spectra of the left and right channel data to obtain aligned video features;

[0143] The scale spectrum, the average spectrum, and the aligned video features are concatenated in dimension to obtain the first sample sequence.

[0144] In some embodiments, the second sequence generating unit is configured to extract descriptive text features of the sample video through a text encoder if there is descriptive text in the sample video;

[0145] The video features of the sample video and the description text features of the sample video are spliced ​​in length to obtain the second sample sequence.

[0146] It should be noted that the training device for the audio generation model provided in the above embodiment only uses the division of the above functional units as an example to illustrate the training of the audio generation model. In actual applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device can be divided into different functional units to complete all or part of the functions described above. In addition, the training device for the audio generation model provided in the above embodiment and the training method embodiment of the audio generation model are of the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0147] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0148] According to another aspect of the present disclosure, there is provided an audio generating device, referring to Figure 7 , the device comprises:

[0149] The second data acquisition unit 701 is configured to acquire mono audio data;

[0150] a proportional spectrum generating unit 702 configured to perform multiple rounds of iterative processing on the monophonic audio data based on an audio generative model trained based on a latent space stream matching algorithm to obtain a target proportional spectrum, wherein the target proportional spectrum represents a ratio of a difference to a sum of frequency spectra of left and right channel data in the two-channel stereo data to be generated;

[0151] A left and right channel generating unit 703 is configured to generate left and right channel data based on the target ratio spectrum and the monophonic audio data;

[0152] The stereo sound generating unit 704 is configured to obtain two-channel stereo sound data corresponding to the monophonic audio data based on the left and right channel data.

[0153] In some embodiments, the scale spectrum generation unit is configured to input the scale spectrum obtained in the previous round of iteration into the audio generation model for processing during each round of iteration to obtain an intermediate scale spectrum and vector field, sample the vector field based on the ODE solver, update the intermediate scale spectrum based on the sampling points, and output the scale spectrum and vector field obtained in this round of iteration.

[0154] It should be noted that the audio generation device provided in the above embodiment only uses the division of the above functional units as an example to illustrate audio generation. In actual applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device can be divided into different functional units to complete all or part of the functions described above. In addition, the audio generation device provided in the above embodiment and the audio generation method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0155] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0156] When an electronic device is provided as a terminal, Figure 8 FIG. 8 is a block diagram of a terminal 800 according to an exemplary embodiment. Figure 8The following is a block diagram illustrating the structure of a terminal 800 provided by an exemplary embodiment of the present disclosure. Terminal 800 may be a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. Terminal 800 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other similar names.

[0157] Typically, the terminal 800 includes a processor 801 and a memory 802 .

[0158] Processor 801 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 801 may be implemented in hardware using at least one of the following: a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), or a PLA (Programmable Logic Array). Processor 801 may also include a main processor and a coprocessor. The main processor is used to process data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 801 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing content displayed on the display screen. In some embodiments, processor 801 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0159] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 802 is used to store at least one computer program, which is executed by the processor 801 to implement the method provided in the method embodiment of the present application.

[0160] In some embodiments, terminal 800 may optionally include a peripheral device interface 803 and at least one peripheral device. The processor 801, memory 802, and peripheral device interface 803 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 803 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 804, a display screen 805, a camera assembly 806, an audio circuit 807, and a power supply 808.

[0161] The peripheral device interface 803 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 801 and the memory 802. In some embodiments, the processor 801, the memory 802, and the peripheral device interface 803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 801, the memory 802, and the peripheral device interface 803 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0162] The RF circuit 804 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 804 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. In some embodiments, the RF circuit 804 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The RF circuit 804 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 804 may also include circuitry related to Near Field Communication (NFC), although this application does not limit this.

[0163] Display screen 805 is used to display a user interface (UI). This UI may include graphics, text, icons, videos, or any combination thereof. If display screen 805 is a touchscreen display, it is also capable of detecting touch signals on or above the surface of display screen 805. These touch signals can be input as control signals to processor 801 for processing. Display screen 805 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be a single display screen 805, located on the front panel of terminal 800. In other embodiments, there may be at least two display screens 805, located on different surfaces of terminal 800 or in a foldable design. In still other embodiments, display screen 805 may be a flexible display, located on a curved or foldable surface of terminal 800. Display screen 805 can also be configured as a non-rectangular, irregular shape, also known as a special-shaped screen. Display screen 805 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0164] The camera assembly 806 is used to capture images or videos. In some embodiments, the camera assembly 806 includes a front camera and a rear camera. Typically, the front camera is provided on the front panel of the terminal, and the rear camera is provided on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 806 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0165] The audio circuit 807 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input into the processor 801 for processing, or input into the radio frequency circuit 804 to achieve voice communication. For the purpose of stereo sound collection or noise reduction, there may be multiple microphones, each located in different parts of the terminal 800. The microphone may also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert electrical signals from the processor 801 or the radio frequency circuit 804 into sound waves. The speaker may be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 807 may also include a headphone jack.

[0166] Power supply 808 is used to power various components in terminal 800. Power supply 808 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 808 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is charged via a wired line, while a wireless rechargeable battery is charged via a wireless coil. The rechargeable battery can also support fast charging technology.

[0167] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation on the terminal 800, and the terminal 800 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0168] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions. The instructions can be executed by an electronic device, such as processor 801 of terminal 800, to perform the above-described method for executing a task related to a network activity. Alternatively, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, or the like.

[0169] The above-mentioned computer device can also be implemented as a server. The structure of the server is introduced below:

[0170] Figure 9This is a schematic diagram of the structure of a server provided in an embodiment of the present application. The server 900 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 901 and one or more memories 902, wherein the one or more memories 902 store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 901 to implement the methods provided in the above-mentioned various method embodiments. Of course, the server 900 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 900 may also include other components for implementing device functions, which will not be described in detail here.

[0171] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a computer program. The computer program can be executed by a processor to perform the method in the above embodiment. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, or an optical data storage device.

[0172] In an exemplary embodiment, a computer program product or computer program is also provided, which includes program code, which is stored in a computer-readable storage medium. A processor of a computer device reads the program code from the computer-readable storage medium, and the processor executes the program code, so that the computer device performs the above method.

[0173] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0174] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for an audio generation model, characterized in that: The method comprises: Acquire training data, where the training data includes a plurality of two-channel stereo data, and the training data also includes at least one of a sample video from which the plurality of two-channel stereo data are derived and a description text of the sample video; Obtaining a proportional spectrum of each of the two-channel stereo data, where the proportional spectrum represents a ratio of a difference to a sum of frequency spectra of left and right channel data in the two-channel stereo data; Adding noise to the plurality of ratio spectra based on Gaussian distribution; Based on the sample videos and the two-channel stereo data, a plurality of first sample sequences are generated, where the first sample sequences represent the sample videos, the average frequency spectra of the two-channel stereo data, and the ratio spectrum after adding noise; generating a second sample sequence based on at least one of the sample video and a description text of the sample video; Using the first sample sequence as a main input of a model and the second sample sequence as an auxiliary input of the model, the first sample sequence and the second sample sequence are processed by an audio generative model to obtain a prediction vector field corresponding to each of the first sample sequences; The audio generation model is trained based on the noisy scale spectrum, the prediction vector field corresponding to each of the first sample sequences, and the Gaussian distribution.

2. The method for training an audio generation model according to claim 1, wherein: The audio generation model includes a self-attention layer and a cross-attention layer. The query vector, key vector and value vector of the self-attention layer are all obtained based on the first sample sequence. The query vector of the cross-attention layer is obtained based on the first sample sequence. The key vector and value vector of the cross-attention layer are both obtained based on the second sample sequence.

3. The method for training an audio generation model according to claim 1, wherein: The adding noise to the plurality of proportional spectra comprises: Noise is added to the plurality of scale spectra based on the Gaussian distribution and a scaling factor selected between 0 and 1.

4. The method for training an audio generation model according to claim 1, wherein: The generating a plurality of first sample sequences based on the respective sample videos and the two-channel stereo data comprises: generating the proportional spectrum based on a sum and a difference between frequency spectra of left and right channel data of the two-channel stereo data; Outputting the mean of the frequency spectra of the left and right channel data of the two-channel stereo data as the average frequency spectrum of the two-channel stereo data; Extracting video features of the sample video, and aligning the video features with the frequency spectra of the left and right channel data to obtain aligned video features; The scale spectrum, the average spectrum, and the aligned video features are concatenated in dimension to obtain the first sample sequence.

5. The method for training an audio generation model according to claim 1, wherein: Generating a second sample sequence based on at least one of a video feature of the sample video and a descriptive text feature of the video includes: If the sample video has a description text, extracting the description text features of the sample video through a text encoder; The video features of the sample video and the description text features of the sample video are spliced ​​in length to obtain the second sample sequence.

6. An audio generation method, characterized in that: include: Get mono audio data; performing multiple rounds of iterative processing on the monophonic audio data based on an audio generative model trained based on a latent space stream matching algorithm to obtain a target proportional spectrum, wherein the target proportional spectrum represents a ratio of a difference to a sum of frequency spectra of left and right channel data in the two-channel stereo data to be generated; generating left and right channel data based on the target proportional spectrum and the monophonic audio data; Based on the left and right channel data, two-channel stereo data corresponding to the mono audio data is obtained.

7. The audio generation method according to claim 6, characterized in that The performing multiple rounds of iterative processing on the monophonic audio data based on the audio generation model to obtain a target proportional spectrum includes: During each iteration, the scale spectrum obtained in the previous iteration is input into the audio generation model for processing to obtain an intermediate scale spectrum and vector field. The vector field is sampled based on the ordinary differential equation (ODE) solver, and the intermediate scale spectrum is updated based on the sampling points. The scale spectrum and vector field obtained in this iteration are output.

8. A training device for an audio generation model, characterized in that: include: A first data acquisition unit is configured to acquire training data, wherein the training data includes a plurality of two-channel stereo data, and the training data also includes at least one of a sample video from which the plurality of two-channel stereo data are sourced and a description text of the sample video; a proportional spectrum acquiring unit configured to acquire a proportional spectrum of each of the two-channel stereo data, wherein the proportional spectrum represents a ratio of a difference to a sum of frequency spectra of left and right channel data in the two-channel stereo data; a noise adding unit configured to add noise to the plurality of proportional spectra based on Gaussian distribution; A first sequence generating unit is configured to generate a plurality of first sample sequences based on the sample videos and the two-channel stereo data, wherein the first sample sequences represent the average spectrum of the sample videos and the two-channel stereo data and the ratio spectrum after adding noise; A second sequence generating unit is configured to generate a second sample sequence based on at least one of the sample video and a description text of the sample video; a training unit configured to use the first sample sequence as a main input of a model and the second sample sequence as an auxiliary input of the model, process the first sample sequence and the second sample sequence through an audio generative model, and obtain a prediction vector field corresponding to each of the first sample sequences; The audio generation model is trained based on the noisy scale spectrum, the prediction vector field corresponding to each of the first sample sequences, and the Gaussian distribution.

9. An audio generating device, characterized in that: include: A second data acquisition unit is configured to acquire mono audio data; a proportional spectrum generating unit configured to perform multiple rounds of iterative processing on the monophonic audio data based on an audio generative model trained based on a latent space stream matching algorithm to obtain a target proportional spectrum, wherein the target proportional spectrum represents a ratio of a difference to a sum of frequency spectra of left and right channel data in the two-channel stereo data to be generated; a left and right channel generating unit, configured to generate left and right channel data based on the target ratio spectrum and the monophonic audio data; The stereo sound generating unit is configured to obtain two-channel stereo sound data corresponding to the monophonic audio data based on the left and right channel data.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing program code executable by the processor; The processor is configured to execute the program code to implement the training method of the audio generation model as described in any one of claims 1 to 5; or the audio generation method as described in claim 6 or 7.

11. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the training method of the audio generation model according to any one of claims 1 to 5; Or, the audio generation method as described in claim 6 or 7.

12. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the training method of the audio generation model as described in any one of claims 1 to 5; or the audio generation method as described in claim 6 or 7.

Images