Audio processing method and device, electronic equipment and storage medium

Abstract

The invention provides an audio processing method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the time-frequency feature extraction of a target mixed audio, and obtaining a target time-frequency feature; the target mixed audio comprises a first target audio and a second target audio; inputting the target time-frequency feature into an audio separation model to obtain a first time-frequency feature of the first target audio; inputting the first time-frequency feature and the target time-frequency feature as conditions into a target diffusion model to obtain a second time-frequency feature of the first target audio; and obtaining the first target audio and the second target audio according to the second time frequency feature. According to the method, the audio separation model is firstly used for performing coarse separation, and then on the basis of the coarse separation result and the original mixed audio, fine correction is performed on the coarse separation result by using the strong data distribution learning capability and the high-fidelity generation characteristic of the diffusion model, so that the audio separation quality is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to an audio processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Multi-source separation technology refers to separating audio from different sound sources in mixed audio, and it is an important research direction in the field of signal processing. In the field of speech signal processing, multi-source separation technology has become an important foundation for applications such as speech recognition, speech enhancement, and speech synthesis.

[0003] In related technologies, neural network models, such as band-split RNN (BSRNN) models, are trained to implicitly learn the differences in audio spectrum distribution, temporal correlation, and harmonic structure of different sound sources in a data-driven manner. Then, based on the differences, mask estimation or target sound source reconstruction is performed to output the separation result in one go. However, this method has poor generalization ability, which makes it easy to reduce the separation quality when processing complex audio. Summary of the Invention

[0004] This invention provides an audio processing method, apparatus, electronic device, and storage medium to address the shortcomings of low audio separation quality in the prior art and improve audio separation quality.

[0005] This invention provides an audio processing method, comprising: Time-frequency features are extracted from the target mixed audio to obtain target time-frequency features; the target mixed audio includes a first target audio and a second target audio. The target time-frequency features are input into the audio separation model to obtain the first time-frequency features of the first target audio. The first time-frequency feature and the target time-frequency feature are used as conditions input into the target diffusion model to obtain the second time-frequency feature of the first target audio. Based on the second time-frequency feature, the first target audio and the second target audio are obtained.

[0006] According to an audio processing method provided by the present invention, the step of inputting the first time-frequency feature and the target time-frequency feature as conditions into a target diffusion model to obtain a second time-frequency feature of the first target audio includes: Based on the first time-frequency feature and the target time-frequency feature, the target condition information is determined; The target condition information and noise signal are input into the target diffusion model to obtain the second time-frequency feature.

[0007] According to an audio processing method provided by the present invention, determining target condition information based on the first time-frequency feature and the target time-frequency feature includes: The first time-frequency feature and the target time-frequency feature are input into the feature encoding module to obtain a first encoded feature and a second encoded feature; the first encoded feature is the encoded feature corresponding to the first time-frequency feature, and the second encoded feature is the encoded feature corresponding to the target time-frequency feature; The first encoded feature and the second encoded feature are input into the attention module to obtain the target condition information.

[0008] According to an audio processing method provided by the present invention, the step of inputting the first time-frequency feature and the target time-frequency feature into a feature encoding module to obtain a first encoded feature and a second encoded feature includes: The first time-frequency feature is input into the first encoding layer of the feature encoding module to obtain the first encoded feature; The target time-frequency features are input into the second coding layer of the feature coding module to obtain the second coded features; The first coding layer and the second coding layer share weight parameters.

[0009] According to an audio processing method provided by the present invention, the step of inputting the first encoded feature and the second encoded feature into an attention module to obtain the target condition information includes: The first encoded feature and the second encoded feature are input into the weight calculation unit of the attention module to obtain the target weight matrix; The first encoded feature and the second encoded feature are input into the value feature calculation layer of the attention module to obtain the first value feature and the second value feature; the first value feature is the value feature corresponding to the first time-frequency feature, and the second value feature is the value feature corresponding to the target time-frequency feature; The target weight matrix, the first value feature, and the second value feature are input into the fusion layer of the attention module to obtain the conditional information corresponding to the first time-frequency feature and the conditional information corresponding to the target time-frequency feature; The target condition information is obtained based on the condition information corresponding to the first time-frequency feature and the condition information corresponding to the target time-frequency feature.

[0010] According to an audio processing method provided by the present invention, the step of inputting the target weight matrix, the first value feature, and the second value feature into the fusion layer of the attention module to obtain conditional information corresponding to the first time-frequency feature and conditional information corresponding to the target time-frequency feature includes: The target weight matrix and the first value feature are input into the first fusion sub-layer of the fusion layer to obtain the conditional information corresponding to the first time-frequency feature; The target weight matrix and the second value feature are input into the second fusion sub-layer of the fusion layer to obtain the conditional information corresponding to the target time-frequency feature.

[0011] According to an audio processing method provided by the present invention, the weight calculation unit includes a query feature calculation layer, a key feature calculation layer, and a weight calculation layer; The step of inputting the first encoded feature and the second encoded feature into the weight calculation unit of the attention module to obtain the target weight matrix includes: At the channel dimension, the first encoded feature and the second encoded feature are concatenated to obtain the merged feature; The merged features are input into the query feature calculation layer and the key feature calculation layer to obtain query features and key features; The query features and the key features are input into the weight calculation layer to obtain the target weight matrix.

[0012] According to an audio processing method provided by the present invention, the target diffusion model is trained based on the following steps: Construct the initial model corresponding to the target diffusion model; Obtain the second time-frequency feature of the first sample audio in the sample mixed audio output by the initial model; Based on the second time-frequency feature of the first sample audio, obtain the estimated value of the first sample audio and the estimated value of the second sample audio in the sample mixed audio; Based on the estimated value of the first sample audio, the label value of the first sample audio, the estimated value of the second sample audio, and the label value of the second sample audio, the initial model is iteratively trained to obtain the target diffusion model.

[0013] The present invention also provides an audio processing apparatus, comprising: A time-frequency feature extraction unit is used to extract time-frequency features from the target mixed audio to obtain target time-frequency features; the target mixed audio includes a first target audio and a second target audio. The first separation unit is used to input the target time-frequency features into the audio separation model to obtain the first time-frequency features of the first target audio. The correction unit is used to input the first time-frequency feature and the target time-frequency feature as conditions into the target diffusion model to obtain the second time-frequency feature of the first target audio. The second separation unit is used to acquire the first target audio and the second target audio based on the second time-frequency characteristics.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the audio processing methods described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the audio processing method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the audio processing method described above.

[0017] The audio processing method, apparatus, electronic device, and storage medium provided by this invention extract time-frequency features from the target mixed audio and input them into an audio separation model to obtain a coarse estimate of the time-frequency features of the first target audio. Then, using the coarse estimate and the time-frequency features of the target mixed audio as conditions, the target diffusion model is guided to perform conditional generative repair on the coarse estimate of the time-frequency features to obtain a high-quality second time-frequency feature. Finally, the first target audio and the second target audio are separated based on the second time-frequency feature. Thus, through a two-stage architecture of separation followed by compensation, the powerful data distribution learning ability and high-fidelity generation characteristics of the diffusion model are used to finely correct the separation results of the first stage, effectively solving the problems of incomplete separation, severe sound quality loss, and insufficient generalization ability. Furthermore, by using the coarse estimate and the target mixed features as conditional inputs, the learning difficulty of the diffusion model is reduced and the generated results are prevented from deviating from the real audio, significantly improving the separation purity and fidelity of the first target audio. At the same time, it is ensured that the second target audio can be completely and accurately reconstructed through linear subtraction, achieving high-quality audio separation from end to end and exhibiting strong robustness. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts of the audio processing method provided by the present invention.

[0020] Figure 2 This is a schematic diagram of the audio separation model provided by the present invention.

[0021] Figure 3 This is a schematic diagram of the sub-band partitioning module provided by the present invention.

[0022] Figure 4 This is a schematic diagram of the modeling unit provided by the present invention.

[0023] Figure 5 This is a schematic diagram of the sub-band mask estimation module provided by the present invention.

[0024] Figure 6 This is a schematic diagram of the target diffusion model provided by the present invention.

[0025] Figure 7 This is a schematic diagram of the attention module provided by the present invention.

[0026] Figure 8 This is the second flowchart of the audio processing method provided by the present invention.

[0027] Figure 9 This is a schematic diagram of the audio processing device provided by the present invention.

[0028] Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0030] In audio processing scenarios such as music production, audio post-processing, virtual music education, and karaoke accompaniment extraction, extracting pure vocals from mixed music signals with high quality while preserving complete accompaniment is a key prerequisite for realizing functions such as music re-creation, intelligent teaching, and interactive entertainment. This places high technical requirements on the fidelity, clarity, and robustness of the separated audio tracks.

[0031] To meet the above requirements, traditional audio source separation technology often relies on spectrum analysis and statistical models. However, when processing complex audio, it is difficult to effectively distinguish multi-source audio with severe spectrum overlap or harmonic interleaving, which easily leads to incomplete separation and introduces obvious sound quality loss and artificial noise. As a result, it is prone to problems such as incomplete separation and severe sound quality loss.

[0032] With the rapid development of deep learning technology, audio separation technology based on neural networks has been gradually applied. This technology specifically trains neural network models, such as BSRNN models, to implicitly learn the differences in spectral distribution, temporal correlation, and harmonic structure of audio from different sound sources in a data-driven manner. Then, based on the differences, it performs mask estimation or target sound source reconstruction and outputs the separation result in one go. However, this method has poor generalization ability and is prone to producing audio signals of lower quality than the original audio signals, with significant sound quality loss. It also often results in incomplete separation, leading to the separation of audio tracks containing sound components that do not belong to the original track. Therefore, it has the defects of poor audio separation quality and inability to correctly separate pure audio from a single sound source.

[0033] In response, this application provides an audio processing method that can more accurately separate the audio of a single sound source in mixed audio, thereby separating high-quality audio of a single sound source from the mixed audio. This solves the problems of incomplete separation, large sound quality loss, and difficulty in processing similar timbre components in the prior art, thereby further improving the audio separation quality and robustness.

[0034] It should be noted that this method can be implemented by an audio processing device, which can be an electronic device, a component within an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be learning machines, mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, super mobile personal computers, netbooks, or personal digital assistants, etc., while non-mobile electronic devices can be servers, network attached storage devices, personal computers, or self-service machines, etc. This invention does not impose specific limitations.

[0035] Furthermore, this method can be applied to various scenarios that require audio separation, such as music production, audio post-processing, virtual music education, and karaoke accompaniment extraction.

[0036] Figure 1 This is one of the flowcharts illustrating the audio processing method provided by the present invention, such as... Figure 1 As shown, the method includes steps 110, 120, 130 and 140.

[0037] Step 110: Extract time-frequency features from the target mixed audio to obtain target time-frequency features; the target mixed audio includes a first target audio and a second target audio.

[0038] The target mixed audio here refers to the audio that needs to be separated into two audio sources, namely the first target audio and the second target audio, which are stacked to form a composite audio signal.

[0039] The target mixed audio can be a mixed music signal containing vocals and instrumental accompaniment, where the vocals are the first target audio and the accompaniment is the second target audio. Alternatively, the target mixed audio can also be an ensemble recording containing two instrument tracks, where one instrument track is the first target audio and the other is the second target audio, etc. This embodiment does not specifically limit the specific source of the target mixed audio. The following description uses the example of a mixed music signal containing vocals and instrumental accompaniment, where the first target audio is vocals and the second target audio is accompaniment, to illustrate the method provided in this embodiment.

[0040] Optionally, when it is necessary to perform audio separation processing on the target mixed audio, the time-frequency features of the target mixed audio can be extracted first to obtain the time-frequency features of the target mixed audio, that is, the target time-frequency features.

[0041] In the time-frequency feature extraction process, to improve the stability of subsequent numerical processing and model convergence efficiency, the target mixed audio can first be normalized by normalizing the maximum amplitude value, thus regularizing its dynamic range to a symmetrical interval (e.g., [-1, 1]), resulting in a normalized time-domain signal. Subsequently, through Fourier transform, such as Short-Time Fourier Transform (STFT), the normalized time-domain signal is converted into a spectrum, yielding complex spectral features. These complex spectral features can then be directly used as the target time-frequency features, or further derived from them, such as Mel spectrum, etc. This embodiment does not specifically limit this approach. Mathematically, the complex spectral features can be represented as a complex matrix, where the row dimension corresponds to the frequency units on the frequency axis, and the column dimension corresponds to the time frames on the time axis. Each element in the matrix contains amplitude and phase information.

[0042] Step 120: Input the target time-frequency features into the audio separation model to obtain the first time-frequency features of the first target audio.

[0043] Optionally, after obtaining the target time-frequency feature X through step 110, the target time-frequency feature X can be input into the audio separation model to perform preliminary component deconstruction of the time-frequency features of the target mixed audio using the audio separation model, so as to obtain the mask matrix corresponding to the first target audio, and multiply the mask matrix of the first target audio with the target time-frequency feature, thereby suppressing non-first target audio components and preserving the first target audio components, and finally obtaining the coarse time-frequency representation of the first target audio, that is, the first time-frequency feature of the first target audio.

[0044] The audio separation model here refers to a machine learning model or signal processing model that has been pre-trained and has the ability to separate specific sound sources from mixed audio. It can be built based on one or more neural network models, and this embodiment does not specifically limit it.

[0045] Figure 2 This is a schematic diagram of the audio separation model provided by the present invention; as shown below. Figure 2 As shown, in one optional implementation, the audio separation model can be an improved version of the BSRNN model, which includes at least a sub-band partitioning module for sub-band partitioning, an intra-sub-band and inter-sub-band modeling module for multi-layer interactive modeling of time and frequency dimensions, and a sub-band mask estimation module for sub-band mask estimation. Accordingly, step 120 specifically includes: Step 121: Using the sub-band division module, the target time-frequency features are divided into non-uniform sub-bands according to audio characteristics (such as music characteristics, narrow low frequencies and wide high frequencies). A total of K sub-bands are divided, and the number of K is determined by the different sub-band division methods. The non-uniform sub-bands are then unified to the same dimension N and merged to obtain the sub-band features.

[0046] Step 122: Using the multi-layer sub-band intra- and inter-sub-band modeling modules, the K sub-band features obtained in step 121 are alternately modeled using time-series and frequency-series methods to obtain deep feature vectors with separation characteristics. Step 123: Use the sub-band mask estimation module to estimate the mask of the first target audio by the deep feature vector with separation characteristics obtained in step 122, and obtain the mask matrix corresponding to the first target audio. Finally, multiply the mask matrix corresponding to the first target audio with the target time-frequency feature to obtain the coarse time-frequency representation of the separated first target audio, that is, the first time-frequency feature of the first target audio.

[0047] Figure 3 This is a schematic diagram of the sub-band partitioning module provided by the present invention; as shown below. Figure 3 As shown, in one optional implementation, the subband partitioning module can be a linear transformation unit, which consists of K normalization layers and K linear layers with an output dimension of N. Accordingly, the step 121 of using the subband partitioning module to partition the target time-frequency features into subbands to extract the corresponding subband features specifically includes: Step 121-1, according to the predefined bandwidth allocation strategy, that is... And satisfy The target time-frequency features are non-uniformly divided into K sub-bands along the frequency dimension F to form K sub-band time-frequency features; the bandwidth configuration of each sub-band satisfies specific constraints, namely, the bandwidth of the sub-band in the low-frequency region is narrower and the bandwidth of the sub-band in the high-frequency region is wider.

[0048] Step 121-2: Pass the K sub-band time-frequency features through K normalization layers and linear layers respectively to obtain K sub-band features.

[0049] In one alternative implementation, the multi-layer sub-band and inter-sub-band modeling modules are composed of stacked modeling units built from gated recurrent units (GRUs).

[0050] Figure 4 This is a schematic diagram of the structure of the modeling unit provided by the present invention; for example... Figure 4 As shown, each modeling unit is formed by connecting a temporal modeling subunit for implementing time-dimension interactive modeling and a dimensional modeling subunit for implementing frequency-dimension interactive modeling. The temporal modeling subunit consists of at least one layer normalization layer, at least one bidirectional GRU layer in the time dimension, and at least one fully connected layer. The frequency-dimension interactive modeling subunit consists of at least one layer normalization layer and at least one bidirectional GRU layer in the frequency dimension. Correspondingly, the specific steps in step 122 for obtaining the deep feature vector of the separation characteristics include: Step 122-1: Using the layer normalization layer and the bidirectional GRU layer in the time dimension of the time modeling subunit, the deep feature vector with separation characteristics is modeled in the time dimension. Specifically, for the K subband feature, all its time frame sequences are simultaneously fed into the same set of bidirectional GRU parameters. By splicing the forward and backward hidden states, the temporal dynamic characteristics are captured, and the global fusion feature vector in the time dimension is output.

[0051] Step 122-2: The fully connected layer in the temporal modeling subunit is used to perform dimensionality reduction on the global fusion feature vector obtained in step 122-1. This fully connected layer keeps the number of input channels unchanged and only compresses the internal dimension of the features. Then, the output of this fully connected layer is added element-wise to the input features in step 122-1 through residual connections to obtain the final temporal dimension deep fusion feature vector, so as to effectively alleviate the gradient vanishing problem in deep network training.

[0052] Step 122-3: Using the frequency-dimensional interactive modeling layer normalization layer and the frequency-dimensional bidirectional GRU layer, the time-dimensional deep fusion feature vector obtained in step 122-2 is modeled on the sub-band frequency dimension. At this time, the features of different sub-bands at the same time step are concatenated into a sequence and share the same set of bidirectional GRU parameters to learn the harmonic correlation and spectral envelope characteristics between frequency bands, and output the global fusion feature vector in the frequency dimension.

[0053] Step 122-4: Use a fully connected layer with frequency dimension interaction modeling to perform dimensionality reduction on the global fusion feature vector obtained in step 122-3. In this operation, it is necessary to keep the input channels the same as the channels of the time dimension deep fusion feature vector in step 122-2. Then, add the output of the fully connected layer to the output of step 122-2 through residual connection to obtain an intermediate deep feature vector with separation characteristics.

[0054] Step 122-5: Repeat the process of steps 122-1 to 122-4 multiple times for the intermediate deep feature vector with separation characteristics obtained in step 122-4, until all modeling units have been traversed, and the final deep feature vector with separation characteristics can be obtained.

[0055] Figure 5 This is a schematic diagram of the sub-band mask estimation module provided by the present invention; as shown below. Figure 5 As shown, in one optional implementation, the subband mask estimation module can be composed of K multilayer sensing units. Each multilayer sensing unit consists of a layer normalization layer, two fully connected layers, a rectified linear unit (ReLU) activation function, and a gated linear unit (GLU) activation function or a sigmoid activation function layer. Accordingly, the step of obtaining the mask matrix corresponding to the first target audio in step 123 specifically includes: Step 123-1: Decompose the deep feature vector with separation characteristics obtained in step 122 into K sub-band deep feature vectors; Step 123-2: Normalize the K sub-band deep feature vectors obtained from the splitting in step 123-1 to obtain K normalized features; Step 123-3: Use K fully connected layers and the ReLU activation function to perform nonlinear operations on the K normalized features obtained in step 123-2 to obtain K hidden layer feature vectors. Step 123-4: Use K fully connected layers and the GLU activation function to perform further nonlinear operations on the K hidden layer feature vectors obtained from the nonlinear operation in 123-3 to obtain K mask values, and merge them in K dimensions to obtain the mask matrix corresponding to the first target audio.

[0056] Step 130: Input the first time-frequency feature and the target time-frequency feature as conditions into the target diffusion model to obtain the second time-frequency feature of the first target audio.

[0057] It should be noted that the first time-frequency feature output in step 120 is a preliminary estimate of the first target audio in the time-frequency domain by the audio separation model. It represents the approximate energy distribution and harmonic structure of the first target audio, but may still contain a certain degree of distortion, spectral impairment, or residual components from the second target audio. Diffusion models, as an emerging type of generative model, generate data through a gradual noise addition and denoising process. In recent years, they have made breakthrough progress in image generation and have shown potential in audio generation. The core idea of ​​diffusion models is to gradually transform data into pure noise through a forward diffusion process, and then recover the data from the noise through a reverse diffusion process. This generation method enables diffusion models to generate high-quality, high-fidelity samples. Therefore, the powerful data distribution learning capability of diffusion models can be utilized to further compensate, repair, refine, and enhance the quality of the first time-frequency feature of the first target audio initially separated by the audio separation model. This bridges the quality gap between the separation result output by the audio separation model and the real, pure first target audio, repairs the distortion and residual problems caused by the audio separation model, and thus improves the separation effect.

[0058] Specifically, the first time-frequency feature obtained in step 120 (i.e., the coarsely estimated time-frequency feature of the first target audio) and the target time-frequency feature (i.e., the time-frequency feature of the target mixed audio) are used together as conditional information and input into the conditional injection interface of the target diffusion model so that the target diffusion model can use the conditional information to perform diffusion processing, thereby obtaining the further repaired time-frequency feature of the first target audio, i.e., the second time-frequency feature. The second time-frequency feature is significantly better than the first time-frequency feature in terms of spectral fidelity, harmonic integrity, component purity and auditory perception quality, and can be regarded as a high-quality reconstruction and refined repair result of the first target audio.

[0059] It should be noted that this dual-conditional input design has at least two technical advantages: First, since the first time-frequency feature is relatively close to the true signal distribution of the first target audio, it can significantly reduce the learning difficulty for the target diffusion model to generate the target signal from pure noise, allowing the target diffusion model to concentrate its computational resources on detail restoration and residual suppression, rather than reconstructing the entire audio structure from scratch. Second, introducing the target time-frequency feature as an explicit constraint can provide a reliable contextual reference during the iterative denoising process of the target diffusion model, effectively preventing the generated result from deviating from the true information contained in the target mixed audio in terms of content, energy, or spectral structure, and avoiding unreasonable audio artifacts or component drift. Therefore, this dual-conditional input design allows the target diffusion model to focus on restoring the distortion and residual components in the first time-frequency feature during the generation process, while relying on the contextual constraints of the target mixed feature to prevent the generated result from deviating from the true audio content, thereby significantly improving the audio separation quality and fidelity.

[0060] The target diffusion model here is a conditional generation model based on a diffusion probability model or its variants. It can focus on repairing the distortion and residual components in the first time-frequency features, and rely on the contextual constraints of the target mixed features to prevent the generated results from deviating from the real audio content. It can repair the first time-frequency features of the first target audio initially separated by the audio separation model, and further improve the separation quality of the first target audio.

[0061] The target diffusion model here can be constructed from various neural network models. In one possible embodiment, the target diffusion model can be a model constructed based on an encoder-decoder symmetric structure (also known as Unet). Figure 6 This is a schematic diagram of the target diffusion model provided by the present invention; as shown below. Figure 6 As shown, the target diffusion model can be constructed from a Unet model consisting of seven convolutional blocks (convolutional block 1, convolutional block 2, convolutional block 3, convolutional block 4, convolutional block 5, convolutional block 6, and convolutional block 7) stacked with residual connections. In the encoder path, the spatial resolution of the feature map is progressively compressed and the channel dimension is expanded through downsampling. In the decoder path, the spatial resolution is progressively restored and multi-scale contextual information is fused through upsampling. Specifically, in each inference step of the backdiffusion, the target diffusion model receives a Gaussian noise tensor as an initial sample. Simultaneously, the conditional information formed by the first time-frequency feature and the target time-frequency feature is injected into each hidden layer of the Unet through a cross-connection mechanism to guide the generation process towards a solution space consistent with the first time-frequency feature coarsely separated by the target mixed audio and audio separation models.

[0062] Step 140: Obtain the first target audio and the second target audio based on the second time-frequency feature.

[0063] Optionally, after obtaining the high-quality time-frequency features of the first target audio output by the target diffusion model repair, i.e. the second time-frequency features, the first target audio can be recovered from the second time-frequency features and the second target audio can be reconstructed.

[0064] Specifically, the second target audio can be transformed from the time-frequency domain back to a time-domain waveform signal by applying an inverse time-frequency transformation operation corresponding to the target time-frequency characteristics, thereby obtaining the first target audio. If the time-frequency transformation operation corresponding to the target time-frequency characteristics is a short-time Fourier transform, then an inverse short-time Fourier transform can be used to transform the second target audio from the time-frequency domain back to a time-domain waveform signal to obtain the first target audio. Simultaneously, the target time-frequency characteristics can be subtracted from the second time-frequency characteristics to obtain high-quality time-frequency characteristics of the second target audio. Then, the high-quality time-frequency characteristics of the second target audio can be transformed from the time-frequency domain back to a time-domain waveform signal by applying an inverse time-frequency transformation operation corresponding to the target time-frequency characteristics, thereby obtaining the second target audio.

[0065] The method provided in this embodiment extracts time-frequency features from the target mixed audio and inputs them into an audio separation model to obtain a coarse estimate of the time-frequency features of the first target audio. Then, using the coarse estimate and the time-frequency features of the target mixed audio as conditions, the target diffusion model is guided to perform conditional generative repair on the coarse estimate of the time-frequency features to obtain a high-quality second time-frequency feature. Finally, the first target audio and the second target audio are separated based on the second time-frequency feature. Thus, through a two-stage architecture of separation followed by compensation, the powerful data distribution learning ability and high-fidelity generation characteristics of the diffusion model are used to refine the separation results of the first stage, effectively solving the problems of incomplete separation, severe sound quality loss, and insufficient generalization ability. Furthermore, by using the coarse estimate and the target mixed features as conditional inputs, the learning difficulty of the diffusion model is reduced and the generated results are prevented from deviating from the real audio, significantly improving the separation purity and fidelity of the first target audio. At the same time, it ensures that the second target audio can be completely and accurately reconstructed through linear subtraction, achieving high-quality audio separation from end to end and exhibiting strong robustness.

[0066] In some embodiments, step 130 specifically includes: Step 131: Determine target condition information based on the first time-frequency feature and the target time-frequency feature; Step 132: Input the target condition information and noise signal into the target diffusion model to obtain the second time-frequency feature.

[0067] Optionally, the process of inputting the first time-frequency feature and the target time-frequency feature as conditional inputs into the target diffusion model to obtain the second time-frequency feature can be specifically executed as two logical sub-steps: First, target conditional information for guiding the diffusion model is determined based on the first time-frequency feature and the target time-frequency feature. Specifically, feature transformation operations can be applied to the first time-frequency feature (i.e., the coarsely estimated spectrum output by the audio separation model) and the target time-frequency feature (i.e., the complete spectrum of the target mixed audio) to map them to a unified representation domain of the conditional space of the target diffusion model. This feature transformation operation can be implemented by linear projection, nonlinear activation, self-attention mechanism alignment and fusion, etc. This invention does not specifically limit this. The transformed first time-frequency feature and the transformed target time-frequency feature are aligned in dimension and can jointly constitute the target conditional information.

[0068] It should be noted that the part of the target condition information derived from the target time-frequency features provides the global context and energy distribution benchmark of the target mixed audio, preventing the generated results from deviating from the real acoustic scene; the part derived from the first time-frequency features provides an initial semantic framework close to the real target distribution, which significantly reduces the difficulty of the diffusion model to reconstruct high-quality audio from pure noise.

[0069] Subsequently, the target condition information and randomly sampled noise signals are input into the target diffusion model to initiate reverse denoising generation. This process repairs the first time-frequency features of the first target audio, ensuring that the generated second time-frequency features are semantically consistent with the target time-frequency features and the coarsely separated first time-frequency features, while further enhancing their fidelity and naturalness. The noise signal has a time-frequency resolution completely consistent with the second time-frequency features to be generated.

[0070] The method provided in this embodiment repairs the coarse segmentation time-frequency features of the target audio by introducing noise signals and target condition information jointly determined by the first time-frequency features and the target time-frequency features as input to the target diffusion model. This not only ensures that the audio track of the separated first target audio is close to that of the original clean target audio, thus reducing the learning difficulty of the target diffusion model, but also prevents the target diffusion model from generating a first target audio that deviates from the target mixed audio, effectively ensuring a balance between content fidelity and quality improvement in the generated result.

[0071] In some embodiments, step 131 specifically includes: Step 131-1: Input the first time-frequency feature and the target time-frequency feature into the feature encoding module to obtain the first encoded feature and the second encoded feature; the first encoded feature is the encoded feature corresponding to the first time-frequency feature, and the second encoded feature is the encoded feature corresponding to the target time-frequency feature; Step 131-2: Input the first encoded feature and the second encoded feature into the attention module to obtain the target condition information.

[0072] Optionally, in the process of determining target condition information based on the first time-frequency feature and the target time-frequency feature, it can be achieved through the cascaded processing of the feature encoding module and the attention module. This two-stage processing mechanism can fully explore the complementarity and consistency of the two time-frequency features at the semantic level, thereby generating high-quality guiding conditions and laying a solid foundation for the accurate generation of the target diffusion model.

[0073] Specifically, in the first stage, the first time-frequency feature (i.e., the coarsely estimated time-frequency feature output by the audio separation model) and the target time-frequency feature (i.e., the complete time-frequency feature of the target mixed audio) are jointly input into the feature encoding module to obtain the corresponding first coded feature and second coded feature. The first coded feature specifically refers to the latent space representation obtained after encoding the first time-frequency feature, which focuses on the acoustic structure, harmonic distribution, and energy pattern of the first target audio; the second coded feature specifically refers to the latent space representation obtained after encoding the target time-frequency feature, which focuses on the global context, spectral envelope, and multi-source coexistence relationship of the target mixed audio. The two are consistent in dimension to facilitate interaction and fusion during audio separation processing.

[0074] The feature encoding module here refers to a neural network unit or signal processing unit capable of mapping input time-frequency features to a high-dimensional latent space representation. Its main function is to extract deep abstract features and eliminate redundant noise and inconsistencies in the original features. This feature encoding module can be constructed from an integrated module that simultaneously possesses the first time-frequency feature encoding function and the target time-frequency feature encoding function, or it can be composed of two structurally identical but parameter-independent sub-encoders, namely a first encoding layer with the first time-frequency feature encoding function and a second encoding layer with the target time-frequency feature encoding function. This embodiment does not specifically limit this.

[0075] In the second stage, the first and second encoded features are input into the attention module to generate target conditional information. The attention module here refers to a neural network unit that uses an attention mechanism to dynamically assign weights and filter information between features. Its function is to calculate the relevance weights between the two encoded features and adaptively fuse the features based on these weights, thereby highlighting the important regions shared by both features and suppressing irrelevant or conflicting information components. Specifically, the attention module receives the first and second encoded features as input. First, it calculates the attention weight matrix between the two feature vectors through a learnable linear transformation or a similarity metric function. Each element of this weight matrix represents the semantic matching degree and importance score between a local region of the first encoded feature and the corresponding region of the second encoded feature. Subsequently, the attention module uses this weight matrix to perform weighted aggregation on the first and second encoded features respectively to obtain the target conditional information.

[0076] In one alternative implementation, the attention module may employ a cross-attention or collaborative attention architecture to achieve bidirectional information interaction and joint optimization between two encoded features.

[0077] The method provided in this embodiment maps the first time-frequency feature and the target time-frequency feature to a high-dimensional latent space through a feature encoding module. This effectively improves the abstraction level and robustness of the feature representation, eliminates the negative impact of noise interference and minor perturbations in the underlying spectrum on the generation process, and achieves adaptive alignment and dynamic fusion of the first and second encoded features through the introduction of an attention module. This allows the target conditional information to focus on the high-value regions shared by the two features, while suppressing artifact regions caused by separation errors in the audio separation model and residual accompaniment regions unrelated to the target. Compared to the traditional method of simply concatenating or linearly combining the two time-frequency features, the target conditional information generated in this embodiment has stronger discriminative and guiding properties, which can significantly improve the accuracy of the diffusion model in capturing the distribution of the target sound source during the reverse denoising process. As a result, the final generated second time-frequency feature achieves lower distortion, higher inter-spectral separation ratio, and better perceived sound quality.

[0078] In some embodiments, step 131-1 specifically includes: The first time-frequency feature is input into the first encoding layer of the feature encoding module to obtain the first encoded feature; The target time-frequency features are input into the second coding layer of the feature coding module to obtain the second coded features; The first coding layer and the second coding layer share weight parameters.

[0079] Optionally, the process of inputting the first time-frequency feature and the target time-frequency feature into the feature encoding module to obtain the first encoded feature and the second encoded feature can be implemented by setting up two symmetrical and weight-shared encoding layers in parallel within the feature encoding module. This ensures that the two features obtain consistent encoding methods while significantly reducing the number of model parameters and the risk of overfitting, thus laying a unified feature space foundation for the effective computation of the attention mechanism.

[0080] Specifically, the first time-frequency feature (i.e., the coarsely estimated time-frequency feature output by the audio separation model) is input into the first encoding layer of the feature encoding module, which is specifically used to process the input of this branch, so as to map the first time-frequency feature to a high-dimensional latent space, thereby obtaining the first encoded feature feature_V. The target time-frequency feature (i.e., the complete time-frequency feature of the original mixed audio) is input into the second encoding layer of the feature encoding module, which is specifically used to process the input of this branch, so as to map the target time-frequency feature to a high-dimensional latent space, thereby obtaining the second encoded feature feature_X.

[0081] Among them, the first coding layer and the second coding layer are coding layers with symmetrical structure and shared weights.

[0082] In one alternative implementation, the first encoding layer can adopt a two-layer stacked residual network (also known as ResNet) structure, with each layer containing a convolutional kernel, a normalization layer, and a nonlinear activation function layer, retaining key information in the input features through residual connections. In another implementation, the first encoding layer can also be implemented as a single fully connected layer, flattening the input two-dimensional time-frequency features and projecting them onto a latent vector space of a predetermined dimension. This embodiment does not specifically limit the network topology of the first encoding layer; any neural network layer that can effectively map the first time-frequency features to the first encoded features constitutes a reasonable implementation.

[0083] The second coding layer is structurally identical to the first coding layer, meaning it has the same layer configuration, the same number of neurons, the same activation function type, and the same connection method, to ensure that the transformation logic of the two coding layers for the input features is consistent.

[0084] The method provided in this embodiment, with its weight-sharing design between the first and second encoding layers, achieves dual optimization of parameter efficiency and feature consistency at the model architecture level. This not only significantly reduces the model's storage and computational overhead and shortens training and inference time, but more importantly, it forces the two encoding branches to generate feature representations with uniform distribution characteristics. This makes the first and second encoded features comparable and fusionable in the latent space, creating ideal conditions for the attention module to accurately capture the common regions and complementary information between the two features. Compared to the problems of independent parameters and feature space misalignment that may result from independent encoding layers, the target condition information generated in this embodiment shows significant improvements in discriminability and robustness. This allows the diffusion model to more accurately locate the region to be repaired during the reverse denoising process, and the final output second time-frequency feature exhibits superior performance in terms of structural fidelity and component purity.

[0085] In some embodiments, step 131-2 specifically includes: The first encoded feature and the second encoded feature are input into the weight calculation unit of the attention module to obtain the target weight matrix; The first encoded feature and the second encoded feature are input into the value feature calculation layer of the attention module to obtain the first value feature and the second value feature; the first value feature is the value feature corresponding to the first time-frequency feature, and the second value feature is the value feature corresponding to the target time-frequency feature; The target weight matrix, the first value feature, and the second value feature are input into the fusion layer of the attention module to obtain the conditional information corresponding to the first time-frequency feature and the conditional information corresponding to the target time-frequency feature; The target condition information is obtained based on the condition information corresponding to the first time-frequency feature and the condition information corresponding to the target time-frequency feature.

[0086] Figure 7 This is a schematic diagram of the structure of the attention module provided by the present invention; as shown below. Figure 7 As shown, the module includes at least a value feature calculation layer for value feature calculation, a weight calculation unit for weight calculation, and a fusion layer for feature fusion. Optionally, the process of inputting the first and second encoded features into the attention module to obtain target condition information can be achieved through multiple stages of value calculation, weight generation, and hierarchical fusion. This allows for the full exploitation of the correlation between the two encoded features in the spatial, frequency, and channel dimensions, thereby generating condition information with strong discriminative power and high guidance capability.

[0087] Specifically, the first and second encoded features are first input into the weight calculation unit of the attention module, and then the weight calculation unit calculates the target weight matrix using an attention mechanism. This reflects the semantic correlation between the first time-frequency feature and the target time-frequency feature in the corresponding time-frequency region.

[0088] like Figure 7 As shown, in one possible implementation, the weight calculation unit includes a channel concatenation layer for performing feature concatenation along the channel dimension, a query feature calculation layer for calculating query features, a key feature calculation layer for calculating key features, and a weight calculation layer for weight mapping. Accordingly, the specific calculation steps for the target weight matrix include: concatenating the first encoded feature and the second encoded feature along the channel dimension to obtain a merged feature; inputting the merged feature to the query feature calculation layer and the key feature calculation layer to obtain query features and key features; and inputting the query features and the key features to the weight calculation layer to obtain the target weight matrix.

[0089] Specifically, when calculating the target weight matrix, the first encoded feature and the second encoded feature can be concatenated in the channel dimension through the channel concatenation layer to obtain the merged feature. This achieves explicit fusion of the two feature tensors without changing the feature map spatial resolution, providing a unified input representation for subsequent joint attention calculation.

[0090] After obtaining the merged features, the query feature calculation layer and the key feature calculation layer can be used to project the merged features to obtain the query feature output by the query feature calculation layer and the key feature output by the key feature calculation layer. The query feature is used to characterize the characteristics of the target region that needs to be focused on, and the key feature is used to characterize the overall contextual characteristics of the candidate region. In one possible implementation, the query feature calculation layer and the key feature calculation layer can be constructed using a fully connected layer (also known as FC).

[0091] After obtaining the query features (Query) and key features (Key), they can be input into the weight calculation layer to calculate the target weight matrix through an adaptive attention mechanism. This reflects the semantic correlation between the first time-frequency feature and the target time-frequency feature in the corresponding time-frequency region.

[0092] The method provided in this embodiment first explicitly fuses features in the channel dimension, and then generates query and key vectors based on the fused features respectively. This embodiment can adaptively mine and quantify the deep semantic dependencies of two features by utilizing joint context information while keeping the time-frequency spatial structure unchanged, thereby generating a target weight matrix that can accurately reflect the time-frequency consistency of the two features, which significantly improves the accuracy and effectiveness of feature fusion.

[0093] While calculating the target weight matrix, the first encoded feature (feature_V) and the second encoded feature (feature_X) can be input to the value feature calculation layer to obtain the value feature corresponding to the first time-frequency feature, i.e., the first value feature (Value_V), and the value feature corresponding to the target time-frequency feature, i.e., the second value feature (Value_X). Here, the value feature calculation layer can be constructed from a single fully connected layer capable of calculating both the value feature corresponding to the first time-frequency feature and the value feature corresponding to the target time-frequency feature. Alternatively, it can be constructed from two structurally independent fully connected layers, one for calculating the value feature corresponding to the first time-frequency feature and the other for calculating the value feature corresponding to the target time-frequency feature. This embodiment does not specifically limit this approach.

[0094] After obtaining the target weight matrix After the first value feature Value_V and the second value feature Value_X, the target weight matrix is... The first-value feature Value_V and the second-value feature Value_X are input to the fusion layer of the attention module. The fusion layer applies the target weight matrix to the first-value feature and the second-value feature respectively, thereby obtaining the conditional information out_V corresponding to the first time-frequency feature and the conditional information out_X corresponding to the target time-frequency feature. The specific calculation formula is as follows: ; ; ; in, Let be the feature dimension of the key feature.

[0095] Subsequently, the conditional information out_V corresponding to the first time-frequency feature and the conditional information out_X corresponding to the target time-frequency feature are concatenated again along the channel dimension to form the final target conditional information; alternatively, the two can be fused into a single conditional vector through element-wise addition, weighted averaging, or linear projection to obtain the target conditional information. This allows the target conditional information to simultaneously incorporate the detailed structure of the first time-frequency feature and the global constraints of the target time-frequency feature, providing a highly discriminative and high-fidelity guiding signal for the reverse generation process of the target diffusion model.

[0096] The method provided in this embodiment uses a query-key-value attention mechanism to achieve dynamic alignment and adaptive fusion of the first and second encoded features. Compared with simple channel splicing or linear combination, it can more accurately capture the local correlation between the two features in the time-frequency dimension, focusing the target condition information on the high-value region shared by both features. At the same time, it suppresses artifact regions caused by the separation error in the first stage and accompaniment residue regions unrelated to the target, significantly improving the target localization accuracy and detail restoration capability of the diffusion model in the reverse denoising process. Ultimately, the output second time-frequency feature has been significantly improved in terms of structural fidelity, spectral continuity, and component purity.

[0097] In some embodiments, the step of inputting the target weight matrix, the first value feature, and the second value feature into the fusion layer of the attention module to obtain the conditional information corresponding to the first time-frequency feature and the conditional information corresponding to the target time-frequency feature specifically includes: The target weight matrix and the first value feature are input into the first fusion sub-layer of the fusion layer to obtain the conditional information corresponding to the first time-frequency feature; The target weight matrix and the second value feature are input into the second fusion sub-layer of the fusion layer to obtain the conditional information corresponding to the target time-frequency feature.

[0098] Optionally, the process of inputting the target weight matrix, the first value feature, and the second value feature into the fusion layer of the attention module to obtain the conditional information corresponding to the first time-frequency feature and the conditional information corresponding to the target time-frequency feature can be implemented by two independent fusion sub-layers within the fusion layer. This parallel fusion architecture provides the ability to independently control information from different sources, enhancing the flexibility and interpretability of condition generation.

[0099] Specifically, the first fusion sublayer receives the target weight matrix and the first value feature Value_X as input, and performs a weighted aggregation operation to generate the conditional information out_X corresponding to the first time-frequency feature. This out_X represents the relevant components of the first target audio that are enhanced and the noise residue components that are suppressed in the first value feature under the guidance of the target attention weight.

[0100] The second fusion sublayer receives the same target weight matrix and second-value feature Value_V as input, and performs independent weighted aggregation operations to generate conditional information out_V corresponding to the target time-frequency feature. This out_V represents the global context information and multi-source constraint information retained in the second-value feature under the guidance of the target attention weights.

[0101] The method provided in this embodiment, through a parallel architecture of dual fusion sub-layers, achieves independent extraction and customized control of the corresponding conditional information of the first time-frequency feature and the target time-frequency feature. This enables the diffusion model to simultaneously receive guiding signals with different semantic focuses. The first fusion sub-layer enhances the ability to repair the details of the first target audio, while the second fusion sub-layer ensures the consistency constraint of the global context. The synergistic effect of the two effectively solves the problem in the prior art where the generated result deviates from the real audio due to the lack of explicit constraints. The final output second time-frequency feature achieves a better balance between local detail fidelity and overall structural rationality, significantly improving the quality of multi-source audio separation.

[0102] In some embodiments, the target diffusion model can be trained based on the following steps: First, according to a predetermined network architecture, such as a conditional diffusion model based on the U-Net skeleton, the model parameters are initialized to obtain an untrained initial model. The weights of this initial model are typically initialized randomly or using pre-trained model parameters. It should be noted that the specific parameter size, layer configuration, and initialization strategy of the initial model can be flexibly adjusted according to the amount of training data and task complexity; this embodiment does not impose specific limitations on these aspects.

[0103] In addition, a batch of mixed audio samples is sampled from the training dataset. This batch of data includes the time-frequency features of the corresponding mixed audio samples, the first sample audio label (such as the pure human voice spectrum), and the second sample audio label (such as the pure accompaniment spectrum).

[0104] Subsequently, Gaussian noise is progressively added to the first sample audio label to generate a series of noisy samples. During the inverse denoising process, the initial model uses the time-frequency features of the mixed audio samples and the first time-frequency feature of the first sample audio (i.e., the coarsely estimated time-frequency feature of the first sample audio output by the audio separation model) as input conditions. After iterative denoising, it outputs the second time-frequency feature of the first sample audio. Then, an inverse time-frequency transform is performed on the second time-frequency feature to obtain the estimated time-domain waveform of the first sample audio (also called the estimated value). Simultaneously, in the time domain, the estimated time-domain waveform of the first sample audio is subtracted from the estimated time-domain waveform of the mixed audio samples to obtain the estimated time-domain waveform of the second sample audio.

[0105] Subsequently, the L2 reconstruction loss between the estimated value and the label value of the first sample audio is calculated to supervise the model's accurate modeling of the human voice distribution; the L2 loss between the estimated value and the label value of the second sample audio is calculated to constrain the integrity of the accompaniment separation. The two losses are weighted and summed to obtain the total loss, and the initial model is iteratively trained using the total loss until the loss converges or a termination condition such as reaching a certain number of iterations is met. Finally, the updated model parameters are saved to obtain the trained target diffusion model.

[0106] The method provided in this embodiment constructs a multi-task training framework that includes diffusion denoising loss of the first sound source audio and fidelity loss of the second sound source audio. This enables the target diffusion model to improve the generation quality of the first sound source audio while taking into account the integrity of the separation of the second sound source audio. Furthermore, it can gradually learn the high-dimensional distribution characteristics of complex audio data from random initialization, significantly enhancing the generalization ability to unseen sound source styles.

[0107] The following uses a scenario of separating vocals and accompaniment as an example to describe the method provided in this embodiment. In this scenario, the target mixed audio is a mixed music signal, the first target audio is vocal audio, and the second target audio is accompaniment audio.

[0108] Figure 8 This is a second schematic flowchart of the audio processing method provided by the present invention; as shown below. Figure 8 As shown, the method includes: Step 810: Normalize the maximum amplitude of the mixed music signal, and after normalizing it to the range of [-1,1], convert it into a spectrum through short-time Fourier transform to obtain complex spectral features (also known as complex features). Step 820: Input the complex spectral feature X into the audio separation model (such as the vocal and accompaniment separation model used for vocal and accompaniment separation) to obtain the vocal mask, and multiply the vocal mask with the spectrogram of the mixed music signal to obtain the initial vocal spectrum; Step 830: Input the complex spectral feature X and the initial human voice spectrum output by the audio separation model into a module that can perform self-attention alignment and fusion, also known as the self-attention alignment and fusion module, which is constructed by the feature encoding module and the attention module to obtain target condition information; Step 840: Input the target condition information and noise signal into the target diffusion model so that the target diffusion model can repair the initial human voice spectrum under the guidance of the target condition information to obtain the repaired human voice spectrum; Step 850: Perform an inverse short-time Fourier transform (ISFT) on the restored vocal spectrum to obtain the final vocal audio; subtract the complex spectral feature X from the restored vocal spectrum to obtain the accompaniment spectrum; perform an inverse short-time Fourier transform on the accompaniment spectrum to obtain the final accompaniment audio.

[0109] The audio processing apparatus provided by the present invention will be described below. The audio processing apparatus described below can be referred to in correspondence with the audio processing method described above.

[0110] Figure 9 This is a schematic diagram of the audio processing device provided by the present invention; as shown below. Figure 9 As shown, the device includes: The time-frequency feature extraction unit 910 is used to extract time-frequency features from the target mixed audio to obtain target time-frequency features; the target mixed audio includes a first target audio and a second target audio. The first separation unit 920 is used to input the target time-frequency features into the audio separation model to obtain the first time-frequency features of the first target audio. The correction unit 930 is used to input the first time-frequency feature and the target time-frequency feature as conditions into the target diffusion model to obtain the second time-frequency feature of the first target audio. The second separation unit 940 is used to acquire the first target audio and the second target audio based on the second time-frequency characteristics.

[0111] The apparatus provided in this embodiment extracts time-frequency features from the target mixed audio and inputs them into an audio separation model to obtain a coarse estimate of the time-frequency features of the first target audio. Then, using the coarse estimate and the time-frequency features of the target mixed audio as conditions, it guides the target diffusion model to perform conditional generative repair on the coarse estimate of the time-frequency features to obtain a high-quality second time-frequency feature. Finally, the first target audio and the second target audio are separated based on the second time-frequency feature. Thus, through a two-stage architecture of separation followed by compensation, the powerful data distribution learning ability and high-fidelity generation characteristics of the diffusion model are used to refine the separation results of the first stage, effectively solving the problems of incomplete separation, severe sound quality loss, and insufficient generalization ability. Furthermore, by using the coarse estimate and the target mixed features as conditional inputs, the learning difficulty of the diffusion model is reduced and the generated results are prevented from deviating from the real audio, significantly improving the separation purity and fidelity of the first target audio. At the same time, it ensures that the second target audio can be completely and accurately reconstructed through linear subtraction, achieving high-quality audio separation from end to end and exhibiting strong robustness.

[0112] In some embodiments, the correction unit is specifically used for: Based on the first time-frequency feature and the target time-frequency feature, the target condition information is determined; The target condition information and noise signal are input into the target diffusion model to obtain the second time-frequency feature.

[0113] In some embodiments, the correction unit is further configured to: The first time-frequency feature and the target time-frequency feature are input into the feature encoding module to obtain a first encoded feature and a second encoded feature; the first encoded feature is the encoded feature corresponding to the first time-frequency feature, and the second encoded feature is the encoded feature corresponding to the target time-frequency feature; The first encoded feature and the second encoded feature are input into the attention module to obtain the target condition information.

[0114] In some embodiments, the correction unit is further configured to: The first time-frequency feature is input into the first encoding layer of the feature encoding module to obtain the first encoded feature; The target time-frequency features are input into the second coding layer of the feature coding module to obtain the second coded features; The first coding layer and the second coding layer share weight parameters.

[0115] In some embodiments, the correction unit is further configured to: The first encoded feature and the second encoded feature are input into the weight calculation unit of the attention module to obtain the target weight matrix; The first encoded feature and the second encoded feature are input into the value feature calculation layer of the attention module to obtain the first value feature and the second value feature; the first value feature is the value feature corresponding to the first time-frequency feature, and the second value feature is the value feature corresponding to the target time-frequency feature; The target weight matrix, the first value feature, and the second value feature are input into the fusion layer of the attention module to obtain the conditional information corresponding to the first time-frequency feature and the conditional information corresponding to the target time-frequency feature; The target condition information is obtained based on the condition information corresponding to the first time-frequency feature and the condition information corresponding to the target time-frequency feature.

[0116] In some embodiments, the correction unit is further configured to: The target weight matrix and the first value feature are input into the first fusion sub-layer of the fusion layer to obtain the conditional information corresponding to the first time-frequency feature; The target weight matrix and the second value feature are input into the second fusion sub-layer of the fusion layer to obtain the conditional information corresponding to the target time-frequency feature.

[0117] In some embodiments, the weight calculation unit includes a query feature calculation layer, a key feature calculation layer, and a weight calculation layer; correspondingly, the correction unit is further configured to: At the channel dimension, the first encoded feature and the second encoded feature are concatenated to obtain the merged feature; The merged features are input into the query feature calculation layer and the key feature calculation layer to obtain query features and key features; The query features and the key features are input into the weight calculation layer to obtain the target weight matrix.

[0118] In some embodiments, the apparatus further includes a model training unit, specifically used for: Construct the initial model corresponding to the target diffusion model; Obtain the second time-frequency feature of the first sample audio in the sample mixed audio output by the initial model; Based on the second time-frequency feature of the first sample audio, obtain the estimated value of the first sample audio and the estimated value of the second sample audio in the sample mixed audio; Based on the estimated value of the first sample audio, the label value of the first sample audio, the estimated value of the second sample audio, and the label value of the second sample audio, the initial model is iteratively trained to obtain the target diffusion model.

[0119] The apparatus provided by the present invention is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0120] Figure 10 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 10 As shown, the electronic device may include: a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040, wherein the processor 1010, the communications interface 1020, and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 can call logical instructions in the memory 1030 to execute an audio processing method, which includes: extracting time-frequency features from target mixed audio to obtain target time-frequency features; the target mixed audio includes a first target audio and a second target audio; inputting the target time-frequency features into an audio separation model to obtain a first time-frequency feature of the first target audio; inputting the first time-frequency feature and the target time-frequency feature as conditions into a target diffusion model to obtain a second time-frequency feature of the first target audio; and obtaining the first target audio and the second target audio based on the second time-frequency feature.

[0121] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0122] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the audio processing method provided by the above methods. The method includes: extracting time-frequency features from target mixed audio to obtain target time-frequency features; the target mixed audio includes a first target audio and a second target audio; inputting the target time-frequency features into an audio separation model to obtain a first time-frequency feature of the first target audio; inputting the first time-frequency feature and the target time-frequency feature as conditions into a target diffusion model to obtain a second time-frequency feature of the first target audio; and obtaining the first target audio and the second target audio based on the second time-frequency feature.

[0123] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the audio processing methods provided by the above methods. The method includes: extracting time-frequency features from target mixed audio to obtain target time-frequency features; the target mixed audio includes a first target audio and a second target audio; inputting the target time-frequency features into an audio separation model to obtain a first time-frequency feature of the first target audio; inputting the first time-frequency feature and the target time-frequency feature as conditions into a target diffusion model to obtain a second time-frequency feature of the first target audio; and obtaining the first target audio and the second target audio based on the second time-frequency feature.

[0124] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0125] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An audio processing method, characterized in that, include: Time-frequency features of the target mixed audio are extracted. The target mixed audio includes a first target audio and a second target audio; The target time-frequency features are input into the audio separation model to obtain the first time-frequency features of the first target audio. The first time-frequency feature and the target time-frequency feature are used as conditions input into the target diffusion model to obtain the second time-frequency feature of the first target audio. Based on the second time-frequency feature, the first target audio and the second target audio are obtained.

2. The audio processing method according to claim 1, characterized in that, The step of inputting the first time-frequency feature and the target time-frequency feature as conditions into the target diffusion model to obtain the second time-frequency feature of the first target audio includes: Based on the first time-frequency feature and the target time-frequency feature, the target condition information is determined; The target condition information and noise signal are input into the target diffusion model to obtain the second time-frequency feature.

3. The audio processing method according to claim 2, characterized in that, The step of determining the target condition information based on the first time-frequency feature and the target time-frequency feature includes: The first time-frequency feature and the target time-frequency feature are input into the feature encoding module to obtain a first encoded feature and a second encoded feature; the first encoded feature is the encoded feature corresponding to the first time-frequency feature, and the second encoded feature is the encoded feature corresponding to the target time-frequency feature; The first encoded feature and the second encoded feature are input into the attention module to obtain the target condition information.

4. The audio processing method according to claim 3, characterized in that, The step of inputting the first time-frequency feature and the target time-frequency feature into the feature encoding module to obtain the first encoded feature and the second encoded feature includes: The first time-frequency feature is input into the first encoding layer of the feature encoding module to obtain the first encoded feature; The target time-frequency features are input into the second coding layer of the feature coding module to obtain the second coded features; The first coding layer and the second coding layer share weight parameters.

5. The audio processing method according to claim 3, characterized in that, The step of inputting the first encoded feature and the second encoded feature into the attention module to obtain the target condition information includes: The first encoded feature and the second encoded feature are input into the weight calculation unit of the attention module to obtain the target weight matrix; The first encoded feature and the second encoded feature are input into the value feature calculation layer of the attention module to obtain the first value feature and the second value feature; the first value feature is the value feature corresponding to the first time-frequency feature, and the second value feature is the value feature corresponding to the target time-frequency feature; The target weight matrix, the first value feature, and the second value feature are input into the fusion layer of the attention module to obtain the conditional information corresponding to the first time-frequency feature and the conditional information corresponding to the target time-frequency feature; The target condition information is obtained based on the condition information corresponding to the first time-frequency feature and the condition information corresponding to the target time-frequency feature.

6. The audio processing method according to claim 5, characterized in that, The step of inputting the target weight matrix, the first value feature, and the second value feature into the fusion layer of the attention module to obtain the conditional information corresponding to the first time-frequency feature and the conditional information corresponding to the target time-frequency feature includes: The target weight matrix and the first value feature are input into the first fusion sub-layer of the fusion layer to obtain the conditional information corresponding to the first time-frequency feature; The target weight matrix and the second value feature are input into the second fusion sub-layer of the fusion layer to obtain the conditional information corresponding to the target time-frequency feature.

7. The audio processing method according to claim 5, characterized in that, The weight calculation unit includes a query feature calculation layer, a key feature calculation layer, and a weight calculation layer; The step of inputting the first encoded feature and the second encoded feature into the weight calculation unit of the attention module to obtain the target weight matrix includes: At the channel dimension, the first encoded feature and the second encoded feature are concatenated to obtain the merged feature; The merged features are input into the query feature calculation layer and the key feature calculation layer to obtain query features and key features; The query features and the key features are input into the weight calculation layer to obtain the target weight matrix.

8. The audio processing method according to any one of claims 1-7, characterized in that, The target diffusion model is trained based on the following steps: Construct the initial model corresponding to the target diffusion model; Obtain the second time-frequency feature of the first sample audio in the sample mixed audio output by the initial model; Based on the second time-frequency feature of the first sample audio, obtain the estimated value of the first sample audio and the estimated value of the second sample audio in the sample mixed audio; Based on the estimated value of the first sample audio, the label value of the first sample audio, the estimated value of the second sample audio, and the label value of the second sample audio, the initial model is iteratively trained to obtain the target diffusion model.

9. An audio processing device, characterized in that, include: The time-frequency feature extraction unit is used to extract time-frequency features from the target mixed audio to obtain the target time-frequency features; The target mixed audio includes a first target audio and a second target audio; The first separation unit is used to input the target time-frequency features into the audio separation model to obtain the first time-frequency features of the first target audio. The correction unit is used to input the first time-frequency feature and the target time-frequency feature as conditions into the target diffusion model to obtain the second time-frequency feature of the first target audio. The second separation unit is used to acquire the first target audio and the second target audio based on the second time-frequency characteristics.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the audio processing method as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the audio processing method as described in any one of claims 1 to 8.

Images