A multi-channel audio upmix method, apparatus, device and storage medium

Abstract

This application discloses a multi-channel audio upmixing method, apparatus, device, and storage medium, relating to the field of signal processing technology. The method includes: converting multi-track reference audio and stereo audio into Mel spectrograms; compressing the Mel spectrograms to a low-dimensional continuous latent space to obtain latent vectors; determining audio style features and audio content features based on stereo audio and multi-track reference audio using an encoder; performing noise prediction based on a diffusion network, the noisy latent vectors, the current time step, the audio style features, and the audio content features; comparing the prediction results with the actual noise; training the diffusion network based on the comparison results; using the trained diffusion network to output a target latent vector based on the target multi-track reference audio and target stereo audio; decoding the target latent vector into a multi-track Mel spectrogram; and converting the multi-track Mel spectrogram into a multi-channel audio waveform to complete the multi-channel audio upmixing. This achieves high-quality upmixing adaptation for different multi-channel systems.

Description

Technical Field

[0001] This invention relates to the field of signal processing technology, and in particular to a multi-channel audio upmixing method, apparatus, device, and storage medium. Background Technology

[0002] In the current audio-visual content market, stereo materials still dominate, and multi-speaker playback environments such as home theaters, VR audio-visual, and in-car surround sound are widely used. When traditional stereo signals are directly adapted to multi-channel systems, there are generally core pain points such as the sound field being limited to the front two channels, the side and rear surround channels not being effectively activated, the sound image positioning only covering a limited angle, and the lack of immersive spatial enclosure, which cannot unleash the spatial rendering potential of multi-speaker arrays.

[0003] Existing stereo-to-multichannel upmixing technologies are mainly divided into two categories: one is environment generation technology, which extracts or synthesizes recording environment information and distributes it to the surround channels; the other is multichannel conversion technology, which generates additional playback channels when the number of speakers exceeds the original number of channels. Although most mainstream audio materials such as music and movies are encoded in standard formats such as stereo, 5.1, and 7.1, both traditional technologies have significant limitations when adapting to different multichannel systems: environment generation technology has insufficient accuracy in extracting environmental information, surround channel signals are prone to distortion or disconnection from the main channels, and it is difficult to restore the spatial hierarchy of the real sound field; multichannel conversion technology can only achieve a simple expansion of the number of channels and cannot dynamically optimize the sound image distribution for different systems.

[0004] Looking further, the two core methods of traditional multi-channel mixing also have shortcomings in scene adaptation: limiting the main signal to the front channel for translation, and the surround channel only carrying simple environmental signals, can only create a single experience of "the audience is in the audience seat and the sound source is concentrated in front", lacking an immersive sense of enclosure; although the all-source method realizes the translation of the signal between all speakers, it does not distinguish the spatial weight of the main sound source and the ambient sound, which can easily lead to blurred sound image positioning, reduced identification of the core sound source, and reduced artistry of the upmixed audio.

[0005] Therefore, how to address the shortcomings of traditional upmixing technology, such as insufficient adaptability, low integration of environmental and main channel signals, and difficulty in achieving both sound image positioning accuracy and spatial immersion, is a problem that needs to be considered. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a multi-channel audio upmixing method, apparatus, device, and storage medium, which can solve the shortcomings of traditional upmixing techniques, such as insufficient adaptability, low integration of environmental and main channel signals, and difficulty in simultaneously achieving sound image positioning accuracy and spatial immersion, thereby achieving high-quality upmixing adaptation under different multi-channel systems. The specific solution is as follows: Firstly, this application discloses a multi-channel audio upmixing method, comprising: The multitrack reference audio is downmixed into a stereo version to obtain stereo audio, and the multitrack reference audio and the stereo audio are converted into Mel spectrograms. The Mel spectrogram is compressed into a low-dimensional continuous latent space to obtain latent vectors. The encoder determines audio style features and audio content features based on the stereo audio and multitrack reference audio. The latent vectors include the multitrack reference audio latent vector and the target stereo audio latent vector. Noise prediction is performed based on a diffusion network, a noisy latent vector, the current time step, audio style features, and audio content features. The prediction results are compared with the actual noise, and the diffusion network is trained based on the comparison results to obtain the trained diffusion network. The actual noise is the noise added to the latent vector to obtain the noisy latent vector. The trained post-diffusion network outputs a target latent vector based on the target multitrack reference audio and the target stereo audio. The target latent vector is decoded into a multitrack Mel spectrogram, and the multitrack Mel spectrogram is converted into a multichannel audio waveform to complete multichannel audio upmixing.

[0007] Optionally, before downmixing the multitrack reference audio to a stereo version to obtain stereo audio, the method further includes: The bass channel of the original multi-channel data is removed using a multi-track music dataset, and the corresponding removed data is resampled. The loudness of the sampled data is then normalized to obtain normalized data. A vector-based amplitude translation algorithm is used to map dual-channel audio track data to virtual positioning to obtain artificially synthesized multi-channel audio. A multitrack reference audio is determined based on the normalized data and / or the multichannel audio.

[0008] Optionally, the step of downmixing the multitrack reference audio to a stereo version to obtain stereo audio includes: Based on the Dolby Digital Audio standard, multi-track reference audio is downmixed into a stereo version to obtain stereo audio.

[0009] Optionally, the noise prediction based on the diffusion network, the noisy latent vector, the current time step, audio style features, and audio content features includes: The audio content features are input into the diffusion network by channel cascading at the input end, and the audio style features are input into the deep network of the diffusion network through cross-attention mechanism or feature linear modulation, so that the diffusion network can perform noise prediction based on the noisy latent vector, the current time step, the audio style features, and the audio content features.

[0010] Optionally, comparing the corresponding prediction results with the actual noise and training the diffusion network based on the comparison results to obtain the trained diffusion network includes: The corresponding prediction results are compared with the actual noise to determine the mean square error between the prediction results and the actual noise. The mean square error is defined as the loss function; The diffusion network is back-optimized based on the loss function to obtain the trained diffusion network.

[0011] Optionally, the step of using the trained post-diffusion network to output a target latent vector based on the target multitrack reference audio and the target stereo audio includes: Based on the target multitrack reference audio and the target stereo audio, determine the corresponding target audio style features and target audio content features; The target latent vector is generated using the trained post-diffusion network based on the target audio style features, the target audio content features, and a Gaussian noise map with the same dimension as the target audio.

[0012] Optionally, converting the multitrack Mel spectrogram into a multichannel audio waveform includes: The target latent vector is decoded into a multitrack Mel spectrogram based on a variational autoencoder. The multitrack Mel spectrogram is converted into a multichannel audio waveform using a pre-trained vocoder.

[0013] Secondly, this application discloses a multi-channel audio upmixing device, comprising: An audio conversion module is used to downmix a multitrack reference audio to a stereo version to obtain stereo audio, and to convert the multitrack reference audio and the stereo audio into a Mel spectrogram. The feature determination module is used to compress the Mel spectrogram to a low-dimensional continuous latent space to obtain latent vectors, and to determine audio style features and audio content features based on the stereo audio and multitrack reference audio by an encoder; the latent vectors include multitrack reference audio latent vectors and target stereo audio latent vectors. The training module is used to predict noise based on the diffusion network, the noisy latent vector, the current time step, audio style features, and audio content features. The prediction results are compared with the actual noise, and the diffusion network is trained according to the comparison results to obtain the trained diffusion network. The actual noise is the noise added to the latent vector to obtain the noisy latent vector. The upmixing completion module is used to utilize the trained post-diffusion network to output a target latent vector based on the target multitrack reference audio and the target stereo audio, decode the target latent vector into a multitrack Mel spectrogram, and convert the multitrack Mel spectrogram into a multichannel audio waveform to complete the multichannel audio upmixing.

[0014] Thirdly, this application discloses an electronic device, including: Memory, used to store computer programs; A processor is used to execute computer programs to implement the steps of the multi-channel audio upmixing method described above.

[0015] Fourthly, this application discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned multi-channel audio upmixing method.

[0016] In this application, when performing multi-channel audio upmixing, the multi-track reference audio is first downmixed into a stereo version to obtain stereo audio. The multi-track reference audio and the stereo audio are then converted into Mel spectrograms. The Mel spectrograms are compressed into a low-dimensional continuous latent space to obtain latent vectors. An encoder determines audio style features and audio content features based on the stereo audio and the multi-track reference audio. The latent vectors include the latent vectors of the multi-track reference audio and the target stereo audio. Noise prediction is performed based on a diffusion network, the noisy latent vectors, the current time step, the audio style features, and the audio content features. The prediction results are compared with the actual noise, and the diffusion network is trained based on the comparison results to obtain a trained diffusion network. The actual noise is the noise added to the latent vectors to obtain the noisy latent vectors. Finally, the trained diffusion network outputs a target latent vector based on the target multi-track reference audio and the target stereo audio. The target latent vector is decoded into a multi-track Mel spectrogram, and the multi-track Mel spectrogram is converted into a multi-channel audio waveform to complete the multi-channel audio upmixing. As can be seen, this application achieves high-quality mapping of stereo materials into multi-channel audio with a specific spatial layout by decoupling the "content features" and "spatial style features" of the audio. By reconstructing the sound field in the global latent space through a diffusion model, and combining this with a high-precision encoder for phase estimation and waveform synthesis, the imaging clarity of the main sound source and the natural enveloping feel of spatial reverberation are ensured. This achieves accurate conversion of stereo materials into a multi-channel sound field, fully adapting to the playback needs of multi-speaker environments. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] Figure 1 This is a flowchart of a multi-channel audio upmixing method disclosed in this application; Figure 2 This is a schematic diagram of a multi-channel audio upmixing device disclosed in this application; Figure 3 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Existing stereo-to-multichannel upmixing technologies are mainly divided into two categories: one is environment generation technology, which extracts or synthesizes recording environment information and distributes it to the surround channels; the other is multichannel conversion technology, which generates additional playback channels when the number of speakers exceeds the original number of channels. Although most mainstream audio materials such as music and movies are encoded in standard formats such as stereo, 5.1, and 7.1, both traditional technologies have significant limitations when adapting to different multichannel systems: environment generation technology has insufficient accuracy in extracting environmental information, surround channel signals are prone to distortion or disconnection from the main channels, and it is difficult to restore the spatial hierarchy of the real sound field; multichannel conversion technology can only achieve a simple expansion of the number of channels and cannot dynamically optimize the sound image distribution for different systems. Furthermore, the two core methods of traditional multi-channel mixing also have shortcomings in scene adaptation: limiting the main signal to the front channel for translation, and the surround channels only carrying simple environmental signals, can only create a single experience of "the audience is in the audience seats and the sound source is concentrated in front", lacking an immersive sense of enclosure; although the omnidirectional source method realizes the translation of the signal between all speakers, it does not distinguish the spatial weight of the main sound source and the ambient sound, which can easily lead to blurred sound image positioning, reduced identification of the core sound source, and reduced artistry of the upmixed audio. In order to solve the above technical problems, this application discloses a multi-channel audio upmixing method, device, equipment and storage medium, which can solve the defects of traditional upmixing technology such as insufficient adaptation flexibility, low integration of environmental and main channel signals, and difficulty in balancing sound image positioning accuracy and spatial immersion, and achieve high-quality upmixing adaptation under different multi-channel systems.

[0021] See Figure 1 As shown, this embodiment of the invention discloses a multi-channel audio upmixing method, including: Step S11: Downmix the multitrack reference audio into a stereo version to obtain stereo audio, and convert the multitrack reference audio and the stereo audio into a Mel spectrogram.

[0022] In this embodiment, a multi-channel reference is first determined: a 7.1.4 multi-channel system is used as the standard layout, including front left, front right, rear left, and rear right speakers, which are evenly distributed on a circle with a radius of 1 meter and a height of 1.2 meters. Then, the bass channel of the original multi-channel data is removed using a multi-track music dataset, and the corresponding removed data is resampled. The loudness of the sampled data is normalized to obtain normalized data. A vector-based amplitude translation algorithm is used to map the dual-channel audio track data to virtual positioning to obtain artificially synthesized multi-channel audio. A multi-track reference audio is determined based on the normalized data and / or the multi-channel audio. Specifically, for multi-track music datasets such as Slakh2100 or MUSDB, the bass channel of the original multi-channel data is removed (retaining 7.4 channels for spatial training), the sampling rate is unified to 44.1 kHz, and loudness normalization based on LUFS (Loudness Units relative to Full Scale, an international standard for measuring sound loudness in audio processing) is performed (e.g., unified to -14 LUFS or -23 LUFS) to eliminate the interference of energy differences between different tracks on model convergence. For data with only two-channel tracks, a vector-based amplitude panning (VBAP) algorithm is used to map them to multiple virtual locations in a 7.1.4 standard layout to expand the spatial diversity of the training samples. The synthetic multi-channel audio output after mapping will be used as reference data to expand the training samples, increasing the spatial diversity of the dataset.

[0023] Next, based on the Dolby Digital Audio standard, the multitrack reference audio is downmixed into a stereo version to obtain stereo audio. That is, according to the AC-3 digital audio compression standard, all prepared multitrack reference data are downmixed into a stereo version. Then, both the multitrack reference audio and the target stereo audio are first converted into Mel spectrograms. This transforms the sound into an "image" form that AI (Artificial Intelligence) excels at processing. It is evident that this solution constructs a dual-modal paired training set that conforms to industry downmixing standards such as AC-3, and its inference pipeline is highly integrated. This system can be seamlessly embedded as a standardized plugin or microservice into existing Digital Audio Workstations (DAWs) or cloud-based streaming transcoding engines to achieve large-scale, batch, industrial-grade automated production of multichannel audio.

[0024] Step S12: Compress the Mel spectrogram to a low-dimensional continuous latent space to obtain latent vectors. Then, use an encoder to determine audio style features and audio content features based on the stereo audio and multitrack reference audio. The latent vectors include the multitrack reference audio latent vectors and the target stereo audio latent vectors.

[0025] In this embodiment, a pre-trained VAE (Variational Autoencoder) encoder is used to compress the high-dimensional Mel spectrogram into a low-dimensional continuous latent space. For multi-track audio, each track is stacked in the channel dimension and encoded together. Multi-channel reference audio: After passing through the VAE encoder, it is compressed into a low-dimensional latent vector (from the multi-track reference audio, containing spatial layout and timbre style. This is the ultimate goal that the diffusion model strives to approximate). Stereo input: After passing through the VAE encoder, a vector containing the core musical content is obtained. Style encoding: The multi-channel reference audio is simultaneously passed through an independent style encoder to extract vectors containing only the spatial layout and timbre style. In this way, this application innovatively introduces a "dual encoder architecture". By extracting the core content conditions of the target stereo through the bottom branch and by extracting the global style conditions of the reference multi-channel audio, the acoustic content and the three-dimensional spatial layout are physically decoupled in the latent space.

[0026] Step S13: Based on the diffusion network, the noisy latent vector, the current time step, audio style features, and audio content features, perform noise prediction, compare the corresponding prediction results with the actual noise, and train the diffusion network according to the corresponding comparison results to obtain the trained diffusion network; the actual noise is the noise added to the latent vector to obtain the noisy latent vector.

[0027] In this embodiment, the audio content features are input into the diffusion network through channel cascading at the input end, and audio style features are input into the deep network of the diffusion network through cross-attention mechanism or feature linear modulation, so that the diffusion network can predict noise based on the noisy latent vector, the current time step, the audio style features, and the audio content features. Then, the corresponding prediction results are compared with the actual noise to determine the mean square error between the prediction results and the actual noise; the mean square error is determined as the loss function; and the diffusion network is back-optimized based on the loss function to obtain the trained diffusion network. Specifically, when training the diffusion network, the inputs are: the training target latent variable (i.e., the latent features of the real multi-channel data), the mixed state after adding random noise at the nth time step, and the current time step t. Dual condition control: The generation process is jointly controlled by two conditions. Content condition: Ensures that the generated audio is highly consistent with the original stereo input in terms of core content such as melody and timbre; the content condition ensures that the core musical content of the generated audio is consistent with the original stereo. Optionally, this is typically injected via channel concatenation at the U-Net input. Style conditions: impart a precise sense of spatial distribution to the generated audio 7.1.4 system. Style conditions impart a precise sense of spatial distribution to the generated audio. Optionally, this is dynamically injected into the deep network of U-Net (a convolutional neural network architecture for image segmentation) through cross-attention or feature-wise linear modulation (FiLM). Additionally, conditional signals are randomly discarded during training, and during inference, the strength of the final generated audio's adherence to spatial style and stereo content is flexibly controlled by adjusting the CFG (Classifier-Free Guidance) scalar weights. Thus, this application creatively introduces classifier-free guidance weights at the inference end. Mixing engineers or automated systems can linearly and precisely control the strength of the final generated audio's adherence to the "reference spatial style" by adjusting a single scalar parameter, meeting the stringent requirements of different industry standards.

[0028] In summary, the training process of the diffusion model is as follows: Adding noise: Gradually add noise to the standard answer (latent vector) to obtain noise states z of different degrees. n .

[0029] Conditional injection: z n The current time step, content conditions, and style conditions are input into the diffusion U-Net network.

[0030] Noise prediction: U-Net needs to predict z based on these conditions. nThe noise contained therein.

[0031] Loss Calculation: Compare the noise predicted by U-Net with the actual added noise to calculate the mean squared error (loss function L). This scheme uses the standard diffusion model noise prediction loss, combined with a conditional control mechanism.

[0032] Parameter updates: The parameters of U-Net are continuously optimized through backpropagation. The goal of training is to make U-Net predict noise more and more accurately, meaning that the audio it recovers after denoising becomes closer and closer to the real 7.1.4 audio.

[0033] In each denoising time step of the inference phase, this scheme employs a classifier-free guidance strategy to extrapolate and control the generation direction. By adjusting the size of the hyperparameters, the system can flexibly control the extent to which the generated multi-channel audio approximates the input stereo content and spatial style without retraining the network parameters, thus achieving an optimal balance between generation quality and conditional compliance. The final total loss is the aforementioned diffusion loss. In this way, through the cross-attention mechanism within U-Net, the mixing style of the reference audio (such as the wide reverberation of a concert hall and the compact sound image of a recording studio) is dynamically routed and directly transferred to the target stereo audio. The entire process does not require model fine-tuning for new audio, achieving true plug-and-play functionality and high personalization.

[0034] Step S14: Using the trained post-diffusion network, output the target latent vector based on the target multitrack reference audio and the target stereo audio, decode the target latent vector into a multitrack Mel spectrogram, and convert the multitrack Mel spectrogram into a multi-channel audio waveform to complete the multi-channel audio upmixing.

[0035] In this embodiment, the corresponding target audio style features and target audio content features are determined based on the target multitrack reference audio and the target stereo audio. The target latent vector is generated using the trained diffusion network based on the target audio style features, target audio content features, and a Gaussian noise map with the same dimension as the target audio. The target latent vector is decoded into a multitrack Mel spectrogram using a variational autoencoder. A pre-trained vocoder is then used to convert the multitrack Mel spectrogram into a multi-channel audio waveform. In a specific embodiment, a pre-trained VAE decoder (parameter frozen) is used to decode the clean multi-channel latent features output by the diffusion model after multi-step denoising back into a multitrack Mel spectrogram. Waveform synthesis: A pre-trained BigVGAN vocoder is used to synthesize the Mel spectrogram of each track into a high-fidelity time-domain waveform, ensuring the clarity and phase accuracy of the final output. In this way, this application employs a heterogeneous dimensional conditional injection strategy in the generative network, eliminating the rigid dependence on the physical topology of specific channels. Seamless mapping across multiple terminals: The generated multi-channel latent variables can be flexibly mapped and adapted to home theaters (7.1.4 physical speakers), in-vehicle panoramic sound cabins, and binocular rendering terminals in VR / AR (virtual reality) / Augmented Reality (combined with HRTF (Head-Related Transfer function)) within a unified mathematical framework, solving the problem of sound field adaptation for cross-terminal playback.

[0036] In summary, this application, when performing multi-channel audio upmixing, firstly, downmixes the multi-track reference audio into a stereo version to obtain stereo audio, and then converts the multi-track reference audio and the stereo audio into Mel spectrograms. The Mel spectrograms are then compressed into a low-dimensional continuous latent space to obtain latent vectors. An encoder determines audio style features and audio content features based on the stereo audio and the multi-track reference audio. The latent vectors include the latent vectors of the multi-track reference audio and the target stereo audio. Noise prediction is performed based on a diffusion network, the noisy latent vectors, the current time step, the audio style features, and the audio content features. The prediction results are compared with the actual noise, and the diffusion network is trained based on the comparison results to obtain a trained diffusion network. The actual noise is added to the latent vectors to obtain the noisy latent vectors. Finally, the trained diffusion network outputs a target latent vector based on the target multi-track reference audio and the target stereo audio. The target latent vector is decoded into a multi-track Mel spectrogram, and the multi-track Mel spectrogram is converted into a multi-channel audio waveform to complete the multi-channel audio upmixing. As can be seen, this application achieves high-quality mapping of stereo materials into multi-channel audio with a specific spatial layout by decoupling the "content features" and "spatial style features" of the audio. By reconstructing the sound field in the global latent space through a diffusion model, and combining this with a high-precision encoder for phase estimation and waveform synthesis, the imaging clarity of the main sound source and the natural enveloping feel of spatial reverberation are ensured. This achieves accurate conversion of stereo materials into a multi-channel sound field, fully adapting to the playback needs of multi-speaker environments.

[0037] As can be seen from the previous embodiment, this application discloses a multi-channel audio upmixing method that can achieve high-quality upmixing adaptation under different multi-channel systems. Next, the multi-channel audio upmixing method will be described in conjunction with a specific embodiment.

[0038] In one specific embodiment, during the training phase: the system inputs a multi-channel reference spectrogram and a downmixed stereo spectrogram in parallel. The style encoder and VAE encoder respectively extract the style condition c. cont Target latent variable z0 and content condition c cont After being processed with noise, z0 is input into Diffusion U-Net, which predicts noise under dual conditions. The U-Net parameters are updated by calculating the mean square error (MSE) between the predicted noise and the actual noise.

[0039] Inference Phase: The user inputs the stereo audio to be upmixed, and a multi-channel reference audio track providing a spatial template (both converted to spectrograms). The system extracts c... cont and c styleThe system samples pure Gaussian noise consistent with the target dimension. Guided by dual conditions, Diffusion U-Net performs multi-step inverse denoising, outputting the fused multi-channel implicit representation. Finally, it passes through a VAE decoder and vocoder sequentially to output the final 7.1.4 multi-channel audio waveform.

[0040] In this way, this solution reconstructs the sound field in the global latent space through a diffusion model, and uses a high-precision vocoder for phase estimation and waveform synthesis, ensuring the imaging clarity of the main sound sources (such as vocals and solo instruments) and the natural sense of envelopment of spatial reverberation. Spatial Regeneration of Massive Existing Assets: This invention can losslessly and smoothly convert massive amounts of classic stereo two-channel assets from streaming platforms or music libraries into the 7.1.4 multi-channel format that conforms to modern immersive standards with extremely high acoustic fidelity, greatly expanding the content ecosystem of spatial audio.

[0041] See Figure 2 As shown, an embodiment of the present invention discloses a multi-channel audio upmixing device, comprising: The audio conversion module 11 is used to downmix the multitrack reference audio into a stereo version to obtain stereo audio, and to convert the multitrack reference audio and the stereo audio into a Mel spectrogram. The feature determination module 12 is used to compress the Mel spectrogram to a low-dimensional continuous latent space to obtain latent vectors, and to determine audio style features and audio content features based on the stereo audio and multitrack reference audio by an encoder; the latent vectors include multitrack reference audio latent vectors and target stereo audio latent vectors. Training module 13 is used to perform noise prediction based on the diffusion network, the noisy latent vector, the current time step, audio style features, and audio content features. The corresponding prediction results are compared with the actual noise, and the diffusion network is trained according to the comparison results to obtain the trained diffusion network. The actual noise is the noise added to the latent vector to obtain the noisy latent vector. The upmixing completion module 14 is used to utilize the trained post-diffusion network to output a target latent vector based on the target multitrack reference audio and the target stereo audio, decode the target latent vector into a multitrack Mel spectrogram, and convert the multitrack Mel spectrogram into a multichannel audio waveform to complete the multichannel audio upmixing.

[0042] In summary, this application, when performing multi-channel audio upmixing, firstly, downmixes the multi-track reference audio into a stereo version to obtain stereo audio, and then converts the multi-track reference audio and the stereo audio into Mel spectrograms. The Mel spectrograms are then compressed into a low-dimensional continuous latent space to obtain latent vectors. An encoder determines audio style features and audio content features based on the stereo audio and the multi-track reference audio. The latent vectors include the latent vectors of the multi-track reference audio and the target stereo audio. Noise prediction is performed based on a diffusion network, the noisy latent vectors, the current time step, the audio style features, and the audio content features. The prediction results are compared with the actual noise, and the diffusion network is trained based on the comparison results to obtain a trained diffusion network. The actual noise is added to the latent vectors to obtain the noisy latent vectors. Finally, the trained diffusion network outputs a target latent vector based on the target multi-track reference audio and the target stereo audio. The target latent vector is decoded into a multi-track Mel spectrogram, and the multi-track Mel spectrogram is converted into a multi-channel audio waveform to complete the multi-channel audio upmixing. As can be seen, this application achieves high-quality mapping of stereo materials into multi-channel audio with a specific spatial layout by decoupling the "content features" and "spatial style features" of the audio. By reconstructing the sound field in the global latent space through a diffusion model, and combining this with a high-precision encoder for phase estimation and waveform synthesis, the imaging clarity of the main sound source and the natural enveloping feel of spatial reverberation are ensured. This achieves accurate conversion of stereo materials into a multi-channel sound field, fully adapting to the playback needs of multi-speaker environments.

[0043] In some specific embodiments, the apparatus is further configured to remove the bass channel of the original multichannel data using a multitrack music dataset, resample the corresponding removed data, normalize the loudness of the sampled data to obtain normalized data, map the dual-channel audio track data to virtual positioning using a vector-based amplitude shift algorithm to obtain artificially synthesized multichannel audio, and determine a multitrack reference audio based on the normalized data and / or the multichannel audio.

[0044] In some specific embodiments, the audio conversion module 11 can be used to downmix multitrack reference audio into a stereo version based on the Dolby Digital Audio standard to obtain stereo audio.

[0045] In some specific embodiments, the training module 13 can be used to input the audio content features into the diffusion network by channel cascading at the input end of the diffusion network, and input the audio style features into the deep network of the diffusion network through cross-attention mechanism or feature linear modulation, so that the diffusion network can perform noise prediction based on the noisy latent vector, the current time step, the audio style features and the audio content features.

[0046] In some specific embodiments, the training module 13 can be used to compare the corresponding prediction results with the actual noise to determine the mean square error between the prediction results and the actual noise; determine the mean square error as a loss function; and perform reverse optimization on the diffusion network based on the loss function to obtain the trained diffusion network.

[0047] In some specific embodiments, the upmixing completion module 14 can be used to determine the corresponding target audio style features and target audio content features based on the target multitrack reference audio and the target stereo audio; and use the trained post-diffusion network to generate the target latent vector based on the target audio style features, the target audio content features, and a Gaussian noise map with the same dimension as the target audio.

[0048] In some specific embodiments, the upmixing completion module 14 can be used to decode the target latent vector into a multitrack Mel spectrogram based on a variational autoencoder; and to convert the multitrack Mel spectrogram into a multichannel audio waveform using a pre-trained vocoder.

[0049] Furthermore, embodiments of this application also disclose an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0050] Figure 3 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the multi-channel audio upmixing method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0051] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0052] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0053] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the multi-channel audio upmixing method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0054] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned multi-channel audio upmixing method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0055] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0056] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0057] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0058] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0059] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multi-channel audio upmixing method, characterized in that, include: The multitrack reference audio is downmixed into a stereo version to obtain stereo audio, and the multitrack reference audio and the stereo audio are converted into Mel spectrograms. The Mel spectrogram is compressed into a low-dimensional continuous latent space to obtain latent vectors. The encoder determines the audio style features and audio content features based on the stereo audio and multitrack reference audio. The latent vectors include multitrack reference audio latent vectors and target stereo audio latent vectors; Noise prediction is performed based on the diffusion network, the noisy latent vector, the current time step, audio style features, and audio content features. The prediction results are compared with the actual noise, and the diffusion network is trained based on the comparison results to obtain the trained diffusion network. The actual noise is the noise added to the latent vector to obtain the noisy latent vector; The trained post-diffusion network outputs a target latent vector based on the target multitrack reference audio and the target stereo audio. The target latent vector is decoded into a multitrack Mel spectrogram, and the multitrack Mel spectrogram is converted into a multichannel audio waveform to complete multichannel audio upmixing.

2. The multi-channel audio upmixing method according to claim 1, characterized in that, Before downmixing the multitrack reference audio to a stereo version to obtain stereo audio, the process also includes: The bass channel of the original multi-channel data is removed using a multi-track music dataset, and the corresponding removed data is resampled. The loudness of the sampled data is then normalized to obtain normalized data. A vector-based amplitude translation algorithm is used to map dual-channel audio track data to virtual positioning to obtain artificially synthesized multi-channel audio. A multitrack reference audio is determined based on the normalized data and / or the multichannel audio.

3. The multi-channel audio upmixing method according to claim 1, characterized in that, The process of downmixing multitrack reference audio into a stereo version to obtain stereo audio includes: Based on the Dolby Digital Audio standard, multi-track reference audio is downmixed into a stereo version to obtain stereo audio.

4. The multi-channel audio upmixing method according to claim 1, characterized in that, The noise prediction based on the diffusion network, the noisy latent vector, the current time step, audio style features, and audio content features includes: The audio content features are input into the diffusion network by channel cascading at the input end, and the audio style features are input into the deep network of the diffusion network through cross-attention mechanism or feature linear modulation, so that the diffusion network can perform noise prediction based on the noisy latent vector, the current time step, the audio style features, and the audio content features.

5. The multi-channel audio upmixing method according to claim 1, characterized in that, The step of comparing the corresponding prediction results with the actual noise, and training the diffusion network based on the comparison results to obtain the trained diffusion network includes: The corresponding prediction results are compared with the actual noise to determine the mean square error between the prediction results and the actual noise. The mean square error is defined as the loss function; The diffusion network is back-optimized based on the loss function to obtain the trained diffusion network.

6. The multi-channel audio upmixing method according to claim 1, characterized in that, The step of using the trained diffusion network to output a target latent vector based on the target multitrack reference audio and the target stereo audio includes: Based on the target multitrack reference audio and the target stereo audio, determine the corresponding target audio style features and target audio content features; The target latent vector is generated using the trained post-diffusion network based on the target audio style features, the target audio content features, and a Gaussian noise map with the same dimension as the target audio.

7. The multi-channel audio upmixing method according to any one of claims 1 to 6, characterized in that, The step of converting the multi-track Mel spectrogram into a multi-channel audio waveform includes: The target latent vector is decoded into a multitrack Mel spectrogram based on a variational autoencoder. The multitrack Mel spectrogram is converted into a multichannel audio waveform using a pre-trained vocoder.

8. A multi-channel audio upmixing device, characterized in that, include: An audio conversion module is used to downmix a multitrack reference audio to a stereo version to obtain stereo audio, and to convert the multitrack reference audio and the stereo audio into a Mel spectrogram. The feature determination module is used to compress the Mel spectrogram to a low-dimensional continuous latent space to obtain latent vectors, and to determine audio style features and audio content features based on the stereo audio and multitrack reference audio by the encoder. The latent vectors include multitrack reference audio latent vectors and target stereo audio latent vectors; The training module is used to predict noise based on the diffusion network, the noisy latent vector, the current time step, audio style features, and audio content features. The corresponding prediction results are compared with the actual noise, and the diffusion network is trained according to the corresponding comparison results to obtain the trained diffusion network. The actual noise is the noise added to the latent vector to obtain the noisy latent vector; The upmixing completion module is used to utilize the trained post-diffusion network to output a target latent vector based on the target multitrack reference audio and the target stereo audio, decode the target latent vector into a multitrack Mel spectrogram, and convert the multitrack Mel spectrogram into a multichannel audio waveform to complete the multichannel audio upmixing.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program to implement the steps of the multi-channel audio upmixing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the multi-channel audio upmixing method as described in any one of claims 1 to 7.

Images