Model training method, model-based audio processing method and corresponding device

Abstract

The invention discloses a model training method and a model-based audio processing method and corresponding devices, and the method comprises the steps: extracting a first background sound audio in a stereo audio sample, carrying out the time domain addition of a left channel background sound audio and a right channel background sound audio, and obtaining a channel mixed audio, determining first phase information of the sound channel mixed audio and second phase information of the first background audio; determining a monaural input signal based on the first phase information and the first background sound audio, and determining a stereo sound effect target signal based on the second phase information and the first background sound audio; and performing model training on the initial preset model based on the audio training data, wherein the trained preset model is used as an audio processing model for predicting stereo effect audio based on monophonic audio. The stereo effect target signal is constructed, the unique spatial characteristics of the background sound audio in each sound channel are extracted, the stereo effect is enhanced, and the stereo effect target signal is used as a supervision signal, so that the preset model can better learn the difference characteristics of the left and right sound channels.

Description

Technical Field

[0001] This disclosure relates to the fields of audio processing technology and model technology, specifically to a model training method and a model-based audio processing method and corresponding apparatus. Background Technology

[0002] Mono-to-stereo rendering aims to synthesize spatialized stereo audio from a given mono input to enhance immersion and realism, and is widely used in music production, games, and virtual reality.

[0003] In related technologies, a combination of a mid-signal and a predicted side-signal can be constructed to enable the model to learn subtle differences between the left and right audio channels. The side-signal captures the difference information between the two channels, while the mid-signal fuses the common information from both channels. This characteristic makes it difficult to clearly deconstruct the stereo effect structure, leading to ambiguity in the supervision signal for model training. This, in turn, affects the model's training performance and results in poor spatial perception reproduction of the generated stereo audio. Summary of the Invention

[0004] This content section is provided to briefly introduce the concepts, which will be described in detail in the subsequent detailed description section. This content section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] Firstly, this disclosure provides a model training method, the model training method comprising: Extract the first background audio from the stereo audio sample, the first background audio including the left channel background audio and the right channel background audio; The left channel background audio and the right channel background audio are added together in the time domain to obtain a channel mixed audio, and the first phase information of the channel mixed audio and the second phase information of the first background audio are determined. A mono input signal is determined based on the first phase information and the first background audio, and a stereo target signal is determined based on the second phase information and the first background audio, wherein the mono input signal and the stereo target signal together form audio training data; The initial preset model is trained based on the audio training data, and the trained preset model serves as an audio processing model for predicting stereo audio based on mono audio.

[0006] Secondly, this disclosure provides a model-based audio processing method, the audio processing method comprising: The mono audio is split into human voice audio and second background audio. Based on the second background sound audio and the audio processing model, stereo sound audio is obtained, which includes left channel sound audio and right channel sound audio. The audio processing model is trained based on the model training method described in the first aspect above. The left channel audio, the second background audio, and the human voice audio are mixed to obtain the left channel audio. The right channel audio, the second background audio, and the human voice audio are mixed to obtain the right channel audio. Stereo audio is obtained based on the left channel audio and the right channel audio.

[0007] Thirdly, this disclosure provides a model training apparatus, the model training apparatus comprising: The extraction module is used to extract the first background audio from the stereo audio sample, the first background audio including the left channel background audio and the right channel background audio; The mixing module is used to perform time-domain addition of the left channel background audio and the right channel background audio to obtain the channel mixed audio, and to determine the first phase information of the channel mixed audio and the second phase information of the first background audio. The determining module is used to determine a mono input signal based on the first phase information and the first background audio, and to determine a stereo target signal based on the second phase information and the first background audio, wherein the mono input signal and the stereo target signal together form audio training data; The model training module is used to train an initial preset model based on the audio training data, wherein the trained preset model serves as an audio processing model for predicting stereo audio based on mono audio.

[0008] Fourthly, this disclosure provides a model-based audio processing apparatus, the audio processing apparatus comprising: The splitting module is used to split mono audio into human voice audio and second background audio. The model processing module is used to obtain stereo audio based on the second background sound audio and the audio processing model. The stereo audio includes left channel audio and right channel audio. The audio processing model is trained based on the model training method described in the first aspect above. An audio processing module is used to mix the left channel sound effect audio, the second background sound audio, and the human voice audio to obtain left channel audio, mix the right channel sound effect audio, the second background sound audio, and the human voice audio to obtain right channel audio, and obtain stereo audio based on the left channel audio and the right channel audio.

[0009] Fifthly, this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first or second aspect.

[0010] Sixthly, this disclosure provides an electronic device, comprising: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method in the first or second aspect.

[0011] In a seventh aspect, this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first or second aspect.

[0012] The above technical solution involves temporally adding the left and right channel background audio samples in a stereo audio sample to obtain a mixed audio channel. It then determines the first phase information of the mixed audio channel and the second phase information of the background audio channel. Based on the first phase information and the first background audio channel, a mono input signal is determined, and based on the second phase information and the first background audio channel, a stereo effect target signal is determined. This allows for the generation of audio training data based on the mono input signal and the stereo effect target signal. Finally, the initial preset model is trained using the audio training data to obtain an audio processing model capable of predicting stereo audio effects based on mono audio. This method constructs input data for the model based on the phase information of the mixed audio channels and the background audio. By preserving the original phase information, it avoids phase cancellation problems in the audio training data, reduces amplitude loss, and constructs a stereo target signal based on the phase information and background audio. It extracts the unique spatial characteristics of the background audio in each channel, enhances the stereo effect, and serves as a supervision signal, enabling the preset model to better learn the differences between the left and right channels. This improves the prediction effect of the audio processing model on stereo audio, so that when generating stereo audio, the audio processing model can improve the spatial perception reproduction effect of stereo audio.

[0013] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0014] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings: Figure 1 This is a flowchart illustrating a model training method according to an exemplary embodiment of the present disclosure; Figure 2 This is a schematic diagram illustrating an audio processing architecture according to an exemplary embodiment of the present disclosure; Figure 3 This is a schematic flowchart illustrating a model-based audio processing method according to an exemplary embodiment of the present disclosure; Figure 4 This is a comparative schematic diagram illustrating a stereo audio signal according to an exemplary embodiment of the present disclosure; Figure 5 This is a block diagram illustrating a model training apparatus according to an exemplary embodiment of the present disclosure; Figure 6 This is a block diagram illustrating a model-based audio processing apparatus according to an exemplary embodiment of the present disclosure; Figure 7 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0017] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0018] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0019] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0020] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0021] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0022] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0023] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0024] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0025] Meanwhile, it is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0026] It's important to note that the perception of stereo primarily relies on interaural time difference (ITD) and interaural level difference (ILD). ITD is the time difference between the arrival of the same sound at both ears, while ILD is the amplitude difference between the arrival of the same sound at both ears. As spatial audio becomes increasingly important in multimedia experiences, the need for methods to convert mono recordings into perceptibly believable stereo audio is growing. Examples include hand-designed signal processing techniques or predefined translation heuristics implemented in digital audio workstations (DAWs), but these lack flexibility and generalization capabilities.

[0027] With the development of model technology, especially generative models, model-based methods for synthesizing stereo audio from mono input have emerged. For example, additional modal information or spatial priors can be introduced to enhance stereo perception, or end-to-end diffusion-based frameworks can be used to directly predict stereo audio from mono input without additional guidance. However, in the absence of clear spatial constraints, the model is prone to degenerate into copying the input signal, thus lacking effective stereo perception.

[0028] Another method combines predicted side signals with a mid-signal. This involves constructing a combination of the mid-signal and predicted side signals, and applying a complex mask to the mid-signal to allow the model to learn subtle differences between the left and right channels. The mid-signal is the sum of the left and right channel signals, containing identical content in both channels, such as centered vocals or instruments. The side signals are the difference between the left and right channel signals, containing only the differences, such as stereo effects or image-shifted instruments. In other words, the side signals capture the differences between the two channels, while the mid-signal incorporates the common information. This characteristic makes it difficult to explicitly deconstruct the stereo effect structure. Consequently, the supervision signal for model training is ambiguous, making it difficult to clearly express the independent stereo effect structure of each sound effect. Furthermore, this method lacks explicit modeling of spatial asymmetry between channels, limiting its ability to represent realistic stereo effects. This affects the model's training performance, resulting in poor spatial perception reproduction in the generated stereo audio.

[0029] In view of this, the present disclosure provides a model training method and a model-based audio processing method and corresponding apparatus to solve the above-mentioned technical problems.

[0030] The embodiments of this disclosure will be further explained below with reference to the accompanying drawings.

[0031] Figure 1This is a flowchart illustrating a model training method according to an exemplary embodiment of the present disclosure, with reference to... Figure 1 The model training method may include the following steps: S101: Extract the first background audio from the stereo audio sample. The first background audio includes the left channel background audio and the right channel background audio.

[0032] For example, stereo audio samples can be obtained from an audio dataset, which can be proprietary or open-source; this disclosure makes no restriction on this. If each stereo audio sample in the audio dataset includes stereo mixed audio and independent audio, the human voice audio in the independent audio can be directly subtracted from the stereo mixed audio to obtain the background audio. If each stereo audio sample in the audio dataset does not include independent audio, the human voice audio can be extracted from the stereo mixed audio using an audio separation model, and then the extracted human voice audio can be subtracted from the stereo mixed audio to obtain the background audio. The specific method can be determined according to the actual situation; this disclosure makes no restriction on this.

[0033] It is important to note that multiple stereo audio samples can be selected, covering a variety of music styles, to increase the richness of the audio training data and thus improve the generalization ability of the audio processing model.

[0034] S102: Add the left channel background audio and the right channel background audio in the time domain to obtain the channel mixed audio, and determine the first phase information of the channel mixed audio and the second phase information of the first background audio.

[0035] For example, the time-domain signal of the first background audio. Where t represents the time variable, This represents the time-domain signal of the left channel background audio. This represents the time-domain signal of the right channel background audio. This represents a two-dimensional real number array containing 2×N elements, where N is a positive integer.

[0036] Then, the time-domain signals of the left channel background audio and the right channel background audio are summed in the time domain according to the following formula to obtain the time-domain signal of the mixed audio. :

[0037] Among them, the mixed audio is equivalent to a single mono audio.

[0038] In one possible manner, determining the first phase information of the mixed audio channels and the second phase information of the first background audio channel includes: extracting the first phase information from the audio signal information obtained after performing a short-time Fourier transform on the mixed audio channels; extracting the left channel phase information from the audio signal information obtained after performing a short-time Fourier transform on the left channel background audio channel, and extracting the right channel phase information from the audio signal information obtained after performing a short-time Fourier transform on the right channel background audio channel; the second phase information includes the left channel phase information and the right channel phase information.

[0039] For example, the time-domain signal of mixed audio channels. After applying the Short-Time Fourier Transform (STFT), audio signal information describing the amplitude and phase information of the signal at different time points and frequencies is obtained. From this, the first phase information at different frequencies f and different time points t can be extracted. This is to facilitate the subsequent calculation of mono input data based on phase information in the model.

[0040] Correspondingly, the time-domain signal of the left channel background audio After applying the short-time Fourier transform, the left channel phase information at different frequencies f and different time points t is extracted. The time-domain signal of the right channel background audio After applying the short-time Fourier transform, the right channel phase information at different frequencies f and different time points t is extracted. This is so that the target output of the model can be calculated subsequently based on phase information.

[0041] It should be noted that the amplitude and phase information mentioned above are amplitude and phase information in the frequency domain.

[0042] Of course, other methods can be used to extract phase information from time-domain signals, and this disclosure does not limit this.

[0043] S103: Determine the mono input signal based on the first phase information and the first background audio, and determine the stereo target signal based on the second phase information and the first background audio, wherein the mono input signal and the stereo target signal together form audio training data.

[0044] S104: Train the initial preset model based on the audio training data, wherein the trained preset model serves as the audio processing model for predicting stereo audio based on mono audio.

[0045] Using the above method, the input data for the model is constructed based on the phase information of the mixed audio channels and the background audio. Since the original phase information is preserved, the phase cancellation problem in the audio training data can be avoided, and the amplitude loss can be reduced. Furthermore, the stereo effect target signal is constructed based on the phase information of the background audio and the background audio, and the unique spatial characteristics of the background audio in each channel are extracted to enhance the stereo effect. As a supervision signal, it enables the preset model to better learn the difference features of the left and right channels, thereby improving the prediction effect of the audio processing model on the stereo audio. In order to improve the spatial perception restoration effect of the stereo audio when generating stereo audio, the audio processing model can be used to improve the stereo audio.

[0046] In one possible manner, determining a mono input signal based on first phase information and first background audio includes: determining a first amplitude of the left channel background audio in the frequency domain and a second amplitude of the right channel background audio in the frequency domain, and adding the first amplitude and the second amplitude to obtain a total frequency domain amplitude; determining a mono input signal based on the first phase information and the total frequency domain amplitude.

[0047] For example, after applying the short-time Fourier transform to the time-domain signals of the left and right channel background audio, the amplitudes of the left channel background audio at different frequencies f and different time points t can be extracted. And the amplitude of the right channel background audio at different frequencies f and different time points t Adding the two together gives the total frequency domain amplitude. Where L represents the left channel and R represents the right channel.

[0048] The combined amplitude is then calculated based on the following formula. The phase information of the time-domain signal of the mixed audio is combined to construct a complex spectrum, and then an inverse short-time Fourier transform (ISTFT) is performed on it to obtain the time-domain signal, which serves as the final mono input signal. :

[0049] in, Phase information characterizing the time-domain signal of complex-form mixed audio channels.

[0050] Therefore, using the aforementioned mono input signal as the model input in the audio training data can effectively avoid the phase cancellation problem that occurs when directly averaging the waveform in the time domain, thereby reducing amplitude loss.

[0051] Of course, other methods can also be used to convert time-domain signals to frequency-domain signals, and the specific method can be determined according to the requirements. This disclosure does not impose any restrictions on this.

[0052] In one possible manner, the stereo target signal is determined based on the second phase information and the first background audio, including: determining the first amplitude of the left channel background audio in the frequency domain and the second amplitude of the right channel background audio in the frequency domain, and determining the left channel amplitude component and the right channel amplitude component based on the first amplitude and the second amplitude; and determining the stereo target signal based on the second phase information, the left channel amplitude component, and the right channel amplitude component.

[0053] For example, to model spatial contrast, a difference map between the amplitudes of the left and right channels can be defined first. This difference map can then be decomposed into two non-negative amplitude components. The amplitude components corresponding to the two channels can be combined with the original phase information of their respective channels, thereby highlighting the differences in energy distribution between the left and right channels and enhancing the stereo effect.

[0054] In one possible manner, determining the left channel amplitude component and the right channel amplitude component based on the first amplitude and the second amplitude includes: subtracting the second amplitude from the first amplitude to obtain a first amplitude difference, and taking the larger of the first amplitude difference and 0 as the left channel amplitude component; subtracting the first amplitude from the second amplitude to obtain a second amplitude difference, and taking the larger of the second amplitude difference and 0 as the right channel amplitude component.

[0055] For example, the difference graph between the amplitudes of the left and right channels can be defined based on the following calculation formula. :

[0056] Therefore, based on the following calculation formula, It is decomposed into two non-negative amplitude components. :

[0057] Of course, the two non-negative amplitude components can also be obtained directly based on the following calculation formula. :

[0058] The specific choice can be made according to needs, and this disclosure does not impose any restrictions.

[0059] in, It preserves the portion of channel C that dominates in the time-frequency region while suppressing the influence of the other channel, thus effectively isolating the spatial characteristics of each channel.

[0060] In one possible manner, the second phase information includes the left channel phase information of the left channel background audio and the right channel phase information of the right channel background audio. Determining the stereo target signal based on the second phase information, the left channel amplitude component, and the right channel amplitude component includes: multiplying the left channel phase information by the left channel amplitude component to obtain a first complex spectrum; performing an inverse short-time Fourier transform on the first complex spectrum to obtain the left channel sound target signal; multiplying the right channel phase information by the right channel amplitude component to obtain a second complex spectrum; performing an inverse short-time Fourier transform on the second complex spectrum to obtain the right channel sound target signal; and determining the left channel sound target signal and the right channel sound target signal as the stereo target signal.

[0061] For example, the complex spectrum can be obtained by combining the amplitude components corresponding to the two channels with their original phase information based on the following calculation formula, that is, reconstructing the background sound stereo effect in the frequency domain and converting it back to the time domain to obtain the final stereo effect target signal. :

[0062] Among them, stereo sound target signal That is, the time-domain signal of the background stereo sound effect corresponding to the left and right channels, and this signal is used as the target output of the model.

[0063] Of course, other methods can also be used to convert time-domain signals to frequency-domain signals, and the specific method can be determined according to the requirements. This disclosure does not impose any restrictions on this.

[0064] Therefore, based on stereo audio samples, corresponding mono audio samples are constructed as model inputs, while corresponding background stereo effects for the left and right channels are constructed as training targets, forming training data pairs. Through this pairing of input and target audio training data, the unique spatial characteristics of each channel can be preserved, and the differences in energy distribution between the left and right channels can be highlighted, thereby enhancing the stereo effect.

[0065] It is worth noting that, in order to improve the efficiency of model training, a complete stereo audio can be divided into multiple audio segments according to a preset duration, and the audio segments can be used as stereo audio samples to construct corresponding training data pairs. Alternatively, the mono audio and background stereo effects obtained based on the complete stereo audio can be divided into multiple one-to-one mono audio segments and background stereo effect segments according to a preset duration, and the one-to-one mono audio segments and background stereo effect segments can be used as a training data pair. The preset duration can be set according to the needs, and this disclosure does not impose any restrictions on it.

[0066] Furthermore, it is possible to construct such as Figure 2The deep learning-based mono audio to stereo sound rendering framework shown can be divided into three models: a pre-trained audio separation model to separate human voice audio and obtain background audio, an audio processing model to generate channel-specific background stereo sound effects, and a pre-trained vocoder to restore the Mel spectrum representation output by the audio processing model to time-domain audio.

[0067] Among the possible approaches, the audio processing model is the conditional stream matching model.

[0068] For example, the Conditional Flow Matching (CFM) model can model stereo rendering as a conditional generation task based on the principle of flow matching. Through the Conditional Flow Matching (CFM) mechanism, the audio processing model learns to generate background stereo effects for the left and right channels that conform to spatial perception characteristics, starting from random noise and under the condition of mono background audio.

[0069] It is worth noting that the goal of flow matching is to learn a probabilistic path that evolves over time t. This path will distribute the data. Mapped to a standard Gaussian baseline distribution This process is achieved by solving the following divine ordinary differential equation:

[0070] in, This represents the trajectory starting from the initial point x at time t. Characterizes a time-dependent vector field that describes the state space at each time point of a data point x. The direction and magnitude of change, and the evolution of the overall data distribution over time, form a probability distribution path. Ultimately from the target distribution The sampling can be achieved by solving the initial value problem of the above calculation formula 1.

[0071] In one possible approach, training an initial preset model based on audio training data includes: repeatedly performing the following steps until preset training completion conditions are met to obtain a trained preset model: randomly sampling a first noise vector from a Gaussian noise vector; determining an input vector at each time step based on the first noise vector and the stereo target signal, and determining a reference velocity vector at each time step based on the input vector at each time step; inputting a first mono condition and the input vector at each time step into the preset model to obtain a predicted velocity vector at each time step, where the first mono condition is the Mel spectrum corresponding to the mono input signal; determining a model loss based on the difference between the predicted velocity vector and the reference velocity vector at each time step, and updating the parameters of the preset model based on the model loss, so that the preset model learns the optimal flow path from noise to stereo distribution under given mono conditions.

[0072] For example, the model loss is determined based on the difference between the predicted velocity vector and the reference velocity vector at each time step, and the parameters of the preset model are updated based on the model loss so that the preset model learns the optimal flow path from noise to stereo effect distribution under given mono conditions. In this way, the trained audio processing model can predict stereo effect audio based on mono audio.

[0073] In one possible approach, based on the first noise vector and the stereo target signal, the input vector at each time step is determined, and based on the input vector at each time step, the reference velocity vector at each time step is determined, including: for each time step, multiplying the time step scalar with the Mel spectrum corresponding to the stereo target signal to obtain a first product, determining the difference between the upper limit threshold of the preset time interval and the time step scalar, multiplying the difference with the first noise vector to obtain a second product, determining the sum of the first product and the second product as the input vector at the time step, and differentiating the input vector to obtain the reference velocity vector at the time step.

[0074] For example, based on a mono input signal and stereo sound target signal The training data pair is first processed by converting the two signals into a Mel-spectrum representation. Specifically, the mono input signal... and stereo sound target signal The transformation can be performed using a short-time Fourier transform, and then the corresponding Mel spectrum representation can be calculated. and , or as Figure 2 The Mel filter bank shown can be transformed, for example, to extract a 128-dimensional Mel spectrum, but this disclosure does not limit this process.

[0075] For example, such as Figure 2 As shown, from a random noise To begin, learn a streaming model that represents the Mel spectrum of a given mono input signal. Under the given conditions, generate the corresponding stereo sound effect X1 (i.e. ), that is, based on conditional information The random noise X0 is mapped to the stereo effect X1.

[0076] For example, based on a time-dependent stream mapping , representing the transition between two density functions. At each time step scalar t∈[0,1], the stream function... It is a linear interpolation between X0 and X1, that is, the input vector at each time step:

[0077] Furthermore, based on the flow model, Equation 2 is replaced with Equation 1 to obtain the reference velocity vector. :

[0078] In this implementation, a neural network can be defined. To fit this vector field During training, the preset model is represented by the Mel spectrum of the mono input signal. As a conditional input, it guides the generation of background stereo effects. This conditionalization strategy enables the trained audio processing model to generate corresponding background stereo effects based on the input mono audio without relying on any additional modal information as guidance. Furthermore, by utilizing the rich time-frequency structure encoded in the Mel spectrum, the model can learn to capture spatial cues related to stereo effects, especially background features related to interaural intensity differences, thereby generating more natural stereo effects.

[0079] Of course, other methods can also be used to convert time-domain signals to frequency-domain signals, and the specific method can be determined according to the requirements. This disclosure does not impose any restrictions on this.

[0080] Furthermore, during training, the conditional flow matching loss function can be used to optimize the model, minimizing the difference between the model's predicted output and the target output. For example, optimization can be performed based on the following loss function by calculating the L2 loss between the model's predicted flow function and the target flow function:

[0081] in, Indicates the model loss. (·) indicates that the variables X, t, and probability distribution are related. Take the overall expectation, ( The predicted output of the model is represented by , which indicates the evolution of the stream function from the initial noise X0 to the generated stereo effect. This represents the model's target output. The goal is to achieve a vector field that the model learns. Approximate ideal vector field .

[0082] This can be understood as follows: in each parameter update iteration, the model calculates the corresponding loss value at several time steps t, and averages these losses as the overall loss for that iteration. If the overall loss is greater than a preset threshold, subsequent iterations of training continue; if the overall loss is less than or equal to the threshold, training terminates, thus obtaining the final audio processing model.

[0083] Using the above method, the loss function enables the model to optimize its generation process, minimizing the difference between the two at several time steps t.

[0084] Of course, other loss functions can also be used to calculate the model loss, and the specific method can be determined according to the requirements. This disclosure does not impose any restrictions on this.

[0085] It is worth noting that the model input can be divided into several small blocks, such as 4×4 blocks, for encoding. Furthermore, the preset model can adopt a dual-axis Transformer architecture with a feature dimension of 384, containing 12 modules, each with 12 attention heads, and incorporating rotational positional embedding on both the time and frequency axes. For conditional modeling, this embodiment can use the time-frequency features as conditional inputs after projection processing via a 2D patch embedder and an MLP (Multi-Layer Perceptron). Specific settings can be configured according to requirements, and this disclosure does not impose any limitations on this.

[0086] This embodiment provides a model training method, including: extracting a first background sound audio from stereo audio samples, the first background sound audio including a left channel background sound audio and a right channel background sound audio; performing time-domain addition on the left channel background sound audio and the right channel background sound audio to obtain a channel mixed audio, and determining a first phase information of the channel mixed audio and a second phase information of the first background sound audio, the second phase information including the left channel phase information of the left channel background sound audio and the right channel phase information of the right channel background sound audio; determining a mono input signal based on the first phase information and the first background sound audio, and determining a stereo effect target signal based on the left channel phase information, the right channel phase information, and the first background sound audio, wherein the mono input signal and the stereo effect target signal together form audio training data; and training an initial preset model based on the audio training data, wherein the trained preset model serves as an audio processing model for predicting stereo effect audio based on mono audio.

[0087] Figure 3 This is a model-based audio processing method illustrated in an exemplary embodiment of the present disclosure, with reference to... Figure 3 The audio processing method may include the following steps: S301: Perform audio splitting on the mono audio to obtain human voice audio and second background audio.

[0088] In one possible approach, the mono audio is split into human voice audio and a second background sound audio, including: extracting human voice from the mono audio using an audio separation model to obtain human voice audio; and subtracting the human voice audio from the mono audio to obtain the second background sound audio.

[0089] For example, such as Figure 2 As shown, human voice audio can be extracted based on an audio separation model, and then the background audio can be obtained by subtracting the human voice audio from the mono audio, in order to simplify the audio data to be processed and improve the generation efficiency of stereo effects.

[0090] Of course, in other possible ways, it is also possible to extract human voice audio and background sound audio directly from mono audio through a model, and this disclosure does not limit this.

[0091] S302: Based on the second background audio and audio processing model, stereo audio is obtained, which includes left channel audio and right channel audio.

[0092] The audio processing model was trained using the aforementioned model training method.

[0093] S303: Mix the left channel sound effect audio, the second background sound audio, and the human voice audio to obtain the left channel audio; mix the right channel sound effect audio, the second background sound audio, and the human voice audio to obtain the right channel audio; and obtain stereo audio based on the left channel audio and the right channel audio.

[0094] The method described above, based on a residual inference strategy, does not directly generate a complete stereo waveform. Instead, the audio processing model directly predicts the stereo background sound effects of the left and right channels based on the mono background sound audio, effectively enhancing the modeling ability for asymmetric spatial cues. This prediction is then added to the mono voice audio and background sound audio to form the final stereo audio. Because the model retains the original mono phase information during prediction, it better restores the detailed energy distribution between the left and right channels, improving the stereo effect. Furthermore, it does not rely on additional spatial priors or complex parameter searches, making it more practical and scalable, and adaptable to mono-to-stereo conversion tasks in various scenarios.

[0095] In one possible approach, stereo audio is obtained based on a second background sound audio and an audio processing model, including: randomly sampling a second noise vector from a Gaussian noise vector; inputting the second noise vector and a second mono condition into the audio processing model, so that the audio processing model outputs a second stereo effect prediction signal based on the second noise vector under the second mono condition, where the second mono condition is the Mel spectrum of the second background sound audio; and performing a time-domain transformation on the second stereo effect prediction signal using a vocoder to obtain the stereo audio.

[0096] For example, combining the following formula and Figure 2 The processing steps shown yield the final stereo audio. :

[0097]

[0098] in, This represents the human voice audio extracted from a mono audio stream. This indicates background audio, and... and It is copied into two channels to fit the format of the final mixed stereo audio calculation. For example... Figure 2 As shown, in the reasoning process, the background audio is first calculated. The Mel spectrum is used as the conditional input cond to the model. Then, initial noise is sampled from a Gaussian noise vector as the variable Z, and the trained audio processing model is used to predict the background stereo effect in the latent space. .in, This means that during the model inference process, a numerical solver is used to solve ordinary differential equations, which evolve step by step over the time interval 0 to 1, ultimately yielding the model's prediction at time step 1. Finally, a pre-trained vocoder is used to generate the stereo sound effect for the output background audio. The stereo waveform color is converted from the frequency domain to the time domain, i.e., stereo audio. .

[0099] Then, the stereo audio, original background audio, and vocal audio are mixed in proportion to obtain stereo audio. This not only better reproduces the subtle energy distribution between the left and right channels, improving the stereo effect, but also eliminates the need for additional spatial priors or complex parameter searches, making it more practical and scalable, and adaptable to various mono-to-stereo conversion tasks.

[0100] It should be noted that because the Mel spectrum of stereo sound effects is specially constructed, it may differ from the actual complex spectrum, resulting in unnatural phase information. Therefore, to enhance phase consistency and improve perceptual realism, the phase of the mixed two-channel audio can be uniformly replaced with the phase of the original single-channel audio, instead of relying on the phase predicted by the vocoder, to obtain the final rendered stereo audio.

[0101] It is worth noting that by comparing the baseline stereo audio, stereo audio predicted by other prediction models, and the stereo audio obtained in this embodiment, a series of detection metrics were used, representing the left and right channels and the center channel respectively. These metrics included Mel-spectral distance (measuring the fidelity of the Mel spectrum), robustness to signal scale changes, scale-invariant signal-to-noise ratio (SNR) for evaluating waveform quality, Frechet audio distance (evaluating perceptual similarity of audio by comparing feature distributions in the pre-trained embedding model), and stereo imaging-related statistics. Furthermore, to further analyze the stereo imaging effect, the amplitude difference stereo width between channels could be calculated, and tools could be used to evaluate the audio production quality, such as sharpness, fidelity, dynamic range, frequency equalization, and spatiality. Among these, the amplitude difference stereo width between channels, production quality, Frechet audio distance between the left and right channels, scale-invariant SNR, and Mel-spectral distance showed better performance.

[0102] In addition, such as Figure 4 As shown, the stereo audio in this embodiment is more similar to the reference stereo audio in terms of waveform structure, indicating that it is closer to the real stereo audio and that the audio processing model has a better effect in spatial perception.

[0103] Based on the same concept, this embodiment provides a model training device, such as... Figure 5 As shown, the model training device 500 includes: Extraction module 501 is used to extract the first background sound audio from the stereo audio sample, the first background sound audio including the left channel background sound audio and the right channel background sound audio; The mixing module 502 is used to perform time-domain addition of the left channel background audio and the right channel background audio to obtain a channel mixed audio, and to determine the first phase information of the channel mixed audio and the second phase information of the first background audio. The determining module 503 is used to determine a mono input signal based on the first phase information and the first background audio, and to determine a stereo target signal based on the second phase information and the first background audio, wherein the mono input signal and the stereo target signal together form audio training data; The model training module 504 is used to train an initial preset model based on the audio training data, wherein the trained preset model serves as an audio processing model for predicting stereo audio based on mono audio.

[0104] Optionally, the determining module 503 is used to: Determine the first amplitude of the left channel background audio in the frequency domain and the second amplitude of the left channel background audio in the frequency domain, and add the first amplitude and the second amplitude to obtain the sum of the frequency domain amplitudes; The mono input signal is determined based on the first phase information and the sum of the frequency domain amplitudes.

[0105] Optionally, the determining module 503 is used to: Determine the first amplitude of the left channel background audio in the frequency domain and the second amplitude of the right channel background audio in the frequency domain, and determine the left channel amplitude component and the right channel amplitude component based on the first amplitude and the second amplitude. The stereo target signal is determined based on the second phase information, the left channel amplitude component, and the right channel amplitude component.

[0106] Optionally, the determining module 503 is used to: Subtracting the second amplitude from the first amplitude yields the first amplitude difference, and the larger of the first amplitude difference and 0 is taken as the left channel amplitude component. The second amplitude is subtracted from the first amplitude to obtain the second amplitude difference, and the larger of the second amplitude difference and 0 is taken as the right channel amplitude component.

[0107] Optionally, the second phase information includes the left channel phase information of the left channel background audio and the right channel phase information of the right channel background audio, and the determining module 503 is used for: The left channel phase information is multiplied by the left channel amplitude component to obtain a first complex spectrum. The first complex spectrum is then subjected to an inverse short-time Fourier transform to obtain the left channel sound effect target signal. The right channel phase information and the right channel amplitude component are multiplied to obtain the second complex spectrum. The second complex spectrum is then subjected to an inverse short-time Fourier transform to obtain the right channel sound effect target signal. The left channel sound effect target signal and the right channel sound effect target signal are determined as the stereo sound effect target signal.

[0108] Optionally, the mixing module 502 is used for: The first phase information is extracted from the audio signal information obtained after performing a short-time Fourier transform on the mixed audio channels. The left channel phase information is extracted from the audio signal information obtained by performing a short-time Fourier transform on the left channel background audio, and the right channel phase information is extracted from the audio signal information obtained by performing a short-time Fourier transform on the right channel background audio; the second phase information includes the left channel phase information and the right channel phase information.

[0109] Optionally, the model training module 504 is used for: Repeat the following steps until the preset training completion conditions are met to obtain the pre-trained preset model: The first noise vector is randomly determined from the Gaussian noise vector; Randomly sample the first noise vector from the Gaussian noise vector; Based on the first noise vector and the stereo sound target signal, the input vector at each time step is determined, and the reference velocity vector at each time step is determined based on the input vector at each time step. The first mono condition and the input vector at each time step are input together into the preset model to obtain the predicted velocity vector at each time step. The first mono condition is the Mel spectrum corresponding to the mono input signal. The model loss is determined based on the difference between the predicted velocity vector and the reference velocity vector at each time step, and the parameters of the preset model are updated based on the model loss, so that the preset model learns the optimal flow path from noise to stereo effect distribution under given mono conditions.

[0110] Optionally, the model training module 504 is used for: For each time step, the time step scalar is multiplied by the Mel spectrum corresponding to the stereo target signal to obtain a first product. The difference between the upper limit threshold of the preset time interval and the time step scalar is determined, and the difference is multiplied by the first noise vector to obtain a second product. The sum of the first product and the second product is determined as the input vector under the time step, and the derivative of the input vector is calculated to obtain the reference velocity vector under the time step.

[0111] Optionally, the audio processing model is a conditional flow matching model.

[0112] Based on the same concept, this embodiment provides a model-based audio processing device, such as... Figure 6 As shown, the audio processing device 600 includes: The splitting module 601 is used to split the mono audio to obtain human voice audio and second background sound audio. The model processing module 602 is used to obtain stereo audio based on the second background sound audio and the audio processing model. The stereo audio includes left channel audio and right channel audio. The audio processing model is trained based on the above-mentioned model training method. The audio processing module 603 is used to mix the left channel sound effect audio, the second background sound audio, and the human voice audio to obtain the left channel audio, mix the right channel sound effect audio, the second background sound audio, and the human voice audio to obtain the right channel audio, and obtain stereo audio based on the left channel audio and the right channel audio.

[0113] Optionally, the model processing module 602 is used for: Randomly sample a second noise vector from the Gaussian noise vector; The second noise vector and the second mono condition are input into the audio processing model so that the audio processing model outputs a second stereo effect prediction signal based on the second noise vector under the second mono condition, where the second mono condition is the Mel spectrum of the second background audio. The second stereo sound prediction signal is converted in the time domain by a vocoder to obtain the stereo sound audio.

[0114] Optionally, the splitting module 601 is used for: The human voice audio is obtained by extracting human voice from the mono audio using an audio separation model; The second background audio is obtained by subtracting the human voice audio from the mono audio.

[0115] Based on the same concept, embodiments of this disclosure also provide a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of any of the above-described model training methods or audio processing methods based on large models.

[0116] Based on the same concept, this disclosure also provides an electronic device that may include: A storage device on which computer programs are stored; A processing device for executing a computer program stored in a storage device to implement the steps of any of the above-described model training methods or audio processing methods based on large models.

[0117] Based on the same concept, embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described model training methods or audio processing methods based on large models.

[0118] The following is for reference. Figure 7 The diagram illustrates a structural schematic of an electronic device 700 suitable for implementing embodiments of the present disclosure. Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0119] like Figure 7 As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0120] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0121] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the methods of embodiments of this disclosure.

[0122] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0123] In some implementations, communication can be conducted using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can be interconnected with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0124] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0125] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: extract a first background sound audio from stereo audio samples, the first background sound audio including a left channel background sound audio and a right channel background sound audio; perform time-domain addition on the left channel background sound audio and the right channel background sound audio to obtain a channel mixed audio, and determine a first phase information of the channel mixed audio and a second phase information of the first background sound audio; determine a mono input signal based on the first phase information and the first background sound audio, and determine a stereo effect target signal based on the second phase information and the first background sound audio, wherein the mono input signal and the stereo effect target signal together form audio training data; and train an initial preset model based on the audio training data, wherein the trained preset model serves as an audio processing model for predicting stereo effect audio based on mono audio.

[0126] Alternatively, the aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: split the mono audio to obtain a human voice audio and a second background sound audio; obtain stereo audio based on the second background sound audio and an audio processing model, wherein the stereo audio includes a left channel audio and a right channel audio, and the audio processing model is trained based on the aforementioned model training method; mix the left channel audio, the second background sound audio, and the human voice audio to obtain a left channel audio, mix the right channel audio, the second background sound audio, and the human voice audio to obtain a right channel audio, and obtain stereo audio based on the left channel audio and the right channel audio.

[0127] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0128] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0129] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules are not, in some cases, intended to limit the functionality of the module itself.

[0130] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0131] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0132] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0133] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0134] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

Claims

1. A model training method, characterized in that, The model training method includes: Extract the first background audio from the stereo audio sample, the first background audio including the left channel background audio and the right channel background audio; The left channel background audio and the right channel background audio are added together in the time domain to obtain a channel mixed audio, and the first phase information of the channel mixed audio and the second phase information of the first background audio are determined. A mono input signal is determined based on the first phase information and the first background audio, and a stereo target signal is determined based on the second phase information and the first background audio, wherein the mono input signal and the stereo target signal together form audio training data; The initial preset model is trained based on the audio training data, and the trained preset model serves as an audio processing model for predicting stereo audio based on mono audio.

2. The model training method according to claim 1, characterized in that, The step of determining the mono input signal based on the first phase information and the first background audio includes: Determine the first amplitude of the left channel background audio in the frequency domain and the second amplitude of the right channel background audio in the frequency domain, and add the first amplitude and the second amplitude to obtain the total frequency domain amplitude; The mono input signal is determined based on the first phase information and the sum of the frequency domain amplitudes.

3. The model training method according to claim 1, characterized in that, The step of determining the stereo target signal based on the second phase information and the first background audio includes: Determine the first amplitude of the left channel background audio in the frequency domain and the second amplitude of the right channel background audio in the frequency domain, and determine the left channel amplitude component and the right channel amplitude component based on the first amplitude and the second amplitude. The stereo target signal is determined based on the second phase information, the left channel amplitude component, and the right channel amplitude component.

4. The model training method according to claim 3, characterized in that, The determination of the left channel amplitude component and the right channel amplitude component based on the first amplitude and the second amplitude includes: Subtracting the second amplitude from the first amplitude yields the first amplitude difference, and the larger of the first amplitude difference and 0 is taken as the left channel amplitude component. The second amplitude is subtracted from the first amplitude to obtain the second amplitude difference, and the larger of the second amplitude difference and 0 is taken as the right channel amplitude component.

5. The model training method according to claim 3, characterized in that, The second phase information includes the left channel phase information of the left channel background audio and the right channel phase information of the right channel background audio. Determining the stereo target signal based on the second phase information, the left channel amplitude component, and the right channel amplitude component includes: The left channel phase information is multiplied by the left channel amplitude component to obtain a first complex spectrum. The first complex spectrum is then subjected to an inverse short-time Fourier transform to obtain the left channel sound effect target signal. The right channel phase information and the right channel amplitude component are multiplied to obtain the second complex spectrum. The second complex spectrum is then subjected to an inverse short-time Fourier transform to obtain the right channel sound effect target signal. The left channel sound effect target signal and the right channel sound effect target signal are determined as the stereo sound effect target signal.

6. The model training method according to claim 1, characterized in that, Determining the first phase information of the mixed audio channels and the second phase information of the first background audio channel includes: The first phase information is extracted from the audio signal information obtained after performing a short-time Fourier transform on the mixed audio channels. The left channel phase information is extracted from the audio signal information obtained by performing a short-time Fourier transform on the left channel background audio, and the right channel phase information is extracted from the audio signal information obtained by performing a short-time Fourier transform on the right channel background audio; the second phase information includes the left channel phase information and the right channel phase information.

7. The model training method according to any one of claims 1-6, characterized in that, The step of training the initial preset model based on the audio training data includes: Repeat the following steps until the preset training completion conditions are met to obtain the pre-trained preset model: Randomly sample the first noise vector from the Gaussian noise vector; Based on the first noise vector and the stereo sound target signal, the input vector at each time step is determined, and the reference velocity vector at each time step is determined based on the input vector at each time step. The first mono condition and the input vector at each time step are input together into the preset model to obtain the predicted velocity vector at each time step. The first mono condition is the Mel spectrum corresponding to the mono input signal. The model loss is determined based on the difference between the predicted velocity vector and the reference velocity vector at each time step, and the parameters of the preset model are updated based on the model loss, so that the preset model learns the optimal flow path from noise to stereo effect distribution under given mono conditions.

8. The model training method according to claim 7, characterized in that, The step of determining the input vector at each time step based on the first noise vector and the stereo sound target signal, and determining the reference velocity vector at each time step based on the input vector at each time step, includes: For each time step, the time step scalar is multiplied by the Mel spectrum corresponding to the stereo target signal to obtain a first product. The difference between the upper limit threshold of the preset time interval and the time step scalar is determined, and the difference is multiplied by the first noise vector to obtain a second product. The sum of the first product and the second product is determined as the input vector under the time step, and the derivative of the input vector is calculated to obtain the reference velocity vector under the time step.

9. The model training method according to any one of claims 1-6, characterized in that, The audio processing model is a conditional stream matching model.

10. A model-based audio processing method, characterized in that, The audio processing method includes: The mono audio is split into human voice audio and second background audio. Based on the second background sound audio and the audio processing model, stereo sound audio is obtained, wherein the stereo sound audio includes left channel sound audio and right channel sound audio, and the audio processing model is trained based on the model training method described in any one of claims 1-9. The left channel audio, the second background audio, and the human voice audio are mixed to obtain the left channel audio. The right channel audio, the second background audio, and the human voice audio are mixed to obtain the right channel audio. Stereo audio is obtained based on the left channel audio and the right channel audio.

11. The model-based audio processing method according to claim 10, characterized in that, The stereo audio effect obtained based on the second background audio and audio processing model includes: Randomly sample a second noise vector from the Gaussian noise vector; The second noise vector and the second mono condition are input into the audio processing model so that the audio processing model outputs a second stereo effect prediction signal based on the second noise vector under the second mono condition, where the second mono condition is the Mel spectrum of the second background audio. The second stereo sound prediction signal is converted in the time domain by a vocoder to obtain the stereo sound audio.

12. The model-based audio processing method according to claim 10 or 11, characterized in that, The process of splitting the mono audio to obtain human voice audio and second background sound audio includes: The human voice audio is obtained by extracting human voice from the mono audio using an audio separation model; The second background audio is obtained by subtracting the human voice audio from the mono audio.

13. A model training device, characterized in that, The model training device includes: The extraction module is used to extract the first background audio from the stereo audio sample, the first background audio including the left channel background audio and the right channel background audio; The mixing module is used to perform time-domain addition of the left channel background audio and the right channel background audio to obtain the channel mixed audio, and to determine the first phase information of the channel mixed audio and the second phase information of the first background audio. The determining module is used to determine a mono input signal based on the first phase information and the first background audio, and to determine a stereo target signal based on the second phase information and the first background audio, wherein the mono input signal and the stereo target signal together form audio training data; The model training module is used to train an initial preset model based on the audio training data, wherein the trained preset model serves as an audio processing model for predicting stereo audio based on mono audio.

14. A model-based audio processing device, characterized in that, The audio processing device includes: The splitting module is used to split mono audio into human voice audio and second background audio. The model processing module is used to obtain stereo audio based on the second background sound audio and the audio processing model, wherein the stereo audio includes left channel audio and right channel audio, and the audio processing model is trained based on the model training method described in any one of claims 1-9. An audio processing module is used to mix the left channel sound effect audio, the second background sound audio, and the human voice audio to obtain left channel audio, mix the right channel sound effect audio, the second background sound audio, and the human voice audio to obtain right channel audio, and obtain stereo audio based on the left channel audio and the right channel audio.

15. A computer-readable medium having a computer program stored thereon, characterized in that, When executed by a processing device, the computer program performs the steps of the method described in any one of claims 1-12.

16. An electronic device, characterized in that, include: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method according to any one of claims 1-12.

17. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-12.

Images