Audio separation model training method, audio separation method, electric equipment and medium

Abstract

The invention discloses an audio separation model training method, an audio separation method, electric equipment and a computer readable storage medium. The method comprises the following steps: determining an estimated output audio signal according to an obtained mixed audio signal sample based on an audio separation model; performing energy estimation processing on the estimated output audio signal and the background audio signal sample, and determining a target energy weighted loss function; performing frequency division processing on the estimated output audio signal and the background audio signal sample, and determining a time domain frequency division loss function; determining a fusion loss function of the audio separation model according to the time domain frequency division loss function and the target energy weighted loss function; and training an audio separation model according to the fusion loss function. Therefore, the audio separation model is trained based on the loss function fused by the target energy weighted loss function and the time domain frequency division loss function, the convergence speed can be increased, and the accuracy of the audio separation model for separating the mixed audio is improved.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to an audio separation model training method, an audio separation method, an electrical device, and a computer-readable storage medium. Background Technology

[0002] Background separation methods based on neural networks can extract background music components from mixed audio signals and remove human voices. However, in related technologies, neural network-based background separation methods suffer from unstable training and slow convergence in lightweight models, making them difficult to adapt to environments with limited computing power. Summary of the Invention This application provides an audio separation model training method, an audio separation method, an electrical device, and a computer-readable storage medium.

[0003] This application provides an audio separation model training method, the method comprising: Based on the audio separation model, an estimated output audio signal is determined according to the acquired mixed audio signal samples, wherein the mixed audio signal samples include background audio signal samples and target audio signal samples corresponding to the background audio signal samples; The estimated output audio signal and the background audio signal samples are subjected to energy estimation processing to determine the target energy weighted loss function; The estimated output audio signal and the background audio signal samples are subjected to frequency division processing to determine the time-domain frequency division loss function; The fusion loss function of the audio separation model is determined based on the time-domain frequency division loss function and the target energy weighted loss function. The audio separation model is trained based on the fusion loss function.

[0004] Thus, the implementation method of this application can obtain a fusion loss function by fusing the target energy weighted loss function and the time-domain frequency division loss function, and train the audio separation model based on the fusion loss function, thereby accelerating the convergence speed. Ultimately, the model gradually learns more accurate separation rules, resulting in a trained and stable audio separation model. Compared to training the audio separation model based solely on the frequency loss function, the implementation method of this application can fuse the target energy weighted loss function and the time-domain frequency division loss function. Through a frequency band weighting strategy, it guides the audio separation model to accurately optimize the target frequency band, accelerating the convergence speed. Furthermore, the weighting strategy based on the target audio energy guides the audio separation model to prioritize optimizing regions with high target audio energy, thereby improving the accuracy of the audio separation model in separating mixed audio to a certain extent.

[0005] In some embodiments, the background audio signal sample includes multiple frames of background audio frame signal samples, and the target audio signal sample includes multiple frames of target audio frame signal samples, wherein each frame of the background audio frame signal sample corresponds to one frame of the target audio frame signal sample, and the method further includes: Each background audio frame signal sample and the corresponding target audio frame signal sample are mixed with a preset energy ratio to determine multiple mixed audio frame signal samples, thereby determining the mixed audio signal sample.

[0006] In this way, each background audio frame signal sample and its corresponding target audio frame signal sample are mixed at a preset energy ratio to determine multiple mixed audio frame signal samples, thus defining the mixed audio signal sample. This allows the mixed audio signal sample to retain the true mixing characteristics of the target audio in the target audio signal sample and the background audio in the background audio frame signal sample, ensuring that the separation rules learned by the audio separation model are consistent with the actual mixed audio characteristics, and improving the generalization ability of the audio separation model, enabling it to adapt to audio separation requirements with various energy distributions.

[0007] In some implementations, the estimated output audio signal includes multiple frames of estimated background audio frame signals, and the step of performing energy estimation processing on the estimated output audio signal and the background audio signal samples to determine the target energy-weighted loss function includes: Energy estimation processing is performed on the target audio frame signal sample and the background audio frame signal sample of each frame to determine the target energy weight of each frame. The error loss for each frame is determined based on the estimated background audio frame signal for each frame and the background audio frame signal sample for each frame. The target energy weighted loss function is determined based on the error loss of each frame and the target energy weight of each frame.

[0008] Thus, energy estimation is performed on each target audio frame signal sample and each background audio frame signal sample to determine the target energy weight for each frame. Based on the estimated background audio frame signal and each background audio frame signal sample, the error loss for each frame is determined. Based on the error loss and the target energy weight for each frame, the target energy weighted loss function is determined. In this way, the target energy weight for each frame can be determined based on the proportion of the target audio's energy in the mixed audio energy. Combined with the error loss of each estimated background audio frame signal and each background audio frame signal sample, the weights of the target energy weighted loss function can be dynamically adjusted at the frame level. This makes the weight allocation more closely match the actual audio characteristics, guiding the audio separation model to prioritize optimizing regions with high target audio energy, thereby improving the accuracy of the audio separation model in separating mixed audio to a certain extent.

[0009] In some implementations, the step of performing frequency division processing on the estimated output audio signal and the background audio signal samples to determine the time-domain frequency division loss function includes: Based on preset frequency division parameters, the estimated output audio signal and the background audio signal sample are respectively subjected to frequency division processing to determine the estimated background sub-band signal and the background sub-band signal sample. The estimated background sub-band signal includes multiple frames of estimated background sub-band frame signal, and the background sub-band signal sample includes multiple frames of background sub-band frame signal sample. The time-domain frequency division loss function is determined based on the estimated background sub-band frame signal for each frame, the background sub-band frame signal sample for each frame, and the preset sub-band weight.

[0010] Thus, based on preset frequency division parameters, frequency division processing is performed on the estimated output audio signal and background audio signal samples respectively to determine the estimated background sub-band signal and background sub-band signal samples. The estimated background sub-band signal includes multiple frames of estimated background sub-band signals, and the background sub-band signal samples include multiple frames of background sub-band signal samples. A time-domain frequency division loss function is determined based on each frame of estimated background sub-band signal, each frame of background sub-band signal samples, and preset sub-band weights. In this way, based on the preset frequency division parameters, the estimated output audio signal can be decomposed into multiple estimated background sub-band signals, and the background audio signal samples can be decomposed into multiple background sub-band signal samples. This provides a precise comparative basis for subsequent sub-band level error calculation. Furthermore, based on the error loss of each set of estimated background sub-band frames and each frame of background sub-band signal samples under each sub-band, combined with the preset sub-band weights, the time-domain frequency division loss function can be obtained. This strengthens the model's optimization guidance in the target audio frequency band and improves the accuracy of the audio separation model in separating mixed audio to a certain extent.

[0011] In some implementations, determining the fusion loss function of the audio separation model based on the time-domain frequency division loss function and the target energy-weighted loss function includes: The fusion loss function is determined based on the time-domain frequency division loss function, the preset time-domain frequency division loss weight, the target energy weighted loss function, and the preset target energy weighted loss weight.

[0012] Thus, based on the time-domain frequency division loss function, the preset time-domain frequency division loss weights, the target energy-weighted loss function, and the preset target energy-weighted loss weights, a fusion loss function is determined. This integrates the advantages of the time-domain frequency division loss function and the target energy-weighted loss function, allowing the fusion loss function to simultaneously guide model optimization training stability and target frequency band separation accuracy, thereby improving the accuracy of the audio separation model in separating mixed audio to a certain extent.

[0013] In some embodiments, the method further includes: The fusion loss function is determined based on the target audio signal sample, the background audio signal sample, and the estimated output audio signal.

[0014] Thus, based on the target audio signal samples, background audio signal samples, and estimated output audio signal, a fusion loss function is determined. Furthermore, by considering the distribution of target audio energy in the current sub-band, the energy distribution within each frequency band is extracted based on the target audio signal samples, background audio signal samples, and estimated output audio signal. A target energy weight for the corresponding frequency band is generated based on this energy distribution, and within that frequency band, the target energy weight is used to weight the temporal error. Ultimately, a more refined fusion loss function is obtained without increasing the number of model parameters. This allows the fusion loss function to accurately pinpoint the error region and more precisely guide the audio separation model to eliminate interference from the target audio.

[0015] In some implementations, determining the fusion loss function based on the target audio signal sample, the background audio signal sample, and the estimated output audio signal includes: Based on preset frequency division parameters, the target audio signal sample, the background audio signal sample, and the estimated output audio signal are subjected to frequency division processing respectively to determine the target sub-band signal sample, the background sub-band signal sample, and the estimated background sub-band signal. The target sub-band signal sample includes multiple frames of target sub-band frame signal samples, the background sub-band signal sample includes multiple frames of the background sub-band frame signal samples, and the estimated background sub-band signal includes multiple frames of the estimated background sub-band frame signal. Energy estimation processing is performed on the target sub-band frame signal sample and the background sub-band frame signal sample of each frame to determine the target sub-band energy weight of each frame. The time-domain frequency division loss function is determined based on the estimated background subband frame signal for each frame, the background subband frame signal sample for each frame, the target subband energy weight, and the preset subband weight.

[0016] Thus, based on preset frequency division parameters, frequency division processing is performed on the target audio signal sample, background audio signal sample, and estimated output audio signal respectively to determine the target sub-band signal sample, background sub-band signal sample, and estimated background sub-band signal. The target sub-band signal sample includes multiple frames of target sub-band frame signal samples, the background sub-band signal sample includes multiple frames of background sub-band frame signal samples, and the estimated background sub-band signal includes multiple frames of estimated background sub-band frame signals. Energy estimation processing is performed on each frame of target sub-band frame signal sample and each frame of background sub-band frame signal sample to determine the target sub-band energy weight for each frame. Based on each frame of estimated background sub-band frame signal, each frame of background sub-band frame signal sample, the target sub-band energy weight, and the preset sub-band weight, the time-domain frequency division loss function is determined. In this way, target sub-band energy weights can be introduced into each frequency band. Then, within each frequency band, the error between each frame of estimated background sub-band frame signal and each frame of background sub-band frame signal sample is subjected to dual weighting of target sub-band energy weights and preset sub-band weights. This simultaneously handles the importance of frequency bands and energy distribution, obtaining a more refined fusion loss function. This allows the fusion loss function to accurately lock the error region, thereby more accurately guiding the audio separation model to eliminate interference from the target audio.

[0017] This application provides an audio separation method, characterized in that the method is based on an audio separation model, the audio separation model is trained using an audio separation model training method, and the method includes: Acquire a target mixed audio signal, wherein the target mixed audio signal includes a target audio signal and a background audio signal; Based on the audio separation model, audio separation processing is performed on the target mixed audio signal to separate the background audio signal from the target mixed audio signal.

[0018] Thus, a target mixed audio signal is acquired, which includes a target audio signal and a background audio signal. Based on an audio separation model, audio separation processing is performed on the target mixed audio signal to separate the background audio signal from the target mixed audio signal. In this way, by training the audio separation model using the audio separation model training method, the target mixed audio signal can be input into the audio separation model, enabling the audio separation model to perform audio separation processing on the target mixed audio signal to separate the background audio signal from the target mixed audio signal, achieving accurate audio signal separation.

[0019] This application provides an electrical device including a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the steps of the above-described method.

[0020] This application provides a computer-readable storage medium storing a computer program that, when executed by one or more processors, implements the steps of the above-described method.

[0021] The electrical device and computer-readable storage medium provided in this application, based on an audio separation model, determine an estimated output audio signal from acquired mixed audio signal samples. The mixed audio signal samples include background audio signal samples and target audio signal samples corresponding to the background audio signal samples. Energy estimation processing is performed on the estimated output audio signal and background audio signal samples to determine a target energy-weighted loss function. Frequency division processing is performed on the estimated output audio signal and background audio signal samples to determine a time-domain frequency division loss function. A fusion loss function for the audio separation model is determined based on the time-domain frequency division loss function and the target energy-weighted loss function. The audio separation model is then trained based on the fusion loss function. Thus, this application embodiment can obtain a fusion loss function by fusing the target energy-weighted loss function and the time-domain frequency division loss function, and train the audio separation model based on the fusion loss function, accelerating convergence and ultimately enabling the model to gradually learn more accurate separation rules, resulting in a trained and stable audio separation model. Compared to training an audio separation model solely based on a frequency loss function, the implementation method of this application can integrate a target energy weighted loss function and a time-domain frequency division loss function. Through a frequency band weighting strategy, the audio separation model is guided to accurately optimize the target frequency band, accelerating the convergence speed. Furthermore, a weighting strategy based on the target audio energy guides the audio separation model to prioritize optimizing regions with high target audio energy, thereby improving the accuracy of the audio separation model in separating mixed audio to a certain extent.

[0022] Additional aspects and advantages of embodiments of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of embodiments of this application. Attached Figure Description

[0023] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, wherein: Figure 1 This is one of the flowcharts illustrating the audio separation model training method in certain embodiments of this application; Figure 2 This is a second schematic flowchart of the audio separation model training method according to certain embodiments of this application; Figure 3 This is the third flowchart illustrating the audio separation model training method of some embodiments of this application; Figure 4 This is the process for obtaining the target energy-weighted loss function in certain embodiments of this application; Figure 5 This is the fourth flowchart illustrating the audio separation model training method of some embodiments of this application; Figure 6 This is a process for obtaining the time-domain frequency division loss function in certain embodiments of this application; Figure 7 This is the fifth flowchart illustrating the audio separation model training method of some embodiments of this application; Figure 8 This is one of the processes for obtaining the fusion loss function in certain embodiments of this application; Figure 9 This is the sixth flowchart illustrating the audio separation model training method of some embodiments of this application; Figure 10 This is the seventh flowchart illustrating the audio separation model training method of some embodiments of this application; Figure 11 This is the second process for obtaining the fusion loss function in some embodiments of this application; Figure 12 This is the audio separation model training process in some embodiments of this application; Figure 13 This is a flowchart illustrating an audio separation method according to certain embodiments of this application. Detailed Implementation

[0024] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of this application, and should not be construed as limiting the embodiments of this application.

[0025] The neural network-based background separation method can accurately extract the background music component from a mixed audio signal containing human voices and background music, while effectively removing the human voice part. It is widely used in various scenarios such as karaoke systems, smart in-vehicle entertainment, and mobile audio editing.

[0026] However, in actual deployment, the relevant technologies face the challenge of scenario adaptation. When the model needs to be deployed in environments with limited computing power, such as in-vehicle systems and mobile devices, a lightweight model architecture with fewer parameters is required. However, existing neural network-based background audio separation methods often perform poorly in lightweight models because lightweight models have limited fitting capabilities, while the frequency domain information has high dimensionality and complex structure, which can easily lead to overfitting or underfitting. This results in instability and slow convergence during training, thus affecting the accuracy of the separation results.

[0027] Based on the above issues, please refer to Figure 1 This application provides an audio separation model training method, the method comprising: 01: Based on the audio separation model, the estimated output audio signal is determined according to the acquired mixed audio signal samples. The mixed audio signal samples include background audio signal samples and target audio signal samples corresponding to the background audio signal samples. 02: Perform energy estimation processing on the estimated output audio signal and background audio signal samples to determine the target energy weighted loss function; 03: Perform frequency division processing on the estimated output audio signal and background audio signal samples to determine the time-domain frequency division loss function; 04: Determine the fusion loss function of the audio separation model based on the time-domain frequency division loss function and the target energy weighted loss function; 05: Train the audio separation model based on the fusion loss function.

[0028] This application provides an audio separation model training apparatus. The audio separation model training method of this application can be implemented by the audio separation model training apparatus of this application. Specifically, the audio separation model training apparatus includes a determination module and a training module. The determination module is used to determine an estimated output audio signal based on the audio separation model and acquired mixed audio signal samples, wherein the mixed audio signal samples include background audio signal samples and target audio signal samples corresponding to the background audio signal samples. The determination module is also used to perform energy estimation processing on the estimated output audio signal and background audio signal samples to determine a target energy weighted loss function. The determination module is also used to perform frequency division processing on the estimated output audio signal and background audio signal samples to determine a time-domain frequency division loss function. The determination module is also used to determine a fusion loss function of the audio separation model based on the time-domain frequency division loss function and the target energy weighted loss function. The training module is used to train the audio separation model based on the fusion loss function.

[0029] This application also provides an electrical device, which includes a memory and a processor. The audio separation model training method of this application can be implemented by the electrical device of this application. Specifically, the memory stores a computer program, and the processor is used to determine an estimated output audio signal based on the audio separation model and acquired mixed audio signal samples, wherein the mixed audio signal samples include background audio signal samples and target audio signal samples corresponding to the background audio signal samples. The processor is also used to perform energy estimation processing on the estimated output audio signal and background audio signal samples to determine a target energy weighted loss function. The processor is also used to perform frequency division processing on the estimated output audio signal and background audio signal samples to determine a time-domain frequency division loss function. The processor is also used to determine a fusion loss function of the audio separation model based on the time-domain frequency division loss function and the target energy weighted loss function. The processor is used to train the audio separation model based on the fusion loss function.

[0030] Specifically, the audio separation model is a model based on neural networks that separates background audio and target audio from mixed audio signals. For example, it can separate the background music that does not contain human voices from a mixed audio that contains human voices and background music.

[0031] Mixed audio signal samples are training data for the audio separation model, and can be generated by mixing background audio signal samples and target audio signal samples in a certain proportion.

[0032] Among them, background audio signal samples refer to background audio such as pure accompaniment and ambient sound; target audio signal samples refer to data that have differences in energy distribution and frequency distribution compared with background audio data. Target audio refers to human voice data such as pure singing voice and speaking voice. Background audio signal samples and target audio signal samples can be used as the target output benchmark of the audio separation model, and are used to compare with the estimated output audio signal of the model to calculate the loss function.

[0033] The estimated output audio signal is the predictive target output of the audio separation model after processing the mixed input audio signal samples. The estimated output audio signal closely approximates the true signal sample and is used to calculate the loss function by comparing it with the model's target output benchmark. For example, when the audio separation model is used to separate pure background audio, the background audio signal sample serves as the model's target output benchmark; when the audio separation model is used to separate pure human voices or the sound of a specific instrument, the target audio signal sample serves as the model's target output benchmark. The following illustrations use an audio separation model that removes human voices and outputs background audio as an example.

[0034] The target energy-weighted loss function is an error loss function calculated based on the energy distribution corresponding to the target audio. It is used to ensure that the audio separation model focuses on regions with high target audio energy during training. For example, based on the non-uniformity of human voice energy distribution and the perceptual characteristics of the human auditory system, an error loss function can be calculated based on the energy distribution corresponding to human voice. This allows the audio separation model to focus on regions with high human voice energy during training, reducing human voice residue in the output background audio and improving listening quality.

[0035] The time-domain frequency division loss function is the sum of error loss functions calculated separately for each frequency band audio signal based on the importance of each frequency band. It is used to ensure the stability of the audio separation model trained based on time-domain loss and the attention to the frequency band corresponding to the target audio. For example, it guides the audio separation model to accurately optimize the low-mid frequency region corresponding to human voice.

[0036] The fusion loss function refers to the target loss function formed by combining the target energy weighted loss function and the time-domain frequency division loss function according to a preset logic. It combines the advantages of both the target energy weighted loss function and the time-domain frequency division loss function, achieving a balance between training stability, separation accuracy, and listening quality.

[0037] By inputting mixed audio signal samples into the audio separation model, the model can output an estimated output audio signal through a neural network structure, achieving preliminary separation prediction. Based on the estimated output audio signal, background audio signal samples, and target audio signal samples, the target energy weighted loss function and time-domain frequency division loss function of the audio separation model are calculated respectively. Then, based on the target energy weighted loss function, the audio separation model is guided to prioritize optimizing regions with high target audio energy. For example, it can be guided to prioritize optimizing regions with high human voice energy. Based on the time-domain frequency division loss function, the audio separation model can achieve accurate optimization of the target frequency band. For example, it can prioritize optimizing the low and mid frequencies concentrated in human voices.

[0038] By fusing the target energy-weighted loss function and the time-domain frequency division loss function, a fusion loss function is obtained by calculating the comprehensive loss. The audio separation model can be trained based on the fusion loss function. With the fusion loss function as the optimization target, the parameters of the audio separation model are adjusted through the backpropagation algorithm to continuously reduce the fusion loss value. Ultimately, the model gradually learns more accurate separation rules, resulting in a trained and stable audio separation model. Thus, with only mixed audio signals as input, the predicted output audio signal can be directly obtained in resource-constrained environments such as in-vehicle systems and mobile devices, thereby improving the accuracy of the predicted output audio signal and enhancing the user's subjective listening experience.

[0039] Compared to training an audio separation model solely based on a frequency loss function, the implementation method of this application can integrate a target energy weighted loss function and a time-domain frequency division loss function. Through a frequency band weighting strategy, the audio separation model is guided to accurately optimize the target frequency band, accelerating the convergence speed. Furthermore, a weighting strategy based on the target audio energy guides the audio separation model to prioritize optimizing regions with high target audio energy, thereby improving the accuracy of the audio separation model in separating mixed audio to a certain extent.

[0040] It should be noted that the mixed audio signal sample, background audio signal sample, target audio signal sample, and estimated output audio signal in the embodiments of this application are all time-domain signals. When the audio separation model is a frequency-domain model, the input frequency-domain mixed audio signal sample and the output frequency-domain estimated output audio signal need to be converted into corresponding time-domain signals before calculating the mixing loss function, such as by performing an inverse short-time Fourier transform.

[0041] In summary, based on the audio separation model, the estimated output audio signal is determined according to the acquired mixed audio signal samples. The mixed audio signal samples include background audio signal samples and target audio signal samples corresponding to the background audio signal samples. Energy estimation processing is performed on the estimated output audio signal and background audio signal samples to determine the target energy weighted loss function. Frequency division processing is also performed on the estimated output audio signal and background audio signal samples to determine the time-domain frequency division loss function. Based on the time-domain frequency division loss function and the target energy weighted loss function, the fusion loss function of the audio separation model is determined. The audio separation model is then trained based on the fusion loss function. This embodiment of the application obtains the fusion loss function by fusing the target energy weighted loss function and the time-domain frequency division loss function, and trains the audio separation model based on the fusion loss function, accelerating the convergence speed and ultimately enabling the model to gradually learn more accurate separation rules, resulting in a trained and stable audio separation model. Compared to training an audio separation model solely based on a frequency loss function, the implementation method of this application can integrate a target energy weighted loss function and a time-domain frequency division loss function. Through a frequency band weighting strategy, the audio separation model is guided to accurately optimize the target frequency band, accelerating the convergence speed. Furthermore, a weighting strategy based on the target audio energy guides the audio separation model to prioritize optimizing regions with high target audio energy, thereby improving the accuracy of the audio separation model in separating mixed audio to a certain extent.

[0042] Please see Figure 2 In some embodiments, the background audio signal sample includes multiple frames of background audio frame signal samples, and the target audio signal sample includes multiple frames of target audio frame signal samples, with each frame of background audio frame signal sample corresponding to one frame of target audio frame signal sample. The method further includes: 06: Mix each background audio frame signal sample and the corresponding target audio frame signal sample with a preset energy ratio to determine multiple mixed audio frame signal samples, and then determine the mixed audio signal sample.

[0043] In some implementations, the determining module is further configured to mix each background audio frame signal sample and the corresponding target audio frame signal sample with a preset energy ratio to determine multiple mixed audio frame signal samples, thereby determining the mixed audio signal sample.

[0044] In some implementations, the processor is further configured to mix each background audio frame signal sample and the corresponding target audio frame signal sample with a preset energy ratio to determine multiple mixed audio frame signal samples, thereby determining the mixed audio signal sample.

[0045] Specifically, the background audio signal sample includes multiple background audio frame signal samples, wherein the background audio frame signal sample is the smallest unit of data obtained after dividing the background audio signal sample into short audio frames.

[0046] The target audio signal sample includes multiple target audio frame signal samples. The target audio frame signal sample is the smallest unit of data obtained by dividing the target audio signal sample into short audio frames. Each background audio frame signal sample corresponds to one target audio frame signal sample, which is used to ensure that the background audio signal sample and the target audio signal sample match one-to-one in the time dimension.

[0047] The preset energy ratio refers to the pre-set energy ratio between the target audio frame signal sample and the background audio frame signal sample. It can be determined according to the target audio energy distribution characteristics of the actual application scenario. For example, if the energy ratio of the target audio frame signal sample is high, the mixed audio frame signal sample can cover the situation where the energy ratio of human voice is high in the a cappella scenario, thereby improving the generalization ability of the audio separation model.

[0048] The mixed audio frame signal sample is a short-time audio frame data formed by superimposing the corresponding target audio frame signal sample and the background audio frame signal sample according to a preset energy ratio, and is used to construct the mixed audio signal sample.

[0049] Based on each pair of time-synchronized background audio frame signal samples and target audio frame signal samples, each background audio frame signal sample and its corresponding target audio frame signal sample can be mixed with a preset energy ratio. This process splices all the mixed single-frame audio frame signal samples in their original time order to form a complete mixed audio signal sample. This preserves the true mixing characteristics of the target audio in the target audio signal sample and the background audio in the background audio frame signal sample, such as mixed audio containing human voices and background music. This ensures that the separation rules learned by the audio separation model are consistent with the actual mixed audio characteristics and improves the generalization ability of the audio separation model, enabling it to adapt to audio separation requirements with various energy distributions, such as a cappella and strong accompaniment.

[0050] In one example, clean vocal and accompaniment signals can be obtained from publicly available audio datasets or manually collected samples, and uniformly resampled to 44.1kHz. At the same time, standardization processing is performed to eliminate energy differences between different audios. Then, by combining across songs, the processed vocal and accompaniment signals from different song segments can be mixed. Paired background music signals and vocal signals are used as label data. That is, in this embodiment, background audio signal samples and target audio signal samples are paired, labeled, and mixed to generate mixed audio signal samples, so as to introduce diversity during training and improve the generalization ability of the model.

[0051] Furthermore, in the embodiments of this application, the mixed audio signal sample, background audio signal sample, target audio signal sample, and estimated output audio signal are all time-domain signals. When the audio separation model is a frequency-domain model, the mixed audio signal sample needs to be converted into a frequency-domain signal by a short-time Fourier transform before being input into the model. In addition, the output frequency-domain estimated output audio signal needs to be converted into a corresponding time-domain signal by an inverse short-time Fourier transform before the mixing loss function is calculated.

[0052] In this way, each background audio frame signal sample and its corresponding target audio frame signal sample are mixed at a preset energy ratio to determine multiple mixed audio frame signal samples, thus defining the mixed audio signal sample. This allows the mixed audio signal sample to retain the true mixing characteristics of the target audio in the target audio signal sample and the background audio in the background audio frame signal sample, ensuring that the separation rules learned by the audio separation model are consistent with the actual mixed audio characteristics, and improving the generalization ability of the audio separation model, enabling it to adapt to audio separation requirements with various energy distributions.

[0053] Please see Figure 3In some implementations, the estimated output audio signal includes multiple frames of estimated background audio frame signals. Step 02 (performing energy estimation processing on the estimated output audio signal and background audio signal samples to determine the target energy weighted loss function) includes: 021: Perform energy estimation processing on each target audio frame signal sample and each background audio frame signal sample to determine the target energy weight for each frame; 022: Determine the error loss for each frame based on the estimated background audio frame signal for each frame and the background audio frame signal sample for each frame; 023: Determine the target energy weighted loss function based on the error loss of each frame and the target energy weight of each frame.

[0054] In some implementations, the determining module is further configured to perform energy estimation processing on each target audio frame signal sample and each background audio frame signal sample to determine the target energy weight for each frame. The determining module is further configured to determine the error loss for each frame based on the estimated background audio frame signal and each background audio frame signal sample. The determining module is further configured to determine the target energy weighted loss function based on the error loss for each frame and the target energy weight for each frame.

[0055] In some implementations, the processor is further configured to perform energy estimation processing on each target audio frame signal sample and each background audio frame signal sample to determine the target energy weight for each frame. The processor is also configured to determine the error loss for each frame based on the estimated background audio frame signal and each background audio frame signal sample. The processor is further configured to determine a target energy weighted loss function based on the error loss for each frame and the target energy weight for each frame.

[0056] Specifically, the estimated output audio signal includes multiple estimated background audio frame signals, which are the smallest unit of data obtained by dividing the estimated output audio signal into short-time audio frames.

[0057] Energy estimation processing can quantify the energy of the target audio or background audio in the estimated background audio frame signal by calculating the square of the amplitude or the effective value of the estimated background audio frame signal in each frame, so as to determine the target energy weight of each frame based on the energy.

[0058] The target energy weight refers to the proportional coefficient calculated based on the energy of the target sub-band signal sample and the corresponding background sub-band signal sample. It is used to characterize the proportion of target audio energy to total energy within a frame. For example, in an a cappella scene, the energy of human voice is relatively high. To obtain background audio with less residual human voice, it is necessary to focus on the region with a high proportion of human voice energy during model training, that is, to increase the target energy weight of the corresponding frame.

[0059] Error loss is used to quantify the difference between a single frame estimated background audio frame signal and the corresponding background audio frame signal sample, such as mean squared error (MSE), mean absolute error (L1 loss), etc.

[0060] By performing energy estimation processing on each target audio frame signal sample and each background audio frame signal sample, the energy corresponding to the target audio and the energy corresponding to the background audio can be obtained. The sum of the energy corresponding to the target audio and the energy corresponding to the background audio is determined as the energy of the mixed audio. Then, based on the proportion of the energy corresponding to the target audio in the energy of the mixed audio, the target energy weight of each frame is determined, realizing frame-level dynamic adjustment of the weight, making the weight allocation more in line with the actual audio characteristics, and providing data support for guiding the audio separation model to prioritize the optimization of regions with high target audio energy.

[0061] By calculating the error loss of each frame's estimated background audio frame signal and each frame's background audio frame signal sample, and combining the target energy weight of each frame, the weighted error loss of all frames can be accumulated to determine the target energy weighted loss function. This guides the audio separation model to prioritize optimizing regions with high target audio energy, thereby improving the accuracy of the audio separation model in separating mixed audio to a certain extent.

[0062] In one example, such as Figure 4 In the process of obtaining the target energy-weighted loss function shown, it is based on the paired vocal and accompaniment tag data. and This involves analyzing the target audio signal sample and the corresponding background audio signal sample. Energy estimation is performed on each target audio frame signal sample and each background audio frame signal sample to extract the energy spectrum of the human voice portion and obtain the total energy of the current frame. This allows for the extraction of the energy of the human voice audio, i.e., the target audio, within that frame. That is, the energy value of the human voice in each frame, and the energy of the background audio in that frame. If a frame is a cappella, the vocal energy value is high, and the corresponding target energy weight is close to 1; if a frame has a high background audio energy value, the target energy weight is lower. The weight coefficient for the corresponding audio frame is determined by normalizing the total energy of the current frame and based on the vocal energy value. :

[0063] Next, the background audio frame signal is estimated by calculating each frame. and each frame of background audio frame signal sample The mean squared error loss, combined with the weighting coefficient The weighted loss of each frame can be accumulated to calculate the average weighted loss and determine the final target energy weighted loss function. :

[0064] Compared to audio separation models that are prone to vocal remnants in a cappella scenarios, the implementation method of this application allows the model to pay more attention to areas with a high proportion of vocal energy during training, thereby reducing vocal remnants and improving the listening quality of the separated accompaniment.

[0065] Furthermore, when the audio separation model needs to output the signal corresponding to the target audio, instead of acquiring the energy proportion of the human voice audio (i.e., the target audio) in that frame, the target energy weight of the frame containing the target audio signal can be reduced. This guides the audio separation model to prioritize optimizing regions with low target audio energy, thereby improving the accuracy of the audio separation model in separating mixed audio to some extent.

[0066] Thus, energy estimation is performed on each target audio frame signal sample and each background audio frame signal sample to determine the target energy weight for each frame. Based on the estimated background audio frame signal and each background audio frame signal sample, the error loss for each frame is determined. Based on the error loss and the target energy weight for each frame, the target energy weighted loss function is determined. In this way, the target energy weight for each frame can be determined based on the proportion of the target audio's energy in the mixed audio energy. Combined with the error loss of each estimated background audio frame signal and each background audio frame signal sample, the weights of the target energy weighted loss function can be dynamically adjusted at the frame level. This makes the weight allocation more closely match the actual audio characteristics, guiding the audio separation model to prioritize optimizing regions with high target audio energy, thereby improving the accuracy of the audio separation model in separating mixed audio to a certain extent.

[0067] Please see Figure 5 In some implementations, step 03 (performing frequency division processing on the estimated output audio signal and background audio signal samples to determine the time-domain frequency division loss function) includes: 031: Based on the preset frequency division parameters, the estimated output audio signal and the background audio signal samples are divided by frequency to determine the estimated background sub-band signal and the background sub-band signal sample. The estimated background sub-band signal includes multiple frames of estimated background sub-band frame signal, and the background sub-band signal sample includes multiple frames of background sub-band frame signal sample. 032: Determine the time-domain frequency division loss function based on the estimated background sub-band frame signal for each frame, the background sub-band frame signal sample for each frame, and the preset sub-band weights.

[0068] In some embodiments, the determining module is further configured to perform frequency division processing on the estimated output audio signal and the background audio signal samples based on preset frequency division parameters, respectively, to determine the estimated background sub-band signal and background sub-band signal samples, wherein the estimated background sub-band signal includes multiple frames of estimated background sub-band frame signals, and the background sub-band signal samples include multiple frames of background sub-band frame signal samples. The determining module is further configured to determine a time-domain frequency division loss function based on each frame of estimated background sub-band frame signal, each frame of background sub-band frame signal samples, and preset sub-band weights.

[0069] In some embodiments, the processor is further configured to perform frequency division processing on the estimated output audio signal and the background audio signal samples based on preset frequency division parameters, respectively, to determine the estimated background sub-band signal and background sub-band signal samples, wherein the estimated background sub-band signal includes multiple frames of estimated background sub-band frame signals, and the background sub-band signal samples include multiple frames of background sub-band frame signal samples. The processor is further configured to determine a time-domain frequency division loss function based on each frame of estimated background sub-band frame signal, each frame of background sub-band frame signal samples, and preset sub-band weights.

[0070] Specifically, the preset crossover parameters are pre-set parameters used to divide audio frequency bands. For example, 800Hz can be divided into low frequency and mid frequency, and 4000Hz can be divided into mid frequency and high frequency. That is, low frequency is 0-800Hz, mid frequency is 800-4000Hz, and high frequency is above 4000Hz.

[0071] The estimated background subband signal is a series of continuous audio signals obtained by dividing the estimated output audio signal into multiple frequency bands based on preset frequency division parameters. Each estimated background subband signal corresponds to a specific frequency range.

[0072] Background subband signal samples are multiple frequency bands of continuous audio signals obtained by dividing the background audio signal samples based on preset frequency division parameters. Each background subband signal sample also corresponds to a specific frequency range and is one-to-one with the estimated background subband signal.

[0073] The estimated background subband signal includes multiple frames of estimated background subband frame signals, and the background subband signal samples include multiple frames of background subband frame signal samples, which can be used to calculate the subband level error.

[0074] Preset subband weights are weighting coefficients pre-set based on the importance of frequency bands divided by preset frequency division parameters. They are used to adjust the degree of influence of different subband error losses on model training. For example, in an audio separation model for accompaniment separation, the low and mid frequency bands where vocals are concentrated can be given higher weights, while the high frequency bands can be given lower weights.

[0075] Based on the audio separation requirements in real-world scenarios, such as adapting to the differences in spectral characteristics between vocals and accompaniment, preset frequency division parameters can be pre-set. Based on these preset parameters, frequency filtering methods such as filter banks can be used to decompose the estimated output audio signal into multiple estimated background sub-band signals and background audio signal samples into multiple background sub-band signal samples, such as low-frequency sub-bands, mid-frequency sub-bands, and high-frequency sub-bands. This provides a precise comparative basis for subsequent sub-band level error calculation. Furthermore, based on the error loss of each estimated background sub-band frame signal and each background sub-band frame signal sample under each sub-band, combined with preset sub-band weights, the weighted frame-level sub-band error loss, i.e., the time-domain frequency division loss function, can be obtained, thereby strengthening the model's optimization guidance in the target audio frequency band.

[0076] In one example, based on preset frequency division parameters, the cutoff frequencies of the low-pass filter, band-pass filter, and high-pass filter can be clearly defined. For example, the low-pass filter cutoff frequency is 800Hz, the band-pass filter cutoff frequency is 800Hz-4000Hz, and the high-pass filter cutoff frequency is 4000Hz. That is, sub-band 1 is the low-frequency band from 0Hz to 800Hz, mainly including the fundamental frequency region of human voice; sub-band 2 is the mid-frequency band from 800Hz to 4000Hz, mainly including the region with rich human voice timbre; and sub-band 3 is the high-frequency band above 4000Hz, mainly for instrument overtones.

[0077] like Figure 6 In the time-domain frequency division loss function acquisition process shown, the estimated background sub-band frame signal within each sub-band is calculated separately. and each frame of background subband signal samples MSE loss involves dividing the vocal and accompaniment signals through sub-band filters to extract the time-domain sub-band signals corresponding to each frequency band. Time-domain loss, such as MSE loss or L1 loss, is calculated independently within each sub-band. Weighting coefficients are then assigned based on the importance of different frequency bands, i.e., pre-defined sub-band weights. Finally, the weighted sum of the losses from each sub-band yields the time-domain frequency division loss function. :

[0078] in, Indicates the number of sub-bands; Indicates the first The preset sub-band weight coefficients for each sub-band.

[0079] The time-domain frequency division loss reflects the differences in importance between different frequency bands and also takes into account the refined error feedback at the frame level, which can provide a stable and targeted optimization target for training audio separation models.

[0080] Compared to training an audio separation model solely based on a frequency loss function, the implementation method of this application guides the audio separation model to accurately optimize the target frequency band through a frequency band weighting strategy, thereby improving the accuracy of the audio separation model in separating mixed audio to a certain extent.

[0081] Furthermore, when the audio separation model needs to output the signal corresponding to the target audio, instead of setting corresponding weighting coefficients for the importance of different frequency bands, the preset sub-band weight of the sub-band where the target audio signal is located can be reduced. This guides the audio separation model to prioritize optimizing the region where the non-target audio is located, thereby improving the accuracy of the audio separation model in separating mixed audio to a certain extent.

[0082] Thus, based on preset frequency division parameters, frequency division processing is performed on the estimated output audio signal and background audio signal samples respectively to determine the estimated background sub-band signal and background sub-band signal samples. The estimated background sub-band signal includes multiple frames of estimated background sub-band signals, and the background sub-band signal samples include multiple frames of background sub-band signal samples. A time-domain frequency division loss function is determined based on each frame of estimated background sub-band signal, each frame of background sub-band signal samples, and preset sub-band weights. In this way, based on the preset frequency division parameters, the estimated output audio signal can be decomposed into multiple estimated background sub-band signals, and the background audio signal samples can be decomposed into multiple background sub-band signal samples. This provides a precise comparative basis for subsequent sub-band level error calculation. Furthermore, based on the error loss of each set of estimated background sub-band frames and each frame of background sub-band signal samples under each sub-band, combined with the preset sub-band weights, the time-domain frequency division loss function can be obtained. This strengthens the model's optimization guidance in the target audio frequency band and improves the accuracy of the audio separation model in separating mixed audio to a certain extent.

[0083] Please see Figure 7 In some implementations, step 04 (determining the fusion loss function of the audio separation model based on the time-domain frequency division loss function and the target energy weighted loss function) includes: 041: Determine the fusion loss function based on the time-domain frequency division loss function, the preset time-domain frequency division loss weight, the target energy weighted loss function, and the preset target energy weighted loss weight.

[0084] In some implementations, the determining module is further configured to determine the fusion loss function based on the time-domain frequency division loss function, the preset time-domain frequency division loss weight, the target energy weighted loss function, and the preset target energy weighted loss weight.

[0085] In some implementations, the processor is further configured to determine a fusion loss function based on a time-domain frequency division loss function, a preset time-domain frequency division loss weight, a target energy weighted loss function, and a preset target energy weighted loss weight.

[0086] Specifically, the preset time-domain frequency division loss weight is a pre-set coefficient used to adjust the influence of the time-domain frequency division loss function in the fusion loss. It is an adjustable hyperparameter. The larger the weight, the stronger the guiding effect of the time-domain frequency division loss function on model training, that is, the stronger the effect of guiding the audio separation model to accurately optimize the target frequency band.

[0087] The preset target energy weighted loss weight is a pre-set coefficient used to adjust the influence of the target energy weighted loss function in the fusion loss. Its value range is consistent with the preset time-domain frequency division loss weight. It is an adjustable hyperparameter. The larger the weight, the stronger the guiding effect of the target energy weighted loss function on model training, that is, the stronger the effect of guiding the audio separation model to prioritize optimizing regions with high target audio energy.

[0088] The preset time-domain frequency division loss weight and the preset target energy weighted loss weight are used to control the relative weights between the time-domain frequency division loss function and the target energy weighted loss function.

[0089] By pre-setting the time-domain frequency division loss weight and the target energy weighted loss weight, the time-domain frequency division loss function and the target energy weighted loss function are weighted and fused to obtain a fused loss function. This integrates the advantages of the time-domain frequency division loss function and the target energy weighted loss function, and guides the model to optimize training stability, target frequency band separation accuracy and output target audio listening quality, thereby improving the accuracy of audio separation model in separating mixed audio to a certain extent.

[0090] In one example, such as Figure 8 The fusion loss function shown is obtained through linear weighting. The process is as follows and is applicable to most training scenarios. By calculating the time-domain frequency division loss function and the target energy weighted loss function separately, and then weighting and fusing the time-domain frequency division loss function and the target energy weighted loss function based on preset time-domain frequency division loss weights and preset target energy weighted loss weights, a fused loss function can be obtained. :

[0091] in, The preset time-domain frequency division loss weights are used; Weight the loss by setting a preset target energy; This is the time-domain frequency division loss function; The target energy weighted loss function.

[0092] pass and The weighting ratio can achieve a precise balance between optimizing the time-domain frequency division loss function and the target energy weighted loss function. For example, if the audio separation model training is unstable, the weighting ratio can be appropriately increased. If noticeable vocal residue is found in a cappella performances, the volume can be increased appropriately. This ensures that the model always iterates in the optimal direction, so that the audio separation model training takes into account both stability and convergence speed, and can focus on the separation accuracy of the target frequency band and the listening quality of the output target audio, thereby improving the accuracy of the audio separation model in separating mixed audio to a certain extent.

[0093] Thus, based on the time-domain frequency division loss function, the preset time-domain frequency division loss weights, the target energy-weighted loss function, and the preset target energy-weighted loss weights, a fusion loss function is determined. This integrates the advantages of the time-domain frequency division loss function and the target energy-weighted loss function, allowing the fusion loss function to simultaneously guide model optimization training stability and target frequency band separation accuracy, thereby improving the accuracy of the audio separation model in separating mixed audio to a certain extent.

[0094] Please see Figure 9 In some implementations, the method further includes: 042: Determine the fusion loss function based on the target audio signal sample, the background audio signal sample, and the estimated output audio signal.

[0095] In some implementations, the determining module is also used to determine a fusion loss function based on the target audio signal sample, the background audio signal sample, and the estimated output audio signal.

[0096] In some implementations, the processor is also configured to determine a fusion loss function based on the target audio signal sample, the background audio signal sample, and the estimated output audio signal.

[0097] Specifically, the fusion loss function can also incorporate target audio signal samples, which are combined with background audio signal samples and the estimated output audio signal to determine the result.

[0098] In calculating the time-domain frequency division loss function, compared to independently calculating the time-domain frequency division loss function and the target energy weighted loss function and then weighting and summing them to obtain the fusion loss function, we can further consider the distribution of target audio energy in the current sub-band. Based on the target audio signal sample, background audio signal sample, and estimated output audio signal, for the target audio signal sample of each frequency band, we extract the energy distribution within that frequency band and generate the target energy weight for the corresponding frequency band based on the energy distribution. Within that frequency band, we use the target energy weight to weight the time-domain error. Finally, we summarize the weighted errors of all frequency bands, thereby obtaining a more refined fusion loss function without increasing the number of model parameters. This allows the fusion loss function to accurately lock the error region and more accurately guide the audio separation model to eliminate interference from the target audio, such as reducing the amount of vocal residue in scenarios like a cappella or weak background music.

[0099] Thus, based on the target audio signal samples, background audio signal samples, and estimated output audio signal, a fusion loss function is determined. Furthermore, by considering the distribution of target audio energy in the current sub-band, based on the target audio signal samples, background audio signal samples, and estimated output audio signal, the energy distribution within each frequency band is extracted, and a target energy weight for the corresponding frequency band is generated based on this energy distribution. Within that frequency band, the target energy weight is used to weight the temporal error, ultimately obtaining a more refined fusion loss function without increasing the number of model parameters. This allows the fusion loss function to accurately pinpoint the error region and more precisely guide the audio separation model to eliminate interference from the target audio.

[0100] Please see Figure 10 In some implementations, step 042 (determining the fusion loss function based on the target audio signal sample, the background audio signal sample, and the estimated output audio signal) includes: 0421: Based on preset frequency division parameters, frequency division processing is performed on the target audio signal sample, background audio signal sample and estimated output audio signal respectively to determine the target sub-band signal sample, background sub-band signal sample and estimated background sub-band signal. The target sub-band signal sample includes multiple frames of target sub-band frame signal sample, the background sub-band signal sample includes multiple frames of background sub-band frame signal sample, and the estimated background sub-band signal includes multiple frames of estimated background sub-band frame signal. 0422: Perform energy estimation processing on the target sub-band frame signal samples and the background sub-band frame signal samples of each frame to determine the target sub-band energy weight of each frame; 0423: Determine the fusion loss function based on the estimated background subband frame signal for each frame, the background subband frame signal sample for each frame, the target subband energy weight, and the preset subband weight.

[0101] In some embodiments, the determining module is further configured to perform frequency division processing on the target audio signal sample, background audio signal sample, and estimated output audio signal based on preset frequency division parameters, respectively, to determine the target sub-band signal sample, background sub-band signal sample, and estimated background sub-band signal. The target sub-band signal sample includes multiple frames of target sub-band frame signal samples, the background sub-band signal sample includes multiple frames of background sub-band frame signal samples, and the estimated background sub-band signal includes multiple frames of estimated background sub-band frame signals. The determining module is further configured to perform energy estimation processing on each frame of target sub-band frame signal sample and each frame of background sub-band frame signal sample to determine the target sub-band energy weight for each frame. The determining module is further configured to determine the time-domain frequency division loss function based on each frame of estimated background sub-band frame signal, each frame of background sub-band frame signal sample, the target sub-band energy weight, and the preset sub-band weight.

[0102] In some embodiments, the processor is further configured to perform frequency division processing on the target audio signal sample, background audio signal sample, and estimated output audio signal based on preset frequency division parameters, respectively, to determine the target sub-band signal sample, background sub-band signal sample, and estimated background sub-band signal. The target sub-band signal sample includes multiple frames of target sub-band frame signal samples, the background sub-band signal sample includes multiple frames of background sub-band frame signal samples, and the estimated background sub-band signal includes multiple frames of estimated background sub-band frame signals. The processor is further configured to perform energy estimation processing on each frame of target sub-band frame signal sample and each frame of background sub-band frame signal sample to determine the target sub-band energy weight for each frame. The processor is further configured to determine a time-domain frequency division loss function based on each frame of estimated background sub-band frame signal, each frame of background sub-band frame signal sample, the target sub-band energy weight, and the preset sub-band weight.

[0103] Specifically, the target sub-band signal sample refers to the continuous audio data obtained by dividing the target audio signal sample into frequency bands according to preset frequency division parameters. It corresponds one-to-one with the background sub-band signal sample and the estimated background sub-band signal, and is used to provide a frequency band-level reference for the target audio energy distribution.

[0104] The target subband energy weight is a proportionality coefficient calculated based on the energy of a single target subband frame signal sample and the corresponding background subband frame signal sample. It is used to characterize the proportion of target audio energy in the total energy within the frequency band and frame.

[0105] like Figure 11 In the process of obtaining the fusion loss function by combining target audio energy perception within the frequency band shown, based on the preset frequency division parameters, frequency division processing is performed on the target audio signal sample, background audio signal sample and estimated output audio signal, and energy estimation processing is performed on the target sub-band frame signal sample and the corresponding background sub-band frame signal sample of each frame. This allows the target sub-band energy weight of each frame to be obtained, so as to extract the target audio energy distribution in each frequency band and improve the accuracy of the audio separation model in separating mixed audio to a certain extent.

[0106] Understandably, the energy distribution of target audio in different frequency bands may be different. For example, there is more human voice in the low and mid frequencies, but some frames have higher energy. By obtaining the energy weight of the target sub-band of each frame, the invalid constraint of global energy weighting on the frequency band where the non-target audio is located can be avoided, so that the loss calculation of each frequency band can be adapted to the energy characteristics of its own signal.

[0107] like Figure 11 The fusion loss function is obtained by combining the target audio energy sensing within the frequency band shown. The process involves introducing target sub-band energy weights within each frequency band. This allows for a dual weighting of the error between the estimated background sub-band frame signal and the sample background sub-band frame signal within each frequency band, using both target sub-band energy weights and preset sub-band weights. This can be understood as first selecting the frequency band where the target audio is located, and then selecting the frame with the highest target audio energy within that frequency band. This simultaneously addresses both frequency band importance and energy distribution, thereby obtaining a more refined fusion loss function without increasing the number of model parameters. This enables the fusion loss function to accurately pinpoint the error region and more precisely guide the audio separation model to eliminate interference from the target audio, such as reducing vocal remnants in scenarios like a cappella or weak background music.

[0108] In one example, the fusion loss function It can also be obtained using the following formula:

[0109] in, Indicates the number of sub-bands in the frequency band; Indicates the first Preset sub-band weighting coefficients for each frequency band; Indicates the first Target energy weights in each frequency band; and Representing the background subband signal sample and the estimated background subband signal at the th... Time-domain signals of each frequency band.

[0110] Thus, based on preset frequency division parameters, frequency division processing is performed on the target audio signal sample, background audio signal sample, and estimated output audio signal respectively to determine the target sub-band signal sample, background sub-band signal sample, and estimated background sub-band signal. The target sub-band signal sample includes multiple frames of target sub-band frame signal samples, the background sub-band signal sample includes multiple frames of background sub-band frame signal samples, and the estimated background sub-band signal includes multiple frames of estimated background sub-band frame signals. Energy estimation processing is performed on each frame of target sub-band frame signal sample and each frame of background sub-band frame signal sample to determine the target sub-band energy weight for each frame. Based on each frame of estimated background sub-band frame signal, each frame of background sub-band frame signal sample, the target sub-band energy weight, and the preset sub-band weight, the time-domain frequency division loss function is determined. In this way, target sub-band energy weights can be introduced into each frequency band. Then, within each frequency band, the error between each frame of estimated background sub-band frame signal and each frame of background sub-band frame signal sample is subjected to dual weighting of target sub-band energy weights and preset sub-band weights. This simultaneously handles the importance of frequency bands and energy distribution, obtaining a more refined fusion loss function. This allows the fusion loss function to accurately lock the error region, thereby more accurately guiding the audio separation model to eliminate interference from the target audio.

[0111] the following Figure 12The training process of the audio separation model shown below will be explained using this example: First, the mixed audio signal samples are preprocessed, i.e., audio data preprocessing. The preprocessed mixed audio signal samples are then input into the audio separation model, which enables the audio separation model to output the estimated output audio signal through the neural network structure, thus achieving preliminary separation prediction, i.e., audio data postprocessing. Then, based on the estimated output audio signal, background audio signal sample, and target audio signal sample, the target energy weighted loss function (human voice energy weighted loss function) and the time-domain frequency division loss function of the audio separation model are calculated respectively. The target energy weighted loss function and the time-domain frequency division loss function are then fused to obtain the comprehensive loss, i.e., the fusion loss function. Finally, the audio separation model is trained based on the fusion loss function. With the fusion loss function as the optimization objective, the parameters of the audio separation model are adjusted through the backpropagation algorithm to continuously reduce the fusion loss value. Ultimately, the model gradually learns more accurate separation rules, resulting in a trained and stable audio separation model.

[0112] This application provides an audio separation method, characterized in that the method is based on an audio separation model, which is trained using an audio separation model training method. The method includes: 07: Acquire the target mixed audio signal, which includes the target audio signal and the background audio signal; 08: Based on the audio separation model, audio separation processing is performed on the target mixed audio signal to separate the background audio signal from the target mixed audio signal.

[0113] This application provides an audio separation apparatus. The audio separation method of this application can be implemented by the audio separation apparatus of this application. Specifically, the audio separation apparatus includes an acquisition module and a separation module. The acquisition module is used to acquire a target mixed audio signal, which includes a target audio signal and a background audio signal. The separation module is used to perform audio separation processing on the target mixed audio signal based on an audio separation model, so as to separate the background audio signal from the target mixed audio signal.

[0114] In some implementations, the processor is further configured to acquire a target mixed audio signal, which includes a target audio signal and a background audio signal. The processor is also configured to perform audio separation processing on the target mixed audio signal based on an audio separation model to separate the background audio signal from the target mixed audio signal.

[0115] Specifically, by training the audio separation model based on the audio separation model training method, the target mixed audio signal can be input into the audio separation model so that the audio separation model can perform audio separation processing on the target mixed audio signal to separate the background audio signal from the target mixed audio signal, thereby achieving accurate separation of the audio signal.

[0116] In one example, the acquired target mixed audio signal is directly input into the trained audio separation model. The model employs an end-to-end processing mode, requiring no manual intervention or additional post-processing steps. First, the model extracts features from the target mixed audio signal. Based on the spectral and energy feature patterns learned during training, it quickly identifies which parts of the signal belong to human voice and which parts belong to background audio. Subsequently, the human voice features are suppressed, while the background audio features are preserved and restored, accurately separating the target audio signal, namely the human voice and the background audio signal. Finally, the model directly outputs the restored background audio signal, with the output format consistent with the input format, which can be directly used for subsequent playback, editing and other scenarios.

[0117] Thus, a target mixed audio signal is acquired, which includes a target audio signal and a background audio signal. Based on an audio separation model, audio separation processing is performed on the target mixed audio signal to separate the background audio signal from the target mixed audio signal. In this way, by training the audio separation model using the audio separation model training method, the target mixed audio signal can be input into the audio separation model, enabling the audio separation model to perform audio separation processing on the target mixed audio signal to separate the background audio signal from the target mixed audio signal, achieving accurate audio signal separation.

[0118] This application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements the steps of the training method for the audio separation model as described above.

[0119] It is understood that a computer program includes computer program code. Computer program code can be in the form of source code, object code, executable files, or some intermediate form. Computer-readable storage media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc.

[0120] In this specification, the terms "specifically," "furthermore," "particularly," "understandably," etc., refer to specific features, structures, materials, or characteristics described in connection with embodiments or examples that are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0121] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of executable request code comprising one or more steps for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0122] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for training an audio separation model, characterized in that, The method includes: Based on the audio separation model, an estimated output audio signal is determined according to the acquired mixed audio signal samples, wherein the mixed audio signal samples include background audio signal samples and target audio signal samples corresponding to the background audio signal samples; The estimated output audio signal and the background audio signal samples are subjected to energy estimation processing to determine the target energy weighted loss function; The estimated output audio signal and the background audio signal samples are subjected to frequency division processing to determine the time-domain frequency division loss function; The fusion loss function of the audio separation model is determined based on the time-domain frequency division loss function and the target energy weighted loss function. The audio separation model is trained based on the fusion loss function.

2. The method of claim 1, wherein, The background audio signal sample includes multiple frames of background audio frame signal samples, and the target audio signal sample includes multiple frames of target audio frame signal samples. Each frame of the background audio frame signal sample corresponds to one frame of the target audio signal sample. The method further includes: Each background audio frame signal sample and the corresponding target audio frame signal sample are mixed with a preset energy ratio to determine multiple mixed audio frame signal samples, thereby determining the mixed audio signal sample.

3. The method according to claim 2, characterized in that, The estimated output audio signal includes multiple frames of estimated background audio frame signals. The step of performing energy estimation processing on the estimated output audio signal and the background audio signal samples to determine the target energy-weighted loss function includes: Energy estimation processing is performed on the target audio frame signal sample and the background audio frame signal sample of each frame to determine the target energy weight of each frame. The error loss for each frame is determined based on the estimated background audio frame signal for each frame and the background audio frame signal sample for each frame. The target energy weighted loss function is determined based on the error loss of each frame and the target energy weight of each frame.

4. The method according to claim 3, characterized in that, The step of performing frequency division processing on the estimated output audio signal and the background audio signal samples to determine the time-domain frequency division loss function includes: Based on preset frequency division parameters, the estimated output audio signal and the background audio signal sample are respectively subjected to frequency division processing to determine the estimated background sub-band signal and the background sub-band signal sample. The estimated background sub-band signal includes multiple frames of estimated background sub-band frame signal, and the background sub-band signal sample includes multiple frames of background sub-band frame signal sample. The time-domain frequency division loss function is determined based on the estimated background sub-band frame signal for each frame, the background sub-band frame signal sample for each frame, and the preset sub-band weight.

5. The method according to claim 4, characterized in that, The step of determining the fusion loss function of the audio separation model based on the time-domain frequency division loss function and the target energy weighted loss function includes: The fusion loss function is determined based on the time-domain frequency division loss function, the preset time-domain frequency division loss weight, the target energy weighted loss function, and the preset target energy weighted loss weight.

6. The method according to claim 5, characterized in that, The method further includes: The fusion loss function is determined based on the target audio signal sample, the background audio signal sample, and the estimated output audio signal.

7. The method according to claim 6, characterized in that, The step of determining the fusion loss function based on the target audio signal sample, the background audio signal sample, and the estimated output audio signal includes: Based on preset frequency division parameters, the target audio signal sample, the background audio signal sample, and the estimated output audio signal are subjected to frequency division processing respectively to determine the target sub-band signal sample, the background sub-band signal sample, and the estimated background sub-band signal. The target sub-band signal sample includes multiple frames of target sub-band frame signal samples, the background sub-band signal sample includes multiple frames of the background sub-band frame signal samples, and the estimated background sub-band signal includes multiple frames of the estimated background sub-band frame signal. Energy estimation processing is performed on the target sub-band frame signal sample and the background sub-band frame signal sample of each frame to determine the target sub-band energy weight of each frame. The time-domain frequency division loss function is determined based on the estimated background subband frame signal for each frame, the background subband frame signal sample for each frame, the target subband energy weight, and the preset subband weight.

8. An audio separation method, characterized in that, The method is based on an audio separation model, which is trained using the audio separation model training method described in any one of claims 1-7. The method includes: Acquire a target mixed audio signal, wherein the target mixed audio signal includes a target audio signal and a background audio signal; Based on the audio separation model, audio separation processing is performed on the target mixed audio signal to separate the background audio signal from the target mixed audio signal.

9. An electrical appliance, characterized in that, The method includes a memory and a processor, wherein the memory stores a computer program, which, when executed by the processor, implements the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by one or more processors, implements the method of any one of claims 1-8.

Images