Audio separation method, device, electronic device and storage medium

Abstract

The present application provides an audio separation method, device, electronic device and storage medium. The audio separation method includes: inputting the obtained mixed audio into a pre-trained audio separation model, dividing the mixed audio into multiple first sub-band features, and merging the multiple first sub-band features to determine the merged sub-band features; performing time series modeling on the merged sub-band features based on the audio separation model to obtain a time series feature sequence of the merged sub-band features, and then performing band-level modeling on the time series feature sequence to obtain a band feature sequence of the time series feature sequence; performing feature processing on the band feature sequence based on the audio separation model to determine the full-band mask feature, and separating the target audio from the mixed audio based on the full-band mask feature and the mixed audio. The target audio in the mixed audio is separated by the audio separation model, thereby improving the accuracy and efficiency of single audio separation.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to an audio separation method, device, electronic device and storage medium. Background Art

[0002] Track separation technology is divided into traditional track separation technology and neural network track separation technology. Traditional track separation technology such as non-negative matrix decomposition, Bayesian method, repetitive structure method, etc. These methods are usually based on signal processing and mathematical models, such as short-time Fourier transform (STFT) and related algorithms. These methods rely on the characteristics of frequency domain and time domain, and separate different tracks through filtering, phase difference and other technologies. It is necessary to manually design algorithms for extracting audio features, which may require domain expertise and experience. It is necessary to manually design algorithms for extracting audio features, which may require domain expertise and experience. It may be more sensitive to the quality and accuracy of data, and require more prior knowledge and manual adjustment. This makes these methods relatively inflexible in adapting to complex and diverse audio scenes, and performs poorly in processing complex audio scenes and multi-tracks. The complex audio signal structure and changes may exceed the modeling capabilities of traditional methods. Some traditional methods are more sensitive to the quality and accuracy of data, and they may require more prior knowledge and manual adjustment. Therefore, how to improve the accuracy of audio separation has become a technical issue that cannot be underestimated. Summary of the invention

[0003] In view of this, the purpose of the present application is to provide an audio separation method, device, electronic device and storage medium, which separate the target audio in the mixed audio through an audio separation model, thereby improving the accuracy and efficiency of single audio separation.

[0004] The present application provides an audio separation method, the audio separation method comprising:

[0005] Inputting the obtained mixed audio into a pre-trained audio separation model, dividing the mixed audio into a plurality of first sub-frequency band features, and merging the plurality of first sub-frequency band features to determine a merged sub-frequency band feature;

[0006] Based on the audio separation model, the combined sub-frequency band features are subjected to time series modeling to obtain a time series feature sequence of the combined sub-frequency band features, and then the time series feature sequence is subjected to frequency band level modeling to obtain a frequency band feature sequence of the time series feature sequence;

[0007] The frequency band feature sequence is feature processed based on the audio separation model to determine a full-band mask feature, and the target audio in the mixed audio is separated based on the full-band mask feature and the mixed audio.

[0008] In a possible implementation manner, the step of inputting the acquired mixed audio into a pre-trained audio separation model and dividing the mixed audio into a plurality of first sub-band features includes:

[0009] The audio separation model divides the mixed audio based on a preset frequency band division parameter to determine a plurality of first sub-frequency bands;

[0010] Performing normalization processing on each of the first sub-frequency bands to determine a first sub-frequency band after the normalization processing;

[0011] The first sub-frequency band after normalization is subjected to deformable convolution processing to determine a first feature, and the first feature is subjected to full connection processing to generate the first sub-frequency band feature.

[0012] In a possible implementation manner, performing time series modeling on the merged sub-band features based on the audio separation model to obtain a time series feature sequence of the merged sub-band features includes:

[0013] Normalizing the merged sub-band features based on the audio separation model to determine the normalized merged sub-band features;

[0014] The normalized merged sub-band features are input into the bidirectional long short-term memory network layer for feature processing to generate the second feature;

[0015] Performing deformable convolution processing on the second feature to determine a third feature, and performing full connection processing on the third feature to generate a fourth feature;

[0016] The fourth feature and the combined sub-frequency band feature are subjected to feature addition processing to generate a time series feature sequence of the combined sub-frequency band feature.

[0017] In a possible implementation manner, the performing feature processing on the frequency band feature sequence based on the audio separation model to determine a full-band mask feature, and separating the target audio from the mixed audio based on the full-band mask feature and the mixed audio includes:

[0018] Segmenting the frequency band feature sequence based on the audio separation model to determine a plurality of second sub-frequency band features; wherein the number of the second sub-frequency band features is consistent with the number of the first sub-frequency band features;

[0019] Performing normalization processing on each of the second sub-frequency band features to determine the normalized second sub-frequency band features;

[0020] Performing multi-layer perceptron processing on the second sub-frequency band features after the normalization processing to determine the second sub-frequency band features after the multi-layer perceptron processing;

[0021] Performing deformable convolution processing on the second sub-band features processed by the multi-layer perceptron to determine the second sub-band features processed by the deformable convolution, and merging a plurality of the second sub-band features processed by the deformable convolution to generate the full-band mask features;

[0022] The full-band mask feature is multiplied by the mixed audio to determine the target audio.

[0023] In a possible implementation manner, the audio separation model is determined by the following steps:

[0024] Acquire sample separation audio, sample residual audio, and sample mixed audio containing the sample separation audio in the sample audio data set;

[0025] Inputting the sample mixed audio into the residual neural network model, performing audio separation processing on the sample mixed audio, and separating the predicted separated audio present in the sample mixed audio;

[0026] The network parameters of the residual neural network model are adjusted based on the loss value between the predicted separated audio and the sample separated audio to determine the audio separation model.

[0027] In a possible implementation manner, adjusting the network parameters of the residual neural network model based on the loss value between the predicted separated audio and the sample separated audio to determine the audio separation model includes:

[0028] If the loss value between the predicted separated audio and the sample separated audio is less than or equal to a preset loss value threshold, determining the residual neural network model as the audio separation model;

[0029] If the loss value between the predicted separated audio and the sample separated audio is greater than the preset loss value threshold, the network parameters of the residual neural network model are adjusted until the loss value is less than or equal to the preset loss value threshold, then the adjustment of the network parameters of the residual neural network model is stopped to determine the audio separation model.

[0030] In a possible implementation manner, the sample audio data set is determined by the following steps:

[0031] If the energy value difference between the predicted separated audio and the sample mixed audio is greater than or equal to a preset energy value, the sample mixed audio is a clean sample residual audio;

[0032] If the energy difference between the predicted separated audio and the sample residual audio is less than a preset energy value, the sample mixed audio is a clean sample separated audio;

[0033] When the sample mixed audio includes the sample separated audio and the sample residual audio, marking the sample mixed audio as a pseudo label;

[0034] The sample audio dataset is constructed based on the sample mixed audio with multiple pseudo labels, multiple clean sample residual audio and multiple clean sample separated audio.

[0035] The present application also provides an audio separation device, the audio separation device comprising:

[0036] A frequency band cutting module, used for inputting the obtained mixed audio into a pre-trained audio separation model, dividing the mixed audio into a plurality of first sub-frequency band features, and merging the plurality of first sub-frequency band features to determine a merged sub-frequency band feature;

[0037] A modeling module, configured to perform time series modeling on the combined sub-frequency band features based on the audio separation model to obtain a time series feature sequence of the combined sub-frequency band features, and then perform frequency band level modeling on the time series feature sequence to obtain a frequency band feature sequence of the time series feature sequence;

[0038] A separation module is used to perform feature processing on the frequency band feature sequence based on the audio separation model to determine a full-band mask feature, and separate the target audio from the mixed audio based on the full-band mask feature and the mixed audio.

[0039] An embodiment of the present application also provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the audio separation method as described above are performed.

[0040] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the audio separation method as described above are executed.

[0041] An audio separation method, device, electronic device and storage medium provided by an embodiment of the present application, the audio separation method comprising: inputting the acquired mixed audio into a pre-trained audio separation model, dividing the mixed audio into a plurality of first sub-band features, and merging the plurality of first sub-band features to determine a merged sub-band feature; performing time series modeling on the merged sub-band feature based on the audio separation model to obtain a time series feature sequence of the merged sub-band feature, and then performing frequency band-level modeling on the time series feature sequence to obtain a frequency band feature sequence of the time series feature sequence; performing feature processing on the frequency band feature sequence based on the audio separation model to determine a full-band mask feature, and separating the target audio from the mixed audio based on the full-band mask feature and the mixed audio. The target audio from the mixed audio is separated by the audio separation model, thereby improving the accuracy and efficiency of single audio separation.

[0042] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A flowchart of an audio separation method provided in an embodiment of the present application;

[0045] Figure 2 This is one of the structural schematic diagrams of an audio separation device provided in an embodiment of the present application;

[0046] Figure 3 A second structural diagram of an audio separation device provided in an embodiment of the present application;

[0047] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The drawings in the embodiments clearly and completely describe the technical solutions in the embodiments of the present application. It should be understood that the drawings in the present application are only for the purpose of illustration and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowcharts used in the present application illustrate the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art, under the guidance of the content of the present application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0049] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0050] First, the application scenarios to which the present application is applicable are introduced. The present application can be applied in the field of audio processing technology.

[0051] According to research, track separation technology is divided into traditional track separation technology and neural network track separation technology. Traditional track separation technology, such as non-negative matrix decomposition, Bayesian method, repetitive structure method, etc., are usually based on signal processing and mathematical models, such as short-time Fourier transform (STFT) and related algorithms. These methods rely on the characteristics of frequency domain and time domain, and separate different tracks through filtering, phase difference and other technologies. It is necessary to manually design algorithms for extracting audio features, which may require domain expertise and experience. It is necessary to manually design algorithms for extracting audio features, which may require domain expertise and experience. It may be more sensitive to the quality and accuracy of data, and require more prior knowledge and manual adjustment. This makes these methods relatively inflexible in adapting to complex and diverse audio scenes, and performs poorly in processing complex audio scenes and multi-tracks. The complex audio signal structure and changes may exceed the modeling capabilities of traditional methods. Some traditional methods are more sensitive to the quality and accuracy of data, and they may require more prior knowledge and manual adjustment. Therefore, how to improve the accuracy of audio separation has become a technical issue that cannot be underestimated.

[0052] Based on this, an embodiment of the present application provides an audio separation method, which separates the target audio in the mixed audio through an audio separation model, thereby improving the accuracy and efficiency of single audio separation.

[0053] See also Figure 1 , Figure 1 This is a flow chart of an audio separation method provided in an embodiment of the present application. Figure 1 As shown in , the audio separation method provided by the embodiment of the present application includes:

[0054] S101: Input the obtained mixed audio into a pre-trained audio separation model, divide the mixed audio into a plurality of first sub-frequency band features, and merge the plurality of first sub-frequency band features to determine a merged sub-frequency band feature.

[0055] In this step, the obtained mixed audio is input into the audio separation model, the mixed audio is divided into a plurality of first sub-frequency band features, and the plurality of first sub-frequency band features are merged to determine a merged sub-frequency band feature.

[0056] In a possible implementation manner, the step of inputting the acquired mixed audio into a pre-trained audio separation model and dividing the mixed audio into a plurality of first sub-band features includes:

[0057] A: The audio separation model segments the mixed audio based on preset frequency band segmentation parameters to determine a plurality of first sub-frequency bands.

[0058] Here, the audio separation model divides the mixed audio according to a preset frequency band division parameter to determine a plurality of first sub-frequency bands.

[0059] The audio separation model performs frequency band segmentation according to the frequency band segmentation parameters set before training, such as 0~1KHZ, 1KHZ~2KHZ...40KHZ~50KHZ.

[0060] B: performing normalization processing on each of the first sub-frequency bands to determine a first sub-frequency band after the normalization processing.

[0061] Here, normalization processing is performed on each first sub-frequency band to determine a first sub-frequency band after normalization processing.

[0062] C: performing deformable convolution processing on the first sub-frequency band after normalization processing to determine the first feature, and performing full connection processing on the first feature to generate the first sub-frequency band feature.

[0063] Here, deformable convolution processing is performed on the normalized first sub-frequency band to determine the first feature, and full connection processing is performed on the first feature to generate the first sub-frequency band feature.

[0064] In this scheme, the audio is divided into frequency bands according to the hyperparameters set before training, such as 0~1KHZ, 1KHZ~2KHZ...40KHZ~50KHZ, and then the divided signals are normalized and fully connected to generate feature vectors, which are then merged to form full-band feature vectors. Since the feature dimensions of each first sub-band are the same, the RNN network with the same input and output can be shared for training, so there is no need to build a separate network for each band, which reduces the size of the model. Then, using prior knowledge, more computer resources are invested in fine-grained calculations for the frequency band intervals where the target sound source is concentrated, and coarse-grained calculations are performed for the frequency band intervals where the target sound source is not concentrated, thereby improving the training speed and the performance of the model to separate the audio tracks. In addition, each first sub-band can be calculated in parallel, which further improves the model training speed.

[0065] S102: Based on the audio separation model, time series modeling is performed on the merged sub-frequency band features to obtain a time series feature sequence of the merged sub-frequency band features, and then frequency band level modeling is performed on the time series feature sequence to obtain a frequency band feature sequence of the time series feature sequence.

[0066] In this step, time series modeling is performed on the merged sub-band features according to the audio separation model to obtain a time series feature sequence of the merged sub-band features, and then frequency band level modeling is performed on the time series feature sequence to obtain a frequency band feature sequence of the time series feature sequence.

[0067] Among them, when using the merged sub-band features to model the time series, the next step of frequency band level modeling is entered through data processing and feature extraction after convolution. After data processing and feature extraction after convolution, the output containing both time series features and frequency band features is obtained.

[0068] In a possible implementation manner, performing time series modeling on the merged sub-band features based on the audio separation model to obtain a time series feature sequence of the merged sub-band features includes:

[0069] a: normalizing the merged sub-band features based on the audio separation model to determine the normalized merged sub-band features.

[0070] Here, the merged sub-band features are normalized according to the audio separation model to determine the normalized merged sub-band features.

[0071] b: The normalized merged sub-band features are input into the bidirectional long short-term memory network layer for feature processing to generate the second feature.

[0072] Here, the normalized merged sub-band features are input into the bidirectional long short-term memory network layer for feature processing to generate the second feature.

[0073] c: Perform deformable convolution processing on the second feature to determine the third feature, and perform full connection processing on the third feature to generate the fourth feature.

[0074] Here, the second feature is subjected to deformable convolution processing to determine the third feature, and the third feature is subjected to full connection processing to generate the fourth feature.

[0075] d: performing feature addition processing on the fourth feature and the combined sub-frequency band feature to generate a time series feature sequence of the combined sub-frequency band feature.

[0076] Among them, the process of performing frequency band level modeling on the time series feature sequence is consistent with the above-mentioned process of performing time series modeling on the merged sub-band features, and will not be described in detail in this part.

[0077] S103: performing feature processing on the frequency band feature sequence based on the audio separation model to determine full-band mask features, and separating target audio from the mixed audio based on the full-band mask features and the mixed audio.

[0078] In this step, the frequency band feature sequence is feature processed according to the audio separation model to determine the full-band mask feature, and the target audio in the mixed audio is separated according to the full-band mask feature and the mixed audio.

[0079] In a possible implementation manner, the performing feature processing on the frequency band feature sequence based on the audio separation model to determine a full-band mask feature, and separating the target audio from the mixed audio based on the full-band mask feature and the mixed audio includes:

[0080] (1): Based on the audio separation model, the frequency band feature sequence is segmented to determine a plurality of second sub-band features; wherein the number of the second sub-band features is consistent with the number of the first sub-band features.

[0081] Here, the frequency band feature sequence is segmented according to the audio separation model to determine a plurality of second sub-frequency band features.

[0082] (2): Perform normalization processing on each of the second sub-frequency band features to determine the normalized second sub-frequency band features.

[0083] Here, each second sub-frequency band feature is normalized to determine the normalized second sub-frequency band feature.

[0084] (3): Perform multi-layer perceptron processing on the second sub-frequency band features after normalization processing to determine the second sub-frequency band features after multi-layer perceptron processing.

[0085] Here, the second sub-frequency band features after normalization are processed by a multi-layer perceptron to determine the second sub-frequency band features after the multi-layer perceptron processing.

[0086] (4): Performing deformable convolution processing on the second sub-band features after the multi-layer perceptron processing to determine the second sub-band features after the deformable convolution processing, and merging multiple second sub-band features after the deformable convolution processing to generate the full-band mask features.

[0087] Here, deformable convolution is performed on the second sub-band features processed by the multi-layer perceptron to determine the second sub-band features processed by the deformable convolution, and multiple second sub-band features processed by the deformable convolution are merged to generate full-band mask features.

[0088] The audio separation model uses deformable convolution to replace traditional convolution in frequency band segmentation, time series and band-level modeling, and mask generation to improve model performance. Deformable convolution first inputs an original feature map, and then obtains an offset through a convolution. That is, the offset of the deformable convolution is generated by a convolution layer, the input of which is the original feature image, and the output is the offset.

[0089] (5): Multiply the full-band mask feature by the mixed audio to determine the target audio.

[0090] Here, the full-band mask feature is multiplied by the mixed audio to determine the target audio.

[0091] In this scheme, the frequency band feature sequence is further divided into K second sub-band features according to the configuration file. After data processing and convolution extraction of multiple second sub-band features, sub-band masks are obtained, which are then merged into a full-band mask. The full-band mask is multiplied by the mixed audio input to obtain the separated target audio.

[0092] In a possible implementation manner, the audio separation model is determined by the following steps:

[0093] I: sample separation audio, sample residual audio, and sample mixed audio containing the sample separation audio are obtained in the sample audio data set.

[0094] Here, sample separation audio, sample residual audio, and sample mixed audio containing the sample separation audio are obtained in the sample audio data set.

[0095] The sample residual audio is the audio obtained by separating the sample separated audio from the sample mixed audio.

[0096] In a possible implementation manner, the sample audio data set is determined by the following steps:

[0097] 1: If the energy value difference between the predicted separated audio and the sample mixed audio is greater than or equal to the preset energy value, the sample mixed audio is a clean sample residual audio; if the energy difference between the predicted separated audio and the sample residual audio is less than the preset energy value, the sample mixed audio is a clean sample separated audio.

[0098] Here, a data filtering method based on energy value is used to detect whether the separated signal is clean or noisy / distorted. If the energy value difference (in decibels) between the sample mixed audio and the predicted separated audio is greater than 30dB, the sample mixed audio is considered to be clean sample residual audio. If the energy difference between the sample mixed audio and the separated sample residual audio is greater than 30dB, the sample mixed audio is clean sample separated audio.

[0099] Second, when the sample mixed audio includes the sample separated audio and the sample residual audio, the sample mixed audio is marked as a pseudo label.

[0100] Here, when the sample mixed audio includes the sample separated audio and the sample residual audio, the sample mixed audio is labeled as a pseudo label.

[0101] 3: Construct the sample audio dataset based on the sample mixed audio with multiple pseudo labels, multiple clean sample residual audios and multiple clean sample separated audios.

[0102] Here, a sample audio dataset is constructed based on the sample mixed audio with multiple pseudo labels, multiple clean sample residual audios, and multiple clean sample separated audios.

[0103] In this scheme, two data enhancement methods are adopted to generate pseudo-label data in real time during training and mix it with real labeled data, reducing the demand for data volume: If the random number is less than the set silent probability, the target track will be removed from the mixed track to obtain a silent segment. If the random number is less than the set mixing probability, a certain number of other tracks will be randomly selected and mixed into the target track. Both methods transform the existing data set during the training and learning process, which will not take up hard disk space, but can greatly improve the richness and diversity of the data.

[0104] II: Input the sample mixed audio into the residual neural network model, perform audio separation processing on the sample mixed audio, and separate the predicted separated audio existing in the sample mixed audio.

[0105] Here, the sample mixed audio is input into the residual neural network model, and the sample mixed audio is subjected to audio separation processing to separate the predicted separated audio present in the sample mixed audio.

[0106] III: Adjusting the network parameters of the residual neural network model based on the loss value between the predicted separated audio and the sample separated audio to determine the audio separation model.

[0107] Here, the network parameters of the residual neural network model are adjusted according to the loss value between the predicted separated audio and the sample separated audio to determine the audio separation model.

[0108] In a possible implementation manner, adjusting the network parameters of the residual neural network model based on the loss value between the predicted separated audio and the sample separated audio to determine the audio separation model includes:

[0109] i: If the loss value between the predicted separated audio and the sample separated audio is less than or equal to a preset loss value threshold, the residual neural network model is determined as the audio separation model.

[0110] Here, if the loss value between the predicted separated audio and the sample separated audio is less than or equal to a preset loss value threshold, the residual neural network model is determined as the audio separation model.

[0111] ii: If the loss value between the predicted separated audio and the sample separated audio is greater than the preset loss value threshold, the network parameters of the residual neural network model are adjusted until the loss value is less than or equal to the preset loss value threshold, then the adjustment of the network parameters of the residual neural network model is stopped to determine the audio separation model.

[0112] Here, if the loss value between the predicted separated audio and the sample separated audio is greater than the preset loss value threshold, the network parameters of the residual neural network model are adjusted until the loss value is less than or equal to the preset loss value threshold, then the adjustment of the network parameters of the residual neural network model is stopped to determine the audio separation model.

[0113] In this scheme, the musical instrument separated by the audio separation model can be the erhu, and the frequency range of the erhu is 300-8KHZ, and the frequency bands with the largest frequency peak and the most common sounds are 315, 400, 500, 800, 1K, 1.25, 1.6K, 2KHZ, so the parameters are set to give a finer-grained frequency band segmentation scheme for 315-2KHZ, and a coarser-grained frequency band segmentation scheme for 0-315HZ and 2K-8KHZ.

[0114] An audio separation method provided by an embodiment of the present application includes: inputting the acquired mixed audio into a pre-trained audio separation model, dividing the mixed audio into multiple first sub-band features, and merging the multiple first sub-band features to determine the merged sub-band features; performing time series modeling on the merged sub-band features based on the audio separation model to obtain a time series feature sequence of the merged sub-band features, and then performing frequency band-level modeling on the time series feature sequence to obtain a frequency band feature sequence of the time series feature sequence; performing feature processing on the frequency band feature sequence based on the audio separation model to determine a full-band mask feature, and separating the target audio from the mixed audio based on the full-band mask feature and the mixed audio. The target audio in the mixed audio is separated by the audio separation model, thereby improving the accuracy and efficiency of single audio separation.

[0115] See also Figure 2 , Figure 3 , Figure 2 This is one of the structural schematic diagrams of an audio separation device provided in an embodiment of the present application; Figure 3 This is a second structural diagram of an audio separation device provided in an embodiment of the present application. Figure 2 As shown in , the audio separation device 200 includes:

[0116] The frequency band cutting module 210 is used to input the obtained mixed audio into a pre-trained audio separation model, divide the mixed audio into a plurality of first sub-frequency band features, and merge the plurality of first sub-frequency band features to determine a merged sub-frequency band feature;

[0117] A modeling module 220 is used to perform time series modeling on the combined sub-frequency band features based on the audio separation model to obtain a time series feature sequence of the combined sub-frequency band features, and then perform frequency band level modeling on the time series feature sequence to obtain a frequency band feature sequence of the time series feature sequence;

[0118] The separation module 230 is used to perform feature processing on the frequency band feature sequence based on the audio separation model to determine a full-band mask feature, and separate the target audio from the mixed audio based on the full-band mask feature and the mixed audio.

[0119] In a possible implementation manner, when the frequency band cutting module 210 is used to input the acquired mixed audio into a pre-trained audio separation model and divide the mixed audio into a plurality of first sub-band features, the frequency band cutting module 210 is specifically used to:

[0120] The audio separation model divides the mixed audio based on a preset frequency band division parameter to determine a plurality of first sub-frequency bands;

[0121] Performing normalization processing on each of the first sub-frequency bands to determine a first sub-frequency band after the normalization processing;

[0122] The first sub-frequency band after normalization is subjected to deformable convolution processing to determine a first feature, and the first feature is subjected to full connection processing to generate the first sub-frequency band feature.

[0123] Further, when the modeling module 220 is used to perform time series modeling on the merged sub-band features based on the audio separation model to obtain a time series feature sequence of the merged sub-band features, the modeling module 220 is specifically used to:

[0124] Normalizing the merged sub-band features based on the audio separation model to determine the normalized merged sub-band features;

[0125] The normalized merged sub-band features are input into the bidirectional long short-term memory network layer for feature processing to generate the second feature;

[0126] Performing deformable convolution processing on the second feature to determine a third feature, and performing full connection processing on the third feature to generate a fourth feature;

[0127] The fourth feature and the combined sub-frequency band feature are subjected to feature addition processing to generate a time series feature sequence of the combined sub-frequency band feature.

[0128] Further, when the separation module 230 is used to perform feature processing on the frequency band feature sequence based on the audio separation model to determine the full-band mask feature, and separate the target audio from the mixed audio based on the full-band mask feature and the mixed audio, the separation module 230 is specifically used to:

[0129] Segmenting the frequency band feature sequence based on the audio separation model to determine a plurality of second sub-frequency band features; wherein the number of the second sub-frequency band features is consistent with the number of the first sub-frequency band features;

[0130] Performing normalization processing on each of the second sub-frequency band features to determine the normalized second sub-frequency band features;

[0131] Performing multi-layer perceptron processing on the second sub-frequency band features after the normalization processing to determine the second sub-frequency band features after the multi-layer perceptron processing;

[0132] Performing deformable convolution processing on the second sub-band features processed by the multi-layer perceptron to determine the second sub-band features processed by the deformable convolution, and merging a plurality of the second sub-band features processed by the deformable convolution to generate the full-band mask features;

[0133] The full-band mask feature is multiplied by the mixed audio to determine the target audio.

[0134] Further, such as Figure 3 As shown, the audio separation device 200 further includes a training module 240, and the training module 240 is used to:

[0135] Acquire sample separation audio, sample residual audio, and sample mixed audio containing the sample separation audio in the sample audio data set;

[0136] Inputting the sample mixed audio into the residual neural network model, performing audio separation processing on the sample mixed audio, and separating the predicted separated audio present in the sample mixed audio;

[0137] The network parameters of the residual neural network model are adjusted based on the loss value between the predicted separated audio and the sample separated audio to determine the audio separation model.

[0138] Further, when the training module 240 is used to adjust the network parameters of the residual neural network model based on the loss value between the predicted separated audio and the sample separated audio to determine the audio separation model, the training module 240 is specifically used to:

[0139] If the loss value between the predicted separated audio and the sample separated audio is less than or equal to a preset loss value threshold, determining the residual neural network model as the audio separation model;

[0140] If the loss value between the predicted separated audio and the sample separated audio is greater than the preset loss value threshold, the network parameters of the residual neural network model are adjusted until the loss value is less than or equal to the preset loss value threshold, then the adjustment of the network parameters of the residual neural network model is stopped to determine the audio separation model.

[0141] Further, such as Figure 3 As shown, the audio separation device 200 further includes a data set construction module 250. The data set construction module 250 determines the sample audio data set through the following steps:

[0142] If the energy value difference between the predicted separated audio and the sample mixed audio is greater than or equal to a preset energy value, the sample mixed audio is a clean sample residual audio;

[0143] If the energy difference between the predicted separated audio and the sample residual audio is less than a preset energy value, the sample mixed audio is a clean sample separated audio;

[0144] When the sample mixed audio includes the sample separated audio and the sample residual audio, marking the sample mixed audio as a pseudo label;

[0145] The sample audio dataset is constructed based on the sample mixed audio with multiple pseudo labels, multiple clean sample residual audio and multiple clean sample separated audio.

[0146] An audio separation device provided in an embodiment of the present application includes: a frequency band cutting module for inputting the acquired mixed audio into a pre-trained audio separation model, dividing the mixed audio into a plurality of first sub-band features, and merging the plurality of first sub-band features to determine a merged sub-band feature; a modeling module for performing time series modeling on the merged sub-band features based on the audio separation model to obtain a time series feature sequence of the merged sub-band features, and then performing frequency band level modeling on the time series feature sequence to obtain a frequency band feature sequence of the time series feature sequence; a separation module for performing feature processing on the frequency band feature sequence based on the audio separation model to determine a full-band mask feature, and separating the target audio from the mixed audio based on the full-band mask feature and the mixed audio. The target audio from the mixed audio is separated by the audio separation model, thereby improving the accuracy and efficiency of single audio separation.

[0147] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4As shown in , the electronic device 400 includes a processor 410 , a memory 420 and a bus 430 .

[0148] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, the above-mentioned Figure 1 The steps of the audio separation method in the method embodiment shown, the specific implementation method can be found in the method embodiment, and will not be repeated here.

[0149] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The steps of the audio separation method in the method embodiment shown, the specific implementation method can be found in the method embodiment, and will not be repeated here.

[0150] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0151] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0152] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0153] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0154] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0155] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An audio separation method, characterized in that: The audio separation method comprises: Inputting the obtained mixed audio into a pre-trained audio separation model, dividing the mixed audio into a plurality of first sub-frequency band features, and merging the plurality of first sub-frequency band features to determine a merged sub-frequency band feature; Based on the audio separation model, the combined sub-frequency band features are subjected to time series modeling to obtain a time series feature sequence of the combined sub-frequency band features, and then the time series feature sequence is subjected to frequency band level modeling to obtain a frequency band feature sequence of the time series feature sequence; Performing feature processing on the frequency band feature sequence based on the audio separation model to determine full-band mask features, and separating the target audio from the mixed audio based on the full-band mask features and the mixed audio; The performing time series modeling on the merged sub-frequency band features based on the audio separation model to obtain a time series feature sequence of the merged sub-frequency band features includes: Normalizing the merged sub-band features based on the audio separation model to determine the normalized merged sub-band features; The normalized merged sub-band features are input into the bidirectional long short-term memory network layer for feature processing to generate the second feature; Performing deformable convolution processing on the second feature to determine a third feature, and performing full connection processing on the third feature to generate a fourth feature; Perform feature addition processing on the fourth feature and the combined sub-frequency band feature to generate a time series feature sequence of the combined sub-frequency band feature; The process of performing frequency band level modeling on the time series feature sequence is consistent with the process of performing time series modeling on the merged sub-frequency band features.

2. The audio separation method according to claim 1, characterized in that: The step of inputting the obtained mixed audio into a pre-trained audio separation model and dividing the mixed audio into a plurality of first sub-band features includes: The audio separation model divides the mixed audio based on a preset frequency band division parameter to determine a plurality of first sub-frequency bands; Performing normalization processing on each of the first sub-frequency bands to determine a first sub-frequency band after the normalization processing; The first sub-frequency band after normalization is subjected to deformable convolution processing to determine a first feature, and the first feature is subjected to full connection processing to generate the first sub-frequency band feature.

3. The audio separation method according to claim 1, characterized in that: The step of performing feature processing on the frequency band feature sequence based on the audio separation model to determine a full-band mask feature, and separating the target audio from the mixed audio based on the full-band mask feature and the mixed audio includes: Segmenting the frequency band feature sequence based on the audio separation model to determine a plurality of second sub-frequency band features; wherein the number of the second sub-frequency band features is consistent with the number of the first sub-frequency band features; Performing normalization processing on each of the second sub-frequency band features to determine the normalized second sub-frequency band features; Performing multi-layer perceptron processing on the second sub-frequency band features after the normalization processing to determine the second sub-frequency band features after the multi-layer perceptron processing; Performing deformable convolution processing on the second sub-band features processed by the multi-layer perceptron to determine the second sub-band features processed by the deformable convolution, and merging a plurality of the second sub-band features processed by the deformable convolution to generate the full-band mask features; The full-band mask feature is multiplied by the mixed audio to determine the target audio.

4. The audio separation method according to claim 1, characterized in that: The audio separation model is determined by the following steps: Acquire sample separation audio, sample residual audio, and sample mixed audio containing the sample separation audio in the sample audio data set; Inputting the sample mixed audio into the residual neural network model, performing audio separation processing on the sample mixed audio, and separating the predicted separated audio present in the sample mixed audio; The network parameters of the residual neural network model are adjusted based on the loss value between the predicted separated audio and the sample separated audio to determine the audio separation model.

5. The audio separation method according to claim 4, characterized in that: The adjusting the network parameters of the residual neural network model based on the loss value between the predicted separated audio and the sample separated audio to determine the audio separation model includes: If the loss value between the predicted separated audio and the sample separated audio is less than or equal to a preset loss value threshold, determining the residual neural network model as the audio separation model; If the loss value between the predicted separated audio and the sample separated audio is greater than the preset loss value threshold, the network parameters of the residual neural network model are adjusted until the loss value is less than or equal to the preset loss value threshold, then the adjustment of the network parameters of the residual neural network model is stopped to determine the audio separation model.

6. The audio separation method according to claim 4, characterized in that: The sample audio data set is determined by the following steps: If the energy value difference between the predicted separated audio and the sample mixed audio is greater than or equal to a preset energy value, the sample mixed audio is a clean sample residual audio; If the energy difference between the predicted separated audio and the sample residual audio is less than a preset energy value, the sample mixed audio is a clean sample separated audio; When the sample mixed audio includes the sample separated audio and the sample residual audio, marking the sample mixed audio as a pseudo label; The sample audio dataset is constructed based on the sample mixed audio with multiple pseudo labels, multiple clean sample residual audio and multiple clean sample separated audio.

7. An audio separation device, characterized in that: The audio separation device comprises: A frequency band cutting module, used for inputting the obtained mixed audio into a pre-trained audio separation model, dividing the mixed audio into a plurality of first sub-frequency band features, and merging the plurality of first sub-frequency band features to determine a merged sub-frequency band feature; A modeling module, configured to perform time series modeling on the combined sub-frequency band features based on the audio separation model to obtain a time series feature sequence of the combined sub-frequency band features, and then perform frequency band level modeling on the time series feature sequence to obtain a frequency band feature sequence of the time series feature sequence; A separation module, configured to perform feature processing on the frequency band feature sequence based on the audio separation model to determine a full-band mask feature, and separate the target audio from the mixed audio based on the full-band mask feature and the mixed audio; When the modeling module is used to perform time series modeling on the merged sub-band features based on the audio separation model to obtain a time series feature sequence of the merged sub-band features, the modeling module is specifically used to: Normalizing the merged sub-band features based on the audio separation model to determine the normalized merged sub-band features; The normalized merged sub-band features are input into the bidirectional long short-term memory network layer for feature processing to generate the second feature; Performing deformable convolution processing on the second feature to determine a third feature, and performing full connection processing on the third feature to generate a fourth feature; Perform feature addition processing on the fourth feature and the combined sub-frequency band feature to generate a time series feature sequence of the combined sub-frequency band feature; The process of performing frequency band level modeling on the time series feature sequence is consistent with the process of performing time series modeling on the merged sub-frequency band features.

8. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to perform the steps of the audio separation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the audio separation method according to any one of claims 1 to 6 are executed.

Images