Audio processing method, training method of sound source separation model, and electronic device

Abstract

The application provides an audio processing method, a training method of a sound source separation model and an electronic device. The audio processing method comprises: fully extracting audio features of to-be-processed audio data by using a two-dimensional convolution-based feature extraction network, a frequency band interaction layer and a frequency point interaction layer of a sound source separation model, and then performing sound source separation based on the fully extracted audio features. The scheme adds the frequency band interaction layer and the frequency point interaction layer on the basis of the two-dimensional convolution-based feature extraction network, extracts frequency band interaction features between multiple sub-frequency bands and global frequency point interaction features in each sub-frequency band, so that the advantages of small calculation amount of the two-dimensional convolution-based feature extraction network are retained, and the defects of insufficient extraction capability of the two-dimensional convolution-based feature extraction network for global interaction features are made up, so that the entire sound source separation process can achieve the effect of relatively high separation precision and relatively short processing delay.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio processing, and in particular to an audio processing method, a training method of a sound source separation model, and an electronic device. BACKGROUND

[0002] Sound source separation refers to separating an audio signal of a certain type of sound source from mixed audio, such as separating (extracting) an audio signal of a certain or several types of musical instruments from a mixed audio of multiple musical instruments, or separating a human voice from a song with accompaniment. Sound source separation is often applied to online processing scenarios, and such online real-time processing has a high requirement for latency. However, a traditional sound source separation method cannot meet the above short-latency requirement due to a large amount of computation and long latency. SUMMARY

[0003] The present application provides an audio processing method, a training method of a sound source separation model, and an electronic device, which can shorten processing latency while ensuring separation accuracy.

[0004] In a first aspect, an audio processing method is provided. The method includes: obtaining to-be-processed audio data, the to-be-processed audio data including audio signals of multiple sound sources; processing the to-be-processed audio data by using a sound source separation model to obtain an audio signal of at least one target sound source; the at least one target sound source is at least one sound source of the multiple sound sources; the sound source separation model includes a feature extraction network, a frequency band interaction layer, and a frequency point interaction layer; the feature extraction network is constructed by using a two-dimensional convolutional base network, and is configured to perform feature extraction on the to-be-processed audio data to obtain a first audio feature set; the frequency band interaction layer is configured to extract frequency band interaction features between multiple sub-bands corresponding to the to-be-processed audio data from the first audio feature set, so as to obtain a second audio feature set; the frequency point interaction layer is configured to extract global frequency point interaction features in each sub-band in the second audio feature set, so as to obtain a third audio feature set; and the sound source separation model is further configured to separate the audio signal of the at least one target sound source from the third audio feature set.

[0005] In the technical solution of the present application, the frequency band interaction layer and the frequency point interaction layer are added on the basis of the two-dimensional convolutional feature extraction network, the frequency band interaction features between the multiple sub-bands are extracted, and the global frequency point interaction features in each sub-band are extracted, so that the advantages of small amount of computation of the two-dimensional convolutional feature extraction network are retained, and the defect of insufficient extraction capability of the two-dimensional convolutional feature extraction network for global interaction features is made up, so that the entire sound source separation process can achieve a relatively high separation accuracy and a relatively short processing latency.

[0006] In some implementations of the first aspect, the plurality of sub-bands are obtained by dividing the to-be-processed audio data along a frequency dimension, or the plurality of sub-bands are obtained by dividing the first extracted result output by the first network layer along a frequency dimension, the first network layer being any one of the layers of the feature extraction network. In this implementation, the plurality of sub-bands corresponding to the to-be-processed audio data can be divided before entering the feature extraction network, during feature extraction by the feature extraction network, or before inputting the extracted features to the frequency band interaction layer after the feature extraction network finishes extracting features, so that the adjustment can be made flexibly according to actual application scenarios.

[0007] In some implementations of the first aspect, when the feature extraction network has the capability of extracting frequency band interaction features between the plurality of sub-bands, so that the first audio feature set contains the frequency band interaction features between the plurality of sub-bands, the source separation model does not include the frequency band interaction layer; and / or when the feature extraction network has the capability of extracting global frequency point interaction features of each frequency point corresponding to the to-be-processed audio data, so that the first audio feature set contains the global frequency point interaction features of each frequency point, the source separation model does not include the frequency point interaction layer.

[0008] For some special feature extraction networks, in combination with the segmentation operation (operation of dividing the plurality of sub-bands), some traditional feature extraction networks can also have the capability of extracting global interaction features, so for this type of feature extraction networks, the network structure can be appropriately simplified, so as to further reduce the amount of calculation and shorten the time delay without affecting the feature extraction effect.

[0009] In some implementations of the first aspect, the method further includes: matching, by a reshape operation, a target dimension with an action dimension of a base network for constructing the frequency band interaction layer; and / or matching, by the reshape operation, a target dimension of the frequency point interaction layer with an action dimension of a base network for constructing the frequency point interaction layer. In this implementation, the reshape operation enables different types of base networks to act on the corresponding target dimensions.

[0010] In some implementations of the first aspect, when the audio source separation model is used to process the audio data to obtain the audio signal of the first target audio source, the method can include: when the first target audio source belongs to a preset audio source category, using a frequency band interaction layer to extract frequency band interaction features between multiple sub-bands from the first audio feature set to obtain a second audio feature set, using a frequency point interaction layer to extract frequency point interaction features within each sub-band in the second audio feature set to obtain a third audio feature set, and using the audio source separation model to separate the audio signal of the first target audio source from the third audio feature set; the first target audio source is any one of the at least one target audio source; or, when the first target audio source does not belong to the preset audio source category, using the audio source separation model to separate the audio signal of the first target audio source from the first audio feature set.

[0011] In this implementation, only the target audio source belonging to the preset audio source category has the frequency band interaction features and the frequency point interaction features extracted, and the target audio source not belonging to the preset audio source category does not have the frequency band interaction features and the frequency point interaction features extracted, so that the extraction of the frequency band interaction features and the frequency point interaction features for the target audio source not belonging to the preset audio source category can be adaptively omitted, and the processing delay is further shortened.

[0012] In one example, the preset audio source category is used to represent a full-band audio source category; or, the output value of the discriminative network corresponding to the audio source in the preset audio source category is less than or equal to a preset threshold. That is, the prior knowledge can be used to confirm whether the possible audio source belongs to the full-band audio source category or the non-full-band audio source category, so as to determine the preset audio source category in an enumerated manner; and the feature vector of the audio source can be input to the discriminative network, so that the discriminative network outputs a corresponding output value, and the preset threshold is set, and only the audio source category with the output value less than or equal to the preset threshold is the preset audio source category. In this example, an example of the setting manner of the preset audio source category is given, which can enumerate the audio source categories or determine them by discriminative scoring.

[0013] In some implementations of the first aspect, the frequency band interaction layer and / or the frequency point interaction layer are constructed using a two-dimensional convolutional base network, an attention network, or a fully connected (FC) layer.

[0014] It should be understood that although the attention network has a higher amount of calculation than the two-dimensional convolution in this implementation, since this is only an additional network layer outside the feature extraction network, the overall amount of calculation of the entire sound source separation model is still much smaller than the feature extraction network constructed entirely by the one-dimensional convolution network in the traditional scheme. This implementation gives an example of the frequency band interaction layer and the frequency point interaction layer. The frequency band interaction layer can be added to supplement the extraction of the frequency band interaction features between multiple sub-bands, and the frequency point interaction layer can be added to supplement the extraction of the frequency point interaction features in each sub-band. It should also be understood that since the feature extraction is performed by the frequency band interaction layer first, and then the feature extraction is performed by the frequency point interaction layer, although the frequency point interaction layer extracts the frequency point interaction features in each sub-band, these frequency point features have been affected by the previous feature extraction network and the frequency point interaction layer, and have generated new global features. Therefore, the frequency point interaction features extracted here are also global features.

[0015] In one example, the frequency band interaction layer is constructed by using a 1x1 convolution layer; and the frequency point interaction layer is constructed by using an FC layer.

[0016] In this example, the frequency band interaction layer is constructed by using a 1x1 convolution, which performs channel fusion, changes the proportion of each sub-band in the channel to fuse the channel features, and thus learns the nonlinear features between the sub-bands through the activation function. Moreover, the 1x1 convolution has a very small amount of calculation. The frequency point interaction layer is constructed by using an FC layer, which can learn the interaction information between all frequency points in each sub-band at each time step by virtue of the full connection characteristics of the FC layer, so that each frequency point refers to the information of other frequency points at the current time step, that is, learns the global frequency point interaction information (features) by virtue of the FC layer.

[0017] In a second aspect, a training method of a sound source separation model is provided. The training method comprises: obtaining training data, the training data comprising to-be-trained audio data and an audio signal label of at least one known sound source corresponding to the to-be-trained audio data, the to-be-trained audio data being mixed audio of multiple sound sources synthesized by an audio signal of the at least one known sound source and other audio signals; inputting the to-be-trained audio data into an initial sound source separation model, updating a weight parameter of the initial sound source separation model according to a difference between a predicted audio signal of the at least one known sound source output by the initial sound source separation model and the audio signal label of the at least one known sound source, thereby obtaining a trained sound source separation model; the initial sound source separation model comprising a feature extraction network, a frequency band interaction layer and a frequency point interaction layer; the feature extraction network being constructed by using a two-dimensional convolutional base network, and being configured to perform feature extraction on the to-be-trained audio data to obtain a first audio feature set; the frequency band interaction layer being configured to extract frequency band interaction features between multiple sub-bands corresponding to the to-be-trained audio data from the first audio feature set, thereby obtaining a second audio feature set; the frequency point interaction layer being configured to extract frequency point interaction features in each sub-band in the second audio feature set, thereby obtaining a third audio feature set; and the sound source separation model being further configured to separate the predicted audio signal of the at least one known sound source from the third audio feature set.

[0018] The trained sound source separation model obtained by the training method of the second aspect can be applied to the audio processing method of the first aspect. It should also be understood that the description of the sound source separation model in the first aspect can be applied to the second aspect. The difference between the two aspects is that the first aspect is the inference stage of the trained model used for inference, and the second aspect is the training stage of the model. For brevity, no further description is given.

[0019] In combination with the second aspect, in some implementations of the second aspect, the frequency band interaction layer and / or the frequency point interaction layer are constructed by using a two-dimensional convolutional base network, an attention mechanism network or a fully connected layer.

[0020] In combination with the second aspect, in some implementations of the second aspect, the multiple sub-bands are obtained by dividing the to-be-trained audio data along a frequency dimension, or the multiple sub-bands are obtained by dividing a first extraction result output by a first network layer along a frequency dimension, the first network layer being any layer of the feature extraction network.

[0021] In a third aspect, a sound source separation apparatus is provided. The apparatus comprises a unit composed of software and / or hardware for executing any method of the first aspect.

[0022] In a fourth aspect, a training apparatus is provided. The apparatus comprises a unit composed of software and / or hardware for executing any method of the second aspect.

[0023] In a fifth aspect, an electronic device is provided, the electronic device comprising: one or more processors, and a memory; the memory coupled with the one or more processors, the memory configured to store computer program code comprising computer instructions, the one or more processors configured to invoke the computer instructions to enable the electronic device to implement any of the methods of the first aspect and the second aspect.

[0024] In a sixth aspect, a chip system is provided, the chip system applied to an electronic device, the chip system comprising one or more processors, the one or more processors configured to invoke computer instructions to enable the electronic device to implement any of the methods of the first aspect and the second aspect.

[0025] Optionally, the chip system further comprises a memory, the memory being electrically connected with the processor.

[0026] Optionally, the chip system can further comprise a communication interface.

[0027] In a seventh aspect, a computer readable storage medium is provided, the computer readable storage medium comprising instructions that, when executed on an electronic device, enable the electronic device to implement any of the methods of the first aspect and the second aspect.

[0028] In an eighth aspect, a computer program product is provided, the computer program product comprising a computer program that, when executed by an electronic device, enables the electronic device to implement any of the methods of the first aspect and the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is a schematic diagram of an applicable scenario of an embodiment of the present application.

[0030] Figure 2 is a schematic diagram of an execution process of an audio processing method of an embodiment of the present application.

[0031] Figure 3 is a structural schematic diagram of a feature extraction network of an embodiment of the present application.

[0032] Figure 4 is a schematic flowchart of an audio processing method of an embodiment of the present application.

[0033] Figure 5 is a schematic diagram of an execution process of feature extraction using a source separation model of an embodiment of the present application.

[0034] Figure 6 is a schematic diagram of a processing process of a frequency band interaction layer of an embodiment of the present application.

[0035] Figure 7Fig. 2 is a schematic diagram of another processing procedure of a frequency band interaction layer according to an embodiment of the present application.

[0036] Figure 8 Fig. 3 is a schematic diagram of another processing procedure of a frequency band interaction layer according to an embodiment of the present application.

[0037] Figure 9 Fig. 4 is a schematic diagram of a processing procedure of a frequency point interaction layer according to an embodiment of the present application.

[0038] Figure 10 Fig. 5 is a schematic diagram of another processing procedure of a frequency point interaction layer according to an embodiment of the present application.

[0039] Figure 11 Fig. 6 is a schematic diagram of another processing procedure of a frequency point interaction layer according to an embodiment of the present application.

[0040] Figure 12 Fig. 7 is a schematic diagram of an execution procedure of an audio processing method according to an embodiment of the present application.

[0041] Figure 13 Fig. 8 is a comparison diagram of results of separating drums from a same piece of mixed audio using different schemes.

[0042] Figure 14 Fig. 9 is a schematic diagram of a non-full-band sound source according to an embodiment of the present application.

[0043] Figure 15 Fig. 10 is a schematic flowchart of another audio processing method according to an embodiment of the present application.

[0044] Figure 16 Fig. 11 is a schematic flowchart of a training method of a sound source separation model according to an embodiment of the present application.

[0045] Figure 17 Fig. 12 is a schematic diagram of a software architecture of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] The schemes of the embodiments of the present application will be described below with reference to the accompanying drawings.

[0047] Figure 1 Fig. 13 is a schematic diagram of an applicable scenario according to an embodiment of the present application. As shown in Fig. 13(a), assume that a piece of original music is a mixed audio, which includes audio of vocals, drums, bass, piano, guitar and other instruments. Any one of the instruments is a sound source, and the vocals is also a sound source. That is, the mixed audio of the original music contains audio signals from multiple sound sources. The so-called sound source separation is to separate audio signals of one or more sound sources in the mixed audio, for example, to separate audio signals of drums from the mixed audio. Figure 1 Fig. 13(b) shows a schematic diagram of a sound source separation model according to an embodiment of the present application.Figure 1 For example, in (a), the human voice and the drum sound are separated. The human voice has harmonic characteristics and belongs to a non-full-band source, while the drum sound is a full-band source. However, it should be understood that there is no limitation on the number of separated source categories and the types of sources. For example, the audio signal to be separated can also be the audio signal of a guitar. The guitar has both full-band characteristics and harmonic characteristics. The above source types can be separated from the mixed audio signal by using the solution of the present application, and other cases are not listed one by one.

[0048] For example, in (a), the human voice and the drum sound are separated. The human voice has harmonic characteristics and belongs to a non-full-band source, while the drum sound is a full-band source. However, it should be understood that there is no limitation on the number of separated source categories and the types of sources. For example, the audio signal to be separated can also be the audio signal of a guitar. The guitar has both full-band characteristics and harmonic characteristics. The above source types can be separated from the mixed audio signal by using the solution of the present application, and other cases are not listed one by one. Figure 1 For example, in (a), the human voice and the drum sound are separated. The human voice has harmonic characteristics and belongs to a non-full-band source, while the drum sound is a full-band source. However, it should be understood that there is no limitation on the number of separated source categories and the types of sources. For example, the audio signal to be separated can also be the audio signal of a guitar. The guitar has both full-band characteristics and harmonic characteristics. The above source types can be separated from the mixed audio signal by using the solution of the present application, and other cases are not listed one by one. Figure 1 For example, in (a), the human voice and the drum sound are separated. The human voice has harmonic characteristics and belongs to a non-full-band source, while the drum sound is a full-band source. However, it should be understood that there is no limitation on the number of separated source categories and the types of sources. For example, the audio signal to be separated can also be the audio signal of a guitar. The guitar has both full-band characteristics and harmonic characteristics. The above source types can be separated from the mixed audio signal by using the solution of the present application, and other cases are not listed one by one.

[0049] That is, the source separation can be to separate the audio signal of a specified source from the mixed audio signal composed of audio signals of multiple sources, or to separate the audio signal of a specified source from the mixed audio signal composed of multiple sources and further composed of other sources.

[0050] The separated audio signal can be further applied to other audio processing, Figure 1 For example, in (c), the separated audio signal is used for spatial audio. However, it should be understood that the separated audio signal can also be directly played or used for other purposes, and there is no limitation. As shown in (c), after the human voice and the drum sound are respectively amplified, the human voice is further separated into the center human voice and the background human voice (for example, the main singer and the backup singer in a song), and the drum sound is combined with the time domain data of the mixed audio to calculate the time domain data of other source audio, that is, to calculate the part of the time domain data of the mixed audio except the human voice and the drum sound. Figure 1 The time domain data of other audio is combined with the background human voice and the drum sound to perform sound field expansion, and then combined with the center human voice to obtain the processed audio. Such a processing process needs to be performed online during the playing of the song, that is, the source separation needs to be performed in real time and continuously to extract the audio signal of a specified source, and then the spatial audio (processed audio) is generated by using the flow shown in (c).

[0051] Figure 1

[0052] ​​Therefore, if the separation of the sound source is not fast and timely enough, a long time delay will be caused, the spatial audio cannot be generated in time, and the hearing experience of the user is affected. In addition, if the separation accuracy is too low, the timbre of the separated audio signal will be distorted, and the hearing experience of the user is affected.

[0053] To solve the problem, the present application proposes a new audio processing method, which can have relatively high separation accuracy and relatively short processing time delay. The details will be described below in combination with the drawings.

[0054] Figure 2 FIG. 1 is a schematic diagram of an execution process of an audio processing method according to an embodiment of the present application. As shown in the figure, after the to-be-processed audio data is input into a sound source separation model, the processed audio data can be output after being processed by the sound source separation model. The to-be-processed audio data is a mixed audio containing audio signals of multiple sound sources, and the processed audio data is an audio signal of one or more sound sources separated from the input mixed audio. Figure 2

[0055] The sound source separation model includes a feature extraction network, a frequency band interaction layer, and a frequency point interaction layer. The feature extraction network is used to extract high-order spectral features of the mixed audio, the frequency band interaction layer is used to extract global features between different frequency bands, and the frequency point interaction layer is used to extract global features between different frequency points in the same frequency band.

[0056] It should be noted that in the embodiments of the present application, the global mainly refers to the frequency dimension global, that is, the frequency dimension full coverage, or it can be understood as the full frequency band.

[0057] ​In the traditional scheme, the sound source separation model only includes a feature extraction network. When the feature extraction network is entirely constructed by using a one-dimensional convolutional network as a basic network, the one-dimensional convolutional network can cover the entire data range of the data to be processed, and thus has the ability to extract local features and global features. However, due to such full coverage, the operation amount of the one-dimensional convolutional network is large each time, and the operation amount of the entire processing process is too large, thereby causing a problem of too long time delay. When the feature extraction network is entirely constructed by using a two-dimensional convolutional network as a basic network, the operation of the two-dimensional convolutional network is based on the size of the convolution kernel each time, thereby greatly reducing the operation amount of the entire processing process. However, this also brings a new problem. Because the two-dimensional convolutional network is limited by the size of the convolution kernel, the operation each time is limited to the data range that can be covered by the size of the convolution kernel, and thus only local features can be extracted, but the extraction ability of global interaction features (mainly the full-band features in the frequency dimension in the present application) is poor, resulting in that the extracted global interaction features are not sufficient. Based on such insufficiently extracted features, the separated sound source signals are prone to spectral holes in the full-band, causing the timbre of the separated audio signals to change, and affecting the auditory sensation of the user. For example, for the drum sound of the above-mentioned percussive music, the features of the percussive music in the low frequency band are relatively more obvious, and if only local features are used for separation, the audio signals of other sound sources in the high frequency band are likely to be mistaken for the audio signals of the drum sound, or the audio signals of the drum sound in the high frequency band are likely to be mistaken for the audio signals of other sound sources. If the local features and the global features are combined, the features of the audio signals in the high frequency band can be separated by combining the features of the low frequency band drum sound of the percussive music, thereby improving the separation precision. The present application scheme is based on this, and thus the sound source separation model is designed to have a structure of a two-dimensional convolutional network + a one-dimensional convolutional network, so as to sufficiently extract local features and global features. In short, because the two-dimensional convolutional network is limited by the size of the convolution kernel, it is easier to extract local features, but it is difficult to extract global features, thereby causing the separation precision to be too low, spectral holes to occur, and the timbre of the sound source to change. In addition, because the human ear is more sensitive to sounds between 2 kilohertz (khz) and 5 khz, spectral holes in this interval are more likely to be felt by the user, so that the user feels that the timbre of the audio signal is distorted (for example, it may be drum sound, but it sounds like the sound of other musical instruments).

[0058] Furthermore, for sound sources with harmonic characteristics, such as human voice, global features are not obvious. Therefore, using only a two-dimensional convolutional network to extract local features can achieve relatively high accuracy. When using the scheme in this application, the feature extraction network is constructed using a two-dimensional convolutional network, which can fully extract local features, that is, fully extract the features of non-full-band sound sources with harmonic characteristics, such as human voice. The subsequent frequency band interaction layer and frequency point interaction layer supplement the extraction of global features after the local features have been fully extracted. Due to the lack of obvious global features, the frequency band interaction layer and frequency point interaction layer may extract relatively few global features. Based on the feature vectors extracted in this feature extraction process, the audio signal can still be separated with high accuracy after subsequent separation. In other words, for non-full-band sound sources, since the extraction of global features in this application is a supplementary extraction after the local features have been fully extracted, it does not affect the accuracy of local feature extraction. The only difference is that the number of extracted global features is relatively small due to the lack of obvious global features, thus ensuring separation accuracy and preventing a decrease in accuracy due to the extraction of global features.

[0059] Based on the above analysis, this application adds a frequency band interaction layer and a frequency point interaction layer to the feature extraction network built on a two-dimensional convolutional network as the base network. This compensates for the insufficient extraction of global interaction features by the two-dimensional convolutional network. Furthermore, since these are only additional network layers, rather than permeating the entire feature extraction network, the computational cost of this audio source separation model is still far less than that of the traditional feature extraction network built entirely on a one-dimensional convolutional network. In addition, to extract frequency band interaction features, the audio data to be processed needs to be segmented. The audio data or the extraction results generated during the feature extraction process are divided into multiple sub-bands in the frequency dimension, each of which is a sub-band in the full time dimension. For example, assuming the duration of the audio data to be processed is 10 seconds, then the duration of all sub-bands is 10 seconds. It should be understood that the above values ​​are for illustrative purposes and are not intended to limit the approach. Figure 13 As shown in (a), the spectrum of a standard drum sound is divided into multiple segments along the vertical axis, with each segment having the same length along the horizontal axis. The vertical axis represents the frequency dimension, and the horizontal axis represents the time dimension. Figure 13 As shown in (a), an example of dividing the frequency dimension into three sub-bands is also given. It can be seen that the time dimensions of sub-bands 1 to 3 are completely consistent, with the same start and end times, while the frequency dimensions cover different frequency bands respectively. It should be understood that this is only for reference. Figure 13 The sub-bands will be explained. Figure 13 The rest will be discussed in detail below, and will not be repeated here.

[0060] The two-dimensional convolution feature extraction network has good ability to extract local features, and thus can fully extract harmonic features of different scales of the audio data to be processed.

[0061] It should be understood that the dimensions of the sub-bands in the frequency dimension can be the same or different, that is, when dividing along the frequency dimension, it can be equal division or unequal division. Equal division can make it more convenient to integrate the processed data of all sub-bands in the subsequent integration, and unequal division is beneficial to flexible adjustment for different situations of different audio data to be processed, for example, a relatively important frequency band can be divided more finely, and a relatively unimportant frequency band can be divided more coarsely. For example, assuming that the characteristics of a certain sound source are mainly concentrated in the high frequency band, the high frequency band can be divided into N1 sub-bands along the frequency dimension, and the low frequency band can be divided into N2 sub-bands along the frequency dimension, and the dimension of the frequency dimension of each sub-band in the N1 sub-bands is greater than the dimension of the frequency dimension of each sub-band in the N2 sub-bands.

[0062] In some implementations, the sound source separation model can also only include the feature extraction network and the frequency point interaction layer. The reason is that the present application scheme will segment the audio data to be processed, so when the feature extraction network itself has a network layer capable of extracting the interaction features between the frequency bands, the interaction features between the frequency bands can be extracted under the premise of segmentation, so that the frequency band interaction layer can be removed. It should be understood that in this case, not canceling the frequency band interaction layer will only make the feature extraction more sufficient and will not bring any degradation. It should also be understood that in the traditional scheme, even if the feature extraction network itself has a network layer capable of extracting the interaction features between the frequency bands, because the audio data to be processed is not segmented, the interaction features between the frequency bands cannot be extracted. In addition, in order to solve the problem of poor extraction ability of the traditional two-dimensional convolution-based feature extraction network, the frequency band interaction layer and the frequency point interaction layer are added to supplement the features that the traditional two-dimensional convolution-based feature extraction network cannot extract. On this basis, it is found in the test process that some special feature extraction networks still have good separation effect after removing the frequency band interaction layer, and then analyzing this phenomenon, it is found that these special feature extraction networks have a common characteristic that they have a network layer capable of extracting the interaction features between the frequency bands. Based on the principle analysis, because the audio data to be processed will be segmented in the early stage of the present application, such special feature extraction networks can extract the features between the frequency bands, so the feature extraction effect is still relatively good after removing the frequency band interaction layer. Therefore, the scheme of this feature extraction network which extracts both audio features and frequency band interaction features is still different from the traditional scheme which only extracts audio features by the feature extraction network.

[0063] According to the similar logic, in other implementations, when the feature extraction network is capable of extracting global frequency bin interaction features, the sound source separation model can also only include the feature extraction network and the frequency band interaction layer.

[0064] Based on the performance analysis of the feature extraction network, the above-mentioned way of omitting part of the network layers in the sound source separation model can simplify the model, further reduce the amount of calculation, and shorten the time delay without affecting the sufficiency of the extracted features.

[0065] The to-be-processed audio data can be frequency domain data obtained by performing Fourier transform on time domain audio data.

[0066] The size of the one-dimensional convolution kernel is proportional to the number of features. In a sound source separation task, the number of features of the one-dimensional convolution is usually 1024 resolution, and the resolution here refers to the number of fast Fourier transform (FFT) points. The size of the two-dimensional convolution kernel is independent of the number of features. Taking a convolution kernel with a size of 3x3 as an example, when the number of features of the two-dimensional convolution is equal to that of the one-dimensional convolution, the amount of calculation of the two-dimensional convolution is about 3 / F times that of the one-dimensional convolution. It can be seen that the amount of calculation of the two-dimensional convolution is much smaller than that of the one-dimensional convolution. Therefore, although the frequency band interaction layer and the frequency bin interaction layer are added to the two-dimensional convolution feature extraction network in the present application, the amount of calculation of the sound source separation model is still much smaller than that of the one-dimensional convolution feature extraction network. It is a scheme for effectively improving the extraction capability of the two-dimensional convolution feature extraction network without increasing too much amount of calculation, and can also be understood as a scheme for greatly reducing the amount of calculation while ensuring the feature extraction capability comparable to that of the one-dimensional convolution feature extraction network.

[0067] Figure 3 is a structural schematic diagram of a feature extraction network according to an embodiment of the present application. Figure 3 is an example structure of the feature extraction network according to the present application, which is called U-Net structure. The U-Net structure includes a contraction path and an expansion path. The contraction path includes feature extraction and down sampling, which is equivalent to distilling and refining the features. The expansion path includes feature map up sampling and recovery. Here, the U-Net structure includes three times of down sampling and three times of up sampling as an example. As shown in Figure 3As shown, the U-Net structure will pass the input data through three times of down-sampling to enter the center, and then output the processed data after three times of up-sampling. Before each down-sampling or up-sampling, the processing of the basic network is performed once. Taking a two-dimensional convolutional basic network as an example, that is, in the U-Net structure, it includes: the two-dimensional convolutional basic network, the down-sampling layer, the two-dimensional convolutional basic network, the down-sampling layer, the two-dimensional convolutional basic network, the down-sampling layer, the two-dimensional convolutional basic network, the center layer, the two-dimensional convolutional basic network, the up-sampling layer, the two-dimensional convolutional basic network, the up-sampling layer, the two-dimensional convolutional basic network, the up-sampling layer, and the two-dimensional convolutional basic network.

[0068] In the traditional scheme, the U-Net structure feature extraction network can be constructed based on a one-dimensional convolutional basic network. As can be seen, the one-dimensional convolutional basic network needs to be repeatedly used for operation multiple times, which leads to a further increase in the amount of calculation, and the overall amount of calculation is larger. Therefore, the U-Net structure feature extraction network is constructed based on a two-dimensional convolutional basic network in the present application, so that the amount of calculation is much smaller than the U-Net structure feature extraction network constructed based on the one-dimensional convolutional basic network. The weakening of the extraction ability of the global interaction feature caused thereby is overcome by increasing the frequency band interaction layer and the frequency point interaction layer, so as to reduce the amount of calculation and shorten the processing time delay of the sound source separation under the premise of ensuring the separation precision.

[0069] Figure 4 is a schematic flow chart of an audio processing method according to an embodiment of the present application. The following describes each step shown in Figure 4 .

[0070] S401, obtaining to-be-processed audio data.

[0071] The to-be-processed audio data includes audio signals of multiple sound sources. The sound source can be, for example, a certain musical instrument or human voice, etc. The sound source can also be referred to as a sound source, can be referred to as a sound source, and can also be referred to as a sound type or a sound category.

[0072] The to-be-processed audio data can be frequency domain data obtained by performing Fourier transform on a time domain audio signal.

[0073] The manner of obtaining the to-be-processed audio data can be an online acquisition manner, for example, the frequency domain to-be-processed audio data can be obtained by performing Fourier transform on the audio signal collected in real time by a microphone or a microphone, etc. The manner of obtaining the to-be-processed audio data can also be an offline acquisition manner, for example, the to-be-processed audio data can be read from a storage module, or downloaded from a network through a communication interface, etc. There is no limitation.

[0074] S402, processing the to-be-processed audio data by using the sound source separation model to obtain an audio signal of at least one target sound source.

[0075] The at least one target sound source is at least one sound source in the multiple sound sources. The target sound source can be understood as a sound source that is desired to be separated out, for example Figure 1 Taking the example of desiring to separate out human voice and drum sound, the human voice and the drum sound can be an example of the at least one target sound source, and this example can also be an example in which the at least one target sound source includes multiple target sound sources, and the multiple target sound sources are respectively the human voice and the drum sound. It should be understood that the at least one target sound source specifically includes which sound source or sound sources, and the number of sound sources included is not limited. It should also be understood that the number of the at least one target sound source needs to be less than the number of the multiple sound sources of the to-be-processed audio data, and if the numbers are equal, it is equivalent to not performing sound source separation.

[0076] In an implementation manner, the sound source separation model includes a feature extraction network, a frequency band interaction layer, and a frequency point interaction layer; the feature extraction network is constructed by using a two-dimensional convolutional base network, is used for feature extraction on the to-be-processed audio data, and obtains a first audio feature set; the frequency band interaction layer is used for extracting frequency band interaction features between multiple sub-bands corresponding to the to-be-processed audio data from the first audio feature set, thereby obtaining a second audio feature set; the frequency point interaction layer is used for extracting global frequency point interaction features in each sub-band in the second audio feature set, thereby obtaining a third audio feature set; and the sound source separation model is further used for separating out the audio signal of the at least one target sound source from the third audio feature set. Figure 2 and Figure 5 The sound source separation model in the foregoing can be an example of the sound source separation model.

[0077] The multiple sub-bands are sub-bands in a full time dimension, that is, the time dimensions of the multiple sub-bands are the same. For related content about division along the frequency dimension, reference can be made to the related description in the foregoing, and details are not described herein again for the sake of brevity.

[0078] In the scheme of the present application, by dividing the sub-bands, the frequency band interaction layer in the sound source separation model can extract the frequency band interaction features between the multiple sub-bands.

[0079] In an implementation, the plurality of sub-bands are obtained by dividing the to-be-processed audio data along a frequency dimension, or the plurality of sub-bands are obtained by dividing the first extracted result output by the first network layer along a frequency dimension, the first network layer being any one of the layers of the feature extraction network. In this implementation, the plurality of sub-bands corresponding to the to-be-processed audio data can be divided before entering the feature extraction network, during the feature extraction by the feature extraction network, or after the feature extraction by the feature extraction network and before inputting the extracted feature to the frequency band interaction layer, so that the adjustment can be made flexibly according to the actual application scenario.

[0080] As described above, for some special feature extraction networks, in combination with the segmentation operation (operation of dividing the plurality of sub-bands), some traditional feature extraction networks can also have the ability to extract global interaction features, so for this kind of feature extraction networks, the network structure can be appropriately simplified, so as to further reduce the calculation amount and shorten the time delay without affecting the feature extraction effect.

[0081] In an implementation, in a case where the feature extraction network has the ability to extract the frequency band interaction features between the plurality of sub-bands so that the first audio feature set contains the frequency band interaction features between the plurality of sub-bands, the sound source separation model does not include the frequency band interaction layer; and / or in a case where the feature extraction network has the ability to extract the global frequency point interaction features of each frequency point corresponding to the to-be-processed audio data so that the first audio feature set contains the global frequency point interaction features of each frequency point, the sound source separation model does not include the frequency point interaction layer.

[0082] In one example, the sound source separation model includes a feature extraction network and a frequency bin interaction layer; the feature extraction network is constructed by using a two-dimensional convolutional base network, and the feature extraction network includes a 1x1 convolutional layer, which is used to extract features of a plurality of sub-bands input into the feature extraction network to obtain a first audio feature set; the frequency bin interaction layer is used to extract global frequency bin interaction features in each sub-band in the first audio feature set to obtain a third audio feature set; the sound source separation model is further used to separate an audio signal of at least one target sound source from the third audio feature set. In this implementation manner, since the feature extraction network itself contains a 1x1 two-dimensional convolutional layer, and the input into the feature extraction network is a segmented sub-band, the 1x1 two-dimensional convolutional layer in the feature extraction network can already play the role of the frequency bin interaction layer and can extract frequency bin interaction features between a plurality of sub-bands, so in this case, the frequency bin interaction layer in the sound source separation model can be omitted, and therefore the implementation manner can only include the feature extraction network and the frequency bin interaction layer. This implementation manner can further simplify the model structure when the feature extraction network contains a 1x1 convolutional layer, further reduce the calculation amount, and thus shorten the processing delay of sound source separation. It should be understood that, assuming that a certain feature extraction network has a plurality of 1x1 convolutional layers, and the sub-bands are divided on the basis of the output result of the network layer at the front of the feature extraction network, the plurality of sub-bands can still be extracted to frequency bin interaction features under the action of the subsequent 1x1 convolutional layer. In combination with the sound source separation model in Figure 12 , Figure 12 , the feature extraction network ( Figure 12 taking U-Net as an example) is divided into sub-bands before the feature extraction network U-Net. If the U-Net itself contains a 1x1 convolution, then during the feature extraction of the U-Net, both local features and frequency bin interaction features can be extracted, and therefore, under such a premise, the frequency bin interaction layer can be removed, that is, the sound source separation model in Figure 12 only includes the U-Net and the frequency bin interaction layer. It should be understood that this is only to illustrate what kind of situation can simplify the frequency bin interaction layer by means of Figure 12 , and other contents of Figure 12 will be described in detail below, and will not be described here.

[0083] In an implementation, the frequency band interaction layer and / or the frequency point interaction layer is constructed using a two-dimensional convolutional base network, an attention network, or a fully connected (FC) layer. It should be understood that although the attention network has a higher calculation amount than the two-dimensional convolution in this implementation, since this is only an additional network layer outside the feature extraction network, the overall calculation amount of the entire sound source separation model is still much smaller than the feature extraction network constructed entirely by a one-dimensional convolutional network in the traditional scheme. This implementation gives an example of the frequency band interaction layer and the frequency point interaction layer. The frequency band interaction layer is added to supplement the extraction of frequency band interaction features between multiple sub-bands, and the frequency point interaction layer is added to supplement the extraction of frequency point interaction features in each sub-band. It should also be understood that since the feature extraction is performed by the frequency band interaction layer first, and then the feature extraction is performed by the frequency point interaction layer, although the frequency point interaction layer extracts frequency point interaction features in each sub-band, these frequency point features have been affected by the previous feature extraction network and the frequency point interaction layer, and generate new global features. Therefore, the frequency point interaction features extracted here are also global frequency point interaction features.

[0084] Figures 6-8 is an example of a frequency band interaction layer, Figures 9-11 is an example of a frequency point interaction layer, and the details are described below. For brevity, they will not be expanded here. The frequency band interaction features extracted by the frequency band interaction layer can be understood as follows: the calculation process of the frequency band interaction layer combines the data of other sub-bands in the calculation of each sub-band, and the features obtained by considering other sub-bands are called frequency band interaction features, and the frequency band interaction layer is used to extract such features. The frequency point interaction layer extracts global frequency point interaction features, which can be understood as follows: the calculation process of the frequency point interaction layer combines the data of other frequency points in the calculation of each frequency point in each sub-band. Since the data of each frequency point in this sub-band has combined the data of other sub-bands after the action of the previous frequency band interaction layer, the data of each frequency point in this sub-band combines the data of other sub-bands, so that the frequency point interaction layer extracts global frequency point interaction features. The frequency point interaction features obtained by considering the global frequency points are called global frequency point interaction features.

[0085] In one example, the frequency band interaction layer is constructed by using a 1x1 convolution layer, and the frequency point interaction layer is constructed by using an FC layer. In this example, the frequency band interaction layer is constructed by using a 1x1 convolution, channel fusion is performed, channel feature fusion is performed by changing the proportion of each sub-band in the channel, thereby learning the nonlinear features between the sub-bands through an activation function, and the calculation amount of the 1x1 convolution is very small; the frequency point interaction layer is constructed by using an FC layer, and the interaction information between all frequency points in each sub-band at each time step can be learned by using the full connection characteristics of the FC layer, thereby enabling each frequency point to refer to the information of other frequency points at the current time step, that is, learning the global frequency point interaction information.

[0086] It should be understood that the time step refers to the time unit of the frequency spectrum data of the audio.

[0087] It should be noted that when different types of basic networks are used to construct the frequency band interaction layer and / or the frequency point interaction layer, because the dimension order acted on by different types of basic networks is different, and the dimensions that need to be acted on also differ, reshape operation may be needed to enable different types of basic networks to act on the target dimension. In the present application scheme, the target dimension of the frequency band interaction layer is the channel dimension, and the target dimension of the frequency point interaction layer is the frequency dimension.

[0088] In one implementation, the method further includes: when the target dimension of the frequency band interaction layer and / or the frequency point interaction layer does not match the action dimension of the basic network used to construct the frequency band interaction layer and / or the frequency point interaction layer, performing reshape operation to match the target dimension and the action dimension.

[0089] For example, the attention network acts on the last dimension, and the frequency band interaction layer needs to act on the channel dimension (which is an example of the target dimension), and the channel dimension is not in the last dimension of the initial dimension of the initial input data of the frequency band interaction layer, so the initial dimension needs to be reshaped first to adjust the channel dimension to the last dimension, so that the attention network can act on the channel dimension. Other cases can be referred to Figures 6-11 The related description will not be repeated.

[0090] It should also be understood that the action dimension of the 1x1 convolution layer is the second dimension, and the action dimensions of the attention network and the FC layer are the last dimension. Therefore, in the frequency band interaction layer, when the 1x1 convolution layer is used, the action dimension of the 1x1 convolution layer is the second dimension, which is exactly the target dimension (channel dimension) of the frequency band interaction layer, and the two are consistent, so the reshape operation is not needed. In the frequency band interaction layer, when the attention network or the FC layer is used, the action dimensions of the two are the last dimension, which is not the target dimension (channel dimension) of the frequency band interaction layer, but the frequency dimension, and the two are not consistent, so the reshape operation is needed to adjust the channel dimension to the last dimension before processing by the frequency band interaction layer, and then restore the dimension after processing by the reshape operation. In the frequency point interaction layer, when the 1x1 convolution layer is used, the action dimension of the 1x1 convolution layer is the second dimension, which is not the target dimension (frequency dimension) of the frequency point interaction layer, and the two are not consistent, so the reshape operation is needed to adjust the frequency dimension to the second dimension before processing by the frequency point interaction layer, and then restore the dimension after processing by the reshape operation. In the frequency point interaction layer, when the attention network or the FC layer is used, the action dimensions of the two are the last dimension, which is exactly the target dimension (frequency dimension) of the frequency band interaction layer, and the two are consistent, so the reshape operation is not needed. It can also be seen that because the 1x1 convolution acts on the second dimension, it matches the action dimension of the feature extraction network, so when the feature extraction network contains a 1x1 convolution, the frequency band interaction layer can be omitted.

[0091] In actual application, some sound sources have the hollow characteristic by themselves, that is, such sound sources have harmonic characteristics, so that such sound sources become non-full-band sound sources. For example, human voice is a sound source with harmonic characteristics. It can be seen from the schematic diagram of the non-full-band sound source of Figure 14 This type of non-full-band sound source has a hollow in its own spectrum, for example, human voice is a non-full-band sound source with harmonic characteristics, so it is not necessary to further extract the frequency band interaction features and frequency point interaction features of the sound source in the to-be-processed audio data of such sound source. Because the global features of such sound source are not obvious, in terms of the spectrum, it is like Figure 14 a piece of discontinuous line segment in the frequency dimension, and there is almost no vertical line along the vertical axis (frequency dimension) from the beginning to the end like the drum sound (an example of a full-band sound source) in Figure 13 Therefore, it is not necessary to make such a line segment more coherent by analyzing the continuity of the line segment, that is, it is not necessary to make the frequency dimension global features more sufficient by extracting the frequency band interaction features and the global frequency point interaction features, which is embodied in the spectrum, that is, it is not necessary to make the line in the frequency dimension more coherent and complete, because the non-full-band sound source refers to a sound source that does not occupy the entire dimension range in the frequency dimension.

[0092] In an implementation manner, the step S402 can comprise: in a case where the first target sound source belongs to a preset sound source category, extracting, by using a frequency band interaction layer, frequency band interaction features between multiple sub frequency bands from the first audio feature set, so as to obtain a second audio feature set, extracting, by using a frequency point interaction layer, global frequency point interaction features in each sub frequency band in the second audio feature set, so as to obtain a third audio feature set, and separating, by using the sound source separation model, an audio signal of the first target sound source from the third audio feature set; the first target sound source is any one of at least one target sound source; or, in a case where the first target sound source does not belong to the preset sound source category, separating, by using the sound source separation model, the audio signal of the first target sound source from the first audio feature set. In this implementation manner, only the target sound source belonging to the preset sound source category will extract the frequency band interaction features and the global frequency point interaction features, and the target sound source not belonging to the preset sound source category will not extract the frequency band interaction features and the global frequency point interaction features, so that the extraction of the frequency band interaction features and the frequency point interaction features for the target sound source not belonging to the preset sound source category can be adaptively saved, and the processing time delay is further shortened. Figure 12 is an example of this implementation manner, which will not be repeated here for brevity.

[0093] In an example, the preset sound source category is used to represent a full-band sound source category; or, the output value of the discrimination network corresponding to the sound source in the preset sound source category is less than or equal to a preset threshold. That is, the prior knowledge can be used to confirm whether the possible sound source belongs to the full-band sound source category or the non-full-band sound source category, so as to determine the preset sound source category in an enumerated manner; or, the feature vector of the sound source can be input to the discrimination network, so that the discrimination network outputs a corresponding output value, and the preset threshold is set, and only the sound source category with the output value less than or equal to the preset threshold is the preset sound source category. In this example, an example of the setting manner of the preset sound source category is given, which can enumerate the sound source categories or determine them by discrimination scoring. Figure 15 is an example of this example, which will not be repeated here for brevity.

[0094] Figure 4 The method shown in the figure adds a frequency band interaction layer and a frequency point interaction layer on the basis of the two-dimensional convolution feature extraction network, extracts the frequency band interaction features between multiple sub frequency bands and the frequency point interaction features in each sub frequency band, respectively, so as to compensate for the defect of the insufficient extraction ability of the two-dimensional convolution feature extraction network for the global interaction features while retaining the advantage of small calculation amount of the two-dimensional convolution feature extraction network, so that the whole sound source separation process can achieve the effect of relatively high separation precision and relatively short processing time delay.

[0095] Figure 5 is a schematic diagram of an execution process of feature extraction by using a sound source separation model according to an embodiment of the present application.Figure 5 This is an example of the execution process for obtaining the third audio feature set in step S402. The audio source separation model includes a two-dimensional convolutional feature extraction network, a frequency band interaction layer, and a frequency point interaction layer. For example... Figure 5 As shown, a two-dimensional convolutional feature extraction network can be used to extract the spectral features of the audio data to be processed; these can be high-order spectral features. It should be understood that a two-dimensional convolutional feature extraction network can fully extract the harmonic features at different scales of the audio data to be processed, that is, extract local features.

[0096] Then, higher-order spectral features or the spectrum can be input into the frequency band interaction layer. However, before inputting them into the frequency band interaction layer, the spectrum or higher-order spectral features of the audio data to be processed need to be divided into multiple sub-bands along the frequency dimension. Here, we take dividing it into sub-bands (band 1) to sub-band N as an example, where N is an integer greater than 1. It can be seen that... Figure 5 The diagram shows that you can either use a feature extraction network to perform preliminary two-dimensional feature extraction before dividing the frequency bands, or divide the original spectrum (i.e., the spectrum of the audio data to be processed) into sub-bands, or divide the frequency bands into intermediate results (feature vectors) output by any layer during feature extraction by the feature extraction network, as long as the division of sub-bands is completed before input to the frequency band interaction layer.

[0097] After entering the frequency band interaction layer, it extracts the frequency band interaction features between multiple sub-bands, thus obtaining the frequency band interaction results output by N channels (i.e., channel 1 to channel N). Extracting global interaction features between different sub-bands avoids the fragmentation of information between different sub-bands. Combined with... Figure 13 The three sub-bands shown in (a) are designed to prevent feature extraction from being isolated from each other, which would result in relatively sufficient extraction for one sub-band but inability to combine data from different sub-bands, especially with obvious fragmentation near the boundaries of different sub-bands. It should be understood that this introduces... Figure 13 This is to illustrate that information fragmentation will result in a clear sense of boundary between sub-bands. Other details will be explained below and will not be repeated here.

[0098] After entering the frequency interaction layer, for any input channel i, the global frequency interaction features between frequency points in the sub-band corresponding to channel i can be extracted. The dimension of the frequency interaction result is the same as the dimension of a single sub-band, thus ensuring that the details of the global features are not lost.

[0099] Figure 5is an example of obtaining the third audio feature set in step S402, first extracting local features using a feature extraction network (two-dimensional convolution), then dividing the local feature set (first feature set) into N sub-bands along the frequency dimension, and inputting the N sub-bands into the frequency band interaction layer. Each sub-band in the frequency band interaction layer corresponds to a channel, so as to extract the interaction features between different sub-bands, and then combine the frequency band interaction features (frequency band interaction results in Figure 5 ) with the previously obtained local feature set into a new feature set (the first feature set in the above), and input them into the frequency point interaction layer again in the same sub-band division manner. The frequency point interaction layer performs extraction of global frequency point interaction features (frequency point interaction results in Figure 5 ) in each channel (since the channel corresponds to the sub-band, here it is equivalent to each sub-band), and combines the obtained frequency point interaction features into the second feature set to obtain the third feature set described above. Then, the sound source separation model can perform sound source separation based on the third feature set through channel selection to obtain the audio signal of the target sound source.

[0100] Figure 5 The frequency band interaction layer in Figures 6-8 may adopt any one of Figure 5 , or be constructed by using other suitable network layers. Figures 9-11 The frequency point interaction layer in may adopt any one of

[0101] , or be constructed by using other suitable network layers. It will not be listed one by one. Figure 6 Figure 6 Figure 6 is a schematic diagram of the processing process of a frequency band interaction layer in an embodiment of the present application. Taking the frequency band interaction layer constructed based on 1x1 two-dimensional convolution as an example. With the help of 1x1 convolution, the channel fusion feature can be performed, and the proportion of each sub-band in the output channel can be adaptively selected, so as to learn the interaction features between the sub-bands. As shown in

[0102] , the frequency band features (sub-band 1-sub-band N) are subjected to 1x1 convolution, and the activation function relu(x)=max(0,x) is used, and the frequency band interaction results are output. The output frequency band interaction results can satisfy the formula below. wherein n∈[1,N] is used to represent the channel number, N represents the total number of channels, the frequency band interaction layer includes N convolution kernels, the size of each convolution kernel is 1x1, the dimension of each convolution kernel is Nx1x1, and there are N output channels. relu(x) is an activation function, w and b represent the weight vector and the bias vector of the convolution kernel respectively. represents the element of the i-th row and the j-th column of the n-th channel of the frequency band interaction layer output. It should be understood that i and j are random values of rows and columns, and the value range depends on the size of each sub-band. In addition, the so-called training model process refers to the process of updating the model parameters, that is, updating the parameter values in the weight vector and the bias vector.

[0103] Figure 7 is a schematic diagram of another processing process of a frequency band interaction layer of an embodiment of the present application. Figure 7 Taking the construction of the frequency band interaction layer based on the attention network as an example. The characteristics of the attention network make it easier to fully learn the interaction features between the sub-bands. Although the calculation amount is relatively large, since it is only an additional network layer outside the feature extraction network, the increase in the calculation amount of the entire sound source separation model is not obvious, and the overall calculation amount of the sound source separation model is still much smaller than that of the traditional scheme of constructing the feature extraction network completely using the attention network. In addition, since the last dimension of the dimension of the input sub-band is the frequency dimension, and the extraction of the frequency band interaction feature needs to be extracted in the channel dimension, the dimension of the sub-band needs to be reshaped first before the extraction of the frequency band interaction feature, and after the frequency band interaction feature is obtained, the dimension of the sub-band is restored by reshaping again. As shown in Figure 7 , the initial dimension of the sub-band is (b, c*N, t, f) before inputting into the frequency band interaction layer, where b represents the number of samples input into the frequency band interaction layer, or the batch size, t represents the number of time steps, c represents the number of channels before dividing the sub-band, N represents the number of sub-bands after dividing the sub-band, Q, K and V are all calculation parameters under the attention mechanism, representing query, key and value respectively, and the weight dimension is (c*N, c*N), so the attention network needs to be applied to the last dimension, that is, the channel dimension. Therefore, the initial dimension (b, c*N, t, f) needs to be reshaped into (b, t, f, c*N), but it should be understood that here the reshape only needs to ensure that the last dimension is the channel dimension, and there is no limitation on how to set the front several dimensions, for example, it can also be reshaped into (b, f, t, c*N).

[0104] As shown in Figure 7 , Q=input*W Q +b Q , K=input*W K +b K , V=input*W V +b VThe input represents data input to the attention layer. The output data can satisfy the following formula.

[0105] Wherein, the output represents output data, T is a transpose operator, and softmax is a regression function.

[0106] Figure 7 The self-attention mechanism in the attention mechanism is adopted.

[0107] As shown in Figure 7 , after obtaining the data output by the attention layer, reshape is performed again to restore the same initial dimension.

[0108] It should be noted that in the embodiments of the present application, the first reshape is to enable the selected base network to act on the target dimension required by the network layer, and the second reshape is to restore the dimension for better connection with the subsequent network structure. For example, if the frequency band interaction layer selects the base network shown in Figure 7 , reshape is performed to enable the network to act on the channel dimension, but in order to better connect with the subsequent network, reshape is performed again after the layer to restore the dimension.

[0109] Figure 8 is a schematic diagram of another processing process of the frequency band interaction layer in the embodiments of the present application. Figure 8 Taking the frequency band interaction layer based on FC as an example, the construction based on FC layer is similar to the construction based on 1x1 convolution. However, since the last dimension of the dimension of the input sub-band is the frequency dimension, and the extraction of the frequency band interaction feature needs to be extracted in the channel dimension, the sub-band dimension needs to be reshaped first before the extraction of the frequency band interaction feature in this mode, and after the frequency band interaction feature is obtained, the dimension of the sub-band is restored by reshaping again. As shown in Figure 8 , the initial dimension of the sub-band is (b, c*N, t, f) before the sub-band is input to the frequency band interaction layer, and the related explanation is referred to Figure 7 , which will not be repeated. Therefore, the initial dimension (b, c*N, t, f) needs to be reshaped to (b, t, f, c*N), but it should be understood that the reshape here only needs to ensure that the last dimension is the channel dimension, and there is no limitation on how to set the front several dimensions, for example, it can also be reshaped to (b, f, t, c*N).

[0110] As shown in Figure 8As shown, the sub-band features after reshaping, after being processed in the channel dimension by the FC layer, output data can satisfy the formula output = relu(input * W). FC +b FC ), where output represents the output data, input represents the data input to the FC layer, and W FC Let b be a two-dimensional matrix of dimension (c*N, c*N). FC The dimension is c*N.

[0111] like Figure 8 As shown, after obtaining the data output by the FC layer, it is reshaped again to restore it to the same dimension as the initial dimension.

[0112] Figures 6 to 8 Examples of three frequency band interaction layers are given. Constructing frequency band interaction layers based on 1x1 convolutions or fully connected (FC) layers has the advantages of low computational cost and fast operation. Constructing frequency band interaction layers based on attention mechanisms has a relatively high computational cost, but it can learn more comprehensive frequency band interaction features. Although the computational cost is relatively high, since only the frequency band interaction layer has a high computational cost, the overall computational cost of the entire sound source separation model is still much lower than that of the traditional approach of constructing the entire model based on one-dimensional convolutional networks.

[0113] Figure 9 This is a schematic diagram illustrating the processing procedure of a frequency point interaction layer according to an embodiment of this application. The frequency point interaction layer needs to learn global interaction information among frequency points, that is, global dependency information; therefore, it can be constructed using an FC (Functionally Connected) layer. Figure 9 This is an example of using a fully connected (FC) layer to construct a frequency interaction layer. Within each time step of a sub-band, the FC layer is applied to calculate the output. In the output, each frequency point references information from other frequencies at the current time step through the FC layer, thus learning global interaction information. For example... Figure 9 The arbitrary sub-band (sub-band n) shown is passed through an FC layer, and the activation function relu(x) = max(0,x) is used to output the frequency interaction results within that sub-band. The output frequency interaction results can satisfy the formula below.

[0114] Where n∈[1,N] represents the sub-band number, N represents the total number of sub-bands, and F represents the feature dimension, that is, the total dimension of the frequency point features. relu(x) is the activation function, and w and b represent the weight vector and bias vector of the FC layer, respectively. ik Let w represent the element in the i-th row and j-th column of the weight matrix. The row number of w depends only on the input F, therefore w corresponds to i. This represents the element in the k-th row and j-th column of the n-th sub-band of the input. represents the element in the i-th row and the j-th column of the n-th channel of the frequency band interaction layer output.

[0115] Figure 10 is a schematic diagram of another processing process of a frequency point interaction layer of an embodiment of the present application. The frequency point interaction layer can also be constructed using an attention network, Figure 10 that is, an example of constructing a frequency point interaction layer using an attention network. Figure 10 The introduction of related parameters can refer to Figure 7 and will not be repeated. Since the frequency point interaction layer is to extract features in the frequency dimension, reshape operation is not required in the processing process of the frequency point interaction layer in Figure 10 Figure 10 The weight dimensions of Q, K and V in are (f, f), therefore, the attention will be performed on the last dimension of the input data (b, c*N, t, f), that is, the frequency, and therefore reshape is not required. As shown in Figure 10 the output data can satisfy the formula The introduction of related parameters is described above and will not be repeated.

[0116] Figure 11 is a schematic diagram of another processing process of a frequency point interaction layer of an embodiment of the present application. The frequency point interaction layer can also be constructed using a 1x1 convolution layer, Figure 11 that is, an example of constructing a frequency point interaction layer using a 1x1 convolution layer. This implementation is similar to the construction based on the FC layer. However, since the last dimension of the dimension of the input sub-band is the frequency dimension, and the 1x1 convolution will act on the second dimension when extracting the frequency point interaction feature, and since the second dimension of the initial dimension of the sub-band (b, c*N, t, f) is the channel dimension, not the frequency dimension, therefore, in this mode, the initial dimension of the sub-band needs to be reshaped first before the frequency point interaction feature is extracted, and after the frequency point interaction feature is obtained, the dimension of the sub-band is reshaped again to restore the dimension of the sub-band. As shown in Figure 11 before the sub-band is input into the frequency band interaction layer, the initial dimension of the sub-band (b, c*N, t, f) is reshaped into (b, f, t, c*N) and then 1x1 convolution is performed. It should be understood that when reshaping, only the second dimension is required to be the frequency dimension f.

[0117] As shown in Figure 11 the data output by the reshaped sub-band feature after the processing of the 1x1 convolution layer can satisfy the formula output = relu(input*W conv +b conv ), wherein output represents the output data, input represents the data input into the 1x1 convolution layer, W conv ​and b conv These are the weight vector and the bias vector, respectively.

[0118] like Figure 11 As shown, after obtaining the data output from the 1x1 convolutional layer, it is reshaped again to restore it to the same dimension as the initial layer.

[0119] Figures 9 to 11 Examples of three frequency interaction layers are given. Constructing frequency interaction layers based on 1x1 convolutions or fully connected (FC) layers has the advantages of low computational cost and fast operation. Constructing frequency interaction layers based on attention mechanisms has a relatively high computational cost, but it can learn more comprehensive global frequency interaction features. Although the computational cost is relatively high, since only the frequency interaction layer has a high computational cost, the overall computational cost of the entire sound source separation model is still much lower than that of the traditional approach that is entirely based on one-dimensional convolutional networks.

[0120] Figure 12 This is a schematic diagram illustrating the execution process of an audio processing method according to an embodiment of this application. Figure 12 yes Figure 4 An example of the execution process of the method shown. Figure 12 As shown, the input audio data is a mixed audio signal containing multiple sound sources. After Fourier transform, a 2048-point FFT spectrum is obtained (an example of the audio data to be processed). The 2048-point FFT spectrum is divided into four sub-bands along the frequency dimension (an example of multiple sub-bands), sub-band 1 to sub-band 4, each sub-band having a 256-point FFT. The four sub-bands are then input into a U-Net structure built on a two-dimensional convolutional network (an example of a feature extraction network). The output data (an example of the first audio feature set) is used to determine whether the target sound source belongs to a preset category (an example of a preset sound source category, such as a full-band sound source). If the target sound source belongs to the preset category, the U-Net is further checked for the presence of a 1x1 convolutional layer; otherwise, the output data is directly input into the channel selection module (e.g., a 3x3 convolutional layer with two channels). When determining whether U-Net contains a 1x1 convolutional layer, if the result is "yes," it means that U-Net already has the ability to extract features from the band interaction layer, therefore the band interaction layer is skipped. Figure 12 Taking the frequency band interaction layer constructed using a 1x1 convolutional layer as an example, the data is further input into the frequency point interaction layer (…). Figure 12For example, when the judgment result is "yes", it means that the U-Net has the ability to extract the frequency band interaction layer feature, and the data obtained after the frequency band interaction layer extracts the frequency band interaction feature (an example of the second audio feature set) is further input into the frequency point interaction layer. The data obtained after the frequency point interaction layer extracts the global feature of the frequency point interaction (an example of the third audio feature set) is input into the channel selection module.

[0121] The channel selection module separates the audio signal corresponding to the target sound source from the third audio feature set, that is, outputs the ratio data of the target sound source corresponding to the audio signal in the mixed audio. Figure 12 For example, the channel selection module corresponds to two channels. It should be understood that this refers to selecting two channels from N channels, and each channel outputs one audio signal, and the two channels output the audio signals of two target sound sources, respectively. Therefore, by adjusting the proportion of each channel through the channel selection module, the separation of the audio signal can be realized, for example, the proportion of the channel of the non-target sound source can be set to 0.

[0122] It should be understood that, Figure 12 It should be understood that,

[0123] The judgment of whether the target sound source is of a preset category can also be omitted, so that the audio signals of all sound sources need to pass through the frequency band interaction layer and the frequency point interaction layer. The judgment of whether the feature extraction network contains a 1x1 convolution can also be removed, because the feature extraction network can directly know whether it contains a 1x1 convolution layer after being determined, so whether to set the frequency band interaction layer can be determined when constructing the initial sound source separation model. The feature extraction network can also use other neural network structures capable of extracting audio features other than the U-Net structure.

[0124] Using the method shown in Figure 12 The audio signals of the human voice and the drum sound extracted by the method shown in the figure can be directly played or further processed, for example, the spatial audio processing shown in (c) can be performed. Figure 1

[0125] Figure 13 ​This is a comparison chart showing the results of separating drum sounds from the same mixed audio using different methods. For example... Figure 13 As shown in (a), the spectrum of a standard drum sound is a clear, full-range audio signal. The term "full-range" is reflected in this spectrum. Figure 13 The lines along the vertical axis (frequency dimension) shown in (a) are continuous.

[0126] When using a traditional U-Net feature extraction network based on a two-dimensional convolutional network to extract audio features from mixed audio and then separating the drum sound audio signal, the two-dimensional convolution can only extract local features. This manifests in the spectrum as small line segments along the frequency dimension (i.e., the vertical direction). This inevitably means that these small line segments may not be able to be pieced together to form a coherent and complete line segment, resulting in many holes in the spectrum of the separated drum sound audio signal. Figure 13 The vertical lines shown in (b) are mixed and discontinuous, for example... Figure 13 In regions S1 and S2 shown in (b), there are gaps in the vertical lines. When such gaps appear, the separated drum sound will be distorted (it will not sound like a drum), and the human ear may perceive the existence of these gaps, that is, perceive the distorted drum sound.

[0127] When adopting the scheme of this application, a feature extraction network constructed using a two-dimensional convolutional network is first used to extract local features, that is, to extract small line segments in the frequency dimension. Then, a frequency band interaction layer and a frequency point interaction layer are used to extract global features, so that these small line segments can be accurately spliced ​​into a coherent and complete line segment. This can effectively reduce spectral holes and present a more complete and coherent signal. Figure 13 The effect shown in (c) is that the vertical lines are clear and continuous, for example... Figure 13 In regions S1 and S2 shown in (c), the vertical lines are continuous and there are no gaps, which is consistent with... Figure 13 The same area shown in (b) forms a sharp contrast. Therefore, what is separated is an accurate, undistorted drum sound. From Figure 13 As can also be seen in (c), the frequency band interaction features and frequency point interaction features of this application are both intended to supplement the correlation between the extracted local features, which can be understood as the connection relationship between the above-mentioned small line segments.

[0128] It should be noted that the above assumption is that the 2D convolutional network extracts small vertical line segments. The frequency band interaction layer and frequency point interaction layer are used to extract the relationships between these small line segments to more vividly illustrate the effect of the proposed solution. However, this does not mean that the 2D convolutional network necessarily extracts small vertical line segments; this depends on the spectral characteristics of the sound source itself and is not a limitation. It is sufficient to understand that the proposed solution first uses a 2D convolutional network to extract local features, and then uses the frequency band interaction layer and frequency point interaction layer to supplement the extraction of global features, thereby ensuring that features across the entire frequency band are fully extracted.

[0129] It should also be understood that if a feature extraction network based on a one-dimensional convolutional network in a traditional approach is used, a similar spectral effect to the approach in this application can be achieved, meaning that the separated drum sound audio signal can also exhibit similar characteristics. Figure 13 The effect is similar to that shown in (c). The reason is that one-dimensional convolution is a full-band operation, meaning that each operation covers the entire frequency dimension. To make it easier to understand, you can imagine that each operation extracts a series of lines that run through the entire frequency dimension. Figure 14 The process involves extracting the entire length (full frequency band) of a line segment along the vertical axis, requiring repeated operations to extract this full-length line segment, which necessitates a very large computational load. However, the proposed solution retains the advantage of low computational cost of 2D convolution while compensating for its shortcomings, thus achieving high separation accuracy with minimal computational cost.

[0130] Figure 14 This is a schematic diagram of a non-full-band audio source according to an embodiment of this application. Figure 14 Taking human voice as an example, such as Figure 14 The spectrum diagram of human voice shown illustrates that harmonic characteristics manifest as horizontal waves in the spectrum. The vertical axis (frequency dimension) lacks continuous lines, while the horizontal axis (time dimension or data dimension) depends on whether the voice is interrupted. For example... Figure 13 The S3 region shows a vertically oriented cavity. (Comparison) Figure 14 and Figure 14 It can be seen that a so-called full-band sound source can be understood to some extent as a sound source with continuous lines in the frequency dimension, while a so-called non-full-band sound source can be understood to some extent as a sound source without continuous lines in the frequency dimension. Therefore, for the latter, even without extracting global features, the continuity of the frequency dimension segments will not be affected, because a non-full-band sound source itself does not have continuous lines in the frequency dimension.

[0131] When using the scheme described in this application to separate human voice, the feature extraction network is sufficient to extract the desired features. Figure 15The local features in the frequency dimension shown, combined with its non-full-band characteristics, do not require further extraction of global features using the frequency band interaction layer and the frequency point interaction layer. Even if the scheme of the present application is adopted and the extraction of frequency band interaction features and global frequency point interaction features is performed after the feature extraction network, the features extracted by the frequency band interaction layer and the frequency point interaction layer will be very few, so that the harmonic features are still retained and will not cause the filling of the self-hollow consequences.

[0132] Figure 15 is a schematic flowchart of another audio processing method of an embodiment of the present application. Figure 4 is Figure 16 An example of the method shown is to first divide the frequency band and then use the feature extraction network to extract features, and a judgment is added as to whether to use the frequency band interaction layer and the frequency point interaction layer to extract features, so that for non-full-band sound sources, the above two interaction layers do not need to be processed, further reducing the amount of calculation and shortening the processing delay.

[0133] S1501, obtaining audio data to be processed.

[0134] This step is an example of step S401.

[0135] S1502, extracting audio features using a feature extraction network.

[0136] This step is an example of extracting features using the feature extraction network in the sound source separation model to obtain the first audio feature set in step S403.

[0137] S1503, obtaining and judging whether the category of the target sound source belongs to the preset sound source category, when the judgment result is "yes", executing step S1504; when the judgment result is "no", executing step S1505.

[0138] In one implementation mode, a preset sound source category set can be created, which lists full-band sound source categories. Only the target sound source belonging to the preset sound source category set will be input to the frequency band interaction layer and the frequency point interaction layer.

[0139] In another implementation mode, the target sound source is created as a category feature vector, and the category feature vector is input into a discriminant network. The output value of the discriminant network is judged. When the output value is less than or equal to a preset threshold, it is equivalent to that the target sound source belongs to the preset sound source category, otherwise it does not belong to the preset sound source category. The preset threshold can be 0.5 for example, but it should be understood that other numerical values such as 0.6 can also be used, and there is no limitation.

[0140] S1504, respectively extracting frequency band interaction features and frequency point interaction features using the frequency band interaction layer and the frequency point interaction layer.

[0141] S1505, separate the audio signal of the target sound source through the channel selection.

[0142] For the convenience of understanding, the following will be described in conjunction with specific examples.

[0143] Assuming that the target sound source is human voice, which is a non-full-band audio with harmonic characteristics, when performing step S1503, if the preset sound source category is the set of full-band sound source categories, the human voice will be determined as not belonging to the preset sound source category, and step S1505 will be performed instead of step S1504. In this case, if the judgment method of whether it belongs to the preset sound source category is to compare the relationship between the discriminant network and the preset threshold, when performing step S1503, the feature vector of the sound source category of human voice is constructed, the constructed feature vector is input into the discriminant network to obtain an output value, and the relationship between the output value and the preset threshold is compared. At this time, since the corresponding output value of the human voice is greater than the preset threshold, it is determined as not belonging to the preset sound source category, and thus based on the determination result, step S1505 is performed instead of step S1504.

[0144] Assuming that the target sound source is drum sound, which is a full-band audio, when performing step S1503, if the preset sound source category is the set of full-band sound source categories, the drum sound will be determined as belonging to the preset sound source category, and step S1504 and step S1505 will be performed. In this case, if the judgment method of whether it belongs to the preset sound source category is to compare the relationship between the discriminant network and the preset threshold, when performing step S1503, the feature vector of the sound source category of drum sound is constructed, the constructed feature vector is input into the discriminant network to obtain an output value, and the relationship between the output value and the preset threshold is compared. At this time, since the corresponding output value of the drum sound is less than the preset threshold, it is determined as belonging to the preset sound source category, and thus based on the determination result, step S1504 and step S1505 are performed.

[0145] Figure 16 is a schematic flow chart of a training method of a sound source separation model according to an embodiment of the present application. The following will introduce each step shown in Figure 12 .

[0146] S1601, obtaining training data.

[0147] The training data includes audio data to be trained and an audio signal label of at least one known sound source corresponding to the audio data to be trained, and the audio data to be trained is mixed audio of multiple sound sources synthesized by an audio signal of at least one known sound source and other audio signals.

[0148] It should be understood that the training stage is supervised learning, and the real data needs to be known, so that the predicted data output by the model can be compared with the real data, and the model weight parameters are adjusted reversely based on this, so that the predicted data output by the model is closer and closer to the real data.

[0149] Each to-be-trained audio data and the audio signal label of at least one known sound source corresponding thereto can constitute a set of training data. In the scheme of the present application, each training is to compare the audio signal (that is, the predicted data) output by the model after processing the to-be-trained audio data with the audio signal label (the real data) of at least one known sound source, and reversely update the model weight parameters, so that the audio signal output by the model is closer and closer to the audio signal label of at least one known sound source.

[0150] S1602, input the to-be-trained audio data to the initial sound source separation model, update the weight parameters of the initial sound source separation model according to the difference between the predicted audio signal of at least one known sound source output by the initial sound source separation model and the audio signal label of at least one known sound source, and obtain the trained sound source separation model.

[0151] In an implementation manner, the initial sound source separation model comprises a feature extraction network, a frequency band interaction layer and a frequency point interaction layer; the feature extraction network is constructed by using a two-dimensional convolutional base network, and is used for performing feature extraction on the to-be-trained audio data to obtain a first audio feature set; the frequency band interaction layer is used for extracting frequency band interaction features between a plurality of sub-bands corresponding to the to-be-trained audio data from the first audio feature set, so as to obtain a second audio feature set; the frequency point interaction layer is used for extracting frequency point interaction features in each sub-band in the second audio feature set, so as to obtain a third audio feature set; and the sound source separation model is further used for separating the predicted audio signal of at least one known sound source from the third audio feature set.

[0152] The related content of the plurality of sub-bands can be referred to the above, and will not be described herein again.

[0153] It should be understood that the description of the inference stage for the structure of the sound source separation model in the above can be referred to the training stage, and will not be described herein again for the sake of brevity.

[0154] In an implementation manner, the plurality of sub-bands are obtained by dividing the to-be-trained audio data along the frequency dimension, or the plurality of sub-bands are obtained by dividing the first extraction result output by the first network layer along the frequency dimension, and the first network layer is any layer of the feature extraction network.

[0155] In one example, the initial source separation model comprises a feature extraction network and a frequency bin interaction layer; the feature extraction network is constructed by using a two-dimensional convolutional base network, and the feature extraction network comprises a 1x1 convolutional layer for extracting features of a plurality of sub-bands to obtain a first audio feature set; the frequency bin interaction layer is used to extract frequency bin interaction features in each sub-band in the first audio feature set to obtain a third audio feature set; the initial source separation model is further used to separate a predicted audio signal of at least one known source from the third audio feature set. As can be seen from this example, for the feature extraction network with the 1x1 convolutional layer, it can be determined in the training stage that the frequency band interaction layer is removed, so when such a source separation model is applied to the audio processing method of the present application after being trained (that is, in the inference stage), it is no longer necessary to determine whether to perform processing through the frequency band interaction layer.

[0156] In one implementation, the frequency band interaction layer and / or the frequency bin interaction layer is constructed by using a two-dimensional convolutional base network, an attention mechanism network or a fully connected layer.

[0157] In one example, the frequency band interaction layer is constructed by using a 1x1 convolutional layer; and the frequency bin interaction layer is constructed by using an FC layer. Figure 16 An example of this example is given.

[0158] The initial source separation model can be a completely untrained model, or can be a model in which a frequency band interaction layer and a frequency bin interaction layer are added on the basis of a feature extraction network that has been trained (that is, a feature extraction network that has the ability to extract local features in audio data), and the latter can reduce the number of training rounds to a certain extent.

[0159] Figure 4 The method provides a new source separation model, which not only uses a feature extraction network with relatively small computational complexity to extract audio features, but also uses a frequency band interaction layer and a frequency bin interaction layer to respectively supplement the extraction of frequency band interaction features and frequency bin interaction features, and through training of the new source separation model, the source separation model has good source separation capability, and the trained source separation model can be applied to Figures 5-12 The audio processing method.

[0160] The method of the embodiments of the present application is introduced above in combination with the drawings. It should be understood that although each step in the flowchart involved in each embodiment as described above is displayed in sequence, these steps are not necessarily executed in sequence as shown in the figure. Unless explicitly stated herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential but can be executed in rotation or alternation with at least part of other steps or stages in other steps. The device of the embodiments of the present application is introduced below in combination with the drawings.

[0161] The present application also provides a training device, which comprises an acquisition unit and a training unit. The device can be integrated in a server, a cloud server, or other electronic devices capable of training a neural network model, i.e., can be integrated in a training device. The device can be used to execute any of the above training methods of the sound source separation model. For example, the acquisition unit can be used to execute steps S1601-S1602, and the training unit can be used to execute step S1603. In an implementation manner, the device can further comprise a storage unit for storing relevant data. The storage unit can be integrated in any of the above units, or can be a unit independent of all the above units.

[0162] The present application also provides a sound source separation device, which comprises an acquisition unit and a processing unit. The device can be integrated in a mobile phone, a notebook computer, a tablet computer, a car machine, a smart wearable device, or other electronic devices capable of processing an audio signal by using a neural network model, i.e., can be integrated in an inference device. The device can be used to execute any of the above audio processing methods. For example, the acquisition unit can be used to execute step S401, and the processing unit can be used to execute step S402. The device can also be used to execute Figure 15 and Figure 17 each process. In an implementation manner, the device can further comprise a storage unit for storing relevant data. The storage unit can be integrated in any of the above units, or can be a unit independent of all the above units.

[0163] Figure 17 is a software architecture schematic diagram of an electronic device according to an embodiment of the present application. Figure 17 The execution flow of the present application scheme can be introduced immediately from the perspective of the internal execution process of the electronic device. As shown in FIG. 1, the electronic device can comprise a processor 1001, a bus 1002, a storage medium 1003, a display unit 1004, a sensor 1005, a communication interface 1006, and the like. Figure 17As shown, the application layer includes various applications. Here, we take applications such as camera, video, browser, and music as an example. However, it should be understood that in practice, it may include only some of these applications or other applications as well. There is no limitation.

[0164] Any application capable of playing audio data can adopt the solution proposed in this application. For example, it could be listening to music online in a browser or creating songs using a music application, etc., and will not be listed one by one.

[0165] The framework layer includes an audio stream service module and an audio stream management module. The audio stream service module is mainly used to store and forward audio streams and realize cross-application transmission, while the audio stream management module is mainly used to manage related drivers such as speakers, set some management parameters, or make related settings for process scheduling.

[0166] It should be understood that Figure 17 The software architecture mainly shows the application layer and framework layer, but in practice, electronic devices can also include other software layers, such as the native layer, the hardware abstraction layer (HAL layer), etc., without any limitation.

[0167] The hardware layer includes the processor, the audio source separation model deployed in the hardware layer, and the speakers.

[0168] Suppose a user selects spatial audio effects in a music application and starts playing a song, then the music application will transmit the song's audio stream ( Figure 17 The audio stream #1 is sent to the audio stream service module in the framework layer, and then to the processor in the hardware layer. Upon receiving the audio stream, the processor executes the relevant steps of this application, including converting the time-domain audio stream #1 into frequency-domain data (assuming the result is data #2, an example of the audio data to be processed), inputting it into the sound source separation model, and processing it to obtain data #3, which is the separated audio signal. After obtaining data #3, the processor performs spatial audio processing to obtain spatial audio data (data #4), and then sends data #4 to the audio stream service module, which transmits it to the music application. The music application plays the song online with spatial audio effects. The music application can also call the speaker in the hardware layer through the audio stream management module and the audio stream service module to complete the playback.

[0169] It should be understood that the above is merely an example of a processing flow to illustrate the solution. In practical applications, other situations may exist. For example, the processor may first divide the signal into frequency bands and then input the frequency domain data of multiple sub-bands (i.e., data #2 is the segmented data) into the audio source separation model. Other situations will not be listed one by one.

[0170] The processor can include one or more processing units, such as a central processing unit (CPU), an application processor (AP), a modem processing unit, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video code, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), other general purpose processors, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, and / or the like. The general purpose processor can be a microprocessor or the processor can also be any conventional processor. Different processing units can be independent devices or integrated in one or more processors.

[0171] The sound source separation model can be stored in the memory, ​ The memory, not shown in the figure, can be an internal storage unit of the electronic device, such as a hard disk or a memory of the electronic device, or an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like. Optionally, the memory can include both the internal storage unit and the external storage device of the electronic device. The memory can be used to store an operating system, an application program, a boot loader, data, and other programs, such as program codes of the computer program, and the like. The memory can also be used to temporarily store data that has been output or will be output.

[0172] The memory can also be provided in the processor to store instructions and data. In some embodiments, the memory in the processor is a cache memory. The memory can hold instructions or data that have just been used or are frequently used by the processor. If the processor 1 needs to use the instructions or data again, it can directly call from the memory. This avoids repeated access and reduces the waiting time of the processor, thereby improving the efficiency of the system.

[0173] It should be noted that the information interaction, execution process and the like between the above apparatuses / units are based on the same concept as the method embodiments of the present application, and the specific functions and the technical effects brought by the same can be referred to the method embodiments part, which will not be repeated here.

[0174] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to complete all or part of the above-described functions. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for convenient distinction, and do not limit the protection scope of the present application. The specific working process of the unit and module in the system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0175] The embodiment of the present application further provides an electronic device, which comprises one or more processors and a memory; the memory is coupled with the one or more processors, and the memory is used for storing computer program codes, the computer program codes comprise computer instructions, and the one or more processors invoke the computer instructions to enable the electronic device to execute the steps in any of the foregoing methods.

[0176] The embodiment of the present application further provides a chip system, which is applied to an electronic device, and the chip system comprises one or more processors, and the one or more processors are used for invoking computer instructions to enable the electronic device to execute the steps in any of the foregoing methods. Optionally, the chip system further comprises a memory, and the memory is electrically connected with the processor. Optionally, the chip system can further comprise a communication interface.

[0177] The embodiments of the present application further provide a computer readable storage medium, which stores instructions, and when the instructions are executed by an electronic device, any method described above can be implemented. The computer readable medium can at least include any entity or device capable of carrying the computer program code (instructions) to the photographing device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a U disk, a mobile hard disk, a magnetic disk or an optical disk, etc. In some jurisdictions, according to legislation and patent practice, the computer readable medium can not be an electrical carrier signal and a telecommunication signal.

[0178] The embodiments of the present application further provide a computer program product, which includes a computer program, and when the computer program is executed by an electronic device, any method described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable code, or some intermediate form.

[0179] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0180] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software mode depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0181] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely schematic, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed ones can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0182] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0183] It should be understood that when used in the specification and the appended claims of the present application, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0184] It should also be understood that the term "and / or" used in the description of the present application and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0185] In addition, in the description of the present application and the appended claims, the terms "first", "second", "third" and the like are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0186] In the present application, the reference "one embodiment" or "some embodiments" and the like means that the specific features, structures or characteristics described in connection with the embodiment are included in one or more embodiments of the present application. Therefore, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in further some embodiments" and the like appearing in different places in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "include", "contain", "have" and their variants mean "include but not limited to", unless otherwise specifically emphasized.

[0187] The above-described embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. An audio processing method applied to an electronic device, characterized in that, include: Acquire audio data to be processed, wherein the audio data to be processed includes audio signals from multiple sound sources; The audio data to be processed is processed using a sound source separation model to obtain an audio signal of at least one target sound source; the at least one target sound source is at least one of the multiple sound sources. The audio source separation model includes a feature extraction network, a frequency band interaction layer, and a frequency point interaction layer; the feature extraction network is constructed using a two-dimensional convolutional network and is used to extract features from the audio data to be processed to obtain a first audio feature set. The frequency band interaction layer is used to extract frequency band interaction features between multiple sub-frequency bands corresponding to the audio data to be processed from the first audio feature set, thereby obtaining a second audio feature set; during the calculation process of the frequency band interaction layer, the calculation of each sub-frequency band is combined with the data of other sub-frequency bands; the features obtained considering other sub-frequency bands are the frequency band interaction features; the multiple sub-frequency bands are obtained by dividing the audio data to be processed along the frequency dimension, or the multiple sub-frequency bands are obtained by dividing the first extraction result output by the first network layer along the frequency dimension, where the first network layer is any layer of the feature extraction network; The frequency point interaction layer is used to extract global frequency point interaction features in each sub-band of the second audio feature set, thereby obtaining the third audio feature set; The calculation process of the frequency point interaction layer ensures that the calculation of each frequency point within each sub-band is combined with the data of other frequency points; the frequency point interaction features obtained by considering the global frequency points are called global frequency point interaction features. The sound source separation model is also used to separate the audio signal of the at least one target sound source from the third set of audio features.

2. The method according to claim 1, characterized in that, When the feature extraction network has the ability to extract frequency band interaction features between the multiple sub-bands, such that the first audio feature set contains frequency band interaction features between the multiple sub-bands, the audio source separation model does not include the frequency band interaction layer; and / or, when the feature extraction network has the ability to extract global frequency point interaction features corresponding to each frequency point of the audio data to be processed, such that the first audio feature set contains global frequency point interaction features of each frequency point, the audio source separation model does not include the frequency point interaction layer.

3. The method according to claim 1, characterized in that, The method further includes: The target dimension of the frequency band interaction layer is matched with the functional dimension of the base network that constructs the frequency band interaction layer by means of a reshape operation; and / or, the target dimension of the frequency point interaction layer is matched with the functional dimension of the base network that constructs the frequency point interaction layer by means of a reshape operation.

4. The method according to any one of claims 1 to 3, characterized in that, The process of using a sound source separation model to process the audio data to obtain audio signals from at least one target sound source includes: When the first target audio source belongs to a preset audio source category, the frequency band interaction layer is used to extract the frequency band interaction features between the multiple sub-frequency bands from the first audio feature set, thereby obtaining the second audio feature set. The frequency point interaction layer is then used to extract the global frequency point interaction features within each sub-frequency band of the second audio feature set, thereby obtaining the third audio feature set. Finally, the audio signal of the first target audio source is separated from the third audio feature set using the audio source separation model. The first target audio source is any one of the at least one target audio source; or... If the first target audio source does not belong to the preset audio source category, the audio signal of the first target audio source is separated from the first audio feature set using the audio source separation model.

5. The method according to claim 4, characterized in that, The preset sound source category is used to represent a full-band sound source category; or, the output value of the discrimination network corresponding to the sound source in the preset sound source category is less than or equal to a preset threshold.

6. The method according to any one of claims 1 to 3, characterized in that, The frequency band interaction layer and / or the frequency point interaction layer are constructed using a two-dimensional convolutional basic network, an attention mechanism network, or a fully connected FC layer.

7. A training method for a sound source separation model, characterized in that, include: Acquire training data, which includes audio data to be trained and audio signal labels of at least one known sound source corresponding to the audio data to be trained. The audio data to be trained is a mixed audio of multiple sound sources synthesized by combining the audio signals of the at least one known sound source with other audio signals. The audio data to be trained is input into an initial audio source separation model. Based on the difference between the predicted audio signal of the at least one known audio source output by the initial audio source separation model and the audio signal label of the at least one known audio source, the weight parameters of the initial audio source separation model are updated, thereby obtaining the trained audio source separation model. The initial audio source separation model includes a feature extraction network, a frequency band interaction layer, and a frequency point interaction layer. The feature extraction network is constructed using a two-dimensional convolutional network and is used to extract features from the audio data to be trained, obtaining a first audio feature set. The frequency band interaction layer is used to extract frequency band interaction features between multiple sub-frequency bands corresponding to the audio data to be trained from the first audio feature set, thereby obtaining a second audio feature set; during the calculation process of the frequency band interaction layer, the calculation of each sub-frequency band is combined with the data of other sub-frequency bands; the features obtained considering other sub-frequency bands are the frequency band interaction features; the multiple sub-frequency bands are obtained by dividing the audio data to be trained along the frequency dimension, or the multiple sub-frequency bands are obtained by dividing the first extraction result output by the first network layer along the frequency dimension, where the first network layer is any layer of the feature extraction network; the frequency point interaction layer is used to extract the frequency point interaction features within each sub-frequency band in the second audio feature set, thereby obtaining a third audio feature set; The calculation process of the frequency interaction layer ensures that the calculation of each frequency point within each sub-band is combined with the data of other frequency points; the frequency interaction features obtained by considering the global frequency points are called global frequency interaction features; the sound source separation model is also used to separate the predicted audio signal of the at least one known sound source from the third audio feature set.

8. The method according to claim 7, characterized in that, The frequency band interaction layer and / or the frequency point interaction layer are constructed using a two-dimensional convolutional basic network, an attention mechanism network, or a fully connected FC layer.

9. An electronic device, characterized in that, The electronic device includes: one or more processors, and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 6, or 7 to 8.

10. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being configured to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 6, or 7 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as claimed in any one of claims 1 to 6, or 7 to 8.

Images