Audio separation method, apparatus, device, storage medium, and program product

Abstract

The application provides an audio separation method, device, equipment, storage medium and program product. The method comprises the following steps: performing feature extraction processing on mixed audio to obtain first audio features; the mixed audio comprises human voice audio and background audio; a voiceprint feature of human voice is extracted from the first audio features to obtain second audio features containing the voiceprint feature; the first audio features and the second audio features are fused to obtain third audio features; and human voice audio and background audio are extracted from the mixed audio based on the third audio features. The above method can improve the separation effect of human voice and background sound.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio processing, and particularly relates to an audio separation method and device, equipment, a storage medium and a program product. BACKGROUND

[0002] In the field of music production and audio processing, the separation of vocals and background music has always been an important research topic. Traditional source separation techniques often rely on spectral analysis and statistical models, but in complex musical environments, the effect is limited. With the rapid development of deep learning technology, neural network-based audio separation technology has been gradually applied, but there are still some challenges in separating vocals and background music in music data, especially when the instrument sound component is similar to the vocal sound in the spectral structure, which will cause the model to mistakenly separate the instrument sound component into the vocal sound, and the separation effect is not ideal. SUMMARY

[0003] Based on the above technical status, the present application proposes an audio separation method, device, equipment, storage medium and program product, which can improve the separation effect of vocals and background music.

[0004] The first aspect of the present application proposes an audio separation method, comprising:

[0005] performing feature extraction processing on the mixed audio to obtain first audio features; the mixed audio includes vocal audio and background audio;

[0006] extracting a voiceprint feature of the vocals from the first audio features to obtain second audio features containing the voiceprint feature;

[0007] fusing the first audio features and the second audio features to obtain third audio features;

[0008] extracting vocal audio and background audio from the mixed audio based on the third audio features.

[0009] In some implementations, the extracting vocal audio and background audio from the mixed audio based on the third audio features comprises:

[0010] generating a vocal mask and a background mask based on the third audio features;

[0011] extracting vocal audio and background audio from the mixed audio based on the vocal mask and the background mask.

[0012] In some implementations, a feature extraction process is performed on the mixed audio to obtain first audio features, a voiceprint feature of a human voice is extracted from the first audio features to obtain second audio features containing the voiceprint feature, the first audio features and the second audio features are fused to obtain third audio features, and a human voice mask and a background mask are generated based on the third audio features, including:

[0013] An audio feature of the mixed audio is input into an audio separation model to obtain a human voice mask and a background mask output by the audio separation model;

[0014] The audio separation model includes:

[0015] A first feature modeling module is configured to perform a feature extraction process on the mixed audio to obtain first audio features;

[0016] A voiceprint perception module is configured to extract a voiceprint feature of a human voice from the first audio features to obtain second audio features containing the voiceprint feature;

[0017] A second feature modeling module is configured to fuse the first audio features and the second audio features to obtain third audio features;

[0018] A mask estimation module is configured to generate a human voice mask and a background mask based on the third audio features.

[0019] In some implementations, the feature extraction process performed on the mixed audio to obtain the first audio features includes:

[0020] The audio feature of the mixed audio is subband-divided to obtain a plurality of subband audio features;

[0021] The plurality of subband audio features are jointly modeled in a time dimension and a frequency dimension to obtain the first audio features.

[0022] In some implementations, the joint modeling of the plurality of subband audio features in the time dimension and the frequency dimension to obtain the first audio features includes:

[0023] The plurality of subband audio features are contextually jointly modeled in the time dimension to obtain first global features;

[0024] The first global features are dimensionally reduced, and the first global features after the dimension reduction are superimposed with the plurality of subband audio features to obtain second global features;

[0025] The second global features are contextually jointly modeled in the frequency dimension to obtain third global features;

[0026] Dimensionality reduction is performed on the third global feature, and the third global feature after dimensionality reduction is superimposed with the second global feature to obtain a first audio feature.

[0027] In some implementations, the sub-band division on the audio feature of the mixed audio to obtain a plurality of sub-band audio features includes:

[0028] The sub-band division on the audio feature of the mixed audio is performed in a manner that the bandwidth of the divided sub-band audio feature is proportional to the frequency of the sub-band audio feature.

[0029] In some implementations, the extraction of the voiceprint feature of the human voice from the first audio feature to obtain a second audio feature containing the voiceprint feature includes:

[0030] The first audio feature is input into a voiceprint perception model, and the voiceprint perception model extracts a voiceprint feature of a human voice from the first audio feature to obtain a second audio feature containing the voiceprint feature.

[0031] The voiceprint perception model extracts a voiceprint feature from an input audio feature sample and performs speaker recognition training based on the extracted audio feature containing the voiceprint feature.

[0032] In some implementations, the fusion of the first audio feature and the second audio feature to obtain a third audio feature includes:

[0033] The first audio feature and the second audio feature are spliced to obtain a spliced feature.

[0034] The spliced feature is jointly modeled in the time dimension and the frequency dimension to obtain a third audio feature.

[0035] In some implementations, the joint modeling of the spliced feature in the time dimension and the frequency dimension to obtain a third audio feature includes:

[0036] The spliced feature is contextually jointly modeled in the time dimension to obtain a first fusion feature.

[0037] The first fusion feature is dimensionally reduced, and the first fusion feature after dimensionality reduction is superimposed with the spliced feature to obtain a second fusion feature.

[0038] The second fusion feature is contextually jointly modeled in the frequency dimension to obtain a third fusion feature.

[0039] The third fusion feature is dimensionally reduced, and the third fusion feature after the dimension reduction is superimposed with the second fusion feature to obtain a third audio feature.

[0040] In some implementations, the generating a vocal mask and a background mask based on the third audio feature includes:

[0041] From the third audio feature, a feature corresponding to each subband is extracted; the subband is determined by subband division on an audio feature of the mixed audio;

[0042] Based on the feature corresponding to each subband, an estimation result of a vocal mask and a background mask corresponding to each subband is determined;

[0043] The estimation results of the vocal mask and the background mask corresponding to each subband are merged to determine the vocal mask and the background mask.

[0044] The second aspect of the present application proposes an audio separation device, comprising:

[0045] A first feature processing unit is configured to perform feature extraction processing on mixed audio to obtain a first audio feature; the mixed audio includes vocal audio and background audio;

[0046] A voiceprint feature extraction unit is configured to obtain a second audio feature containing a voiceprint feature by extracting the voiceprint feature of the vocal audio from the first audio feature;

[0047] A second feature processing unit is configured to fuse the first audio feature and the second audio feature to obtain a third audio feature;

[0048] An audio separation unit is configured to extract vocal audio and background audio from the mixed audio based on the third audio feature.

[0049] The third aspect of the present application proposes an electronic device, comprising a memory and a processor;

[0050] The memory is connected with the processor, and is configured to store a program;

[0051] The processor is configured to realize the audio separation method by running the program in the memory.

[0052] The fourth aspect of the present application proposes a storage medium, and the storage medium stores a computer program; when the computer program is run by a processor, the audio separation method is realized.

[0053] The fifth aspect of the present application provides a computer program product comprising computer program instructions which, when executed by a processor, cause the processor to perform the audio separation method described above.

[0054] The audio separation method provided by the present application extracts a voiceprint feature from a first audio feature of mixed audio, and fuses a second audio feature containing the voiceprint feature with the first audio feature, and then separates a third audio feature obtained by fusion from user audio. The third audio feature used for audio separation contains the voiceprint feature of the human voice in the mixed audio, thereby facilitating accurate identification of the human audio component from the mixed audio based on the third audio feature, thereby realizing more accurate separation of human voice and background sound and improving the quality of audio separation. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0056] Figure 1 A flowchart of an audio separation method provided by an embodiment of the present application.

[0057] Figure 2 A structural diagram of an audio separation model provided by an embodiment of the present application.

[0058] Figure 3 A structural diagram of another audio separation model provided by an embodiment of the present application.

[0059] Figure 4 A structural diagram of a sub-band division module provided by an embodiment of the present application.

[0060] Figure 5 A structural diagram of a sub-band intra and inter modeling module provided by an embodiment of the present application.

[0061] Figure 6 A structural diagram of two sub-band intra and inter modeling modules connected in series provided by an embodiment of the present application.

[0062] Figure 7 A structural diagram of a voiceprint perception model provided by an embodiment of the present application.

[0063] Figure 8 A structural diagram of a mask estimation module provided by an embodiment of the present application.

[0064] Figure 9A structural schematic diagram of an audio separation device provided by an embodiment of the present application.

[0065] Figure 10 A structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0066] The technical solution of the embodiment of the present application is applicable to an audio separation application scenario, and is particularly applicable to an application scenario of separating human voice audio and background audio from mixed audio. The technical solution of the embodiment of the present application can more accurately separate human voice audio and background audio from mixed audio.

[0067] In the field of music production and audio processing, the separation of human voice and background audio has always been an important research topic. Traditional audio source separation techniques often rely on spectral analysis and statistical models, but in a complex music environment, the effect is limited.

[0068] With the rapid development of deep learning technology, neural network-based audio separation technology has gradually been applied, but there are still certain challenges in separating human voice and background audio in music data. The existing neural network-based music human voice and background audio separation method only relies on a data-driven way to implicitly learn the difference between human voice and background audio to complete the separation task. However, for unseen data, due to the generalization problem, the separation effect is poor, especially when the instrument sound component is similar to the human voice in the spectral structure, the model will mistakenly separate the instrument sound component into the human voice, and the separation effect is not ideal.

[0069] To solve the above technical problems, the present application provides an audio separation method based on voiceprint perception. This method takes into account the uniqueness and stability of voiceprint features, which helps the model to identify the human voice component in music and effectively separate it, thereby achieving more accurate separation of human voice and background audio.

[0070] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0071] The audio separation method proposed in the embodiments of the present application can be exemplarily applied to an audio processing device or system, such as a computer, a server, a workstation, a smart terminal, a handheld terminal, a wearable device, etc. with audio processing function, or a system composed of these devices and an audio processing system inside these devices, which can be executed by a processor in the above-mentioned devices or systems.

[0072] The embodiment of the present application first proposes an audio separation method, as shown in Figure 1 The method comprises:

[0073] S101, performing feature extraction processing on the mixed audio to obtain first audio features.

[0074] The mixed audio comprises human voice audio and background audio, that is, the mixed audio can be obtained by mixing the human voice audio and the background audio.

[0075] When the mixed audio is obtained, first, the time domain signal of the mixed audio is extracted. For example, the signal of the mixed audio is subjected to amplitude maximum value normalization processing, and the signal amplitude of the mixed audio is normalized to the range of [-1, 1]. Then, the normalized mixed audio is converted into a frequency spectrum graph through short-time Fourier transform (STFT), and a logarithmic power spectrum (lps) feature is obtained through the following formula: lps = ln(Y2), wherein Y represents the frequency spectrum graph of the mixed audio.

[0076] The logarithmic power spectrum feature can be used as the first audio feature of the mixed audio. In other embodiments, the logarithmic power spectrum feature can be further processed, such as convolution processing based on context interaction, to further extract deep features from the logarithmic power spectrum feature, and the processed features can be used as the first audio feature. In addition, other audio feature extraction methods can be used to extract audio features from the mixed audio, and the extracted audio features can be used as the first audio feature.

[0077] S102, extracting a voiceprint feature of the human voice from the first audio feature to obtain a second audio feature comprising the voiceprint feature.

[0078] Specifically, the voiceprint feature of the human voice refers to the voiceprint feature of the human voice embodied in the human voice audio in the mixed audio. The voiceprint feature can be used to reflect the voiceprint characteristics of the speaker generating the human voice audio, and can be used to identify the human voice audio.

[0079] Common voiceprint features include short-time energy, zero-crossing rate, mel-frequency cepstral coefficient, etc.

[0080] For example, in the embodiment of the present application, the voiceprint feature of the human voice is extracted from the first audio feature by a voiceprint extraction model to obtain a second audio feature comprising the voiceprint feature.

[0081] That is, the first audio feature is input into a pre-trained voiceprint extraction model, and the voiceprint feature of the human voice is extracted from the first audio feature by the voiceprint extraction model to obtain a second audio feature comprising the voiceprint feature.

[0082] The voiceprint extraction model can adopt a Gaussian mixture model (GMM), a support vector machine (SVM), a deep neural network (DNN), or the like. When the voiceprint extraction model is trained, the audio features of the sample audio to be extracted are input into the voiceprint extraction model, the voiceprint features are extracted from the audio features by the voiceprint extraction model, and speaker recognition is performed based on the voiceprint features output by the voiceprint extraction model. When the voiceprint features output by the voiceprint extraction model can accurately identify the speaker, it is considered that the voiceprint extraction model can accurately extract the voiceprint features of the speaker.

[0083] It should be noted that the voiceprint features of the human voice are extracted from the first audio features in the embodiments of the present application, and the voiceprint features of each speaker audio contained in the mixed audio are extracted, that is, the voiceprint features of the human voice of each speaker are extracted when the mixed audio contains human voice audios of several speakers, and finally the second audio features containing the voiceprint features are obtained, which are second audio features containing the voiceprint features of each speaker in the mixed audio.

[0084] S103, the first audio features and the second audio features are fused to obtain third audio features.

[0085] Specifically, after the voiceprint features are extracted from the first audio features, the second audio features containing the voiceprint features are fused with the original first audio features to obtain third audio features.

[0086] The fusion can be context feature interaction after splicing to obtain the third audio features. Alternatively, the fusion can also be superposition processing of the second audio features and the first audio features at corresponding positions.

[0087] S104, based on the third audio features, human voice audio and background audio are extracted from the mixed audio.

[0088] Specifically, the third audio features obtained through the above processing contain complete mixed audio features (first audio features) and voiceprint features (second audio features), and therefore can be used for separating human voice audio and background audio from the mixed audio.

[0089] In some embodiments, human voice masks and background masks are generated based on the third audio features.

[0090] Then, based on the human voice masks and the background masks, human voice audio and background audio are extracted from the mixed audio.

[0091] Exemplarily, the third audio feature can be input into a pre-trained audio separation model such as BSRNN, etc., and the human voice mask and the background mask can be separated from the third audio feature by processing the third audio feature by the model. Then, the human voice mask and the background mask are multiplied with the spectrogram of the mixed audio respectively to obtain the spectrum of the human voice audio and the spectrum of the background audio. The time-domain human voice audio and the time-domain background audio can be obtained by performing inverse Fourier transform on the spectrum of the human voice audio and the spectrum of the background audio.

[0092] As can be seen from the above, the audio separation method proposed in the embodiments of the present application extracts the voiceprint feature from the first audio feature of the mixed audio, fuses the second audio feature containing the voiceprint feature with the first audio feature, and then separates the third audio feature obtained by fusion. The third audio feature used for audio separation contains the voiceprint feature of the human voice in the mixed audio, which is conducive to accurately identifying the human voice audio component from the mixed audio based on the third audio feature, thereby realizing more accurate separation of human voice and background sound and improving the quality of audio separation.

[0093] In another embodiment of the present application, an audio separation model is also proposed. Through the audio separation model, the audio separation method proposed in the above embodiments of the present application can be performed.

[0094] Referring to Figure 2 The audio separation model includes a first feature modeling module, a voiceprint perception module, a second feature modeling module, and a mask estimation module.

[0095] The first feature modeling module is configured to perform feature extraction processing on the mixed audio to obtain a first audio feature.

[0096] The voiceprint perception module is configured to extract a voiceprint feature of human voice from the first audio feature to obtain a second audio feature containing the voiceprint feature.

[0097] The second feature modeling module is configured to fuse the first audio feature and the second audio feature to obtain a third audio feature.

[0098] The mask estimation module is configured to generate a human voice mask and a background mask based on the third audio feature.

[0099] Next, the specific implementation process of the audio separation method proposed in the embodiments of the present application will be introduced in combination with the specific structure of each module of the above audio separation model.

[0100] Before processing the mixed audio input audio separation model, the signal of the mixed audio is first amplitude maximum normalized, and the signal amplitude of the mixed audio is normalized to the range of [-1, 1]. Then, the normalized mixed audio is converted into a spectrogram by a short-time Fourier transform (STFT), and a log power spectrum (lps) feature is obtained based on the spectrogram. As shown in Figure 2 , the log power spectrum (lps) feature is input as an audio feature of the mixed audio into the audio separation model for processing.

[0101] In some embodiments, as shown in Figure 3 , the first feature modeling module includes a sub-band division module and an intra-sub-band and inter-sub-band modeling module.

[0102] The sub-band division module is configured to divide the audio feature of the mixed audio into sub-bands to obtain a plurality of sub-band audio features.

[0103] The intra-sub-band and inter-sub-band modeling module is configured to jointly model the plurality of sub-band audio features in the time dimension and the frequency dimension to obtain the first audio feature.

[0104] In some embodiments, as shown in Figure 4 , the sub-band division module is a linear transformation unit composed of K layer normalization layers and K fully connected layers with an output dimension of N.

[0105] The sub-band division module divides the audio feature of the mixed audio into sub-bands to obtain a plurality of sub-band audio features, specifically:

[0106] First, the sub-band division module divides the lps feature of the mixed audio into sub-bands according to a predefined bandwidth to obtain K sub-band lps features.

[0107] The predefined bandwidth is and satisfies F is the frequency range of the mixed audio.

[0108] Then, the K sub-band lps features obtained are respectively passed through K layer normalization layers and fully connected layers to obtain K sub-band audio features.

[0109] In other embodiments, when dividing the audio feature of the mixed audio into sub-bands, the audio feature of the mixed audio is divided into sub-bands according to a relationship that the bandwidth of the divided sub-band audio feature is proportional to the frequency of the sub-band audio feature, to obtain a plurality of sub-band audio features. That is, the audio feature of the mixed audio is divided into non-uniform sub-bands in a manner that the sub-band bandwidth is narrower in the low frequency band and the sub-band bandwidth is wider in the high frequency band, and the non-uniform sub-bands are unified to the same dimension N for merging to obtain each sub-band audio feature.

[0110] In another embodiment, referring to Figure 5 The above-mentioned intra-subband and inter-subband modeling module is a modeling unit composed of a GRU, which is composed of a layer normalization layer, a time-dimension bidirectional GRU layer, a full connection layer, a layer normalization layer, a frequency-dimension bidirectional GRU layer, and a full connection layer.

[0111] Based on the structure of the above-mentioned intra-subband and inter-subband modeling module, the intra-subband and inter-subband modeling module jointly models the time dimension and frequency dimension of the plurality of subband audio features to obtain the first audio feature, which specifically includes the following A1-A4 processing:

[0112] A1, the layer normalization layer and the time-dimension bidirectional GRU layer are used to model the context joint of the plurality of subband audio features in the time dimension, and K subbands share the same bidirectional GRU layer to obtain the first global feature.

[0113] A2, the full connection layer is used to reduce the dimension of the first global feature obtained in step A1, the input channel is kept unchanged with the channel of the subband audio feature, and then the first global feature after the dimension reduction processing is superimposed with the plurality of subband audio features to obtain the second global feature.

[0114] A3, the layer normalization layer and the frequency-dimension bidirectional GRU layer are used to model the context joint of the second global feature obtained in step A2 in the subband frequency dimension, and the time dimension shares the same bidirectional GRU layer to obtain the third global feature.

[0115] A4, the full connection layer is used to reduce the dimension of the third global feature obtained in step A3, the input channel is kept unchanged with the channel of the second global feature in A2, and then the third global feature after the dimension reduction processing is superimposed with the second global feature to obtain the first audio feature.

[0116] In another embodiment, the above-mentioned intra-subband and inter-subband modeling module can be a series combination of two or more intra-subband and inter-subband modeling modules, as shown in Figure 6 Two intra-subband and inter-subband modeling modules can be connected in series to jointly model the audio features, so that the first audio feature contains more feature information, that is, the first audio feature is more accurate.

[0117] In another embodiment, the above-mentioned voiceprint perception module obtains a second audio feature containing the voiceprint feature by extracting the voiceprint feature of the human voice from the first audio feature, which is specifically realized by a voiceprint perception model.

[0118] That is, inputting the first audio feature into the voiceprint perception model, obtaining a second audio feature containing the voiceprint feature by extracting the voiceprint feature of the human voice from the first audio feature by the voiceprint perception model;

[0119] Wherein, the voiceprint perception model extracts the voiceprint feature from the input audio feature sample, and performs speaker recognition training based on the extracted audio feature containing the voiceprint feature.

[0120] Specifically, when training the voiceprint perception model described above, input the audio feature of the sample audio to be extracted into the voiceprint perception model, extract the voiceprint feature from the audio feature by the voiceprint perception model, and perform speaker recognition based on the voiceprint feature output by the voiceprint perception model. When the voiceprint feature output by the voiceprint perception model can accurately identify the speaker, it can be considered that the voiceprint perception model can accurately extract the voiceprint feature of the speaker.

[0121] In another embodiment, the voiceprint perception model described above is stacked by three identical residual convolutional coding units. Referring to Figure 7 As shown, the residual convolutional coding unit is composed of a two-dimensional convolution layer, a layer normalization layer, a ReLU activation function, an FSMN (Feedforward Sequential Memory Networks) unit, a layer normalization layer and a residual module. Based on the structure of the voiceprint perception model described above, the voiceprint perception model extracts the voiceprint feature of the human voice from the first audio feature to obtain the second audio feature containing the voiceprint feature, which specifically includes the following steps B1-B4:

[0122] B1, the two-dimensional convolution layer performs convolution operation on the first audio feature by a set of two-dimensional convolution filters with size k x k to obtain a local feature vector.

[0123] B2, the layer normalization layer and the ReLU activation function process the local feature vector obtained in step B1 to obtain a second feature vector.

[0124] B3, the FSMN unit with size h is used to process the second feature vector obtained in step B2 to obtain a third feature vector.

[0125] B4, the layer normalization layer and the residual module are used to process the third feature vector and superimpose the processing result with the first audio feature to obtain the second audio feature containing the voiceprint feature.

[0126] In another embodiment, the second feature modeling module fuses the first audio feature and the second audio feature by first concatenating the first audio feature and the second audio feature to obtain a concatenated feature, and then jointly modeling the concatenated feature in the time dimension and the frequency dimension to obtain a third audio feature.

[0127] That is, the first audio feature and the second audio feature are first concatenated to realize the merging of the two. Then, the concatenated feature is contextually jointly modeled in the time dimension and the frequency dimension, so as to fully fuse the contextual feature information in the concatenated feature from the time dimension and the frequency dimension to obtain a third audio feature.

[0128] In another embodiment, the second feature modeling module described above can adopt the structure of the intra-subband and inter-subband modeling module shown in Figure 5 Based on the structure described above, when the second feature modeling module jointly models the concatenated feature in the time dimension and the frequency dimension, it specifically includes the processing steps of C1-C4:

[0129] C1, using a layer normalization layer and a bidirectional GRU layer in the time dimension to perform contextually joint modeling processing on the concatenated feature after the first audio feature and the second audio feature are concatenated in the time dimension, and K subbands share the same bidirectional GRU layer to obtain a first fused feature.

[0130] C2, using a fully connected layer to perform dimension reduction processing on the first fused feature obtained in step C1, keeping the input channel unchanged with the channel of the concatenated feature after the first audio feature and the second audio feature are concatenated, and then superimposing the first fused feature after the dimension reduction processing and the concatenated feature to obtain a second fused feature.

[0131] C3, using a layer normalization layer and a bidirectional GRU layer in the frequency dimension to perform contextually joint modeling processing on the second fused feature obtained in step C2 in the subband frequency dimension, and the time dimension shares the same bidirectional GRU layer to obtain a third fused feature.

[0132] C4, using a fully connected layer to perform dimension reduction processing on the third fused feature obtained in step C3, keeping the input channel unchanged with the channel of the second fused feature obtained in step C2, and then superimposing the third fused feature after the dimension reduction processing and the second fused feature to obtain a third audio feature.

[0133] In another embodiment, the second feature modeling module described above adopts the form of two intra-subband and inter-subband modeling modules in series, that is, the second feature modeling module is composed of two layers of intra-subband and inter-subband modeling modules. Figure 6

[0134] ​Based on the structure form, the first layer sub-band and sub-band modeling module performs the above C1-C4 processing on the spliced features after the first audio feature and the second audio feature are spliced, then the first layer sub-band and sub-band modeling module superimposes the third fusion features after dimension reduction processing and the second fusion features, and takes the superimposed fusion features as the input of the second layer sub-band and sub-band modeling module. The second layer sub-band and sub-band modeling module processes the superimposed fusion features according to the above steps C1-C4, and finally obtains the third audio feature.

[0135] In this way, through multiple sub-band and sub-band modeling processing, the final obtained third audio feature contains more feature information, that is, the third audio feature is more accurate.

[0136] In other embodiments, the second feature modeling module can also be composed of more layers of sub-band and sub-band modeling modules, so as to realize more in-depth feature extraction and obtain more accurate third audio features.

[0137] In another embodiment, referring to Figure 8 As shown in the figure, the mask estimation module is composed of K multi-layer perception units, each of which is composed of a layer normalization layer, two fully connected layers, a ReLU activation function, and a sigmoid activation function.

[0138] Based on the above structure form, the mask estimation module generates a vocal mask and a background mask based on the third audio feature, and specifically includes the following steps D1-D4 processing process:

[0139] D1, the third audio feature is divided into K sub-bands to obtain the respective features corresponding to each sub-band. Among them, the sub-band division method of the third audio feature is the same as the sub-band division method of the audio feature of the mixed audio introduced in the above embodiment.

[0140] D2, the layer normalization layer respectively normalizes the respective features corresponding to the K sub-bands obtained in step D1 to obtain K normalized features.

[0141] D3, the K fully connected layers and the ReLU activation function are used to process the K normalized features obtained in step D2 to obtain K hidden layer feature vectors.

[0142] D4, the K fully connected layers and the sigmoid activation function are used to process the K hidden layer feature vectors obtained in step D3 to obtain K mask values, each of which includes a vocal mask and a background mask. Finally, the estimation results of the respective vocal masks and background masks corresponding to the K sub-bands are merged to obtain the final vocal mask and background mask.

[0143] Corresponding to the audio separation method described above, the embodiments of the present application also provide an audio separation device, as shown in Figure 9 The device comprises:

[0144] A first feature processing unit 100 is configured to perform feature extraction processing on mixed audio to obtain first audio features; the mixed audio comprises human voice audio and background audio;

[0145] A voiceprint feature extraction unit 110 is configured to obtain second audio features containing voiceprint features by extracting voiceprint features of human voice from the first audio features;

[0146] A second feature processing unit 120 is configured to fuse the first audio features and the second audio features to obtain third audio features;

[0147] An audio separation unit 130 is configured to extract human voice audio and background audio from the mixed audio based on the third audio features.

[0148] In some implementations, the audio separation unit 130 extracts human voice audio and background audio from the mixed audio based on the third audio features, comprising:

[0149] Generating human voice masks and background masks based on the third audio features;

[0150] Extracting human voice audio and background audio from the mixed audio based on the human voice masks and the background masks.

[0151] In some implementations, the first feature processing unit 100 performs feature extraction processing on mixed audio to obtain first audio features, the voiceprint feature extraction unit 110 obtains second audio features containing voiceprint features by extracting voiceprint features of human voice from the first audio features, the second feature processing unit 120 fuses the first audio features and the second audio features to obtain third audio features, and the audio separation unit 130 generates human voice masks and background masks based on the third audio features, comprising:

[0152] Inputting audio features of mixed audio into an audio separation model to obtain human voice masks and background masks output by the audio separation model;

[0153] The audio separation model comprises:

[0154] A first feature modeling module is configured to perform feature extraction processing on mixed audio to obtain first audio features;

[0155] a voiceprint perception module configured to obtain second audio features containing voiceprint features by extracting the voiceprint features of human voice from the first audio features;

[0156] a second feature modeling module configured to obtain third audio features by fusing the first audio features and the second audio features;

[0157] a mask estimation module configured to generate human voice masks and background masks based on the third audio features.

[0158] In some implementations, the first feature processing unit 100 performs feature extraction processing on mixed audio to obtain first audio features, including:

[0159] performing sub-band division on audio features of the mixed audio to obtain a plurality of sub-band audio features;

[0160] performing joint modeling of the time dimension and the frequency dimension on the plurality of sub-band audio features to obtain the first audio features.

[0161] In some implementations, the first feature processing unit 100 performs joint modeling of the time dimension and the frequency dimension on the plurality of sub-band audio features to obtain the first audio features, including:

[0162] performing context joint modeling processing of the time dimension on the plurality of sub-band audio features to obtain first global features;

[0163] performing dimension reduction processing on the first global features, and superimposing the first global features after the dimension reduction processing and the plurality of sub-band audio features to obtain second global features;

[0164] performing context joint modeling processing of the frequency dimension on the second global features to obtain third global features;

[0165] performing dimension reduction processing on the third global features, and superimposing the third global features after the dimension reduction processing and the second global features to obtain the first audio features.

[0166] In some implementations, the first feature processing unit 100 performs sub-band division on audio features of mixed audio to obtain a plurality of sub-band audio features, including:

[0167] performing sub-band division on the audio features of the mixed audio according to a relationship that the bandwidth of the divided sub-band audio features is proportional to the frequency of the sub-band audio features to obtain a plurality of sub-band audio features.

[0168] In some implementations, the voiceprint feature extraction unit 110 obtains second audio features containing voiceprint features by extracting the voiceprint features of human voice from the first audio features, including:

[0169] inputting the first audio feature into a voiceprint perception model, obtaining a second audio feature containing a voiceprint feature by extracting the voiceprint feature of the human voice from the first audio feature through the voiceprint perception model;

[0170] The voiceprint perception model is obtained by extracting a voiceprint feature from an input audio feature sample and performing speaker recognition training based on the extracted audio feature containing the voiceprint feature.

[0171] In some implementations, the second feature processing unit 120 fuses the first audio feature and the second audio feature to obtain a third audio feature, including:

[0172] splicing the first audio feature and the second audio feature to obtain a spliced feature;

[0173] jointly modeling the spliced feature in time dimension and frequency dimension to obtain a third audio feature.

[0174] In some implementations, the second feature processing unit 120 jointly models the spliced feature in time dimension and frequency dimension to obtain a third audio feature, including:

[0175] context joint modeling processing the spliced feature in time dimension to obtain a first fused feature;

[0176] dimension reduction processing the first fused feature, and superimposing the first fused feature after the dimension reduction processing and the spliced feature to obtain a second fused feature;

[0177] context joint modeling processing the second fused feature in frequency dimension to obtain a third fused feature;

[0178] dimension reduction processing the third fused feature, and superimposing the third fused feature after the dimension reduction processing and the second fused feature to obtain a third audio feature.

[0179] In some implementations, the audio separation unit 130 generates a human voice mask and a background mask based on the third audio feature, including:

[0180] extracting a feature corresponding to each subband from the third audio feature; each subband is determined by subband division on the audio feature of the mixed audio;

[0181] determining an estimation result of the human voice mask and the background mask corresponding to each subband based on the feature corresponding to each subband;

[0182] The estimation results of the respective sub-band corresponding voice mask and background mask are combined to determine the voice mask and the background mask.

[0183] The audio separation device provided by the embodiment is of the same application concept as the audio separation method provided by the above-mentioned embodiments of the present application, can execute the audio separation method provided by any of the above-mentioned embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method. The technical details not described in detail in the embodiment can be seen from the specific processing content of the audio separation method provided by the above-mentioned embodiments of the present application, and will not be described here.

[0184] The functions implemented by each of the above units can be implemented by the same or different processors, and the embodiments of the present application are not limited.

[0185] It should be understood that the units in the above device can be implemented in the form of processor calling software. For example, the device includes a processor connected with a memory, the memory stores instructions, and the processor calls the instructions stored in the memory to implement any of the above methods or to implement the functions of the units of the device, wherein the processor can be a general processor, such as a CPU or a microprocessor, and the memory can be an internal memory or an external memory of the device. Alternatively, the units in the device can be implemented in the form of hardware circuit. The functions of part or all of the units can be implemented by designing the hardware circuit. The hardware circuit can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of part or all of the units are implemented by designing the logical relationship of elements in the circuit. For another example, in another implementation, the hardware circuit can be implemented by a PLD. Taking an FPGA as an example, it can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to implement the functions of part or all of the units. All units of the above device can be implemented in the form of processor calling software, or all units can be implemented in the form of hardware circuit, or part of the units can be implemented in the form of processor calling software, and the remaining part can be implemented in the form of hardware circuit.

[0186] In the embodiments of the present application, the processor is a circuit with signal processing capability. In one implementation, the processor can be a circuit with instruction reading and running capability, such as a CPU, a microprocessor, a GPU, or a DSP, etc. In another implementation, the processor can implement certain functions through a logic relationship of a hardware circuit, which is fixed or can be reconfigured. For example, the processor is a hardware circuit implemented by an ASIC or a PLD, such as an FPGA, etc. In the reconfigurable hardware circuit, the processor loads a configuration document to implement the hardware circuit configuration. It can be understood that the processor loads instructions to implement the functions of the above units.

[0187] It can be seen that each unit in the above apparatus can be one or more processors (or processing circuits) configured to implement the above methods, such as a CPU, a GPU, an NPU, a TPU, a DPU, a microprocessor, a DSP, an ASIC, an FPGA, or a combination of at least two of these processor forms.

[0188] In addition, each unit in the above apparatus can be integrated together or can be independently implemented. In one implementation, these units are integrated together to implement a SOC. The SOC can include at least one processor for implementing any of the above methods or functions of the units of the apparatus. The at least one processor can be different, such as including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0189] Another embodiment of the present application also proposes an electronic device, as shown in Figure 10 The device includes:

[0190] a memory 200 and a processor 210;

[0191] The memory 200 is connected with the processor 210, and is configured to store programs.

[0192] The processor 210 is configured to implement the audio separation method disclosed in any of the above embodiments by running the programs stored in the memory 200.

[0193] Specifically, the above electronic device can further include a bus, a communication interface 220, an input device 230, and an output device 240.

[0194] The processor 210, the memory 200, the communication interface 220, the input device 230, and the output device 240 are connected with each other through the bus. Among them:

[0195] The bus can include a path for communicating information among the various components of the computer system.

[0196] The processor 210 can be a general purpose processor, such as a central processing unit (CPU), a microprocessor, or the like, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of programs of the present application. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components.

[0197] The processor 210 can include a main processor, and can further include a baseband chip, a modem, or the like.

[0198] The memory 200 stores programs for implementing the technical solutions of the present application, and can also store an operating system and other key services. Specifically, the programs can include program codes, and the program codes include computer operation instructions. More specifically, the memory 200 can include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk memory, a flash, and the like.

[0199] The input device 230 can include a device that receives data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor, and the like.

[0200] The output device 240 can include a device that allows information to be output to a user, such as a display screen, a printer, a speaker, and the like.

[0201] The communication interface 220 can include a device using any transceiver to communicate with other devices or communication networks, such as an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), and the like.

[0202] The processor 210 executes the programs stored in the memory 200 and calls other devices, which can be used to implement each step of any one of the audio separation methods provided by the above-mentioned embodiments of the present application.

[0203] The embodiment of the present application also provides a chip, which comprises a processor and a data interface, wherein the processor reads and runs a program stored on a memory through the data interface to execute the audio separation method described in any of the above embodiments, and the specific processing process and advantages can be referred to the description of the audio separation method.

[0204] In addition to the above method and device, the embodiment of the present application can also be a computer program product, which comprises computer program instructions, and the computer program instructions make the processor execute the steps in the audio separation method described in any of the above embodiments of the present application when the processor runs.

[0205] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0206] In addition, the embodiment of the present application can also be a storage medium, which stores a computer program, and the computer program makes the processor execute the steps in the audio separation method described in any of the above embodiments of the present application.

[0207] For each of the above method embodiments, in order to simply describe, it is expressed as a combination of a series of actions, but those skilled in the art should know that the present application is not limited to the order of the described actions, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application.

[0208] It should be noted that each of the embodiments in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0209] The steps in the method of each embodiment of the present application can be adjusted, combined and reduced in sequence according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0210] The modules and sub-modules in the device and terminal of the embodiments of the present application can be combined, divided, and deleted according to actual needs.

[0211] In several embodiments provided by the present application, it should be understood that the disclosed terminal, device, and method can be implemented by other manners. For example, the terminal embodiments described above are only illustrative, for example, the division of the modules or sub-modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of sub-modules or modules can be combined or integrated into another module, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed ones can be indirect coupling or communication connection through some interfaces, devices, or modules, which can be electrical, mechanical, or other forms.

[0212] The modules or sub-modules described as separate components can or can not be physically separated, and the components as modules or sub-modules can or can not be physical modules or sub-modules, that is, can be located in one place, or can be distributed to a plurality of network modules or sub-modules. Part or all of the modules or sub-modules can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0213] In addition, the functional modules or sub-modules in each embodiment of the present application can be integrated in one processing module, or each module or sub-module can exist physically, or two or more modules or sub-modules can be integrated in one module. The integrated module or sub-module can be realized in the form of hardware or software functional module or sub-module.

[0214] The skilled person can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software, or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in general terms in the above description. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0215] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be directly implemented by hardware, software units executed by a processor, or a combination of both. The software units can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0216] Finally, it should be noted that, in the description above, relative terms such as first and second, etc. are merely used to distinguish one entity or action from another, without necessarily requiring or implying any such actual relationship or order between such entities or actions. Also, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0217] The above description of disclosed embodiments provides enabling teaching for making or using the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An audio separation method, characterized by, The method comprises: performing feature extraction processing on the mixed audio to obtain first audio features; the mixed audio comprises human voice audio and background audio; extracting a voiceprint feature of the human voice from the first audio features to obtain second audio features comprising the voiceprint feature; concatenating the first audio features and the second audio features to obtain concatenated features, and jointly modeling the concatenated features in the time dimension and the frequency dimension to obtain third audio features; generating a human voice mask and a background mask based on the third audio features; extracting human voice audio and background audio from the mixed audio based on the human voice mask and the background mask.

2. The method of claim 1, wherein, The method comprises: performing feature extraction processing on the mixed audio to obtain first audio features, extracting a voiceprint feature of the human voice from the first audio features to obtain second audio features comprising the voiceprint feature, concatenating the first audio features and the second audio features to obtain concatenated features, and jointly modeling the concatenated features in the time dimension and the frequency dimension to obtain third audio features; generating a human voice mask and a background mask based on the third audio features, comprising: inputting audio features of the mixed audio into an audio separation model to obtain a human voice mask and a background mask output by the audio separation model; The audio separation model comprises: a first feature modeling module configured to perform feature extraction processing on the mixed audio to obtain first audio features; a voiceprint perception module configured to extract a voiceprint feature of the human voice from the first audio features to obtain second audio features comprising the voiceprint feature; a second feature modeling module configured to concatenate the first audio features and the second audio features to obtain concatenated features, and jointly model the concatenated features in the time dimension and the frequency dimension to obtain third audio features; 3. The method according to claim 1 or 2, characterized in that, a mask estimation module configured to generate a human voice mask and a background mask based on the third audio features. The method comprises: performing sub-band division on the audio features of the mixed audio to obtain a plurality of sub-band audio features; 4. The method of claim 3, wherein, jointly modeling the plurality of sub-band audio features in the time dimension and the frequency dimension to obtain the first audio features. The method comprises: performing context joint modeling processing in the time dimension on the plurality of sub-band audio features to obtain first global features; performing dimension reduction processing on the first global features, and superimposing the first global features after the dimension reduction processing and the plurality of sub-band audio features to obtain second global features; performing context joint modeling processing in the frequency dimension on the second global features to obtain third global features; 5. The method of claim 3, wherein, performing dimension reduction processing on the third global features, and superimposing the third global features after the dimension reduction processing and the second global features to obtain the first audio features. The method comprises: performing sub-band division on the audio features of the mixed audio to obtain a plurality of sub-band audio features; Based on the principle that the bandwidth of the sub-band audio features is proportional to the frequency of the sub-band audio features, the audio features of the mixed audio are divided into sub-bands to obtain multiple sub-band audio features.

6. The method of claim 1 or 2, wherein, The step of extracting voiceprint features from the first audio features to obtain a second audio feature containing the voiceprint features includes: The first audio feature is input into the voiceprint perception model, and the voiceprint perception model extracts the voiceprint features of the human voice from the first audio feature to obtain a second audio feature containing the voiceprint features. The voiceprint perception model is obtained by extracting voiceprint features from input audio feature samples and training speaker recognition based on the extracted audio features containing voiceprint features.

7. The method of claim 1, wherein, The joint modeling of the spliced ​​features in terms of time and frequency dimensions yields a third audio feature, including: The spliced ​​features are subjected to contextual joint modeling in the time dimension to obtain the first fused feature; The first fusion feature is subjected to dimensionality reduction processing, and the dimensionality-reduced first fusion feature is superimposed with the spliced ​​feature to obtain the second fusion feature; The second fused feature is subjected to context joint modeling processing along the frequency dimension to obtain the third fused feature; The third fusion feature is subjected to dimensionality reduction processing, and the dimensionality-reduced third fusion feature is superimposed with the second fusion feature to obtain the third audio feature.

8. The method of claim 1 or 2, wherein, The generation of the voice mask and background mask based on the third audio feature includes: From the third audio features, features corresponding to each sub-band are extracted; each sub-band is determined by dividing the audio features of the mixed audio into sub-bands. Based on the features corresponding to each sub-band, the estimation results of the voice mask and background mask corresponding to each sub-band are determined. The estimation results of the voice mask and background mask corresponding to each sub-band are merged to determine the voice mask and background mask.

9. An audio separating apparatus, characterized by comprising: include: The first feature processing unit is used to perform feature extraction processing on the mixed audio to obtain the first audio feature; the mixed audio includes human voice audio and background audio. The voiceprint feature extraction unit is used to extract the voiceprint features of a human voice from the first audio feature to obtain a second audio feature containing the voiceprint features. The second feature processing unit is used to concatenate the first audio feature and the second audio feature to obtain a concatenated feature, and to perform joint modeling of the concatenated feature in terms of time dimension and frequency dimension to obtain a third audio feature. An audio separation unit is used to generate a voice mask and a background mask based on the third audio feature; Based on the human voice mask and the background mask, human voice audio and background audio are extracted from the mixed audio.

10. An electronic device, comprising: Including memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the audio separation method as described in any one of claims 1 to 8 by running a program in the memory.

11. A storage medium, characterized by The storage medium stores a computer program, which, when executed by a processor, implements the audio separation method as described in any one of claims 1 to 8.

12. A computer program product, characterised in that, comprising computer program instructions which, when executed by a processor, cause the processor to perform the audio separation method of any one of claims 1 to 8.

Images