Sound source separation method and device and storage medium

Abstract

The invention relates to a sound source separation method and device and a storage medium. The sound source separation method comprises the following steps: acquiring frequency domain characteristics of mixed audio; sequentially dividing the frequency domain features into a plurality of sub-band frequency domain features according to different frequency bands; fusing each sub-band frequency domain feature in the plurality of sub-band frequency domain features with a corresponding global sub-band frequency domain feature according to a division sequence, performing global sub-band frequency domain feature identification on the global sub-band frequency domain feature fused with the sub-band frequency domain feature, and taking an identification result as a global sub-band frequency domain feature fused with the next sub-band frequency domain feature, repeating the processes of fusion and global sub-band frequency domain feature recognition until global sub-band frequency domain feature recognition of recognition times is completed, and obtaining a sub-band frequency domain feature recognition result of the mixed audio; and based on the sub-band frequency domain feature identification result of the mixed audio, sound source separation is carried out on different frequency band sound sources included in the mixed audio. According to the invention, separation of multi-band sound sources can be realized.

Description

Technical Field

[0001] The present disclosure relates to the field of sound source processing, and in particular to a sound source separation method, device and storage medium. Background Art

[0002] Sound source separation technology is a sound source processing method that extracts specified types of signals from a mixed audio signal.

[0003] In the related technologies, the sound source separation technology based on deep learning methods can usually be divided into mapping networks and masking networks, which are prone to residual noise or excessive separation leading to distortion of audio or a certain frequency band. At the same time, the mainstream deep learning framework has the problem of large model size and large computational complexity, making it difficult to perform real-time separation tasks on processors such as the central processing unit (CPU) and graphics processing unit (GPU) of small smart mobile terminals. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a sound source separation method, device and storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, a sound source separation method is provided, the sound source separation method comprising: obtaining frequency domain features of mixed audio, the mixed audio including audio of different frequency band sound sources; dividing the frequency domain features into a plurality of sub-band frequency domain features in sequence according to different frequency bands; fusing each sub-band frequency domain feature of the plurality of sub-band frequency domain features with the corresponding global sub-band frequency domain feature in the order of division, performing global sub-band frequency domain feature recognition on the global sub-band frequency domain feature fused with the sub-band frequency domain feature, and using the recognition result as a signal to be fused with the next sub-band frequency domain feature. The above fusion and global sub-band frequency domain feature recognition process is repeated until the global sub-band frequency domain feature recognition for the number of recognitions is completed to obtain the sub-band frequency domain feature recognition result of the mixed audio; wherein, the N-th recognition of the global sub-band frequency domain feature corresponds to the division of the N-th sub-band frequency domain feature, and N is a positive integer greater than or equal to 1 and less than the number of recognitions of the global sub-band frequency domain feature; based on the sub-band frequency domain feature recognition result of the mixed audio, the different frequency band sound sources included in the mixed audio are separated.

[0006] In one embodiment, the frequency domain features are divided into a plurality of sub-band frequency domain features in accordance with different frequency bands, including: the frequency domain features are respectively input into different sub-band division networks in a sound source separation network to obtain a plurality of sub-band frequency domain features; the different sub-band division networks are used to divide the sub-band frequency domain features corresponding to different frequency bands.

[0007] In one embodiment, the global sub-band frequency domain feature recognition is based on the global sub-band frequency domain feature recognition network recognition in the sound source separation network; the global sub-band frequency domain feature recognition network includes N+1 groups of sub-networks, and each of the first N groups of sub-networks in the N+1 groups of sub-networks includes a batch normalization layer, a one-dimensional convolution layer, a two-dimensional convolution layer, and an activation function; the N+1th group of sub-networks in the N+1 groups of sub-networks includes a batch normalization layer and a one-dimensional convolution layer.

[0008] In one embodiment, the sound source separation of the different frequency band sound sources included in the mixed audio based on the sub-band frequency domain feature recognition result of the mixed audio comprises: performing sound source separation on the sub-band frequency domain feature recognition result of the mixed audio based on the time convolution network TCN in the sound source separation network within the mth period, and obtaining a first separation result of sound source separation of the different frequency band sound sources included in the mixed audio, wherein the time length corresponding to the first separation result is the time length having a first degree of overlap with the mth period from the starting time of the mth period, and the m is a positive integer greater than or equal to 1; starting from the end time of the duration corresponding to the first separation result, the sub-band frequency domain feature recognition result of the mixed audio is separated based on the TCN within the m+1th period to obtain a second separation result of sound source separation of the different frequency band sound sources included in the mixed audio, and the duration of the second separation result is the duration having the first degree of overlap with the m+1th period from the start time of the m+1th period; the above process is repeated until the separation of the different frequency band sound sources included in the mixed audio is completed.

[0009] In one embodiment, the sound source separation network is pre-trained in the following manner: a first mixed audio training data set and a second mixed audio training data set are obtained, wherein the first mixed audio training data set includes multi-band sound source mixed audio data collected in an environment doped with noise data, and the second mixed audio training data set includes audio data obtained by mixing multi-band sound source data collected in an environment without noise data; model training is performed based on the first mixed audio training data set and the second mixed audio training data set to obtain a sound source separation network.

[0010] In one embodiment, the model training is performed based on the first mixed audio training data set and the second mixed audio training data set to obtain a sound source separation network, including: determining a first loss function corresponding to the first mixed audio training data set, and determining a second loss function corresponding to the second mixed audio training data set; weighting the first loss function and the second loss function, and performing model training constraints based on the weighted loss function to obtain a sound source separation network; wherein the weight of the first loss function is greater than the weight of the second loss function.

[0011] In one implementation, the first loss function and the second loss function are determined based on a mean square error loss function and a signal-to-noise ratio loss function.

[0012] In one implementation, the audio of the sound sources in different frequency bands includes audio corresponding to a human voice frequency band and audio corresponding to an accompaniment frequency band.

[0013] According to a second aspect of an embodiment of the present disclosure, a sound source separation device is provided, comprising: an acquisition unit, configured to acquire frequency domain features of a mixed audio, wherein the mixed audio includes audio of sound sources of different frequency bands; a division unit, configured to divide the frequency domain features into a plurality of sub-band frequency domain features in sequence according to different frequency bands; an identification unit, configured to fuse each sub-band frequency domain feature of the plurality of sub-band frequency domain features with the corresponding global sub-band frequency domain feature in the order of division, perform global sub-band frequency domain feature identification on the global sub-band frequency domain feature fused with the sub-band frequency domain feature, and use the identification result as a match to the next sub-band frequency domain feature The global sub-band frequency domain features of the mixed audio are fused, and the above fusion and global sub-band frequency domain feature recognition processes are repeatedly performed until the global sub-band frequency domain feature recognition for the number of recognitions is completed to obtain the sub-band frequency domain feature recognition result of the mixed audio; wherein the N-th recognition of the global sub-band frequency domain feature corresponds to the division of the N-th sub-band frequency domain feature, and N is a positive integer greater than or equal to 1 and less than the number of recognitions of the global sub-band frequency domain feature; a separation unit is used to perform sound source separation on the different frequency band sound sources included in the mixed audio based on the sub-band frequency domain feature recognition result of the mixed audio.

[0014] In one embodiment, the division unit divides the frequency domain features into multiple sub-band frequency domain features in sequence according to different frequency bands in the following manner: the frequency domain features are respectively input into different sub-band division networks in the sound source separation network to obtain multiple sub-band frequency domain features; the different sub-band division networks are used to divide the sub-band frequency domain features corresponding to different frequency bands.

[0015] In one embodiment, the global sub-band frequency domain feature recognition is based on the global sub-band frequency domain feature recognition network recognition in the sound source separation network; the global sub-band frequency domain feature recognition network includes N+1 groups of sub-networks, and each of the first N groups of sub-networks in the N+1 groups of sub-networks includes a batch normalization layer, a one-dimensional convolution layer, a two-dimensional convolution layer, and an activation function; the N+1th group of sub-networks in the N+1 groups of sub-networks includes a batch normalization layer and a one-dimensional convolution layer.

[0016] In one embodiment, the separation unit performs sound source separation on the different frequency band sound sources included in the mixed audio based on the sub-band frequency domain feature recognition result of the mixed audio in the following manner: within the mth period, the sub-band frequency domain feature recognition result of the mixed audio is performed based on the time convolution network TCN in the sound source separation network to obtain a first separation result of sound source separation on the different frequency band sound sources included in the mixed audio, and the duration corresponding to the first separation result is the duration having a first degree of overlap with the mth period from the start time of the mth period. , where m is a positive integer greater than or equal to 1; starting from the end time of the duration corresponding to the first separation result, performing sound source separation on the sub-band frequency domain feature recognition result of the mixed audio based on the TCN within the m+1th cycle, and obtaining a second separation result of sound source separation of the different frequency band sound sources included in the mixed audio, wherein the duration of the second separation result is the duration having the first degree of overlap with the m+1th cycle from the start time of the m+1th cycle; repeating the above process until the separation of the different frequency band sound sources included in the mixed audio is completed.

[0017] In one embodiment, the sound source separation device also includes: a training unit, which is used to train the sound source separation network in the following manner: obtaining a first mixed audio training data set and a second mixed audio training data set, the first mixed audio training data set including multi-band sound source mixed audio data collected in an environment doped with noise data, and the second mixed audio training data set including audio data obtained by mixing multi-band sound source data collected in an environment without noise data; performing model training based on the first mixed audio training data set and the second mixed audio training data set to obtain the sound source separation network.

[0018] In one embodiment, the training unit performs model training based on the first mixed audio training data set and the second mixed audio training data set to obtain a sound source separation network in the following manner: determine a first loss function corresponding to the first mixed audio training data set, and determine a second loss function corresponding to the second mixed audio training data set; weight the first loss function and the second loss function, and based on the weighted loss function, perform model training constraints to obtain a sound source separation network; wherein the weight of the first loss function is greater than the weight of the second loss function.

[0019] In one implementation, the first loss function and the second loss function are determined based on a mean square error loss function and a signal-to-noise ratio loss function.

[0020] In one implementation, the audio of the sound sources in different frequency bands includes frequency domain features of the audio corresponding to the human voice frequency band and the audio corresponding to the accompaniment frequency band.

[0021] According to a third aspect of the present disclosure, a sound source separation device is provided, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to: execute the sound source separation method described in the aforementioned first aspect or any one of the embodiments of the first aspect.

[0022] According to a fourth aspect of the present disclosure, a storage medium is provided, wherein the storage medium stores instructions, and when the instructions are executed by a processor, the sound source separation method described in the first aspect or any one of the embodiments of the first aspect is executed.

[0023] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: audio including sound sources of different frequency bands is divided into multiple sub-band frequency domain features, and is sequentially fused with global sub-band frequency domain features to obtain a sub-band frequency domain feature recognition result of the mixed audio, and a more distinguishing feature representation is extracted. Sound source separation is performed based on this recognition result, so as to achieve more accurate separation of audio including sound sources of different frequency bands.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0026] Figure 1 The figure is a flow chart of a sound source separation method according to an exemplary embodiment.

[0027] Figure 2 The figure is a flow chart of a sub-band division method according to an exemplary embodiment.

[0028] Figure 3 The diagram is a schematic diagram of a global sub-band frequency domain feature recognition process according to an exemplary embodiment.

[0029] Figure 4 The present invention is a flowchart of a method for performing sound source separation on sub-band frequency domain feature recognition results based on TCN according to an exemplary embodiment.

[0030] Figure 5 The present invention is a flowchart of a method for training a sound source separation network according to an exemplary embodiment.

[0031] Figure 6 The invention is a flowchart of a method for determining a weighted loss function and constraining model training according to an exemplary embodiment.

[0032] Figure 7 It is a schematic diagram of a sound source separation network training process in a vocal accompaniment separation scenario according to an exemplary embodiment.

[0033] Figure 8 The present invention is a block diagram of a sound source separation device according to an exemplary embodiment.

[0034] Fig. 9 The present invention is a block diagram of a sound source separation device according to an exemplary embodiment. DETAILED DESCRIPTION

[0035] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.

[0036] The sound source separation method provided by the embodiments of the present disclosure can be applied to application scenarios involving audio separation of sound sources of different frequency bands, for example, to the scenario of separating human voice and accompaniment in mixed audio.

[0037] In the related technology, the sound source separation method based on deep learning method can generally be divided into mapping network and masking network. The effect of mapping network in separating human voice is slightly poor, and it is easy to cause residual noise; while the masking network can achieve a relatively clean separation effect, it may be prone to over-separation, resulting in distortion or distortion of a certain frequency band. At the same time, mainstream deep learning frameworks such as Transformer network, Unet network, MobileNet-v2 network, Conformer network, etc. are large in model size and computational complexity, making it difficult to perform real-time separation tasks on CPU, GPU and other processors of small smart mobile terminals.

[0038] In view of this, an embodiment of the present disclosure provides a sound source separation method. In the sound source separation method provided by the present disclosure, the audio including sound sources of different frequency bands is divided into a plurality of sub-band frequency domain features, and is sequentially fused with the global sub-band frequency domain features to obtain a sub-band frequency domain feature recognition result of the mixed audio. Based on the recognition result, the sound sources of the different frequency bands included in the mixed audio are separated.

[0039] Figure 1 The figure is a flow chart of a sound source separation method according to an exemplary embodiment.

[0040] In step S11, the frequency domain features of the mixed audio are obtained.

[0041] The mixed audio includes audios of different frequency bands, for example, audios containing human voice and accompaniment.

[0042] In step S12, the frequency domain features are divided into a plurality of sub-band frequency domain features in sequence according to different frequency bands.

[0043] The frequency domain features of the audio signal are divided into several sub-band frequency domain features of different frequency band intervals, each of which represents the audio information within a specific frequency range, so as to analyze and process the audio frequency domain features of different frequency ranges. The frequency band division intervals are determined during the training process of the sound source separation network.

[0044] In step S13, each sub-band frequency domain feature of the multiple sub-band frequency domain features is fused with the corresponding global sub-band frequency domain feature in the order of division, and the global sub-band frequency domain feature fused with the sub-band frequency domain feature is identified, and the identification result is used as the global sub-band frequency domain feature to be fused with the next sub-band frequency domain feature.

[0045] For example, the frequency domain features are divided into N sub-band frequency domain features in sequence according to different frequency bands, where N is a positive integer greater than or equal to 1. The first sub-band frequency domain feature in the division order is fused with the corresponding global sub-band frequency domain feature, where the global sub-band frequency domain feature corresponding to the first sub-band frequency domain feature is the frequency domain feature of the original mixed audio. The global sub-band frequency domain feature fused with the first sub-band frequency domain feature is identified for the first time, and the identification result is used as the global sub-band frequency domain feature corresponding to the second sub-band frequency domain feature and is fused with the second sub-band frequency domain feature. The global sub-band frequency domain feature fused with the first sub-band frequency domain feature and the second sub-band frequency domain feature is identified for the second time, and the identification result is used as the global sub-band frequency domain feature corresponding to the third sub-band frequency domain feature and is fused with the third sub-band frequency domain feature. By analogy, the result of each global sub-band frequency domain feature recognition is used as the global sub-band frequency domain feature to be fused with the next sub-band frequency domain feature.

[0046] In step S14, the above fusion and global sub-band frequency domain feature recognition process is repeated until the global sub-band frequency domain feature recognition of the recognition times is completed, and the sub-band frequency domain feature recognition result of the mixed audio is obtained.

[0047] Among them, the Nth recognition of the global sub-band frequency domain feature corresponds to the division of the Nth sub-band frequency domain feature. In each global sub-band frequency domain feature recognition process, each sub-band frequency domain feature divided by frequency band is fused with the global sub-band frequency domain feature. After the fused feature is recognized, the fused feature is used as the global sub-band frequency domain feature fused with the next sub-band frequency domain feature. This global sub-band frequency domain feature recognition process is repeated N times until the recognition result includes each sub-band frequency domain feature divided by frequency band, and the final sub-band frequency domain feature recognition result of the mixed audio is obtained.

[0048] In step S15, based on the sub-band frequency domain feature recognition result of the mixed audio, sound source separation is performed on the different frequency band sound sources included in the mixed audio.

[0049] By performing time domain modeling and feature extraction on the sub-band frequency domain feature recognition results of the mixed audio, the sound sources of different frequency bands in the mixed audio can be separated, thereby obtaining the separated audio signal.

[0050] In the sound source separation method provided by the embodiment of the present disclosure, the frequency domain features of the audio including sound sources of different frequency bands are obtained, and the obtained frequency domain features are divided into a plurality of sub-band frequency domain features in sequence according to different frequency bands. The sub-band frequency domain features are fused with the global sub-band frequency domain features identified by the global sub-band frequency domain features in the order of division, and the fused frequency domain features are learned and transmitted as the global sub-band frequency domain features to be fused with the sub-band frequency domain features of the next frequency band. The global sub-band frequency domain feature recognition is completed until the number of recognitions is completed, and the sub-band frequency domain feature recognition result of the mixed audio is obtained. This recognition result is a more discriminative feature representation, which prepares for further separation of the sound source.

[0051] The sound source separation method provided by the embodiment of the present disclosure can effectively separate the human voice and accompaniment in the mixed audio by dividing the sound source separation network of the sub-band frequency domain characteristics. At the same time, the sound source separation method provided by the embodiment of the present disclosure adopts a lightweight network structure and an optimized model design, which can meet the needs of sound source separation on a mobile terminal.

[0052] The rich audio information in the frequency domain features mainly exists in the mid- and low-frequency range, and this frequency band contains most of the information of the music features. Therefore, after obtaining the frequency domain features of the audio including the sound sources of different frequency bands, in order to improve the network's separation effect on the mid- and low-frequency features, a sub-band division method is adopted. The embodiment of the present disclosure provides a sub-band division method. In the sub-band division method provided by the present disclosure, the frequency domain features are divided into a plurality of sub-band frequency domain features in sequence according to different frequency bands.

[0053] Figure 2 The figure is a flow chart of a sub-band division method according to an exemplary embodiment.

[0054] In step S21, the frequency domain features are respectively input into different sub-band division networks in the sound source separation network to obtain a plurality of sub-band frequency domain features.

[0055] In step S22, different sub-band division networks are used to divide sub-band frequency domain features corresponding to different frequency bands.

[0056] In the disclosed embodiment, the frequency domain characteristics of the audio including the sound sources of different frequency bands are divided into a plurality of sub-bands according to different frequency band ranges. For example, the sub-band widths are set to 2kHz, 4kHz, 8kHz, and 12kHz, respectively, and the set sub-band widths are determined according to the frequency characteristics of the sound sources to be separated during the network training process.

[0057] The subband division network is composed of a band division network (Band split) and a one-dimensional convolutional layer (Conv1D), wherein Band split is composed of two layers of linear layers (Linear). Linear is mainly used to map subband frequency domain features to the required dimensions. Through linear transformation, the features can be dimensionally adjusted and transformed to adapt to subsequent processing requirements. The introduction of Linear can improve the adaptability and generalization ability of the network, so that the model can better learn and express frequency domain features of different frequency bands. Conv1D is used in the subband division network to perform feature processing and compensation on each subband frequency domain feature, further enhancing the characterization ability of each subband frequency domain feature. The subband division method in the disclosed embodiment enables the subband division network to better distinguish the audio features of different frequency bands.

[0058] After the frequency domain features are divided into a plurality of sub-band frequency domain features according to different frequency band ranges, the sub-band frequency domain features are fused with the global sub-band frequency domain features of the global sub-band frequency domain feature recognition according to the division order, and the global sub-band frequency domain feature recognition is performed.

[0059] In one embodiment, if Figure 3 As shown, the global sub-band frequency domain feature recognition network includes N+1 groups of sub-networks, wherein the first N groups of sub-networks correspond to the division of N sub-band frequency domain features. Each of the first N groups of sub-networks in the N+1 groups of sub-networks includes a batch normalization layer (BatchNorm), a one-dimensional convolution layer (Conv1D), a two-dimensional convolution layer (Conv2D), and an activation function.

[0060] BatchNorm is used to normalize the input of each global sub-band frequency domain feature recognition in the sub-network to accelerate the convergence speed and improve the robustness of the global sub-band frequency domain feature recognition network. It helps the global sub-band frequency domain feature recognition network to better learn and fit the data by reducing the internal covariate shift and gradient vanishing problems.

[0061] Conv1D is used in the global sub-band frequency domain feature recognition network to perform feature processing and compensation on the frequency domain features obtained by fusing the sub-band frequency domain features with the global sub-band frequency domain features, so as to further enhance the representation capability of the features.

[0062] Conv2D is used in the global sub-band frequency domain feature recognition network to extract frequency domain related features from the input features. It extracts relevant information from the frequency domain features by sliding the convolution kernel on the input feature map and performing convolution operations. Since audio data has rich features in the frequency domain, the use of a two-dimensional convolutional layer can effectively capture patterns and correlations at different frequencies.

[0063] The activation function introduces nonlinear transformation, allowing the network to learn more complex patterns and representations. In the global sub-band frequency domain feature recognition network, the commonly used activation functions include pReLU (Parametric Rectified Linear Unit) and so on.

[0064] The N+1th group of sub-networks in the N+1 groups of sub-networks includes BatchNorm and Conv1D.

[0065] In the last group of networks, Conv2D is no longer used, only BatchNorm and Conv1D are included. This is because in the previous N groups of sub-networks, BatchNorm, Conv1D, Conv2D, and activation functions have been used to process and compensate for frequency domain features with different frequency band distinctions. The operation of the N+1 group of sub-networks is only to use the batch normalization layer for normalization and a one-dimensional convolution layer for data buffering to prepare for the subsequent use of TCN to separate frequency domain features.

[0066] In the disclosed embodiment, by dividing the sub-band frequency domain features corresponding to different frequency bands and performing compensation processing for each sub-band, the feature representation on different frequency bands is enhanced, the feature differences of audio signals on different frequency bands can be better captured, and the modeling capability of frequency band features can be improved. The global sub-band frequency domain features are gradually integrated and identified with the sub-band frequency domain features of different frequency bands, so that the network can more accurately identify the frequency domain signals on different frequency bands. Therefore, the method in the disclosed embodiment can achieve a clearer and more accurate separation effect in the separation task.

[0067] At the same time, since audio data has a high-dimensional feature representation, under the traditional neural network-based sound source separation method, the network needs to process a large amount of feature information, resulting in a large amount of calculation. However, the sound source separation network can effectively reduce the amount of calculation by dividing the sub-band frequency domain features corresponding to different frequency bands and fusing and identifying them step by step with the global sub-band frequency domain features. By dividing the frequency domain features of audio including sound sources of different frequency bands into multiple sub-bands according to different frequency band ranges, the network can perform local modeling on each sub-band, reducing the dimension of the features and the parameter scale of the network. The computational complexity of the network is reduced, and the training and reasoning speed of the model is accelerated. Therefore, the sound source separation network in the embodiment of the present disclosure can reduce the computational load while ensuring the sound source separation effect in practical applications.

[0068] By dividing the sub-band frequency domain features corresponding to different frequency bands and fusing and identifying them with the global sub-band frequency domain features step by step, the sub-band frequency domain feature recognition result of the mixed audio is obtained. This recognition result is a more distinguishing feature representation. Based on the sub-band frequency domain feature recognition result of the mixed audio, the sound sources of different frequency bands included in the mixed audio are separated.

[0069] Figure 4 The present invention is a flowchart of a method for performing sound source separation on sub-band frequency domain feature recognition results based on TCN according to an exemplary embodiment.

[0070] In step S41, within the mth cycle, the sub-band frequency domain feature recognition result of the mixed audio is subjected to sound source separation based on the temporal convolutional network TCN in the sound source separation network to obtain a first separation result of sound source separation of different frequency band sound sources included in the mixed audio.

[0071] In step S42, starting from the end time of the duration corresponding to the first separation result, the sub-band frequency domain feature recognition result of the mixed audio is separated based on TCN within the m+1th period to obtain a second separation result of the sound source separation of different frequency band sound sources included in the mixed audio.

[0072] In step S43, the above process is repeatedly performed until the separation of the sound sources of different frequency bands included in the mixed audio is completed.

[0073] In the disclosed embodiment, the sub-band frequency domain feature recognition results of the mixed audio are separated from the sound source based on the time convolution network (TCN). Among them, the network structure for separating the recognition results includes: BatchNorm, TCN and Conv1D. TCN is a time series convolution network for processing time series data. It accepts the sub-band frequency domain feature recognition results of the mixed audio as input, and performs time domain modeling and further feature extraction on the features through convolution operations. Since TCN is a model based on convolution operations, a certain delay will be introduced. In order to achieve real-time separation, an overlapping processing strategy can be used. That is, a certain overlapping area is maintained between adjacent cycles, so that the audio data of adjacent cycles are partially overlapped to improve the separation effect and smooth the output. For a continuous audio data stream, the separation results of each cycle need to be spliced ​​to obtain the overall separation result. This can be achieved by processing the overlapping area.

[0074] The above process is repeated to continuously separate the sound source based on the separation result of the current cycle and the sub-band frequency domain feature recognition result of the next cycle. The separation result of each cycle is used as the input of the next cycle, and the accuracy of separation is gradually improved based on the information in the time domain and frequency domain.

[0075] Here, m is a positive integer greater than or equal to 1.

[0076] Through this process, TCN achieves real-time separation of sound sources in different frequency bands in mixed audio. Each cycle uses the previous separation results and the current sub-band frequency domain characteristics for separation, and continuously iterates to improve the separation effect. In this way, accurate real-time separation of sound sources in each frequency band of mixed audio can be achieved.

[0077] Figure 5 The present invention is a flowchart of a method for training a sound source separation network according to an exemplary embodiment.

[0078] In step S51, a first mixed audio training data set and a second mixed audio training data set are obtained.

[0079] Among them, the first mixed audio training data set includes multi-band sound source mixed audio data collected in an environment mixed with noise data, and the second mixed audio training data set includes audio data obtained by mixing multi-band sound source data collected in an environment without noise data.

[0080] In step S52, model training is performed based on the first mixed audio training data set and the second mixed audio training data set to obtain a sound source separation network.

[0081] In the disclosed embodiment, the first mixed audio training set refers to the original audio including sound sources of different frequency bands. The second mixed audio data training set is audio data obtained by synthesizing pure sound source data of different frequency bands without noise. Traditional separation networks usually only use the original mixed audio features for model learning. However, the sound source usually contains different types of audio, resulting in incomplete alignment of the mixed audio with the audio to be separated. This will have a negative impact on the performance of the algorithm.

[0082] In order to alleviate the impact of this problem, a second mixed audio training data set is introduced in the training method in the embodiment of the present disclosure. The second mixed audio training data set is audio data obtained by mixing multi-band sound source data collected in a noise-free data environment, which can enable the model to better adapt to the characteristics of the mixed audio. During the training process, the first mixed audio training data set and the second mixed audio training data set are trained together as inputs, so that the model can better adapt to different types of mixed audio. This can improve the generalization ability of the model, enhance its separation effect in practical applications and reduce the mismatch problem in the training data. It should be noted that the first and second mixed audio training data sets are only used in the model training stage and are not included in the test reasoning part in actual application.

[0083] In deep model training methods, the loss function is an indicator used to measure the difference between the model's prediction results and the true label. Specifically, the model training process is to minimize the loss function through back-propagation algorithms and optimization algorithms, such as gradient descent. By continuously adjusting the model's parameters, the value of the loss function gradually decreases, and the model gradually learns better separation capabilities.

[0084] Figure 6 The invention is a flowchart of a method for determining a weighted loss function and constraining model training according to an exemplary embodiment.

[0085] In step S61, a first loss function corresponding to the first mixed audio training data set is determined, and a second loss function corresponding to the second mixed audio training data set is determined.

[0086] In step S62, the first loss function and the second loss function are weighted, and based on the weighted loss function, model training constraints are performed to obtain a sound source separation network.

[0087] Among them, the weight of the first loss function is greater than the weight of the second loss function.

[0088] In the disclosed embodiment, the calculation result of the weighted loss function can be used as an evaluation indicator in the training process to monitor the training progress and performance of the model. In addition, by setting the weights of different loss functions, their relative importance in model training can be controlled. In this way, the losses of different parts can be reasonably adjusted according to the characteristics of the task and the optimization goals, so that the model can be better learned and optimized. It should be noted that the setting of weights needs to be experimented and tuned, and selected according to the characteristics of specific tasks and data sets. Reasonable weight settings can help the model achieve better performance and improve its generalization ability in practical applications.

[0089] Therefore, in the method of the embodiment of the present disclosure, the weight of the first loss function is greater than the weight of the second loss function, which can help the model pay more attention to the separation effect of the original audio including the sound sources of different frequency bands. This can ensure that the model can more accurately separate the sound sources of different frequency bands in the mixed audio, and improve the effect of the model in practical applications and user experience.

[0090] At the same time, setting a smaller weight for the second loss function can reduce the focus on the noise-free audio data training set, which can prevent the model from overfitting the training data and causing audio distortion. Although the weight of the second loss function is smaller, it still plays a certain role in helping the model adapt to misaligned music data, improving its generalization ability and robustness, and ensuring that the model achieves a good balance between the two.

[0091] In one embodiment, in the audio separation task, the difference between the separation result and the real human voice or accompaniment can be measured by using the mean square error (MSE) as the loss function. MSE is a commonly used indicator to measure the difference between the predicted value and the true value. It calculates the mean of the square of the difference between the predicted value and the true value and takes the square root thereof.

[0092] MSE can measure the overall accuracy of the separation results, that is, the average difference between the separation results and the true values. By minimizing MSE, the model can better approximate the real human voice and accompaniment.

[0093] In addition, in order to measure the sound quality and clarity of the separation results and minimize the distortion of the sound source, the signal-to-noise ratio (SNR) is introduced as part of the loss function. SNR measures the ratio between the useful signal and the noise in the separation result, that is, the ratio between the human voice and the accompaniment and the background noise in the separation result.

[0094] By introducing SNR into the error function, the model can better suppress background noise, minimize sound source distortion, and improve the clarity and audibility of useful signals in the separation results.

[0095] By using MSE and SNR as the calculation of the loss function, the accuracy and sound quality of the separation results can be considered at the same time, thereby obtaining a better audio separation effect.

[0096] In one embodiment, the sound source separation method and the sound source separation network training method in the embodiments of the present disclosure are used to separate audio including sound sources of different frequency bands, for example, to separate human voice and accompaniment in mixed audio.

[0097] In the present disclosure, Figure 7 As shown, the frequency domain features of the original mixed audio containing human voice and accompaniment are extracted by Fourier transform as the first mixed audio training set. The frequency domain features of clean human voice and accompaniment are extracted and synthesized by Fourier transform, and the synthesized frequency domain features are used as the second mixed audio data training set.

[0098] In each training batch, N first mixed audios and N second mixed audios are used as features sent to the sound source separation network, that is, each batch contains 2N groups of features.

[0099] For example, set the batch size to 16 and the number of mixed audios N to 8.

[0100] Through the sound source separation network, the separated vocal and accompaniment audio are obtained.

[0101] In the disclosed embodiment, the loss function mainly includes two parts: the loss function of the first mixed audio includes: vocal loss L-vo-s, accompaniment loss L-ac-s, and the loss function of the second mixed audio includes: vocal loss L-vo-p, accompaniment loss L-ac-p.

[0102] Calculate the loss function for the first mixed audio:

[0103]

[0104] Among them, the mean square error loss function L including the frequency domain characteristics of human voice and accompaniment MSE And the signal-to-noise ratio loss function L in the time domain SNR In addition, in order to improve the accuracy of training, the weighting coefficients w1 and w2 are 0.5;

[0105] Calculate the loss function for the second mixed audio:

[0106]

[0107] Among them, in order to avoid sound source distortion to the greatest extent, the weighted coefficients w1 and w2 of the loss function are 0.99.

[0108] Through the calculation of the above two parts of the loss function, the overall loss function of the model training is:

[0109] L=a·L1+b·L2

[0110] In order to pay more attention to the separation effect of the original audio including the audio sources of different frequency bands and reduce the attention to the noise-free audio data training set, the weight of the first loss function is set to be greater than the weight of the second loss function. For example, the weight a of the loss function of the first mixed audio is set to 0.7, and the weight b of the loss function of the second mixed audio is set to 0.3.

[0111] In the disclosed embodiment, the Adam optimizer is used to train the model, that is, the training iteration process is repeated until a preset number of training rounds is reached or a stop condition is met. The Adam optimizer combines the momentum method and the adaptive learning rate mechanism to more effectively update the model parameters.

[0112] In the disclosed embodiment, after iterative training, a sound source separation network for the scene of separating human voice and accompaniment is obtained, and the separation network obtained by the training is used to separate human voice and accompaniment.

[0113] The original mixed audio containing the different frequency band sound sources to be separated is subjected to Fourier transform to obtain frequency domain features, and the obtained frequency domain features are divided into self-contained frequency domain features of different frequency bands and fused and recognized with the global sub-band frequency domain features. The recognition result is input into the time convolution network TCN, and the human voice and accompaniment are separated based on this recognition result to obtain the separated audio of the human voice and accompaniment. For example, in the TCN network, the length of the mth cycle is 50ms, the first overlap between the mth cycle and the m+1th cycle is 25ms, and the length of the separation result is 25ms.

[0114] The signal-to-distortion ratio (SDR) is used to evaluate the separation effect of the human voice and the accompaniment audio. The separation results obtained in the embodiment of the present disclosure are that the human voice SDR is 7.0±0.3 and the accompaniment SDR is 15.0±0.3, and nearly pure human voice and accompaniment audio are obtained.

[0115] The parameters are set according to the training results of the sound source separation network, for example, the Adam optimizer is applied and the learning rate is set to 0.001. In the global sub-band frequency domain feature recognition, the receptive field of the two-dimensional convolution layer is (1,3), and the TCN is implemented by two-dimensional convolution, and its receptive field size is (3,3). Each TCN contains three layers of two-dimensional convolution.

[0116] In the disclosed embodiment, the audio separation network is trained by using a first mixed audio training data set containing frequency domain features of original mixed audio of human voice and accompaniment and a second mixed audio training data set obtained by mixing frequency domain features of pure human voice and accompaniment audio. Based on a weighted loss function, model training constraints are performed to obtain a sound source separation network.

[0117] In the actual human voice and accompaniment audio separation scenario, the trained sound source separation network is used to separate the human voice and the accompaniment. By dividing the sub-band frequency domain features, not only can the characteristic differences between the human voice and the accompaniment in different frequency bands be better captured, and a clearer and more accurate separation result can be obtained, but also different divided frequency bands can be set according to the characteristics of different music types, which is suitable for the separation of more types of music. For example, folk music, DJ music, classical music, pop music, etc. In addition, by performing local modeling on each sub-band through the network, the dimension of the features and the parameter scale of the network are reduced, and the computational complexity of the network is reduced, so that real-time separation tasks can be performed on CPU, GPU and other processors of small smart mobile terminals.

[0118] In one embodiment, by using the sound source separation method in the embodiment of the present disclosure, two sets of audio, human voice and accompaniment, are obtained. On this basis, human voice and accompaniment can be enhanced separately according to different application scenarios. For example, the following method is used to enhance human voice and accompaniment separately: S = α·V + β·A

[0119] Among them, S is the audio obtained by the enhancement operation, V is the human voice, and A is the accompaniment, which are graded and adjusted by parameters α and β.

[0120] At the same time, in order to highlight the listening effect of level changes, the vocal accompaniment enhancement effect can be set to level 5 and applied to different scenarios, such as melody extraction, pitch estimation, temperament estimation, music transcription, karaoke, intelligent sound effects, music teaching, audio editing, etc.

[0121] Based on the same concept, the embodiment of the present disclosure also provides a sound source separation device.

[0122] It is understandable that the sound source separation device provided in the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0123] Figure 8 is a block diagram of a sound source separation device according to an exemplary embodiment. Figure 8 The device includes an acquisition unit 101, a division unit 102, a recognition unit 103, a separation unit 104 and a training unit 105.

[0124] The acquisition unit 101 is used to acquire frequency domain features of mixed audio, where the mixed audio includes audios of sound sources in different frequency bands;

[0125] A division unit 102, configured to divide the frequency domain features into a plurality of sub-band frequency domain features in sequence according to different frequency bands;

[0126] The identification unit 103 is used to fuse each sub-band frequency domain feature of the multiple sub-band frequency domain features with the corresponding global sub-band frequency domain feature in the order of division, perform global sub-band frequency domain feature identification on the global sub-band frequency domain feature fused with the sub-band frequency domain feature, and use the identification result as the global sub-band frequency domain feature fused with the next sub-band frequency domain feature, repeatedly perform the above fusion and global sub-band frequency domain feature identification process until the global sub-band frequency domain feature identification of the identification number is completed, and obtain the sub-band frequency domain feature identification result of the mixed audio;

[0127] The separation unit 104 is used to separate the sound sources of different frequency bands included in the mixed audio based on the sub-band frequency domain feature recognition result of the mixed audio.

[0128] The training unit 105 is used to train a sound source separation network in the following manner: obtain a first mixed audio training data set and a second mixed audio training data set, the first mixed audio training data set includes multi-band sound source mixed audio data collected in an environment doped with noise data, and the second mixed audio training data includes audio data obtained by mixing multi-band sound source data collected in an environment without noise data.

[0129] In one embodiment, the division unit divides the frequency domain features into multiple sub-band frequency domain features according to different frequency bands in the following manner: the frequency domain features are respectively input into different sub-band division networks in the sound source separation network to obtain multiple sub-band frequency domain features; different sub-band division networks are used to divide the sub-band frequency domain features corresponding to different frequency bands.

[0130] In one embodiment, global sub-band frequency domain feature recognition is based on global sub-band frequency domain feature recognition network recognition in a sound source separation network; the global sub-band frequency domain feature recognition network includes N groups of sub-networks, and each sub-network in the first N-1 groups of sub-networks in the N groups of sub-networks includes a batch normalization layer, a one-dimensional convolution layer, a two-dimensional convolution layer, and an activation function; the Nth group of sub-networks in the N groups of sub-networks includes a batch normalization layer and a one-dimensional convolution layer.

[0131] In one embodiment, the separation unit performs sound source separation on different frequency band sound sources included in the mixed audio based on the sub-band frequency domain feature recognition result of the mixed audio in the following manner: performing sound source separation on the sub-band frequency domain feature recognition result of the mixed audio based on the time convolution network TCN in the sound source separation network within the mth period, and obtaining a first separation result of sound source separation on different frequency band sound sources included in the mixed audio, wherein the duration corresponding to the first separation result is the duration having a first degree of overlap with the mth period from the start time of the mth period, and m is a positive integer greater than or equal to 1; starting from the end time of the duration corresponding to the first separation result, performing sound source separation on the sub-band frequency domain feature recognition result of the mixed audio based on TCN within the m+1th period, and obtaining a second separation result of sound source separation on different frequency band sound sources included in the mixed audio, wherein the duration of the second separation result is the duration having a first degree of overlap with the m+1th period from the start time of the m+1th period; and repeating the above process until the separation of different frequency band sound sources included in the mixed audio is completed.

[0132] In one embodiment, the training unit is used to train a sound source separation network in the following manner: obtain a first mixed audio training data set and a second mixed audio training data set, the first mixed audio training data set including multi-band sound source mixed audio data collected in an environment doped with noise data, and the second mixed audio training data including audio data obtained by mixing multi-band sound source data collected in an environment without noise data; perform model training based on the first mixed audio training data set and the second mixed audio training data set to obtain a sound source separation network.

[0133] In one embodiment, the training unit performs model training based on a first mixed audio training data set and a second mixed audio training data set to obtain a sound source separation network in the following manner, including: determining a first loss function corresponding to the first mixed audio training data set, and determining a second loss function corresponding to the second mixed audio training data set; weighting the first loss function and the second loss function, and based on the weighted loss function, performing model training constraints to obtain a sound source separation network; wherein the weight of the first loss function is greater than the weight of the second loss function.

[0134] In one embodiment, the first loss function and the second loss function are determined based on a mean square error loss function and a signal-to-noise ratio loss function.

[0135] In one embodiment, the audio of the sound sources in different frequency bands includes frequency domain characteristics of the audio corresponding to the human voice frequency band and the audio corresponding to the accompaniment frequency band.

[0136] Fig. 9 2 is a block diagram of a device 200 for sound source separation according to an exemplary embodiment. For example, the device 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0137] Reference Fig. 9 , the device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0138] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.

[0139] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0140] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.

[0141] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0142] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0143] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0144] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0145] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0146] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0147] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0148] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0149] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.

[0150] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.

[0151] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0152] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0153] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.

Claims

1. A sound source separation method, characterized in that: include: . Acquire frequency domain features of mixed audio, where the mixed audio includes audio from sound sources in different frequency bands; Dividing the frequency domain features into a plurality of sub-band frequency domain features in sequence according to different frequency bands; Each sub-band frequency domain feature of the multiple sub-band frequency domain features is fused with the corresponding global sub-band frequency domain feature in the order of division, and global sub-band frequency domain feature recognition is performed on the global sub-band frequency domain feature fused with the sub-band frequency domain feature, and the recognition result is used as the global sub-band frequency domain feature fused with the next sub-band frequency domain feature, and the above fusion and global sub-band frequency domain feature recognition process is repeatedly performed until the global sub-band frequency domain feature recognition of the recognition number is completed, and the sub-band frequency domain feature recognition result of the mixed audio is obtained; The Nth identification of the global sub-band frequency domain feature corresponds to the division of the Nth sub-band frequency domain feature, and N is a positive integer greater than or equal to 1 and less than the number of identifications of the global sub-band frequency domain feature; Based on the sub-band frequency domain feature recognition result of the mixed audio, sound source separation is performed on the different frequency band sound sources included in the mixed audio.

2. The sound source separation method according to claim 1, characterized in that: The step of dividing the frequency domain features into a plurality of sub-band frequency domain features in sequence according to different frequency bands includes: Inputting the frequency domain features into different sub-band division networks in the sound source separation network respectively to obtain multiple sub-band frequency domain features; The different sub-band division networks are used to divide sub-band frequency domain characteristics corresponding to different frequency bands.

3. The sound source separation method according to claim 1, characterized in that: The global sub-band frequency domain feature recognition is based on the global sub-band frequency domain feature recognition network recognition in the sound source separation network; The global sub-band frequency domain feature recognition network includes N+1 groups of sub-networks, and each sub-network in the first N groups of sub-networks in the N+1 groups of sub-networks includes a batch normalization layer, a one-dimensional convolution layer, a two-dimensional convolution layer, and an activation function; The N+1th group of sub-networks in the N+1 groups of sub-networks includes a batch normalization layer and a one-dimensional convolutional layer.

4. The sound source separation method according to claim 1, characterized in that: The step of separating the sound sources of the different frequency bands included in the mixed audio based on the sub-band frequency domain feature recognition result of the mixed audio comprises: In the mth period, based on the time convolution network TCN in the sound source separation network, the sub-band frequency domain feature recognition result of the mixed audio is separated into sound sources, and a first separation result of the sound source separation of the different frequency band sound sources included in the mixed audio is obtained, wherein the time length corresponding to the first separation result is the time length having a first degree of overlap with the mth period from the start time of the mth period, and m is a positive integer greater than or equal to 1; Starting from the end time of the duration corresponding to the first separation result, performing sound source separation on the sub-band frequency domain feature recognition result of the mixed audio based on the TCN within the m+1th period, obtaining a second separation result of sound source separation of the different frequency band sound sources included in the mixed audio, wherein the duration of the second separation result is the duration having the first degree of overlap with the m+1th period from the start time of the m+1th period; The above process is repeatedly performed until the separation of the sound sources of different frequency bands included in the mixed audio is completed.

5. The sound source separation method according to any one of claims 2 to 4, characterized in that: The sound source separation network is pre-trained in the following manner: Acquire a first mixed audio training data set and a second mixed audio training data set, wherein the first mixed audio training data set includes multi-band sound source mixed audio data collected in an environment doped with noise data, and the second mixed audio training data set includes audio data obtained by mixing multi-band sound source data collected in an environment without noise data; Model training is performed based on the first mixed audio training data set and the second mixed audio training data set to obtain a sound source separation network.

6. The sound source separation method according to claim 5, characterized in that: The performing model training based on the first mixed audio training data set and the second mixed audio training data set to obtain a sound source separation network includes: Determine a first loss function corresponding to the first mixed audio training data set, and determine a second loss function corresponding to the second mixed audio training data set; Weighting the first loss function and the second loss function, and performing model training constraints based on the weighted loss functions to obtain a sound source separation network; The weight of the first loss function is greater than the weight of the second loss function.

7. The sound source separation method according to claim 6, characterized in that: The first loss function and the second loss function are determined based on a mean square error loss function and a signal-to-noise ratio loss function.

8. The sound source separation method according to claim 1, characterized in that: The audios of the sound sources of different frequency bands include the audios corresponding to the human voice frequency band and the audios corresponding to the accompaniment frequency band.

9. A sound source separation device, characterized in that: include: An acquisition unit, configured to acquire frequency domain features of mixed audio, wherein the mixed audio includes audios of sound sources in different frequency bands; A division unit, used to divide the frequency domain features into a plurality of sub-band frequency domain features in sequence according to different frequency bands; an identification unit, configured to fuse each sub-band frequency domain feature of the multiple sub-band frequency domain features with the corresponding global sub-band frequency domain feature in the order of division, perform global sub-band frequency domain feature identification on the global sub-band frequency domain feature fused with the sub-band frequency domain feature, and use the identification result as the global sub-band frequency domain feature fused with the next sub-band frequency domain feature, repeatedly perform the above-mentioned fusion and global sub-band frequency domain feature identification process until the global sub-band frequency domain feature identification of the identification number is completed, and obtain the sub-band frequency domain feature identification result of the mixed audio; The Nth identification of the global sub-band frequency domain feature corresponds to the division of the Nth sub-band frequency domain feature, and N is a positive integer greater than or equal to 1 and less than the number of identifications of the global sub-band frequency domain feature; The separation unit is used to separate the sound sources of the different frequency bands included in the mixed audio based on the sub-band frequency domain feature recognition result of the mixed audio.

10. The sound source separation device according to claim 9, characterized in that: The division unit divides the frequency domain features into a plurality of sub-band frequency domain features in sequence according to different frequency bands in the following manner: Inputting the frequency domain features into different sub-band division networks in the sound source separation network respectively to obtain multiple sub-band frequency domain features; The different sub-band division networks are used to divide sub-band frequency domain characteristics corresponding to different frequency bands.

11. The sound source separation device according to claim 9, characterized in that: The global sub-band frequency domain feature recognition is based on the global sub-band frequency domain feature recognition network recognition in the sound source separation network; The global sub-band frequency domain feature recognition network includes N+1 groups of sub-networks, and each sub-network in the first N groups of sub-networks in the N+1 groups of sub-networks includes a batch normalization layer, a one-dimensional convolution layer, a two-dimensional convolution layer, and an activation function; The N+1th group of sub-networks in the N+1 groups of sub-networks includes a batch normalization layer and a one-dimensional convolutional layer.

12. The sound source separation device according to claim 9, characterized in that: The separation unit performs sound source separation on the different frequency band sound sources included in the mixed audio based on the sub-band frequency domain feature recognition result of the mixed audio in the following manner: In the mth period, based on the time convolution network TCN in the sound source separation network, the sub-band frequency domain feature recognition result of the mixed audio is separated into sound sources, and a first separation result of the sound source separation of the different frequency band sound sources included in the mixed audio is obtained, wherein the time length corresponding to the first separation result is the time length having a first degree of overlap with the mth period from the start time of the mth period, and m is a positive integer greater than or equal to 1; Starting from the end time of the duration corresponding to the first separation result, performing sound source separation on the sub-band frequency domain feature recognition result of the mixed audio based on the TCN within the m+1th period, obtaining a second separation result of sound source separation of the different frequency band sound sources included in the mixed audio, wherein the duration of the second separation result is the duration having the first degree of overlap with the m+1th period from the start time of the m+1th period; The above process is repeatedly performed until the separation of the sound sources of different frequency bands included in the mixed audio is completed.

13. The sound source separation device according to any one of claims 10 to 12, characterized in that: The sound source separation device also includes: The training unit is used to train the sound source separation network in the following manner: Acquire a first mixed audio training data set and a second mixed audio training data set, wherein the first mixed audio training data set includes multi-band sound source mixed audio data collected in an environment doped with noise data, and the second mixed audio training data set includes audio data obtained by mixing multi-band sound source data collected in an environment without noise data; Model training is performed based on the first mixed audio training data set and the second mixed audio training data set to obtain a sound source separation network.

14. The sound source separation device according to claim 13, characterized in that: The training unit performs model training based on the first mixed audio training data set and the second mixed audio training data set to obtain a sound source separation network in the following manner: Determine a first loss function corresponding to the first mixed audio training data set, and determine a second loss function corresponding to the second mixed audio training data set; Weighting the first loss function and the second loss function, and performing model training constraints based on the weighted loss functions to obtain a sound source separation network; The weight of the first loss function is greater than the weight of the second loss function.

15. The sound source separation device according to claim 14, characterized in that: The first loss function and the second loss function are determined based on a mean square error loss function and a signal-to-noise ratio loss function.

16. The sound source separation device according to claim 9, characterized in that: The audios of the sound sources of different frequency bands include the audios corresponding to the human voice frequency band and the audios corresponding to the accompaniment frequency band.

17. A sound source separation device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the method described in any one of claims 1 to 8.

18. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions in the storage medium are executed by a processor of the terminal, the terminal is enabled to execute the method according to any one of claims 1 to 8.

Images