Multi-channel voice separation method and device based on neural network
By constructing a feature extraction module based on STFT and IPD and a dual-path fast converter neural network of SCBAM and ConFastformer, combining a multi-channel speech separation model with crossed linear attention and full-belt modeling unit, the speech separation technology in the existing technology is solved The lightweight and generalization problems achieve high-quality voice separation in complex acoustic environments and adapt to the needs of mobile devices.
Patent Information
- Application Number
- CN202411092778.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-08-09
AI Technical Summary
The existing microphone array-based speech separation technology cannot meet the requirements of lightweight, high robustness and high generalization, and adapt to complex and variable acoustic environments. At the same time, the model parameter quantity and calculation of multi-channel speech separation algorithm based on neural networks The amount is large and difficult to adapt to mobile devices with limited computing power or memory.
The feature extraction module based on STFT and IPD is adopted, combined with SCBAM and ConFastformer to build an initial dual-path fast converter neural network, and a multi-channel speech separation model is constructed through cross-linear attention and full-belt modeling units, and the Tanh and Sigmoid activation functions are used. The mask module is optimized to build a lightweight multi-channel speech separation model.
It realizes high-quality speech separation under different hybrid conditions, shows excellent generalization ability, is suitable for complex acoustic environments, and reduces the number of parameters and calculations of the model to adapt to mobile devices.
Smart Images

Figure CN119049500A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of speech separation and extraction, and in particular to a multi-channel speech separation method and device based on a neural network. Background Art
[0002] In daily life, we are always surrounded by various types of sounds, and the sounds of interest are often inevitably intertwined with the additional noise from other sound sources and the reverberation caused by reflections. Therefore, separating the sounds of interest from the background interference is of vital importance to ensure smooth communication between people and accurate transmission of commands during human-computer interaction.
[0003] In the related technology, speech separation can be divided into single-channel and multi-channel (i.e., microphone array) speech separation according to the number of channels or microphones used. Among them, the multi-channel speech separation technology can utilize the spatial information in the microphone array structure to perform multi-channel speech separation in a far-field complex acoustic environment, and can also perform multi-channel speech separation based on a neural network, such as by using multiple stacked TCN (The Comcast Network), LSTM (Long Short-Term Memory) or Transformer modules.
[0004] However, among the related technologies, the speech separation technology based on microphone array cannot meet the requirements of lightweight, high robustness and high generalization, and cannot adapt to the complex and changeable acoustic environment. In the multi-channel speech separation algorithm based on neural network, the model has large number of parameters and calculation amount, which makes it difficult to adapt to mobile devices with limited computing power or memory, and it is urgently needed to be improved. Summary of the invention
[0005] The present application provides a multi-channel speech separation method and device based on a neural network to solve the problem that the speech separation technology based on microphone array in the related technology cannot meet the requirements of lightweight, high robustness and high generalization, and cannot adapt to the complex and changeable acoustic environment. In the multi-channel speech separation algorithm based on the neural network, the model has a large number of parameters and a large amount of calculation, and it is difficult to adapt to mobile devices with limited computing power or memory.
[0006] The first aspect of the present application provides a multi-channel speech separation method based on a neural network, which is applied to the model construction stage, wherein the method includes the following steps: based on STFT (Short-Time Fourier Transform) and IPD (Interchannel Phase Difference), constructing a feature extraction module of an initial multi-channel speech separation model; based on feature coding and SCBAM (Shuffle Convolutional Block Attention Module), constructing a spectrum modeling unit of an initial dual-path fast converter neural network; based on cross linear attention and ConFastformer (Convolution-augmented Fastformer, convolution enhanced fast converter) respectively construct the subband modeling unit and the full-band modeling unit of the initial dual-path fast converter neural network; based on the spectrum modeling unit, the subband modeling unit and the full-band modeling unit, construct the initial dual-path fast converter neural network of the initial multi-channel speech separation model; based on the fully connected layer of the Tanh activation function and the fully connected layer of the Sigmoid activation function, construct the mask module of the initial multi-channel speech separation model; based on the feature extraction module, the initial dual-path fast converter neural network and the mask module, construct the initial multi-channel speech separation model; input the target multi-channel speech into the initial multi-channel speech separation model, take the target mask as the optimization target of the initial multi-channel speech separation model, optimize the target parameters of the initial dual-path fast converter neural network, update the initial multi-channel speech separation model based on the optimized dual-path fast converter neural network, obtain the multi-channel speech separation model, and obtain the separated speech corresponding to the target multi-channel speech based on the multi-channel speech separation model.
[0007] Optionally, in one embodiment of the present application, the target multi-channel speech is input into the initial multi-channel speech separation model, the target mask is used as the optimization target of the initial multi-channel speech separation model, and the target parameters of the initial dual-path fast converter neural network are optimized, including: combining the target multi-channel speech and the feature extraction module to obtain the spectrum and frequency domain spatial features of the target multi-channel speech; and obtaining the target feature map of the target multi-channel speech based on the real part of the spectrum, the imaginary part of the spectrum and the frequency domain spatial features.
[0008] Optionally, in one embodiment of the present application, the target multi-channel speech is input into the initial multi-channel speech separation model, the target mask is used as the optimization target of the initial multi-channel speech separation model, and the target parameters of the initial dual-path fast converter neural network are optimized, which also includes: combining the target feature map and the spectrum modeling unit to obtain the spectral local information and spectral spatial information of the target multi-channel speech; based on the spectral local information and the spectral spatial information, the time context information of the target multi-channel speech is obtained according to the sub-band modeling unit; based on the time context information, the frequency context information of the target multi-channel speech is obtained according to the full-band modeling unit; based on the frequency context information, a separable high-dimensional feature representation of the target multi-channel speech is obtained.
[0009] Optionally, in one embodiment of the present application, the target multi-channel speech is input into the initial multi-channel speech separation model, the target mask is used as the optimization target of the initial multi-channel speech separation model, and the target parameters of the initial dual-path fast converter neural network are optimized, which also includes: obtaining the complex mask of the separated speech corresponding to the target multi-channel speech according to the separable high-dimensional feature representation and the mask module; and training the initial multi-channel speech separation model in combination with the complex mask and the target mask to optimize the target parameters of the initial dual-path fast converter neural network.
[0010] Optionally, in one embodiment of the present application, obtaining the separated speech corresponding to the target multi-channel speech based on the multi-channel speech separation model includes: converting the initial separated speech to obtain the time domain waveform of the initial separated speech; and obtaining the separated speech based on the time domain waveform.
[0011] The second aspect of the present application provides a multi-channel speech separation method based on a neural network, which is applied to the use stage of the model, wherein the method includes the following steps: obtaining the multi-channel speech to be separated; inputting the multi-channel speech to be separated into a pre-built multi-channel speech separation model to obtain the separated speech corresponding to the multi-channel speech to be separated, wherein the pre-built multi-channel speech separation model is obtained from the multi-channel speech to be separated.
[0012] In a third aspect, an embodiment of the present application provides a multi-channel speech separation device based on a neural network, which is applied to the model construction stage, wherein the device includes: a first construction module, which is used to construct a feature extraction module of an initial multi-channel speech separation model based on STFT and IPD; a second construction module, which is used to construct a spectrum modeling unit of an initial dual-path fast converter neural network based on feature coding and SCBAM; a third construction module, which is used to construct a sub-band modeling unit and a full-band modeling unit of the initial dual-path fast converter neural network according to cross linear attention and ConFastformer respectively; a fourth construction module, which is used to construct an initial dual-path fast converter neural network of the initial multi-channel speech separation model based on the spectrum modeling unit, the sub-band modeling unit and the full-band modeling unit; and a fifth construction module. A block is used to construct a mask module of the initial multi-channel speech separation model based on a fully connected layer of a Tanh activation function and a fully connected layer of a Sigmoid activation function; a sixth construction module is used to construct the initial multi-channel speech separation model based on the feature extraction module, the initial dual-path fast converter neural network and the mask module; a first generation module is used to input the target multi-channel speech into the initial multi-channel speech separation model, use the target mask as the optimization target of the initial multi-channel speech separation model, optimize the target parameters of the initial dual-path fast converter neural network, update the initial multi-channel speech separation model based on the optimized dual-path fast converter neural network, obtain the multi-channel speech separation model, and obtain the separated speech corresponding to the target multi-channel speech based on the multi-channel speech separation model.
[0013] Optionally, in one embodiment of the present application, the first generation module includes: a first generation unit, used to combine the target multi-channel speech and the feature extraction module to obtain the spectrum and frequency domain spatial features of the target multi-channel speech; a second generation unit, used to obtain the target feature map of the target multi-channel speech based on the real part of the spectrum, the imaginary part of the spectrum and the frequency domain spatial features.
[0014] Optionally, in one embodiment of the present application, the first generation module also includes: a third generation unit, used to combine the target feature map and the spectrum modeling unit to obtain the spectral local information and spectral spatial information of the target multi-channel speech; a fourth generation unit, used to obtain the time context information of the target multi-channel speech according to the sub-band modeling unit based on the spectral local information and the spectral spatial information; a fifth generation unit, used to obtain the frequency context information of the target multi-channel speech according to the full-band modeling unit based on the time context information; and a sixth generation unit, used to obtain a separable high-dimensional feature representation of the target multi-channel speech based on the frequency context information.
[0015] Optionally, in one embodiment of the present application, the first generation module further includes: an acquisition unit, used to acquire the complex mask of the separated speech corresponding to the target multi-channel speech according to the separable high-dimensional feature representation and the mask module; an optimization unit, used to train the initial multi-channel speech separation model in combination with the complex mask and the target mask to optimize the target parameters of the initial dual-path fast converter neural network.
[0016] Optionally, in one embodiment of the present application, the first generating module includes: a conversion unit, used to convert the initial separated speech to obtain the time domain waveform of the initial separated speech; and a seventh generating unit, used to obtain the separated speech based on the time domain waveform.
[0017] The fourth aspect of the present application provides a multi-channel speech separation device based on a neural network, which is applied to the use stage of the model, wherein the device includes: an acquisition module, used to acquire the multi-channel speech to be separated; a second generation module, used to input the multi-channel speech to be separated into a pre-built multi-channel speech separation model to obtain the separated speech corresponding to the multi-channel speech to be separated, wherein the pre-built multi-channel speech separation model is obtained from the multi-channel speech to be separated.
[0018] The fifth aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the neural network-based multi-channel speech separation method as described in the above embodiment.
[0019] The sixth aspect embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the program is executed by a processor, it implements the above-mentioned neural network-based multi-channel speech separation method.
[0020] The seventh aspect embodiment of the present application provides a computer program product, including a computer program, which, when executed, implements the above-mentioned neural network-based multi-channel speech separation method.
[0021] The embodiment of the present application can construct an initial multi-channel speech separation model based on the feature extraction module, the initial dual-path fast converter neural network and the mask module, and use the target mask as the training target, optimize the target parameters, obtain the multi-channel speech separation model, and then obtain the separated speech corresponding to the target multi-channel speech based on the multi-channel speech separation model, wherein the multi-channel speech separation model constructed by the embodiment of the present application has a more lightweight network structure, can accurately and stably achieve high-quality speech separation, and exhibits excellent generalization ability under different mixing conditions, showing good practical value and technical prospects. Thus, it solves the problem that the speech separation technology based on microphone arrays in the related technology cannot meet the requirements of lightweight, high robustness and high generalization, and adapt to complex and changeable acoustic environments, while in the multi-channel speech separation algorithm based on neural networks, the model has a large number of parameters and calculations, and is difficult to adapt to mobile devices with limited computing power or memory.
[0022] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0024] Figure 1 A flowchart of a multi-channel speech separation method based on a neural network provided according to an embodiment of the present application;
[0025] Figure 2 A block diagram of a spectrum modeling unit framework provided according to an embodiment of the present application;
[0026] Figure 3 A schematic block diagram of a ConFastformer network framework composed of a multi-scale convolution module and a multi-head linear attention according to an embodiment of the present application;
[0027] Figure 4 A schematic block diagram of a cross-linear attention structure provided according to an embodiment of the present application;
[0028] Figure 5 A flowchart of mask estimation according to an embodiment of the present application;
[0029] Figure 6 A flowchart of the working principle of a multi-channel speech separation method based on a neural network provided according to an embodiment of the present application;
[0030] Figure 7A block diagram of a multi-channel speech separation device based on a neural network provided according to an embodiment of the present application;
[0031] Figure 8 A flowchart of another multi-channel speech separation method based on a neural network provided according to an embodiment of the present application;
[0032] Fig. 9 A block diagram of another multi-channel speech separation device based on a neural network provided according to an embodiment of the present application;
[0033] Fig.10 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION
[0034] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0035] The following describes the multi-channel speech separation method and device based on neural network in the embodiment of the present application with reference to the accompanying drawings. In view of the fact that the speech separation technology based on microphone array mentioned in the above background technology cannot meet the requirements of lightweight, high robustness and high generalization, and adapt to the complex and changeable acoustic environment, and in the multi-channel speech separation algorithm based on neural network, the number of parameters and the amount of calculation of the model are large, and it is difficult to adapt to the problem of mobile devices with limited computing power or memory, the present application provides a multi-channel speech separation method based on neural network, in which an initial multi-channel speech separation model can be constructed based on a feature extraction module, an initial dual-path fast converter neural network and a mask module, and the target mask is used as a training target to optimize the target parameters to obtain a multi-channel speech separation model, and then the separated speech corresponding to the target multi-channel speech is obtained based on the multi-channel speech separation model, wherein the multi-channel speech separation model constructed in the embodiment of the present application has a more lightweight network structure, can accurately and stably achieve high-quality speech separation, and exhibits excellent generalization ability under different mixing conditions, showing good practical value and technical prospects. This solves the problem in related technologies that the microphone array-based speech separation technology cannot meet the requirements of lightweight, high robustness and high generalization, and cannot adapt to complex and changeable acoustic environments; while in the neural network-based multi-channel speech separation algorithm, the model has a large number of parameters and a large amount of calculation, making it difficult to adapt to mobile devices with limited computing power or memory.
[0036] Specifically, Figure 1 The present invention provides a flowchart of a multi-channel speech separation method based on a neural network according to an embodiment of the present application.
[0037] like Figure 1 As shown, the multi-channel speech separation method based on neural network is applied to the model construction stage, wherein the method includes the following steps:
[0038] In step S101, a feature extraction module of an initial multi-channel speech separation model is constructed based on STFT and IPD.
[0039] It is understandable that STFT can transform time domain signals into time-frequency domain, and IPD is a frequency domain spatial feature commonly used in research such as multi-channel speech enhancement, multi-channel speech separation and sound source localization. It can effectively reflect subtle changes in the direction of arrival of the sound source. Since the speakers are located in different directions, the IPD feature helps the model identify non-overlapping speaker speech segments in the mixed speech, and then extract cleaner speaker direction information and sound information from it to help speech separation.
[0040] As a possible implementation method, the embodiment of the present application can construct a feature extraction module of the initial multi-channel speech separation model based on STFT and IPD.
[0041] In step S102, based on feature coding and SCBAM, a spectrum modeling unit of an initial dual-path fast converter neural network is constructed.
[0042] In the actual implementation process, the spectrum modeling unit of the initial dual-path fast converter neural network of the embodiment of the present application is composed of feature encoding and SCBAM modules, and its structural diagram is as follows: Figure 2 shown.
[0043] Specifically, the embodiment of the present application first fuses the input feature F through feature encoding and upgrades the input feature. The feature encoding consists of a two-dimensional convolution layer, batch normalization, and a ReLU activation function, which is used to encode the network input feature and obtain a high-dimensional feature representation. The formula of the calculation process can be but is not limited to:
[0044] F enc =BN(ReLU(Conv(F))),
[0045] Among them, Conv represents a two-dimensional convolution operation, the convolution kernel size is 3×3, the step size is 1, and the same padding strategy is used to ensure that the feature map size remains unchanged. The ReLu activation function is used to increase the nonlinear representation ability of the network. BN represents batch normalization calculation, which is used to normalize the data according to the batch size, smooth the training process, and avoid gradient explosion or gradient disappearance during network training.
[0046] Furthermore, the embodiment of the present application uses SCBAM, which is composed of a local feature extraction branch and a convolutional attention branch, to perform local feature extraction and spatial feature extraction respectively. The specific implementation process is as follows: First, the embodiment of the present application divides the input feature into two parts along the feature channel, which are represented as
[0047] Among them, in the local feature extraction branch of the embodiment of the present application, firstly, a two-dimensional convolution module with a convolution kernel size of 1×1 is used to transform the input feature dimension into B×T×F to increase the depth and nonlinear representation ability of the network; then, a depth-separable convolution unit with a convolution kernel size of 3×3 is used to extract local spectral features; finally, a 1×1 convolution dimensionality reduction feature channel is used to restore the original feature dimension. Among them, in the embodiment of the present application, the specific structure of the depth-separable convolution is as follows Figure 2 As shown in the figure, it consists of a channel-by-channel convolution module with a convolution kernel size of 3×3 and a point-by-point convolution module with a convolution kernel size of 1×1. The convolution attention branch is composed of cascaded spatial attention and channel attention modules. The calculation process can be but is not limited to:
[0048]
[0049] in, represents the input features of the convolutional attention module, represents the output features of the convolutional attention module, CA(·) and SA(·) represent the channel attention module and the spatial attention module, respectively.
[0050] In this embodiment of the present application, the channel attention module can effectively model the correlation between feature channels, and its specific structure is as follows: Figure 2 As shown. Assume the input feature is Among them, C, N and L represent the feature channel, length and width respectively. First, the global information of each channel is obtained through average pooling and maximum pooling operations, and two vectors of dimension C×1×1 are output; then the feature vector is compressed and restored using a compression excitation module consisting of two fully connected layers; then the two vectors are added and the weight of each channel is generated through the Sigmoid activation function; finally, the learned weight is applied to the original feature map to obtain the channel attention calculation output. The mathematical representation of this process can be, but is not limited to, as:
[0051] CA(G)=σ(FC(AvgPool(G))+MaxPool(G)),
[0052] Among them, σ(·) represents the Sigmoid activation function, FC(·) represents the compression-excitation fully connected module, AvgPool(·) and MaxPool(·) represent the average pooling and maximum pooling calculations along the feature channel direction, respectively.
[0053] Furthermore, in the embodiment of the present application, the spatial attention module is used to emphasize the important areas in the feature space, and its specific structure is as follows: Figure 2 As shown. First, the feature channel is reduced in dimension through global pooling and maximum pooling, and then the two pooling outputs are spliced along the feature channel into a feature map of size 2×N×L; then a convolutional layer is used for learning, and the Sigmoid activation function is used to generate the weight of each spatial position; finally, the learned weight is applied to the original feature map to obtain the spatial attention calculation output. The spatial attention calculation process can be, but is not limited to:
[0054] SA(G)=σ(Conv([AvgPool(G);MaxPool(G)])),
[0055] Among them, Conv(·) represents a two-dimensional convolutional layer with a convolution kernel size of 7×7, which is used to learn the spatial attention weights.
[0056] Furthermore, the embodiment of the present application can splice the output of the local feature extraction branch and the convolution attention branch along the feature channel, use the channel shuffle operation to achieve feature fusion, and obtain the final output feature of the spectrum modeling unit. Among them, in the embodiment of the present application, the local feature extraction branch uses deep separable convolution to extract the local pattern of the feature map, while the convolution attention branch can effectively use the global information to adjust the feature map, giving greater weight to the areas and feature channels that contribute to speech separation. The two branches complement each other, so that the spectral modeling unit has strong feature extraction and spectral modeling capabilities.
[0057] In step S103, a sub-band modeling unit and a full-band modeling unit of an initial dual-path fast converter neural network are respectively constructed according to cross linear attention and ConFastformer.
[0058] In some embodiments, the sub-band modeling unit of the embodiment of the present application first converts the dimension of the input feature into F×B×T, and then uses ConFastformer to model the last feature dimension, that is, to model the time dependency of all sub-bands. In the embodiment of the present application, ConFastformer mainly consists of two fully connected layers, a multi-scale convolution module and a multi-head linear attention. The network structure is as follows: Figure 3 As shown. Assume the input feature is First, the input features are dimensionally upgraded through a fully connected layer with LN (Layer Normalization) and ReLu activation functions. The formula for this process can be, but is not limited to, expressed as:
[0059]
[0060] in, represents the output feature of the fully connected layer, FC(·) represents the fully connected layer, H represents the output channel of the fully connected layer, and LN(·) represents the layer normalization calculation.
[0061] Further, in the embodiment of the present application, the multi-scale convolution module is composed of three channel-by-channel convolution modules of different sizes, which are used to extract local feature patterns and simulate the correlation between adjacent time-frequency features. A jump connection mechanism is used between each channel-by-channel convolution module to fuse feature information of different scales. The formula of the multi-scale convolution module modeling process can be expressed as, but not limited to:
[0062]
[0063] in, Indicates that the convolution kernel size is k i Channel-wise one-dimensional convolution, represents the output of the i-th channel-by-channel convolution module, is the final output of the multi-scale convolution module.
[0064] Furthermore, the embodiment of the present application can use a multi-head linear attention mechanism to model the output of a multi-scale convolution module. Specifically, the embodiment of the present application first converts the input features into a query vector Q, a key vector K and a value vector V through a linear layer; then, according to the preset number of attention heads h, Q, K and V are divided into h parts respectively, denoted as {Q i ,K i ,V i}, where i = 0, ..., h-1, and the linear attention Z of each head is calculated accordingly i ; Then the embodiment of the present application concatenates the output of each attention head along the feature channel, and fuses all attention head features through a linear layer, and combines them with the query vector Q and the multi-scale convolution module output After adding, we get the multi-head linear attention output Finally, the fully connected layer is used to transform the feature dimension into F×H×T and restore it to the original feature dimension to obtain the output feature of the ConFastformer module.
[0065] Furthermore, the embodiment of the present application can use the cross-linear attention module to fuse the local features obtained by the spectrum modeling unit with the sub-band features output by the ConFastformer module. In the embodiment of the present application, the cross-linear attention is improved based on the linear attention mechanism of Fastformer. The network block diagram is as follows: Figure 4 As shown. For different types of input features G to be fused 1 and G 2 , first use LN to normalize the input features, and then use the linear layer to transform the input features G 1 Convert it into Q vector and input feature G 2 Converted into K vector and V vector, in order to simplify the calculation, the K vector in the proposed cross linear attention is equal to the V vector. Then according to the preset number of attention heads h, Q, K and V are divided into h parts respectively, expressed as {q i ,K i ,V i}, where i = 0, ..., h-1, and the linear attention Z of each head is calculated accordingly i The output of each attention head is concatenated along the feature channel, and then the features of each attention head are fused through a linear layer and added to the query vector Q to obtain the cross linear attention output G. 12 .
[0066] In some embodiments, the full-band modeling unit of the embodiment of the present application first converts the feature dimension into T×B×F, and the remaining calculation steps are consistent with the sub-band modeling process. Through full-band modeling, all sub-band information at each time point can be fused, thereby effectively extracting global frequency information. The full-band modeling unit independently processes the full-band information of each frame, and the feature parameters of each frame are shared. Recently, more and more studies have found that modeling the frequency dependency of speech helps to restore spectral details, thereby improving the quality of speech separation. In addition, for multi-channel speech signals, it becomes particularly difficult to accurately obtain the spatial feature information of the speaker from the sub-band features in the case of severe interference noise or reverberation, and the direct path response of the sound source has a highly correlated characteristic in frequency, which helps to solve the problem of spatial information ambiguity caused by strong reverberation and low signal-to-noise ratio. Since the DP-RTF shape and amplitude of sound sources in different directions are significantly different in the frequency dimension, modeling the frequency context information is also beneficial to extract the speaker's direction information from it, further improving the quality of separated speech.
[0067] In step S104, an initial dual-path fast transformer neural network of an initial multi-channel speech separation model is constructed based on the spectrum modeling unit, the sub-band modeling unit and the full-band modeling unit.
[0068] As a possible implementation method, the embodiment of the present application can construct an initial dual-path fast converter neural network of an initial multi-channel speech separation model based on a spectrum modeling unit, a sub-band modeling unit and a full-band modeling unit.
[0069] In step S105, a mask module of an initial multi-channel speech separation model is constructed based on a fully connected layer of a Tanh activation function and a fully connected layer of a Sigmoid activation function.
[0070] It can be understood by those skilled in the art that the embodiment of the present application can convert the output result of the initial dual-path fast converter neural network into a complex mask for estimating the separated speech through a mask module, and apply the mask to the complex spectrum of the mixed speech signal. The specific method can be:
[0071] In the embodiment of the present application, the output of the initial dual-path fast converter neural network is divided into two paths, one for phase estimation and the other for amplitude estimation. In the phase estimation process of the embodiment of the present application, the output result of the initial dual-path fast converter neural network is sent to a fully connected layer, and the output is mapped to between -1 and 1 through the Tanh activation function as a complex mask for preliminary estimation, and then the phase mask is calculated based on the complex mask, and the formula can be but is not limited to expressed as:
[0072]
[0073] in, represents DPFNN output, Tanh(·) represents Tanh activation function, and denote the real and imaginary parts of the preliminary estimated complex mask, respectively. represents the estimated phase mask, and P represents the number of speakers in the mixed speech.
[0074] Furthermore, in the embodiment of the present application, the phase mask and the reference channel mixed speech phase spectrum are element-by-element multiplied to obtain the phase of the separated speech, and the formula thereof can be, but is not limited to, expressed as:
[0075] S phase =M phase ·Y phase ,
[0076] in, and They represent the reference channel mixed speech phase spectrum and separated speech phase spectrum respectively.
[0077] In the amplitude estimation process of the embodiment of the present application, the output of the DPFNN is sent to another fully connected layer with a Sigmoid activation function to obtain an amplitude mask, and the amplitude mask is multiplied element by element with the mixed speech amplitude spectrum of the reference channel to obtain the amplitude of the separated speech. The formula can be, but is not limited to, expressed as:
[0078]
[0079] in, and They represent the reference channel mixed speech amplitude spectrum and the separated speech amplitude spectrum respectively.
[0080] Furthermore, the embodiment of the present application can combine the amplitude spectrum and the phase spectrum of the separated speech, and the final complex spectrum of the separated speech can be expressed as but not limited to:
[0081]
[0082] in, and Respectively represent the real and imaginary parts of the complex spectrum of the separated speech.
[0083] In step S106, an initial multi-channel speech separation model is constructed based on the feature extraction module, the initial dual-path fast transformer neural network and the mask module.
[0084] As a possible implementation method, the embodiment of the present application can construct an initial multi-channel speech separation model based on a feature extraction module, an initial dual-path fast converter neural network and a mask module.
[0085] In step S107, the target multi-channel speech is input into the initial multi-channel speech separation model, the target mask is used as the optimization target of the initial multi-channel speech separation model, the target parameters of the initial dual-path fast converter neural network are optimized, and the initial multi-channel speech separation model is updated based on the optimized dual-path fast converter neural network to obtain the multi-channel speech separation model, and the separated speech corresponding to the target multi-channel speech is obtained based on the multi-channel speech separation model.
[0086] It can be understood that the target multi-channel speech in the embodiment of the present application comes from the LibriSpeech (library speech) dataset, and the noise data comes from the WHAM (WSJ0 Hipster Ambient Mixtures, a dataset containing a mixture of various environmental noises and speech) dataset.
[0087] During the actual implementation process, the embodiment of the present application can input the target multi-channel speech into the initial multi-channel speech separation model, and use the target mask as the learning target, optimize the target parameters of the initial dual-path fast converter neural network, and then obtain the multi-channel speech separation model based on the optimized dual-path fast converter neural network, and then use the constructed multi-channel speech separation model to obtain the separated speech corresponding to the target multi-channel speech.
[0088] Optionally, in one embodiment of the present application, the target multi-channel speech is input into an initial multi-channel speech separation model, the target mask is used as the optimization target of the initial multi-channel speech separation model, and the target parameters of the initial dual-path fast converter neural network are optimized, including: combining the target multi-channel speech and the feature extraction module to obtain the spectrum and frequency domain spatial features of the target multi-channel speech; and obtaining the target feature map of the target multi-channel speech based on the real part of the spectrum, the imaginary part of the spectrum and the frequency domain spatial features.
[0089] In the actual implementation process, the embodiment of the present application can pass the target multi-channel speech through the feature extraction module to calculate the spectrum and frequency domain spatial features of the target multi-channel speech, and then stack the real part of the spectrum, the imaginary part of the spectrum and the frequency domain spatial features along the feature channel dimension to obtain the target feature map of the target multi-channel speech. The specific process can be:
[0090] In the embodiment of the present application, it is assumed that there are P speakers and M microphones in the recording environment, and the target multi-channel speech signal received by the microphone array can be expressed as:
[0091]
[0092] Among them, s p (n) represents the speech signal of the pth speaker, * represents the convolution operator, h p (n) represents the RIR (Room Impulse Response) from the pth speaker to the microphone array, and v(n) represents the additive noise received by the microphone array.
[0093] Furthermore, the embodiment of the present application uses STFT to transform the time-domain multi-channel speech signal y(n) of the target multi-channel speech into the time-frequency domain. At this time, the formula of the target multi-channel speech signal can be re-expressed as:
[0094]
[0095] Among them, Y(t,f) represents the multi-channel mixed speech spectrum, S p (t,f), and denote the speech signal, direct path signal and reverberation signal spectra of the pth speaker respectively, t and f denote the time and frequency indexes respectively, and H(f) denotes the frequency response of RIR.
[0096] Furthermore, the embodiment of the present application uses IPD as an input feature, wherein the calculation formula of IPD may be but is not limited to:
[0097] IPD ij =∠Y i (t,f)-∠Y j (t,f),
[0098] Among them, ∠Y i (t,f) and ∠Y j (t, f) represent the phases of the mixed signals of the i-th channel and the j-th channel respectively.
[0099] In some embodiments, the process of obtaining the target feature map based on the real part of the spectrum, the imaginary part of the spectrum and the frequency domain spatial features in the embodiment of the present application can be:
[0100] It is understandable that in order to better utilize the spectral information of the target multi-channel speech and effectively reconstruct the amplitude and phase of the separated speech, the embodiment of the present application can use the real part and the imaginary part of the spectrum Y(t,f) of the target multi-channel speech as the input spectral features. In addition, since IPD is a periodic function in the embodiment of the present application, in order to avoid the problem of feature representation ambiguity, cosIPD and sinIPD can be used as input spatial features. Furthermore, the embodiment of the present application can combine spectral features and spatial features to obtain a target feature map of the target multi-channel speech, wherein the target feature map can be, but is not limited to, represented as:
[0101] F=[Y real ; Yi imag ; cosIPD; sinIPD],
[0102] Among them, Y real and Y imag Respectively represent the real and imaginary parts of the mixed signal spectrum Y(t,f), It represents the model input features calculated by the feature extraction module. Cin, T and F represent the number of feature channels, frames and frequencies respectively.
[0103] Optionally, in one embodiment of the present application, the target multi-channel speech is input into an initial multi-channel speech separation model, the target mask is used as the optimization target of the initial multi-channel speech separation model, and the target parameters of the initial dual-path fast converter neural network are optimized, which also includes: combining the target feature map and the spectrum modeling unit to obtain the spectral local information and spectral spatial information of the target multi-channel speech; based on the spectral local information and the spectral spatial information, the time context information of the target multi-channel speech is obtained according to the sub-band modeling unit; based on the time context information, the frequency context information of the target multi-channel speech is obtained according to the full-band modeling unit; based on the frequency context information, a separable high-dimensional feature representation of the target multi-channel speech is obtained.
[0104] It will be understood by those skilled in the art that the embodiments of the present application can combine the target feature map and the spectrum modeling unit to obtain the spectral local information and spectral spatial information of the target multi-channel speech, and based on the spectral local information and the spectral spatial information, obtain the time context information and frequency context information of the target multi-channel speech, and then obtain a separable high-dimensional feature representation of the target multi-channel speech.
[0105] In some embodiments, the embodiment of the present application can extract the local spectrum information and the spectrum space information of the target multi-channel speech from the target feature map through the spectrum modeling unit. The specific process can be: first, the target feature map F is fused and the input features are upgraded through feature coding, and then the local spectrum information and the spectrum space information are extracted using SCBAM. The formula can be, but is not limited to, expressed as:
[0106]
[0107] in, represents the output features of the feature encoding module, B represents the output channel of the feature encoding module, Encoder(·) represents the feature encoding module in the spectral modeling unit, SCBAM(·) represents the shuffled convolutional attention module, Represents the output features of the spectrum modeling unit.
[0108] In some embodiments, the embodiments of the present application can respectively pass the time context information and the frequency context information through the sub-band modeling unit and the full-band modeling unit to obtain a separable high-dimensional feature representation of the target multi-channel speech. The specific process can be: in the sub-band modeling unit of the embodiment of the present application, the dimension of the input feature is first converted into F×B×T, and then the last feature dimension is modeled using ConFastformer, that is, the time dependency of all sub-bands is modeled; then the local features obtained by the cross-linear attention fusion spectrum modeling unit and the sub-band features output by the ConFastformer module are expressed as follows, but not limited to:
[0109]
[0110] in, represents the input features of the sub-band modeling unit, Permute(·) represents the feature dimension conversion operation, CLA(·) represents the cross linear attention unit, F T ' dim represents the output features of the ConFastformer module, Represents the final output features of the sub-band modeling unit.
[0111] Among them, in the embodiment of the present application, the subband modeling unit can process each frequency subband independently, and all frequency subband parameters are shared. In the speech separation task, since the speech of the speaker is not completely overlapped in the time-frequency domain, if the temporal context information can be effectively modeled, the non-overlapping parts of the speech segments before and after the overlapping segments can be used to obtain clearer speaker features to improve the performance of the speech separation model.
[0112] Furthermore, in the full-band modeling unit of the embodiment of the present application, the feature dimension is first converted into T×B×F, and the remaining calculation steps are consistent with the sub-band modeling process. It can be understood that the embodiment of the present application can fuse all sub-band information at each time point through full-band modeling, thereby effectively extracting global frequency information. In addition, the full-band modeling unit of the embodiment of the present application can independently process the full-band information of each frame, and the feature parameters of each frame are shared. Recently, more and more studies have found that modeling the frequency dependency of speech helps to restore spectral details, thereby improving the quality of speech separation.
[0113] In addition, for multi-channel speech signals, in the case of severe interference noise or reverberation, it becomes particularly difficult to accurately obtain the spatial feature information of the target speaker from the sub-band features, and the direct path response of the sound source has a highly correlated characteristic in frequency, which can help solve the problem of spatial information ambiguity caused by strong reverberation and low signal-to-noise ratio. Since the relative transfer function shape and amplitude of sound sources in different directions are significantly different in the frequency dimension, modeling the frequency context information is also conducive to extracting the speaker direction information from it, further improving the quality of separated speech.
[0114] Optionally, in one embodiment of the present application, the target multi-channel speech is input into an initial multi-channel speech separation model, the target mask is used as the optimization target of the initial multi-channel speech separation model, and the target parameters of the initial dual-path fast converter neural network are optimized, which also includes: obtaining the complex mask of the separated speech corresponding to the target multi-channel speech based on the separable high-dimensional feature representation and mask module; and training the initial multi-channel speech separation model in combination with the complex mask and the target mask to optimize the target parameters of the initial dual-path fast converter neural network.
[0115] In the actual implementation process, the embodiment of the present application can convert the separable high-dimensional feature representation into a complex mask for separating speech through a mask module, and apply the mask to the spectrum of the target multi-channel speech, train the initial multi-channel speech separation model, and obtain the target parameters of the optimized initial multi-channel speech separation model. The process can be:
[0116] The embodiment of the present application uses the target mask as training, wherein the target mask can be understood as an improved complex mask, which can be set by technicians in this field according to actual conditions, and the present application does not make specific restrictions. The implementation process is as follows: Figure 5 The output of the initial dual-path fast converter neural network is divided into two paths, one for amplitude estimation and the other for phase estimation.
[0117] Among them, in the phase estimation of the embodiment of the present application, the output features of the initial dual-path fast converter neural network are sent to a fully connected layer, and the output is mapped to between -1 and 1 through the Tanh activation function as a complex mask for preliminary estimation, and then the phase mask is calculated based on the complex mask, and its formula can be but is not limited to expressed as:
[0118]
[0119] in, represents DPFNN output, Tanh(·) represents Tanh activation function, and denote the real and imaginary parts of the preliminary estimated complex mask, respectively. represents the estimated phase mask, and P represents the number of speakers in the mixed speech.
[0120] Furthermore, in the embodiment of the present application, the phase mask is element-by-element multiplied by the phase spectrum of the mixed speech of the reference channel to obtain the phase spectrum of the separated speech, and the formula thereof can be, but is not limited to, expressed as:
[0121] S phase =M phase ·Y phase ,
[0122] in, and They represent the reference channel mixed speech phase spectrum and separated speech phase spectrum respectively.
[0123] In the embodiment of the present application, in the amplitude estimation, the output of the initial dual-path fast converter neural network is sent to another fully connected layer with a Sigmoid activation function to obtain an amplitude mask, and the amplitude mask is multiplied element by element with the reference channel mixed speech amplitude spectrum to obtain the amplitude spectrum of the separated speech, and the formula can be but is not limited to:
[0124]
[0125] in, and They represent the reference channel mixed speech amplitude spectrum and the separated speech amplitude spectrum respectively.
[0126] Furthermore, the embodiment of the present application can combine the amplitude spectrum and the phase spectrum of the separated speech, and the final complex spectrum of the separated speech can be expressed as but not limited to:
[0127]
[0128] in, and Respectively represent the real and imaginary parts of the complex spectrum of the separated speech.
[0129] Optionally, in one embodiment of the present application, separated speech corresponding to the target multi-channel speech is obtained based on the multi-channel speech separation model, including: converting the initial separated speech to obtain the time domain waveform of the initial separated speech; and obtaining the separated speech based on the time domain waveform.
[0130] In some embodiments, the embodiment of the present application may transform the time domain waveform of the initial separated speech through ISTFT (Inverse Short-Time Fourier Transform), and the specific process may be:
[0131] It can be understood that ISTFT is the inverse process of STFT, which is used to convert the complex spectrum of the signal in the time-frequency domain into a time-domain waveform. Therefore, the embodiment of the present application can first perform an IDFT (Inverse Discrete Fourier Transform) operation on each frame of the complex spectrum, and its formula can be, but is not limited to, expressed as:
[0132]
[0133] in, Represents the time domain signal of the mth frame after IDFT.
[0134] Then the embodiment of the present application can use the comprehensive window The purpose of this step is to reduce the adverse effects of the truncation effect and suppress the subjective auditory discontinuity of the frame edge. The comprehensive window may include, but is not limited to, a root mean square Hanning window and a Blackman window, etc., and this application does not make specific restrictions. Specifically, the embodiment of this application can use a comprehensive window z(n) to After weighting, the output is in, It can be expressed as, but not limited to:
[0135]
[0136] Finally, the embodiment of this application can Overlap and add to get the final time domain waveform
[0137] The working principle of the multi-channel speech separation method based on neural network proposed in the embodiment of the present application is introduced in detail below with reference to a specific embodiment.
[0138] Figure 6 The present invention is a flowchart of the working principle of a multi-channel speech separation method based on a neural network provided according to an embodiment of the present application.
[0139] Step S601: Acquire target multi-channel speech.
[0140] That is to say, the embodiment of the present application can obtain the target multi-channel speech.
[0141] Step S602: construct a feature extraction module based on STFT and IPD.
[0142] That is to say, the embodiment of the present application can use STFT to transform the time domain multi-channel speech signal y(n) of the target multi-channel speech into the time-frequency domain, and use IPD as the input feature to construct a feature extraction module, and then use the feature extraction module to obtain the target feature map of the target multi-channel speech.
[0143] Step S603: constructing an initial dual-path fast converter neural network based on the spectrum modeling unit, the sub-band modeling unit and the full-band modeling unit.
[0144] That is to say, the embodiment of the present application can utilize the spectral modeling unit, the sub-band modeling unit, and the full-band modeling unit to construct an initial dual-path fast converter neural network, wherein the embodiment of the present application can extract the spectral local information and spectral spatial information of the target multi-channel speech from the target feature map through the spectral modeling unit, and pass the time context information and the frequency context information through the sub-band modeling unit and the full-band modeling unit to obtain a separable high-dimensional feature representation of the target multi-channel speech.
[0145] Step S604: construct a mask module.
[0146] That is to say, the embodiment of the present application can construct a mask module, pass the separable high-dimensional feature representation through the mask module, convert it into a complex mask for separating speech, and apply the mask to the spectrum of the target multi-channel speech, train the initial multi-channel speech separation model, and obtain the target parameters of the optimized initial multi-channel speech separation model.
[0147] Step S605: transforming the initial separated speech to obtain the time domain waveform of the initial separated speech.
[0148] That is to say, the embodiment of the present application can transform the time domain waveform of the initial separated speech through ISTFT.
[0149] Step S606: Output the separated speech based on the time domain waveform.
[0150] That is to say, the embodiment of the present application can use a comprehensive window to weight the time domain waveform and then output it to output the final separated speech.
[0151] According to the multi-channel speech separation method based on neural network proposed in the embodiment of the present application, an initial multi-channel speech separation model can be constructed based on the feature extraction module, the initial dual-path fast converter neural network and the mask module, and the target mask is used as the training target to optimize the target parameters to obtain the multi-channel speech separation model, and then the separated speech corresponding to the target multi-channel speech is obtained based on the multi-channel speech separation model, wherein the multi-channel speech separation model constructed in the embodiment of the present application has a more lightweight network structure, can accurately and stably achieve high-quality speech separation, and exhibits excellent generalization ability under different mixing conditions, showing good practical value and technical prospects. Thus, the problem that the speech separation technology based on microphone array in the related technology cannot meet the requirements of lightweight, high robustness and high generalization, and adapt to complex and changeable acoustic environments, and in the multi-channel speech separation algorithm based on neural network, the model has large number of parameters and calculation amount, and it is difficult to adapt to mobile devices with limited computing power or memory.
[0152] Next, a multi-channel speech separation device based on a neural network proposed in an embodiment of the present application is described with reference to the accompanying drawings.
[0153] Figure 7 A block diagram of a multi-channel speech separation device based on a neural network provided in an embodiment of the present application.
[0154] like Figure 7 As shown, the neural network-based multi-channel speech separation device 10 is applied to the model construction stage, wherein the device 10 includes: a first construction module 100, a second construction module 200, a third construction module 300, a fourth construction module 400, a fifth construction module 500, a sixth construction module 600 and a first generation module 700.
[0155] The first construction module 100 is used to construct a feature extraction module of an initial multi-channel speech separation model based on STFT and IPD.
[0156] The second building module 200 is used to build a spectrum modeling unit of an initial dual-path fast converter neural network based on feature coding and SCBAM.
[0157] The third building module 300 is used to respectively build a sub-band modeling unit and a full-band modeling unit of the initial dual-path fast converter neural network according to the cross linear attention and ConFastformer.
[0158] The fourth building module 400 is used to build an initial dual-path fast transformer neural network of an initial multi-channel speech separation model based on the spectrum modeling unit, the sub-band modeling unit and the full-band modeling unit.
[0159] The fifth building module 500 is used to build a mask module of the initial multi-channel speech separation model based on the fully connected layer of the Tanh activation function and the fully connected layer of the Sigmoid activation function.
[0160] The sixth building module 600 is used to build an initial multi-channel speech separation model based on the feature extraction module, the initial dual-path fast transformer neural network and the mask module.
[0161] The first generation module 700 is used to input the target multi-channel speech into the initial multi-channel speech separation model, use the target mask as the optimization target of the initial multi-channel speech separation model, optimize the target parameters of the initial dual-path fast converter neural network, update the initial multi-channel speech separation model based on the optimized dual-path fast converter neural network, obtain the multi-channel speech separation model, and obtain the separated speech corresponding to the target multi-channel speech based on the multi-channel speech separation model.
[0162] Optionally, in one embodiment of the present application, the first generation module 700 includes: a first generation unit and a second generation unit.
[0163] The first generating unit is used to combine the target multi-channel speech and the feature extraction module to obtain the spectrum and frequency domain space features of the target multi-channel speech.
[0164] The second generating unit is used to obtain a target feature map of the target multi-channel speech based on the real part of the spectrum, the imaginary part of the spectrum and the frequency domain spatial features.
[0165] Optionally, in one embodiment of the present application, the first generation module 700 further includes: a third generation unit, a fourth generation unit, a fifth generation unit and a sixth generation unit.
[0166] Among them, the third generation unit is used to combine the target feature map and the spectrum modeling unit to obtain the spectrum local information and spectrum spatial information of the target multi-channel speech.
[0167] The fourth generating unit is used to obtain the time context information of the target multi-channel speech according to the sub-band modeling unit based on the local spectrum information and the spatial spectrum information.
[0168] The fifth generating unit is used to obtain the frequency context information of the target multi-channel speech according to the full-band modeling unit based on the time context information.
[0169] The sixth generating unit is used to obtain a separable high-dimensional feature representation of the target multi-channel speech based on the frequency context information.
[0170] Optionally, in one embodiment of the present application, the first generating module 700 further includes: an acquiring unit and an optimizing unit.
[0171] The acquisition unit is used to acquire the complex mask of the separated speech corresponding to the target multi-channel speech according to the separable high-dimensional feature representation and the mask module.
[0172] An optimization unit is used to train an initial multi-channel speech separation model by combining the complex mask and the target mask to optimize target parameters of the initial dual-path fast transformer neural network.
[0173] Optionally, in one embodiment of the present application, the first generation module 700 includes: a conversion unit and a seventh generation unit.
[0174] The conversion unit is used to convert the initial separated speech to obtain the time domain waveform of the initial separated speech.
[0175] The seventh generating unit is used to obtain separated speech based on the time domain waveform.
[0176] It should be noted that the aforementioned explanation of the embodiment of the multi-channel speech separation method based on a neural network is also applicable to the multi-channel speech separation device based on a neural network in this embodiment, and will not be repeated here.
[0177] According to the multi-channel speech separation device based on neural network proposed in the embodiment of the present application, an initial multi-channel speech separation model can be constructed based on the feature extraction module, the initial dual-path fast converter neural network and the mask module, and the target mask is used as the training target to optimize the target parameters to obtain the multi-channel speech separation model, and then the separated speech corresponding to the target multi-channel speech is obtained based on the multi-channel speech separation model, wherein the multi-channel speech separation model constructed in the embodiment of the present application has a more lightweight network structure, can accurately and stably achieve high-quality speech separation, and exhibits excellent generalization ability under different mixing conditions, showing good practical value and technical prospects. Thus, the problem that the speech separation technology based on microphone array in the related technology cannot meet the requirements of lightweight, high robustness and high generalization, and adapt to complex and changeable acoustic environments, and in the multi-channel speech separation algorithm based on neural network, the model has large parameters and calculation amount, and it is difficult to adapt to mobile devices with limited computing power or memory.
[0178] The above embodiment describes the model construction phase, and the following describes an embodiment of the use phase.
[0179] Figure 8 The present invention provides a flowchart of another multi-channel speech separation method based on a neural network according to an embodiment of the present application.
[0180] like Figure 8 As shown, the multi-channel speech separation method based on neural network is applied to the use stage of the model, wherein the method comprises the following steps:
[0181] In step S801, multi-channel speech to be separated is obtained.
[0182] As a possible implementation method, the embodiment of the present application can obtain multi-channel speech to be separated.
[0183] For example, the embodiment of the present application may assume that there are P speakers and M microphones in the recording environment, and the target multi-channel speech signal received by the microphone array may be expressed as:
[0184]
[0185] Among them, s p (n) represents the speech signal of the pth speaker, * represents the convolution operator, h p (n) represents the RIR from the pth speaker to the microphone array, and v(n) represents the additive noise received by the microphone array.
[0186] In step S802, the multi-channel speech to be separated is input into a pre-constructed multi-channel speech separation model to obtain separated speech corresponding to the multi-channel speech to be separated, wherein the pre-constructed multi-channel speech separation model is obtained from the multi-channel speech to be separated.
[0187] In the actual implementation process, the embodiment of the present application can input the multi-channel speech to be separated into a pre-built multi-channel speech separation model to obtain the separated speech corresponding to the multi-channel speech to be separated. The specific process can be:
[0188] First, the embodiment of the present application can obtain a target feature map of the target multi-channel speech based on a feature extraction module; then use a dual-path fast converter neural network to obtain a separable high-dimensional feature representation of the target multi-channel speech; then use a mask module to obtain the complex mask of the separated speech corresponding to the target multi-channel speech, and then obtain the final separated speech.
[0189] According to the multi-channel speech separation method based on neural network proposed in the embodiment of the present application, the multi-channel speech to be separated can be input into a pre-built multi-channel speech separation model to obtain the separated speech corresponding to the multi-channel speech to be separated, wherein the embodiment of the present application can learn the local spectrum information and local spatial information of the multi-channel speech to be separated based on the spectrum modeling module, and use the sub-band modeling unit and the full-band modeling unit to extract the time context features and the frequency context features, which can accurately and stably achieve high-quality speech separation, show excellent generalization ability, and show good practical value and technical prospects. Thus, the problem that the speech separation technology based on microphone array in the related technology cannot meet the requirements of lightweight, high robustness and high generalization, and adapt to the complex and changeable acoustic environment, and in the multi-channel speech separation algorithm based on neural network, the model has large number of parameters and calculation amount, and it is difficult to adapt to the problems of mobile devices with limited computing power or memory.
[0190] Next, a multi-channel speech separation device based on a neural network proposed in an embodiment of the present application is described with reference to the accompanying drawings.
[0191] Fig. 9 A block diagram of another neural network-based multi-channel speech separation device provided according to an embodiment of the present application.
[0192] like Fig. 9 As shown, the neural network-based multi-channel speech separation device 20 is applied to the use stage of the model, wherein the device 20 includes: an acquisition module 100 and a second generation module 200.
[0193] The acquisition module 100 is used to acquire multi-channel speech to be separated.
[0194] The second generation module 200 is used to input the multi-channel speech to be separated into a pre-constructed multi-channel speech separation model to obtain the separated speech corresponding to the multi-channel speech to be separated, wherein the pre-constructed multi-channel speech separation model is obtained from the multi-channel speech to be separated.
[0195] It should be noted that the aforementioned explanation of the embodiment of the multi-channel speech separation method based on a neural network is also applicable to the multi-channel speech separation device based on a neural network in this embodiment, and will not be repeated here.
[0196] According to the multi-channel speech separation device based on neural network proposed in the embodiment of the present application, the multi-channel speech to be separated can be input into the pre-built multi-channel speech separation model to obtain the separated speech corresponding to the multi-channel speech to be separated, wherein the embodiment of the present application can learn the local spectrum information and local spatial information of the multi-channel speech to be separated based on the spectrum modeling module, and use the sub-band modeling unit and the full-band modeling unit to extract the time context features and the frequency context features, which can accurately and stably achieve high-quality speech separation, show excellent generalization ability, and show good practical value and technical prospects. Thus, the problem that the speech separation technology based on microphone array in the related technology cannot meet the requirements of lightweight, high robustness and high generalization, and adapt to the complex and changeable acoustic environment, and in the multi-channel speech separation algorithm based on neural network, the model has large number of parameters and calculation, and it is difficult to adapt to the problems of mobile devices with limited computing power or memory.
[0197] Fig.10 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. The electronic device may include:
[0198] A memory 1001 , a processor 1002 , and a computer program stored in the memory 1001 and executable on the processor 1002 .
[0199] When the processor 1002 executes the program, the multi-channel speech separation method based on neural network provided in the above embodiment is implemented.
[0200] Furthermore, the electronic device further comprises:
[0201] The communication interface 1003 is used for communication between the memory 1001 and the processor 1002 .
[0202] The memory 1001 is used to store computer programs that can be executed on the processor 1002 .
[0203] The memory 1001 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.
[0204] If the memory 1001, the processor 1002 and the communication interface 1003 are implemented independently, the communication interface 1003, the memory 1001 and the processor 1002 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0205] Optionally, in a specific implementation, if the memory 1001, the processor 1002 and the communication interface 1003 are integrated on a chip, the memory 1001, the processor 1002 and the communication interface 1003 can communicate with each other through an internal interface.
[0206] The processor 1002 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.
[0207] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned multi-channel speech separation method based on a neural network.
[0208] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, implements the above-mentioned neural network-based multi-channel speech separation method.
[0209] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0210] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0211] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0212] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.
[0213] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of multiple of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0214] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0215] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0216] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. A multi-channel speech separation method based on neural network, characterized in that: Applied to the model construction phase, wherein the method comprises the following steps: Based on short-time Fourier transform (STFT) and inter-channel phase difference (IPD), a feature extraction module of the initial multi-channel speech separation model is constructed; Based on feature encoding and shuffled convolutional attention SCBAM, the spectrum modeling unit of the initial dual-path fast transformer neural network is constructed; Constructing a sub-band modeling unit and a full-band modeling unit of the initial dual-path fast transformer neural network according to cross linear attention and convolution enhanced fast transformer ConFastformer respectively; Based on the spectrum modeling unit, the sub-band modeling unit and the full-band modeling unit, construct an initial dual-path fast transformer neural network of the initial multi-channel speech separation model; Based on the fully connected layer of the Tanh activation function and the fully connected layer of the Sigmoid activation function, a mask module of the initial multi-channel speech separation model is constructed; Constructing the initial multi-channel speech separation model based on the feature extraction module, the initial dual-path fast transformer neural network and the mask module; The target multi-channel speech is input into the initial multi-channel speech separation model, the target mask is used as the optimization target of the initial multi-channel speech separation model, the target parameters of the initial dual-path fast converter neural network are optimized, and the initial multi-channel speech separation model is updated based on the optimized dual-path fast converter neural network to obtain a multi-channel speech separation model, and the separated speech corresponding to the target multi-channel speech is obtained based on the multi-channel speech separation model.
2. The multi-channel speech separation method based on neural network according to claim 1, characterized in that: The step of inputting the target multi-channel speech into the initial multi-channel speech separation model, taking the target mask as the optimization target of the initial multi-channel speech separation model, and optimizing the target parameters of the initial dual-path fast converter neural network comprises: Combining the target multi-channel speech with the feature extraction module, obtaining the spectrum and frequency domain spatial features of the target multi-channel speech; A target feature map of the target multi-channel speech is obtained based on the real part of the spectrum, the imaginary part of the spectrum and the frequency domain spatial features.
3. The multi-channel speech separation method based on neural network according to claim 2 is characterized in that: The step of inputting the target multi-channel speech into the initial multi-channel speech separation model, taking the target mask as the optimization target of the initial multi-channel speech separation model, and optimizing the target parameters of the initial dual-path fast converter neural network also includes: Combining the target feature map with the spectrum modeling unit, obtaining the spectrum local information and spectrum spatial information of the target multi-channel speech; Based on the local spectrum information and the spatial spectrum information, obtaining the temporal context information of the target multi-channel speech according to the sub-band modeling unit; Based on the time context information, obtaining frequency context information of the target multi-channel speech according to the full-band modeling unit; Based on the frequency context information, a separable high-dimensional feature representation of the target multi-channel speech is obtained.
4. The multi-channel speech separation method based on neural network according to claim 3 is characterized in that: The step of inputting the target multi-channel speech into the initial multi-channel speech separation model, taking the target mask as the optimization target of the initial multi-channel speech separation model, and optimizing the target parameters of the initial dual-path fast converter neural network also includes: According to the separable high-dimensional feature representation and the mask module, obtaining a complex mask of the separated speech corresponding to the target multi-channel speech; The initial multi-channel speech separation model is trained in combination with the complex mask and the target mask to optimize target parameters of the initial dual-path fast transformer neural network.
5. The multi-channel speech separation method based on neural network according to claim 1 is characterized in that: The step of obtaining the separated speech corresponding to the target multi-channel speech based on the multi-channel speech separation model includes: Converting the initial separated speech to obtain a time domain waveform of the initial separated speech; The separated speech is obtained based on the time domain waveform.
6. A multi-channel speech separation method based on neural network, characterized in that: Applied to the use phase of the model, wherein the method comprises the following steps: Obtain multi-channel speech to be separated; The multi-channel speech to be separated is input into a pre-constructed multi-channel speech separation model to obtain the separated speech corresponding to the multi-channel speech to be separated, wherein the pre-constructed multi-channel speech separation model is obtained from the multi-channel speech to be separated.
7. A multi-channel speech separation device based on a neural network, characterized in that: Applied to the model construction stage, wherein the device comprises: The first building module is used to build a feature extraction module of an initial multi-channel speech separation model based on STFT and IPD; The second building module is used to build a spectrum modeling unit of an initial dual-path fast converter neural network based on feature coding and SCBAM; A third building module is used to respectively build a sub-band modeling unit and a full-band modeling unit of the initial dual-path fast converter neural network according to cross linear attention and ConFastformer; a fourth building block, configured to build an initial dual-path fast transformer neural network of the initial multi-channel speech separation model based on the spectrum modeling unit, the sub-band modeling unit and the full-band modeling unit; A fifth building module is used to build a mask module of the initial multi-channel speech separation model based on a fully connected layer of a Tanh activation function and a fully connected layer of a Sigmoid activation function; a sixth construction module, configured to construct the initial multi-channel speech separation model based on the feature extraction module, the initial dual-path fast transformer neural network and the mask module; A generation module is used to input the target multi-channel speech into the initial multi-channel speech separation model, use the target mask as the optimization target of the initial multi-channel speech separation model, optimize the target parameters of the initial dual-path fast converter neural network, update the initial multi-channel speech separation model based on the optimized dual-path fast converter neural network, obtain the multi-channel speech separation model, and obtain the separated speech corresponding to the target multi-channel speech based on the multi-channel speech separation model.
8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-channel speech separation method based on a neural network as described in any one of claims 1-5 or 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the neural network-based multi-channel speech separation method as described in any one of claims 1-5 or 6.
10. A computer program product, characterized in that It comprises a computer program, which, when executed, is used to implement the multi-channel speech separation method based on neural network as described in any one of claims 1-5 or 6.
Citation Information
Patent Citations
Voice separation method and method and electronic device
CN110070882A
Speech enhancement method fusing Transform and U-net network
CN114141238A
Voice separation method and device based on multi-channel full convolution time domain network
CN117373477A
Model training and multi-channel speech enhancement method, device and equipment
CN118098254A
Apparatus and method for extracting sound source from multi-channel audio signal
US20190180142A1
Cited By
Multi-channel voice separation method and device based on multi-scale feature channel fusion
CN119252272A
Multi-channel speech enhancement method based on multi-scale spatial information and spectrum feature fusion
CN119889338A
Filtering method and system after voice separation based on sub-band envelope features
CN119964593A
Speech enhancement method and device based on full-band and sub-band fusion, terminal and medium
CN121054024A
Cross-modal speech separation method and system based on multi-scale semantic aggregation strategy
CN122314006A
