Multi-channel speech separation method and device based on neural network

Abstract

The present application relates to the field of speech separation and extraction technology, and in particular to a multi-channel speech separation method and device based on a neural network, wherein the method comprises: constructing a feature extraction module; constructing an initial dual-path fast converter neural network based on a spectrum modeling unit, a sub-band modeling unit, and a full-band modeling unit; constructing a mask module; constructing an initial multi-channel speech separation model based on the feature extraction module, the initial dual-path fast converter neural network, and the mask module; inputting the target multi-channel speech into the initial multi-channel speech separation model, taking the target mask as the optimization target, optimizing the target parameters, and then obtaining a multi-channel speech separation model to achieve separation of the target multi-channel speech. Thus, the problem in the related art that it cannot meet the requirements of lightweight, high robustness, and high generalization, adapt to complex and changeable acoustic environments, and the large number of parameters and calculations make it difficult to adapt to mobile devices with limited computing power or memory is solved.

Description

Technical Field

[0001] The present application relates to the technical field of speech separation and extraction, and in particular to a multi-channel speech separation method and device based on a neural network. Background Art

[0002] In daily life, we are constantly surrounded by various types of sounds. Sounds of interest are often inevitably intertwined with additional noise from other sources and reverberation caused by reflections. Therefore, separating sounds of interest from background interference is crucial for ensuring smooth communication between people and accurate transmission of commands during human-computer interaction.

[0003] In related technologies, speech separation can be divided into single-channel and multi-channel (i.e., microphone array) speech separation according to the number of channels or microphones used. Among them, multi-channel speech separation technology can utilize the spatial information in the microphone array structure to perform multi-channel speech separation in a far-field complex acoustic environment, or it can perform multi-channel speech separation based on a neural network, such as by using multiple stacked TCN (The Comcast Network), LSTM (Long Short-Term Memory) or Transformer modules.

[0004] However, among related technologies, the speech separation technology based on microphone arrays cannot meet the requirements of lightweight, high robustness and high generalization, and cannot adapt to complex and changeable acoustic environments. In the multi-channel speech separation algorithm based on neural networks, the model has a large number of parameters and computational complexity, which makes it difficult to adapt to mobile devices with limited computing power or memory, and urgently needs improvement. Summary of the Invention

[0005] The present application provides a multi-channel speech separation method and device based on a neural network to solve the problems in related technologies, that is, the speech separation technology based on microphone arrays cannot meet the requirements of lightweight, high robustness and high generalization, and adapt to complex and changeable acoustic environments. In the multi-channel speech separation algorithm based on neural networks, the model has a large number of parameters and a large amount of calculation, which makes it difficult to adapt to mobile devices with limited computing power or memory.

[0006] The first embodiment of the present application provides a multi-channel speech separation method based on a neural network, which is applied to the model construction stage, wherein the method includes the following steps: based on STFT (Short-Time Fourier Transform) and IPD (Interchannel Phase Difference), constructing a feature extraction module of an initial multi-channel speech separation model; based on feature coding and SCBAM (Shuffle Convolutional Block Attention Module), constructing a spectrum modeling unit of an initial dual-path fast converter neural network; based on cross linear attention and ConFastformer (Convolution-augmented Fastformer, convolution enhanced fast converter) respectively construct the subband modeling unit and the full-band modeling unit of the initial dual-path fast converter neural network; based on the spectrum modeling unit, the subband modeling unit and the full-band modeling unit, construct the initial dual-path fast converter neural network of the initial multi-channel speech separation model; based on the fully connected layer of the Tanh activation function and the fully connected layer of the Sigmoid activation function, construct the mask module of the initial multi-channel speech separation model; based on the feature extraction module, the initial dual-path fast converter neural network and the mask module, construct the initial multi-channel speech separation model; input the target multi-channel speech into the initial multi-channel speech separation model, use the target mask as the optimization target of the initial multi-channel speech separation model, optimize the target parameters of the initial dual-path fast converter neural network, update the initial multi-channel speech separation model based on the optimized dual-path fast converter neural network, obtain the multi-channel speech separation model, and obtain the separated speech corresponding to the target multi-channel speech based on the multi-channel speech separation model.

[0007] Optionally, in one embodiment of the present application, the target multi-channel speech is input into the initial multi-channel speech separation model, the target mask is used as the optimization target of the initial multi-channel speech separation model, and the target parameters of the initial dual-path fast converter neural network are optimized, including: combining the target multi-channel speech and the feature extraction module to obtain the spectrum and frequency domain spatial features of the target multi-channel speech; and obtaining the target feature map of the target multi-channel speech based on the real part of the spectrum, the imaginary part of the spectrum and the frequency domain spatial features.

[0008] Optionally, in one embodiment of the present application, the target multi-channel speech is input into the initial multi-channel speech separation model, the target mask is used as the optimization target of the initial multi-channel speech separation model, and the target parameters of the initial dual-path fast converter neural network are optimized, which also includes: combining the target feature map and the spectrum modeling unit to obtain the spectral local information and spectral spatial information of the target multi-channel speech; based on the spectral local information and the spectral spatial information, the time context information of the target multi-channel speech is obtained according to the sub-band modeling unit; based on the time context information, the frequency context information of the target multi-channel speech is obtained according to the full-band modeling unit; based on the frequency context information, a separable high-dimensional feature representation of the target multi-channel speech is obtained.

[0009] Optionally, in one embodiment of the present application, the target multi-channel speech is input into the initial multi-channel speech separation model, the target mask is used as the optimization target of the initial multi-channel speech separation model, and the target parameters of the initial dual-path fast converter neural network are optimized, which also includes: obtaining the complex mask of the separated speech corresponding to the target multi-channel speech according to the separable high-dimensional feature representation and the mask module; and training the initial multi-channel speech separation model in combination with the complex mask and the target mask to optimize the target parameters of the initial dual-path fast converter neural network.

[0010] Optionally, in one embodiment of the present application, obtaining the separated speech corresponding to the target multi-channel speech based on the multi-channel speech separation model includes: converting the initial separated speech to obtain the time domain waveform of the initial separated speech; and obtaining the separated speech based on the time domain waveform.

[0011] The second aspect of the present application provides a multi-channel speech separation method based on a neural network, which is applied to the use stage of the model, wherein the method includes the following steps: obtaining the multi-channel speech to be separated; inputting the multi-channel speech to be separated into a pre-built multi-channel speech separation model to obtain the separated speech corresponding to the multi-channel speech to be separated, wherein the pre-built multi-channel speech separation model is obtained from the multi-channel speech to be separated.

[0012] The third aspect of the present application provides a multi-channel speech separation device based on a neural network, which is applied to the model construction stage, wherein the device includes: a first construction module for constructing a feature extraction module of an initial multi-channel speech separation model based on STFT and IPD; a second construction module for constructing a spectrum modeling unit of an initial dual-path fast converter neural network based on feature coding and SCBAM; a third construction module for constructing a sub-band modeling unit and a full-band modeling unit of the initial dual-path fast converter neural network according to cross-linear attention and ConFastformer respectively; a fourth construction module for constructing an initial dual-path fast converter neural network of the initial multi-channel speech separation model based on the spectrum modeling unit, the sub-band modeling unit and the full-band modeling unit; a fifth construction module A block is used to construct a mask module of the initial multi-channel speech separation model based on a fully connected layer of a Tanh activation function and a fully connected layer of a Sigmoid activation function; a sixth construction module is used to construct the initial multi-channel speech separation model based on the feature extraction module, the initial dual-path fast converter neural network and the mask module; a first generation module is used to input the target multi-channel speech into the initial multi-channel speech separation model, use the target mask as the optimization target of the initial multi-channel speech separation model, optimize the target parameters of the initial dual-path fast converter neural network, update the initial multi-channel speech separation model based on the optimized dual-path fast converter neural network, obtain a multi-channel speech separation model, and obtain the separated speech corresponding to the target multi-channel speech based on the multi-channel speech separation model.

[0013] Optionally, in one embodiment of the present application, the first generation module includes: a first generation unit, used to combine the target multi-channel speech and the feature extraction module to obtain the spectrum and frequency domain spatial features of the target multi-channel speech; a second generation unit, used to obtain the target feature map of the target multi-channel speech based on the real part of the spectrum, the imaginary part of the spectrum and the frequency domain spatial features.

[0014] Optionally, in one embodiment of the present application, the first generation module further includes: a third generation unit, used to combine the target feature map and the spectrum modeling unit to obtain the spectrum local information and spectrum spatial information of the target multi-channel speech; a fourth generation unit, used to obtain the time context information of the target multi-channel speech according to the sub-band modeling unit based on the spectrum local information and the spectrum spatial information; a fifth generation unit, used to obtain the frequency context information of the target multi-channel speech according to the full-band modeling unit based on the time context information; and a sixth generation unit, used to obtain a separable high-dimensional feature representation of the target multi-channel speech based on the frequency context information.

[0015] Optionally, in one embodiment of the present application, the first generation module further includes: an acquisition unit for acquiring the complex mask of the separated speech corresponding to the target multi-channel speech based on the separable high-dimensional feature representation and the mask module; an optimization unit for training the initial multi-channel speech separation model in combination with the complex mask and the target mask to optimize the target parameters of the initial dual-path fast converter neural network.

[0016] Optionally, in one embodiment of the present application, the first generation module includes: a conversion unit for converting the initial separated speech to obtain a time domain waveform of the initial separated speech; and a seventh generation unit for obtaining the separated speech based on the time domain waveform.

[0017] The fourth aspect of the present application provides a multi-channel speech separation device based on a neural network, which is applied to the use stage of the model, wherein the device includes: an acquisition module for acquiring the multi-channel speech to be separated; a second generation module for inputting the multi-channel speech to be separated into a pre-built multi-channel speech separation model to obtain the separated speech corresponding to the multi-channel speech to be separated, wherein the pre-built multi-channel speech separation model is obtained from the multi-channel speech to be separated.

[0018] The fifth aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the multi-channel speech separation method based on a neural network as described in the above embodiment.

[0019] The sixth aspect of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the above-mentioned neural network-based multi-channel speech separation method.

[0020] The seventh aspect of the present application provides a computer program product, including a computer program, which, when executed, implements the above-mentioned neural network-based multi-channel speech separation method.

[0021] The embodiment of the present application can construct an initial multi-channel speech separation model based on the feature extraction module, the initial dual-path fast converter neural network and the mask module, and use the target mask as the training target to optimize the target parameters to obtain a multi-channel speech separation model, and then obtain the separated speech corresponding to the target multi-channel speech based on the multi-channel speech separation model. Among them, the multi-channel speech separation model constructed by the embodiment of the present application has a more lightweight network structure, can accurately and stably achieve high-quality speech separation, and shows excellent generalization ability under different mixing conditions, showing good practical value and technical prospects. Therefore, it solves the problem that the speech separation technology based on microphone arrays in the related art cannot meet the requirements of lightweight, high robustness and high generalization, and adapt to complex and changeable acoustic environments, while in the multi-channel speech separation algorithm based on neural networks, the model has a large number of parameters and a large amount of calculation, which is difficult to adapt to mobile devices with limited computing power or memory.

[0022] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0024] Figure 1 A flowchart of a multi-channel speech separation method based on a neural network according to an embodiment of the present application;

[0025] Figure 2 A block diagram of a spectrum modeling unit framework provided according to one embodiment of the present application;

[0026] Figure 3 A schematic block diagram of a ConFastformer network framework consisting of a multi-scale convolution module and a multi-head linear attention according to an embodiment of the present application;

[0027] Figure 4 A block diagram of a cross-linear attention structure provided according to one embodiment of the present application;

[0028] Figure 5 A flowchart of mask estimation according to one embodiment of the present application;

[0029] Figure 6 A flowchart of the working principle of a multi-channel speech separation method based on a neural network provided according to one embodiment of the present application;

[0030] Figure 7Schematic diagram of a multi-channel speech separation device based on a neural network according to an embodiment of the present application;

[0031] Figure 8 A flowchart of another multi-channel speech separation method based on a neural network provided according to an embodiment of the present application;

[0032] Figure 9 Schematic diagram of another multi-channel speech separation device based on a neural network according to an embodiment of the present application;

[0033] Figure 10 A schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0035] The following describes a multi-channel speech separation method and device based on a neural network in accordance with an embodiment of the present application with reference to the accompanying drawings. The microphone array-based speech separation technology mentioned in the background art cannot meet the requirements of lightweight, high robustness, and high generalization, and adapt to complex and changing acoustic environments. In the multi-channel speech separation algorithm based on a neural network, the model has a large number of parameters and a large amount of computation, making it difficult to adapt to mobile devices with limited computing power or memory. The present application provides a multi-channel speech separation method based on a neural network. In this method, an initial multi-channel speech separation model can be constructed based on a feature extraction module, an initial dual-path fast converter neural network, and a mask module. The target mask is used as a training target, and the target parameters are optimized to obtain a multi-channel speech separation model. Furthermore, the separated speech corresponding to the target multi-channel speech is obtained based on the multi-channel speech separation model. The multi-channel speech separation model constructed in accordance with the present application has a more lightweight network structure, can accurately and stably achieve high-quality speech separation, and exhibits excellent generalization capabilities under different mixing conditions, demonstrating good practical value and technical prospects. This solves the problem in related technologies that the microphone array-based speech separation technology cannot meet the requirements of lightweight, high robustness and high generalization, and cannot adapt to complex and changeable acoustic environments. In the multi-channel speech separation algorithm based on neural networks, the model has a large number of parameters and computational complexity, making it difficult to adapt to mobile devices with limited computing power or memory.

[0036] Specifically, Figure 1 The present invention provides a flowchart of a multi-channel speech separation method based on a neural network according to an embodiment of the present application.

[0037] like Figure 1 As shown, the multi-channel speech separation method based on neural network is applied to the model construction stage, wherein the method includes the following steps:

[0038] In step S101, a feature extraction module of an initial multi-channel speech separation model is constructed based on STFT and IPD.

[0039] It is understandable that STFT can transform time domain signals into time-frequency domain, and IPD is a frequency domain spatial feature commonly used in research such as multi-channel speech enhancement, multi-channel speech separation and sound source localization. It can effectively reflect subtle changes in the direction of arrival of the sound source. Since the speakers are located in different directions, the IPD feature helps the model identify non-overlapping speaker speech segments in the mixed speech, and then extract relatively clean speaker direction information and sound information from it to help speech separation.

[0040] As a possible implementation method, the embodiment of the present application can construct a feature extraction module of the initial multi-channel speech separation model based on STFT and IPD.

[0041] In step S102, based on feature coding and SCBAM, a spectrum modeling unit of an initial dual-path fast converter neural network is constructed.

[0042] In the actual implementation process, the spectrum modeling unit of the initial dual-path fast converter neural network of the embodiment of the present application is composed of feature encoding and SCBAM modules, and its structural diagram is as follows: Figure 2 shown.

[0043] Specifically, the embodiment of the present application first fuses the input feature F through feature encoding and upgrades the input feature. The feature encoding consists of a two-dimensional convolution layer, batch normalization, and a ReLU activation function to encode the network input feature and obtain a high-dimensional feature representation. The formula of the calculation process can be, but is not limited to,:

[0044] F enc =BN(ReLU(Conv(F))),

[0045] Among them, Conv represents a two-dimensional convolution operation, the convolution kernel size is 3×3, the stride is 1, and the same padding strategy is adopted to ensure that the feature map size remains unchanged. The ReLu activation function is used to increase the nonlinear representation ability of the network. BN represents batch normalization calculation, which is used to normalize the data according to the batch size, smooth the training process, and avoid gradient explosion or gradient disappearance during network training.

[0046] Furthermore, the embodiment of the present application uses SCBAM, which consists of a local feature extraction branch and a convolutional attention branch to perform local feature extraction and spatial feature extraction respectively. The specific implementation process is as follows: First, the embodiment of the present application divides the input features into two parts along the feature channel, which are represented as

[0047] Among them, in the local feature extraction branch of the embodiment of the present application, the input feature dimension is first transformed into B×T×F through a two-dimensional convolution module with a convolution kernel size of 1×1, thereby increasing the depth and nonlinear representation capability of the network; then the local spectral features are extracted through a depth-separable convolution unit with a convolution kernel size of 3×3; finally, a 1×1 convolution dimensionality reduction feature channel is used to restore the original feature dimension. In this embodiment of the present application, the specific structure of the depthwise separable convolution is as follows: Figure 2 As shown in the figure, it consists of a channel-by-channel convolution module with a convolution kernel size of 3×3 and a point-by-point convolution module with a convolution kernel size of 1×1. The convolution attention branch is composed of a cascade of spatial attention and channel attention modules. The calculation process can be, but is not limited to:

[0048]

[0049] in, represents the input features of the convolutional attention module, represents the output features of the convolutional attention module, CA(·) and SA(·) represent the channel attention module and spatial attention module, respectively.

[0050] Among them, in the embodiment of the present application, the channel attention module can effectively model the correlation between feature channels, and its specific structure is as follows Figure 2 As shown. Assume that the input feature is Here, C, N, and L represent the feature channel, length, and width, respectively. First, global information for each channel is obtained through average pooling and maximum pooling operations, outputting two vectors of dimension C×1×1. The feature vectors are then compressed and restored using a compression-excitation module consisting of two fully connected layers. The two vectors are then added together and activated using a Sigmoid function to generate a weight for each channel. Finally, the learned weights are applied to the original feature map to obtain the channel attention calculation output. The mathematical representation of this process can be, but is not limited to, as follows:

[0051] CA(G)=σ(FC(AvgPool(G))+MaxPool(G)),

[0052] Among them, σ(·) represents the Sigmoid activation function, FC(·) represents the compression-excitation fully connected module, AvgPool(·) and MaxPool(·) represent the average pooling and maximum pooling calculations along the feature channel direction, respectively.

[0053] Furthermore, in the embodiment of the present application, the spatial attention module is used to emphasize the important areas in the feature space, and its specific structure is as follows: Figure 2 As shown in the figure. First, the feature channel is reduced in dimension through global pooling and maximum pooling. The two pooling outputs are then concatenated along the feature channel into a feature map of size 2×N×L. Subsequently, a convolutional layer is used for learning, and a Sigmoid activation function is used to generate weights for each spatial position. Finally, the learned weights are applied to the original feature map to obtain the spatial attention calculation output. The spatial attention calculation process can be, but is not limited to:

[0054] SA(G)=σ(Conv([AvgPool(G);MaxPool(G)])),

[0055] Among them, Conv(·) represents a two-dimensional convolutional layer with a convolution kernel size of 7×7, which is used to learn the spatial attention weights.

[0056] Furthermore, the embodiment of the present application can splice the output of the local feature extraction branch and the convolution attention branch along the feature channel, use the channel shuffling operation to achieve feature fusion, and obtain the final output feature of the spectrum modeling unit. Among them, in the embodiment of the present application, the local feature extraction branch uses depthwise separable convolution to extract local patterns of the feature map, while the convolution attention branch can effectively use global information to adjust the feature map, giving greater weight to areas and feature channels that contribute to speech separation. The two branches complement each other, so that the spectral modeling unit has stronger feature extraction and spectral modeling capabilities.

[0057] In step S103, a sub-band modeling unit and a full-band modeling unit of an initial dual-path fast converter neural network are constructed according to the cross-linear attention and ConFastformer, respectively.

[0058] In some embodiments, the sub-band modeling unit of the embodiment of the present application first converts the dimension of the input feature into F×B×T, and then uses ConFastformer to model the last feature dimension, that is, to model the time dependency of all sub-bands. In the embodiment of the present application, ConFastformer mainly consists of two fully connected layers, a multi-scale convolution module and a multi-head linear attention. The network structure is as follows: Figure 3 As shown. Assume that the input feature is First, the input features are dimensionally increased through a fully connected layer with LN (Layer Normalization) and ReLu activation functions. This process can be expressed as, but not limited to:

[0059]

[0060] in, represents the output features of the fully connected layer, FC(·) represents the fully connected layer, H represents the output channel of the fully connected layer, and LN(·) represents the layer normalization calculation.

[0061] Furthermore, in the embodiment of the present application, the multi-scale convolution module is composed of three channel-by-channel convolution modules of different sizes, which are used to extract local feature patterns and simulate the correlation between adjacent time-frequency features. A skip connection mechanism is used between each channel-by-channel convolution module to fuse feature information of different scales. The formula for the modeling process of the multi-scale convolution module can be expressed as, but not limited to:

[0062]

[0063] in, Indicates that the convolution kernel size is k i Channel-wise one-dimensional convolution, represents the output of the i-th channel-by-channel convolution module, is the final output of the multi-scale convolution module.

[0064] Furthermore, the embodiment of the present application can use the multi-head linear attention mechanism to model the output of the multi-scale convolution module Specifically, the embodiment of the present application first converts the input features into a query vector Q, a key vector K and a value vector V through a linear layer; then, according to the preset number of attention heads h, Q, K and V are divided into h parts respectively, denoted as {Q i ,K i ,V i}, where i = 0, ..., h-1, and the linear attention Z of each head is calculated accordingly i ; Then the embodiment of the present application concatenates the output of each attention head along the feature channel and fuses all attention head features through a linear layer, and combines them with the query vector Q and the multi-scale convolution module output After adding, we get the multi-head linear attention output Finally, the fully connected layer is used to convert the feature dimension into F×H×T and restore it to the original feature dimension to obtain the output feature of the ConFastformer module.

[0065] Furthermore, the embodiment of the present application can use the cross-linear attention module to fuse the local features obtained by the spectrum modeling unit and the sub-band features output by the ConFastformer module. In this embodiment of the present application, the cross-linear attention is improved based on the linear attention mechanism of Fastformer. The network block diagram is as follows: Figure 4 As shown. For different types of input features G1 and G2 to be fused, first use LN to normalize the input features, then use the linear layer to convert the input feature G1 into a Q vector, and the input feature G2 into a K vector and a V vector. In order to simplify the calculation, the K vector in the proposed cross linear attention is equal to the V vector. Then, according to the preset number of attention heads h, Q, K and V are divided into h parts respectively, expressed as {q i ,K i ,V i}, where i = 0, ..., h-1, and the linear attention Z of each head is calculated accordingly i The output of each attention head is concatenated along the feature channel and then fused with the features of each attention head through a linear layer and added to the query vector Q to obtain the cross linear attention output G 12 .

[0066] In some embodiments, the full-band modeling unit of the embodiment of the present application first converts the feature dimension into T×B×F, and the remaining calculation steps are consistent with the sub-band modeling process. Through full-band modeling, all sub-band information at each time point can be integrated, thereby effectively extracting global frequency information. The full-band modeling unit independently processes the full-band information of each frame, and the feature parameters of each frame are shared. Recently, more and more studies have found that modeling the frequency dependency of speech helps to restore spectral details, thereby improving the quality of speech separation. In addition, for multi-channel speech signals, in the case of severe interference noise or reverberation, it becomes particularly difficult to accurately obtain the spatial feature information of the speaker from the sub-band features. The direct path response of the sound source has a highly correlated characteristic in frequency. Utilizing this characteristic helps to solve the spatial information ambiguity problem caused by strong reverberation and low signal-to-noise ratio. Since the DP-RTF shape and amplitude of sound sources in different directions are significantly different in the frequency dimension, modeling the frequency context information is also beneficial to extract the speaker direction information from it, further improving the quality of separated speech.

[0067] In step S104, an initial dual-path fast transformer neural network of an initial multi-channel speech separation model is constructed based on the spectrum modeling unit, the sub-band modeling unit and the full-band modeling unit.

[0068] As a possible implementation method, the embodiment of the present application can construct an initial dual-path fast converter neural network of an initial multi-channel speech separation model based on a spectrum modeling unit, a sub-band modeling unit and a full-band modeling unit.

[0069] In step S105, a mask module of the initial multi-channel speech separation model is constructed based on a fully connected layer of a Tanh activation function and a fully connected layer of a Sigmoid activation function.

[0070] Those skilled in the art will appreciate that, in the embodiment of the present application, the output result of the initial dual-path fast converter neural network can be converted into a complex mask for estimating the separated speech through a mask module, and the mask is applied to the complex spectrum of the mixed speech signal. The specific method can be:

[0071] In this embodiment of the present application, the output of the initial dual-path fast converter neural network is divided into two paths: one for phase estimation and the other for amplitude estimation. In this embodiment of the present application, during the phase estimation process, the output of the initial dual-path fast converter neural network is fed into a fully connected layer, where the output is mapped to a value between -1 and 1 using a Tanh activation function as a complex mask for preliminary estimation. The phase mask is then calculated based on the complex mask, which can be expressed as, but not limited to, the following formula:

[0072]

[0073] in, represents the DPFNN output, Tanh(·) represents the Tanh activation function, and denote the real and imaginary parts of the preliminary estimated complex mask, respectively. represents the estimated phase mask, and P represents the number of speakers in the speech mixture.

[0074] Furthermore, in the embodiment of the present application, the phase mask and the reference channel mixed speech phase spectrum can be multiplied element by element to obtain the phase of the separated speech, and the formula can be, but is not limited to, expressed as:

[0075] S phase =M phase ·Y phase ,

[0076] in, and They represent the reference channel mixed speech phase spectrum and separated speech phase spectrum respectively.

[0077] In the amplitude estimation process of the embodiment of the present application, the output of the DPFNN is fed into another fully connected layer with a Sigmoid activation function to obtain an amplitude mask, and the amplitude mask is multiplied element-by-element by the amplitude spectrum of the mixed speech of the reference channel to obtain the amplitude of the separated speech. The formula can be, but is not limited to, expressed as:

[0078]

[0079] in, and They represent the reference channel mixed speech amplitude spectrum and the separated speech amplitude spectrum respectively.

[0080] Furthermore, the embodiment of the present application can combine the amplitude spectrum and the phase spectrum of the separated speech, and the final complex spectrum of the separated speech can be expressed as, but not limited to:

[0081]

[0082] in, and Represent the real and imaginary parts of the complex spectrum of the separated speech respectively.

[0083] In step S106, an initial multi-channel speech separation model is constructed based on the feature extraction module, the initial dual-path fast transformer neural network and the mask module.

[0084] As a possible implementation method, the embodiment of the present application can construct an initial multi-channel speech separation model based on a feature extraction module, an initial dual-path fast converter neural network and a mask module.

[0085] In step S107, the target multi-channel speech is input into the initial multi-channel speech separation model, the target mask is used as the optimization target of the initial multi-channel speech separation model, the target parameters of the initial dual-path fast converter neural network are optimized, and the initial multi-channel speech separation model is updated based on the optimized dual-path fast converter neural network to obtain a multi-channel speech separation model, and the separated speech corresponding to the target multi-channel speech is obtained based on the multi-channel speech separation model.

[0086] It can be understood that the target multi-channel speech in the embodiment of the present application comes from the LibriSpeech (library speech) dataset, and the noise data comes from the WHAM (WSJ0 Hipster Ambient Mixtures, a dataset containing a mixture of various environmental noises and speech) dataset.

[0087] During the actual implementation process, the embodiment of the present application can input the target multi-channel speech into the initial multi-channel speech separation model, and use the target mask as the learning target to optimize the target parameters of the initial dual-path fast converter neural network, and then obtain the multi-channel speech separation model based on the optimized dual-path fast converter neural network, and then realize the use of the constructed multi-channel speech separation model to obtain the separated speech corresponding to the target multi-channel speech.

[0088] Optionally, in one embodiment of the present application, the target multi-channel speech is input into the initial multi-channel speech separation model, the target mask is used as the optimization target of the initial multi-channel speech separation model, and the target parameters of the initial dual-path fast converter neural network are optimized, including: combining the target multi-channel speech and the feature extraction module to obtain the spectrum and frequency domain spatial features of the target multi-channel speech; and obtaining the target feature map of the target multi-channel speech based on the real part of the spectrum, the imaginary part of the spectrum and the frequency domain spatial features.

[0089] In the actual implementation process, the embodiment of the present application can pass the target multi-channel speech through the feature extraction module to calculate the spectrum and frequency domain spatial features of the target multi-channel speech, and then stack the real part of the spectrum, the imaginary part of the spectrum and the frequency domain spatial features along the feature channel dimension to obtain the target feature map of the target multi-channel speech. The specific process can be:

[0090] In this embodiment of the present application, it is assumed that there are P speakers and M microphones in the recording environment. The target multi-channel speech signal received by the microphone array can be expressed as:

[0091]

[0092] Among them, s p (n) represents the speech signal of the pth speaker, * represents the convolution operator symbol, h p (n) represents the RIR (Room Impulse Response) from the p-th speaker to the microphone array, and v(n) represents the additive noise received by the microphone array.

[0093] Furthermore, the embodiment of the present application uses STFT to transform the time-domain multi-channel speech signal y(n) of the target multi-channel speech into the time-frequency domain. At this time, the formula of the target multi-channel speech signal can be re-expressed as:

[0094]

[0095] Among them, Y(t,f) represents the multi-channel mixed speech spectrum, S p (t,f), and denote the speech signal, direct path signal and reverberation signal spectra of the pth speaker respectively, t and f denote the time and frequency indices respectively, and H(f) denotes the frequency response of RIR.

[0096] Furthermore, the embodiment of the present application uses IPD as an input feature, wherein the calculation formula of IPD can be but is not limited to:

[0097] IPD ij =∠Y i (t,f)-∠Y j(t,f),

[0098] Among them, ∠Y i (t,f) and ∠Y j (t, f) represent the phases of the mixed signals of the i-th channel and the j-th channel respectively.

[0099] In some embodiments, the process of obtaining the target feature map based on the real part of the spectrum, the imaginary part of the spectrum, and the frequency domain spatial features in the embodiment of the present application may be:

[0100] It is understandable that, in order to better utilize the spectral information of the target multi-channel speech and effectively reconstruct the amplitude and phase of the separated speech, the embodiment of the present application can use the real part and the imaginary part of the spectrum Y(t,f) of the target multi-channel speech as the input spectral features. In addition, since the IPD in the embodiment of the present application is a periodic function, in order to avoid the problem of feature representation ambiguity, cosIPD and sinIPD can be used as input spatial features. Furthermore, the embodiment of the present application can combine the spectral features and the spatial features to obtain the target feature map of the target multi-channel speech, wherein the target feature map can be, but is not limited to, represented as:

[0101] F=[Y real ; Yi imag ;cosIPD;sinIPD],

[0102] Among them, Y real and Y imag Represent the real and imaginary parts of the mixed signal spectrum Y(t,f), respectively. represents the model input features calculated by the feature extraction module. Cin, T, and F represent the number of feature channels, frames, and frequencies, respectively.

[0103] Optionally, in one embodiment of the present application, the target multi-channel speech is input into the initial multi-channel speech separation model, the target mask is used as the optimization target of the initial multi-channel speech separation model, and the target parameters of the initial dual-path fast converter neural network are optimized, which also includes: combining the target feature map and the spectrum modeling unit to obtain the spectral local information and spectral spatial information of the target multi-channel speech; based on the spectral local information and spectral spatial information, the time context information of the target multi-channel speech is obtained according to the sub-band modeling unit; based on the time context information, the frequency context information of the target multi-channel speech is obtained according to the full-band modeling unit; based on the frequency context information, a separable high-dimensional feature representation of the target multi-channel speech is obtained.

[0104] It will be understood by those skilled in the art that the embodiments of the present application can combine the target feature map and the spectrum modeling unit to obtain the spectral local information and spectral spatial information of the target multi-channel speech, and based on the spectral local information and spectral spatial information, obtain the time context information and frequency context information of the target multi-channel speech, and then obtain a separable high-dimensional feature representation of the target multi-channel speech.

[0105] In some embodiments, the embodiment of the present application can extract the spectral local information and spectral spatial information of the target multi-channel speech from the target feature map through the spectral modeling unit. The specific process can be: first, the target feature map F is fused and the input features are upgraded through feature coding, and then the spectral local information and spectral spatial information are extracted using SCBAM. The formula can be, but is not limited to, expressed as:

[0106]

[0107] in, represents the output features of the feature encoding module, B represents the output channel of the feature encoding module, Encoder(·) represents the feature encoding module in the spectrum modeling unit, SCBAM(·) represents the shuffled convolution attention module, Represents the output features of the spectrum modeling unit.

[0108] In some embodiments, the embodiment of the present application can respectively pass the time context information and the frequency context information through the sub-band modeling unit and the full-band modeling unit to obtain a separable high-dimensional feature representation of the target multi-channel speech. The specific process can be: in the sub-band modeling unit of the embodiment of the present application, the dimension of the input feature is first converted into F×B×T, and then the ConFastformer is used to model the last feature dimension, that is, the time dependency of all sub-bands is modeled; then the local features obtained by the cross-linear attention fusion spectrum modeling unit and the sub-band features output by the ConFastformer module are used. The formula can be, but is not limited to, expressed as:

[0109]

[0110] in, represents the input features of the sub-band modeling unit, Permute(·) represents the feature dimension conversion operation, CLA(·) represents the cross linear attention unit, F T ' dim represents the output features of the ConFastformer module, Represents the final output features of the sub-band modeling unit.

[0111] In this embodiment of the present application, the subband modeling unit can independently process each frequency subband, while all frequency subband parameters are shared. In speech separation tasks, since the speakers' speech does not completely overlap in the time-frequency domain, if the temporal context information can be effectively modeled, the non-overlapping portions of the speech segments before and after the overlapping segments can be used to obtain clearer speaker features, thereby improving the performance of the speech separation model.

[0112] Furthermore, in the full-band modeling unit of the embodiment of the present application, the feature dimension is first converted into T×B×F, and the remaining calculation steps are consistent with the sub-band modeling process. It can be understood that the embodiment of the present application can fuse all sub-band information at each time point through full-band modeling, thereby effectively extracting global frequency information. In addition, the full-band modeling unit of the embodiment of the present application can independently process the full-band information of each frame, and the feature parameters of each frame are shared. Recently, more and more studies have found that modeling the frequency dependency of speech helps to restore spectral details, thereby improving the quality of speech separation.

[0113] Furthermore, for multi-channel speech signals, the present invention's embodiments make it particularly difficult to accurately obtain the target speaker's spatial characteristics from subband features in the presence of severe interference noise or reverberation. However, the direct path response of a sound source is highly correlated in frequency, which can be leveraged to address the spatial information ambiguity that arises in situations with strong reverberation and low signal-to-noise ratios. Because the relative transfer function shapes and amplitudes of sound sources from different directions differ significantly in the frequency dimension, modeling frequency context information also facilitates extracting speaker directional information, further improving the quality of separated speech.

[0114] Optionally, in one embodiment of the present application, the target multi-channel speech is input into the initial multi-channel speech separation model, the target mask is used as the optimization target of the initial multi-channel speech separation model, and the target parameters of the initial dual-path fast converter neural network are optimized. It also includes: obtaining the complex mask of the separated speech corresponding to the target multi-channel speech based on the separable high-dimensional feature representation and mask module; and training the initial multi-channel speech separation model in combination with the complex mask and the target mask to optimize the target parameters of the initial dual-path fast converter neural network.

[0115] In the actual implementation process, the embodiment of the present application can convert the separable high-dimensional feature representation into a complex mask for separating speech through a mask module, and apply the mask to the spectrum of the target multi-channel speech, train the initial multi-channel speech separation model, and obtain the target parameters of the optimized initial multi-channel speech separation model. The process can be:

[0116] The embodiment of the present application uses the target mask as training, wherein the target mask can be understood as an improved complex mask, which can be set by those skilled in the art according to actual conditions, and is not specifically limited in this application. The implementation process is as follows: Figure 5 The output of the initial dual-path fast converter neural network is divided into two paths, one for amplitude estimation and the other for phase estimation.

[0117] In the embodiment of the present application, in phase estimation, the output features of the initial dual-path fast converter neural network are fed into a fully connected layer, and the output is mapped to between -1 and 1 through the Tanh activation function as a complex mask for preliminary estimation. The phase mask is then calculated based on the complex mask, and its formula can be, but is not limited to, expressed as:

[0118]

[0119] in, represents the DPFNN output, Tanh(·) represents the Tanh activation function, and denote the real and imaginary parts of the preliminary estimated complex mask, respectively. represents the estimated phase mask, and P represents the number of speakers in the speech mixture.

[0120] Furthermore, in the embodiment of the present application, the phase mask is element-wise multiplied by the phase spectrum of the mixed speech of the reference channel to obtain the phase spectrum of the separated speech, and the formula thereof can be, but is not limited to, expressed as:

[0121] S phase =M phase ·Y phase ,

[0122] in, and They represent the reference channel mixed speech phase spectrum and separated speech phase spectrum respectively.

[0123] In the embodiment of the present application, in amplitude estimation, the output of the initial dual-path fast converter neural network is fed into another fully connected layer with a Sigmoid activation function to obtain an amplitude mask, and the amplitude mask is element-wise multiplied with the amplitude spectrum of the reference channel mixed speech to obtain the amplitude spectrum of the separated speech. The formula can be, but is not limited to, expressed as:

[0124]

[0125] in, and They represent the reference channel mixed speech amplitude spectrum and the separated speech amplitude spectrum respectively.

[0126] Furthermore, the embodiment of the present application can combine the amplitude spectrum and the phase spectrum of the separated speech, and the final complex spectrum of the separated speech can be expressed as, but not limited to:

[0127]

[0128] in, and Represent the real and imaginary parts of the complex spectrum of the separated speech respectively.

[0129] Optionally, in one embodiment of the present application, the separated speech corresponding to the target multi-channel speech is obtained based on the multi-channel speech separation model, including: converting the initial separated speech to obtain the time domain waveform of the initial separated speech; and obtaining the separated speech based on the time domain waveform.

[0130] In some embodiments, the embodiment of the present application may transform the time domain waveform of the initial separated speech through ISTFT (Inverse Short-Time Fourier Transform), and the specific process may be:

[0131] It can be understood that the ISTFT is the inverse process of the STFT, which is used to convert the complex spectrum of the signal in the time-frequency domain into a time-domain waveform. Therefore, the embodiment of the present application can first perform an IDFT (Inverse Discrete Fourier Transform) operation on each frame of the complex spectrum, and its formula can be, but is not limited to, expressed as:

[0132]

[0133] in, Represents the time domain signal of the mth frame after IDFT.

[0134] Then the embodiment of the present application can use the comprehensive window to The purpose of this step is to reduce the adverse effects of truncation and suppress the subjective auditory discontinuity of the frame edge. The comprehensive window can include, but is not limited to, the root mean square Hanning window and the Blackman window, etc., and this application does not impose specific restrictions. Specifically, the embodiment of this application can use the comprehensive window z(n) to After weighting, the output in, It can be expressed as, but not limited to:

[0135]

[0136] Finally, the embodiment of this application can Overlap and add to get the final time domain waveform

[0137] The working principle of the multi-channel speech separation method based on neural network proposed in the embodiment of the present application is described in detail below with reference to a specific embodiment.

[0138] Figure 6 The present invention provides a flowchart of the working principle of a multi-channel speech separation method based on a neural network according to an embodiment of the present application.

[0139] Step S601: Acquire target multi-channel speech.

[0140] That is to say, the embodiment of the present application can obtain the target multi-channel speech.

[0141] Step S602: Construct a feature extraction module based on STFT and IPD.

[0142] That is to say, the embodiment of the present application can use STFT to transform the time domain multi-channel speech signal y(n) of the target multi-channel speech into the time-frequency domain, and use IPD as the input feature to construct a feature extraction module, and then use the feature extraction module to obtain the target feature map of the target multi-channel speech.

[0143] Step S603: constructing an initial dual-path fast converter neural network based on the spectrum modeling unit, the sub-band modeling unit and the full-band modeling unit.

[0144] That is to say, the embodiment of the present application can use the spectrum modeling unit, the sub-band modeling unit and the full-band modeling unit to construct an initial dual-path fast converter neural network, wherein the embodiment of the present application can extract the spectral local information and spectral spatial information of the target multi-channel speech from the target feature map through the spectrum modeling unit, and pass the time context information and frequency context information through the sub-band modeling unit and the full-band modeling unit to obtain a separable high-dimensional feature representation of the target multi-channel speech.

[0145] Step S604: construct a mask module.

[0146] That is to say, the embodiment of the present application can construct a mask module, pass the separable high-dimensional feature representation through the mask module, convert it into a complex mask for separating speech, and apply the mask to the spectrum of the target multi-channel speech, train the initial multi-channel speech separation model, and obtain the target parameters of the optimized initial multi-channel speech separation model.

[0147] Step S605: converting the initial separated speech to obtain a time domain waveform of the initial separated speech.

[0148] That is to say, the embodiment of the present application can transform the time domain waveform of the initial separated speech through ISTFT.

[0149] Step S606: Output the separated speech based on the time domain waveform.

[0150] That is, the embodiment of the present application can use a comprehensive window to weight the time domain waveform and then output it to output the final separated speech.

[0151] According to the multi-channel speech separation method based on neural network proposed in the embodiment of the present application, an initial multi-channel speech separation model can be constructed based on the feature extraction module, the initial dual-path fast converter neural network and the mask module, and the target mask is used as the training target to optimize the target parameters to obtain the multi-channel speech separation model, and then the separated speech corresponding to the target multi-channel speech is obtained based on the multi-channel speech separation model. Among them, the multi-channel speech separation model constructed in the embodiment of the present application has a more lightweight network structure, can accurately and stably achieve high-quality speech separation, and exhibits excellent generalization ability under different mixing conditions, showing good practical value and technical prospects. Therefore, it solves the problem that the speech separation technology based on microphone array in the related technology cannot meet the requirements of lightweight, high robustness and high generalization, and adapt to complex and changeable acoustic environments, and in the multi-channel speech separation algorithm based on neural network, the model has a large number of parameters and calculation amount, which is difficult to adapt to mobile devices with limited computing power or memory.

[0152] Next, a multi-channel speech separation device based on a neural network proposed in accordance with an embodiment of the present application will be described with reference to the accompanying drawings.

[0153] Figure 7 A block diagram of a multi-channel speech separation device based on a neural network provided in an embodiment of the present application.

[0154] like Figure 7 As shown, the neural network-based multi-channel speech separation device 10 is applied to the model construction stage, wherein the device 10 includes: a first construction module 100, a second construction module 200, a third construction module 300, a fourth construction module 400, a fifth construction module 500, a sixth construction module 600 and a first generation module 700.

[0155] The first construction module 100 is used to construct a feature extraction module of an initial multi-channel speech separation model based on STFT and IPD.

[0156] The second building module 200 is used to build a spectrum modeling unit of an initial dual-path fast converter neural network based on feature coding and SCBAM.

[0157] The third building module 300 is used to respectively construct a sub-band modeling unit and a full-band modeling unit of the initial dual-path fast converter neural network according to the cross linear attention and ConFastformer.

[0158] The fourth building block 400 is configured to build an initial dual-path fast transformer neural network of an initial multi-channel speech separation model based on the spectrum modeling unit, the sub-band modeling unit, and the full-band modeling unit.

[0159] The fifth construction module 500 is used to construct a mask module of the initial multi-channel speech separation model based on a fully connected layer of a Tanh activation function and a fully connected layer of a Sigmoid activation function.

[0160] The sixth building module 600 is used to build an initial multi-channel speech separation model based on the feature extraction module, the initial dual-path fast converter neural network and the mask module.

[0161] The first generation module 700 is used to input the target multi-channel speech into the initial multi-channel speech separation model, use the target mask as the optimization target of the initial multi-channel speech separation model, optimize the target parameters of the initial dual-path fast converter neural network, update the initial multi-channel speech separation model based on the optimized dual-path fast converter neural network, obtain the multi-channel speech separation model, and obtain the separated speech corresponding to the target multi-channel speech based on the multi-channel speech separation model.

[0162] Optionally, in one embodiment of the present application, the first generation module 700 includes: a first generation unit and a second generation unit.

[0163] The first generating unit is used to combine the target multi-channel speech and the feature extraction module to obtain the spectrum and frequency domain spatial features of the target multi-channel speech.

[0164] The second generating unit is used to obtain a target feature map of the target multi-channel speech based on the real part of the spectrum, the imaginary part of the spectrum and the frequency domain spatial features.

[0165] Optionally, in one embodiment of the present application, the first generation module 700 further includes: a third generation unit, a fourth generation unit, a fifth generation unit and a sixth generation unit.

[0166] Among them, the third generation unit is used to combine the target feature map and the spectrum modeling unit to obtain the spectrum local information and spectrum spatial information of the target multi-channel speech.

[0167] The fourth generating unit is configured to obtain the time context information of the target multi-channel speech according to the sub-band modeling unit based on the local spectrum information and the spatial spectrum information.

[0168] The fifth generating unit is configured to obtain frequency context information of the target multi-channel speech based on the time context information and the full-band modeling unit.

[0169] The sixth generating unit is configured to obtain a separable high-dimensional feature representation of the target multi-channel speech based on the frequency context information.

[0170] Optionally, in one embodiment of the present application, the first generating module 700 further includes: an acquiring unit and an optimizing unit.

[0171] The acquisition unit is used to acquire the complex mask of the separated speech corresponding to the target multi-channel speech according to the separable high-dimensional feature representation and the mask module.

[0172] An optimization unit is used to train an initial multi-channel speech separation model by combining the complex mask and the target mask to optimize target parameters of the initial dual-path fast transformer neural network.

[0173] Optionally, in one embodiment of the present application, the first generation module 700 includes: a conversion unit and a seventh generation unit.

[0174] The conversion unit is used to convert the initial separated speech to obtain the time domain waveform of the initial separated speech.

[0175] The seventh generating unit is configured to obtain separated speech based on the time domain waveform.

[0176] It should be noted that the above explanation of the embodiment of the multi-channel speech separation method based on a neural network is also applicable to the multi-channel speech separation device based on a neural network in this embodiment, and will not be repeated here.

[0177] According to the multi-channel speech separation device based on a neural network proposed in the embodiment of the present application, an initial multi-channel speech separation model can be constructed based on a feature extraction module, an initial dual-path fast converter neural network and a mask module, and the target mask is used as a training target to optimize the target parameters to obtain a multi-channel speech separation model, and then the separated speech corresponding to the target multi-channel speech is obtained based on the multi-channel speech separation model. Among them, the multi-channel speech separation model constructed in the embodiment of the present application has a more lightweight network structure, can accurately and stably achieve high-quality speech separation, and exhibits excellent generalization ability under different mixing conditions, showing good practical value and technical prospects. Therefore, it solves the problem that the speech separation technology based on microphone arrays in the related art cannot meet the requirements of lightweight, high robustness and high generalization, and cannot adapt to complex and changeable acoustic environments. In the multi-channel speech separation algorithm based on neural networks, the model has a large number of parameters and a large amount of calculation, which makes it difficult to adapt to mobile devices with limited computing power or memory.

[0178] The above embodiment describes the model construction phase. The following describes an embodiment of the use phase.

[0179] Figure 8This is a flowchart of another neural network-based multi-channel speech separation method provided according to an embodiment of the present application.

[0180] like Figure 8 As shown, the multi-channel speech separation method based on neural network is applied to the use stage of the model, wherein the method includes the following steps:

[0181] In step S801, multi-channel speech to be separated is obtained.

[0182] As a possible implementation method, the embodiment of the present application can obtain multi-channel speech to be separated.

[0183] For example, in the embodiment of the present application, it is assumed that there are P speakers and M microphones in the recording environment. Then, the target multi-channel speech signal received by the microphone array can be expressed as:

[0184]

[0185] Among them, s p (n) represents the speech signal of the pth speaker, * represents the convolution operator symbol, h p (n) represents the RIR from the pth speaker to the microphone array, and v(n) represents the additive noise received by the microphone array.

[0186] In step S802, the multi-channel speech to be separated is input into a pre-built multi-channel speech separation model to obtain separated speech corresponding to the multi-channel speech to be separated, wherein the pre-built multi-channel speech separation model is obtained from the multi-channel speech to be separated.

[0187] In the actual implementation process, the embodiment of the present application can input the multi-channel speech to be separated into a pre-built multi-channel speech separation model to obtain the separated speech corresponding to the multi-channel speech to be separated. The specific process can be:

[0188] First, the embodiment of the present application can obtain a target feature map of the target multi-channel speech based on the feature extraction module; then use the dual-path fast converter neural network to obtain a separable high-dimensional feature representation of the target multi-channel speech; then use the mask module to obtain the complex mask of the separated speech corresponding to the target multi-channel speech, and then obtain the final separated speech.

[0189] According to the multi-channel speech separation method based on neural network proposed in the embodiment of the present application, the multi-channel speech to be separated can be input into a pre-built multi-channel speech separation model to obtain the separated speech corresponding to the multi-channel speech to be separated. Among them, the embodiment of the present application can learn the local spectrum information and local spatial information of the multi-channel speech to be separated based on the spectrum modeling module, and use the sub-band modeling unit and the full-band modeling unit to extract the time context features and frequency context features, which can accurately and stably achieve high-quality speech separation, show excellent generalization ability, and show good practical value and technical prospects. Therefore, it solves the problem in the related art that the speech separation technology based on microphone array cannot meet the requirements of lightweight, high robustness and high generalization, and adapt to complex and changeable acoustic environments, and in the multi-channel speech separation algorithm based on neural network, the model has a large number of parameters and a large amount of calculation, which is difficult to adapt to mobile devices with limited computing power or memory.

[0190] Next, a multi-channel speech separation device based on a neural network proposed in accordance with an embodiment of the present application will be described with reference to the accompanying drawings.

[0191] Figure 9 A block diagram of another neural network-based multi-channel speech separation device provided according to an embodiment of the present application.

[0192] like Figure 9 As shown, the neural network-based multi-channel speech separation device 20 is applied to the model usage stage, wherein the device 20 includes: an acquisition module 100 and a second generation module 200.

[0193] The acquisition module 100 is used to acquire multi-channel speech to be separated.

[0194] The second generation module 200 is used to input the multi-channel speech to be separated into a pre-built multi-channel speech separation model to obtain the separated speech corresponding to the multi-channel speech to be separated, wherein the pre-built multi-channel speech separation model is obtained from the multi-channel speech to be separated.

[0195] It should be noted that the above explanation of the embodiment of the multi-channel speech separation method based on a neural network is also applicable to the multi-channel speech separation device based on a neural network in this embodiment, and will not be repeated here.

[0196] According to the multi-channel speech separation device based on a neural network proposed in the embodiment of the present application, the multi-channel speech to be separated can be input into a pre-built multi-channel speech separation model to obtain the separated speech corresponding to the multi-channel speech to be separated. Among them, the embodiment of the present application can learn the local spectrum information and local spatial information of the multi-channel speech to be separated based on the spectrum modeling module, and use the sub-band modeling unit and the full-band modeling unit to extract the time context features and frequency context features, which can accurately and stably achieve high-quality speech separation, show excellent generalization ability, and show good practical value and technical prospects. Therefore, it solves the problem in the related art that the speech separation technology based on microphone array cannot meet the requirements of lightweight, high robustness and high generalization, and cannot adapt to complex and changeable acoustic environments. In the multi-channel speech separation algorithm based on neural network, the model has a large number of parameters and a large amount of calculation, which makes it difficult to adapt to mobile devices with limited computing power or memory.

[0197] Figure 10 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. The electronic device may include:

[0198] A memory 1001 , a processor 1002 , and a computer program stored in the memory 1001 and executable on the processor 1002 .

[0199] When the processor 1002 executes the program, the multi-channel speech separation method based on neural network provided in the above embodiment is implemented.

[0200] Furthermore, the electronic device further includes:

[0201] The communication interface 1003 is used for communication between the memory 1001 and the processor 1002 .

[0202] The memory 1001 is used to store computer programs that can be run on the processor 1002 .

[0203] The memory 1001 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0204] If the memory 1001, the processor 1002, and the communication interface 1003 are implemented independently, the communication interface 1003, the memory 1001, and the processor 1002 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0205] Optionally, in a specific implementation, if the memory 1001, the processor 1002 and the communication interface 1003 are integrated on a chip, the memory 1001, the processor 1002 and the communication interface 1003 can communicate with each other through an internal interface.

[0206] The processor 1002 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0207] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned neural network-based multi-channel speech separation method.

[0208] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, implements the above-mentioned neural network-based multi-channel speech separation method.

[0209] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0210] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0211] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing a custom logical function or process step, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed in a different order than shown or discussed, including performing functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0212] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or N wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a computer memory.

[0213] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented using hardware, as in another embodiment, it can be implemented using any one or a combination of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0214] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0215] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0216] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A multi-channel speech separation method based on neural network, characterized in that: Applied to the model construction phase, wherein the method includes the following steps: Based on short-time Fourier transform (STFT) and inter-channel phase difference (IPD), a feature extraction module is constructed for the initial multi-channel speech separation model. Based on feature encoding and shuffled convolutional attention (SCBAM), a spectral modeling unit of the initial dual-path fast converter neural network is constructed. Constructing a sub-band modeling unit and a full-band modeling unit of the initial dual-path fast converter neural network according to cross linear attention and convolution enhanced fast converter ConFastformer respectively; constructing an initial dual-path fast transformer neural network of the initial multi-channel speech separation model based on the spectrum modeling unit, the sub-band modeling unit and the full-band modeling unit; Based on the fully connected layer of the Tanh activation function and the fully connected layer of the Sigmoid activation function, a mask module of the initial multi-channel speech separation model is constructed; Constructing the initial multi-channel speech separation model based on the feature extraction module, the initial dual-path fast transformer neural network and the mask module; The target multi-channel speech is input into the initial multi-channel speech separation model, the target mask is used as the optimization target of the initial multi-channel speech separation model, the target parameters of the initial dual-path fast converter neural network are optimized, and the initial multi-channel speech separation model is updated based on the optimized dual-path fast converter neural network to obtain a multi-channel speech separation model, and the separated speech corresponding to the target multi-channel speech is obtained based on the multi-channel speech separation model.

2. The multi-channel speech separation method based on a neural network according to claim 1, wherein Inputting the target multi-channel speech into the initial multi-channel speech separation model, taking the target mask as the optimization target of the initial multi-channel speech separation model, and optimizing the target parameters of the initial dual-path fast converter neural network, comprises: Combining the target multi-channel speech with the feature extraction module, obtaining the spectrum and frequency domain spatial features of the target multi-channel speech; A target feature map of the target multi-channel speech is obtained based on the real part of the spectrum, the imaginary part of the spectrum and the frequency domain spatial features.

3. The multi-channel speech separation method based on neural network according to claim 2, wherein The step of inputting the target multi-channel speech into the initial multi-channel speech separation model, taking the target mask as the optimization target of the initial multi-channel speech separation model, and optimizing the target parameters of the initial dual-path fast converter neural network further includes: Combining the target feature map and the spectrum modeling unit, obtaining spectrum local information and spectrum spatial information of the target multi-channel speech; Based on the local spectrum information and the spatial spectrum information, obtaining the temporal context information of the target multi-channel speech according to the subband modeling unit; Based on the time context information, obtaining frequency context information of the target multi-channel speech according to the full-band modeling unit; Based on the frequency context information, a separable high-dimensional feature representation of the target multi-channel speech is obtained.

4. The multi-channel speech separation method based on neural network according to claim 3, characterized in that The step of inputting the target multi-channel speech into the initial multi-channel speech separation model, taking the target mask as the optimization target of the initial multi-channel speech separation model, and optimizing the target parameters of the initial dual-path fast converter neural network further includes: Obtaining a complex mask of the separated speech corresponding to the target multi-channel speech according to the separable high-dimensional feature representation and the mask module; The initial multi-channel speech separation model is trained in combination with the complex mask and the target mask to optimize target parameters of the initial dual-path fast transformer neural network.

5. The multi-channel speech separation method based on neural network according to claim 1, characterized in that The obtaining, based on the multi-channel speech separation model, separated speech corresponding to the target multi-channel speech includes: Converting the initial separated speech to obtain a time domain waveform of the initial separated speech; The separated speech is obtained based on the time domain waveform.

6. A multi-channel speech separation method based on neural network, characterized in that: The multi-channel speech separation method based on a neural network according to any one of claims 1 to 5 is applied to the use phase of the model, wherein the method comprises the following steps: Obtain multi-channel speech to be separated; The multi-channel speech to be separated is input into a pre-built multi-channel speech separation model to obtain separated speech corresponding to the multi-channel speech to be separated, wherein the pre-built multi-channel speech separation model is obtained from the multi-channel speech to be separated.

7. A multi-channel speech separation device based on a neural network, characterized in that: Applied to the model construction stage, wherein the device includes: The first building module is used to build a feature extraction module of an initial multi-channel speech separation model based on STFT and IPD; The second building block is used to construct a spectrum modeling unit of an initial dual-path fast converter neural network based on feature coding and SCBAM; A third building block is configured to construct a sub-band modeling unit and a full-band modeling unit of the initial dual-path fast converter neural network according to cross linear attention and ConFastformer, respectively; a fourth building block, configured to build an initial dual-path fast transformer neural network of the initial multi-channel speech separation model based on the spectrum modeling unit, the sub-band modeling unit, and the full-band modeling unit; A fifth construction module is used to construct a mask module of the initial multi-channel speech separation model based on a fully connected layer of a Tanh activation function and a fully connected layer of a Sigmoid activation function; a sixth building module, configured to build the initial multi-channel speech separation model based on the feature extraction module, the initial dual-path fast transformer neural network, and the mask module; A generation module is used to input the target multi-channel speech into the initial multi-channel speech separation model, use the target mask as the optimization target of the initial multi-channel speech separation model, optimize the target parameters of the initial dual-path fast converter neural network, update the initial multi-channel speech separation model based on the optimized dual-path fast converter neural network, obtain a multi-channel speech separation model, and obtain the separated speech corresponding to the target multi-channel speech based on the multi-channel speech separation model.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-channel speech separation method based on a neural network as described in any one of claims 1 to 5 or 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the neural network-based multi-channel speech separation method according to any one of claims 1 to 5 or 6.

10. A computer program product, characterized in that The invention comprises a computer program, which, when executed, is used to implement the multi-channel speech separation method based on a neural network as described in any one of claims 1 to 5 or 6.

Images