End-to-end noisy speech separation method based on channel attention mechanism and Transform

Abstract

The invention discloses an end-to-end noisy speech separation method based on a channel attention mechanism and Transform, and the method comprises the steps: constructing a time sequence perception context channel attention layer, and carrying out the effective filtering and screening of a feature channel dimension; meanwhile, due to the existence of time sequence perception and context perception characteristics, the rationality of channel weight distribution under noise is further improved; and secondly, in order to enhance the global expression and modeling capability of the voice potential features output by the encoder, a bidirectional LSTM-Transform encoder layer is provided, a feed-forward layer with bidirectional LSTM further enriches global context information in the multi-head attention features, and the feature encoding effectiveness of the encoder is improved. According to the method, the performance of the voice separation system is improved under complex noise, various separation test indexes are all improved, and in addition, the model complexity is reduced and the effectiveness is improved due to reasonable feature screening, so that the method is suitable for most application scenes related to human-computer interaction.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech separation, and in particular relates to an end-to-end noisy speech separation method based on a channel attention mechanism and a Transformer. Background Art

[0002] The speech separation task originates from the classic "cocktail party problem", that is, it becomes difficult to distinguish and understand what each speaker is saying when there are overlapping speakers and complex background noise. Many scenarios in life (such as conference speech transcription, multimedia processing, etc.) often involve the application of speech separation technology, and most of these scenarios contain a certain amount of background noise. The development of speech technology is not only to improve the quality of the speaker's speech extracted by the speech separation system, but also to significantly and effectively reduce the interference of background noise and improve the robustness of the speech separation system to background noise. This is of great significance in many real-life application scenarios.

[0003] The goal of speech separation is not only to separate the speech of a single speaker from a noisy mixed signal, but also to identify and segment the speech fragments of a single speaker so that they can be used as input for other important downstream tasks. In this process, the speech separation system as an upstream task can be used as a preprocessing module of intelligent devices, and finally make corresponding downstream tasks (such as automatic speech recognition, speech sentiment analysis, etc.) adaptation according to the needs of the user's specific application scenario. Deep learning technology plays an increasingly important role in this process. The application of deep learning technology covers image, natural speech processing, signal processing and other related fields. It is a more profound learning and reference to the way the human brain thinks, and in some fields, the performance of deep learning technology can even be comparable to human intelligence. In particular, with the rapid development and progress of artificial intelligence technology and high-performance hardware devices (such as CPU and GPU), some speech separation technologies based on deep learning are no longer limited by the inherent characteristics of software and hardware, and many speech separation algorithms have been deployed in practical applications. Using deep learning technology, the speech patterns of speakers in noisy mixed speech signals (including between speakers and background noise, and between different speakers) can be well identified, which significantly improves and enhances people and intelligent machines' understanding of speakers' speech.

[0004] Nowadays, many speech separation technologies based on deep learning mainly involve methods of joint analysis from the time domain or time-frequency domain. Although the latter can make full use of the complementarity of the time-frequency domain information of the speech signal, many algorithms often involve high computational complexity and parameter quantity, which has certain constraints for resource-constrained scenarios. In addition, the short-time Fourier transform technology used to transform the time-domain speech signal into the time-frequency domain has the defect that different frequency components in the speech signal have the same receptive field (window size), which to a certain extent also limits the performance improvement of the speech separation system based on the time-frequency domain. The current mainstream speech separation model can be regarded as an encoder + separation module + decoder structure (this structure is derived from the Conv-TasNet model). The encoder is used to obtain the potential feature representation of the input noisy speech signal, and then the separator is responsible for further learning and capturing the speaker's speech pattern from the feature space and reducing the interference of background noise to obtain the mask of the estimated signal. Finally, the decoder maps the speaker's speech obtained by the separation module from the potential feature space to the real speech signal space.

[0005] The end-to-end structure based on Conv-TasNet has not only been widely used in the field of speech separation, but also has a profound impact on other speech tasks (such as speech enhancement) and model architectures in other fields. In the Conv-TasNet network, the separation module captures and represents the dynamic time changes of information in the speech time domain waveform by stacking multiple one-dimensional convolutional layers, and then models the speaker's speech pattern. However, the multi-layer stacked convolution operation makes the feature mask information learned by the separation module redundant, which reduces the effectiveness of the speech separation system to a certain extent. For this reason, it is necessary to reasonably filter and screen the features output by the separation module. In addition, the Conv-TasNet encoder and separation module lack full use of the global context dependency of speech, because background noise based on context information may have a potential role in promoting the performance of the noisy speech separation system in some cases. Both the local and global characteristics of speech are very important for the speech separation system to separate the speech of different speakers and reduce the interference of background noise. Summary of the invention

[0006] The purpose of the present invention is to address the defects and shortcomings of the above-mentioned prior art, and propose an end-to-end noisy speech separation method based on channel attention mechanism and Transformer, and propose to perform channel attention from the channel dimension of the feature, acting on the encoder and separator respectively. In addition, the method of the present invention introduces additional timing perception and context information before the channel attention completes the decision. Time domain perception helps to enhance the speech timing characteristics of the feature, and the presence of context information effectively improves the rationality of channel weight allocation in the channel attention mechanism, and realizes effective filtering and screening of speech features. Secondly, the present invention proposes a bidirectional LSTM-Transformer encoding layer to enhance the global expression and modeling capabilities of the encoder output features, and further improves the effectiveness of the encoder in feature encoding.

[0007] The technical solution adopted by the present invention to solve the technical problem is: an end-to-end noisy speech separation method based on attention mechanism and Transformer, the method comprising the following steps:

[0008] Step 1: Input noisy mixed signal S mix is composed of the clean speech signals S from the two speakers. 1 , S 2 The input mixed signal is first preprocessed by a convolution 0 layer and four cascaded and parameter-shared convolution 1 layers. Then, the temporal-aware context channel attention layer and the bidirectional LSTM-Transformer encoder layer are used to perform feature screening and assign global context information to the preprocessing results, respectively, to obtain a potential feature representation F that is effective for speech signals and has a certain context modeling capability. Encoder ;

[0009] Step 2: Three cascades of separator basic modules consisting of multiple cascaded 1D convolutional layers and temporal-aware contextual channel attention layers gradually process the features F. Encoder , and generate a speech separation mask Mask, and finally represent the potential feature F of the mixed signal output by the mask Mask and the encoder Encoder Perform element-wise multiplication to separate the latent spatial features F corresponding to each speaker's speech and background noise in the mixed signal Separator ;

[0010] Step 3: Feature representation F of the separator output Separator Finally, the decoder, which consists of four cascaded convolutional 1 layers and one convolutional 0 layer corresponding to the encoder front end, completes the mapping from the potential feature space to the real time domain speech signal space and background noise signal space. The decoder finally outputs the estimated three time domain signals, corresponding to the speech signals of the two speakers respectively. and background noise signal

[0011] Step 4: Constrain and optimize the performance of the speech separation system by the overall loss function of speech separation;

[0012] Step 5: Perform a performance evaluation on the proposed end-to-end noisy speech separation method based on channel attention mechanism and Transformer.

[0013] Furthermore, the specific steps of step 1 are:

[0014] Step 1-1: The time-domain noisy mixed speech signal passes through the convolution layer 0, and undergoes a one-dimensional convolution with a convolution kernel size of L / 2=16 and a stride of S=L / 2 to obtain approximately stable and overlapping speech segments s mix-k , the number of output channels is C 0 =512, and use PReLU to obtain the nonlinear expression ability of features;

[0015] Step 1-2: Segment the speech signal s mix-k Furthermore, four cascaded and parameter-shared convolution layers are used to reduce model training parameters while further improving the expressiveness of speech features.

[0016] Step 1-3: The speech nonlinear feature map output after one convolution 0 layer and four convolution 1 layers is passed through the time-series-aware context channel attention layer to screen features in its output channel dimension, filter out redundant features, and improve the effectiveness of features that are more useful for separation;

[0017] Step 1-4: The output result after feature screening based on the attention mechanism in the channel dimension enhances the global modeling ability of the speech separation system for the speech signal. Through the bidirectional LSTM-Transformer encoder layer, an effective latent feature representation F of the speech signal with certain global context information is finally obtained. Encoder .

[0018] Furthermore, in step 1-2, the specific method for implementing four cascaded and parameter-shared convolution layers 1 includes the following steps:

[0019] Step 1-2-1: A single convolution layer consisting of a 1D convolution with an expanded convolution function and a nonlinear activation function PReLU is used to improve the nonlinear expression ability of speech feature mapping. The number of 1D convolution input channels, the number of output channels, the convolution kernel size, the convolution kernel stride, the expanded convolution expansion rate, and the convolution padding number are 512, 512, 3, 1, 1, and 1 respectively;

[0020] Step 1-2-2: Cascade four convolutional layers and share parameters among the four layers.

[0021] Furthermore, in the steps 1-3, the specific method for implementing the time-series-aware context channel attention layer includes the following steps:

[0022] Step 1-3-1: adding a timing-aware feature to the output of step 1-2;

[0023] Step 1-3-2: Introduce contextual relationships into the output of step 1-3-1 to provide reasonable channel weight allocation for the subsequent channel attention module;

[0024] Step 1-3-3: Perform feature weighting and screening on the channel dimension for speech features with temporal perception context.

[0025] Furthermore, in step 1-3-1, the specific method for realizing the speech feature representation with time sequence perception characteristics comprises the following steps:

[0026] Step 1-3-1-1: Average pool the last dimension of the input feature to 1, retaining only the time dimension, and then fully learn the dynamic changes of the input signal in the time domain. This operation can be defined as:

[0027]

[0028] Where N is the number of dimensions of the last dimension of the input feature, x t and t Respectively represent the features at the tth time point, f gap stands for global average pooling;

[0029] Step 1-3-1-2: Perform channel transformation through 1×1 convolution on the result that only retains the time dimension;

[0030] Step 1-3-1-3: ReLU activation to provide nonlinear expression capabilities;

[0031] Step 1-3-1-4: 1×1 convolution again learns the time domain characteristics of speech, and calculates the weights through the Sigmoid function as the temporal perception of the input features;

[0032] Step 1-3-1-5: Perform element-wise multiplication of the learned time-series-aware weights of the input features with the input features to add global time-series-aware features to the time domain signal.

[0033] Furthermore, in step 1-3-2, the specific method for realizing the speech feature with time sequence perception context comprises the following steps:

[0034] Step 1-3-2-1: Increase the number of channels of the two-dimensional output of step 1-3-1-5 through 1×1 convolution;

[0035] Step 1-3-2-2: Merge the last two dimensions output from the previous step;

[0036] Step 1-3-2-3: 1×1 convolution compresses the channel dimension of the two-dimensional output result of step 1-3-2-2 and calculates the context feature;

[0037] Step 1-3-2-4: Apply the Softmax function to calculate the mask of the context feature;

[0038] Step 1-3-2-5: Perform element-wise multiplication of the context feature and the context mask to obtain speech features with temporal-aware context.

[0039] Furthermore, in step 1-3-3, the specific method for implementing feature weighting and screening of speech features with temporal perception context in the channel dimension includes the following steps:

[0040] Step 1-3-3-1: 1×1 convolution transforms the number of channels of the output result of step 1-3-2-5;

[0041] Step 1-3-3-2: Layer normalization enhances speech features and the stability of model training;

[0042] Step 1-3-3-3: Apply 1×1 convolution again to transform the number of channels;

[0043] Step 1-3-3-4: Sigmoid function estimates the attention score in the channel dimension (i.e., the weight assigned to each channel) for the output result of step 1-3-1-5, wherein the acquisition of the attention score involves the global context information of the speech, which makes the allocation of channel weights more reasonable and effective;

[0044] Step 1-3-3-5: perform element-wise multiplication of the output of step 1-3-1-5 and its corresponding channel attention score in the channel dimension, thereby completing the effective screening of time-series-aware contextual speech features in the channel dimension.

[0045] Furthermore, in steps 1-4, the specific method for implementing the bidirectional LSTM-Transformer encoder layer includes the following steps:

[0046] Step 1-4-1: Enhance the long-distance dependency of speech sequence in the output result of step 1-3-3-5 through the multi-head attention mechanism of the Transformer encoder layer;

[0047] Step 1-4-2: Add the temporal perception context speech features outputted from step 1-3-3-5 to the result of step 1-4-1 and apply layer normalization, which helps to ensure the stability of model training and prevent gradient explosion and gradient disappearance while smoothing the output;

[0048] Step 1-4-3: Use a feed-forward layer with a globally aware bidirectional LSTM to increase the global modeling and expression capabilities of speech features;

[0049] Step 1-4-4: Add the outputs of step 1-4-2 and step 1-4-3 and apply layer normalization, thereby obtaining the speech potential feature representation F of the encoder's final output Encoder , for further learning by the separator.

[0050] Furthermore, in step 1-4-1, the specific method for implementing the multi-head attention mechanism includes the following steps:

[0051] Step 1-4-1-1: Define the number of heads h in the multi-head attention mechanism. Different heads are responsible for focusing on different features and information angles of the input features, and capturing the global dependencies in the speech timing features, improving the model's understanding and modeling capabilities of the contextual relationships of speech signals;

[0052] Step 1-4-1-2: Each head calculates K (Key), Q (Query) and V (Value) about the input features. Different heads have different weight matrices for calculating K, Q and V, and their parameters are learnable.

[0053] Step 1-4-1-3: Softmax calculates the attention score of each head, and the attention score is multiplied by V to obtain the feature representation of the input feature with respect to attention;

[0054] Step 1-4-1-4: Concatenate the output results of multiple heads and multiply them with a parameter-learnable weight matrix to finally form a multi-head attention feature representation of the input features.

[0055] Furthermore, in step 1-4-3, the specific method for implementing the feedforward layer with global-aware bidirectional LSTM includes the following steps:

[0056] Step 1-4-3-1: Copy the output of step 1-4-2, and reverse one of the copies according to the hidden time dimension, i.e., in reverse time order;

[0057] Step 1-4-3-2: The speech features in the positive and reverse order of the hidden time dimension are respectively fed into the LSTM network, and the two outputs are concatenated to further enhance the global context expression and modeling capabilities of the multi-channel speech attention features output in step 1-4-1;

[0058] Step 1-4-3-3: ReLU activation and a linear layer further improve the nonlinear expression ability of features.

[0059] Furthermore, in step 2, the specific steps of implementing the separator include:

[0060] Step 2-1: Apply layer normalization to the output of step 1 to improve the stability of model training, and then transform the number of channels by using 1×1 convolution;

[0061] Step 2-2: construct a basic module of the separator by cascading 1D convolutional layers and the temporal-aware contextual channel attention layer constructed in step 1-3;

[0062] Step 2-3: cascading the separator basic modules obtained in step 2-2 three times;

[0063] Step 2-4: Perform feature aggregation and nonlinear mapping on the relevant outputs in the three cascaded separator basic modules to obtain the speech separation mask;

[0064] Step 2-5: Combine the speech separation mask Mask with the encoder output feature F Encoder Multiply them together to get the potential feature estimate F corresponding to the two speaker's speech signals and the background noise signal Separator .

[0065] Furthermore, in step 2-2, the specific method for implementing the separator basic module includes the following steps:

[0066] Step 2-2-1: Construct a 1D convolutional layer;

[0067] Step 2-2-2: The 1D convolutional layer constructed in step 2-2-1 is expanded by a factor of 2. m (where m = 0, 1, 2, 3, 4, 5, 6, 7) are cascaded in sequence, that is, the 1D convolutional layer is stacked eight times;

[0068] Step 2-2-3: Then send the output of step 2-2-2 to the time-series-aware contextual channel attention layer constructed in step 1-3, perform reasonable feature screening in the channel dimension and give the features certain contextual information, thereby completing the construction of the basic module of the separator.

[0069] Furthermore, in step 2-2-1, the specific method for implementing the 1D convolutional layer includes the following steps:

[0070] Step 2-2-1-1: 1×1 convolution performs channel transformation on the input to adapt to the calculation of subsequent layers;

[0071] Step 2-2-1-2: PReLU activation and layer normalization are used to enhance the nonlinearity of features and the stability of model training respectively;

[0072] Step 2-2-1-3: Send the output of step 2-2-1-2 to deep convolution to reduce the amount of calculation and improve the efficiency of convolution kernel parameters;

[0073] Step 2-2-1-4: PReLU activation and layer normalization;

[0074] Step 2-2-1-5: Apply two parallel 1×1 convolutions to perform transformations of different numbers of channels on the output of step 2-2-1-4, where one of the convolution outputs is element-wise added to the output of step 2-2-1-1, and finally two outputs of the 1D convolution layer are obtained.

[0075] Furthermore, in the step 2-4, the specific method of performing feature aggregation and nonlinear mapping on the relevant outputs in the three cascaded separator basic modules to obtain the final speech separation mask includes the following steps:

[0076] Step 2-4-1: Add the outputs of the 1D convolutional layer and the temporal-aware context channel attention layer in the three cascaded separator basic modules to obtain the feature aggregation representation F of the entire separator cascade module sum ;

[0077] Step 2-4-2: PReLU on the aggregated feature F sum Perform nonlinear mapping;

[0078] Step 2-4-3: 1×1 convolution adjusts the number of channels to accommodate the subsequent activation layer;

[0079] Step 2-4-4: Sigmoid calculates the mixed features F output by the encoder respectively Encoder The masks corresponding to the speech of the two speakers and the background noise.

[0080] Furthermore, the specific steps of step 3 include:

[0081] Step 3-1: Separate the feature F output from step 2 Separator After four cascaded transposed convolutional layers, the convolution kernel size, stride, and dilation factor correspond to the four cascaded convolutional layers at the front end of the encoder, and the parameters are not shared, gradually learning the feature representation of the time domain waveform of the real speech signal and background noise signal;

[0082] Step 3-2: Finally, after passing through the transposed convolution layer corresponding to the front-end convolution layer 0 of the encoder, the separated two speaker speech time domain signals and the background noise time domain signal are obtained.

[0083] Furthermore, the specific steps of step 4 include:

[0084] Step 4-1: Define the overall loss function for speech separation.

[0085] Loss total =Loss SI-SNR +αLoss MR ,

[0086] Among them, Loss SI-SNR =-SI-SNR represents the scale-invariant signal-to-noise ratio loss function (SI-SNR), with the input being the separation signal and the corresponding target signal, which is used to reduce the gap between the speech estimation and background noise estimation of the two speakers and the real speaker speech and background noise. MR Corresponding to the multi-resolution (MR) loss, it aims to improve the quality of separated speech and more effectively remove the noise in the speaker's speech based on the consideration of both the time domain and frequency domain information of the speech. α is a manually adjustable hyperparameter used to adjust the contribution and influence of the multi-resolution loss on the performance of the speech separation system.

[0087] Step 4-2: Apply a permutation-invariant training strategy to reduce the dependence of the model separation performance on the speaker order, enhance the generalization ability of the model, and gradually reduce the overall loss of the speech separation algorithm through gradient descent.

[0088] Furthermore, in step 5, a specific method for evaluating the performance of the proposed end-to-end noisy speech separation method based on the channel attention mechanism and Transformer includes the following steps:

[0089] Step 5-1: Compare an end-to-end noisy speech separation method based on channel attention mechanism and Transformer with the existing time-domain speech separation model to verify its overall performance;

[0090] Step 5-2: Compare an end-to-end noisy speech separation method based on channel attention mechanism and Transformer with an end-to-end noisy speech separation method based on Transformer to verify the effectiveness of the time-aware contextual channel attention layer;

[0091] Step 5-3: Compare an end-to-end noisy speech separation method based on a channel attention mechanism and Transformer with an end-to-end noisy speech separation method based on a channel attention mechanism to verify the effectiveness of the bidirectional LSTM-Transformer encoder layer.

[0092] Beneficial effects:

[0093] 1. In order to screen out features that are useful for separation, the present invention proposes to perform channel attention from the channel dimension of the features, acting on the encoder and separator respectively. In addition, the method of the present invention introduces additional timing perception and contextual information before the channel attention completes the decision. Time domain perception helps to enhance the speech timing characteristics of the features. The presence of contextual information effectively improves the rationality of channel weight allocation in the channel attention mechanism, thereby achieving effective filtering and screening of speech features.

[0094] 2. In order to improve the global expression and modeling capabilities of the speech potential features obtained by the encoder in the speech separation system, the present invention constructs a bidirectional LSTM Transformer encoder layer based on the Transformer encoder layer and the feedforward layer with a bidirectional LSTM, which improves the effectiveness of the encoder feature encoding while enhancing the system's global modeling capabilities for speech timing, thereby further enhancing the system's performance in performing speech separation under noisy conditions.

[0095] 3. The present invention also estimates the noise signal, and the estimated noise signal indirectly reflects the performance of the speech separation system, especially the interference of removing noise from the speaker's speech. In addition, the estimated noise helps to strengthen the model to learn the difference and connection between the speaker's speech and noise from the mixed feature representation, and improve the robustness of the model to noise, because in some cases noise also has a certain promoting effect on the effect of speech separation. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Figure 1 It is a schematic diagram of the overall speech separation process of the present invention.

[0097] Figure 2 This is a framework diagram of the time-aware contextual channel attention layer of the present invention.

[0098] Figure 3 This is a diagram of the Transformer encoder layer framework on which the present invention is based.

[0099] Figure 4 This is a framework diagram of the feedforward layer with bidirectional LSTM proposed in the present invention. DETAILED DESCRIPTION

[0100] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings of the specification. It should be noted that, in order to avoid blurring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solution of the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0101] In addition, it should be noted that the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus.

[0102] like Figures 1 to 4 As shown, the present invention proposes an end-to-end noisy speech separation method based on channel attention mechanism and Transformer, in which the time-series-aware contextual channel attention layer further enhances the effectiveness of the output features in the encoder and separator by performing reasonable feature screening and filtering in the channel dimension. In addition, the constructed bidirectional LSTM-Transformer further improves the global expression and modeling capabilities of the encoder output features, and also enriches the contextual information in the potential features of the speech. The above two designs effectively improve the utilization rate of useful features and global contextual information in the speech separation system, and improve the performance of the speech separation system under noise, and specifically include the following steps:

[0103] Step 1: Input noisy mixed signal S mix is the clean speech signal S of the two speakers respectively. 1 , S 2 The input noisy mixed signal is first preprocessed by a convolution 0 layer and four cascaded and parameter-shared convolution 1 layers. Then, the temporal-aware context channel attention layer and the bidirectional LSTM-Transformer encoder layer are used to perform feature screening and assign global context information to the preprocessing results, so as to obtain a potential feature representation F that is effective for speech signals and has certain context modeling capabilities. Encoder ;

[0104] Step 2: Three cascades of separator basic modules consisting of multiple cascaded 1D convolutional layers and temporal-aware contextual channel attention layers gradually process the features F. Encoder , generate the speech separation mask Mask, and finally represent the potential feature F of the mixed signal output by the mask Mask and the encoder Encoder Perform element-wise multiplication to separate the latent spatial features F corresponding to each speaker's speech and background noise in the mixed signal Separator ;

[0105] Step 3: Feature representation F of the separator output SeparatorFinally, the decoder, which consists of four cascaded convolutional 1 layers and one convolutional 0 layer corresponding to the encoder front end, completes the mapping from the potential feature space to the real time domain speech signal space and background noise signal space. The decoder finally outputs the estimated three time domain signals, corresponding to the speech signals of the two speakers respectively. and background noise signal

[0106] Step 4: Constrain and optimize the performance of the speech separation system by the overall loss function of speech separation;

[0107] Step 5: Perform a performance evaluation on the proposed end-to-end noisy speech separation method based on channel attention mechanism and Transformer.

[0108] The present invention uses two noisy speech datasets, namely the WHAM! dataset and the Libri2Mix dataset. For the WHAM! dataset: First, the speech speech of two speakers is randomly selected from the WSJ0 corpus, and then the speech of the two speakers is mixed with a signal-to-noise ratio randomly selected from -5dB to 5dB, thereby constructing a clean speech training set consisting of 49 male speakers and 51 female speakers, each mixed speech containing two speakers, totaling 30 hours. In addition, a 10-hour validation set and a 5-hour test set come from the speeches of another 16 speakers in WSJ0 (different from the training set). Finally, after 8kHz downsampling, a training set containing 20,000 voices, a validation set of 5,000 voices, and a test set of 3,000 voice data are obtained, with an accuracy of 16 bits. The audio noise data in WHAM! comes from a large amount of sampled data from different urban environments in the San Francisco Bay Area at the end of 2018. These collection areas mainly include restaurants, coffee shops, bars, parks and other occasions. For the Libri2Mix dataset: it is a noisy mixed speech dataset generated by mixing the LibriSpeech clean speech set and the WHAM noise set, and each speech contains only two random speakers, of which 13,900, 3,000, and 3,000 are used for model training, verification, and testing. For both the WHAM! and Libri2Mix datasets, the noisy mixed speech signal must first be passed through the convolution 0 layer and four cascaded convolution 1 layers at the front end of the encoder to obtain the preliminary feature representation of the speech signal, and then the features need to be processed in depth and finely through the subsequent layers of the encoder.

[0109] The subsequent layer of step 1 of the present invention specifically includes:

[0110] Step 1-1: The time-domain noisy mixed speech signal passes through the convolution layer 0, and undergoes a one-dimensional convolution with a convolution kernel size of L / 2=16 and a stride of S=L / 2 to obtain approximately stable and overlapping speech segments s mix-k , the number of output channels is C 0 =512, and use PReLU to obtain the nonlinear expression ability of features;

[0111] Step 1-2: Segment the speech signal s mix-k Furthermore, four cascaded and parameter-shared convolution layers are used to reduce model training parameters while further improving the expressiveness of speech features.

[0112] Step 1-3: The speech nonlinear feature map output after one convolution 0 layer and four convolution 1 layers is passed through the time-series-aware context channel attention layer to screen features in its output channel dimension, filter out redundant features, and improve the effectiveness of features that are more useful for separation;

[0113] Step 1-4: The output result after feature screening based on the attention mechanism in the channel dimension enhances the global modeling ability of the speech separation system for the speech signal. This step is mainly achieved through the bidirectional LSTM-Transformer encoder layer, and finally obtains an effective latent feature representation F of the speech signal with certain global context information. Encoder .

[0114] In the above steps 1-2 of the present invention, the specific method for implementing four cascaded and parameter-shared convolution layers 1 includes the following steps:

[0115] Step 1-2-1: A single convolution layer consisting of a 1D convolution with an expanded convolution function and a nonlinear activation function PReLU is used to improve the nonlinear expression ability of speech feature mapping. The number of 1D convolution input channels, the number of output channels, the convolution kernel size, the convolution kernel stride, the expanded convolution expansion rate, and the convolution padding number are 512, 512, 3, 1, 1, and 1 respectively;

[0116] Step 1-2-2: Cascade four convolutional layers and share parameters among the four layers.

[0117] In the above steps 1-3 of the present invention, the specific method for implementing the time sequence aware context channel attention layer includes the following steps:

[0118] Step 1-3-1: Add timing-aware features to the output of step 1-2 above;

[0119] Step 1-3-2: Introduce contextual relationships into the output of step 1-3-1 above to provide reasonable channel weight allocation for the subsequent channel attention module;

[0120] Step 1-3-3: Perform feature weighting and screening on the channel dimension for speech features with temporal perception context.

[0121] In the above step 1-3-1 of the present invention, the specific method for realizing speech feature representation with time sequence perception characteristics includes the following steps: Step 1-3-1-1: average pooling the last dimension of the input feature to 1, retaining only the time dimension, and then fully learning the dynamic changes of the input signal in the time domain. This operation can be defined as:

[0122]

[0123] Where N is the number of dimensions of the last dimension of the input feature, x t and t Respectively represent the features at the tth time point, f gap stands for global average pooling;

[0124] Step 1-3-1-2: Perform channel transformation through 1×1 convolution on the result that only retains the time dimension;

[0125] Step 1-3-1-3: ReLU activation to provide nonlinear expression capabilities;

[0126] Step 1-3-1-4: 1×1 convolution again learns the time domain characteristics of speech, and calculates the weights through the Sigmoid function as the temporal perception of the input features;

[0127] Step 1-3-1-5: Perform element-wise multiplication of the learned time-series-aware weights of the input features with the input features to add global time-series-aware features to the time domain signal.

[0128] In the above step 1-3-2 of the present invention, the specific method for realizing the speech feature with time sequence perception context comprises the following steps:

[0129] Step 1-3-2-1: Increase the number of channels by performing 1×1 convolution on the two-dimensional output of step 1-3-1-5 above;

[0130] Step 1-3-2-2: Merge the last two dimensions output from the previous step;

[0131] Step 1-3-2-3: 1×1 convolution compresses the channel dimension of the two-dimensional output result of the above step 1-3-2-2 and calculates the context feature;

[0132] Step 1-3-2-4: Apply the Softmax function to calculate the mask of the context feature;

[0133] Step 1-3-2-5: Perform element-wise multiplication of the context feature and the context mask to obtain speech features with temporal-aware context.

[0134] In the above step 1-3-3 of the present invention, the specific method for implementing feature weighting and screening of speech features with temporal perception context in the channel dimension includes the following steps: Step 1-3-3-1: 1×1 convolution transforms the number of channels of the output result of the above step 1-3-2-5;

[0135] Step 1-3-3-2: Layer normalization enhances speech features and the stability of model training;

[0136] Step 1-3-3-3: Apply 1×1 convolution again to transform the number of channels;

[0137] Step 1-3-3-4: The Sigmoid function estimates the attention score in the channel dimension (i.e., the weight assigned to each channel) for the output result of the above step 1-3-1-5. Since the acquisition of the attention score involves the global context information of the speech, the allocation of channel weights is more reasonable and effective;

[0138] Step 1-3-3-5: Perform element-wise multiplication of the output of step 1-3-1-5 and its corresponding channel attention score in the channel dimension, thereby completing the effective screening of time-series-aware contextual speech features in the channel dimension.

[0139] In the above steps 1-4 of the present invention, the specific method for implementing the bidirectional LSTM-Transformer encoder layer includes the following steps:

[0140] Step 1-4-1: Use the multi-head attention mechanism of the Transformer encoder layer to enhance the long-distance dependency of the speech sequence in the output of step 1-3-3-5 above;

[0141] Step 1-4-2: Add the temporal perception context speech features output from step 1-3-3-5 to the result of step 1-4-1 and apply layer normalization to smooth the output, which helps to ensure the stability of model training and prevent gradient explosion and gradient disappearance;

[0142] Step 1-4-3: Use a feed-forward layer with a globally aware bidirectional LSTM to increase the global modeling and expression capabilities of speech features;

[0143] Step 1-4-4: Add the outputs of the above steps 1-4-2 and 1-4-3 and apply layer normalization, thereby obtaining the speech potential feature representation finally output by the encoder in the present invention for further learning by the separator.

[0144] In the above step 1-4-1 of the present invention, the specific method for implementing the multi-head attention mechanism includes the following steps: Step 1-4-1-1: define the number of heads h in the multi-head attention mechanism, where different heads are responsible for paying attention to different features and information angles of the input features, and capturing the global dependencies in the speech timing features, thereby improving the model's ability to understand and model the contextual relationships of speech signals;

[0145] Step 1-4-1-2: Each head calculates K (Key), Q (Query) and V (Value) about the input features. Different heads have different weight matrices for calculating K, Q and V, and their parameters are learnable.

[0146] Step 1-4-1-3: Softmax calculates the attention score of each head, and the attention score is multiplied by V to obtain the feature representation of the input feature with respect to attention;

[0147] Step 1-4-1-4: Concatenate the output results of multiple heads and multiply them with a parameter-learnable weight matrix to finally form a multi-head attention feature representation of the input features;

[0148] In the above step 1-4-3 of the present invention, the specific method for implementing the feedforward layer with a global-aware bidirectional LSTM includes the following steps: Step 1-4-3-1: copy the output of the above step 1-4-2, and reverse one of the copies according to the hidden time dimension, that is, in reverse time order;

[0149] Step 1-4-3-2: The speech features in the positive and reverse order of the hidden time dimension are respectively fed into the LSTM network, and the two outputs are concatenated to further enhance the global context expression and modeling capabilities of the multi-channel speech attention features output in step 1-4-1;

[0150] Step 1-4-3-3: ReLU activation and a linear layer further improve the nonlinear expression ability of features.

[0151] In step 2, the basic modules of the separator, feature aggregation and nonlinear mapping to obtain the separation mask are specifically explained:

[0152] Step 2-1: Apply layer normalization to the output of step 1 to improve the stability of model training, and then transform the number of channels by using 1×1 convolution;

[0153] Step 2-2: construct a basic module of the separator by cascading 1D convolutional layers and the temporal-aware contextual channel attention layer constructed in step 1-3;

[0154] Step 2-3: cascading the separator basic modules obtained in step 2-2 three times;

[0155] Step 2-4: Perform feature aggregation and nonlinear mapping on the relevant outputs in the three cascaded separator basic modules to obtain the speech separation mask;

[0156] Step 2-5: Combine the speech separation mask Mask with the encoder output feature F Encoder Multiply them together to get the potential feature estimate F corresponding to the two speaker's speech signals and the background noise signal Separator .

[0157] In the above step 2-2 of the present invention, the specific method for implementing the basic module of the separator includes the following steps: Step 2-2-1: constructing a 1-dimensional convolutional layer;

[0158] Step 2-2-2: Expand the 1D convolutional layer constructed in step 2-2-1 above by a factor of 2. m (where m = 0, 1, 2, 3, 4, 5, 6, 7) are cascaded in sequence, that is, the 1D convolutional layer is stacked eight times;

[0159] Step 2-2-3: Then send the output of the above step 2-2-2 to the time-series-aware context channel attention layer constructed in step 1-3, perform reasonable feature screening in the channel dimension and give the features certain context information, thereby completing the construction of the basic module of the separator.

[0160] In the above step 2-2-1 of the present invention, the specific method for implementing the 1D convolution layer includes the following steps: Step 2-2-1-1: 1×1 convolution performs channel transformation on the input to adapt to the calculation of the subsequent layer;

[0161] Step 2-2-1-2: PReLU activation and layer normalization enhance the nonlinearity of features and the stability of model training respectively;

[0162] Step 2-2-1-3: Send the output of step 2-2-1-2 to the deep convolution to reduce the amount of calculation and improve the efficiency of convolution kernel parameters;

[0163] Step 2-2-1-4: PReLU activation and layer normalization;

[0164] Step 2-2-1-5: Apply two parallel 1×1 convolutions to perform transformations with different numbers of channels on the output of step 2-2-1-4 above. One of the convolution outputs is element-wise added to the output of step 2-2-1-1, and finally two outputs of the 1D convolution layer are obtained.

[0165] In the above step 2-4 of the present invention, the specific method of performing feature aggregation and nonlinear mapping on the relevant outputs in the cascaded three separator basic modules to obtain the final speech separation mask includes the following steps: Step 2-4-1: Adding the outputs of the 1D convolution layer and the temporal perception context channel attention layer in the three cascaded separator basic modules to obtain the feature aggregation representation of the separator cascade module as a whole;

[0166] Step 2-4-2: PReLU performs nonlinear mapping on the aggregated features;

[0167] Step 2-4-3: 1×1 convolution adjusts the number of channels to accommodate the subsequent activation layer;

[0168] Step 2-4-4: Sigmoid calculates the masks corresponding to the two speakers' voices and the background noise in the mixed features respectively.

[0169] The specific steps of step 3 include:

[0170] Step 3-1: The output of step 2 above is passed through four cascaded transposed convolutional layers, whose convolution kernel size, stride, and dilation factor correspond to the four cascaded convolutional 1 layers of the encoder front end, and the parameters are not shared, and the feature representation of the time domain waveform of the real speech signal and background noise signal is gradually learned;

[0171] Step 3-2: Finally, after the transposed convolution corresponding to the front-end convolution layer 0 of the encoder, the separated two speaker speech time domain signals and the background noise time domain signal are obtained.

[0172] The specific steps of step 4 include:

[0173] Step 4-1: Define the overall loss function for speech separation.

[0174] Loss total =Loss SI-SNR +αLoss MR ,

[0175] Among them, Loss SI-SNR =-SI-SNR represents the scale-invariant signal-to-noise ratio loss function (SI-SNR), with the input being the separation signal and the corresponding target signal, which is used to reduce the gap between the speech estimation and background noise estimation of the two speakers and the real speaker speech and background noise. MR Corresponding to the multi-resolution (MR) loss, it aims to improve the quality of separated speech and more effectively remove the noise in the speaker's speech based on the consideration of both the time domain and frequency domain information of the speech. α is a manually adjustable hyperparameter used to adjust the contribution and influence of the multi-resolution loss on the performance of the speech separation system.

[0176] Step 4-2: Apply a permutation-invariant training strategy to reduce the dependence of the model separation performance on the speaker order, enhance the generalization ability of the model, and gradually reduce the overall loss of the speech separation algorithm through gradient descent.

[0177] In the above step 5 of the present invention, a specific method for evaluating the performance of a proposed end-to-end noisy speech separation method based on a channel attention mechanism and a Transformer includes the following steps: Step 5-1: Comparing an end-to-end noisy speech separation method based on a channel attention mechanism and a Transformer with an existing time-domain speech separation model to verify the overall performance of the present invention;

[0178] Step 5-2: Compare an end-to-end noisy speech separation method based on channel attention mechanism and Transformer with an end-to-end noisy speech separation method based on Transformer to verify the effectiveness of the time-aware contextual channel attention layer;

[0179] Step 5-3: Compare an end-to-end noisy speech separation method based on a channel attention mechanism and Transformer with an end-to-end noisy speech separation method based on a channel attention mechanism to verify the effectiveness of the bidirectional LSTM-Transformer encoder layer.

[0180] The embodiment of the present invention is based on the Pytorch deep learning framework, the deep learning acceleration hardware is a single NVIDIA Tesla P100 SXM2 graphics card, and the CPU RTF test uses Intel (R) Xeon (R) Silver 4214R CPU @ 2.40GHz. The original mixed speech is divided into speech segments with a length of 4s, and adjacent speech segments overlap by 2s (this can ensure that adjacent speech has certain timing and contextual association characteristics of the original complete speech). The number of training epochs is set to 200, and the initial learning rate is 0.001. If the performance of the model on the validation set does not decrease after two consecutive epochs, the learning rate of the next iteration is halved, and the minimum learning rate is 1e-8. The Adam optimizer is used to reasonably optimize the method of updating the learnable parameters using gradient descent.

[0181] The experimental evaluation indicators in the embodiments of the present invention mainly include perceptual evaluation of speech quality (PESQ), short-term objective intelligibility (STOI) and scale-invariant signal-to-noise ratio improvement (SI-SNRi).

[0182] First, the present invention focuses on comparing the effects of using different channel attention modules in the basic modules of the encoder and separator on the speech separation performance. These modules are respectively embedded in the encoder, separator, and the combination of the two in the overall speech separation process, and their impact on the speech separation quality is evaluated by three indicators: PESQ, STOI, and SI-SNRi. The three types of channel attention mainly include: basic channel attention, context-aware channel attention, and time-series-aware contextual channel attention module. The experimental results are shown in Table 1:

[0183] Table 1 Performance comparison of adding different channel attention mechanism modules in three positions of the overall separation framework

[0184]

[0185] From Table 1, we can see that with the improvement of the channel attention mechanism from basic channel attention to temporal-aware contextual channel attention, the scores of various indicators have shown significant improvement. The increase in PESQ score means that the model has enhanced its ability to simulate human auditory perception. The improvement in STOI score indicates that the model performs better in maintaining the intelligibility of speech transmission, while the increase in SI-SNRi score reflects the progress of the model in improving the target speech to noise ratio. At the encoder level, the temporal-aware contextual-aware channel attention module shows superior performance compared to the other two modules. This result suggests that the integration of temporal information is crucial to capturing the dynamic characteristics of speech signals, allowing the encoder to process speech information more accurately. At the separator level, the same module also shows the best performance, which shows that in the process of speech signal separation and recovery, the context-aware and temporal-aware characteristics have significant advantages in distinguishing target speech from background noise.

[0186] In addition, when the channel attention mechanism module is applied to both the encoder and the separator, the overall performance of the model reaches its peak. This phenomenon shows that the global optimization of the channel attention mechanism is crucial in the entire speech separation process. The synergy of the time-aware context-aware channel attention module in the encoding and separation stages significantly improves the processing quality of speech signals, which is reflected in the improvement of various evaluation indicators.

[0187] Finally, the present invention also focuses on analyzing the impact of the proposed bidirectional LSTM-Transformer encoder layer on separation performance. The importance of the Transformer encoder layer improved by LSTM is demonstrated through the comparison of experimental waveforms. In the model without LSTM improvement, the estimated speech signal is mixed with noise components, which appear as irregular fluctuations on the waveform, which not only affects the clarity of the speech, but also reduces the accuracy of the speech backend processing. In contrast, the signal waveform processed by the introduction of the bidirectional LSTM-Transformer encoder layer appears smooth and clean. This smoothness shows that the model effectively removes noise during processing and retains the main features of the speech.

[0188] In summary, the end-to-end noisy speech separation method based on the channel attention mechanism and Transformer proposed in this embodiment can significantly and effectively improve the performance of the noisy speech separation system from two main aspects: the channel attention method and its position selection in the separation framework and the bidirectional LSTM-Transformer encoder layer. The experimental results verify the effectiveness and rationality of the time-series-aware context channel attention layer and the bidirectional LSTM-Transformer encoder layer in screening features and improving the global context modeling capabilities of features. Compared with the existing speech separation methods based on deep learning, this embodiment has been improved in multiple measurement indicators, and to a certain extent, the adaptability and robustness of the separation algorithm to various background noises have been enhanced, which has a good learning and reference significance in practical applications.

[0189] The above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. An end-to-end noisy speech separation method based on channel attention mechanism and Transformer, characterized in that: The method comprises the following steps: Step 1: Input noisy mixed signal S mix It is composed of the superposition of the clean speech signals S1 and S2 of the two speakers and the environmental background noise n. First, the input noisy mixed signal is preprocessed by a convolution 0 layer and four cascaded and parameter-shared convolution 1 layers. Then, the preprocessing results are feature-filtered and given global context information using the time-series-aware context channel attention layer and the bidirectional LSTM-Transformer encoder layer to obtain the potential feature representation F that is effective for the speech signal and has a certain context modeling capability. Encoder ; Step 2: Three cascades of separator basic modules consisting of multiple cascaded 1D convolutional layers and temporal-aware contextual channel attention layers gradually process the features F. Encoder , generate the speech separation mask Mask, and finally represent the potential feature F of the mixed signal output by the mask Mask and the encoder Encoder Perform element-wise multiplication to separate the latent space features F corresponding to each speaker's speech and background noise in the mixed signal Separator ; Step 3: Feature representation F of the separator output Separator Finally, the decoder, which consists of four cascaded convolutional 1 layers and one convolutional 0 layer corresponding to the encoder front end, completes the mapping from the potential feature space to the real time domain speech signal space and background noise signal space. The decoder finally outputs the estimated three time domain signals, corresponding to the speech signals of the two speakers respectively. and background noise signal Step 4: Constrain and optimize the performance of the speech separation system by the overall loss function of speech separation; Step 5: Perform a performance evaluation on the proposed end-to-end noisy speech separation method based on channel attention mechanism and Transformer.

2. According to claim 1, the end-to-end noisy speech separation method based on channel attention mechanism and Transformer is characterized in that: The specific steps of step 1 include: Step 1-1: The time-domain noisy mixed speech signal passes through the convolution layer 0, and undergoes a one-dimensional convolution with a convolution kernel size of L / 2=16 and a stride of S=L / 2 to obtain a stable and overlapping speech segment s mix-k , the number of output channels is C0=512, and PreLU is used to obtain the nonlinear expression of features; Step 1-2: Segment the speech signal s mix-k Through four cascaded and parameter-shared convolution layers, the model training parameters are reduced while the expressiveness of speech features is improved; Step 1-3: The speech nonlinear feature map output after one convolution 0 layer and four convolution 1 layers is passed through the time-series-aware context channel attention layer to screen features in its output channel dimension, filter out redundant features, and improve the effectiveness of features useful for separation; Step 1-4: The output result after feature screening based on the attention mechanism in the channel dimension enhances the global modeling ability of the speech separation system for the speech signal, and finally obtains the effective latent feature representation F of the speech signal with global context information through the bidirectional LSTM-Transformer encoder layer. Encoder .

3. According to claim 2, the end-to-end noisy speech separation method based on channel attention mechanism and Transformer is characterized in that: In the steps 1-3, the specific method of implementing the time-series-aware context channel attention layer includes the following steps: Step 1-3-1: adding a timing-aware feature to the output of step 1-2; Step 1-3-2: Introduce contextual relationships into the output of step 1-3-1 to provide reasonable channel weight allocation for the subsequent channel attention module; Step 1-3-3: weight and filter the speech features with temporal perception context in the channel dimension; In step 1-3-1, the specific method for realizing the speech feature representation with time sequence perception characteristics comprises the following steps: Step 1-3-1-1: Average pool the last dimension of the input feature to 1, retaining only the time dimension, and then fully learn the dynamic changes of the input signal in the time domain. This operation is defined as: Where N is the number of dimensions of the last dimension of the input feature, x t and t Respectively represent the features at the tth time point, f gap stands for global average pooling; Step 1-3-1-2: Perform channel transformation through 1×1 convolution on the result that only retains the time dimension; Step 1-3-1-3: ReLU activation to provide nonlinear expression capabilities; Step 1-3-1-4: 1×1 convolution again learns the time domain characteristics of speech, and calculates the weights through the Sigmoid function as the temporal perception of the input features; Step 1-3-1-5: Perform element-wise multiplication of the learned time-series-aware weights of the input features with the input features to add global time-series-aware features to the time domain signal; In step 1-3-2, the specific method for realizing the speech feature with time sequence perception context comprises the following steps: Step 1-3-2-1: Increase the number of channels of the two-dimensional output of step 1-3-1-5 through 1×1 convolution; Step 1-3-2-2: Merge the last two dimensions output from the previous step; Step 1-3-2-3: 1×1 convolution compresses the channel dimension of the two-dimensional output result of step 1-3-2-2 and calculates the context feature; Step 1-3-2-4: Apply the Softmax function to calculate the mask of the context feature; Step 1-3-2-5: perform element-wise multiplication of the context feature and the context mask to obtain speech features with temporal perception context; In step 1-3-3, the specific method for implementing feature weighting and screening of speech features with temporal perception context in the channel dimension includes the following steps: Step 1-3-3-1: 1×1 convolution transforms the number of channels of the output result of step 1-3-2-5; Step 1-3-3-2: Layer normalization enhances speech features and the stability of model training; Step 1-3-3-3: Apply 1×1 convolution again to transform the number of channels; Step 1-3-3-4: Sigmoid function estimates the attention score in the channel dimension for the output result of step 1-3-1-5, that is, the weight assigned to each channel; Step 1-3-3-5: perform element-wise multiplication of the output of step 1-3-1-5 and its corresponding channel attention score in the channel dimension, thereby completing the effective screening of time-series-aware contextual speech features in the channel dimension.

4. According to claim 2, the end-to-end noisy speech separation method based on channel attention mechanism and Transformer is characterized in that: In steps 1-4, the specific method for implementing the bidirectional LSTM-Transformer encoder layer includes the following steps: Step 1-4-1: Enhance the long-distance dependency of speech timing in the output result of step 1-3-3-5 through the multi-head attention mechanism of the Transformer encoder layer; Step 1-4-2: Add the temporal perception context speech features outputted from step 1-3-3-5 to the result of step 1-4-1 and apply layer normalization, which helps to ensure the stability of model training and prevent gradient explosion and gradient disappearance while smoothing the output; Step 1-4-3: Use a feed-forward layer with a globally aware bidirectional LSTM to increase the global modeling and expression capabilities of speech features; Step 1-4-4: Add the outputs of step 1-4-2 and step 1-4-3 and apply layer normalization to obtain the speech potential feature representation F of the encoder's final output Encoder , for the separator to learn; In step 1-4-1, the specific method for implementing the multi-head attention mechanism includes the following steps: Step 1-4-1-1: Define the number of heads h in the multi-head attention mechanism. Different heads are responsible for focusing on different features and information angles of the input features, and capturing the global dependencies in the speech timing features, improving the model's understanding and modeling capabilities of the contextual relationships of speech signals; Step 1-4-1-2: Each head calculates K (Key), Q (Query) and V (Value) about the input features. Different heads have different weight matrices for calculating K, Q and V, and their parameters can be learned. Step 1-4-1-3: Softmax calculates the attention score of each head, and the attention score is multiplied by V to obtain the feature representation of the input feature with respect to attention; Step 1-4-1-4: Concatenate the output results of multiple heads and multiply them with a parameter-learned weight matrix to finally form a multi-head attention feature representation of the input features; In step 1-4-3, the specific method for implementing the feedforward layer with global-aware bidirectional LSTM includes the following steps: Step 1-4-3-1: Copy the output of step 1-4-2, and reverse one of the copies according to the hidden time dimension, i.e., in reverse time order; Step 1-4-3-2: The speech features in the positive and reverse order of the hidden time dimension are respectively fed into the LSTM network, and the two outputs are concatenated to enhance the global context expression and modeling capabilities of the multi-channel speech attention features output in step 1-4-1; Step 1-4-3-3: ReLU activation and a linear layer improve the nonlinear expression ability of features.

5. According to claim 1, the end-to-end noisy speech separation method based on channel attention mechanism and Transformer is characterized in that: In step 2, the specific steps of implementing the separator include: Step 2-1: Apply layer normalization to the output of step 1 to improve the stability of model training, and then transform the number of channels by using 1×1 convolution; Step 2-2: construct a basic module of the separator by multiple cascaded 1D convolutional layers and the temporal-aware context channel attention layer constructed in step 1-3; Step 2-3: cascading the separator basic modules obtained in step 2-2 three times; Step 2-4: Perform feature aggregation and nonlinear mapping on the relevant outputs in the three cascaded separator basic modules to obtain the speech separation mask; Step 2-5: Combine the speech separation mask Mask with the encoder output feature F Encoder Multiply them together to get the potential feature estimate F corresponding to the two speaker's speech signals and the background noise signal Separator .

6. According to claim 5, the end-to-end noisy speech separation method based on channel attention mechanism and Transformer is characterized in that: In step 2-2, the specific method for implementing the separator basic module includes the following steps: Step 2-2-1: Construct a 1D convolutional layer; Step 2-2-2: dilate the 1D convolutional layer constructed in step 2-2-1 by a factor of 2 m , where m = 0, 1, 2, 3, 4, 5, 6, 7, are cascaded in sequence, that is, the 1D convolutional layer is stacked eight times; Step 2-2-3: Then, the output of step 2-2-2 is sent to the time-series-aware context channel attention layer constructed in step 1-3, reasonable feature screening is performed in the channel dimension, and certain context information is given to the features, so as to complete the construction of the basic module of the separator; In step 2-2-1, the specific method for implementing the 1D convolutional layer includes the following steps: Step 2-2-1-1: 1×1 convolution performs channel transformation on the input to adapt to the calculation of subsequent layers; Step 2-2-1-2: PReLU activation and layer normalization are used to enhance the nonlinearity of features and the stability of model training respectively; Step 2-2-1-3: Send the output of step 2-2-1-2 to deep convolution to reduce the amount of calculation and improve the efficiency of convolution kernel parameters; Step 2-2-1-4: PReLU activation and layer normalization; Step 2-2-1-5: Apply two parallel 1×1 convolutions to perform transformations of different numbers of channels on the output of step 2-2-1-4, where one of the convolution outputs is element-wise added to the output of step 2-2-1-1, and finally two outputs of the 1D convolution layer are obtained.

7. According to claim 5, the end-to-end noisy speech separation method based on channel attention mechanism and Transformer is characterized in that: In the step 2-4, the specific method of performing feature aggregation and nonlinear mapping on the relevant outputs in the three cascaded separator basic modules to obtain the final speech separation mask includes the following steps: Step 2-4-1: Add the outputs of the 1D convolutional layer and the temporal-aware context channel attention layer in the three cascaded separator basic modules to obtain the feature aggregation representation F of the entire separator cascade module sum ; Step 2-4-2: PReLU on the aggregated feature F sum Perform nonlinear mapping; Step 2-4-3: 1×1 convolution adjusts the number of channels to accommodate the subsequent activation layer; Step 2-4-4: Sigmoid calculates the mixed features F output by the encoder respectively Encoder The masks corresponding to the two speakers’ speech and background noise.

8. According to claim 1, the end-to-end noisy speech separation method based on channel attention mechanism and Transformer is characterized in that: The specific steps of step 3 include: Step 3-1: Separate the feature F output from step 2 Separator After four cascaded transposed convolutional layers, the convolution kernel size, stride, and dilation factor correspond to the four cascaded convolutional 1 layers at the front end of the encoder, and the parameters are not shared, so as to learn the feature representation that conforms to the time domain waveform of the real speech signal and background noise signal; Step 3-2: Finally, after the transposed convolution corresponding to the front-end convolution layer 0 of the encoder, the separated two speaker speech time domain signals and the background noise time domain signal are obtained.

9. According to claim 1, the end-to-end noisy speech separation method based on channel attention mechanism and Transformer is characterized in that: The specific steps of step 4 include: Step 4-1: Define the overall loss function for speech separation. Loss total =Loss SI-SNR +αLoss MR , Among them, Loss SI-SNR =-SI-SNR represents the scale-invariant signal-to-noise ratio loss function. The input is the separation signal and the corresponding target signal. It is used to reduce the gap between the speech estimation and background noise estimation of the two speakers and the real speaker speech and background noise. Loss MR Corresponding to multi-resolution loss, the quality of separated speech is improved and the noise in the speaker's speech is more effectively removed based on the time domain and frequency domain information of speech. α is a manually adjusted hyperparameter used to adjust the contribution and influence of multi-resolution loss on the performance of the speech separation system. Step 4-2: Apply a permutation-invariant training strategy to reduce the model separation performance’s dependence on speaker order, enhance the model’s generalization ability, and reduce the overall loss of the speech separation algorithm through gradient descent.

10. According to claim 1, the end-to-end noisy speech separation method based on channel attention mechanism and Transformer is characterized in that: In step 5, a specific method for evaluating the performance of the proposed end-to-end noisy speech separation method based on the channel attention mechanism and Transformer includes the following steps: Step 5-1: Compare an end-to-end noisy speech separation method based on channel attention mechanism and Transformer with the existing time-domain speech separation model to verify its overall performance; Step 5-2: Compare an end-to-end noisy speech separation method based on channel attention mechanism and Transformer with an end-to-end noisy speech separation method based on Transformer to verify the effectiveness of the time-aware contextual channel attention layer; Step 5-3: Compare an end-to-end noisy speech separation method based on a channel attention mechanism and Transformer with an end-to-end noisy speech separation method based on a channel attention mechanism to verify the effectiveness of the bidirectional LSTM-Transformer encoder layer.

Images